Holy shit that was the most annoying bug. I had the game crashing, cause being an invalid pointer (0xDDDDDDDD) in vstdlib.dll. I wasn't calling any functions from there or even linking it. The crash always happened after one of my functions returned. Otherwise it was completely random. It would happen in the menu, when I pressed a button or after a few seconds in-game. Sometimes it never happened.
I spent over an hour trying to debug the crash, did some tests and found out that it stopped when I commented out calls to my Lua hooks. I suspected the reason to be either luabind or the hook module not existing. However, it started again later, after I had rewritten the Lua hooks in C++. Again, I spent some time debugging, when it disappeared completely. I added the hooks back, played around with the code and the game didn't crash at all. Couple hours ago the crashing came back.
Well, this time I was determined to find the cause. I took out IDA, attached the debugger to GMod and launched. The crash was indeed in vstdlib.dll, the function was trying to use the invalid pointer. It was a virtual method of some class. I checked the vtable for hints, found "VCvarQuery001". I compared it to the vstdlib's Mac version (which contains debug info), and found out that the guilty method was ICvar::FindCommandBase. This was really helpful and I managed to fix the bug. The cause?
local lua_run = cvar.ConCommand( "ir_lua_run", lua_run_c )
local lua_open = cvar.ConCommand( "ir_lua_open", lua_open_c )
Can you spot it? It took me a while.
I'm creating the ConCommands as locals. They get added to a linked list when created. The linked list is traversed by ICvar::FindCommandBase. Yep, the garbage collector got the concommands, Luabind deleted the C++ objects, but the destructor doesn't automatically remove it from the linked list.
The fix was simple, I just added a destructor to the ConCommand wrapper that unregisters the command from the list.