1. Post #1
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    If you don't know what a pointer is, or what the heap is, don't bother with this thread.

    So yea, this module lets you allocate and deallocate memory from the heap.

    Download: http://www.fortfn.co.uk/files/gm_heap.dll
    Source: http://pastebin.com/Nm4tdyJn

    Usage:

    local ptr = heap.Allocate( 10 ) -- Allocate 10 bytes
    heap.Set( ptr, 0, 10 ) -- Set all of our newly allocated memory to 0
    for i=1, 10 do
        heap.SetByte( ptr+i - 1, i ) -- Set each byte to a value
    end
    for i=1, 10 do
        print( heap.GetByte( ptr+i - 1 ) ) -- Print it
    end
    
    local ptr2 = heap.Allocate( 10 ) -- Allocate another 10 bytes
    heap.Set( ptr2, 0, 10 ) -- Set all of our newly allocated memory to 0
    heap.Copy( ptr, ptr2, 5 ) -- Copy 5 bytes across
    
    print( "---" )
    
    heap.Deallocate( ptr ) -- Deallocate our original 10 bytes
    for i=1, 10 do
        print( heap.GetByte( ptr2+i - 1 ) ) -- Print it
    end
    
    heap.Deallocate( ptr2 ) -- Deallocate our last 10 bytes

    Code:
    1
    2
    3
    4
    5
    6
    7
    8
    10
    ---
    1
    2
    3
    4
    5
    0
    0
    0
    0
    0
    Now of course, using a module for storing an array of 10 bytes is completely stupid, but suppose you want to store an array of 1,000,000 bytes (~1 mb of data). Lua will not offer you the speed to manage, access and set the data in a table of such size but this module should do.
    Reply With Quote Edit / Delete Reply United Kingdom Show Events Useful Useful x 5Dumb Dumb x 4 (list)

  2. Post #2
    Box collector
    haza55's Avatar
    October 2005
    545 Posts
    Can we get perf times for both a 1mb table(4 bytes per element) and 1mb of data with your module.

    And perhaps 10mb, 100mb and 500mb tests.

    Edited:

    Also you need to encapsulate the pointers, so the do bounds checking or something.

  3. Post #3
    Gold Member
    Gbps's Avatar
    December 2008
    3,636 Posts
    This could be very unsafe if it wanted to be.
    Reply With Quote Edit / Delete Reply United States Show Events Agree Agree x 4Dumb Dumb x 2 (list)

  4. Post #4
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    Awsome module, but i cant find any use for it

    Edited:

    Actaully lets store the players data in the heap! :pseudo:

  5. Post #5
    Gold Member
    LuaStoned's Avatar
    September 2007
    1,351 Posts
    As haza said we need proper testing before :)

  6. Post #6
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    It's only as unsafe as the scripts which use it. And I'll do some speed tests later.

  7. Post #7
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    I will use this for storing items in my gamemode, as i will have around.... 5k+ items placed on the map it will be a good test of this

    Edited:

    Deallocating seems to take about... forever on 1mb data

  8. Post #8
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    I did some basic testing.

    require( "heap" )
    
    local B = 1
    local KB = B * 1024
    local MB = KB * 1024
    
    local cnt = 10 * MB
    
    local function cmdTestLuaTable( ply, com, args )
    	debug.sethook()
    	local start = os.time()
    	local tbl = {}
    	for i=1, cnt do
    		tbl[i] = i % 255
    	end
    	local o = 0
    	for i=1, cnt do
    		o = o + tbl[i]
    	end
    	print( o )
    	print( "Lua table took " .. (os.time()-start) .. " seconds to test!" )
    end
    concommand.Add( "heap_testluatable", cmdTestLuaTable )
    
    local function cmdTestHeapTable( ply, com, args )
    	debug.sethook()
    	local start = os.time()
    	local ptr = heap.Allocate( cnt )
    	local Set = heap.SetByte
    	local Get = heap.GetByte
    	for i=1, cnt do
    		Set( ptr + i - 1, i % 255 )
    	end
    	local o = 0
    	for i=1, cnt do
    		o = o + Get( ptr + i - 1 )
    	end
    	print( o )
    	heap.Deallocate( ptr )
    	print( "Heap array took " .. (os.time()-start) .. " seconds to test!" )
    end
    concommand.Add( "heap_testarray", cmdTestHeapTable )

    Code:
    ] heap_testluatable 
    not enough memory
    ] heap_testarray 
    1331684080
    Heap array took 5 seconds to test!
    Seems like the limit for the amount of data to be stored in a lua table is between 1mb and 10mb. Both loops cause Garry's infinite loop detector to be triggered off (hence debug.sethook). I guess there's no way to make looping ~1,000,000,000 times in Lua any faster (without LuaJIT). So I think this module is only good if you need to store ALOT of memory and need to access it fast (like mine and overv's minecraft in gmod projects).

    I'm not gonna bother going higher than 10mb since obviously you can't test lua tables, and I'd get bored waiting.

  9. Post #9
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    I just got a hl2.exe error after testing that one, its pretty unstable, i think its the deallocating

  10. Post #10
    Gold Member
    Gbps's Avatar
    December 2008
    3,636 Posts
    I hope you don't mind, but I was already working on a module that will probably depreciate this, so don't be offended if it does :saddowns:.

  11. Post #11
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    I hope you don't mind, but I was already working on a module that will probably depreciate this, so don't be offended if it does :saddowns:.
    Go right ahead, my implementation is pretty crappy to be fair.

    Edited:

    I just got a hl2.exe error after testing that one, its pretty unstable, i think its the deallocating
    Can you show me your code?

  12. Post #12
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    All i did was some heap.Allocate some heap.SetByte and some heap.Deallocate, it crashes when you Dealloc it, your example crashes too, i think its your de-allocation that crash it

  13. Post #13
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    It could be, but my example works fine on my PC. Are you testing on a win32 machine?

  14. Post #14
    Box collector
    haza55's Avatar
    October 2005
    545 Posts
    You should use malloc and free. new and delete are probally performing rtti checks.
    Reply With Quote Edit / Delete Reply Australia Show Events Agree Agree x 1 (list)

  15. Post #15
    Gold Member
    AzuiSleet's Avatar
    September 2007
    758 Posts
    This module seems rather useless, because unless you're going to read a value from memory, Lua can do the exact thing. It abstracts the ability to call malloc/free by giving you garbage collected values like tables.

    If you want to store 5000 items use a table with unique ids as keys. It will be many times faster than any system you could possibly create with the scope of this module. If you're worried about memory constraints, move it into SQL and memcached so you can cache item data and anything else.

    Also, as for your benchmark about not being able to allocate the table, it's because of the way tier0 handles memory allocation. It's not optimized for the way Lua works, which was one of the issues being investigated. (Many allocations of small objects will take up larger buckets, large allocations will fail because of this. The bucket system isn't exactly suited for Lua)
    Reply With Quote Edit / Delete Reply United States Show Events Agree Agree x 1 (list)

  16. Post #16
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    This is better in some cases when lua runs out of memory, its just that its.... crashy

  17. Post #17
    Box collector
    haza55's Avatar
    October 2005
    545 Posts
    Yeap, the source engine mallocs hunks(heap chunk). It then deals out small proportions of that memory. This is generally faster because we don't have to do a syscall, wait for the operating system to allocate a virtual block of memory and return(and all the extra descriptors placed on that memory etc etc).

  18. Post #18
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    This module seems rather useless, because unless you're going to read a value from memory, Lua can do the exact thing. It abstracts the ability to call malloc/free by giving you garbage collected values like tables.

    If you want to store 5000 items use a table with unique ids as keys. It will be many times faster than any system you could possibly create with the scope of this module. If you're worried about memory constraints, move it into SQL and memcached so you can cache item data and anything else.

    Also, as for your benchmark about not being able to allocate the table, it's because of the way tier0 handles memory allocation. It's not optimized for the way Lua works, which was one of the issues being investigated. (Many allocations of small objects will take up larger buckets, large allocations will fail because of this. The bucket system isn't exactly suited for Lua)
    It's not useless when you want to store 250,000+ items.

  19. Post #19
    Ningaglio's Avatar
    February 2009
    938 Posts
    Cool, this is just like ( warning )     the cheat engine     and you can ( Warning )     speed hack by changing the speed values and update timers     but sadly i think garry or valve fixed it :(:
    Reply With Quote Edit / Delete Reply France Show Events Dumb Dumb x 12 (list)

  20. Post #20
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    Cool, this is just like ( warning )     the cheat engine     and you can ( Warning )     speed hack by changing the speed values and update timers     but sadly i think garry or valve fixed it :(:
    What?

    If you want to use this to cheat, you still have to find the memory addresses of stuff you want to change and get them into lua. Using cheat engine would be easier than using this to cheat.

  21. Post #21
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    Even with free and malloc it crashes, im running 32 bit vista (probably vista fucks it up)

  22. Post #22
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    FINALLY i figured it out, you have to add a NULL at the end of the memory you allocated before you deallocate it or it will fuck up and unallocate a few thusand bytes before it comes to a NULL char

    Edited:

    Wait a little. doing more tests

    Edited:

    Confirmed

  23. Post #23
    Gold Member
    thomasfn's Avatar
    July 2008
    2,959 Posts
    FINALLY i figured it out, you have to add a NULL at the end of the memory you allocated before you deallocate it or it will fuck up and unallocate a few thusand bytes before it comes to a NULL char

    Edited:

    Wait a little. doing more tests

    Edited:

    Confirmed
    That's weird. Oh well. I started making a new "safe" version of this ages ago, but never got it finished.

  24. Post #24
    King of the Oil Refinery
    Tobba's Avatar
    December 2008
    6,212 Posts
    you can just specify the size of what it shall deallocate, like
    delete [Lua()->GetInterger(2)] ptrMem;