next »

[JWki]

did someone say "can I have a memory system that supports leak detection with IDE integration please"



Got a leak? Just double click the message and you'll know exactly where in your code the allocation came from.

[oahda]

I've barely used it, so I'm not sure, but I think the Xcode tools (Instruments?) can do that? o:

[JWki]

I've barely used it, so I'm not sure, but I think the Xcode tools (Instruments?) can do that? o:

Jep, there's plenty of external tools that can perform leak detection and the like, however this is an extremely lightweight solution that is simply part of the general allocator system.
And it doesn't require any bigger setup to enable than this:



Also I've waited for an opportunity to do this:



The advantage over external tools is that it works out of the box on all platforms, unlike most tools, and doesn't require any additional setup nor does it slow down the program execution (looking at you Valgrind). Also scales pretty much seamlessly from simple leak detection like in the screenshot to handing the user a full callstack of the program state at the point that the leaky allocation was made.
To give credit where it's due, the system is pretty much what Stefan Reinalter covers in his excellent blog at blog.molecular-matters.com.

[oahda]

Ooh. How does the allocator know that something is leaking tho, how does it know there's no delete to clean it up somewhere else later?

[JWki]

Ooh. How does the allocator know that something is leaking tho, how does it know there's no delete to clean it up somewhere else later?

Because you have to free any memory coming from a memory arena via that same memory arena. Raw new and delete are forbidden.
When I want constructor and destructor calls, I use simple macros (the same can be achieved using a simple template function but a macro does the job and is nicer in this case).

[oahda]

Yeah, I didn't necessarily mean the keyword delete, just the concept of deleting/freeing.

So are you figuring out somehow if a corresponding free is missing? If so, how are you doing that? If not, what am I not getting? :c

[JWki]

Yeah, I didn't necessarily mean the keyword delete, just the concept of deleting/freeing.

So are you figuring out somehow if a corresponding free is missing? If so, how are you doing that? If not, what am I not getting? :c

So, this code will result in the arena reporting unfreed memory at shutdown:

Code:

arena = &mallocArena;
for (int i = 0; i < 16; ++i) {
    Foo* foo = LC_NEW(Foo, arena)(i);
    //LC_DELETE(foo, arena);
}

Uncommenting the LC_DELETE, uhm, 'call', will satisfy the system.
The principle is simple, the arena internally keeps track of all allocations in some data structure - it adds an entry for each allocation that is removed when freeing the memory.
All that LC_NEW and LC_DELETE do is allocate memory from the arena that holds a Foo, and calling placement new respectively call the destructor and have the arena free the memory.

Internally, as shown in an earlier snippet, a memory arena is composed of various policy classes that implement the orthogonal functionalities of leak detection / memory tracking, tagging of allocations and freed memory, bounds checking, etc.
When you allocate memory via an arena, it will allocate enough space to hold debug information as dictated by the policy classes and also call the corresponding hooks for tracking.

In my current testing code for this, I lazily fill an array with allocation entries that contain information about the address the memory was allocated at, the size etc. When an allocation is freed, I mark the entry as dead. At shutdown, I iterate over all entries and output non-dead entries.

[oahda]

Ah, that makes sense. How come you're not using RAII or something instead? Smart pointers?

(and what does the LC stand for?)

[JWki]

Ah, that makes sense. How come you're not using RAII or something instead? Smart pointers?

(and what does the LC stand for?)

Well for starters this system is mainly to be able to plug in different allocation strategies - this means NOT using the general purpose allocator like new does for example because in a lot of situations there is a specialized allocation strategy that beats the crap out of the fastest malloc implementation. For example, EVERYTHING that you only need during a frame can be allocated using a linear allocator that is being reset every frame, configured to a reasonable maximum size. There's nothing faster than bumping a pointer to allocate some memory. It's like stack allocation except the stack can be half a gig if you want it to.
Essentially, knowing the allocation lifetime and the deallocation pattern (and in a game engine you should know in about 99 percent of the cases) allows you to super easy get magnitudes of performance increase compared to using malloc/new.

Obviously you can still use std:: smartpointers with a custom deleter, but in my experience it's a better strategy to clearly define ownership semantics for everything explicitly - game objects are owned by the game world, components are owned by systems, yadda yadda - this reduces failure points to a few clearly defined spots in the code, and holes in such a design are easily found using the leak detection measures that I have.
There's not a lot of places in my code where object lifetime is even a problem - most objects are high level stuff that lives on the stack and has application lifetime anyways. "Logical" objects (game objects, components, etc) with high granularity lifetime are referenced using handles and managed explicitly.

Oh and the LC is just the main namespace for the project so it's also the prefix for all the macros. Just found out that something in MSVC already uses that for macros tho so have to come up with a different one.

[oahda]

Yeah, it's also used for C localisation (LC_ALL et cetera) so was wondering if it had anything to do with that first.

Tried to read up on bumping. So you basically make your own "stack on the heap" by reserving a chunk of memory and then keep track of what's actually used use some kind of placement allocation to put stuff in there? Maybe allocate another chunk occasionally if you're running out of space?

[JWki]

Yeah, it's also used for C localisation (LC_ALL et cetera) so was wondering if it had anything to do with that first.

Tried to read up on bumping. So you basically make your own "stack on the heap" by reserving a chunk of memory and then keep track of what's actually used use some kind of placement allocation to put stuff in there? Maybe allocate another chunk occasionally if you're running out of space?

Yeah it works like a stack - you can either have individual frees (but only in exactly reversed order of how you allocated them) or just reset the whole stack at once.
The same way you can make all kinds of allocators - memory pools are amongst the better known ones, you take an allocated block of memory and subdivide them up into chunks of uniform size, keep track of them with an in-place linked list and get O(1) allocation and freeing, very useful for uniform size things (you can also use them for things with any size as long as it's less than the block size, but waste some memory).
They all come down to preallocating some memory, then managing how memory is allocated within these blocks.

[Ordnas]

Recently achieved a major milestone in a project I've been working on. Quite satisfying to get there as I've been working toward it for a while.

http://www.entropicsoftware.com/aie/sat/overview.html

Garthy, my compliments for the Satellite Chocolate Board! I like the "Use the temperature sensor to turn on/off an infrared-controlled airconditioner.", during the summer it could be very helpful! 

[Garthy]

Recently achieved a major milestone in a project I've been working on. Quite satisfying to get there as I've been working toward it for a while.

http://www.entropicsoftware.com/aie/sat/overview.html

Garthy, my compliments for the Satellite Chocolate Board!

Thanks for checking it out and for the kind words.

It's something I'm quite proud of as a combined electronics/software project. I'm a software engineer by profession but I've picked up a bit of electronics recently.

There were two parts to the milestone:

- The electronics side: Fully build the first prototype and make it work.
 
- The software side: Two things: (i) be able to write simple code that shows off the various components of the board; and (ii) get the supporting software to the point where you can write a led blinker (essentially "hello world" for electronics) and get it running on the board in under a minute.

And then write it all up and demonstrate it. I've been working toward this point for a while, and I finally made it.

ARM microcontrollers can be a bit tricky to use, so one of my goals was to come up with a library/system that would make it much easier to work with.

The led blinker might not sound like much, but it involves working with an ARM microcontroller and a Max V CPLD, both of which are very powerful but can be hard to use. The idea was to make it easy to use when using the library.

Video demo of the one-minute led blinker here:

http://www.entropicsoftware.com/aie/sat/one-minute-led-blinker.html

The other pages have demos of the rest of the board. Best starting point is here:

http://www.entropicsoftware.com/aie/sat/overview.html

I like the "Use the temperature sensor to turn on/off an infrared-controlled airconditioner.", during the summer it could be very helpful! 

Thanks. One piece of feedback I've received for the overview page was to list some possible projects that it could be used for. Since it's a general-purpose board it can be a little unclear as to the sorts of things that are possible. As in, how general is general-purpose?

Now all I need is an infrared-controlled airconditioner.

[oahda]

Wrote a JSON parser as a simple enough challenge to test out my understanding of context-free grammars and making a lexer and parser according to the rules of the grammar that I wrote based on the JSON spec. It's so much easier to parse something once the grammar has already been outlined with such notation. Now maybe in the future I have a chance of successfully writing a parser/compiler for a programming language without too many headaches, should I feel like it!

[ProgramGamer]

Cool!

I've been reading up on WASM recently to see what it's about, and it's really interesting. The current pipeline looks like it was meant to allow asm.js to run faster, but you can just compile any language to it with the right setup, so it looks like there's a lot more potential for doing cool stuff with it. Inb4 they make a WASM build target for Game Maker lol

Plus, JavaScript Sucks^TM

[oahda]

It's still to weird to me that WASM/asm.js is even a thing… But I guess that's what powers Emscripten, so I'm glad!

[ProgramGamer]

Something about compiling directly to the bytecode of a VM that ships with every major web browser seems really appealing to me, especially as a game developer. I kinda want to learn to make a compiler to it. I don't know in or for what language, but just making a compiler for WASM sounds cool to me.

I don't know, maybe I'm just crazy lol

[oahda]

It does make sense when you put it like that!

Surely there must already exist one, tho?

[ProgramGamer]

Yeah, you can already compile asm.js code to WASM bytecode and run it in all major browsers, but I wouldn't want to program in javascript for that. They do have a Unity demo running a simple tank game on their website though.

Link

[oahda]

What would you prefer to use?