next »

[ferreiradaselva]

Working on a renderer. Hierarchical bone animation seems to be in good shape:



Next up is model loading so I don't have to type in all those vertices, joints, weights, normals, etc. by hand...

Very good job, Bender Bending Rodríguez.

Hi, just discovered this thread and read back a few pages.

Welcome to the hidden nerd club.

[Ordnas]

Hi, just discovered this thread and read back a few pages.

Jumping in to the middle of the conversation, Ill say that I'm a fan of stl, but hate the syntax. To get around this, I created a wrapper class that accesses the stl functions. The initial motivation behind this was so I could easily adapt to anyone's memory system. All I need to do is modify the wrapper to adapt to the new system. In use, I always access my wrapper class and not stl directly.

I've been working on a destruction engine which I'd like to adapt to other systems so the approach has worked well.



Cheers,
H
 

I do not know if wrapping STL just for changing the syntax it is something I would do. Personally I am not convinced about the trade-off to add an other layer of abstraction and increase the probability of making an human mistake, also the library is platform-indipendent so I would not have any beneficial. Just my personal idea, eh. Probably other programmers are doing STL-wrappers and do not have any problem at all  

[popawheelie]

Hi, just discovered this thread and read back a few pages.

Jumping in to the middle of the conversation, Ill say that I'm a fan of stl, but hate the syntax. To get around this, I created a wrapper class that accesses the stl functions. The initial motivation behind this was so I could easily adapt to anyone's memory system. All I need to do is modify the wrapper to adapt to the new system. In use, I always access my wrapper class and not stl directly.

I've been working on a destruction engine which I'd like to adapt to other systems so the approach has worked well.




Cheers,
H
 

I do not know if wrapping STL just for changing the syntax it is something I would do. Personally I am not convinced about the trade-off to add an other layer of abstraction and increase the probability of making an human mistake, also the library is platform-indipendent so I would not have any beneficial. Just my personal idea, eh. Probably other programmers are doing STL-wrappers and do not have any problem at all  

Agree. I do not believe in adding unnecessary abstraction either. However in this case, I wanted to be able to easily adapt to anyone else's memory system. That not necessarily being STL.

[JWki]

Care to outline how that wrapper works? I'm not certain I'm understanding the purpose.

[popawheelie]

Care to outline how that wrapper works? I'm not certain I'm understanding the purpose.

Sure. Here is a snippet of code:

template<typename Type>
class BxArray
{
public:
   BxArray() { }
   BxArray( BxSIZET Size )              { mArray.resize( Size ); }
   BxArray( BxSIZET Size, Type InitVal) { mArray.resize( Size, InitVal ); }
   ~BxArray() { }

   void append( const Type& data )  { mArray.push_back( data ); }
   void clear()                         { mArray.clear(); }
   Type& operator [] ( BxSIZET index )  { return mArray[ index ]; }

private:

   std::vector<Type> mArray;
};

Of course this class is a lot bigger, and includes a lot of other functions, but you should get the idea. The class is nothing more than an abstraction to std::vector.

I create some typedefs to simplify further:

typedef BxArray<short>              BxShortArray;
typedef BxArray<unsigned short>     BxUShortArray;
typedef BxArray<int>                BxIntArray;

I use these types through out the code. Now if I ever want to use another memory system, all I need to do is rewrite this class. Rest of the code is untouched. If you plan on never using anything other than stl, than this is probably all mute.

The system Im writing is designed to piggyback on other game engines. If the parent game engine has its own memory manager, than I can easily adapt to it and use it.

[Garthy]

I will write a wrapper around anything if I feel it results in an improvement, or makes it a better fit for the system I am working on.

I tend to make heavy use of external libraries and have found that they tend to be constructed with different methodologies depending on the preferences of their authors. Sometimes an established library is old or has a genuinely bad interface, and a wrapper can hide this. Sometimes certain operations are problematic or must be called with certain caveats- in this case enforcing the required conditions through a wrapper can protect against undefined behaviour.

In my own projects, I have happily abstracted over even std::vector. So even the STL isn't off-limits if I think there is a possible improvement. For std::vector, personally, I find operator[] having undefined behaviour when called with an out-of-range value to be a poor design decision, especially when the undefined behaviour is almost always reading raw memory before or after the vector, and exceptions are an appropriate mechanism for reporting such failures. This behaviour results in errors that can go unseen for a time and be a nightmare to debug. I override operator[] to call at() instead. I add a separate call for when I definitely want a raw call. I can then selectively use the raw call when I am optimising a region of code and have confirmed each access will be within range.

I could just call at() all the time instead, but I am compelled to ask: Why should I? The flaw, I feel is in the existing interface. I fix the flaw, rather than tolerate it.

Similarly, std::list doesn't provide a convenient access or delete by index. I completely understand that it is inefficient, access is O(N), loops are O(N^2), and that you really don't want to do this without a good reason. However, when I do have a good reason, I'm not happy being left to write my own call to do it or writing the code using the existing mechanisms by hand each time as some sort of perverse punishment for daring to need to solve a problem best solved by indexing into a list. For what it's worth, I can't remember the exact problem I was trying to solve at the time, but it involved maintaining data which required efficient high-frequency insertions and erasures, but also infrequent accesses and deletions by index that could be considerably slower. At the time I remember laughing that I had finally found a legitimate need to index into a list, but it ceased being funny once I discovered the interface was not going to allow me to do that without additional effort due to a deliberate design decision. In this case though, an external function would probably may have been appropriate, since it is a fairly uncommon case in general, and the burden for the code being less clear might have been acceptable. I think I wrote the wrapper class at the time though as the situation had ticked me off.

Naturally though, I'll adapt to the environment I'm working in. I understand that not everyone feels the same way on these issues. I haven't worked on an external codebase that wraps std::vector as I do, so I know it is uncommon.

Anyway, these are just my personal thoughts and preferences. I understand that it is all a tradeoff and that others may weigh up the exact same concerns and come to the opposite conclusion.

[qMopey]

Problem I have with these kinds of wrappers is two things. First is dependencies. A codebase utilizing these wrappers is harder to move, causing lots of problems during code maturation. Sometimes it is desireable to change code’s environment because the code solves a useful problem. As time goes on, maybe it needs to move into a new game engine, or out of an engine into an editor. Maybe it needs to live inside of a 3rd party plugin, or even on the cloud. Minimizing dependencies is critical as code becomes older, simply to make the code easier to work with, and less likely to break.

Second is API degredation. As one API layer wraps another information is inevitably lost. The wrapper will in some way lop multiple error codes into one, or suppress warnings. The wrapper will always hide certain code paths, or add on extraneous ones. It’s just how it goes; typically wrappers fail in one or more ways, and the more layers the more likely little pieces will break off and be forgotten.

In the end the decision of “does this wrapper justify its own existence” becomes intricate, only answerable by a mystical sort of experience driven intuition. Or so it seems most of the time :p

That said, writing code that is intended to be wrapped is something I find myself doing a lot lately. Instead of trying to build a monolithic technology that includes many “engine” wrappers, instead an “engine” can delegate tasks to other services, and interact with those services via wrappers. I hate seeing those “ExFile” or “SCXHandle” low level constructs ubiquitously sprinkled over a codebase. It tangles everything together needlessly. Instead I’ve been favoring more code duplication just to lower dependencies. Multiple systems have their own file IO wrappers or handle implementations, and plugin to the “engine” with a wrapper. The “engine” basically being the place gameplay happens.

[Garthy]

Would you be happy to elaborate on a couple of your points? These two in particular:

A codebase utilizing these wrappers is harder to move, causing lots of problems during code maturation.

Minimizing dependencies is critical as code becomes older, simply to make the code easier to work with, and less likely to break.

[InfiniteStateMachine]

If you had some api that wraps vector and for some reason find that another project can't have the STL wouldn't you just not compile the STL vector wrapper and thus avoid the STL dependency?

[Ordnas]

What about re-implementing an array structure from scratch instead of using the STL std::vector? It should not be too hard to re-implement std::vector even for a simple project.

[Schrompf]

What about *not* redoing work which has already been done by someone way smarter than you? An exception-safe vector with proper move or in-place construction, reallocation optimisations for trivially copyable types and so on is not an easy undertaking.

A simple dynamic array is quickly done. You'll discover its various bugs and shortcomings only after a while.

[JWki]

What about *not* redoing work which has already been done by someone way smarter than you? An exception-safe vector with proper move or in-place construction, reallocation optimisations for trivially copyable types and so on is not an easy undertaking.

A simple dynamic array is quickly done. You'll discover its various bugs and shortcomings only after a while.

What about doing a better job solving the concrete problem at hand than someone smart who had to support every possible problem?
I need containers that have actual, GOOD support for custom allocators (not the nightmare that STL has) but I don't need exception safety (exceptions are turned off) or reallocation optimisations (all my types are trivially copyable usually so memcpy is how reallocation works by default). Most of the time I don't even need or want reallocation but I still want a "dynamic" array.

[Schrompf]

Well, I'm on your side with the allocators - those could certainly be way better. Other than that: you're wrong. Just reimplementing something because the default solution does all of it and then some, but some of which you don't need, is... silly. And your types are definitely not all trivially copyable, your custom vector being the very example.

[Garthy]

There are times when implementing a domain-specific solution makes sense, and there are times when it does not. It takes more time to write your own. Can you write a solution that brings sufficient benefit to the specific problem you are trying to solve, whilst taking away that same time from other productive endeavours, but also factoring in the time you will save by using a solution tailor-made to the problem?

Sometimes the answer will be "yes", and the right thing to do is to implement your own solution. Sometimes the answer will be "no", and the right thing to do is to use the tools you have, and sink your time elsewhere.

I could only honestly disagree with an absolute answer here. Both are correct given the right circumstances, and either could be superior to the other, depending on the problem. I don't think anyone here is seriously arguing for either extreme, are they?

[Schrompf]

I was talking about reimplementing a dynamic array. I have written my own share of custom containers for specific purposes over the years, but you're right: "specific purpose" is the key. "I need this and that in addition to..." is a good reason for rolling your own. "I don't need this and that, so I threw it all away and rolled my own." is just stupid.

Last week a colleague argued in all honesty that he prefers the old GCC4.6 "because the newer ones are so bloated". I was speechless.

This is probably a C++-specific issue. We use C++ to be in control of everything. And in some brains this "being in control" doesn't stop at what sequence of instructions ends up in the executable. They absolutely need to have control over the source, too, and then some. I can live with a certain amount of "Not invented here" syndrome, but heck... at least reflect on it and be conscious about it.

[JWki]

I was talking about reimplementing a dynamic array. I have written my own share of custom containers for specific purposes over the years, but you're right: "specific purpose" is the key. "I need this and that in addition to..." is a good reason for rolling your own. "I don't need this and that, so I threw it all away and rolled my own." is just stupid.

Last week a colleague argued in all honesty that he prefers the old GCC4.6 "because the newer ones are so bloated". I was speechless.

This is probably a C++-specific issue. We use C++ to be in control of everything. And in some brains this "being in control" doesn't stop at what sequence of instructions ends up in the executable. They absolutely need to have control over the source, too, and then some. I can live with a certain amount of "Not invented here" syndrome, but heck... at least reflect on it and be conscious about it.

I was writing my own response while you were posting this so here is that (touches on the full control part because being in control of what is executed in the application isn't really what happens when you're using STL):

Well, I'm on your side with the allocators - those could certainly be way better. Other than that: you're wrong. Just reimplementing something because the default solution does all of it and then some, but some of which you don't need, is... silly. And your types are definitely not all trivially copyable, your custom vector being the very example.

We'll just have to disagree here. A solution that does everything will have some overhead to be able to do everything and that's fine because if I want everything it's nice to have it but if I have a choice between including a multi-k loc template header that blows up my compile times, requires me to enable exceptions to work in the intended fashion and also executes code that I don't need to be executed (i.e. decides that my types are not safe to copy trivially which is something that the STL and I sometimes disagree on), or I just write a 200 line thing that does exactly what I want for all my use cases and nothing else, doesn't make use of complex metaprogramming so it doesn't blow up my compile times, and ON TOP OF THAT is ACTUALLY portable in the sense that the behaviour will be IDENTICAL on every compiler and every platform (and I do NOT mean the surface behaviour as in the spec, I mean the actual code that will run), then I'm happy to write and test a 200 line datastructure.

And yes containers are not trivially copyable but I don't put containers in other containers usually. And even if there are some types that require non-trivial copy, I find it much nicer to have an EXTRA data structure just for that instead of hiding it all behind the same interface (and I realize that might be a controversial opinion) so it is clear that there's a certain overhead that comes with using that structure. I very very rarely have types that are not trivially copyable though, I try very hard to ensure that they are. State for entire systems may not be in most cases but I don't see a reason to put system state into a container tbqh.


Look I agree that reinventing wheels isn't something one should pursue but when I need a different shape of wheel then I have to make that wheel. I haven't found to need more than maybe five or so different container types for my purposes, heck my last tech prototype (~10 loc) didn't contain any formalized containers, and I also have no use for most of the advanced features of the STL containers because the way I write C++ code just doesn't require them, but I do have use for a unified implementation and an interface that is streamlined for my purposes. I use the standard stuff for work and uni projects because in the former case they are just what we use so there's little point in overthrowing hundreds of thousands of lines of code for whatever reason, and in the latter case there's usually not too much time and other people are used to just using them so to avoid arguments like this one I just go along with it. But for my own stuff, I've found that the benefits of having full control over both interface and implementation and the reduced complexity of both greatly outweigh the disadvantages when writing my own data structures. Not to forget that being able to write somewhat standard containers is something that I think a programmer should be able to do so I just consider it practice.


EDIT: Oh and considering the NIH syndrome part, I for my part AM very aware that I am suffering that a bit more than a lot of other people I talk to and probably partly to an unhealthy degree, but I'd disagree that that's the major cause for my viewpoint in this issue for the reasons stated above.

[InfiniteStateMachine]

Just getting ready to go to work so I didn't get to read and catch up on anything but I have a quick example on why you might implement your own dynamic array.

When removing an object, a optimization is often to swap the remove target with the item at the end of the array and reduce the length by one. When people use the stl they tend to have to do this manually every time. It's nice to have it part of the implementation so it ends up just being a less error-prone method call.

Will catch up later at lunch

[popawheelie]

Just getting ready to go to work so I didn't get to read and catch up on anything but I have a quick example on why you might implement your own dynamic array.

When removing an object, a optimization is often to swap the remove target with the item at the end of the array and reduce the length by one. When people use the stl they tend to have to do this manually every time. It's nice to have it part of the implementation so it ends up just being a less error-prone method call.

Will catch up later at lunch

I ran in to this optimization in some third party code, and was unaware of it until I looked at the source. I needed the array to remain in order for some reason and things were amiss.

But it's a great optimization if you can use it. I implemented it in my stl vector wrapper. Generally I don't add a lot of extra functionality to wrappers, but they are great for those simple tweaks.

[ThemsAllTook]

When removing an object, a optimization is often to swap the remove target with the item at the end of the array and reduce the length by one.

Neat, I'd never thought of doing that. Good trick for unordered arrays!

[qMopey]

Would you be happy to elaborate on a couple of your points? These two in particular:

A codebase utilizing these wrappers is harder to move, causing lots of problems during code maturation.

Minimizing dependencies is critical as code becomes older, simply to make the code easier to work with, and less likely to break.

I can try an example. Say we like a particular compression algorithm inside of an unused legacy game engine. We want to use this algorithm in a game. It is our job to dive into the legacy game engine and pull out this algorithm and place it onto mobile devices. This is essential so the game can fit important pieces into RAM on devices without much RAM. Turns out instead of using portable arrays via pointer and integer pairs, it uses a funky std::vector wrapper. The wrapper has some custom optimization for removing elements of unordered arrays by swapping the last element with the removed index.

Luckily this vector wrapper is the *only* dependency the compression algorithm has. We copy + paste the algorithm along with the vector wrapper into our game. Works just fine, and we placed the vector wrapper into the game.

A few hours later we turn ok debug mode and realize the compression algorithm is way too slow in debug mode. After some profiling it turns out the array access operator hidden away in the wrapper was doing unnecessary debug checks, making debug mode too slow for practical day to day development. We decide to implement a basic std::vector replacement in around 200 lines of code. It is not templated. It’s just here to get the job done. This solution works for now.

Later on the compression algorithm is moves to cloud instances. Turns out the costs of running game servers is too high and can be reduced by using this compression algorithm to lower network bandwidth when players load levels from the server. We set out to move our code onto the servers. Everything seems ok at first.

Eventually someone runs a stress test on the cloud and finds the vector implementation makes raw calls to new, and is causing memory fragmentation on the server when a server instance runs for a long period of time. Now we must adjust the vector implementation to use a new type of allocator. Turns out the original wrapper is getting in the way, and we decide to remove it entirely to make it really easy to use the new allocator. Deleting the wrapper looked trivial. Turns out a bug was introduced regarding array index deletions; the wrapper used to oo the clever swap optimization. It takes three days to track down this bug.

Now imagine instead of one array wrapper this was instead 30 different dependencies on many different wrappers or little tools. Now imagine not one guy performed our job in the above story, but 7 different engineers over a 15 year period.

Now imagine an alternative: the compression algorithm was originally written using pointer arrays. It also exposed a configurable allocator with the C preprocessor. All above problems are trivially avoided as the library becomes self contained with little to no dependencies.

I didn’t even mention C++ templates, but once engineers start actually profiling compile times they realize the less templates code the better. Reducing template usage is also a real problem as code matures.