next »

[Klaim]

Holy balls, thank you sir.
It will take time to process this huge volume of info though, will probably reread it from time to time.

I believe it's a bit hard to follow which is why I want to use it as a draft and make it better.

The only thing I fear so far is that how much maintainability it will need in the end.

It's why I don't recommand the architecture I'm using to everybody. It requires a wide knowldege, just because all the tools I gathered are totally separate and you have to know about them, how to use them, etc.
Also, it's high-grade C++. Unfortunately it means not everybody can maintain this code. But it also mean that I, myself, can do more with less, so it's ok in my case.
Knowledge of bleeding-edge techniques to manage concurrency is clearly a requirements.

- Edit - Also it feels your approach is rather complex, which might bring issues to the table.

First, keep in mind that it's the summary of several months of hard work, shaped as a braindump, so it should look more complex than it really is.
It's my fault for giving too much information. It happen a lot.

Half of what I said is just principle I used to manage complexity. The other half is tools I've built to follow these principles to manage complexity. What I mean is that I described the tools but not much the usage.
I used all my knowledge and learning to put the complex part in tools, so that the game-specific code is isolated and works fine.

When I use my system, it's mostly like using normal code, just cut in tasks and pushed in scheduling functions. I don't have to think about concurrency other than when I choose if the data types have to be updated concurrently or not.

So basically, it's only setting up a framework to achieve these goals:
 - scale with the hardware concurrency capabilities;
 - efficient enough (for my specific game);
 - easy to write game code (it should be written mostly like normal sequential code);

I think the explanation would be better with some diagram and some code example though.
I could paste real usage examples of usage here, but it all looks dumb code.

Like, here is how I initialize the client systems:

Code:

Client::Client( NetRush& netrush_app )
: m_netrush_app( netrush_app )
{
using namespace netrush::system;
using namespace netrush::view;

NR_LOG( "#### Client Creation... ####" );

auto graphic_config = m_netrush_app.config().get_sub( "netrush.graphic" );
m_graphic_system = std::make_unique<GraphicSystem>( m_netrush_app.task_scheduler(), "NetRush", graphic_config );
m_input_engine = std::make_unique<InputEngine>( m_netrush_app.task_scheduler(), m_graphic_system->engine().window_handle() );
m_zoneview_system = std::make_unique<zoneview::ZoneViewSystem>( *m_graphic_system, *m_input_engine );

// Exit when Escape key is pressed.
m_input_engine->schedule( InputTask( "exit_on_escape", [&]( InputState& input_state )
{
if( input_state.keyboard().is_down( KEY_ESCAPE ) )
m_netrush_app.request_exit();
}).reschedule().until( [&]{ return m_netrush_app.is_exiting(); } ) );

auto zone_view = m_zoneview_system->create_zoneview( zone::IdGenerator().make_id<zone::Zone>() );

NR_LOG( "#### Client Creation - DONE. ####" );
}

Here is how I setup the graphic update tasks (m_tasks is a TaskChain<GraphicData>)

Code:

void GraphicEngine::schedule_default_tasks()
{
m_tasks.push_back( GraphicTask( names::GRAPHIC_WINDOW_UPDATE
, []( const GraphicUpdateInfo& )
{
update_windows();
}).reschedule() );

m_tasks.push_back( GraphicTask( names::GRAPHIC_RENDERING_TASK
, []( const GraphicUpdateInfo& info )
{
UCX_ASSERT_NOT_NULL( Ogre::Root::getSingletonPtr() );
Ogre::Root::getSingleton().renderOneFrame( static_cast<Ogre::Real>( info.delta_fsecs ) );
}).reschedule() );

}

Here is how the input system is initialize itself:

Code:

m_state = std::make_unique<State>( window_handle );

static const auto INPUT_SYNC_FREQUENCY = milliseconds(16);

// Launch the input udpate task.
auto task = system::AsyncTask( names::TASK_INPUT_TASKS, [&]()
{
m_state->update();
m_tasks.execute( m_state->current_state() );
});
task.reschedule().interval( INPUT_SYNC_FREQUENCY );

m_task_end_sync.sync_with_task( task );
// Now we're ready to schedule a task for calling our task chain.
task_scheduler.schedule( std::move(task) );

AsyncTask is a typedef for Task<InputData>.
The m_task_end_sync.sync_with_task( task ); just make sure that if the input system is destroyed, it will first notify the task to end and wait for it, terminating in synch with it. This makes everything very deterministic. I use it in each core system construction/destruction for synchronizing with update tasks.

As you can see, other than classic modern c++ idioms, all the complexity is hidden. What you feel is complex is actually the implementation of the tools to hide complexity, which really are glue code using boost/tbb/c++11 libs. I also don't even use futures much yet because they becomes interesting only since the lat boost release.

If you look at this code and feel it's complicated, then really don't even try to do the same.

I believe managing concurrent code will be far simpler once we get some tools that are proposed for the next C++ major standard (c++17 is the current target) so currently we still have to explain how to tools work before you use it and build them because there's nothing standard.

[Garthy]

Klaim:

Yikes, that seems enormously complex, and I did have trouble absorbing it all. It would definitely have the potential to be a massive project. I hope the amount of work you've sunk into it thus far will let you obtain the gains you are after!

It sounds like you're approaching developing such a system in the right way though, with plenty of unit testing as you build the supporting framework up. I don't think it would be practically doable without this. Best of luck!

...

Also, quite a few engines and frameworks have issues when resources are created from different threads. In the case I mentioned earlier I was using OpenGL directly, and when I ran into the issues with textures and models, I "cheated" and just shifted the resource creation back to the main thread, leaving the bulk of the real work in the separate tasks. I'm very much a proponent of using multithreading to solve the easiest 90% of a task, and leaving the difficult last 10% the heck alone.

[Klaim]

Yikes, that seems enormously complex, and I did have trouble absorbing it all.

Don't mix the complexity of the message with the complexity of the system though. I just wrapped library stuffs under tools to make things simple. Using all this right now feels easy.

It would definitely have the potential to be a massive project. I hope the amount of work you've sunk into it thus far will let you obtain the gains you are after!

I hope too and I already begin to see the benefits! The goal to make easy to manage concurrency is, so far in my current use, achieved. But you're right that if you don't have the same kind of background than me it can be a massive project. I expect this kind of systems to become standard in some ways later so that people can use them without understanding the details, as it helps a lot but not everybody can manage to implement all that.

The hard part really was understanding. Like, understanding what is necessary, like ABXY, which mean you have to take a classic game engine design (I re-read parts of Game Engine Architecture) and have to think about how each part work, what can and cannot be parallelized and what kind of paralellization works. I also did a lot of concrete research by implementing 1 file versions of each system with highly concurrent tests before beginning putting the system in my game.

I was actually fearing loosing too much time on these tools but once I got it in place I was actually surprised to not need anything else (other than future.then() and some refactoring once I got variadic templates available - which would make the implementations far shorter )
3 months was actually short compared to what I was thinking and if it would have been 2 months more I would have quit. I also am surprised that I can fix the concurrency bugs but I think it just helps to have a clear understanding of the nature of concurrency in current computers.

Also, I don't know yet if I did'nt totally kill performance but I made sure everything is tunable when I'll need to optimize. Performance is the point I can't test right now because I need to have the basic blocks of the gameplay finished before trying to stress test the game and see the limits. That will be an interesting experiment.

It sounds like you're approaching developing such a system in the right way though, with plenty of unit testing as you build the supporting framework up. I don't think it would be practically doable without this. Best of luck!

I agree, I actually learnt to use unit tests with this project and my open source project (which I had to work on more in the last monts). Unit tests are vital when you build unit systems, really. I even add unit tests today because a lot are missing, and I discover bugs I couldn't see in my current usage. So yeah.

Also, quite a few engines and frameworks have issues when resources are created from different threads. In the case I mentioned earlier I was using OpenGL directly, and when I ran into the issues with textures and models, I "cheated" and just shifted the resource creation back to the main thread, leaving the bulk of the real work in the separate tasks. I'm very much a proponent of using multithreading to solve the easiest 90% of a task, and leaving the difficult last 10% the heck alone.

I would go the same way (or even not using concurrency at all until I discover that I want to parallelize one specific processing). By the way I didn't understand why you tried to manipulate thread priority? Isn't it like a dead-end? It's so hardcore.

[Garthy]

By the way I didn't understand why you tried to manipulate thread priority?

Re thread priority, I had a set of important and non-important tasks (for simplicity I'm including the main thread and incidental threading as important tasks). With thread priority the same, it was slowing down the processing on the important tasks whenever non-important tasks ended up on the same core as the important ones. This had a very noticeable effect on the frame rate.

I dropped the priority on the non-important tasks, and the problem vanished instantly. The unimportant tasks took longer to complete, of course, but that was expected. The important tasks and main threads went back to being smooth again.

Unfortunately, I then tried it out under Windows (original version was on Linux), and that's when I learnt an important lesson about making too many assumptions on the quality of schedulers. On Windows, the unimportant threads were starved entirely, rather than getting a reduced timeslice. It took a bit of messing around to get the effect I wanted there.

[Klaim]

I see, in this case it's not the threads that must have priority but the tasks, so that a thread pool task scheduler can sort the tasks in the thread task queues.
That's the kind of hardcore shit to setup that I typically don't want at all to do myself so I was using TBB tools for all this kind of stuff. But in the end, the systems that really need high frequency are using their own threads (what I call system 1).