next »

[nikolail]

Honestly big thanks for frequent devblogs even thou i rarely understand half of the words, but your attitude is so pleasent to watch, professional and still honest 

Keep your chin up, many of us cheering for you guys!

[Gregg Williams]

Honestly big thanks for frequent devblogs even thou i rarely understand half of the words, but your attitude is so pleasent to watch, professional and still honest 

Keep your chin up, many of us cheering for you guys!

Cheers, hopefully some of my recent technically complex posts will start to die down, and we will get some more artwork and videos going again Just right now, things are pretty deep with this whole networking effort.

[nikolail]

Don't get me wrong, i do love reading complex coding problems, devblogs rarely dive on those subjects like you do.

[Gregg Williams]

Alright finished up the BufferPool class. The general purpose of which is to create a pool of buffers ;p You create an instance usually static for a given set of classes, give it an initial capacity, and size. Then as you need buffers you just acquire them out of the pool, and finally release them back into the pool. This like the Socket Pool is important to prevent tons and tons of runtime allocations and deallocations of memory on the server. If either pool is completely emptied, than they add another round of capacity.

So like my net state class which encapsulates a TCP/IP socket connection and various state, upon creation requests a buffer from the net state buffer pool, in order to handle socket receives. Once that connection shuts down, the buffer as well as the socket go back into their pools instead of being deleted.

I also figured out how to disable Nagle on the tcp sockets using asio. Nagle is a buffering algorithm that is part of TCP/IP with the purpose of preventing the send of a bunch of small packets. This is important since TCP packet overhead is 40 bytes by itself, now imagine something like a telnet session, and each character typed results in a 1 byte packet send, which in reality becomes 41 bytes. The problem of course with Nagle is that it prevents sends until the buffer is full. So for a game, you usually want to turn it off.

However, you still want to actually buffer data when you can to minimize sends and tcp/ip overhead. A single server update cycle might result in tons of packets to be sent to a player, and if no buffering exists every packet would be an individual send with tcp/ip overhead tacked on. Its better to queue these up, and send them only when you have complete buffers. While also at the end of the update cycle, doing a flush essentially to make sure any buffers that aren't full, still get sent out, vs hanging around. This way you ultimately get buffering, plus make sure anything small still goes out, vs being stuck in a buffer and the player waiting around and never receiving it.

You also need to deal with the case of where the socket isn't ready for any more data to be written to it yet.

This is what I'm currently working on implementing at the moment. This is probably the last real super low level part of the whole networking portion of things. Other than maybe the message pump. After this its all writing nice little packets, handlers, and defining a game specific protocol. At some point I'll also will need to do some load testing I suppose, to make sure everything stays stable, and I don't end up with any deadlocks.

[Gregg Williams]

So after way to many crashes, the whole send queue, and buffering scheme seems to be working

The biggest root of my problem really being I had src and destination parameters to memcpy backwards o.O

Next up figuring out where I'm using boost date_time wrong, or replacing it, and starting in on packets.

[s_l_m]

Hey, I'm posting this partially so this topic shows up in my replies and partially to tell you it looks great.

[Gregg Williams]

Hey, I'm posting this partially so this topic shows up in my replies and partially to tell you it looks great.

Cheers and on that note I fixed my boost date_time problem. Bloody > instead of < doh

So yeah onto packets, writers, readers, and handlers.

[Gregg Williams]

Alright so, basic packet reader, packet writer, and packet class is done. I've choose to skip compression for the time being, as its fairly simple to add in down the road, and for initial development just adds more time and complication. Much like encryption, which in all likely hood will just be done over SSL vs a roll it yourself solution.

I'm able to create nice little test packets, send them, decode them, and so forth.

Code:

class TestPacket : public Packet
{
public:
    TestPacket() : Packet(0x10, 5)
    {
        m_Writer->Write((int)0xDEADBEEF); 
    }
};

Above the packet constructor is taking a packetID, and the length in bytes for the packet. To support variable length packets, there is also a constructor that just takes a packetID.

So a simple example flow for the server to send to all connected clients would be something like:

TestPacket p;
NetState::BroadCast(p);

The next big step is to get some sort of system for packet handlers, and overall message pumps working.

That way you have a message pump that basically loops while there are bytes received, and determines if the data matches a packet id, the length expected for the packet, and if all data is currently received for that packet. If so a packet reader is built, and then passed off to a handler which is pretty much just a registered callback per packet type. The handler decodes the packet data, and performs whatever other logic is needed.

For now though I take a break, and go check out my beta invite to SpyParty which looks very interesting

[Gregg Williams]

Well handlers are now done

Nothing special to it at all really, just a struct, with a packet id, length, and a boost function for the callback.

Then a handler class that goes like PacketHandler handlers[0x100], along with a register function that creates a packet handler struct, and adds it to the array based on the packet id.

Tomorrow I'll wrap up the message pumps, which isn't anything fancy, and then I'll be ready to more or less move back to actual feature development.

Just instead of for example asking the client to generate a path when I click on a movement grid tile, I'll instead send a packet asking the server if I can move to X destination that I clicked. The server will then validate that, generate a path, and send me back a set of moves to make, along with possibly some other information like on Move X to stop and query the server for a reaction fire event. That or possibly just verify a path client side generated, and then have the client send a packet per step it takes. Some fun decisions to make in that realm. Likely per step will happen, as it could very well affect visibility, and other pieces.

I'll probably do a hair more network wrap up, by making a server and client network class just to encapsulate a bunch of junk, and make it easy to separate once I want to have a dedicated server app as well.

[Gregg Williams]

Alright Message Pump and server class is done

Things seem to generally be working fairly well. A few deadlocks aside, thanks to forgetting to unlock some mutex objects when bailing out a loop early. Server is able to accept connections, process valid vs invalid packets, and send responses.

Client side needs a bit of cleanup, but is working as well.

At this point, the core networking layer is probably around 95% complete, and I can get back to higher level feature development, and higher level networking constructs. E.G things like starting a new game instance on the server, and the protocol of packets the game will be using.

From a scaling perspective, I'm fairly hopeful that one single game server could handle all of the connections needed, but it looks like I can likely do horizontal scaling anyways if I need to. This would be accomplished by having lobby servers separate, and once a match is started the lobby server could just ask a game server selected based on some criteria, to create a new game instance with X parameters, and then tell each client to disconnect from the lobby server, and connect to the game server at IP XYZ and to join game instance W. Since game server instances in this regard, are only handling game logic and processing, not any of the other junk.

The more I think about it, the more splitting out various aspects into their own servers makes sense. Things like authentication, lobby, patching, etc can all really be their own servers and code bases. (Even if each process is running on the same physical host) This also allows us to re-use all those other servers for future games since their functionality is generally generic vs game specific.

[Gregg Williams]

Installed Linux Mint (lol) today into a vm, I've wanted to try an support linux as a platform for M.I.N.T though the graphics performance so far seems to be really terrible in the VM unlike say my Windows VM, so I might need a different solution. =/

[Gregg Williams]

Alright wrote and cleaned up the whole networking side for the client today. Also spent a good hour or two doing design on how things are suppose to flow.

For instance, for a typical online session the flow likely works out to something like:

1. Connect to a authentication/lobby server (these might get split out)
2. Create or join a game.
3. Configure all the various game settings, setup your army, etc, mark that you are ready. Server verifies valid settings.
4. Lobby server, picks a game server, and asks to create a new game instance, and passes all of this setting data as well as the players.
5. Game server, responds back with a new game instance ID, and authentication tokens or something per player.
6. Lobby server tells each client to disconnect from the lobby, and connect to game server at IP XYZ, using the provided game instance ID and authentication token. (Auth token is possibly tied to IP, and is only valid for a small window of time)
7. Waiting state until all players connect, or timeouts happen.
8. Server informs connected clients to enter into the game state, and into deployment mode.
9. Server tells the player what unit to deploy.
10. Client deploys unit, and asks for approval.
11. Server approves/rejects deployment, and if approves, sends next unit to deploy.
12. Once all player deployments are done, server picks a player to go first/establishes turn order.
13. Game now is live, and follows turn sequence.

This type of flow can be different of course if your doing SP, or LAN, or even online auto matchmaking/ladder stuff, but it shouldn't diverge by to much I imagine. (Other than SP, that could diverge quite a bit)

[ericmbernier]

I just discovered this via your sig in another post. It is looking great so far. I look forward to following along

[Gregg Williams]

Well no progress today. Instead of making the little jeep/hummer move around in cooler ways, I spent 6 hours trying to make a jeep in physical reality move correctly. Or more like fighting with grease inserts, and differentials. I guess I won, but it took way to long, I should probably stitch shut my hand that is bleeding, and I have a wicked headache, which is probably from all the brake cleaner and pb blaster.

Hopefully tomorrow will return to our regularly scheduled M.I.N.T development

[marvinhawkins]

You had me at giant mech....

[Gregg Williams]

Sorry guys avoiding the computer and only hitting forums a tad from the ipad.. the hand wound is pretty nasty and swollen. In true indie fashion I super glued my self back together which saved the keyboard from drowning in blood.. , but I think I'm going to be spending a few days away from development =/

[Impmaster]

You. Super. Glued. Yourself. Together. That is so metal. 

[nikolail]

That's manly way to do it! 

[Gregg Williams]

Alright, after 5 days of zero work, and going out of my mind I think its time to resume. Glue seems to have done the job more or less, still probably another week to fully heal, but seems decent enough to get back to making progress.

I'd love to be able to show off some new art, but alas a ton of decisions are flying around in that area, and Kawe is also distracted with a little side job he couldn't pass up doing, so I don't expect anything new on that front for a few weeks. I'll no doubt end up throwing together some  temporary huds and interface stuff though.

[Gregg Williams]

Built a standalone server version, and fixed a number of errors in the packet handling code. Most of my time today actually got spent cleaning the windows server machine, and various VM's from a bunch of trojan horses that I have no idea how actually came to be.