next »

[vittorioromeo]

Hello everyone!

This is going to be a DevLog about the implementation of one particular feature for my game Open Hexagon: online leaderboards.


First of all, what is Open Hexagon?

It's a fast-paced arcade and rhythm game heavily inspired by Super Hexagon, created with its author's permission (Terry Cavanagh). You control a small triangle spinning around the center of the screen, and your goal is to avoid various obstacles that hone in towards the center.

Here's a trailer:




Open Hexagon started as a clone of Super Hexagon, but it evolved into much more: it features new mechanics such as a 180° swap and curving walls*, and -- most importantly -- it was completely designed with *modding and customization in mind.

Users can create custom levels using Lua scripts and publish them to the Steam Workshop. Fans have created mind-blowing levels that push the engine to its boundaries.

Here's a short compilation of some very creative user-made levels:




I've also had players thank me for getting them into the world of programming, which feels amazing.

Despite being published on Steam, the game is also fully open-source (on GitHub), and anyone can contribute to it or build it themselves for free. My intention is to have the source always freely available so that people can learn from it and/or improve the game.


Now, onto the online leaderboards.

The main goals for this feature are:

- After playing a level, the user's final score is automatically submitted to a server.

- While selecting levels, the user can see his ranking in a global leaderboard.

Stretch goals:

- Download replays of top scores directly in-game from the server, and view them in-game.

Sounds simple enough... but it isn't. Fortunately, I've worked hard to make the game logic fully deterministic and have implemented an input-based replay system already, but there absolutely are zero online capabilities in the game.

Also, I want to make cheating as hard as possible. It sucks when players work hard for high scores, just to have the leaderboard ruined by a hacker.

Therefore, I came up with the following list of things to consider:

- Need a database, somewhere, maybe my VPS or have to rent cloud servers.
    - Not a big deal, but costs and logistics are something to consider.

- Need to completely separate game logic and rendering logic.
    - Definitely a big deal. I started working on this game almost a decade ago, and the codebase is not in a great state, despite my continuous efforts to improve it and clean it up.

- Need headless version of Open Hexagon which only runs game logic without rendering.
    - Again, big deal. The code for the game logic is intertwined with some rendering code.

- Need to somehow encrypt/decrypt replay files to avoid tampering.
    - I have no prior experience in this, but libsodium's Key Exchange and Steamworks Authentication should help.

- Need Open Hexagon client to produce and send replay clients to the server.
    - What happens if the replay fails to be sent? When to retry?
    - What happens if a client spams the server with replays? How to prevent DDOS attacks?

- Need Open Hexagon server to listen for replay files, and run them to verify validity.
    - What happens on version mismatch between client and server?
    - What about malicious replays? Someone could create a basic custom level without obstacles, run for 10 hours, then send a 10 hour replay that will stop the server from processing legit replays.
    - Need to visually notify the client that (1) replay was received and (2) is being processed.

- Need to improve menu rendering to show leaderboards.
    - Must avoid frequent calls to the server caused by browsing through menu entries.

- What happens if a custom level author updates a level?
    - If it's just a "refactoring" change that doesn't affect the behavior of the level, the leaderboard should remain valid
    - Otherwise, existing replays will fail to work. How to detect it? Periodically re-run replays? Or do not detect it at all?

- How to attach player identity to a leaderboard?
    - Custom login system backed up by custom database? Would be a lot of work.
    - Use Steamworks API? Probably easier, but completely ties the game to the Steam platform.

Happy to hear your thoughts on any of the points above.


First goals

My first goal will be to implement a flag in the game in order to run it in headless mode, accepting a replay file as input, running it, and printing out information about the replayed playthrough.

Ideally, I should be able to invoke Open Hexagon from the command line like this:

Code:

./SSVOpenHexagon.exe -headless <path to replay>

And get some sort of output like this:

Code:

Replay <path to replay> completed.

Level: <level>
Player: <name>
Final time: <time>

I will keep this DevLog updated as a way of motivating myself, documenting my progress, and getting feedback. Hope you enjoy!

[ThemsAllTook]

Neat! This is one of my favorite topics. Some thoughts:

- Need to somehow encrypt/decrypt replay files to avoid tampering.

I'm thinking that since you're already planning to send inputs to a server and have it simulate the result, any security upstream of there is in the realm of "make tampering take more effort" rather than "make tampering impossible". Encryption could give you verification that the list of inputs was encoded with a key you've provided, but that key is still under the user's control on some level. I'm imagining a hierarchy of security measures and consequences that would look something like this:

• Inputs are sent unencrypted: Replays could be trivially constructed and sent from outside the game.
  
• Inputs are encrypted with a key embedded in your executable: Replays could be constructed and sent from outside the game if the key was extracted and the encryption mechanism was known or reverse engineered.
  
• Inputs are encrypted with keys exchanged between the game and server: Constructing a replay outside the game now requires an initial communication step, and adds the potential for some timing-related security by discarding messages encrypted with a key that's older than some threshold. Requires more reverse engineering, or instrumentation of the game executable to modify the message payload.
  
  

Past this point, I don't think there's much you can do - since the user's CPU has to execute your instructions at some point and can execute any other arbitrary code, there's always going to be the potential for tampering. I thought I had an argument against encrypting your messages, but now that I've written all of this out, I think it's a solid plan. Someone dedicated could still modify the game to, say, slow down time and get a high score that way, but that takes specialized knowledge and an amount of effort that's probably not going to seem worth it just to get their name high up on a leaderboard? If your player base were to get large enough, someone would eventually do it, but that's probably not a concern for anything except the most widely-played of games.

Sounds simple enough... but it isn't. Fortunately, I've worked hard to make the game logic fully deterministic and have implemented an input-based replay system already, but there absolutely are zero online capabilities in the game.
[...]
- Need to completely separate game logic and rendering logic.
    - Definitely a big deal. I started working on this game almost a decade ago, and the codebase is not in a great state, despite my continuous efforts to improve it and clean it up.

- Need headless version of Open Hexagon which only runs game logic without rendering.
    - Again, big deal. The code for the game logic is intertwined with some rendering code.

Some special concerns here: Is your code deterministic not only on the same computer where the inputs were generated, but when they're replayed somewhere else, potentially on a different CPU architecture? If you're doing any floating point calculations, it might not be. Small rounding differences between floating point calculations on different CPUs and operating systems can give different results that will potentially affect the game simulation, causing replays to desync. Integer math has known consistent rules everywhere, so using fixed point numbers or something similar for any fractional calculations might be necessary.

- How to attach player identity to a leaderboard?

Do you allow players to write freeform text for name entry? If so, this could open up possible concerns about profanity (if you care), hate speech, links to malicious websites, spam, etc. One game I released had leaderboards (with very weak verification that no one bothered to cheat because the player base was pretty small), and one of the prominent players chose "POOPNUGGET" as their name. Pretty tame, I know, but it still revealed to me the potential for much worse things to be written.

Definitely sounds like you're on the right track, so I'll be interested to hear about your progress!

[vittorioromeo]

Neat! This is one of my favorite topics. Some thoughts:

- Need to somehow encrypt/decrypt replay files to avoid tampering.

I'm thinking that since you're already planning to send inputs to a server and have it simulate the result, any security upstream of there is in the realm of "make tampering take more effort" rather than "make tampering impossible". Encryption could give you verification that the list of inputs was encoded with a key you've provided, but that key is still under the user's control on some level. I'm imagining a hierarchy of security measures and consequences that would look something like this:

• Inputs are sent unencrypted: Replays could be trivially constructed and sent from outside the game.
  
• Inputs are encrypted with a key embedded in your executable: Replays could be constructed and sent from outside the game if the key was extracted and the encryption mechanism was known or reverse engineered.
  
• Inputs are encrypted with keys exchanged between the game and server: Constructing a replay outside the game now requires an initial communication step, and adds the potential for some timing-related security by discarding messages encrypted with a key that's older than some threshold. Requires more reverse engineering, or instrumentation of the game executable to modify the message payload.
  
  

Past this point, I don't think there's much you can do - since the user's CPU has to execute your instructions at some point and can execute any other arbitrary code, there's always going to be the potential for tampering. I thought I had an argument against encrypting your messages, but now that I've written all of this out, I think it's a solid plan. Someone dedicated could still modify the game to, say, slow down time and get a high score that way, but that takes specialized knowledge and an amount of effort that's probably not going to seem worth it just to get their name high up on a leaderboard? If your player base were to get large enough, someone would eventually do it, but that's probably not a concern for anything except the most widely-played of games.

Sounds simple enough... but it isn't. Fortunately, I've worked hard to make the game logic fully deterministic and have implemented an input-based replay system already, but there absolutely are zero online capabilities in the game.
[...]
- Need to completely separate game logic and rendering logic.
    - Definitely a big deal. I started working on this game almost a decade ago, and the codebase is not in a great state, despite my continuous efforts to improve it and clean it up.

- Need headless version of Open Hexagon which only runs game logic without rendering.
    - Again, big deal. The code for the game logic is intertwined with some rendering code.

Some special concerns here: Is your code deterministic not only on the same computer where the inputs were generated, but when they're replayed somewhere else, potentially on a different CPU architecture? If you're doing any floating point calculations, it might not be. Small rounding differences between floating point calculations on different CPUs and operating systems can give different results that will potentially affect the game simulation, causing replays to desync. Integer math has known consistent rules everywhere, so using fixed point numbers or something similar for any fractional calculations might be necessary.

- How to attach player identity to a leaderboard?

Do you allow players to write freeform text for name entry? If so, this could open up possible concerns about profanity (if you care), hate speech, links to malicious websites, spam, etc. One game I released had leaderboards (with very weak verification that no one bothered to cheat because the player base was pretty small), and one of the prominent players chose "POOPNUGGET" as their name. Pretty tame, I know, but it still revealed to me the potential for much worse things to be written.

Definitely sounds like you're on the right track, so I'll be interested to hear about your progress!

Thanks a lot! I found your reply very valuable.

First of all, I know that any security measure I put in place is never going to be enough to deter the most determined cheaters, but hopefully my measures will stop 99.9% of them

The "slowing down time" approach is an interesting one. What I am considering to counteract it is the following:
- The client must be connected to the server at all times, using heartbeats
- When a client starts a game session, it sends a packet to the server
- The server receives the packet, and stores a timestamp
- When the client ends a game session and sends a replay, the server compares the time with the stored timestamp
- If the replay time doesn't match the difference between timestamps (with some margin of error), reject the replay

What do you think?

I will also think about restricting name entry to avoid terrible names, malicious websites, or advertisements.

[vittorioromeo]

I just finished writing up the second devlog

As TIGSource doesn't accept Markdown (sigh) and any online Markdown to BBCode converter I found failed miserably, I have posted it on my website where you can read a properly-rendered version:

https://vittorioromeo.info/index/blog/oh_leaderboard_devlog_1.html

Any way to embed it or to use Markdown in TIGSource?

[vittorioromeo]

Finally have a working implementation! Here's a devlog in video format which discusses:

- Server/client model
- Encryption and key exchange
- Threading model
- Anti-cheat measures





Enjoy!

[vittorioromeo]

After many many hours of work, the Open Hexagon 2.0.5 beta (introducing Online Leaderboards) is now officially released to the public for testing and feedback!

Read the full announcement and patch notes here: https://steamcommunity.com/games/1358090/announcements/detail/5225840313088142682

[vittorioromeo]

Squashed some final bugs and polished everything up. Changed compilation settings to use strict floating point mode to avoid simulation divergence between client and server. And finally officially released out of beta:
https://steamcommunity.com/games/1358090/announcements/detail/3064113661648232669

This devlog is officially done