next »

[JWki]

Hi all, I was wondering about how to efficiently save chunks of map data. My map is mainly composed of 8x8x8m smooth voxel terrain, where the data for each chunk is stored as a 32bit-float[8][8][8]. The data is procedurally generated, and the player can modify to their hearts content, so without compression each chunk contains 8*8*8*4bytes=2kB of data. Right now I have the program multithreaded such that the main thread handles openGL, physics and game logic, which'll periodically send requests for changes in terrain to a voxel managing thread. The voxel thread loads chunks if they exist on file, generates them if they don't, skins the voxels, then sends a packet with the finished triangles back to the main thread. I'm saving data by just writing the 2kb to a file identified by the coords of the chunk, this seems quite performant, but occasionally creates visible lag between the player digging terrain and the effects appearing (because the voxel thread can get stuffed with load requests, which due the order required for skinning to work, must be completed before skinning begins). The potential problem is that in about 2 minutes of gameplay you can easily create ~500 2kb files, and I was wandering if it could be sped up in some way, or because of hard drive fragging this might be unethical. Is there a way to package little files into a large file in a way that random access will be performant, even if this large file gets very large?

tldr; saving potentially thousands of 2kB files to disk, is this a problem? And if so is there a way to better it? (While remaining as fast as possible).

Why exactly are you writing to disk all the time?

[Oats]

Is there a reason I shouldn't? I just save a chunk whenever the player moves a certain distance from it, an load whenever the player moves near a spot where there should be a chunk. My map is total possible size is the integer-limit^3, so I can't store the whole map, or even just the space the player has been near, since that could easily go up 100s of MB. So I save when ever I clear a chunk.

[JWki]

Is there a reason I shouldn't? I just save a chunk whenever the player moves a certain distance from it, an load whenever the player moves near a spot where there should be a chunk. My map is total possible size is the integer-limit^3, so I can't store the whole map, or even just the space the player has been near, since that could easily go up 100s of MB. So I save when ever I clear a chunk.

Writing 500 files in 2 minutes seems quite excessive tho.
So yeah first thing to optimize is probably getting that down to fewer files - batching up chunks so you have a file per column for example. I'd go as far as to say try to use a single file. Memory mapped I/O could help here to speed up access times.
Also, are you only saving chunks that have been modified since generation? Because if not that could also help.

EDIT: Upon further consideration, using a single file is probably going to limit your world size in undesired ways, so that's probably not an option. However you can try batching up chunks to save as mentioned above. Try batching by column, or n*n columns.

Oh and you can also try RLE compression for chunks that are mostly consistent (like air).

[Oats]

Some useful suggestions, I think I'm gonna try doing batches sticking chunks into larger groups, and run length encoding seems a sensible route, thanks! 

[gimymblert]

Hi all, I was wondering about how to efficiently save chunks of map data. My map is mainly composed of 8x8x8m smooth voxel terrain, where the data for each chunk is stored as a 32bit-float[8][8][8]. The data is procedurally generated, and the player can modify to their hearts content, so without compression each chunk contains 8*8*8*4bytes=2kB of data. Right now I have the program multithreaded such that the main thread handles openGL, physics and game logic, which'll periodically send requests for changes in terrain to a voxel managing thread. The voxel thread loads chunks if they exist on file, generates them if they don't, skins the voxels, then sends a packet with the finished triangles back to the main thread. I'm saving data by just writing the 2kb to a file identified by the coords of the chunk, this seems quite performant, but occasionally creates visible lag between the player digging terrain and the effects appearing (because the voxel thread can get stuffed with load requests, which due the order required for skinning to work, must be completed before skinning begins). The potential problem is that in about 2 minutes of gameplay you can easily create ~500 2kb files, and I was wandering if it could be sped up in some way, or because of hard drive fragging this might be unethical. Is there a way to package little files into a large file in a way that random access will be performant, even if this large file gets very large?

tldr; saving potentially thousands of 2kB files to disk, is this a problem? And if so is there a way to better it? (While remaining as fast as possible).

Minecraft faced the same problem, they effectively choose to pack them into region file
http://minecraft.gamepedia.com/Region_file_format

[Oats]

@gimymbert that's a good resource, it just seems like kind of a pain to implement though, but I'll implement something similar soon.

[Sik]

It pretty much boils down to "only save the chunks that have changed" (i.e. don't save those you can just recompute from scratch). The header's purpose is pretty much just to tell it which chunks it has saved.

[BorisTheBrave]

The key takeaways from that Minecraft link are:

1) Arrange chunks heirarchically (in Minecraft: sections are 16x16x1, live inside chunks 16x16x256, live inside regions 512x512x256). Note the vertical and horizontal differences, if that's relevant to your game
2) Quantize your data - you don't need a float per vertex.
3) For the largest sizes of chunks, compress, as you don't need to load / unload them very often.
4) Skip chunks that are entirely empty, (which is lots when you've partitioned into horizontal slices).

Obviously, this is a bit much to implement right away. It'd probably be simpler just to have two files. One a containing all the chunk data (say, blocks of 32x32x256), and one to keep an index of where to find where each chunk is in the file. The second file is going to be vastly smaller and rarely changes, so you can just load the entirety of it into memory while running the game. That'll work until you get truly huge maps.

[gimymblert]

2) Quantize your data - you don't need a float per vertex.

Wait what? you don't need to store vertex at all, they are implicit to the underlying data.

Depending on how you store material , you could also store a bit array of 32x32 x 1 int32
It seems that you can speed up generation by only generating carving noise (ie filling and negative space voxel) and use the generation to specify as a query for only visible voxel or at removing time.

[BorisTheBrave]

Wait what? you don't need to store vertex at all, they are implicit to the underlying data.

Tbh, I have no idea why oats is using floats. Was kinda assuming some marching cubes type thing, which is per-vertex, not per-cell-center.

Point still stands, 32bits per cell doesn't make sense.

[gimymblert]

I guess he is using floats for isosurfaces (ie percent of matter in each voxel) as he says "smooth", still don't need to store vertex (only region's position in int). But I missed that part since I was focusing on "minecraft like", so my point stand except boolean grid (though it can be an optimization's look up depending on the impact it has on cache and other things)

[Oats]

@sik, neat, took like a minute to implement too.

@BorisTheBear,
1) Splitting my chunks into horizontal slices won't achieve much, since the game is pretty solidly filled in all directions.
2)No idea what this means, as gimymblert says I only store one value between -1 and 1 per cell, then at load time approximate the surface where the interpolation is 0, using a simplification of marching cubes.
3)Yeah this sounds sensible

Yeah 32bit precision is overkill, I've been meaning to switch it to using single bytes instead.
Thanks for the material!

[bateleur]

2)No idea what this means, as gimymblert says I only store one value between -1 and 1 per cell, then at load time approximate the surface where the interpolation is 0, using a simplification of marching cubes.

Makes sense, but the point is that the "value between -1 and 1" does not need as much information content as a 32-bit float for your algorithm to work. So, for example, a 16-bit fixed point representation might be enough, or even 8-bit fixed point. Why bother? Because there are obvious large efficiency savings.

[Oats]

@bateleur Yup, literally what I wrote at the end of my post ^^

[Pishtaco]

I need some https/ssl/whatever help.

I know nothing about networking. I have a website, http://dood.al, that's hosted on nearlyfreespeech.net. I want to put this program on it (when it's finished), a software oscilloscope that should be able to take input from the user's microphone. But it turns out that Chrome will only give access to the microphone if you're on an HTTPS site.

According to this thread on nearlyfreespeech.net, there's a script I can run on the site that will take care of this automatically. It will get a free certificate from Let's Encrypt, install it and renew it when necessary. Now firstly, this seems a bit too good to be true; and secondly, I don't actually know what any of this stuff means, and what effect it will have. I would like it to be so that when someone types dood.al/oscilloscope into their browser, they get the oscilloscope and the microphone will work; on the other hand I don't want the way people currently access the site to stop working.

On the nearlyfreespeech control panel there's also a button saying "enable TLS". If I press it it says "TLS cannot be enabled for doodal.nfshost.com until it is configured for dood.al or the site's canonical type is set to off." 

Any help much appreciated.

[BorisTheBrave]

Let's Encrypt is the real deal. Their certificates don't prove your identity, but they work fine for all the security stuff like TLS.

Canonical type is this: https://faq.nearlyfreespeech.net/section/customization/canonicaltype#canonicaltype

Basically, it sounds like your current setup has redirection turned on, and this is incompatible with TLS (probably because it is redirecting to the http version of your site). I would either look for another Enable TLS button applying to the dood.al domain instead of doodal.nfshost.com, or turn off that redirection as explained in the link.

Also read this: https://faq.nearlyfreespeech.net/section/customization/enforcessl#enforcessl

[Pishtaco]

Let's Encrypt is the real deal. Their certificates don't prove your identity, but they work fine for all the security stuff like TLS.

Canonical type is this: https://faq.nearlyfreespeech.net/section/customization/canonicaltype#canonicaltype

Basically, it sounds like your current setup has redirection turned on, and this is incompatible with TLS (probably because it is redirecting to the http version of your site). I would either look for another Enable TLS button applying to the dood.al domain instead of doodal.nfshost.com, or turn off that redirection as explained in the link.

Also read this: https://faq.nearlyfreespeech.net/section/customization/enforcessl#enforcessl

Thanks very much for the reply. Unfortunately I'm still struggling with it. Are you saying it's set up now to redirect anything to "http://dood.al", and this means it's incompatible with TLS? Right now the Enable TLS button is only next to "doodal.nfshost.com", there isn't one next to "dood.al".

Currently "dood.al" is the canonical name, and is set to "hard". On the first link it says if the canonical name is set to "off ... The network attempts to avoid the use of a site's canonical name" - that doesn't sound great, since I would like "dood.al" to appear in the browser bar when someone is using the site. (At least, that was my reasoning when I set it up - I don't really understand what this is about.)

I had seen the second link - the six month thing is why I'm worried about getting something wrong here. If I set up https and redirect automatically to it, I suppose that will happen to all pages on the domain. Does that mean I need to setup a new https version of existing sites, or is this something I don't need to worry about?

And my general question is - how is this enable TLS button related to the script for using Let's Encrypt? Do I need to do both?

Thanks again for helping with this.

[oahda]

Most lightweight, easy to use (header only / no linking) C++ library for cross-platform (Mac/Linux/Windows) file system management (adding, removing, copying and browsing files and directories)? Found this, but that's not its only purpose, so it might be too much.

EDIT:
Ah, of course there's one coming for C++17... q_q

[JWki]

Most lightweight, easy to use (header only / no linking) C++ library for cross-platform (Mac/Linux/Windows) file system management (adding, removing, copying and browsing files and directories)? Found this, but that's not its only purpose, so it might be too much.

EDIT:
Ah, of course there's one coming for C++17... q_q

The one coming is boost::filesystem essentially isn't it?

[oahda]

Most lightweight, easy to use (header only / no linking) C++ library for cross-platform (Mac/Linux/Windows) file system management (adding, removing, copying and browsing files and directories)? Found this, but that's not its only purpose, so it might be too much.

EDIT:
Ah, of course there's one coming for C++17... q_q

The one coming is boost::filesystem essentially isn't it?

Yeah, and since this is only for a local tool anyway, I might be okay with linking with Boost's file system lib for now, so I was just downloading Boost here... But I can't figure out how the heck to do Boost stuff. I think I need to both compile the file system lib somehow and then link against it?

EDIT:
Figured it out.