next »

[Alec]

I'm curious what you folks you use for managing "PAK files", if you do. (i.e. compressing your data directories into a ZIP or equivalent file, and pretending its still a real directory structure)

I've looked into PhysFs. It scares me, because it hasn't been updated since 2005. (?)

MoleBox is nifty in that you don't have to change your code at all to use it. But it costs money and isn't cross platform.

[ravuya]

While PhysFS hasn't updated much since 2005, I doubt archive files have changed much since 1988.

I've seen some games use zziplib but I dunno how easy that is to use.

[DrDerekDoctors]

I use Allegro's datapacking functions which are cross-platform and haven't caused me any problems using them with OpenGL and FMOD.

[Alec]

Not interested in compression so much as just lumping the files together and making the code think its still reading things normally.

[DrDerekDoctors]

Yuss, that's what the Allegro pack functions do.

[Alec]

Hmm, don't you use specific functions to load from allegro pack files? (like pack_fopen)

MoleBox is cool cause you can use any normal method (fopen, std::ifstream) to read from the packed data.

[DrDerekDoctors]

Yuss, you do you particular functions although getting a handle to a file stream ain't exactly hard. As I say, I've had no problems getting it to interface with non-Allegro libraries like FMOD. I'm sure there are better solutions out there, but this is the one that I know about.

[Alec]

Its not hard, but its more work than doing not having to change anything.

Plus its allegro... a standalone library just for doing the pack stuff would be cool.

I wonder what Moonpod does for their stuff. cough

[DrDerekDoctors]

Is Moonpod cross-platform then? And yes, I agree that the drag of another DLL in with it is sucky.

[Madgarden]

The Allegro source code is open and free, and so I'm sure you could rip out the packfile code (which has endian awareness BTW, FTW) and adapt it to your needs.

Allegro can easily be static-linked as well, so no need for DLL hell.

[Alec]

Is Moonpod cross-platform then?

No, but I think they rolled their own code for it, and I'm curious how they did it.

[ravuya]

SDL has some similar code (SDL_rwops) for reading from concatenated package files.

[PoV]

Previously I've rolled my own, since I seem to have trust issues with other peoples code.  I haven't done it yet with my current game, but all my file IO is through a custom interface, so it'll be painless to add later.

How about some random advice?   

- For terms: Archives are a directory/dictionary/list of filenames with offsets to chunks, and many data chunks (files).  Compression comes later.
- Directories and subdirectories are just longer filenames.  Unlimited file name length means unlimited directories.
- Padding the start of data chunks to 4 byte boundaries has performance advantages on many platforms when in RAM.
- For random access, compress chunks individually.  Compressing everything at once tends to save more space, but you trade off random access.
- If there's room in RAM, cache the archive.  At the very least, caching the directory/dictionary can improve load times by not having to read it every time you request a file.
- If there's more room in RAM, compress the archive as a whole, and cache the uncompressed copy after loading.  But keep in mind, you need slightly more free memory than the sum of the uncompressed size and the compressed size.
- If running off a CD/DVD, seek time sucks.  If you can't cache everything important or commonly used in RAM, replicating your archive on different parts of the disk can improve seek time.  A simple check of what sector the last file request was on disk, and a little math to pick the closest to where the laser sits.
- Seek time on memory cards/flash memory is significantly better than CDs/DVDs, but random access is still slower than continuous access.
- Prefer (if possible) ordering data on disk/archive in the same order it's requested.  Though that's a bigger impact with CDs/DVDs, it's something that can be considered when talking about your own archiving.
- Storing all files in the same directory in the same general part of the archive can achieve the previous point, so long as that data is of "Level 1" or "Level 2" type scope.
- Accessing distinctly common data first, then level specific data can save seek time if the common data is far away on disc.
- Storing or interweaving streamed data together can save seek time (music, video, static geometry).

That's probably more than anyone would want to know about archives.