next »

[bluej774]

When you guys and girls make a game with a decent amount of assets (images, music, etc) and you don't want to leave them just loose, how do you package them? Zip? 7zip? PAK? PK4?  Or do you package them?  Is there a good reason not to?

[Ivan]

I use Physfs to make archives I can access at runtime for my assets. It interates pretty smoothly with standard file i/o and so is a pretty great choice.

http://icculus.org/physfs/


(My engine actually uses it to create a cross-platform "executable" file)

[bluej774]

thx.  Looks like a solid choice.

[Core Xii]

I compile everything into one custom data file.

[moi]

Sometimes I just leave them like that, but I change something in the files or encrypt each file so that it becomes difficult to open with a paint package or sthg.

[Oddball]

I think it all depends on the game and the assets. I only package them up if they are worth protecting/encrypting, but even then a determined hacker will get to them. Some games I've left completely open in case anyone wanted to mod the game, which they did.

[David Pittman]

I package everything into a single file with a custom format. It's a pretty basic format: a header, a table of file entries (filenames and offsets), and then a big old blob of file data. Most of the files in the blob are compressed with zlib, except for music files, which I just copy straight in so that they can be streamed directly from the package file.

At runtime, I initialize the game from a single package file. File requests are first looked up in this package, then on disk if they're not in the package. I might change this system at some point to let me use multiple package files in some given order, but I don't really have a reason yet. It would be useful for patching or modding a game, but not otherwise.

[InvisibleMan]

As has been said, it depends on the game.  For the most part, I have left the files in the open.  Because of this, modding became rather popular in one of the online games that I worked on.  I am planning on making a custom file format to store resources for future projects though.  If we want to officially support modding, we can release tools that allow it, or leave moddable resources outside.

You shouldn't count on custom resource files to prevent hacking.  What it will do however is filter out a lot of the script-kiddy "noise".  You aren't going to stop someone who knows what he is doing, but at least you'll cut down on the amount of "I HACKED THIS!  I AM LEET!" posts.

[Will Vale]

I've used ZIP files before with some success, for a game with a large asset set. We had the ability to stack multiple ZIP files and load raw files from disk as David suggests above. The nice thing about this is that modders can unpackage your files and package their own pretty easily, and you can use raw files or ZIP plus some raw files during development.

We had two performance problems:

1) Disk seek time, which was solved by ordering the files to suit the game's requirements (write out a list of files as they are opened, then repack the ZIP to use this order).

2) Index lookup time. We built our own index by hashing each filename, and storing a table of hash/offset pairs in hash order. We then used a binary chop to get to the right place quickly.

Other than that we were very happy. I think we left streams outside the ZIPs since there weren't so many and it seemed the easiest solution

Cheers,

Will

[Pirate Hearts]

I wrote a custom file format to glomp assets together with no compression.  Then at runtime, I can specify an ordered list of sources in which to look for assets.  These sources may either be package files or folders containing loose assets.  Being able to search through multiple sources allows me to separate content into default engine assets that I want to include with every project (default font for debugging, etc.) and game-specific assets.  Being able to access loose content on the disk is convenient for rapid iteration, but I disable this functionality in my "final release" build to ensure that all my content is packaged properly, as I don't want to include loose assets in a deliverable.

[Skomakar'n]

Is nobody willing to give an example of how their custom formats were built up?

[Pirate Hearts]

Mine's pretty much identical to the Quake .pak file format.  There's a header that tells the size of the directory.  Then there's the directory itself, which is an array that contains the filename of each packed file along with its size and position within the pack file.  Then this directory is followed by the raw data of each file.

[Will Vale]

Here's mine, pretty similar to Kyle's. I guess there aren't that many ways of doing this

Code:

#include "sil/main.h"

namespace sil {
namespace data {

///////////////////////////////////////////////////////////////////////////////
// Library format description
///////////////////////////////////////////////////////////////////////////////

enum
{
/// Identify library version 1
LIBRARY_MAGIC_1 = SIL_FOURCC('S','I','L','1'),

/// Identify library version 2
LIBRARY_MAGIC_2 = SIL_FOURCC('S','I','L','2'),

/// Identify library version 3
LIBRARY_MAGIC_3 = SIL_FOURCC('S','I','L','3'),

/// Current version
LIBRARY_MAGIC = LIBRARY_MAGIC_3,
};

/// File header
struct LibraryHeader
{
/// File ID
Nat32 magic;

/// Total file size (including header)
Nat32 size;

/// One index
struct Index
{
Nat32 count;
Nat32 offset;
};

/// Index for library entries (those with names)
Index entries;

/// Index for blocks
Index blocks;

/// Index for fixups
Index fixups;
};

/// One entry in the index
struct LibraryEntry
{
/// Identifier for the resource
Nat32 name_hash;

/// For looking up the type at runtime
Nat32 type_hash;

/// Location of the resource in the file
Nat32 offset;

union
{
/// Padding
Nat32 pad;

/// For storing the type at runtime (write zero into pad)
const rtti::IType *type;
};
};

/// One block record in the index
struct LibraryBlock
{
/// For looking up the type at runtime
Nat32 type_hash;

/// Location of the block in the file
Nat32 offset;

/// Number of entries in the block
Nat32 count;

/// Padding
Nat32 pad;
};

/// One fixup record in the index
struct LibraryFixup
{
/// Offset to the fixup.
Nat32 offset;
};

}; // namespace data
}; // namespace sil

This is intended for storing flat-packed objects as well as files, which is what the fixups and type hashes are for. You could ignore/remove these if you were only storing files.

Note that I'm storing hashes and not filenames. In a production environment the filenames would be in a separate file attached to the archive, so you could load them for verification in a development build, and throw them out in a final build. I haven't bothered to implement this yet though

Will

[kometbomb]

Is nobody willing to give an example of how their custom formats were built up?

The beginning of the file basically has a header that pretty much says at which byte some file is located, and after the header each file is concatenated. It's trivial to return e.g. FILE*'s that are fseeked to the beginning of the requested file.

However, you have to keep in mind sometimes file formats rely on EOF to know when the file has been completely read, this of course doesn't work if there are ten files after that file in the package.

In that case, you need a way to pass your own handlers to the loader, e.g. in the case of SDL you would use SDL_rwops and pass the handlers to SDL_LoadBMP_RW (the loader needs to have support for that). The idea is that for example an image loader will use those handlers to read data and simply doesn't care where that data is coming from. It just knows a function will return X bytes when asked. Then, your handlers keep track if it's at the end of a subfile in the package and return EOF accordingly.

Here's some SDL_rwops code (in particular, take a look at bnd_read(), bnd_seek() etc. which are the same as fread() and fseek()): http://code.google.com/p/klystron/source/browse/trunk/src/util/bundle.c

And here's the tool that bundles the files into one: http://code.google.com/p/klystron/source/browse/trunk/tools/makebundle/makebundle.c

Since a lot of the data I bundle is of my own formats, I could cheat a bit and made the file formats location agnostic in that they can just take a FILE* as a parameter and know how many bytes to read. But in the end I needed to ad rwops support because you can only cheat so much.

[Mikademus]

We had two performance problems:

1) Disk seek time, which was solved by ordering the files to suit the game's requirements (write out a list of files as they are opened, then repack the ZIP to use this order).

2) Index lookup time. We built our own index by hashing each filename, and storing a table of hash/offset pairs in hash order. We then used a binary chop to get to the right place quickly.

That's some gold advice there! Great thanks!