next »

[bluej774]

[deleted]

[Ivan]

I use Physfs to make archives I can access at runtime for my assets. It interates pretty smoothly with standard file i/o and so is a pretty great choice.

http://icculus.org/physfs/


(My engine actually uses it to create a cross-platform "executable" file)

[Core Xii]

I compile everything into one custom data file.

[moi]

Sometimes I just leave them like that, but I change something in the files or encrypt each file so that it becomes difficult to open with a paint package or sthg.

[Oddball]

I think it all depends on the game and the assets. I only package them up if they are worth protecting/encrypting, but even then a determined hacker will get to them. Some games I've left completely open in case anyone wanted to mod the game, which they did.

[David Pittman]

I package everything into a single file with a custom format. It's a pretty basic format: a header, a table of file entries (filenames and offsets), and then a big old blob of file data. Most of the files in the blob are compressed with zlib, except for music files, which I just copy straight in so that they can be streamed directly from the package file.

At runtime, I initialize the game from a single package file. File requests are first looked up in this package, then on disk if they're not in the package. I might change this system at some point to let me use multiple package files in some given order, but I don't really have a reason yet. It would be useful for patching or modding a game, but not otherwise.

[ITS_Mike]

As has been said, it depends on the game.  For the most part, I have left the files in the open.  Because of this, modding became rather popular in one of the online games that I worked on.  I am planning on making a custom file format to store resources for future projects though.  If we want to officially support modding, we can release tools that allow it, or leave moddable resources outside.

You shouldn't count on custom resource files to prevent hacking.  What it will do however is filter out a lot of the script-kiddy "noise".  You aren't going to stop someone who knows what he is doing, but at least you'll cut down on the amount of "I HACKED THIS!  I AM LEET!" posts.

[Will Vale]

I've used ZIP files before with some success, for a game with a large asset set. We had the ability to stack multiple ZIP files and load raw files from disk as David suggests above. The nice thing about this is that modders can unpackage your files and package their own pretty easily, and you can use raw files or ZIP plus some raw files during development.

We had two performance problems:

1) Disk seek time, which was solved by ordering the files to suit the game's requirements (write out a list of files as they are opened, then repack the ZIP to use this order).

2) Index lookup time. We built our own index by hashing each filename, and storing a table of hash/offset pairs in hash order. We then used a binary chop to get to the right place quickly.

Other than that we were very happy. I think we left streams outside the ZIPs since there weren't so many and it seemed the easiest solution

Cheers,

Will

[J. Kyle Pittman]

I wrote a custom file format to glomp assets together with no compression.  Then at runtime, I can specify an ordered list of sources in which to look for assets.  These sources may either be package files or folders containing loose assets.  Being able to search through multiple sources allows me to separate content into default engine assets that I want to include with every project (default font for debugging, etc.) and game-specific assets.  Being able to access loose content on the disk is convenient for rapid iteration, but I disable this functionality in my "final release" build to ensure that all my content is packaged properly, as I don't want to include loose assets in a deliverable.

[oahda]

Is nobody willing to give an example of how their custom formats were built up?

[J. Kyle Pittman]

Mine's pretty much identical to the Quake .pak file format.  There's a header that tells the size of the directory.  Then there's the directory itself, which is an array that contains the filename of each packed file along with its size and position within the pack file.  Then this directory is followed by the raw data of each file.

[Will Vale]

Here's mine, pretty similar to Kyle's. I guess there aren't that many ways of doing this

Code:

#include "sil/main.h"

namespace sil {
namespace data {

///////////////////////////////////////////////////////////////////////////////
// Library format description
///////////////////////////////////////////////////////////////////////////////

enum
{
	/// Identify library version 1
	LIBRARY_MAGIC_1 = SIL_FOURCC('S','I','L','1'),

	/// Identify library version 2
	LIBRARY_MAGIC_2 = SIL_FOURCC('S','I','L','2'),

	/// Identify library version 3
	LIBRARY_MAGIC_3 = SIL_FOURCC('S','I','L','3'),

	/// Current version
	LIBRARY_MAGIC = LIBRARY_MAGIC_3,
};

/// File header
struct LibraryHeader
{
	/// File ID
	Nat32 magic;

	/// Total file size (including header)
	Nat32 size;

	/// One index
	struct Index
	{
		Nat32 count;
		Nat32 offset;
	};

	/// Index for library entries (those with names)
	Index entries;

	///	Index for blocks
	Index blocks;

	/// Index for fixups
	Index fixups;
};

/// One entry in the index
struct LibraryEntry
{
	/// Identifier for the resource
	Nat32 name_hash;

	/// For looking up the type at runtime
	Nat32 type_hash;

	/// Location of the resource in the file
	Nat32 offset;

	union
	{
		/// Padding
		Nat32 pad;

		/// For storing the type at runtime (write zero into pad)
		const rtti::IType *type;
	};
};

/// One block record in the index
struct LibraryBlock
{
	/// For looking up the type at runtime
	Nat32 type_hash;

	/// Location of the block in the file
	Nat32 offset;

	/// Number of entries in the block
	Nat32 count;

	/// Padding
	Nat32 pad;
};

/// One fixup record in the index
struct LibraryFixup
{
	/// Offset to the fixup. 
	Nat32 offset;
};

}; // namespace data
}; // namespace sil

This is intended for storing flat-packed objects as well as files, which is what the fixups and type hashes are for. You could ignore/remove these if you were only storing files.

Note that I'm storing hashes and not filenames. In a production environment the filenames would be in a separate file attached to the archive, so you could load them for verification in a development build, and throw them out in a final build. I haven't bothered to implement this yet though

Will

[Mikademus]

We had two performance problems:

1) Disk seek time, which was solved by ordering the files to suit the game's requirements (write out a list of files as they are opened, then repack the ZIP to use this order).

2) Index lookup time. We built our own index by hashing each filename, and storing a table of hash/offset pairs in hash order. We then used a binary chop to get to the right place quickly.

That's some gold advice there! Great thanks! 

[BrianSlipBit]

The real benefit to compressing data on disk is that it can effectively half (reduce by ~50%) your load times.  It's done by double buffering.  You decompress out of the last loaded buffer while the next buffer is being loaded off disk.