Data format.

[Ryex]

You're wrong. We will also have to store other objects (such as a Game_Map containing Game_Events, Game_Events containing Game_EventCommands, etc.). A dictionary based format would be universal but also the hardest to implement.

I'm saying that we could abstract those objects to messages with arbitrary names containing raw data like numbers, strings, arrays, and hashes.
messages would be able to reference other messages in the file though

here's what I'm thinking.

you open a file and read the first 4 bytes. this long value is the ID of the first message in the file the next byte read is a type identifier 0-255 (I can't see us needing more that 255 types.)
for the purposes of this explanation lets say that a value of 1 indicates an integer, 2 a floating point number, 3 a string, 4 an array, 5 a hash, 6 an object, and 7 a reference to.
lets say the second byte read had a value of 6. the loader would then read the next 4 bytes to get a long value that was the length in bytes of the string that was the name of the message. it would then read that many bytes to get the name of the message.
the loader would then pass the io stream to a handler that is registered for that message name. the handler would read outs it's data and return the object it created from it. the loader would read the next byte after the message handler had returned assuming that the byte indicated an end to the current message which would be indicated by a value of 255 if it isn't 255 the loader know something went wrong. it would then read the next 4 bytes as the ID of the next message. and it would continue until it had read all the messages in the file. the las message should be the top level object the was dumped and would be returned as such.

this operates in a similar fashion to how my RMPY format works in that any object referenced in the currently read message should have been stated previously in the file. it requires that a handler be written for the dumping and loading of each user defined object you want to load and dump but it ends up being extremely portable

I don't see any problems that could arise. of course, I'm not a professorial so feel free to correct me I'll take it as a learning experience.

[Blizzard]

I don't understand what the messages would be. Is that similar to Ruby's Marshall format where types are indicated with letters? Or is this something else? One of the important things in this format is that it should never require some sort of eval call.

Here is a very, very good article on serialization: http://www.parashift.com/c++-faq-lite/serialization.html

You will come across the problem of multiple objects referencing one object and even circular references.
I actually see no problem in using an already existing format. It is much easier to use a developed and tested format than making up one's own, I already said that. Let's first consider using such formats before we decide to make our own.

[Ryex]

that format avoids the problems of circular references each unique object would only appear once in the file and it should never require an eval call as it can get all the information it need by reading the bytes directly.

I too have no problems using a existing format, but what format do we use?

[Blizzard]

I have no idea. ._. There are so many different formats that it's ridiculous. I'd like to avoid XML, though. Does anybody have any suggestions? If not, somebody needs to research a bit.

[Ryex]

ah I saw this once and I have been trying to find it.
http://msgpack.org/

I say that this is likely the best we'll ever get for size and speed. but we would have to reduce all our data structures to arrays and maps.
the other option is YAML which I talked about in the first post

EDIT:
other options include JSON (of course), google made something called protocall buffers. but in the end it come down to this

if you want to be able to store costume data structures you have three options. use YAML, write a new format, reformat the structures into arrays and maps (if you do this msgpack is the way to go, it the fastest and the most compact)

[Blizzard]

This is great, man. This library works for Ruby, Python and C++. And the license pretty much only requires us to credit to msgpack and include a copy of the license. This is exactly the solution for our problem that we have been looking for. Let's do this.

EDIT: Eh, what about object serialization then? Are we going to dump an identifier integer for the object type and then write a hash with variable names and values? IMO this seems kind of the most logical solution. I'll go check out YAML a bit.

EDIT: Hm, YAML seems way too open, that doesn't really work for us. I guess msgpack is the way to go. We will have to implement Ruby functions in C++ that will load msgpack files into our objects into the game. We will also have to implement Python functions for loading and saving our objects. Actually msgpacks comes down to a compromise. We kind of will make our own format, but we will use msgpack as a backbone.

[Ryex]

sounds good. I agree that yaml is way too open too, hence why I listed that as a con in the first post. but ya msgpack should work great for us we just have to put together a frame work to process objects into messages.