we should decide on at least a general format for passing data between the editor and the engine. keep in mind that the editor is made in Python while the engine runs ruby so useing ruby's marshal or python's pickel isn't going to work. our options follow:
Public Standard Formats
YAML: "YAML is a human friendly data serialization standard for all programming languages."
pros:
- its a public standard and ported to many languages including ruby, python, and C++ (if we want to load the data on that side for what ever reason). this means that if we ever needed to switch languages for the editor or anything like that we're good to go and data loading wouldn't be a problem. 3ed parties could also make external editors ect.
- it's human readable & editable, you could edit the data contained with a text editor if you want to.
- it's uses an external c library (in most implementations, there are a few that use pure ruby or pure python ect.) to load objects so it's fast.
cons:
- it's a public standard people will have no trouble loading the data (theft)
- human readable means directly translates to a much larger file size
- it uses a external library that is system dependent
there are of course other corss languages data standard like JSON ect. but if we go with any of them most of the time these same pros and cons will aplay
Roll our Own
the alternative is to make our own. it's not hard really, now that I'me written a system that serializes objects between the two languages I know most of the particulars, but it can get messy.
if we make out own there are several methods to consider
storage format
Binary or Human Readable?
binary: means smaller file size and a slight speedup.
human readable: means larger file size, slower because it would most likely use regexp
data struture
Message or Object Format?
message format: the dump would be made up of messages with arbitrary names, to load and dump objects object & message handlers would have to be set up for each message and object type. this is the most portable format as it takes the structure with it.
object linking format: the dump is a list of object with it's class name and associated attributes. while this one take less work to set up it also means that you might have to do a bunch of special case handling to makes sure you translate the classes and data structure properly.
the rmpy format I developed for rmpy is an example of what we don;t want to do if we chose to make out own it's slow and unwieldy to set up even if it dose work.
thoughts?
I thought the editor was going to be coded in C++ but based from the work you had already done.
Quote from: Blizzard@Ryex: I think that under these circumstances and your given arguments, Python is definitely a better choice for the editor itself. Just be aware the regardless of the code you wrote for RMPY, it is very likely that the ARC editor will be a lot different from RMPY and a restructured a lot.
The editor does not have high performance requirements. If the underlying rendering system was made in C or C++, it doesn't matter anyway. We should just make sure that the code structure internally is very robust and that we have a properly thought out design. RMPY was already made in Python and it was working fine so it's ok if we use Python for it. This will make porting it to other platforms a lot easier because there practically will be no extra work required.
As for the data format...
We are already using Ruby. I think the best choice would be to go with Ruby Marshall. The problem is that the whole Ruby Marshalling procedure would have to be implemented in Python. Have you thought about a Ruby made editor and wxRuby, Ryex? This would pretty much solve our problem with the interchange format.
I did and it was how I stared off with RMPY but I soon realized that the wxRuby binding wasn't stable or developed enough to be used. and we did get the marshal format working in ruby but for some reason it started failing when reading the files remember? we could never figure out what was wrong.
Then I guess we will either have implement our own format or implement Ruby Marshall properly in Python. Thoughts?
Well see thats why I thought we were coding the editor in C++ since we have already successfully linked Ruby with it. Besides we need to be able to import RMXP projects anyways so...
I'm still thinking whether to make the importer a separate application. I think that would be the best. We'll see.
the current way I import projects in rmpy is it starts a ruby process that loads the project files and then exports them to a format I can read. I then load those files and do what I need
Doesn't that mean we have both Python and Ruby in the editor?
Quote from: Blizzard on March 15, 2011, 12:05:23 pm
Doesn't that mean we have both Python and Ruby in the editor?
well I'm not saying it's good, in fact I feel its unwieldy and stupid but in a sense yes. the editor spawns a new process to run ruby so that the files can be converted. personalty I think providing a script that they can place in their game and then run the game to export the files should be enough if we roll our own formatter. or we could create an import program of some sort. I'm sure we can come up with all sorts of ways to convert existing projects the what format should they end up in? thats the question. from the editors point of view it is extremely unwieldy, slow, and annoying to have to go through an out side process just so that the editor can save the project in a playable format.
I was actually thinking of the external app that it simply generates a full fledged ARC project. Then you can just double click on the ARC project file and it will run your once RMXP project in ARC.
We still need to decide what format we are going to use for the data files. I'm thinking that maybe XML would be a good idea. Then, when people "compile" their project, we can translate everything into a binary format such as Ruby Marshall. What do you think about that? Of course, this will create new problems such as when to read XML and when .rbm (let's just call it that way, it's like .rxdata).
but see there is still that extra step of having to translate the data format to marshal before the game can be played or tested. which I don;t see the point to.
if we created our own binary format then we wouldn't need to that extra step. and it would load just nearly as fast. we could always translate it to marshal for a final release I suppose just to get that extra speed boost but there almost wouldn't be a point.
I'm leaning towards the external converter. Have it do a one-time conversion of an RMXP project into an ARC project, and no more worries.
Not be negative towards RMPY, God knows Ryex put a lot of time into it and it is good, but is it really the appropriate answer for what we are looking for? I realize it will save us a lot of work, but in the end is it really the best way having to use Python just to save some work? I would say that if we had to make the editor in C++, which would obviously take more time, but in the end we eliminated a bunch of problems and had a better product, wouldn't it be worth it?
Another idea would be to figure out what the Marshalling bug was with WxRuby. I never personally had a problem with it, but since this is a community of people who are more familiar with Ruby than any other language, think what that would allow the end user to do if was written in Ruby. Any plug-in system we made would easily allow for any current scripter to customize his/her or editor, and/or create plug-ins for others. That means more plug-ins because more people can make them, and more interest in ARC.
I don't want to convey that I am against RMPY, I am by no means against it, I just don't want to allow ourselves to act as if we are stuck to it, even if it is going to affect the quality of the end-product due to compatibility issues.
Quote from: ForeverZer0 on March 16, 2011, 12:43:54 am
I'm leaning towards the external converter. Have it do a one-time conversion of an RMXP project into an ARC project, and no more worries.
Not be negative towards RMPY, God knows Ryex put a lot of time into it and it is good, but is it really the appropriate answer for what we are looking for? I realize it will save us a lot of work, but in the end is it really the best way having to use Python just to save some work? I would say that if we had to make the editor in C++, which would obviously take more time, but in the end we eliminated a bunch of problems and had a better product, wouldn't it be worth it?
Another idea would be to figure out what the Marshalling bug was with WxRuby. I never personally had a problem with it, but since this is a community of people who are more familiar with Ruby than any other language, think what that would allow the end user to do if was written in Ruby. Any plug-in system we made would easily allow for any current scripter to customize his/her or editor, and/or create plug-ins for others. That means more plug-ins because more people can make them, and more interest in ARC.
I don't want to convey that I am against RMPY, I am by no means against it, I just don't want to allow ourselves to act as if we are stuck to it, even if it is going to affect the quality of the end-product due to compatibility issues.
Thats what I was thinking as well. I'm still keeping in mind that since we have C++ linked with Ruby we could just keep the Ruby marshal format for the data format.
perhaps your misunderstanding.
if we make our own binary format we can do a one time conversion from RMXP and be done. it would only take me three or four hours max to write a working phraser and emitter for both ruby and python and I'm willing to bet that it would be fairly fast (not as fast as marshal most likely but still sufficiently fast).we would just need to get a standard for the data format written down. there is no reason that the engine can't use ruby marshal for save game or anything like that in fact that would be preferred. but realistically it neither a problem nor that much of an inconvenience to write our own format too communicate between the editor and the engine.
also I'm vehemently against using c++ to write the editor.
reasons to use python over c++ and ruby
- we don't need the power
- it's far FAR easier to write in a high level language
- - this means faster edits and easer for others to extend
- - easily portable to other platforms
- - - the only reliable way to distribute a ruby application is a thing called crate and it requires that you be able to build ruby as a static library and bind all the scripts into sql light databases meaning you have to build it anew for every platform and it is for the most part untested.
- - - with python there are utilities that quickly build executables for any platform that are tested and proven to work.
- the libraries bindings are more developed and more stable than ruby's (the more developed is a VERY big thing, all the extras such as opengl contexts and the AUI manager that are key aren't stable or even touched in ruby's binding of wx)
the benefits of using a high level language shouldn't be cast off just so we can use ruby's marshal. their just too large.
just so every one understands this,
we write our own format:
an external program (probably compiled with orca) would be used to transform the project into a ARC project using out format (yes it would only run on window (possibly under wine in Linux) but If you have RMXP you have windows right?). ARC would then be able to load and save the project with no additional trouble.
now I realize that I'm partially bias as I put a TON of work into the existing code base that was once RMPY. but hopeful I'm presenting a logical argument .
While I have developed a certain disliking of Python due to working with Python/C++ systems that really cause a lot of memory problems and general structure problems, I think that the editor should be done in a higher level language because it will make things a lot easier, especially plugins. The only two realistic options (C# and .NET is not an option) are Python and Ruby. The advantages of Ruby is that we won't have to write our own file format while Python would provide us with significantly more stability. My main concern about writing our own format is that I know exactly how problematic things can become if you upgrade the format or if there are conceptual bugs in the format. That's why I would like avoid a new format at all costs. There is no way to make a simple format that also fulfills our requirements.
The converter program can be just a big Python script compiled into an exe with any kind of program like py2exe.
Using Python for the editor is definitely the better choice here, but we have to figure out a good way to solve the data format problem. Since we're 2 vs. 2 here, we have to come to an agreement about the editor first.
hmm? the only data I can conceive us storing are numbers, strings, arrays, and hashes. using a message format where the messages have arbitrary names that can be interrupted by the target any way it wishes I see no problems that could arise.
most problems are encountered when costume data structures are used. if we use a message based system then these structures are suddenly arbitrary and the problems are avoided.
You're wrong. We will also have to store other objects (such as a Game_Map containing Game_Events, Game_Events containing Game_EventCommands, etc.). A dictionary based format would be universal but also the hardest to implement.
I stated what I think we should do but I'll leave the final decision with you three since all three of you know more than I do so I really don't have any valid arguments.
Quote from: Blizzard on March 16, 2011, 03:36:31 am
You're wrong. We will also have to store other objects (such as a Game_Map containing Game_Events, Game_Events containing Game_EventCommands, etc.). A dictionary based format would be universal but also the hardest to implement.
I'm saying that we could abstract those objects to messages with arbitrary names containing raw data like numbers, strings, arrays, and hashes.
messages would be able to reference other messages in the file though
here's what I'm thinking.
you open a file and read the first 4 bytes. this long value is the ID of the first message in the file the next byte read is a type identifier 0-255 (I can't see us needing more that 255 types.)
for the purposes of this explanation lets say that a value of 1 indicates an integer, 2 a floating point number, 3 a string, 4 an array, 5 a hash, 6 an object, and 7 a reference to.
lets say the second byte read had a value of 6. the loader would then read the next 4 bytes to get a long value that was the length in bytes of the string that was the name of the message. it would then read that many bytes to get the name of the message.
the loader would then pass the io stream to a handler that is registered for that message name. the handler would read outs it's data and return the object it created from it. the loader would read the next byte after the message handler had returned assuming that the byte indicated an end to the current message which would be indicated by a value of 255 if it isn't 255 the loader know something went wrong. it would then read the next 4 bytes as the ID of the next message. and it would continue until it had read all the messages in the file. the las message should be the top level object the was dumped and would be returned as such.
this operates in a similar fashion to how my RMPY format works in that any object referenced in the currently read message should have been stated previously in the file. it requires that a handler be written for the dumping and loading of each user defined object you want to load and dump but it ends up being extremely portable
I don't see any problems that could arise. of course, I'm not a professorial so feel free to correct me I'll take it as a learning experience.
I don't understand what the messages would be. Is that similar to Ruby's Marshall format where types are indicated with letters? Or is this something else? One of the important things in this format is that it should never require some sort of eval call.
Here is a very, very good article on serialization: http://www.parashift.com/c++-faq-lite/serialization.html
You will come across the problem of multiple objects referencing one object and even circular references.
I actually see no problem in using an already existing format. It is much easier to use a developed and tested format than making up one's own, I already said that. Let's first consider using such formats before we decide to make our own.
that format avoids the problems of circular references each unique object would only appear once in the file and it should never require an eval call as it can get all the information it need by reading the bytes directly.
I too have no problems using a existing format, but what format do we use?
I have no idea. ._. There are so many different formats that it's ridiculous. I'd like to avoid XML, though. Does anybody have any suggestions? If not, somebody needs to research a bit. :P
ah I saw this once and I have been trying to find it.
http://msgpack.org/
I say that this is likely the best we'll ever get for size and speed. but we would have to reduce all our data structures to arrays and maps.
the other option is YAML which I talked about in the first post
EDIT:
other options include JSON (of course), google made something called protocall buffers. but in the end it come down to this
if you want to be able to store costume data structures you have three options. use YAML, write a new format, reformat the structures into arrays and maps (if you do this msgpack is the way to go, it the fastest and the most compact)
This is great, man. This library works for Ruby, Python and C++. And the license pretty much only requires us to credit to msgpack and include a copy of the license. This is exactly the solution for our problem that we have been looking for. Let's do this.
EDIT: Eh, what about object serialization then? Are we going to dump an identifier integer for the object type and then write a hash with variable names and values? IMO this seems kind of the most logical solution. I'll go check out YAML a bit.
EDIT: Hm, YAML seems way too open, that doesn't really work for us. I guess msgpack is the way to go. We will have to implement Ruby functions in C++ that will load msgpack files into our objects into the game. We will also have to implement Python functions for loading and saving our objects. Actually msgpacks comes down to a compromise. We kind of will make our own format, but we will use msgpack as a backbone.
sounds good. I agree that yaml is way too open too, hence why I listed that as a con in the first post. but ya msgpack should work great for us we just have to put together a frame work to process objects into messages.