Data format.

[Ryex]

we should decide on at least a general format for passing data between the editor and the engine. keep in mind that the editor is made in Python while the engine runs ruby so useing ruby's marshal or python's pickel isn't going to work. our options follow:

Public Standard Formats


YAML: "YAML is a human friendly data serialization standard for all programming languages."

pros:
- its a public standard and ported to many languages including ruby, python, and C++ (if we want to load the data on that side for what ever reason). this means that if we ever needed to switch languages for the editor or anything like that we're good to go and data loading wouldn't be a problem. 3ed parties could also make external editors ect.
- it's human readable & editable, you could edit the data contained with a text editor if you want to.
- it's uses an external c library (in most implementations, there are a few that use pure ruby or pure python ect.) to load objects so it's fast.


cons:
- it's a public standard people will have no trouble loading the data (theft)
- human readable means directly translates to a much larger file size
- it uses a external library that is system dependent

there are of course other corss languages data standard like JSON ect. but if we go with any of them most of the time these same pros and cons will aplay


Roll our Own


the alternative is to make our own. it's not hard really, now that I'me written a system that serializes objects between the two languages I know most of the particulars, but it can get messy.

if we make out own there are several methods to consider

storage format

Binary or Human Readable?

binary: means smaller file size and a slight speedup.

human readable: means larger file size, slower because it would most likely use regexp

data struture

Message or Object Format?

message format: the dump would be made up of messages with arbitrary names, to load and dump objects object & message handlers would have to be set up for each message and object type. this is the most portable format as it takes the structure with it.

object linking format: the dump is a list of object with it's class name and associated attributes. while this one take less work to set up it also means that you might have to do a bunch of special case handling to makes sure you translate the classes and data structure properly.



the rmpy format I developed for rmpy is an example of what we don;t want to do if we chose to make out own it's slow and unwieldy to set up even if it dose work.

thoughts?

[G_G]

I thought the editor was going to be coded in C++ but based from the work you had already done.

[Ryex]

@Ryex: I think that under these circumstances and your given arguments, Python is definitely a better choice for the editor itself. Just be aware the regardless of the code you wrote for RMPY, it is very likely that the ARC editor will be a lot different from RMPY and a restructured a lot.

[Blizzard]

The editor does not have high performance requirements. If the underlying rendering system was made in C or C++, it doesn't matter anyway. We should just make sure that the code structure internally is very robust and that we have a properly thought out design. RMPY was already made in Python and it was working fine so it's ok if we use Python for it. This will make porting it to other platforms a lot easier because there practically will be no extra work required.

As for the data format...
We are already using Ruby. I think the best choice would be to go with Ruby Marshall. The problem is that the whole Ruby Marshalling procedure would have to be implemented in Python. Have you thought about a Ruby made editor and wxRuby, Ryex? This would pretty much solve our problem with the interchange format.

[Ryex]

I did and it was how I stared off with RMPY but I soon realized that the wxRuby binding wasn't stable or developed enough to be used. and we did get the marshal format working in ruby but for some reason it started failing when reading the files remember? we could never figure out what was wrong.

[Blizzard]

Then I guess we will either have implement our own format or implement Ruby Marshall properly in Python. Thoughts?

[G_G]

Well see thats why I thought we were coding the editor in C++ since we have already successfully linked Ruby with it. Besides we need to be able to import RMXP projects anyways so...

[Blizzard]

I'm still thinking whether to make the importer a separate application. I think that would be the best. We'll see.

[Ryex]

the current way I import projects in rmpy is it starts a ruby process that loads the project files and then exports them to a format I can read. I  then load those files and do what  I need

[Blizzard]

Doesn't that mean we have both Python and Ruby in the editor?

[Ryex]

Doesn't that mean we have both Python and Ruby in the editor?

well I'm not saying it's good, in fact I feel its unwieldy and stupid but in a sense yes. the editor spawns a new process to run ruby so that the files can be converted. personalty I think providing a script that they can place in their game and then run the game to export the files should be enough if we roll our own formatter. or we could create an import program of some sort.  I'm sure we can come up with all sorts of ways to convert existing projects the what format should they end up in? thats the question.  from the editors point of view it is extremely unwieldy, slow, and annoying to have to go through an out side process just so that the editor can save the project in a playable format.  

[Blizzard]

I was actually thinking of the external app that it simply generates a full fledged ARC project. Then you can just double click on the ARC project file and it will run your once RMXP project in ARC.

We still need to decide what format we are going to use for the data files. I'm thinking that maybe XML would be a good idea. Then, when people "compile" their project, we can translate everything into a binary format such as Ruby Marshall. What do you think about that? Of course, this will create new problems such as when to read XML and when .rbm (let's just call it that way, it's like .rxdata).

[Ryex]

but see there is still that extra step of having to translate the data format to marshal before the game can be played or tested. which I don;t see the point to.

if we created our own binary format then we wouldn't need to that extra step. and it would load just nearly as fast. we could always translate it to marshal for a final release I suppose just to get that extra speed boost but there almost wouldn't be a point.

[ForeverZer0]

I'm leaning towards the external converter. Have it do a one-time conversion of an RMXP project into an ARC project, and no more worries.
Not be negative towards RMPY, God knows Ryex put a lot of time into it and it is good, but is it really the appropriate answer for what we are looking for?  I realize it will save us a lot of work, but in the end is it really the best way having to use Python just to save some work? I would say that if we had to make the editor in C++, which would obviously take more time, but in the end we eliminated a bunch of problems and had a better product, wouldn't it be worth it?

Another idea would be to figure out what the Marshalling bug was with WxRuby. I never personally had a problem with it, but since this is a community of people who are more familiar with Ruby than any other language, think what that would allow the end user to do if was written in Ruby. Any plug-in system we made would easily allow for any current scripter to customize his/her or editor, and/or create plug-ins for others. That means more plug-ins because more people can make them, and more interest in ARC.

I don't want to convey that I am against RMPY, I am by no means against it, I just don't want to allow ourselves to act as if we are stuck to it, even if it is going to affect the quality of the end-product due to compatibility issues.

[G_G]

I'm leaning towards the external converter. Have it do a one-time conversion of an RMXP project into an ARC project, and no more worries.
Not be negative towards RMPY, God knows Ryex put a lot of time into it and it is good, but is it really the appropriate answer for what we are looking for?  I realize it will save us a lot of work, but in the end is it really the best way having to use Python just to save some work? I would say that if we had to make the editor in C++, which would obviously take more time, but in the end we eliminated a bunch of problems and had a better product, wouldn't it be worth it?

Another idea would be to figure out what the Marshalling bug was with WxRuby. I never personally had a problem with it, but since this is a community of people who are more familiar with Ruby than any other language, think what that would allow the end user to do if was written in Ruby. Any plug-in system we made would easily allow for any current scripter to customize his/her or editor, and/or create plug-ins for others. That means more plug-ins because more people can make them, and more interest in ARC.

I don't want to convey that I am against RMPY, I am by no means against it, I just don't want to allow ourselves to act as if we are stuck to it, even if it is going to affect the quality of the end-product due to compatibility issues.

Thats what I was thinking as well. I'm still keeping in mind that since we have C++ linked with Ruby we could just keep the Ruby marshal format for the data format.

[Ryex]

perhaps your misunderstanding.

if we make our own binary format we can do a one time conversion from RMXP and be done. it would only take me three or four hours max to write a working phraser and emitter for both ruby and python and I'm willing to bet that it would be fairly fast (not as fast as marshal most likely but still sufficiently fast).we would just need to get a standard for the data format written down. there is no reason that the engine can't use ruby marshal for save game or anything like that in fact that would be preferred. but realistically it neither a problem nor that much of an inconvenience to write our own format too communicate between the editor and the engine.  

also I'm vehemently against using c++ to write the editor.

reasons to use python over c++ and ruby
- we don't need the power
- it's far FAR easier to write in a high level language
- - this means faster edits and easer for others to extend
- - easily portable to other platforms
- - - the only reliable way to distribute a ruby application is a thing called crate and it requires that you be able to build ruby as a static library and bind all the scripts into  sql light databases meaning you have to build it anew for every platform and it is for the most part untested.
- - - with python there are utilities that quickly build executables for any platform that are tested and proven to work.
- the libraries bindings are more developed and more stable than ruby's (the more developed is a VERY big thing, all the extras such as opengl contexts and the AUI manager that are key aren't stable or even touched in ruby's binding of wx)

the benefits of using a high level language shouldn't be cast off just so we can use ruby's marshal. their just too large.


just so every one understands this,
we write our own format:
an external program (probably compiled with orca) would be used to transform the project into a ARC project using out format (yes it would only run on window (possibly under wine in Linux) but If you have RMXP you have windows right?). ARC would then be able to load and save the project with no additional trouble.

now I realize that I'm partially bias as I put a TON of work into the existing code base that was once RMPY. but hopeful I'm presenting a logical argument .

[Blizzard]

While I have developed a certain disliking of Python due to working with Python/C++ systems that really cause a lot of memory problems and general structure problems, I think that the editor should be done in a higher level language because it will make things a lot easier, especially plugins. The only two realistic options (C# and .NET is not an option) are Python and Ruby. The advantages of Ruby is that we won't have to write our own file format while Python would provide us with significantly more stability. My main concern about writing our own format is that I know exactly how problematic things can become if you upgrade the format or if there are conceptual bugs in the format. That's why I would like avoid a new format at all costs. There is no way to make a simple format that also fulfills our requirements.

The converter program can be just a big Python script compiled into an exe with any kind of program like py2exe.

Using Python for the editor is definitely the better choice here, but we have to figure out a good way to solve the data format problem. Since we're 2 vs. 2 here, we have to come to an agreement about the editor first.

[Ryex]

hmm? the only data I can conceive us storing are numbers, strings, arrays, and hashes. using a message format where the messages have arbitrary names that can be interrupted by the target any way it wishes I see no problems that could arise.

most problems are encountered when costume data structures are used. if we use a message based system  then these structures are suddenly arbitrary and the problems are avoided.

[Blizzard]

You're wrong. We will also have to store other objects (such as a Game_Map containing Game_Events, Game_Events containing Game_EventCommands, etc.). A dictionary based format would be universal but also the hardest to implement.

[G_G]

I stated what I think we should do but I'll leave the final decision with you three since all three of you know more than I do so I really don't have any valid arguments.