ok, so I just finished writing a serialization module for ruby so far I can only dump objects and not reconstitute them, why? because frankly reconstituting dumped objects is much more complicated then dumping them, plus I haven't had time to write that part yet. this is planed to be a fall back if I can't get ruby marshal working in python.
I have documented the dump format here https://github.com/Ryex/RPG-Maker-PY/wiki/Data-serialization (https://github.com/Ryex/RPG-Maker-PY/wiki/Data-serialization)
I designed this special so that it would avoid recursion and thus throw SystemStackError's if the stack level got too deep. as it is any object that would contain sub objects is queued so that it's sub objects aren't dumped until work on that object is finished.
http://pastebin.com/ETuKh7gu (http://pastebin.com/ETuKh7gu)
man I learned a lot about the workings of ruby while working on this.
Very nice. So you decided going with an own serialization after all?
I'm still hoping to get marshal working in python, but considering the fact that the current bug is that the reader is some how reading the object links out of order I decided I was best to come up with another option.
If I end up using this I'll automate it so that RPG Maker PY will start the ruby loader, which loads the rxdata files with marshal and then dumps with the rmpy format. then it will write to a .dumplog file that RPG Maker Py will read and know that the import worked and will load the dumped files itself.
going the other way it will call the ruby exporter and the process will go in reverse.
the ruby loader and exporter will of course be compiled executables if at all possible, no point in forcing the user to install the right version of ruby just so that the marshal version will be right.
it won't be as fast or as smooth as loading striate into python but it should work.
I just want to say
SUCCESS
I can sterilize objects now (I just worked out all the bugs and did a successful test)
to compare I'll use the Actors.rxdata file with the default RTP database
Dump stats
| stat | marshal | rmpy |
| time (s) | 0.016000986099243164 | 0.10500597953796387 |
| size (b) | 10,981 | 61,377 |
keep in mind that marshal is written in pure c and rmpy in pure ruby also marshal is a binary format and rmpy is not
I think the result is pretty dam impressive
I updated the pastebin if you want to try it
once again I report success. I finished the loading part of the module and it is working perfectly. best part is that I didn't need to use eval once!
Quote from: Ryex on December 15, 2010, 01:05:30 am
to compare I'll use the Actors.rxdata file with the default RTP database
Dump stats
| stat | marshal | rmpy |
| time (s) | 0.016000986099243164 | 0.10500597953796387 |
| size (b) | 10,981 | 61,377 |
keep in mind that marshal is written in pure c and rmpy in pure ruby also marshal is a binary format and rmpy is not
I think the result is pretty dam impressive
I updated the pastebin if you want to try it
and now for the load stats, once again with the actors.rxdata file with the default database
Load stats
| stat | marshal | rmpy |
| time (s) | 0.016000986099243164 | 0.022001028060913086 |
| size (b) | 10,981 | 61,377 |
strange that marshal is the same with both loading and dumping...
I updated the pastebin again if you want to try it
FUCK YA! loading the rmpy serialized objects on the python side works! though some how I broek the ruby module while changing a few things... small bugs to work out I guess
I love the "change one little thing and whole thing goes to hell effect" that seems to be the rule when writing code. It never seems to fail...
Anyways, it sounds like it is coming along nicely. :D
best part about this. I'm getting to know both languages's inner workings. I have a MUCH better picture of how they work and the differences in their implementation. object serialization... I know one thing if I make an engine in python it will NOT store data like RMXP dose.