[incomplete] ruby object serialization

Started by Ryex, December 14, 2010, 03:10:33 am

Previous topic - Next topic

Ryex

ok, so I just finished writing a serialization module for ruby so far I can only dump objects and not reconstitute them, why? because frankly reconstituting dumped objects is much more complicated then dumping them, plus I haven't had time to write that part yet. this is planed to be a fall back if I can't get ruby marshal working in python.
I have documented the dump format here https://github.com/Ryex/RPG-Maker-PY/wiki/Data-serialization

I designed this special so that it would avoid recursion and thus throw SystemStackError's if the stack level got too deep. as it is any object that would contain sub objects is queued so that it's sub objects aren't dumped until work on that object is finished.

http://pastebin.com/ETuKh7gu

man I learned a lot about the workings of ruby while working on this.

I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />

Blizzard

Very nice. So you decided going with an own serialization after all?
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Ryex

I'm still hoping to get marshal working in python, but considering the fact that the current bug is that the reader is some how reading the object links out of order I decided I was best to come up with another option. 

If I end up using this I'll automate it so that RPG Maker PY will start the ruby loader, which loads the rxdata files with marshal and then dumps with the rmpy format. then it will write to a .dumplog file that RPG Maker Py will read and know that the import worked and will load the dumped files itself.
going the other way it will call the ruby exporter and the process will go in reverse.

the ruby loader and exporter will of course be compiled executables if at all possible, no point in forcing the user to install the right version of ruby just so that the marshal version will be right.

it won't be as fast or as smooth as loading striate into python but it should work.
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />

Ryex

December 15, 2010, 01:05:30 am #3 Last Edit: December 15, 2010, 01:13:40 am by Ryex
I just want to say
SUCCESS

I can sterilize objects now (I just worked out all the bugs and did a successful test)
to compare I'll use the Actors.rxdata file with the default RTP database

Dump stats




statmarshalrmpy
time (s)0.0160009860992431640.10500597953796387
size (b)10,98161,377


keep in mind that marshal is written in pure c and rmpy in pure ruby also marshal is a binary format and rmpy is not

I think the result is pretty dam impressive

I updated the pastebin if you want to try it
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />

Ryex

December 16, 2010, 02:25:09 pm #4 Last Edit: December 16, 2010, 02:26:42 pm by Ryex
once again I report success. I finished the loading part of the module and it is working perfectly. best part is that I didn't need to use eval once!
Quote from: Ryex on December 15, 2010, 01:05:30 am
to compare I'll use the Actors.rxdata file with the default RTP database

Dump stats




statmarshalrmpy
time (s)0.0160009860992431640.10500597953796387
size (b)10,98161,377


keep in mind that marshal is written in pure c and rmpy in pure ruby also marshal is a binary format and rmpy is not

I think the result is pretty dam impressive

I updated the pastebin if you want to try it


and now for the load stats, once again with the actors.rxdata file with the default database

Load stats




statmarshalrmpy
time (s)0.0160009860992431640.022001028060913086
size (b)10,98161,377


strange that marshal is the same with both loading and dumping...

I updated the pastebin again if you want to try it
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />

Ryex

FUCK YA! loading the rmpy serialized objects on the python side works! though some how I broek the ruby module while changing a few things... small bugs to work out I guess
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />

ForeverZer0

I love the "change one little thing and whole thing goes to hell effect" that seems to be the rule when writing code. It never seems to fail...

Anyways, it sounds like it is coming along nicely.  :D
I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

Ryex

best part about this. I'm getting to know both languages's inner workings. I have a MUCH better picture of how they work and the differences in their implementation. object serialization... I know one thing if I make an engine in python it will NOT store data like RMXP dose.
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />