[Finished] [ARCed] MsgPack serialization modules for Python and Ruby

[Ryex]

MsgPack serialization modules for Python and Ruby

Description

As a way to store data and transfer it between Python and Ruby we are using MsgPack serialization. MsgPack can only hander primitive types (ints, bools, floats, strings) and two container types (maps, and arrays). as such we need a module that can transform user defined class objects into a format that can be serialized by MsgPack. and then reconstruct these user defined objects. (namely the data structures in the RPG module in RMXP, but the format should handle other objects too)


Priority

Medium.


Prerequisites

None


Assigned

None


Everything else

this kind of task requires knowledge of both languages' object construction, an under standing of how serialization works, and understanding of what cyclic references are any how to handle them. looking at the RMPY serialization module in the editor folder (contained in the files RMPY.rb and RMPY.py for Ruby and Python respectively) may help.

[Blizzard]

Ryex, when do you think you will have a data definition ready?

[Ryex]

... I'll start work on it. it will be done by the end of this next week.

[Blizzard]

That would be great. This is the only dependent part between engine and editor. When this is done, I can finish the engine regardless of the editor.

[nathmatt]

so how exactly is this going to work just wondering because this is the main thing besides knowledge of python that you would need to know add a config for scripts directly in the editor

[Blizzard]

Well, we are basically substituting the Marshal format for database stuff such as actor data. The point is that implementing Marshal would be a lot harder and take a lot longer so we will be using the MsgPack format to avoid that. Save games and other stuff will still be using Marshal. We might give script writers will also have access to MsgPack not just for loading the database data but also for writing their own files.

[nathmatt]

i meant more as data wise will it more like making ruby classes in c or would you add the class to your script and use a data format

so if im reading MsgPack right, it can take classes and serialize them so they can run on other formats ?

[Blizzard]

We will replace the load_data method and make a MsgPack class in C++ that will be exposed to Ruby which will have the same interface as Marshal (dump and load methods). You can use it then the same way you would use Marshal.

[nathmatt]

ok so yea it will take the needed information from the class thats needed to recreate it on another language

[Ryex]

basically yes. it will deconstruct the class object and reduce it to a list of properties and a class name and then use msgpack to serialize the data. when you want to recreate the object it will take the class anme find the right class create an object and then set all the properties. so you will end up with an object that looks and acts the same as the deconstructed object.

[nathmatt]

yea that is what i thought that should be really useful so i would make a python class that would hold all the config information serialize it and load it up on ruby will ruby need a duplicate class with same name to recreate it ?   

[Ryex]

yes

[Blizzard]

Worst case, we could write the MsgPack module in Ruby instead of C++ so you could pretty much use it anywhere, not just in ARC. It would be kind of simpler in C++, though, because C++ is designed for low level operations. I'll see about that.

[Ryex]

well crap, I got dumping done, and I thought I had loading right, but I screwed something up. all the data is correct when you pull the object list out of the ARC file but I messed up converting it back to proper objects.
I'll work on it again later.

on the bright side despite the fact that it's written in pure ruby it runs fast. at least compared to my old RMPY format. good god that was slow. but ya this system actually has a decent speed. less than a second to load up and dump the test project data.

[ForeverZer0]

[Blizzard]

Be careful with that. You don't know whether you dumped the data incorrectly or loaded it incorrectly. Take both possibilities into consideration when trying to fix the problem.

lol sorry, I managed to click modify instead of reply. stupid G-MOD powers letting me edit an admin's post :/

[Ryex]

No I looked at the data after it was dumped. I loaded the msgpack object and inspected it, it's correct.

basically I turned the dumped object into a hash linking an id to an array where the first element was an number that identified the object type and the second element was data to reconstruct the object

the error is somewhere in my process of taking the data from that hash and turning into objects.

I actually probably need to redo system a bit so that numbers just get dumped strait instead of separated off in the hash. (I was lazy and ripped the structure form RMPY) for most numbers it would increase the amount of data written to the file :/

that at least is simple to change. the other problem is going to take a bit of testing.

[Ryex]

ok, it works now. I made some mistakes in the placement of [] that had the system accessing things wrong. also I had the system dumping local object ids instead of converting them id relative to the file which was leading to nil values. tested it and it work perfectly in both detections.

less than a second to load up and dump the test project's data in either direction compared to the 30+ seconds that RMPY needed.

only the ruby version is written right now but the python version should be close behind.

also it is currently working on a white list system which is bad. I need a way to recognize if the system can properly dump the object outside of pre approving it by adding the class name to the list. I'd much prefer to use a black list but I'm really not sure how to effectively do that it would be even better to not use any sort of list at all and just recognize properties of the object that make it non serialize-able and fail if it finds such an object.

I'll commit the files to the SVN shortly

[Blizzard]

Fine by me. I can translate the Ruby version into ARC's engine and port your Ruby code to C++ Ruby code.

I always get confused with the lists. Black list is the one that keeps track of already dumped objects, right? In any case, this is the kind of list that should be used. There should be no properties that are non-serializable. We just have to prevent people from serializing Sprites, etc. I think we need an equivalent of the _dump and _load methods. How about _arc_dump and _arc_load? We can add that those methods should throw errors when using with Sprites, etc.

[Ryex]

a white list is a list of objects that are allowed through, everything else gets blocked (not good for this use case). a black list is a list of objects to block everything else gets through (better than a white list but then you have to make sure everything that can't be serialized get's onto the list.