[Finished] [ARCed] MsgPack serialization modules for Python and Ruby

Ryex · April 29, 2011, 08:20:06 am

MsgPack serialization modules for Python and Ruby

Description

As a way to store data and transfer it between Python and Ruby we are using MsgPack serialization. MsgPack can only hander primitive types (ints, bools, floats, strings) and two container types (maps, and arrays). as such we need a module that can transform user defined class objects into a format that can be serialized by MsgPack. and then reconstruct these user defined objects. (namely the data structures in the RPG module in RMXP, but the format should handle other objects too)

Priority

Medium.

Prerequisites

None

Assigned

None

Everything else

this kind of task requires knowledge of both languages' object construction, an under standing of how serialization works, and understanding of what cyclic references are any how to handle them. looking at the RMPY serialization module in the editor folder (contained in the files RMPY.rb and RMPY.py for Ruby and Python respectively) may help.

Blizzard · August 20, 2011, 04:01:37 pm

Ryex, when do you think you will have a data definition ready?

Ryex · August 20, 2011, 05:31:51 pm

... I'll start work on it. it will be done by the end of this next week.

Blizzard · August 20, 2011, 05:36:37 pm

That would be great. This is the only dependent part between engine and editor. When this is done, I can finish the engine regardless of the editor.

nathmatt · August 21, 2011, 08:56:31 am

so how exactly is this going to work just wondering because this is the main thing besides knowledge of python that you would need to know add a config for scripts directly in the editor

Blizzard · August 21, 2011, 09:15:01 am

Well, we are basically substituting the Marshal format for database stuff such as actor data. The point is that implementing Marshal would be a lot harder and take a lot longer so we will be using the MsgPack format to avoid that. Save games and other stuff will still be using Marshal. We might give script writers will also have access to MsgPack not just for loading the database data but also for writing their own files.

nathmatt · August 21, 2011, 09:25:05 am

i meant more as data wise will it more like making ruby classes in c or would you add the class to your script and use a data format

so if im reading MsgPack right, it can take classes and serialize them so they can run on other formats ?

Blizzard · August 21, 2011, 09:36:12 am

We will replace the load_data method and make a MsgPack class in C++ that will be exposed to Ruby which will have the same interface as Marshal (dump and load methods). You can use it then the same way you would use Marshal.

nathmatt · August 21, 2011, 09:39:17 am

ok so yea it will take the needed information from the class thats needed to recreate it on another language

Ryex · August 21, 2011, 03:05:13 pm

basically yes. it will deconstruct the class object and reduce it to a list of properties and a class name and then use msgpack to serialize the data. when you want to recreate the object it will take the class anme find the right class create an object and then set all the properties. so you will end up with an object that looks and acts the same as the deconstructed object.

nathmatt · August 21, 2011, 04:33:21 pm

yea that is what i thought that should be really useful so i would make a python class that would hold all the config information serialize it and load it up on ruby will ruby need a duplicate class with same name to recreate it ?

Ryex · August 21, 2011, 05:42:22 pm

yes

Blizzard · August 21, 2011, 06:04:30 pm

Worst case, we could write the MsgPack module in Ruby instead of C++ so you could pretty much use it anywhere, not just in ARC. It would be kind of simpler in C++, though, because C++ is designed for low level operations. I'll see about that.

Ryex · August 29, 2011, 01:11:51 am

well crap, I got dumping done, and I thought I had loading right, but I screwed something up. all the data is correct when you pull the object list out of the ARC file but I messed up converting it back to proper objects.
I'll work on it again later.

on the bright side despite the fact that it's written in pure ruby it runs fast. at least compared to my old RMPY format. good god that was slow. but ya this system actually has a decent speed. less than a second to load up and dump the test project data.

ForeverZer0 · August 29, 2011, 01:24:55 am

Blizzard · August 29, 2011, 02:32:01 am

Be careful with that. You don't know whether you dumped the data incorrectly or loaded it incorrectly. Take both possibilities into consideration when trying to fix the problem.

lol sorry, I managed to click modify instead of reply. stupid G-MOD powers letting me edit an admin's post :/

Ryex · August 29, 2011, 04:14:20 am

No I looked at the data after it was dumped. I loaded the msgpack object and inspected it, it's correct.

basically I turned the dumped object into a hash linking an id to an array where the first element was an number that identified the object type and the second element was data to reconstruct the object

the error is somewhere in my process of taking the data from that hash and turning into objects.

I actually probably need to redo system a bit so that numbers just get dumped strait instead of separated off in the hash. (I was lazy and ripped the structure form RMPY) for most numbers it would increase the amount of data written to the file :/

that at least is simple to change. the other problem is going to take a bit of testing.

Ryex · August 30, 2011, 02:50:39 pm

ok, it works now. I made some mistakes in the placement of [] that had the system accessing things wrong. also I had the system dumping local object ids instead of converting them id relative to the file which was leading to nil values. tested it and it work perfectly in both detections.

less than a second to load up and dump the test project's data in either direction compared to the 30+ seconds that RMPY needed.

only the ruby version is written right now but the python version should be close behind.

also it is currently working on a white list system which is bad. I need a way to recognize if the system can properly dump the object outside of pre approving it by adding the class name to the list. I'd much prefer to use a black list but I'm really not sure how to effectively do that it would be even better to not use any sort of list at all and just recognize properties of the object that make it non serialize-able and fail if it finds such an object.

I'll commit the files to the SVN shortly

Blizzard · August 30, 2011, 04:21:11 pm

Fine by me. I can translate the Ruby version into ARC's engine and port your Ruby code to C++ Ruby code.

I always get confused with the lists. Black list is the one that keeps track of already dumped objects, right? In any case, this is the kind of list that should be used. There should be no properties that are non-serializable. We just have to prevent people from serializing Sprites, etc. I think we need an equivalent of the _dump and _load methods. How about _arc_dump and _arc_load? We can add that those methods should throw errors when using with Sprites, etc.

Ryex · August 30, 2011, 04:51:40 pm

a white list is a list of objects that are allowed through, everything else gets blocked (not good for this use case). a black list is a list of objects to block everything else gets through (better than a white list but then you have to make sure everything that can't be serialized get's onto the list.

Blizzard · August 31, 2011, 02:42:07 am

I don't see a point for a list then. If we implement _arc_load and _arc_dump, then we can make it work the same way as _dump and _load to prevent the Renderables to be dumped. Everything else can be dumped without problems.

Ryex · August 31, 2011, 03:18:09 am

ok then.

Ryex · September 05, 2011, 01:10:46 am

ok, so i'm finished with bother versions of the module
I've had to edit the ruby version as there were a few things I missed that needed to be done to make it work across languages. also I had to facilitate a few fringe cases like nested classes and things like the table class.
the python version is a bit slower than the ruby version about 9 times slower on load but that was expected due to python naturally slower looping and the fact the fact that the python version of msgpack unpacks arrays as tuples which are imitable. so I have to write a deep conversion method to loop though nested lists of tuples and convert them to python lists that ate about a third of the load time. dump time on the python version is roughly twice the dump time on the ruby version

also I though I should point this out, despite being written in pure ruby arc files load THREE TIMES FASTER than marshal. you heard me right ARC games load and reconstitute objects three times faster than RMXP. that means that black time when you first open the game will be cut down by 2/3. impressive yes?
the ruby version dose however dump slightly slower so saving will take a bit longer. but hey it only slower by about a 1/3 so it's not that bad.

python's dumping of arc is about 1/5 slower than ruby's so not bad there.

I've tested both and they both work perfectly. I tested by using ruby to load a existing RMXP project and dump it into arc, from there I loaded the arc files back up and dumped back to RMXP and tested the files by loading them with RMXP and test playing. for python I took the arc files dumped from ruby and loaded them, I then dumped back to arc in a different location I then used ruby to load the python dumped files and dump to RMXP and used RMXP to open these new files.

if not for a bug in how my recreation of RMXP's Table class in ruby and how it dumps to marshal the games would be playable. tables when dumped from my ruby class are for some reason consided to be 3d even if both ysize and zsize are 1 thus the default scripts raise "wrong number of arguments" errors trying to use 2d indexes to fetch data.

but this isn't a problem because why on earth would you need to convert ARC files to RMXP anyway

test logs for you viewing pleasure

(Load: 0.321018 Seconds, Dump: 0.508029 Seconds)

ruby rmxp to arc test: Show

(Load: 1.0759999752 Seconds, Dump: 0.654000043869 Seconds)

python load and dump test: Show

(Load: 0.18401 Seconds, Dump: 0.736042 Seconds)

ruby arc to rmxp test: Show

that last one is converting python arc dumped files not ruby arc dumped files but the times are the same as it is the exact same data, the files are as far as I can tell identical

Ok now onto how to use the features

the methods of the ARC_Dump module are as such

dump(obj, io, redirects = {})
obj = the object to be dumped
io = an io object like an open file object
redirects = this is a hash mapping class names to alt names this is used to help with namespace issues with nested classes ect on the python side of ARC_Dump to can map a class name like RPG::System::Words to RGSS_RPG_System_Words and then on the python side of loading it map RGSS_RPG_System_Words to a name space path that will find the proper class object. you have to do the same mapping when you load it in ruby too (python deals with name spaces and nested classes very differently so this is VERY necessarily)

load(io, redirects = {} [,extended_namespace={}])
io = an io object like an open file object
redirects = this is a hash mapping class names to alt names this is used to help with namespace issues with nested classes ect on the python side of ARC_Dump to can map a class name like RPG::System::Words to RGSS_RPG_System_Words and then on the python side of loading it map RGSS_RPG_System_Words to a name space path that will find the proper class object. you have to do the same mapping when you load it in ruby too (python deals with name spaces and nested classes very differently so this is VERY necessarily)

extended_namespace = only available on the python side due to how python deals with namespace this option is here to allow the caller to pass the globals() dictionary extending the namespace within the module temporarily so that the class objects in the loaded object can be found.

all classes have the option of defining the following class methods in ruby or python.

_arc_dump
must return a string or other basic data type that can be packed via msgpack
it could also raise an error to stop the dump process

_arc_load
the methods don't have to do anything.
it accepts the object returned by _arc_dump and should set up the object with the contained data. keep in mind that at this point the only thing that has happened is that the object has been allocated the init method has not been called so any and all instance variables are currently set to nil.
dose not need to return the completed object.

_arc_dump and _arc_load work as a pair, if you define one you MUST define the other or you'll get errors.

_arc_exclude
should return a array or list of strings of the names of instance variables/attributes that should not be dumped with the object (more important for python where everything is public, but default python exclude any attribute prefaced by a _)

_prep_arc_dump
this method is called just before the object is dumped, it can be used to set up data (this is important for the python side of the table object I use it to convert a private attribute containing the numpy version of the data to a normal python list so it can be dumped normally). this method can be in place regardless whether or not _arc_dump or any of the other methods are defined

_post_arc_load
called after the object has been loaded (this is important for the python side of the table object I use it to convert the normal python list of data to a numpy version). this method can be in place regardless of any other methods.

Python is significantly different in how it handles nested classes and instance variables as well as name space thus python classes can define the following global attributes

_arc_class_path
should contain a string with the namespace path to take to find the class object the module and class names should be separated by "::" not "." as is the python default

_arc_instance_variables
should contain a list of strings of the names of attributes to dump with the object if it is provided no other attributes are dumped other wise all attributes in the objects __dict__ that are not linked to function objects or prefaced with a _ (names prefaced with a _ are by convention private in python) are dumped with the object.

I'm committing the files to SVN now

Blizzard · September 05, 2011, 02:24:45 am

So both methods now work in both languages? Then this task can be marked done.

Ryex · September 05, 2011, 02:32:01 am

yep, it all works.

Blizzard · September 11, 2011, 05:24:04 pm

I'll leave this task here as finished, even though the Python version of our new serialization procedure has not been implemented yet. I trust that you don't forget to do that, Ryex. xD

Ryex · September 11, 2011, 05:25:42 pm

working on it now

Blizzard · September 11, 2011, 05:29:40 pm

Alright, cool. If you feel up for it, you can go test the C++ version as well. Compile the engine and call ARC::Data.load instead of load_data in Scene_Title. Don't forget to convert the rxdata files using arc_data_dump.rb before that. You have a bunch of conversion scripts in the same folder.

Ryex · September 11, 2011, 05:36:14 pm

ah dam
you used the pack and unpack methods. now I have to go find their equivalents in python again. I remembers it wasn't too hard to hard to use but the Letter format designations were different.

EDIT: thank goodness I still have the files from the failed python ruby marshal implementation!

Blizzard · September 12, 2011, 02:04:51 am

Use "import struct" and the struct module has the pack and unpack methods. xD

Ryex · September 12, 2011, 12:27:39 pm

ya, I figured that out from my old files from the ruby marshal attempt

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

[Finished] [ARCed] MsgPack serialization modules for Python and Ruby

April 29, 2011, 08:20:06 am Last Edit: September 09, 2011, 07:02:16 pm by Ryex

August 20, 2011, 04:01:37 pm #1

August 20, 2011, 05:31:51 pm #2

August 20, 2011, 05:36:37 pm #3

August 21, 2011, 08:56:31 am #4

August 21, 2011, 09:15:01 am #5

August 21, 2011, 09:25:05 am #6 Last Edit: August 21, 2011, 09:36:29 am by nathmatt

August 21, 2011, 09:36:12 am #7

August 21, 2011, 09:39:17 am #8

August 21, 2011, 03:05:13 pm #9

August 21, 2011, 04:33:21 pm #10

August 21, 2011, 05:42:22 pm #11

August 21, 2011, 06:04:30 pm #12 Last Edit: August 21, 2011, 06:05:38 pm by Blizzard

August 29, 2011, 01:11:51 am #13 Last Edit: August 29, 2011, 01:15:09 am by Ryex

August 29, 2011, 01:24:55 am #14

August 29, 2011, 02:32:01 am #15 Last Edit: August 29, 2011, 04:11:20 am by Ryex

August 29, 2011, 04:14:20 am #16 Last Edit: August 29, 2011, 04:24:32 am by Ryex

August 30, 2011, 02:50:39 pm #17 Last Edit: August 30, 2011, 04:48:44 pm by Ryex

August 30, 2011, 04:21:11 pm #18

August 30, 2011, 04:51:40 pm #19

August 31, 2011, 02:42:07 am #20

August 31, 2011, 03:18:09 am #21

September 05, 2011, 01:10:46 am #22

September 05, 2011, 02:24:45 am #23

September 05, 2011, 02:32:01 am #24

September 11, 2011, 05:24:04 pm #25

September 11, 2011, 05:25:42 pm #26

September 11, 2011, 05:29:40 pm #27

September 11, 2011, 05:36:14 pm #28 Last Edit: September 11, 2011, 05:56:37 pm by Ryex

September 12, 2011, 02:04:51 am #29

September 12, 2011, 12:27:39 pm #30