[Finished] [ARCed] MsgPack serialization modules for Python and Ruby

Blizzard · August 31, 2011, 02:42:07 am

I don't see a point for a list then. If we implement _arc_load and _arc_dump, then we can make it work the same way as _dump and _load to prevent the Renderables to be dumped. Everything else can be dumped without problems.

Ryex · August 31, 2011, 03:18:09 am

ok then.

Ryex · September 05, 2011, 01:10:46 am

ok, so i'm finished with bother versions of the module
I've had to edit the ruby version as there were a few things I missed that needed to be done to make it work across languages. also I had to facilitate a few fringe cases like nested classes and things like the table class.
the python version is a bit slower than the ruby version about 9 times slower on load but that was expected due to python naturally slower looping and the fact the fact that the python version of msgpack unpacks arrays as tuples which are imitable. so I have to write a deep conversion method to loop though nested lists of tuples and convert them to python lists that ate about a third of the load time. dump time on the python version is roughly twice the dump time on the ruby version

also I though I should point this out, despite being written in pure ruby arc files load THREE TIMES FASTER than marshal. you heard me right ARC games load and reconstitute objects three times faster than RMXP. that means that black time when you first open the game will be cut down by 2/3. impressive yes?
the ruby version dose however dump slightly slower so saving will take a bit longer. but hey it only slower by about a 1/3 so it's not that bad.

python's dumping of arc is about 1/5 slower than ruby's so not bad there.

I've tested both and they both work perfectly. I tested by using ruby to load a existing RMXP project and dump it into arc, from there I loaded the arc files back up and dumped back to RMXP and tested the files by loading them with RMXP and test playing. for python I took the arc files dumped from ruby and loaded them, I then dumped back to arc in a different location I then used ruby to load the python dumped files and dump to RMXP and used RMXP to open these new files.

if not for a bug in how my recreation of RMXP's Table class in ruby and how it dumps to marshal the games would be playable. tables when dumped from my ruby class are for some reason consided to be 3d even if both ysize and zsize are 1 thus the default scripts raise "wrong number of arguments" errors trying to use 2d indexes to fetch data.

but this isn't a problem because why on earth would you need to convert ARC files to RMXP anyway

test logs for you viewing pleasure

(Load: 0.321018 Seconds, Dump: 0.508029 Seconds)

ruby rmxp to arc test: Show

(Load: 1.0759999752 Seconds, Dump: 0.654000043869 Seconds)

python load and dump test: Show

(Load: 0.18401 Seconds, Dump: 0.736042 Seconds)

ruby arc to rmxp test: Show

that last one is converting python arc dumped files not ruby arc dumped files but the times are the same as it is the exact same data, the files are as far as I can tell identical

Ok now onto how to use the features

the methods of the ARC_Dump module are as such

dump(obj, io, redirects = {})
obj = the object to be dumped
io = an io object like an open file object
redirects = this is a hash mapping class names to alt names this is used to help with namespace issues with nested classes ect on the python side of ARC_Dump to can map a class name like RPG::System::Words to RGSS_RPG_System_Words and then on the python side of loading it map RGSS_RPG_System_Words to a name space path that will find the proper class object. you have to do the same mapping when you load it in ruby too (python deals with name spaces and nested classes very differently so this is VERY necessarily)

load(io, redirects = {} [,extended_namespace={}])
io = an io object like an open file object
redirects = this is a hash mapping class names to alt names this is used to help with namespace issues with nested classes ect on the python side of ARC_Dump to can map a class name like RPG::System::Words to RGSS_RPG_System_Words and then on the python side of loading it map RGSS_RPG_System_Words to a name space path that will find the proper class object. you have to do the same mapping when you load it in ruby too (python deals with name spaces and nested classes very differently so this is VERY necessarily)

extended_namespace = only available on the python side due to how python deals with namespace this option is here to allow the caller to pass the globals() dictionary extending the namespace within the module temporarily so that the class objects in the loaded object can be found.

all classes have the option of defining the following class methods in ruby or python.

_arc_dump
must return a string or other basic data type that can be packed via msgpack
it could also raise an error to stop the dump process

_arc_load
the methods don't have to do anything.
it accepts the object returned by _arc_dump and should set up the object with the contained data. keep in mind that at this point the only thing that has happened is that the object has been allocated the init method has not been called so any and all instance variables are currently set to nil.
dose not need to return the completed object.

_arc_dump and _arc_load work as a pair, if you define one you MUST define the other or you'll get errors.

_arc_exclude
should return a array or list of strings of the names of instance variables/attributes that should not be dumped with the object (more important for python where everything is public, but default python exclude any attribute prefaced by a _)

_prep_arc_dump
this method is called just before the object is dumped, it can be used to set up data (this is important for the python side of the table object I use it to convert a private attribute containing the numpy version of the data to a normal python list so it can be dumped normally). this method can be in place regardless whether or not _arc_dump or any of the other methods are defined

_post_arc_load
called after the object has been loaded (this is important for the python side of the table object I use it to convert the normal python list of data to a numpy version). this method can be in place regardless of any other methods.

Python is significantly different in how it handles nested classes and instance variables as well as name space thus python classes can define the following global attributes

_arc_class_path
should contain a string with the namespace path to take to find the class object the module and class names should be separated by "::" not "." as is the python default

_arc_instance_variables
should contain a list of strings of the names of attributes to dump with the object if it is provided no other attributes are dumped other wise all attributes in the objects __dict__ that are not linked to function objects or prefaced with a _ (names prefaced with a _ are by convention private in python) are dumped with the object.

I'm committing the files to SVN now

Blizzard · September 05, 2011, 02:24:45 am

So both methods now work in both languages? Then this task can be marked done.

Ryex · September 05, 2011, 02:32:01 am

yep, it all works.

Blizzard · September 11, 2011, 05:24:04 pm

I'll leave this task here as finished, even though the Python version of our new serialization procedure has not been implemented yet. I trust that you don't forget to do that, Ryex. xD

Ryex · September 11, 2011, 05:25:42 pm

working on it now

Blizzard · September 11, 2011, 05:29:40 pm

Alright, cool. If you feel up for it, you can go test the C++ version as well. Compile the engine and call ARC::Data.load instead of load_data in Scene_Title. Don't forget to convert the rxdata files using arc_data_dump.rb before that. You have a bunch of conversion scripts in the same folder.

Ryex · September 11, 2011, 05:36:14 pm

ah dam
you used the pack and unpack methods. now I have to go find their equivalents in python again. I remembers it wasn't too hard to hard to use but the Letter format designations were different.

EDIT: thank goodness I still have the files from the failed python ruby marshal implementation!

Blizzard · September 12, 2011, 02:04:51 am

Use "import struct" and the struct module has the pack and unpack methods. xD

Ryex · September 12, 2011, 12:27:39 pm

ya, I figured that out from my old files from the ruby marshal attempt

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

King of Booze 2		King of Booze: Never Ever
Drinking Game for Android		Never have I ever for Android
Drinking Game for iOS		Never have I ever for iOS

Chaos Project

[Finished] [ARCed] MsgPack serialization modules for Python and Ruby

Blizzard

August 31, 2011, 02:42:07 am #20

Ryex

August 31, 2011, 03:18:09 am #21

Ryex

September 05, 2011, 01:10:46 am #22

Blizzard

September 05, 2011, 02:24:45 am #23

Ryex

September 05, 2011, 02:32:01 am #24

Blizzard

September 11, 2011, 05:24:04 pm #25

Ryex

September 11, 2011, 05:25:42 pm #26

Blizzard

September 11, 2011, 05:29:40 pm #27

Ryex

September 11, 2011, 05:36:14 pm #28 Last Edit: September 11, 2011, 05:56:37 pm by Ryex

Blizzard

September 12, 2011, 02:04:51 am #29

Ryex

September 12, 2011, 12:27:39 pm #30