Zlib: IronRuby VS Ruby

Started by ForeverZer0, August 17, 2011, 11:17:30 pm

Previous topic - Next topic

ForeverZer0

I am having a bit of a problem that I can't seem to figure out.  I am using IronRuby's Zlib Library in a .NET application through embedded to compress strings. Here's a sample of a script I am using, and calling through the application:


load_assembly 'IronRuby.Libraries', 'IronRuby.StandardLibrary.Zlib'

def load(path)
  file = File.open(path, 'rb')
  object = Marshal.load(file)
  file.close
  return object
end

def save(object, path)
  file = File.open(path, 'wb')
  Marshal.dump(object, file)
  file.close
end

def decompress(string)
  return Zlib::Inflate.inflate(string)
end

def compress(string)
  zlib = Zlib::Deflate.new(Zlib::BEST_COMPRESSION)
  compressed = zlib.deflate(string, Zlib::FINISH)
  return compressed
end


This works perfectly fine.  It also works for inflating objects that were Marshalled with Ruby 1.8.6. The problem arises when I compress and Marshal the string with IronRuby, then attempt to read the data in Ruby 1.8.6.  The compressed data is different, and I get "incorrect data check" errors.  It still loads fine in IronRuby, though, so I'm not sure if it has something to do with different versions of the Zlib library, so if is simply an incompatibility between the languages.  I have tried compressing the data in various ways as well, using a "--Zlib::MAX_WBITS", and every possible compression level, all still to no avail.  Its getting rather frustrating...  :???:

Any ideas?
I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

G_G

I remember having issues with this. I finally got it working after downloading a different version of IronRuby. If it helps you can use the libraries from my Price Fix app I posted awhile ago. It has the IronRuby libraries packed with it. http://forum.chaos-project.com/index.php/topic,9777.0.html Hope it'll help.

Blizzard

Yeah, if it cannot be read by Ruby 1.8.6, then it's a new version of Marshall/Zlib for Ruby that's being used by Iron Ruby. Using an older Iron Ruby version should do the trick.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

ForeverZer0

Using IronRuby 1.0.0.0 with .NET 2.0 is still not doing the trick.  It just seems to compress the data differently.  I have also noticed that the compressed data is significantly larger than standard Ruby's.  Either way, the Ruby Zlib library fails at reading it.  It still throws the "incorrect data check" exception.

I'll just say what I'm trying to do.  I created an external RMXP/RMVX script editor.  It has a lot of neat features that the standard one does not have.  Its not a full blown IDE, but it is a definite improvement.  So as you can see, the ability to save the Scripts.rxdata file back into a format that can be read is a pretty pivotal function.  :P

Everything else seems to work perfectly, except when you open RMXP's script editor, all the text for the scripts is blank.  Stranger yet, even if you write new text into them and save it, the data still does not show up.  At first I though perhaps I screwed up the sorta "ID' that RMXP assigns to each script, defined in the first element of each scripts array (in Scripts.rxdata), bur even when I tried loading the file from another project, you can clearly see the data as it is compressed, but any attempt to use Zlib::Inflate on it wil just throw the error. It doesn't do it when you start the editor, so I imagine Enterbrain must have included some catch in it to bypass any thrown Exceptions on the decompressing process.

Either way, I am very disappointed.  It seems my only choice now is to try and find some Zlib library that is not native to .NET, and try to interface it.   :???:
I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

Blizzard

Nah, it's most probably just a version conflict. Enterbrain never did anything smarter than the little bit of encryption in the RGSS.dll. If I remember right, in Ruby 1.8.6 the Marshall version is "48".
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

G_G

Can you run the game with the modified Scripts?

ForeverZer0

August 20, 2011, 07:59:12 pm #6 Last Edit: August 20, 2011, 08:00:34 pm by ForeverZer0
I also just noticed that RMXP uses Ruby 1.8.1.  I knew that before, but for some stupid reason, I have been thinking it is 1.8.6.
The Marshal version is still 4.8, though.

What gets me is that both the IronRuby Zlib version, and the RMXP Zlib version are both 0.6.0.
Is it possible I am messing something up with the "id".  To be honest, I am unsure exactly how it is used.  I am just assigning a random integer to it.  I know of the existence of "magic numbers" that Ruby uses, but I have never looked into how they are used.

EDIT:
@GG: No, it does nothing, same as if you erased all the scripts.


I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

G_G

August 20, 2011, 08:07:22 pm #7 Last Edit: August 20, 2011, 08:08:34 pm by game_guy
Scratch that. Forget this post.

ForeverZer0

No, Scripts.rxdata is an array of arrays.  Each inner array is a three element array:  [ID, "TITLE", "STRING"]

EDIT:

Hahah, I saw your fail before you could delete it  :V:
I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

Blizzard

What was it, what was it? D:
Yeah, you're right. RMXP uses 1.8.1. I keep forgetting that and always keep thinking it's 1.8.6.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

ForeverZer0

Ok, I am seriously at a loss.  I have been trying for 2 days to find why the hell this won't work. Both the Marshal and Zlib versions are exactly the same, and still nothing seems to work.  It has nothing to do with the ID, I eliminated that possibility. Its not a problem with the string encoding, both are in UTF-8. 

Has anybody successfully used Zlib::Inflate in RMXP on a string that was deflated with IronRuby's Zlib?  I'm sorry for the repeated whining, its just I have never been so thoroughly stumped on something that, from what I can see, should work.

I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

Blizzard

Deflate the same string in both versions and compare them. If they differ, something's off.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

ForeverZer0

RMXP:
s = "this is a string"
z = Zlib::Deflate.deflate(s)
p z


Result:


IronRuby:
var test = _engine.Execute(@"
           load_assembly 'IronRuby.Libraries', 'IronRuby.StandardLibrary.Zlib'"
s = 'this is a string'
z = Zlib::Deflate.deflate(s)
return z", _scope);
MessageBox.Show(test.ToString());


Result:


Now that appears to be different string encodings.  I'm not sure if the #ToString() method is messing with the text or not, so to be sure, I Marshall.dump the compressed string, and then Marshall.load it in RMXP to see what it looks like. 

IronRuby Code
_engine.Execute(@"
    load_assembly 'IronRuby.Libraries', 'IronRuby.StandardLibrary.Zlib'
    file = File.open('test.dat', 'wb')
    s = 'this is a string'
    z = Zlib::Deflate.deflate(s)
    Marshal.dump(z, file)
    file.close", _scope);


RMXP Code
test = load_data('test.dat')
p test


Result



And when I attempt to use Zlib::Inflate on the string...

RMXP Code
result = Zlib::Inflate.inflate(test)
p result


Result



The only thing left to do now that I could see that would make a difference is testing all the different compression levels.  I really don't see how that would make a difference, though, since Zlib::Inflate should inflate the string regardless of its compression level.


I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

Blizzard

Yeah, the encoding in the message boxes is different, but both strings loaded in Ruby look different when deflated. You'll have to research some more.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

ForeverZer0

August 21, 2011, 06:48:05 pm #14 Last Edit: August 22, 2011, 12:40:47 am by ForeverZer0
Well Gzip seems to work fine back and fourth between the two.  I can compress a string with IronRuby using it and load it up in RMXP just fine.  Unfortunately, this doesn't help with my issue...


EDIT:
Well, after spending about 5 MORE hours researching Zlib, posting on the IronRuby developer's forum, looking at one or two other projects that messed with the Scripts.rxdata file, and researching string encoding, I have come to the conclusion that currently IronRuby's Zlib library is not compatible with standard Ruby's library.

I have come up with a solution that will allow me use my .NET application to achieve what I need, but it is the most low-down dirty coding I have ever done.  I am going to try it and hide it so that I never have to face the shame of it. It will work, though, and at this point, that's good enough for me...
I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

Blizzard

Don't hide it. This kind of coding will be unavoidable in some real life scenarios. e.g. While we were using Python, PyOgre and CEGUI, we had to do a couple of nasty hacks ourselves to make it work properly. :/ That's the disadvantage when you have to work with an existing system that has design flaws.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Ryex

let me guess. you went the long way around. calling in an third party perhaps exploited a bug or two, did something that really should not of worked had the system been designed properly. all in the name of success?

I've had to do something similar when getting pyglet to render to a WXPython windows opengl context. the result is stable but i really shouldn't of had to go about it the way I did. far as I know I'm the first person to actually do it. or at least the first person to SHARE how they did it.
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.<br /><br />DropBox, the best free file syncing service there is.<br />

ForeverZer0

I'll never tell.

Okay, I lied: ShowHide
Here is what I did:

  • Using my .NET app, I output each script's text into a file, whose name is its unique id (0 - int::MAX), and save it to a single directory

  • I then write a "keyfile" that is basically a list where each key looks something like this: "SCRIPT_ID - SCRIPT_TITLE

  • The last line of the keyfile is the directory of Scripts.rxdata/Scripts.rvdata for the current project

  • On open, the application unpacks RGSS102E.dll, Game.exe, Game.ini, and Game.rgssad to a local directory. The script within it is written to parse the keyfile, load the script, compress, set to an array, and save to the passed directory

  • In my application, it starts Game.exe as a hidden process, and the actual save is done by RMXP

  • After save, all the files are deleted, and the the RMXP files are deleted when the application closes.

I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.

G_G

Could always make a few ruby scripts and use ocra to convert them to executables.
http://ocra.rubyforge.org/

Run them in the background, do a process watch or something, and so on and so fourth. I've done it this way before and it works pretty good.

ForeverZer0

Nah, I thought of that, but then I would have to embed the Ruby interpreter, and the filesize would  shoot up.  This way, its only about 800 KB difference.
I am done scripting for RMXP. I will likely not offer support for even my own scripts anymore, but feel free to ask on the forum, there are plenty of other talented scripters that can help you.