Author Topic: Ruby 1.8 vs 1.9 incompatibilities  (Read 5287 times)

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Ruby 1.8 vs 1.9 incompatibilities
« on: August 20, 2013, 07:40:24 PM »
Hi!

I have been working on a project for almost a year now, whose goal it is to create an open source implementation
of RGSS1 (aiming to fully implement RGSS2/3 in the future).
As for ruby backends, I originally started out with mruby because the API was incredibly easy to work with, but soon realized that there are
still bugs and, more importantly, syntactical differences between mruby (which tries to adhere to the spec) and MRI, which makes playing
games with the standard RMXP scripts impossible (ie. text boxes don't work at all). So I decided to implement an MRI backend
next. In my eternal ignorance I assumed I could just checkout the latest version from git (2.0rc) and work with that. It appears to
at least work decently with the default scripts (have been able to play through "KN_E", the official RMXP sample game), but as
soon as I tried a user created game things didn't work anymore. In this particular case (To the Moon demo), the problem was
following hash constructor syntax in an advanced message script:

Code: [Select]
hash = { "key1", "value1", "key2", "value2" }I soon found out that while this syntax was perfectly ok in 1.8, it got removed in 1.9. Here's a list of other things that changed
between those versions.

I was wondering how the ARC devs are going to deal with this, as the project is described using 1.9 while claiming to be fully
compatible with all RMXP scripts (once converted) out there? Or does the converter you guys wrote take care of the language incompatibilities?

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #1 on: August 20, 2013, 09:15:38 PM »
There are a few things that we took care of, but some things will require a few minor edits.

Here are some references:
http://forum.chaos-project.com/index.php/topic,9665.0.html
http://forum.chaos-project.com/index.php/topic,10474.0.html
http://forum.chaos-project.com/index.php/topic,10382.0.html
http://forum.chaos-project.com/index.php/topic,12180.0.html
http://forum.chaos-project.com/index.php/topic,10294.0.html
http://forum.chaos-project.com/index.php/topic,10284.0.html
http://forum.chaos-project.com/index.php/topic,9447.0.html

Though, most of these are engine edits/fixes, not stuff between Ruby 1.8 and 1.9. There was also that safeguard that you can't edit an array while you are iterating through it, but I can't find that topic anymore. In any case, we want to preserve Ruby 1.9's functionality, not 1.8.
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #2 on: August 20, 2013, 11:28:04 PM »
Thanks for your reply Blizzard.

It seems you guys have hit your heads against some walls already that I didn't even notice. To be honest,
my first reaction when seeing the list of incompatibilities was to just scratch it and write yet another (1.8)
script backend, but it feels like the farther you go back in MRI's commit history, the more hostile towards
embedding it becomes. Not to mention all the speed and general benefits of newer ruby versions I'd have
to throw away... Right now 1.8 doesn't even compile cleanly for me for whatever reason >_>
Seriously, the longer I worked on this project, the more I started questioning Enterbrain's decision to use
Ruby in a game engine (I don't think there was even a valid reason to use it over something like Lua besides
the whole national pride thing). Only very recently has this started becoming a viable choice with the advent
of projects like mruby. Oh well.

By the way, you somewhere wrote that "flash_data" in Tilemaps doesn't do anything: it does, although it's
almost never used. It basically tints specific tiles in a color, so you can eg. show in what range a character
can move/attack/cast a spell. Let me know if you want to know details.

Offline Ryex

  • Arctic Bird of Programming
  • Global Moderator
  • Chaos Ultimate
  • ****
  • Posts: 5131
  • LV: 197
  • Gender: Male
  • Wants to write a compiler for fun
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #3 on: August 21, 2013, 12:23:13 AM »
By the way, you somewhere wrote that "flash_data" in Tilemaps doesn't do anything: it does, although it's
almost never used. It basically tints specific tiles in a color, so you can eg. show in what range a character
can move/attack/cast a spell. Let me know if you want to know details.

Wait, WHAT!?!?! please enlighten us to this compleately undocumented functionality so that we may implimet it and improve acordingly
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.

DropBox, the best free file syncing service there is.
(click to show/hide)

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #4 on: August 21, 2013, 12:37:05 AM »
Yes, please. I have never seen this data being used anywhere.
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Online KK20

  • Master Scripter Fixer
  • Global Moderator
  • Lexima Warrior
  • ****
  • Posts: 2990
  • LV: 369
  • Gender: Male
  • Bringer of Salt
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #5 on: August 21, 2013, 12:41:50 AM »
Quick test:
Code: [Select]
class Spriteset_Map
  #--------------------------------------------------------------------------
  # * Object Initialization
  #--------------------------------------------------------------------------
  def initialize
    # Make viewports
    @viewport1 = Viewport.new(0, 0, 640, 480)
    @viewport2 = Viewport.new(0, 0, 640, 480)
    @viewport3 = Viewport.new(0, 0, 640, 480)
    @viewport2.z = 200
    @viewport3.z = 5000
    # Make tilemap
    @tilemap = Tilemap.new(@viewport1)
    @tilemap.tileset = RPG::Cache.tileset($game_map.tileset_name)
    for i in 0..6
      autotile_name = $game_map.autotile_names[i]
      @tilemap.autotiles[i] = RPG::Cache.autotile(autotile_name)
    end
    @tilemap.map_data = $game_map.data
    @tilemap.priorities = $game_map.priorities
    # Make panorama plane
    @panorama = Plane.new(@viewport1)
    @panorama.z = -1000
    # Make fog plane
    @fog = Plane.new(@viewport1)
    @fog.z = 3000
    # Make character sprites
    @character_sprites = []
    for i in $game_map.events.keys.sort
      sprite = Sprite_Character.new(@viewport1, $game_map.events[i])
      @character_sprites.push(sprite)
    end
    @character_sprites.push(Sprite_Character.new(@viewport1, $game_player))
    # Make weather
    @weather = RPG::Weather.new(@viewport1)
    # Make picture sprites
    @picture_sprites = []
    for i in 1..50
      @picture_sprites.push(Sprite_Picture.new(@viewport3,
        $game_screen.pictures[i]))
    end
    # Make timer sprite
    @timer_sprite = Sprite_Timer.new
    @tilemap.flash_data = Table.new($game_map.width, $game_map.height)
    @tilemap.flash_data[5,5] = 0xf84
    
    # Frame update
    update
  end
.........
Made a yellow-ish square over the tile at map coordinates (5,5). Its level of opacity goes up and down as well.
Code: [Select]
@tilemap.flash_data[5,5] = 0x000
Doing that removed it entirely.

Flashes over everything with a priority of zero, regardless of layer tile is on.
« Last Edit: August 21, 2013, 12:48:00 AM by KK20 »



Other Projects
RPG Maker XP AceUpgrade RMXP to RMVXA performance!
XPA TilemapTilemap rewrite with many features, including custom resolution!


NNID: KK20-CP
Discord: KK20 Tyler#8901
Join the CP Discord Server

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #6 on: August 21, 2013, 12:53:51 AM »
Do you think it simply renders a square over the tile or does it modulate the existing tiles with that color?
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Online KK20

  • Master Scripter Fixer
  • Global Moderator
  • Lexima Warrior
  • ****
  • Posts: 2990
  • LV: 369
  • Gender: Male
  • Bringer of Salt
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #7 on: August 21, 2013, 01:00:01 AM »
You can easily recreate a similar effect with a simple 32x32 sprite changing its opacity levels.
(click to show/hide)
So my guess would be the first one.
« Last Edit: August 21, 2013, 01:01:21 AM by KK20 »



Other Projects
RPG Maker XP AceUpgrade RMXP to RMVXA performance!
XPA TilemapTilemap rewrite with many features, including custom resolution!


NNID: KK20-CP
Discord: KK20 Tyler#8901
Join the CP Discord Server

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #8 on: August 21, 2013, 01:19:00 AM »
Okay, let's start with the documented part:
Quote
flash_data
Refers to the flash data (Table) used when showing range of possible movement in simulation games, etc.
Defines a 2-dimensional array measuring [ horizontal size * vertical size ]. This array must be the same size as the map data.
Each element uses 4 bits to represent a tile's flash color in RGB; for example, 0xf84 flashes in RGB(15,8,4).

The undocumented stuff:
So, 15 is the maximum value a color component can have. The real maximum value this translates to however is 60,
assuming 255 is the absolute (or 60/255, assuming 1.0 is the max as in eg. OGL).
So a flash_data value of 0xF84 translates to a real color of RGB(60, 32, 16). This color is rendered as an additively
blended quad at the tile position. Therefore a value of 0x0 (black) doesn't show up at all.
This quad is rendered directly over the ground (z=0) tiles, so it appears below ones with >0 priority.

Some more unnecessary stuff:
Even though the doc says the flash table x and y dimensions must match the map size, this isn't even true. The way
flash data is apparently sampled is with wrap around, ie. the color for tile (x, y) is not sampled as flash_data[x, y]
but flash_data[x % xsize, y % ysize]. This means that you could attach a flash table with 1x1 size and basically
tint the entire tilemap in the same color.

This is how it looks in my engine:
Code: [Select]
bool sampleFlashColor(Vec4 &out, int x, int y)
{
const int _x = x % flashData->xSize();
const int _y = y % flashData->ySize();

qint16 packed = flashData->at(_x, _y);

if (packed == 0)
return false;

const float max = 60.0 / 255.0;
const float step = max / 0xF;

float b = ((packed & 0x000F) >> 0) * step;
float g = ((packed & 0x00F0) >> 4) * step;
float r = ((packed & 0x0F00) >> 8) * step;

out = Vec4(r, g, b, 1); // normalized!

return true;
}

That is what I was able to determine with a bit of trying out, but I'm 96% it's correct (visually it perfectly matches).

Offline Ryex

  • Arctic Bird of Programming
  • Global Moderator
  • Chaos Ultimate
  • ****
  • Posts: 5131
  • LV: 197
  • Gender: Male
  • Wants to write a compiler for fun
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #9 on: August 21, 2013, 01:24:44 AM »
very interesting

if we implement this functionality we should write a few helper methods to aid in it's use.
for example handling color conversions and generating flashmaps.
I no longer keep up with posts in the forum very well. If you have a question or comment, about my work, or in general I welcome PM's. if you make a post in one of my threads and I don't reply with in a day or two feel free to PM me and point it out to me.

DropBox, the best free file syncing service there is.
(click to show/hide)

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #10 on: August 21, 2013, 08:25:44 AM »
Yeah, that's some interesting information. I'll be sure to implement this functionality when I find some time since we want to mimic RGSS with ARC LE.
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #11 on: August 21, 2013, 02:43:10 PM »
This might be getting off topic a bit, but you guys haven't by chance found out what the "checksum" attached
to each script in Scripts.rxdata is / how it is computed, do you? I'm pretty sure it's related to the Zlib compression,
but playing around with some of the crc32 functions from the ruby module didn't yield anything comparable the
last time I tried...

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #12 on: August 21, 2013, 03:32:16 PM »
The CRC32 checksum is usually used only to check the validity of the data. If the stored checksum and the calculated match, then it's almost 100% sure that there is no data corruption. You can google how it is computed. Here is a C++ implementation.

Code: (cpp) [Select]
static unsigned int crc32_table[256];
static bool crc32_table_created = false;
void create_crc32_table()
{
if (crc32_table_created)
{
return;
}
unsigned int polynome = 0xEDB88320;
unsigned int sum;
for (int i = 0; i < 256; i++)
{
sum = i;
for (int j = 0; j < 8; j++)
{
if ((sum & 1) != 0)
{
sum = (sum >> 1) ^ polynome;
}
else
{
sum >>= 1;
}
}
crc32_table[i] = sum;
}
crc32_table_created = true;
}

unsigned int calc_crc32(unsigned char* data, long size)
{
create_crc32_table();
unsigned int crc = 0xFFFFFFFF;
for (unsigned long i = 0; i < size, i++)
{
crc = (crc >> 8) ^ crc32_table[(crc ^ data[i]) & 0xFF];
}
return ((crc & 0xFFFFFFFF) ^ 0xFFFFFFFF);
}


(for_iter is like a basic )
« Last Edit: August 21, 2013, 03:39:05 PM by Blizzard »
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #13 on: August 21, 2013, 03:43:42 PM »
IDK, but you can easily decompress the separate scripts without having to think about that. CRC32 is usually used only to check the validity of the data. If the stored checksum and the calculated match, then it's almost 100% sure that there is no data corruption.

Well, the idea that it's actually a checksum was more or less a guess by me anyway because there's not much
else a random number could be when some sort of compression is involved nearby. The last time I experiemnted
with it was around September 2012.. I don't really remember how deep I went with it tbh. (I should try again)
And yeah, it hasn't really had any impact on my work whatsoever so I've been ignoring it for all this time ^^
I was simply curious and thought you guys might have found something already..

But if it's there, it means the RMXP engine has to do some sort of checking with it. I actually have been toying
with the idea of writing a script editor that can read directly from and save to Scripts.rxdata files
(because I got tired of booting into my Windows VM everytime I wanted to do changes), so calculating the
correct checksum would be important for such a case.

Edit: I see you edited in a crc32 algorithm (thanks!). But did you actually check that this matches up with
the number found in Scripts.rxdata?

Edit2: I just tried your crc32 code (which probably doesn't differ much from the built in stuff in Ruby's Zlib) on
both encrypted and decrypted data, and neither match with the stored value =/
So I think Enterbrain is doing something slightly different there after all.
« Last Edit: August 21, 2013, 03:55:57 PM by Ancurio »

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #14 on: August 21, 2013, 03:58:57 PM »
As I said, the checksum doesn't have anything to do with Ruby, it's something a compressor adds. There are different implementations for CRC32. e.g. if you use a different polynome, you will get different results already. Either way, it's doesn't really matter from a developer's perspective. It's only used internally to verify compressed data, Zlib checks it automatically.
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #15 on: August 21, 2013, 04:21:30 PM »
As I said, the checksum doesn't have anything to do with Ruby, it's something a compressor adds.

I know that ^^ I think we might be talking past each other?

It's only used internally to verify compressed data, Zlib checks it automatically.

Oh, that's new for me actually.. does that mean a checksum is already embedded in the compressed
bitstream? Then there would be no point in supplying it separately though...

By the way, I just tried something: I set the "checksum" for the first script in a default game (Game_Temp)
to zero using a hex editor, opened the game with RMXP, did some changes in the script and saved it all back.
And lo and behold: RMXP happily read the zero value and actually saved it back into Scripts.rxdata
without either checking it or changing it despite the changes I did in the script. I think my theory of it
being a checksum is crumbling after all =P

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #16 on: August 21, 2013, 04:56:30 PM »
Lol! Well, maybe something was disabled when compiling Zlib into it, you never know. In either case, the checksum is not relevant to anybody.
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.

Offline Ancurio

  • Trained Member
  • *
  • Posts: 16
  • LV: 0
    • View Profile
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #17 on: August 28, 2013, 12:42:47 PM »
Sorry for hijacking this thread again. I used this day's morning to make some research into how
text is drawn onto bitmaps and I think the stuff I found might be useful to you guys as I see you've
hit the same problems when implementing blt and stretch_blt (the blending algorithm appears to be
the same). Do you want me to post here, make a new thread or post in Blizzards original thread about
the topic?

Offline Blizzard

  • This sexy
  • Administrator
  • has over 9000 posts
  • *****
  • Posts: 19917
  • LV: 642
  • Gender: Male
  • Magic midgets.
    • View Profile
    • You're already on it. (-_-')
Re: Ruby 1.8 vs 1.9 incompatibilities
« Reply #18 on: August 28, 2013, 01:18:06 PM »
Post in my original thread. This would be very helpful if you have a more accurate formula.
Check out Daygames and our games:

King of Booze      King of Booze: Never Ever      Pet Bots
Drinking Game for Android      Never have I ever for Android      Pet Bots for Android
Drinking Game for iOS      Never have I ever for iOS      Pet Bots for iOS
Drinking Game on Steam


Quote from: winkio
I do not speak to bricks, either as individuals or in wall form.

Quote from: Barney Stinson
When I get sad, I stop being sad and be awesome instead. True story.