The Z values are calculated in the underlying library, which unlike Ruby is a strongly typed language, meaning you can just change a value around from one type to another as you please. An integer is an integer, a float is a float, and a variable containing one will never be another. I am quite certain that a non-integer value is going to be coerced into an integer before being used by the engine.
Aside from that, it really doesn't make sense for a Z index to have to be a float. For all intended purposes, you have as many Z values to work with as you want. If you are forced to keep a viewport Z within a certain range (i.e. between 2 map layers), then just change the Z index of the sprites on that viewport. The same can be said vice-versa.
To answer your question, no you can't modify the engine to use floats for Z indices by means of Ruby. They are complete different animals.