[XP] RPG Maker XP Online System (RMX-OS)

Started by Blizzard, June 20, 2009, 11:52:23 am

Previous topic - Next topic

Blizzard

I got the source for MySQL and used the Ruby source to compile the mysql C-extension. I remember it being quite a bothersome process until I finally figured it out properly. >.< But I can't remember if in the end which file that was. It might have been libmysql.so in the root directory. Or maybe I just used that one to compile the two SO files that are currently there.

Also, in the newest version of RMX-OS you have both mysql_api.so for both Ruby 1.9 and Ruby 2.0 available. They aren't 9 years old. >_> libmysql.do/dll are only 7 years old. >_>
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Mason Wheeler

Ack, my mistake.  I looked at the wrong thing.  Regardless, the current version is Ruby 2.2.

So this was something you had to build yourself?

Blizzard

Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Mason Wheeler

I just looked at SQL.rb in the server, and saw something worrisome: it's not set up for proper security.  It has a method for escaping strings--a bad idea which should never be done--and no support for parameterized queries, which is the right way to handle the problem that escaping strings is supposed to solve.  That means that the server's database architecture is essentially a SQL injection attack waiting to happen.

If I had an API specification for mysql_api.so I'd fix this myself, but since it's custom-built, I need to ask for Blizzard to update this to support parameterized queries.  Can you do that?

Blizzard

April 02, 2016, 10:12:31 pm #1764 Last Edit: April 02, 2016, 10:13:56 pm by Blizzard
Uhm, no. Injection attacks can't be done. That's what the whole string escaping is for. SQL injection works by adding SQL queries into data so when processing the queries, the string gets turned into an actual command rather than treated as data. And RMX-OS prevents that. It makes sure that all client string data is properly converted into actual string data before actually doing an SQL query with it. Unless you found a specific piece of code that doesn't use the escaping method to prevent SQL injection, RMX-OS shouldn't have that problem. SQL.rb doesn't do the escaping. The calling code does.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Mason Wheeler

Yes, in theory, assuming everyone remembers to use it every time and the escaping routine works perfectly, that is the case.

In practice, neither of those assumptions is valid.  People forget the escaping when concatenating a string into the query, and escaping routines are notoriously difficult to get right.  (Just look at how many different "mysql_escape_strings_for_real_we_swear_we_got_it_right_this_time" routines there are in PHP!)  This is why any developer with experience in database security will tell you it's not worth the risk: you should always use parameterized queries and never use escaping and concatenation instead.

But don't take my word for it; just ask on StackOverflow (or the more specialized security.stackexchange.com) if escaping is a valid substitute for parameters.  They'll say what I just said.  They might also point out that if you use parameters, the DBMS is able to cache the query plan, leading to improved performance on long-running processes such as servers.

Blizzard

I know that, but there's always the problem with double escaping and the need for more code to be able to queue SQL commands in a parametrized manner so it made more sense to organize the code like this rather than writing all the overhead code to make it work. But I agree that parametrized calls (and any safe wrapper calls for that matter) are the way to go.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

whitespirits

just wanting to ask Blizz some advice :)

so I had 3 people on rmx-os today and all seemed good, people did say they had some lag, is this because I'm in the UK on home PC? I do have 200mb broadband. they were America and spain based.

as well it seems if I enter the map first I have perfect speeds? if I join another players I feel a delay of lag. just wanting to know if a vps will help me and what the situation is with how it actually works, I love blizz abs and rpg XPace I just wana stable game :)

Blizzard

Yeah, the server location does affect that.
Also, I made RMX-OS's communication work through TCP rather than UDP. TCP is slower, but reliable.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Mason Wheeler

After looking this over, with a bit of help from whitespirits, it seems that the source of the lag has very little to do with the choice of TCP over UDP.  Instead, it seems to stem from a quirk in Blizz-ABS, the "gmaster" system.

A bit of background:

Most MMOs are designed with the canonical game state--the authoritative answers to the questions of where characters and NPCs are located, what they're doing, and what items, stats and abilities they have--living entirely on the server.  The game client is mostly just a terminal that passes input messages to the server, retrieves update messages from the server, and renders the game according to the state the server describes in the update messages.  This has several advantages.  It allows you to centralize all the processing of the game, it makes it easy to implement the well-known security principle of Never Trust User Input (always verify that any outside input is valid before accepting and acting on it,) which makes cheating much more difficult, and it makes reverse-engineering the game harder because the client doesn't have the game logic.  Running a map server can be a difficult, processor-intensive job, and you'll often have one server for each map.  (Or sometimes more than one per map, depending on how big your game is!)  This is generally OK, because you tend to have a small number of large maps.

RMX-OS is the exact opposite in many ways.  It's not designed as an MMO, but rather it's a system for adding MMO logic onto an existing, well-established single-player RPG engine.  Because RPG Maker is a single-player engine, the concept of canonical game state existing somewhere other than on the client doesn't make sense to it.  The RMX-OS server is a pretty lightweight MMO system, which mostly deals with message passing and saving persistent data to the database; it knows nothing about the game itself.  It doesn't have its own copy of the maps or the game logic; these are all located on the client-side, which means that both cheating and reverse-engineering are fairly simple, unfortunately.  (All a player needs to do is open up their copy of the game in RPG Maker and play around with the maps.)

But this runs into a problem with a system like Blizz-ABS, where NPCs are supposed to be moving around on the shared map and interacting with the users as enemies that users can fight together: you can't have the clients move the NPCs.  (What if two of them moved the same NPC in different ways?  The action would get out of sync!)  And you can't have the server move the NPCs, because it doesn't know about the maps or the game logic.  Therefore, Blizz-ABS decided to have one client move the NPCs and handle canonical map data for each map, designating that client as "gmaster" (Game Master, I assume.)  So now this one client does all the processing locally, and coordinates everything (including other users' actions) with the server.

What this means is that if a laggy client is chosen as gmaster for a map, he'll lag up everyone on that map.

Ideally, this could be solved by moving canonical processing of game data to the server, but that would be tricky: a large amount of RGSS and the game engine would have to be recreated on the server-side, while removing the graphics part of it, and the corresponding code would need to be disabled on the client-side and replaced with messaging.  And it would take a bunch of additional reworking due to the way RPG Maker's map engine is designed with the implicit assumption that only one map is running at a time.  And just to further complicate things, unlike standard MMOs, which feature a small number of large maps, RPG Maker's style tends towards a large number of small maps, which makes multi-server solutions trickier.

It could be done, but it would not be an easy task!

The other possible solution would be for the server to assign gmaster more intelligently.  Right now, it simply assigns gmaster on a first-come-first-served basis: the first person to enter a map becomes gmaster for that map and retains the title for as long as they stay connected and on that map.  If the server had a way of detecting ping times for its clients, though, it could check as part of the server_update cycle to see which client on each map has the lowest latency and reassign gmaster status to keep things running smoothly.  Unfortunately, there doesn't seem to be any built-in way to do that.

One thing that might work: Have the server spawn a new Thread that does the following in a loop:

for each client connected:
   record current time
   send a PING message to client, to which it responds "PONG"
   wait for the response. when it comes, check how long it took.
record all client ping times in a hash of client => time
save this hash to a global value
sleep for a few seconds before running the loop again.


Then the server_update would check to see if the hash has been updated.  If it has, run through the list of maps, find the fastest user for each map, and set them as gmaster.

This wouldn't be perfect, but it would help keep lag down without having to rewrite the entire game engine.

Any thoughts?

orochii

I think you're right. RMX-OS is actually a good easy solution to put online to almost anything in RPG Maker, though that comes up with the price that the thing you made "instantly online" wasn't built to be online.

I have another idea, since RMX-OS has this thing of "first one becomes server for the map", how about making a special client that makes sure to be a map's gmaster? Basically my idea is this: a client that doesn't renders anything neither does anything, just like a dummy character (maybe even not log as a character at all). It just picks the work as gmaster and that's it. As for picking the gmaster as soon as possible, since the client always ping to see when the server is online, make it automatically log in, or even prohibit logging in until one of these special clients is on (someone could develop a script that beats "us" just to mock everyone, I guess xD).

Maybe make these clients able to be gmasters in more than one map. Assume we have 100 maps and 4 PCs. You then make each PC the gmaster of 25 maps each (or make it 20-15-30-40, maybe some maps are designed to accommodate more players than others).

It's pretty much distributing the load into several computers. And it doesn't need that much of a change as far as I know since these client-servers (?) will be just like a regular client.

KK20

May 07, 2016, 10:37:26 pm #1771 Last Edit: May 07, 2016, 10:39:23 pm by KK20
While all this theory talk could work, the underlying question that needs to be addressed is this:

Is it even worth it?

We're talking about RPG Maker XP here, or any of the newer ones for that matter. It wasn't designed for online play. And it surely wasn't designed to be used for an MMO. That's the biggest issue I have with the way RMX-OS is presented; you slap on the sentence "make an MMO" and everyone suddenly thinks they can make the best MMO game ever. And who wouldn't, right? Most--if not all of us--have grown up on social outlets like MMOs. The idea to be able to play a game with your friends now at your creative disposal, and what better than an easy game-making editor than RMXP? I know I tried when I saw this thread for the first time. I dabbled with RMX-OS and BABS, documented ideas, and fantasized about the fun I could have--and this was during the time I had only started using RMXP for a few months with no real programming background (other than a high school Java class). This was 6 years ago...guess what happened to that project?

RMX-OS exists as a medium to allow online connectivity because it's a fucking cool idea. I mean, seriously, who would have thought this could be possible, and without having to rely on shitty Netplay? It was a revolution for its time, especially when the RMXP community was still largely active.

But looking at the bigger picture, RMX-OS is exactly what I said above--it serves as a medium for online connectivity. The idea to use it as a true MMO just doesn't cut it, but an Online Multiplayer is more feasible. If you want to be serious with this, you shouldn't be using RPG Maker in the first place. If you want to just dick around, then you came to the right place.

In short, there's no reason to even bother rewriting some of the RMX-OS code to compensate for this. The RM engine is inherently bad and there's not much that can be done to avoid that. There's no real merit to doing this in my opinion. If you want to tackle it, then go ahead and share the code. But it's like fixing a hole in the ship when the whole thing is on fire.

Other Projects
RPG Maker XP Ace  Upgrade RMXP to RMVXA performance!
XPA Tilemap  Tilemap rewrite with many features, including custom resolution!

Nintendo Switch Friend Code: 8310-1917-5318
Discord: KK20 Tyler#8901

Join the CP Discord Server!

Blizzard

May 08, 2016, 03:25:29 am #1772 Last Edit: May 08, 2016, 03:28:22 am by Blizzard
Yup, you are spot on, Mason. ^_^ That's exactly how I designed RMX-OS.

G-Master stands for Global Master.

The NPC thing is mostly something that I never got around making. Technically one could easily use the Blizz-ABS plugin code for moving monsters around to make NPCs move around as well.

To be honest, before retiring I did play with the idea to change RMX-OS in such a way to be able to run on multiple servers where the maps would simply be distributed between servers since they are isolated entities. It actually wouldn't be that difficult. All you had to do would be sending messages to the client to reconnect to another server if they are changing to a map that currently resides on another server and you would probably need a login server separate from the map servers. The login server would also handle the list of where each map currently resides and handle server synchronization by handing out and revoking map handles to servers.

It's true that a laggy gmaster will lag the map, but I designed a mechanism to try to find a better gmaster if there is too much lag. Unfortunately I didn't write the code to find the best gmaster. But that code is isolated and can easily be changed if somebody is willing to do it. I talked to whitespirits the other day and I think the gmaster solution would help a lot. If you want to make it, feel free to go ahead. I can even add it in the official distribution of RMX-OS if you want and give you credit. :)

There is another better, but also more complicated solution that I thought of. If you would create a dummy RMXP engine, you could run an instance of the game without graphics on the server for every map so gmaster processing would happen on the server. This approach still allows you to use the gmaster system for processing, but it would be server side. Of course, it would probably still not be that cheat-proof, but it's an improvement over the existing system.

@KK20: The RM engine isn't bad. It's just relatively simple and it wasn't designed with online multiplayer in mind. I think that it does a decent job for what it was created.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Mason Wheeler

QuoteTo be honest, before retiring I did play with the idea to change RMX-OS in such a way to be able to run on multiple servers where the maps would simply be distributed between servers since they are isolated entities. It actually wouldn't be that difficult. All you had to do would be sending messages to the client to reconnect to another server if they are changing to a map that currently resides on another server and you would probably need a login server separate from the map servers. The login server would also handle the list of where each map currently resides and handle server synchronization by handing out and revoking map handles to servers.

Interesting.  Did you ever come up with any code for this?

QuoteThere is another better, but also more complicated solution that I thought of. If you would create a dummy RMXP engine, you could run an instance of the game without graphics on the server for every map so gmaster processing would happen on the server. This approach still allows you to use the gmaster system for processing, but it would be server side.

Possibly, but this would involve running possibly hundreds of instances--I've even seen a few games with over 1000 maps!--and Ruby is not a lightweight scripting engine.  That would require a non-trivial resource commitment to make it work.

Blizzard

May 08, 2016, 09:13:25 am #1774 Last Edit: May 08, 2016, 09:15:20 am by Blizzard
Nope, only concepts.

True, but running it on multiple servers could help counter that. You should also take into account that maps where there are no players at a certain moment, wouldn't actually be running an instance. At 100 players max. on the server, that makes only 100 max. instances.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

Denver

May 12, 2016, 03:31:16 pm #1775 Last Edit: May 12, 2016, 04:25:50 pm by Denver
Hey! First of all, thank you very much for this awesome system, Blizzard! Much appreciated!  :clap:

Secondly, I am unable to connect to the server through the Client. I installed Ruby 2.0.0 and WAMP, created the database, and when I run "RMX-OS.rb" it seems to start the server successfully (at least I get the message "RMX-OS Server has started successfully"). But when I start the game (Client) the server is always offline. I am trying to run it on localhost for some tests, so I did not change anything. What could possibly be wrong? I have no clue. Please help!  :(

EDIT:
I managed to get it to work by downloading "RGSS102E.DLL" and copying it into the "Client" folder. But I will have to modify the "Config.ini" file whenever I save the game. :P

Blizzard

You can also rename the DLL to whatever the one is your RMXP version is using. That also works.
Check out Daygames and our games:

King of Booze 2      King of Booze: Never Ever
Drinking Game for Android      Never have I ever for Android
Drinking Game for iOS      Never have I ever for iOS


Quote from: winkioI do not speak to bricks, either as individuals or in wall form.

Quote from: Barney StinsonWhen I get sad, I stop being sad and be awesome instead. True story.

KK20

But the whole "had to put the RGSS DLL in the client project folder" makes no sense though, especially when the game was running fine, just not establishing a connection. You sure that's what fixed it?

Other Projects
RPG Maker XP Ace  Upgrade RMXP to RMVXA performance!
XPA Tilemap  Tilemap rewrite with many features, including custom resolution!

Nintendo Switch Friend Code: 8310-1917-5318
Discord: KK20 Tyler#8901

Join the CP Discord Server!

Denver

Blizzard
Thank you very much, that is a nice workaround, hehehe!

KK20
I did not change anything, only pasted the DLL in the Client folder and voilĂ  - it worked all of a sudden. If I remove that DLL or use a more recent one ("RGSS103J.DLL" for example), it will not work. It was not me who figured this out though (thank you, Mundo RPG Maker).

G_G

Maybe certain RGSS libraries don't have Win32API implemented properly. I dunno.