After looking this over, with a bit of help from whitespirits, it seems that the source of the lag has very little to do with the choice of TCP over UDP. Instead, it seems to stem from a quirk in Blizz-ABS, the "gmaster" system.
A bit of background:
Most MMOs are designed with the canonical game state--the authoritative answers to the questions of where characters and NPCs are located, what they're doing, and what items, stats and abilities they have--living entirely on the server. The game client is mostly just a terminal that passes input messages to the server, retrieves update messages from the server, and renders the game according to the state the server describes in the update messages. This has several advantages. It allows you to centralize all the processing of the game, it makes it easy to implement the well-known security principle of Never Trust User Input (always verify that any outside input is valid before accepting and acting on it,) which makes cheating much more difficult, and it makes reverse-engineering the game harder because the client doesn't have the game logic. Running a map server can be a difficult, processor-intensive job, and you'll often have one server for each map. (Or sometimes more than one per map, depending on how big your game is!) This is generally OK, because you tend to have a small number of large maps.
RMX-OS is the exact opposite in many ways. It's not designed as an MMO, but rather it's a system for adding MMO logic onto an existing, well-established single-player RPG engine. Because RPG Maker is a single-player engine, the concept of canonical game state existing somewhere other than on the client doesn't make sense to it. The RMX-OS server is a pretty lightweight MMO system, which mostly deals with message passing and saving persistent data to the database; it knows nothing about the game itself. It doesn't have its own copy of the maps or the game logic; these are all located on the client-side, which means that both cheating and reverse-engineering are fairly simple, unfortunately. (All a player needs to do is open up their copy of the game in RPG Maker and play around with the maps.)
But this runs into a problem with a system like Blizz-ABS, where NPCs are supposed to be moving around on the shared map and interacting with the users as enemies that users can fight together: you can't have the clients move the NPCs. (What if two of them moved the same NPC in different ways? The action would get out of sync!) And you can't have the server move the NPCs, because it doesn't know about the maps or the game logic. Therefore, Blizz-ABS decided to have one client move the NPCs and handle canonical map data for each map, designating that client as "gmaster" (Game Master, I assume.) So now this one client does all the processing locally, and coordinates everything (including other users' actions) with the server.
What this means is that if a laggy client is chosen as gmaster for a map, he'll lag up everyone on that map.
Ideally, this could be solved by moving canonical processing of game data to the server, but that would be tricky: a large amount of RGSS and the game engine would have to be recreated on the server-side, while removing the graphics part of it, and the corresponding code would need to be disabled on the client-side and replaced with messaging. And it would take a bunch of additional reworking due to the way RPG Maker's map engine is designed with the implicit assumption that only one map is running at a time. And just to further complicate things, unlike standard MMOs, which feature a small number of large maps, RPG Maker's style tends towards a large number of small maps, which makes multi-server solutions trickier.
It could be done, but it would not be an easy task!
The other possible solution would be for the server to assign gmaster more intelligently. Right now, it simply assigns gmaster on a first-come-first-served basis: the first person to enter a map becomes gmaster for that map and retains the title for as long as they stay connected and on that map. If the server had a way of detecting ping times for its clients, though, it could check as part of the server_update cycle to see which client on each map has the lowest latency and reassign gmaster status to keep things running smoothly. Unfortunately, there doesn't seem to be any built-in way to do that.
One thing that might work: Have the server spawn a new Thread that does the following in a loop:
for each client connected:
record current time
send a PING message to client, to which it responds "PONG"
wait for the response. when it comes, check how long it took.
record all client ping times in a hash of client => time
save this hash to a global value
sleep for a few seconds before running the loop again.
Then the server_update would check to see if the hash has been updated. If it has, run through the list of maps, find the fastest user for each map, and set them as gmaster.
This wouldn't be perfect, but it would help keep lag down without having to rewrite the entire game engine.
Any thoughts?