Notice:
1.Reverse engineering violates EULA. This is for knowledge-sharing purposes only.
2.I'm Chinese and some of my expressions may be incorrect/unprecise.
Basic Introduction:
Assembly is built upon machine code; C/C++ is built upon assembly; RGSS is built upon C/C++. It's normally difficult to change one level's behavior from a higher level but relatively simple from a lower level.
So, reverse engineer RGSS, figure out what's going on, and make some patches. Problem solved.
Implementation:
First of all, Learn about IA32 assembly, WIN32API and other things. Then,
1.Find a breach point. For example, api call, memory access, strings, ...
2.Debug, run and test. Figure out the logic.
3.Patch on the key location. WriteProcessMemory will do.
Done.
Examples:
All examples are for RMXP, but should be very similar to VX/VA.
1.Disable what F12 does.
Since it's triggered by F12, we can try conditional breakpoints on GetKeyState/GetAsyncKeyState, and it works. We get this:
push 7B ; argument, keycode for F12
call GetAsyncKeyState
test ax,ax
jge gotoreturn ; if F12 not triggered, return
xxxx ; Resets the game
gotoreturn:
ret
Simply change jge to jmp, Reset will never be reached.
2.Disable 10s hangup
Search for "Hangup", ->
push eax
push "Hangup"
call xxx
mov [xxx],eax
It's like some kind of registration in ruby, so search for references. Distinguish the results. The rest should be easy. Patch, test and run, again patch.
3.Background running
It's about window procedure and messages, WM_ACTIVATEAPP. Should be easy.
4.Disable alt+enter
Again window procedure. Accelerators. Should be easy.
5.Load font
RGSS called EnumFontFamiliesExA and maintained a list at startup. So if we first call AddFontResource and then call EnumFontFamiliesExA as RGSS did with the new font specified, the new font will be added and usable.
End:
With good preparation, things should be just smooth and clear.
Because key addresses in different versions of RGSS .dlls is different, the exact code of patching is not given. It's just the idea.
Hm, I'm pretty sure RGSS is coded in pure C and has no C++ whatsoever.
Yeah, it's just C.
For anyone who happens to stumble upon this thread, let it be known that all but hangup (which happens when Graphics.update isn't called after 10 seconds) can be done already with scripts/Win32API; they exist out there--a couple I made myself.
I will also comment that this didn't really give me any knowledge on how to reverse engineer. It sounds like this thread was written with the idea that the reader has experience doing this before and has all the necessary tools already. Anyone who is capable of doing that is, in my honest opinion, wasting their time with messing with RGSS. It should be the other way around: use RGSS as a stepping stone into the world of reverse engineering. Treat it as a beginner's guide.
It's not just C. Sure large parts such as CRuby, zlib, libpng, libjpeg, libvorbis and libogg are. However the graphics and sound libraries (RGSS1 uses a different one than 2 and 3) are written in C++ . Well and x86/MMX Assembly for certain CPU intensive bitmap manipulations such as hue or sprite effects. Don't just take my word for it you can check it out for yourself as each libraries' source code was published to the web.
The RGSS developers may have made some changes and additions of course so there are a few things you can't find. Obviously things like original features or the actual RGSS itself: the Ruby classes and modules (Tilemap, Audio, etc), RTP support, the property windows, frame rate, and so on.
Back to the language question: the graphics library which is called nxlib (getting closer to the meaning of VX) is written in C++ and I doubt the developers re-implemented it or wrote an C wrapper. In fact because RTTI exists you can easily find the C++ class names and virtual tables.
Also every RGSS-based RPG Maker editor makes heavy use of MFC and the last one (VX Ace) even mentions a C++ only library in its About window: libtheoraplayer by some dude called Kresimir Spes.
Haha, I actually worked on libtheoraplayer. You can still see me being a contributer on github https://github.com/AprilAndFriends/theoraplayer/graphs/contributors . The version used in in VXA integrated the older version which Kresimir started by himself initially. I only joined in on development a bit later. I actually did the majority of API refactoring for v2.x.
Kresimir used to be one of the 3 owners of Cateia Games and I worked for 10 years for Cateia until later 2018. Cateia was sold to Russian mobile giant Playrix last year and rebranded as Playrix Croatia.
But yeah, IIRC RMXP was C-only. I think they might have used C++ on VX and evidently VXA does have C++ code.