The only limitations graphically (aside from the rendering issues) are really on you. There have been scripts made to allow 3d in rmxp, so is it possible? Yes. Likely? No. If you have an artist or are a decent artist yourself then you can find ways of making the 2d really go far, but even that is limited to your knowledge of the engine itself and your ability to modify/work with it. Basically, don't try to make a beautiful game without first putting in the time to really get to know the ins and outs of the engine because you will probably struggle :\
I think there was a topic on scripts that are impossible made by blizzard, which will give you an idea on some of the real limits of rmxp,