Agreed, I find low numbers are much easier to manage. Plus, when you start seeing the bigger numbers, it means it just got real. Final Fantasy XIII I felt did this rather well, the later enemies are powerful enough to do lots of damage to you and the final super-skills give you some massive massive attacks.
Granted I use Chrono Trigger as my model, so low damage on both sides, 999 max HP (all but one of my cast are ordinary-ish mortals), 9999 damage cap, typically two digit damage until you get the later skills.
That said, I love how OTT the Disgaea series is. There is no damage cap, the post-game superbosses can have trillions of HP so you need everything you can get to stomp them.
An example of poor balancing though (a case where the numbers should have been allowed to go higher) - Yiazmat in Final Fantasy XII. 5 billion HP, but you are limited to a maximum damage cap of 6999 at a time for most of the fight. XIII never goes as high in HP, but some bosses have millions of HP. Even so the much higher cap means you blow through the HP faster, making it less of a slog.
That's what it is all about to me. Rather than big numbers / small numbers, it's how well those numbers fit and scale. I don't want to be limited to four-figure damage if my opponent has a dozen zeroes on the end of his HP score!
So to answer the original question, prefer low numbers, high numbers have their place too, the important thing is BALANCE.