I agree, I think if we have a turnbased / ATB system, we should include quicktime events ala paper mario that can trigger different outcomes for an attack.
What I would like to see, too, in terms of battle system if we're gonna have as custom one: One with a perspective ala Golden Sun - it works great and is really not as overused as most other perspective. We'd need rather large animated battlers, then, though - but I think it'd be worth it.
Another important part should be that we massively include tactics in the battle system - no primitive using the same attack 100 times and hoping the hp doesn't run out. I'm thinking multibody enemies, as in you can attack certain parts of the enemy, and they'll have different weaknesses. And, if you "killed" certain body parts, the enemy will not be able to use certain attacks anymore.
This is more for bosses, anyways.
A quest jounal, as you mentioned, is pretty important imo, though I think it should also log the main story. It happened to me far too often that I forgot about a game for some time and then had problems getting back into the story - Tales of Symphonia though had the whole story written down as if the characters kept a diary. That helped alot.
In terms of substance, the game definitely needs some rather large sidequest(s) that are difficult and difficult to find - to give the player a sense of just having discovered something big, like the whole Cursed Weapons or the Maxwell quest in Tales of Symphonia - it both ended in epic boss battles (one was even stronger than the final boss) and gave the player unique abilities.
Blah Blah I'll add more later.