thegammaray: I use Handbrake regularly. FWIW, I rip to MKV so I can include multiple audio/subtitle streams, which I don't think is possible with MP4 files.
I don't know anything about PS3s, but I encode at constant quality, RF:19. I was dissatisfied with the default RF:20, though I might be fooling myself. Are you encoding HD content? According to the Handbrake website/wiki, RF:20+/-1 is good for DVD-quality video, RF:22+/-1 for HD content.
I'd stick to the newer versions of Handbrake, unless you're somehow using current x264 encoders with older Handbrake versions.
The PS3 has a fairly limited ability to play file formats and the general consensus seems to be MP4 is the best for the job. Since I don't care for or need multiple audio streams or subtitles, it seems good enough for me. I'm not even sure the PS3 can do MKVs anyway.
Zooooom! That was the sound of everything you just said going right over my head ;). Like I said, I've never used Handbrake or pretty much an other encoder/ripper before. I don't see anything that resembles the settings you are talking about in the 0.9.3 version I'm currently using. These are standard DVDs, not Blu-Rays or other HD content in any way.
I'd love to use the latest version of Handbrake (assuming that it will produce better results), the problem is, they dropped preset support for the PS3 a couple versions back and I don't know enough about Handbrake or the PS3's file requirements to wing it on my own (hence this post).