Update (March 2012): Oracle appears to have quietly disassembled their JavaRTS development team.
There isn’t a good answer to this question. There certainly isn’t enough critical mass to say there’s a current most-popular RT JVM.
US Defense systems I’m aware of use JavaRTS, IBM’s WebSphere RT, and PERC. Aviation platforms use aicas JamaicaVM. Most of those folks haven’t been too concerned about licensing costs, to date. They are concerned by much higher program risks. I’ve worked with a number of these programs, and I find that the products are of high quality, and there is (at times extraordinary) support from the vendors.
- AN/FPS-85 radar uses JavaRTS, and though this is a relatively low count, it’s a rather high criticality application
- US Navy’s Aegis Weapon System uses PERC
- US Navy’s DDG-1000 uses IBM WebSphere RT
- aicas’ JamaicaVM is in use in avionics platforms, but they are understandably coy about specifics
Many of the vendors have focused on specific clients or domains, and their designs and support infrastructure tend to fall in line with that. If you have a specific application profile in mind, maybe we can get you a better answer.
The high licensing costs you cite reflect the high expectations users of these products have, and the relatively low density of the demand. The developer community (such as it is) is also more fragmentary because systems with real-time or safety-critical constraints are typically of greater sensitivity (defense, aviation, SCADA) or protected proprietary value (automotive) or both. One excellent forum for real-time Java theory and practice is the annual JTRES conference, which is a close-knit group of academics and technically-savvy vendors.
Related on SO: Primitives-only in Java, who uses RTSJ.