A few years back - about the time when JRuby started getting traction - I was also working on a JVM-based Ruby that I called HotRuby (and yes, I do know that there is also another project by that name). The project has been dormant for two years, but recently Charles Nutter asked me about it, and so I promised to make it available. It's up on GitHub here: http://bit.ly/5XrKA2 . I did a talk to the London Ruby User's Group which is also linked from the GitHub wiki.
Nobody has been actively working on this for several years. I did this as an exercise learn Ruby and I stopped, when I decided that I didn't particularly like the language, and I was overwhelmed by the amount of cruft in the core libraries that needed bug-for-bug compatibility to be useful.
The VM is at the state where it can run some simple performance tests, and it can kind-of run RUnit.
An interesting aspect of this project is that it does a two-phase compilation (similar to hotspot). First, it generates byte codes (not JVM ones), and runs an interpreter on that. This is much faster than running an AST interpreter. Then later it selectively performs an "agressive" compilation on hot methods, turning the code in to JVM code. The purpose of this is the same as for Java HotSpot - get up and running fast, and only spent time and effort where it matters. "agressive" compilation means that it specializes methods for the receiver type, so it compiles methods to JVM under the assumption hat "self" has a specific statically known type. This permits access to ivars to be very fast.
Both the interpreted and compiled mode run on the same MOP, so object-layouts are decided early. When a Ruby-level class is instantiated the first time, I generate a class for it in which all the instance variables that are referenced in methods applicable for that objet are turned into real java fields.
The interpreted mode runs at ~MRI performance, and with compilation-to-jvm enabled it can run as much as ~2.5 x YARV performance. Or at least that was the numbers back when I did the presentation/demo for LRUG.
This was before there was anything called invokedynamic. Method lookup in HotRuby is done via a "JVM-bytecode compiled HashMap" thing. Think of the method lookup table as a hash map (sitting at the call site), in which each potential receiver type is a key. Then, I implement something similar to Self's PIC (Polymorphic Inline Caching) by dynamically bytecode-generating a new class whenever this lookup table needs to have a new key inserted. Urgh! It ends up generating a lot of classes, but it is fast because a method invocation turns into a double virtual dispatch and a cast. If the cast fails, then it is because the compiled hash-map needs to be re-generated.
So perhaps this can be used as inspiration for furhter JRuby work, or inspiration for other people. I'm happy to answer any questions you might have.
Kresten