Erjang/OTP 22% there ...

Update: Adding support for file_readdir (listing a directory) suddently jumped this to 78%; so perhaps my measure of progress is not linear with the effort :-) Still, I am happy with it. Next up is to implement that special variant of open_port that reads/writes file descriptors; in this case to handle stderr. Cheers!

I know, 22% doesn't sound like much, but I've been making good progress on OTP. The graph below shows my two measures of progress on Erjang/OTP, the blue line is OTP modules the Erjang compiler can successfully compile to valid JVM byte code, the red line is how far an OTP boot comes before bailing out (measured as #modules loaded by Erjang vs #modules needed to boot OTP on beam as reported by /sw/bin/erl -loader_debug < /dev/null | grep '{return,{ok' | wc -l).

The last 8 days of work has brought Erjang/OTP from 7% to 22% on this measure, and I am yet to discover something "really hard", no reason to believe we cannot go on. But things will likely be slow over the christmas break.

Now I can go on christmas break with some peace. Next up is to complete file I/O as needed by gb_sets, which is the last module to load before it bails.

Wierd exceptions

There has been a few issues with the beam -> jvm compiler, but it is getting more and more stable. One of the funny things is that erlc seems to do CSE (common subexpression elimination) which gets in the way of exception handling.

I.e. in code_server there is code looking like this:

handle_call({load_native_partial,Mod,Bin}, {_From,_Tag}, S) ->
    Result = (catch hipe_unified_loader:load(Mod,Bin)),
    Status = hipe_result_to_status(Result),
    {reply,Status,S};

handle_call({load_native_sticky,Mod,Bin,WholeModule}, {_From,_Tag}, S) ->
    Result = (catch hipe_unified_loader:load_module(Mod,Bin,WholeModule)),
    Status = hipe_result_to_status(Result),
    {reply,Status,S};

Notice how the two last line of each function clause are equal, and they happen to coincide with being an exception handler. The resulting beam code is [pseudo]:

LabelA:
    {catch LabelC}
    hipe_unified_loader:load(Mod,Bin)
    goto LabelC

... other code goes here, maybe from another function clause ...

LabelB:
    {catch LabelC}
    hipe_unified_loader:load_module(Mod,Bin,WholeModule)
LabelC:
    {catch_end}
    Status = hipe_result_to_status(Result),
    {reply,Status,S};

You see the issue? I have to generate java code such that there are two code blocks that are different places in the method, but share the same exception handler [the code at LabelC]. I was wierded out for a while there, but it seems to work now.

Clojure to the Rescue!

Another stumbling block this week has been implementing ets (erlang term store), which is kind of an STM-thing (shared transactional memory) in erlang, that can be used to share data structures across erlang processes.

It is not complete; I am implementing BIFs as I run into them, but the structure is there, and I have a few insert/delete/lookup methods as needed so far by Erlang/OTP booting. There is also a parser/evaluator for match_spec's. A nice future optimization would be to generate JVM bytecode for compiled match_spec [That would be a good subproject for a new contributor].

Anyway, for ets, I have used Clojure's persistent data structures and the transactional cell, clojure.lang.Ref. This handles the transactionsl requirements on ets operations, and further would allow us to implement transactions across ets operations.

As an example, this is the implementation of ets:insert(tid, [tuple, ...]) for set type (implemented as an IPersistentMap).

@Override
protected void insert_many(final ESeq values) {
    LockingTransaction.runInTransaction(new Callable() {
        @Override
        protected Object run() {        
            IPersistentMap map = (IPersistentMap)mapRef.deref();
            for (ESeq seq = values; !seq.isNil(); seq = seq.tail()) {
                ETuple value = values.head().testTuple();
                if (value == null) throw ERT.badarg(values);
                map = map.assoc(get_key(value), value);
            }
            mapRef.set(map);
            return null;
        }});
}

For clojure folks out there: In Erjang an ESeq is similar to a clojure.lang.ISeq, it is the representation of a well-formed list.