It's been quiet on Erjang for a while, and I've been busy with other things. Going to JavaZone two weeks ago however, got me all psyched up again on my little Erlang on the JVM project.
I met some guys from VMWare/SpringSource and we talked about RabbitMQ, so I spent some late nights these past weeks getting things working. The prospects of this? How about a SpringSource developer wanting to use RabbitMQ? Eventually, this would be a great option as an integrated part of a pure Java development environment for say, Eclipse.
It's at the point now where I can run a good deal of the unit tests, though many still fail. You can read more about the progress below.
The major improvements are
- Mnesia is much more stable now
- TCP/IP error handling is much improved
It's great to work specifically on something like RabbitMQ because it already has a large base of existing test cases. Further, RabbitMQ is plain erlang i.e., no NIFs or linked-in drivers, so it is a perfect candidate for Erjang.
Looking forward to demo/talk about this next week at JAOO. Feel free to follow Erjang at GitHub if you want to stay in the loop on this.
Additions/Fixes...
To get to this point, I added a number of new BIFs, and fixed various bugs. You can see the glory details in the commit log here, but here is an outline of the most important fixes/additions:
- Implement
erlang:hibernate
which is used ingen_server2
and a lot of the other infrastructure modules that come with RabbitMQ. Hibernating is implemented as an exception which is caught at the top-level of the process so as to remove the call stack before going to sleep. - Fix a couple of issues with the tcp_inet driver. Connections shutdown procedures didn't work completely, and rabbit depends on the owner dying to close connections. Further, rabbit uses the tcp option
{exit_on_close, false}
to allow outgoing communication to flush before the actual tcp socket is closed, and Erjang had some issues with that too. - Improve the driver select mechanism.
- Implement a few more options for the efile driver. Rabbit uses the
FILE_SETOPT
driver option to configure read-ahead strategies for file usage, and also depends on the integer (file descriptor) returned byFILE_OPEN
. - In ETS, I had to implement a few more bif used specifically by Rabbit/Mnesia, such as
ets:repair_continuation
,ets:update_counter/3
(the 4-tuple variant). - Various fixes to the BEAM -> JVM compiler that I stumbled upon when running Rabbit. There were some cases where the type inference was slightly off, and I optimized the cases when
erlc
uses{x,1023}
as a scratch register.
I still have an issue with some of the functions in io_lib
being non-tail recursive, causing stack overruns. The problem is that this often happens when Erjang tries to print an error message, and so it doesn't print the reason why it dies. And in some cases, the error_logger
process itself dies from a stack overrun, bringing down the entire VM.
Going forward, it looks like the issues yet-to-be-fixed are either simple bugs like variants of arguments that are not supported by my BIF implementations, and (more tricky) some cleanup issues with processes.
Stay tuned! I'll talk more about these things at JAOO next week, QCon/SF and ErlangFactory Lite LA in November, YOW! Australia in December. If you want me to come and speak please let me know.
Booting RabbitMQ on Erjang
Booting rabbit right now happens with the following command line
ej -boot /var/lib/rabbitmq/mnesia/rabbit@renaissance/plugins-scratch/rabbit \ -pa ./ebin -name rabbit2@renaissance.local \ -sasl sasl_error_logger '{file,"sasl.log"}'
The bootup sequence looks like this (running the latest rabbitmq 0.9.1 build from hg) on Erjang/OTP R14.
=INFO REPORT==== 29-Sep-2010::12:03:51 === alarm_handler: {set,{{disk_almost_full,"/"},[]}} =INFO REPORT==== 29-Sep-2010::12:03:51 === alarm_handler: {set,{{disk_almost_full,"/Volumes/StockMeter"},[]}} =INFO REPORT==== 29-Sep-2010::12:03:51 === alarm_handler: {set,{{disk_almost_full,"/Volumes/Flash"},[]}} | =INFO REPORT==== 29-Sep-2010::12:03:51 === alarm_handler: {set,{system_memory_high_watermark,[]}} +---+ +---+ | | | | | | | | | | | | | +---+ +-------+ | | | RabbitMQ +---+ | | | | | | v%%VSN%% +---+ | | | +-------------------+ AMQP 0-9-1 / 0-9 / 0-8 Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd. Licensed under the MPL. See http://www.rabbitmq.com/ node : rabbit2@renaissance.local app descriptor : /Users/krab/Projects/rabbitmq-server/ebin/rabbit.app home dir : /Users/krab cookie hash : pIGTBl0W0HD2xiNguwnLoQ== log : tty sasl log : sasl.log database dir : /Users/krab/Projects/rabbitmq-server/Mnesia.rabbit2@renaissance.local erlang version : 5.8 starting file handle cache server ... =INFO REPORT==== 29-Sep-2010::12:04:05 === Limiting to approx 10140 file handles (9124 sockets) done starting worker pool ...done starting codec correctness check ...done starting database ...found mnesia running id=<2.0.592> Sep 29, 2010 12:04:05 PM erjang.EModuleManager$FunctionInfo$1 invoke INFO: MISSING mnesia_sup:prep_stop/1 =INFO REPORT==== 29-Sep-2010::12:04:05 === application: mnesia exited: stopped type: permanent =INFO REPORT==== 29-Sep-2010::12:04:06 === Process <2.0.2031> exited abnormally without links/monitors exit reason was: shutdown erjang.ErlangExit: shutdown at erjang.m.erlang.ErlProc.exit(ErlProc.java:468) at erjang.m.mnesia_bup.mnesia_bup.install_fallback_master__2(mnesia_bup.S:207) at erjang.m.mnesia_bup.mnesia_bup$FN_install_fallback_master__2.go(Unknown Source) at erjang.EProc.execute(EProc.java:498) at kilim.Task._runExecute(Task.java:400) at kilim.WorkerThread.run(WorkerThread.java:47) done -- external infrastructure ready starting logging server ...done starting exchange type registry ...done starting exchange type direct ...done starting exchange type headers ...done starting exchange type topic ...done starting exchange type fanout ...done starting statistics event manager ...done -- kernel ready starting guid generator ...done starting cluster delegate ...done starting alarm handler ...done starting memory monitor ...done starting node monitor ...done -- core initialized starting empty DB check ... =INFO REPORT==== 29-Sep-2010::12:04:07 === Memory limit set to 1535MB. done starting queue supervisor and queue recovery ... =INFO REPORT==== 29-Sep-2010::12:04:07 === Added vhost <<"/">> =INFO REPORT==== 29-Sep-2010::12:04:07 === Created user <<"guest">> =INFO REPORT==== 29-Sep-2010::12:04:07 === Set user admin flag for user <<"guest">> to true =INFO REPORT==== 29-Sep-2010::12:04:07 === msg_store_transient: using rabbit_msg_store_ets_index to provide index =INFO REPORT==== 29-Sep-2010::12:04:07 === msg_store_persistent: using rabbit_msg_store_ets_index to provide index =ERROR REPORT==== 29-Sep-2010::12:04:07 === msg_store_persistent: rebuilding indices from scratch done starting exchange recovery ...done -- message delivery logic ready starting error log relay ...done starting networking ... =INFO REPORT==== 29-Sep-2010::12:04:07 === started TCP Listener on 0.0.0.0:5672 done -- network listeners available broker running Eshell V5.8 (abort with ^G) (rabbit2@renaissance.local)1>
Running the tests from rabbitmq-java-client
And we can now run some unit tests. I've been poking around in the rabbit-java-client
unit tests. Here are some screen shots...
The main tests for rabbitmq-java-client
gets most of the way there, but currently dies with a bug in Erjang's ETS implementation when running the 500 producers/consumer test. Well well, can be fixed.
/com/rabbitmq/examples/TestMain.class com.rabbitmq.examples.TestMain : javac v50.0 on 1.6.0_20 Channel 0 fully open. Got message (4 left in q): (0) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Got message (3 left in q): (1) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Got message (2 left in q): (2) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Got message (1 left in q): (3) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Got message (0 left in q): (4) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Drained, remaining in batch = 0. com.rabbitmq.examples.TestMain$BatchedTracingConsumer@6db3f829.handleConsumeOk(amq.ctag-jM3S9gxr4aSTUqoQC71+lA==) com.rabbitmq.examples.TestMain$BatchedTracingConsumer@42698403.handleConsumeOk(amq.ctag-lL8JbFN2bq2oWxUgeDTuFQ==) Async message (0,noack): (5) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (0,ack): (6) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (1,noack): (7) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (1,ack): (8) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (2,noack): (9) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (2,ack): (10) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (3,noack): (11) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (3,ack): (12) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (4,noack): (13) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Async message (4,ack): (14) On the third tone, the time will be Wed Sep 29 12:13:33 CEST 2010 Acking batch. com.rabbitmq.examples.TestMain$BatchedTracingConsumer@6db3f829.handleCancelOk(amq.ctag-jM3S9gxr4aSTUqoQC71+lA==) com.rabbitmq.examples.TestMain$BatchedTracingConsumer@42698403.handleCancelOk(amq.ctag-lL8JbFN2bq2oWxUgeDTuFQ==) About to publish to topic queues About to drain q1 Got message (2 left in q): B Got message (1 left in q): C Got message (0 left in q): D Drained, remaining in batch = 7. About to drain q2 Got message (0 left in q): C Drained, remaining in batch = 9. About to drain q3 Got message (1 left in q): C Got message (0 left in q): D Drained, remaining in batch = 8. About to try mandatory/immediate publications Handling return with body one Returned: #method(reply-code=312,reply-text=NO_ROUTE,exchange=mandatoryTestExchange,routing-key=) - props: #contentHeader (content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null) - body: one Handling return with body two Returned: #method (reply-code=312,reply-text=NO_ROUTE,exchange=mandatoryTestExchange,routing-key=) - props: #contentHeader (content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null) - body: two Handling return with body three Returned: #method (reply-code=313,reply-text=NO_CONSUMERS,exchange=mandatoryTestExchange,routing-key=) - props: #contentHeader (content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null) - body: three Handling return with body four Returned: #method (reply-code=313,reply-text=NO_CONSUMERS,exchange=mandatoryTestExchange,routing-key=) - props: #contentHeader (content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null) - body: four Got message (0 left in q): five Drained, remaining in batch = 0. Completed basic.return testing. Got message (4 left in q): (15) On the third tone, the time will be Wed Sep 29 12:13:34 CEST 2010 Got message (3 left in q): (16) On the third tone, the time will be Wed Sep 29 12:13:34 CEST 2010 Got message (2 left in q): (17) On the third tone, the time will be Wed Sep 29 12:13:34 CEST 2010 Got message (1 left in q): (18) On the third tone, the time will be Wed Sep 29 12:13:34 CEST 2010 Got message (0 left in q): (19) On the third tone, the time will be Wed Sep 29 12:13:34 CEST 2010 Drained, remaining in batch = 0. Handling return with body mandatory Returned: #method (reply-code=312,reply-text=NO_ROUTE,exchange=,routing-key=bogus) - props: #contentHeader (content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null) - body: mandatory Handling return with body immediate Returned: #method (reply-code=313,reply-text=NO_CONSUMERS,exchange=,routing-key=bogus) - props: #contentHeader (content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null) - body: immediate Got message (1 left in q): normal Got message (0 left in q): mandatory Drained, remaining in batch = 8. Closing. Leaving TestMain.run(). WON'T write statistics. WILL use server-side auto-acking.