Tutorial: rabbitmq-bash

Message bus systems are an easy way to distribute tasks.

RabbitMQ is a message broker and there are command line tools too to interact with queues.

To install these tools call:

apt-get install amqp-tools

The two main methods are:

  • amqp-consume
  • amqp-publish

First one is a blocking call that waits for incoming messages.

Second one is a call to put a message on a queue.

So we do have "workers" that consume a queue and "masters" that are publishing tasks.

A worker would look like:

amqp-consume -s -q "test" -e "amq.topic" --vhost "/" -r "worker1" --username=guest --password=guest -d ~/onmessage.sh

The parameters are:

Usage: amqp-consume [-dA?] [-s|--server=hostname:port] [--vhost=vhost] 
       [--username=username] [--password=password] 
       [-q|--queue=queue] [-e|--exchange=exchange]
       [-r|--routing-key=routing key] [-d|--declare] [-A|--no-ack] 
       [-?|--help] [--usage] [OPTIONS]...  

So if a message is dropped in the queue "test" and has the routing key "worker1" the command "~/onmessage.sh" is called.

The onmessage script might look like this:

nano ~/onmessage.sh && chmod +x ~/onmessage.sh

With content:

read line
echo "Message: $line"

amqp-consume pipes the content of the message therefore we cannot work with parameters ($1,$2,...) but with read to save the stream into a variable called "line".

To publish a message to the "worker1" following command has to be called:

amqp-publish -e "amq.topic" -r "worker1" -b "this is a test message"

The parameters are:

Usage: amqp-publish [OPTIONS]...
  -e, --exchange=exchange                     the exchange to publish to
  -r, --routing-key=routing key               the routing key to publish with
  -p, --persistent                            use the persistent delivery mode
  -C, --content-type=content type             the content-type for the message
  -E, --content-encoding=content encoding     the content-encoding for the message
  -b, --body=body                             specify the message body

Connection options
  -s, --server=hostname:port                  the AMQP server to connect to
      --vhost=vhost                           the vhost to use when connecting
      --username=username                     the username to login with
      --password=password                     the password to login with

I use this pattern for a download service.

I send urls to workers that wget the target.

You can even put more than one worker on a queue - so the work load is distributed.

Or define a second queue were the workers publish results to implement asynchronous method calls.

I like the amqp-tools because they can be used with any tool or language.