Skip to content

SO Apps 5: Message Queueing and AMQP Info Sources

November 2, 2015

With AMQP (Advanced Message Queueing Protocol) multiple software technologies located anywhere there is an internet connection can now collaborate – sharing data and behavior.

In many scenarios, asynchronous message queueing is the favored mechanism for the communication and collaboration required between the components of distributed systems and service oriented apps. AMQP is in the process of being widely adopted as a standard way to implement this behavior. It was first developed in 2003 at JPMorganChase; was approved as a standard by OASIS in 2012; and was approved as an ISO and IEC International Standard in 2014 (Wikipedia — AMQP).

AMQP is a wire level application protocol, similar in concept to HTTP, but aimed at asynchronous message queueing scenarios rather than request response situations like HTTP. AMQP’s asynchronous message queueing provides the capability to interconnect software components implemented in a number of different technologies (Java, .NET, etc.) which may be running in various cloud or on-premises locations.

Why Use Message Queueing?

In considering AMQP it is useful to first review why message queueing is the favored inter-component collaboration mechanism in distributed systems and service oriented apps. The following summary is paraphrased from Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Wolfe, Copyright 2001 by Pearson Education, Inc. The rationale behind the use of messaging is presented in the section called “Messaging” (pp 53 – 56) of Chapter 2, “Integration Styles”:

There are generally 3 ways that the components of distributed systems can collaborate with each other:

  • Sharing Data – This form of collaboration is achieved by transferring files or using a shared database. Its main limitation is that it does not allow sharing the functionality implemented by each component in the system.
  • Remote Procedure Calls – Data plus the functionality of individual components is shared through remote procedure calls to each other, either with one-way or request-response scenarios. The main limitation here is that both the sending and receiving components must be running simultaneously for a successful collaboration to occur. This tightly couples the components together, often resulting in the overall system becoming unacceptably unreliable.
  • Asynchronous Messaging – Asynchronous message queues combine sharing both data and the functionality of individual components, but without tightly coupling the components involved. With queues the sender and receiver of queued messages do not need to be active simultaneously. And messaging systems can also be designed to provide timely request-response scenarios. Thus, message based distributed systems are typically far more resilient and reliable than distributed systems based on Remote Procedure Calls alone.

The following quote from an article on the International Association of Software Architects (IASA) web site adds more depth – “From the Field: Escaping Appland” by Monty Montgomery, Master Architect at IDesign.

“The demands placed on modern systems mandate elasticity. It is no longer enough to scale out. Your system must also shrink as well. The easiest way to address this need is to use a queuing technology. Queues normalize system load in a very reliable, predictable and efficient way. They absorb unexpected spikes and cyclical rises in load at lower cost. Queues are also the recognized building code for high throughput. And most importantly, queues provide the essential temporal decoupling you will need between your subsystem level microservices to extend technical empathy to DevOps.”

“The rise of numerous lightweight, mature queue-based messaging technologies clearly indicates the value and need for queuing. And it’s no mistake that all of the queuing technologies that matter now also support AMQP. All the modern software systems that you know and love to use employ some type of queuing technology as the backbone of their system architecture. And queuing is of course the cornerstone of the Cloud.”

Here are the most common asynchronous messaging scenarios, from Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications, Copyright 2014 Microsoft. Please see pp 169 – 171 for the details of each below scenario:

  1. Decoupling workloads from each other.
  2. Temporal decoupling, so the sender and received do not have to be active simultaneously.
  3. Load balancing by spreading the load between multiple components, each servicing the same queue.
  4. Load leveling, using a queue as a buffer to prevent spikes in incoming communications from overwhelming the system.
  5. Cross-platform integration.
  6. Asynchronous workflow, allowing the workflow components to be decoupled from each other.
  7. Deferred processing to facilitate schedules and better use of compute resources.
  8. Reliable messaging, guaranteeing the delivery of an enqueued message so that it never gets lost.
  9. Resilient message handling, allowing multiple receive attempts to be made for a given message until at least one attempt is successful.
  10. Non-blocking message receivers.

 

AMQP Info Sources

Here are the AMQP information sources I have found most helpful:

  1. First, there is the OASIS AMQP website. The About page of this website gives an excellent one page rundown of the capabilities, key features, and business cases for the use of AMQP, plus links to other informative pages as well. Don’t miss this quick and useful way of coming up to speed on AMQP basics.
  2. Microsoft has widely adopted AMQP and is actively using it in the Azure Service Bus, Azure Event Hubs, and also in the Azure IoT Hub.
    1. For the Service Bus, “AMQP 1.0 Support in Service Bus” covers the details and has a lot of links to other relevant Microsoft documentation.
    2. For Event Hubs, “Getting Started with Event Hubs” introduces how AMQP is utilized.
    3. For IoT Hubs please see “Azure IoT Hub Developer Guide” for where AMQP fits in and for links to other articles.
  3. Clemens Vasters (Lead IoT Architect at Microsoft) has several excellent video blogs about AMQP. These will give you an in-depth understanding of the AMQP protocol and what it can do. As you will see, AMQP is very full featured and supports a variety of communication modes beyond simply sending a message.
    1. Overview, and a conversation with David Ingham of Microsoft, the co-editor of the AMQP 1.0 spec – “Announcing the General Availability of AMQP 1.0 in Windows Azure Service Bus!
    2. A series of six 15 minute videos exploring the in-depth technical details and capabilities of AMQP.
      1. The AMQP 1.0 Protocol – 1/6 – Overview”.
      2. The AMQP 1.0 Protocol – 2/6 – Core Elements
      3. The AMQP 1.0 Protocol – 3/6 – Message Transfers
      4. The AMQP 1.0 Protocol – 4/6 – Flow Control
      5. The AMQP 1.0 Protocol – 5/6 – Primitive Type Encoding
      6. The AMQP 1.0 Protocol – 6/6 – Composite Types and Messages
  1. Apache has developed Qpid, a messaging framework based on AMQP for Java, C, and Python clients.
  2. Plus, there are other AMQP messaging systems as well — Just google “AMQP client” to list them.

So there you have it! The rationale behind the use of asynchronous message queueing in distributed systems, plus a lot of the basic information you’ll need to understand the capabilities of AMQP and put it to use.

George Stevens

Creative Commons License

dotnetsilverlightprism blog by George Stevens is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Based on a work at dotnetsilverlightprism.wordpress.com.

Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: