Processes across multiple services: A better approach

In the previous post we discussed a naive approach for handling use cases touching several services. The front end service was orchestrating a number of REST calls to the different services from a single method. We identified a number of drawbacks to this approach including brittleness (in that things can break easily if not all services are running properly), difficulty in supporting (it is not always easy to recover from errors), and lack of performance.

A better approach for solving this scenario would address all the problems we identified with the naive solution. One possibility is using an event driven architecture. Continuing with the example of an ecommerce site selling books, when a customer places an order, the system would raise an OrderPlaced event.

In more concrete terms, this means that the front end service would publish a message to a queue. Services subscribed to the queue would then consume the message, and act accordingly.

We can implement this using Azure Service Bus and Azure Functions. The initial message would be sent not to a queue but a topic. In Service Bus, a queue can have only one consumer; a topic, on the other hand, can have many. This is because topics have subscriptions.

Topic subscriptions have two important capabilities:

• They have filters (called rules) that allows them to select the messages they consume, and
• They can forward messages to queues.

We need something to process the messages sent to the topic. This is where Azure Functions come into the picture: they can be triggered by either topic subscriptions or queues. In our case, we are going to trigger them using queues. We are going to use topic subscriptions to route messages by forwarding them to the right queues.

The advantage of forwarding messages from the topic to queues has to do with retries. If we were using only a topic, and a message fails, retrying the message would cause subscriptions to resend the message to services that already had processed it successfully.

The diagram below illustrates a topology where a topic has five subscriptions. The top subscription forwards messages to a queue. The queue, in turn, triggers an Azure Function.

What should happen in each Azure Function? We need a framework that can:

• Invoke the right code to process a message, and
• Send messages to the topic.

For example, there could be a function in charge of handling billing. After receiving a message of type OrderPlaced it raises a command ChargeAcccount to collect money from the customer for their order. After that, we may publish the event OrderPaid that is then routed to the function in charge of handling fulfillments.

What happens in case of failures? If something goes wrong processing a message, depending on the settings in Service Bus, it may get retried a number of times. After those retries, if it's still failing, the message then goes into the queue's dead-letter queue. Using a tool like Service Bus Explorer, you can find a message, modify it if needed, and then retry it.

Does this set up address the problems with the naive approach?

• This approach allows us for recovery. If something goes wrong, we can retry the message from the dead-letter queue.
• If a service is down, we can still process orders. We captured the request in the message sent to the topic.
• Because we are temporally decoupled, we can deploy services at different times.
• The system can be modified. For example, we can add a new subscription to the topic that forwards messages to a new function.