Processes across multiple services

Imagine that we have an ecommerce site that sells books. Initially, it was a monolith, but we are in the process of re-writing it. In the design process, we identified four services:

• A front end for client interaction.
• A billing service for collecting payments from customers.
• A fullfilment service for scheduling deliveries of orders.
• A communication service for handling notifications to customers.

A core use case of our system is a client purchasing a book. One approach to handle this would be: have the front end call the billing system first. On success, procceed to call the fullfilment service to start the process of delivering the book to the client. Finally, if all goes well, proceed to use communication service to email the customer letting them know that they have been billed, and the date when they should expect their package. All the calls just mentioned are REST calls.

In the front end service, there would be a method that looks something similar to this:

public async Task PurchaseAsync(PurchaseRequest request)
{
    var billingResult = 
        await billing.ProcessPaymentAsync(request.PaymentDetails);
 
    if(billingResult.Success)
    {
        var fullfilmentResult = 
            await fullfilment.SendOrderAsync(request.FullfilmentDetails);
 
        await communication.SendDeliveryEmailAsync(
            request.CommunicationDetails, 
            fullfilmentResult.DeliveryDetails);
    }
    else
    {
        await communication.SendPaymentFailEmailAsync(
            request.CommunicationDetails, 
            billingResult.FailPaymentDetails);
    }
}

The system gets deployed, and the site starts processing orders. But then support reaches out saying that customers are not getting their orders. To troubleshoot, we ask:

• Were customers billed for the books they bought? Some were, some weren't.
• Did they receive any email? No, they didn't.

We go into the logs and find a number of errors related to our method. In some cases we find that the customer didn't have sufficient funds (and billingResult.Success was false), but SendPaymentFailEmailAsync threw an error, so we didn't contact the customer letting them know about the situation. In other cases, the customer was successfully charged, but SendOrderAsync threw an error. In others still, it was ProcessPaymentAsync that failed.

The approach has several problems.

• It's not always possible to easily recover. For example, if SendOrderAsync fails, we can not simply replay the call to PurchaseAsync because we would end up double charging the customer.
• If any service is down, then no purchase can be processed.
• Because of the previous point, it may be required that all services be deployed at the same time, thus negating one of the major benefits of microservices.
• PurchaseAsync can become a bottle-neck in terms of performance. For example, the call to process a payment may be an expensive one.
• This approach also adds more friction to changes. For example, what if later on the process of purchasing a book involves other services? We would have to change the logic to this method in the front end service.

Though this scenario may seem far fetched, I've seen it play out in actual systems. The question then is, what would be a better way of doing this? One possibility is to leverage existing libraries for handling precisely these scenarios. NServiceBus is a great example that I'm familiar with. But there are others like Mass Transit or Ubers Cadence.

Another possibility would be to roll out your own framework. In the next post, we'll explore what that framework may look like in Azure.