Background

My blog system "Moonglade" needs to send email notifications to blog owner when there are events like new comments, new pingbacks, and send email to readers when their comments are replied. 

There are two options for sending emails in web application. Build my own infrastructure or use a third-party service like SendGrid. I've explored both options and finally decided to build my own infrastructure. During the last few years, this infrastructure has been redesigned many times. In this post, I will share the journey of how this email infrastructure evolved and the things I learnt. 

Version 1: Inside Moonglade Application

The initial design is to send email directly from Moonglade itself, which is a ASP.NET Core web application. I've built a .NET library that can configure email in XML template and send messages by SMTP. 

It's adding an xml configuration file under application folder like this:

<?xml version="1.0"?>
<MailConfiguration>
  <CommonConfiguration OverrideToAddress="false" ToAddress="overridetest@test.com" />
  <MailMessage MessageType="TestMail" IsHtml="true">
    <MessageSubject>Test Mail on {MachineName.Value}</MessageSubject>
    <MessageBody>
      <![CDATA[
Mail configuration on {MachineName.Value} is good: <br />
Smtp Server: {SmtpServer.Value}<br />
Smtp Port: {SmtpServerPort.Value}<br />
Smtp Username: {SmtpUserName.Value}<br />
Email Display Name: {EmailDisplayName.Value}<br />
Enable SSL: {EnableSsl.Value}<br />
      ]]>
    </MessageBody>
  </MailMessage>
</MailConfiguration>

Then send email in code

var smtpServer = "smtp-mail.outlook.com";
var userName = "Edi.Test@outlook.com";
var password = "***********";
var port = 587;
var toAddress = "Edi.Wang@outlook.com";

var configSource = $"{Directory.GetCurrentDirectory()}\\mailConfiguration.xml";
var emailHelper = new EmailHelper(configSource, smtpServer, userName, password, port)
     .WithTls()
     .WithSenderName("Test Sender")
     .WithDisplayName("Edi.TemplateEmail.TestConsole");

Pros

  • Only need to deploy one application, which saves cost on Azure.
  • Straight forward coding experience

Cons

  • Bad cloud architecture. I cannot scale the email infrastructure alone when there is large amount of traffic. The web server is doing one more job, sending email is not a "Web" thing.
  • Upgrading or fixing bugs in email component requires redeploying the entire web application. 
  • Hard to deal with failures. Without a database or queue, I cannot track failed email messages and resend them. 
  • Potential security and network restriction. In certain scenarios, web servers won't be allowed to access SMTP services, which makes email fail to send.
  • Coding language must be the same withing one application. If some day, there is a much better email library written in another language, it's hard to migrate because other pieces of the application is written in C#.

Version 2: A standalone ASP.NET Web API

The next version separates web workload and email sending workload by building a standalone API, just doing one thing: Send emails. It's a typical architecture design.

Pros

  • Email API can be scaled, upgraded, and deployed separately without affecting the web application.
  • I can choose different versions of .NET and C# or even another language to build the email API.

Cons

  • I have to pay for an extra server instance even when most of the time there is no email sending API calls.
  • Extra deployment and management for another server instance.
  • Still hard to deal with failures. During email API deployment or down time, any API call from web application will fail, and users won't be able to receive emails.

Version 3: Single Azure Function

The most significant problem for using a full ASP.NET Web API is the cost, including operation and maintenance costs and development costs.

First, the App Service Plan that hosts the Web API is a huge overhead. For an ASP.NET Core application, there is currently no way to choose Consumption Plan, so the API will be billed even when it is idle. According to business, the API is only called about ten times a day, and the total processing time is less than 2 minutes. But I have to pay for the rest 1438 minutes every day. 

Secondly, even a simple and ordinary Email notification function still needs a complete basic framework to host it. Although ASP.NET Core provides a very flexible and powerful framework, Azure Function can completely free up these basic tasks. Azure Function only cares about business code, not infrastructure. In general, developers only need to write the business logic of the function, and do not need to write code such as how to start, route, and assign API Keys.

This is fully explained in my previous blog post 'Migrating an ASP.NET Core Web API Project to Azure Function'.

The function code looks like this

public class EmailSendingFunction
{
    [FunctionName("EmailSending")]
    public async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] NotificationRequest request,
        ILogger log, ExecutionContext executionContext)
    {
        T GetModelFromPayload<T>() where T : class
        {
            var json = request.Payload.ToString();
            return JsonSerializer.Deserialize<T>(json);
        }

        log.LogInformation("EmailSending HTTP trigger function processed a request.");

        try
        {

            var configRootDirectory = executionContext.FunctionAppDirectory;
            AppDomain.CurrentDomain.SetData(Constants.AppBaseDirectory, configRootDirectory);
            log.LogInformation($"Function App Directory: {configRootDirectory}");

            IMoongladeNotification notification = new EmailHandler(log)
            {
                AdminEmail = request.AdminEmail,
                EmailDisplayName = request.EmailDisplayName
            };

            switch (request.MessageType)
            {
                case MailMesageTypes.TestMail:
                    await notification.SendTestNotificationAsync();
                    return new OkObjectResult("TestMail Sent");

                //  skip a few code...

                default:
                    throw new ArgumentOutOfRangeException();
            }
        }
        catch (Exception e)
        {
            log.LogError(e, e.Message);
            return new ConflictObjectResult(e.Message);
        }
    }
}

Pros

  • Save cost by a serverless consumption plan
  • Free from writing ASP.NET Core infrastructure code, only focus on business code
  • Easy to migrate to another programming language like Java, Python, NodeJS, or even PowerShell

Cons

  • Still hard to deal with failures. During email function deployment or down time, any API call from web application will fail, and users won't be able to receive emails.

Version 4: Azure Function and SQL Database

To deal with failures, I added a SQL database for storing email messages and their status. The web application no longer makes direct API calls to the Azure Function. Instead, it only writes messages to a database table and forgets about it. It's the Azure Function's responsibility to pick new messages from database and send them. If email sending failed, it will record and retry a few times until mark the message as failed completely and then move to next message. Yes, it is using database as a queue.

In this version, the function code looks like this

[FunctionName("NotificationV2")]
public async Task Run([TimerTrigger("*/2 * * * *")] TimerInfo myTimer, ILogger log)
{
    log.LogInformation($"NotificationV2 Timer trigger function executed at UTC: {DateTime.UtcNow}");
    // logic ...
}

Pros

  • Dealing with failures. As long as the web application is alive and can write messages into database, any deployment or downtime of Azure Function won't lose any email. When the function is restored up running again, it will continue where it's left. 

Cons

  • Using database as a queue requires writing every queue feature manually. Like peek message and retry logic.
  • Azure Function needs to have a timer trigger to query database for new messages. It is an extra consumption because most of the time the query will return no new messages.

Version 4: Azure Function and Storage Queue

Using database as a queue makes me work 996. Azure has provided real message queue services, Azure Storage Queue and Azure Service Bus. For sending emails, I prefer using Storage Queue.

"Azure Queue storage is a service for storing large numbers of messages that can be accessed from anywhere in the world via authenticated calls using HTTP or HTTPS. A single queue message can be up to 64 KB in size, and a queue can contain millions of messages, up to the total capacity limit of a storage account. Queue storage is often used to create a backlog of work to process asynchronously."

If you need more details when choosing architecture design, you can see this document for detailed comparison between these two.

Now I am free from writing basic queue features like enqueue, dequeue, peek, retry, etc. 

When an Azure Function instance picks up a message, it will be invisible to other instances for a period, this is called visibility timeout mechanism. So that when you have multiple instances at scale, the same message won't be processed multiple times.

By default, Azure will try 5 times for failed message and put them into a poison queue automatically. So that a failed message won't block current instance or other instances from picking up following up messages. 

Web application code using Azure.Storage.Queues package.

public async Task EnqueueNotification<T>(MailMesageTypes type, string[] toAddresses, T payload) where T : class
{
    try
    {
        var queue = new QueueClient(_notificationSettings.AzureStorageQueueConnection, "moongladeemailqueue");

        var en = new EmailNotification
        {
            DistributionList = string.Join(';', toAddresses),
            MessageType = type.ToString(),
            MessageBody = JsonSerializer.Serialize(payload, MoongladeJsonSerializerOptions.Default),
        };

        await InsertMessageAsync(queue, en);
    }
    catch (Exception e)
    {
        _logger.LogError(e, e.Message);
        throw;
    }
}

private async Task InsertMessageAsync(QueueClient queue, EmailNotification emailNotification)
{
    if (null != await queue.CreateIfNotExistsAsync())
    {
        _logger.LogInformation($"Azure Storage Queue '{queue.Name}' was created.");
    }

    var json = JsonSerializer.Serialize(emailNotification);
    var bytes = Encoding.UTF8.GetBytes(json);
    var base64Json = Convert.ToBase64String(bytes);

    await queue.SendMessageAsync(base64Json);
}

Function code.

[FunctionName("NotificationStorageQueue")]
public async Task Run(
    [QueueTrigger("moongladeemailqueue", Connection = "moongladestorage")] QueueMessage queueMessage,
    ILogger log,
    Microsoft.Azure.WebJobs.ExecutionContext executionContext)
{
    log.LogInformation($"C# Queue trigger function processed: {queueMessage.MessageId}");
    // logic...
}

Pros

  • Free from writing queue logic
  • Change timed trigger to queue trigger, saving a lot of database and compute consumption

Cons

  • Learning curve for how to use Azure Storage Queue

Conclusion and more

When designing systems for cloud. Keep in mind cloud native patterns. Message queue is a common way to decoupling system parts that are scaled independently. Email sending is one of those scenarios that can use a message queue. By decoupling email component from main application, it can be scaled to handle large amount of traffic, use a different programming language, resilient to failure, and save cost in certain ways.