How does email work

Let's imagine, John has decided to write an email to Kate and tell her about his trip to the mountains. John goes to the computer and opens his favourite email client (Mail User Agent, MUA). The email client can be either a program installed on a computer (e.g. Apple Mail, Microsoft Outlook, etc.) or a web client (e.g. Gmail, etc.). John composes an email, describing all the details of his journey, specifies the Kate's email address in the «To» field (e.g. kate@gmail.com) and presses the «Send» button. Then the fun begins.

The scheme was generated here.

For simplicity, let's divide the whole proccess into 2 stages: sending and receiving an email. All agents responsible for sending have the number 1, and for receiving – the number 2. As you read the article, I recommend looking at this scheme for clarity.

Sending the email

The journey of the email to John's Mail Transfer Agent

The John's email client (MUA) stores the addresses of incoming (where to recieve a new incoming emails) and outgoing mail servers (who should John give new outgoing emails to). If a separate program is used as an email client (e.g. Apple Mail, Microsoft Outlook, etc.), then when you add a mailbox, in addition to the email address and password, you'll be asked to enter the addresses of incoming and outgoing mail servers. Although, if your mailbox is hosted by some popular mail server (e.g. gmail.com), then most likely, the email client already knows about these mail server addresses and won't ask you to enter them.

The John's email client (MUA) goes to the specified address of outgoing mail server (via SMTP on the port 587) and its greeted by the Mail Submission Agent (MSA), which receives emails from users. Mail Submission Agent is like an employee at the post office, whose only task is to receive letters that need to be delivered somewhere. He'll never deliver them by himself. The same way, the Mail Submission Agent does not send emails, it only receives them from users and passes them on to the Mail Transfer Agent (MTA).

So, the Mail Submission Agent has received John's email and passed it to the Mail Transfer Agent.

Sending the email to Kate's Mail Transfer Agent

Mail Transfer Agent receives emails from Mail Submission Agent, saves them to a queue (mail SPOOL, Simultaneous Peripheral Operations Online), and then he sends them in turn.

John's email is next in line. The Mail Transfer Agent takes it and looks at the recipient's domain (kate@gmail.com -> gmail.com). Now the MTA must determine where the address of the mail server is located, where this email should be sent. It's not necessary that the address of the mail server is located in the same place as the website of the mail server. Moreover, must often, these are different servers.

To find out where the mail server of the recipient is located, the Mail Transfer Agent makes a request to the DNS server and obtain an MX record (Mail Exchanger) for the domain gmail.com.

# Where is the mail server of gmail.com?
nslookup -type=mx gmail.com
# mail exchanger = 5 gmail-smtp-in.l.google.com.
# mail exchanger = 10 alt1.gmail-smtp-in.l.google.com.
# mail exchanger = 20 alt2.gmail-smtp-in.l.google.com.
# mail exchanger = 30 alt3.gmail-smtp-in.l.google.com.
# mail exchanger = 40 alt4.gmail-smtp-in.l.google.com.

The DNS server has answered that gmail.com has 5 such addresses with different priorities (the less a number, the higher priority). At first, Mail Transfer Agent should try the first one. If it's now available, then the second one, etc.

Let's also see, whether the mail server and the website are on different servers.

nslookup gmail-smtp-in.l.google.com
# Address: 64.233.165.27

nslookup gmail.com
# Address: 142.250.74.69

As expected, they are located on different servers.

Finally, the Mail Transfer Agent goes to the first address gmail-smtp-in.l.google.com of the mail server (via SMTP on the port 25) and transmits John's email for Kate to it.

telnet gmail-smtp-in.l.google.com 25
# Trying 64.233.165.26...
# Connected to gmail-smtp-in.l.google.com.
# Escape character is '^]'.
# 220 mx.google.com ESMTP m21-20020a194355000000b0050be7106660si2252687lfj.555 - gsmtp

## Hello! I'm John's mail server.
ehlo domain.com
# 250-mx.google.com at your service, [123.123.123.123]
# 250-SIZE 157286400
# 250-8BITMIME
# 250-STARTTLS
# 250-ENHANCEDSTATUSCODES
# 250-PIPELINING
# 250-CHUNKING
# 250 SMTPUTF8

## I want to send an email from John.
mail from: <john@domain.com>
# 250 2.1.0 OK m21-20020a194355000000b0050be7106660si2252687lfj.555 - gsmtp

## To Kate.
rcpt to: <kate@gmail.com>
# 250 2.1.5 OK m21-20020a194355000000b0050be7106660si2252687lfj.555 - gsmtp

## I'm ready to write the message (RFC 2822).
data
# 354  Go ahead m21-20020a194355000000b0050be7106660si2252687lfj.555 - gsmtp
From: "john@domain.com" <john@domain.com>
Subject: Hello Kate!
Message-Id: <ABCDEFGH-1234-5678-ABCD-ABCDEFGHIJKLM@domain.com>
To: <kate@gmail.com>
Content-Type: text/plain; charset=utf-8

Hello, Kate! 
I have only just got back from the mountains. It was so great!

## I've finished writing the message.
.
# 250 2.0.0 Ok: queued as ABC123DEF45

## It's time to say goodbye
quit
# 221 2.0.0 Bye
# Connection closed by foreign host

Hooray! John's Mail Transfer Agent successfully sent the email to Kate's Mail Transfer Agent via SMTP protocol on the port 25.

As you can see, The MTA response always contains a 3-digit code and a short message that is understandable to human beings. The error code can be as follows:

• 2xx – The command was completed successfully. You can continue.
• 3xx – The command was accepted, but the server requires additional information. You should send another command with that information.
• 4xx – The command was failed, but it's temporary. You should try again later.
• 5xx – The command was failed. You should NOT try again.

Take another look at the content of the email that was sent after the data command. It consists of 2 parts separated by an empty line: the headers and the text the the email. Do you see the header with the name To? What do you think will happen if it's changed to another address, for example, to alice@domain.com? Where will the email be sent, to Kate or Alice? It'll be sent to Kate, because the MTA takes the recipient's address only from the rcpt to command. All headers are just part of the email content that is sent to rcpt to.

By the way, maybe, we didn't send the mail to Kate's end point mail server, but only to a relay, which can redirect it to another relay, but eventually this email will reach the desired mail server. Relays are the same Mail Transfer Agents who, instead of storing the email, pass it on. Relays can help distribute the load across multiple servers, preventing overloading of any single server.

The email is currently located in the Mail Transfer Agent of Kate's mail server.

Receiving the email

Kate's Mail Transfer Agent (MTA) accepts John's email, checks it (whether it's spam, whether it really came from the specified domain, etc.) and if everything is OK, then passes it to the Mail Delivery Agent (MDA).

Spam verification is usually performed not by the MTA itself, but by another agent (e.g. Rspamd), which uses various methods to determine whether an email is spam. If spam is detected, Kate's MTA can either refuse to deliver it to Kate and return the error to John's MTA, or add a header to the email, indicating that the email is spam (e.g. X-Spam: Yes).

As a result, Mail Transfer Agent have 2 roles:

1. To receive outgoing emails from Mail Submission Agent and send them to other Mail Transfer Agents.
2. Recieve incoming emails from different Mail Transfer Agents and transfer them to the Mail Delivery Agent.

Mail Delivery Agent saves incoming emails in user's mailboxes. For example, all incoming emails for Kate can be stored in /home/kate as files. You can read more about the email storage formats supported by Dovecot here (it's worth paying attention primarily to maildir and mdbox).

Mail Delivery Agent can have different rules for putting incoming emails into folders. For example, if an incoming email has the header X-Spam: Yes, which was added in the previous step, then put this email in the spam folder. If the mailbox is full or another error occurs, an email is generated to the sender (John) with the reason why this email could not be delivered.

At the moment, the email is already in Kate's mailbox, but she doesn't know about it yet. Kate's mail client (Mail User Agent) must somehow find out about the new email. To do that, the Mail User Agent periodically asks the Mail Retrieval Agent whether new emails have appeared (the frequency is configured in the mail client). Communication between them takes places via IMAP or POP3.

Finally, Kate's email client have received the email from John and made an audio alert. Kate went to her computer and saw a new email from John.

Sending a reply to the email

If Kate decides to reply to John's email, who will the Mail User Agent send it to? The Mail Retrieval Agent is only responsible for receiving emails, it doesn't send them.

The process of sending a reply will look the same, but only in reverse order:

1. Kate composes a reply in the Mail User Agent and presses «Send».
2. Kate's Mail User Agent sends the email to Mail Submission Agent, which is located on Kate's mail server.
3. Mail Submission Agent trasnfers the email to Mail Transfer Agent.
4. Kate's Mail Transfer Agent makes a request to the DNS server and recieves an address of John's mail server.
5. Kate's Mail Transfer Agent sends the email to John's Mail Transfer Agent (possibly, via relays).
6. John's Mail Transfer Agent transfers the email to Mail Delivery Agent.
7. Mail Delivery Agent saves the email in the John's mailbox on disk.
8. John's Mail User Agent asks to Mail Retrieval Agent whether new emails are appeared and receives the email from Kate.
9. John reads Kate's email.

In order not to complicate the scheme above, the process of responding to an email is not displayed on it, therefore, Kate's MSA and John's MDA, Mailbox, MRA are skipped.

IMAP or POP3

IMAP (Internet Message Access Protocol) and POP3 (Post Office Protocol 3) are used by mail clients to receive emails from a mailbox located on the mail server.

POP3 downloads all emails from the mailbox to the local device, and then deletes them from the server. The mail server can be configured so way copies of all emails are stored somewhere, but another device will no longer be able to download them. As a result, all the emails will be available only from one device. If the mail server does not store copies of emails, then if the device breaks down, all the emails that were on it will be lost.

IMAP allows you to work with one mailbox simultaneously from multiple devices (e.g. on a computer, phone and tablet). While reading emails, they remain on the server and are «marked» as opened (moves to another directory). IMAP supports working with different folders in the mailbox (inbox, spam, drafts, sent, trash, archive, etc.) where you can store emails. These folders will be visible on all devices.

Links to the RFCs for further learning

RFCs (Request for Comments documents) defines the standards for the Internet. There are a few RFCs that describes how does email work.

1. Simple Mail Transfer Protocol (SMTP) [RFC 2821]
2. SMTP Service Extensions [RFC 1869]
3. Internet Message Format [RFC 2822]
4. Internet Message Access Protocol (IMAP) [RFC 2060]
5. Post Office Protocol (POP) [RFC 1939]

In the next article, we'll move on to creating our own mail server with an unlimited number of domains and mailboxes.