Mail server scalability

What problems do we come across?

Linear password files

On some systems, every mail delivery and pop3 connection requires a scan through the whole /etc/passwd file.

• This is a problem with many Linux distributions
• FreeBSD uses searchable databases /etc/pwd.db, /etc/spwd.db
• Don't give mail users a Unix account; have a separate user database. (Better for security, too)
• Or, use an external account database such as LDAP or RADIUS

Linear mbox files

If a user keeps their mail on the server, every POP3 or IMAP connection and every mailbox change requires the POP3 daemon to read the entire mail file. 30,000 messages is a 500MB mailbox file that takes 30 seconds to scan.

Maildir is an alternative which stores each message in its own file:

• Solves the problem of reading a large file.
• Creates a new problem of reading a large directory!
• Choice of filesystem and options makes a big difference (e.g. reiserfs or htree)
• "Creating 100,000 files in a single directory took 38 minutes without directory indexing... and 11 seconds with the directory indexing turned on." (Theodore Ts'o)
• Can run out of inodes with millions of small files

Too many files in one directory

• Can happen with either mbox (too many users) or maildir (too many emails for one user)
• Use a hashed directory structure, e.g. /home/12/34/user

CPU limits

• Be sure that CPU is really the limit!
  • use top when server runs slowly, check %cpu user/system, >50% may be a problem
  • Use a graphing/trending tool like munin or cacti to spot heavy usage times, and correlate with high disk/CPU/memory usage
• Put in a faster CPU/multiple CPUs (SMP), and more RAM
• Ensure kernel parameters correctly tweaked (sockets, filehandles)
• Distribute the load across multiple boxes - clustering
• Use an efficient MTA and POP/IMAP server
• Use IMAP rather than POP3 for accounts with webmail, and for large mailboxes

Disk performance

• Use softupdates (FreeBSD) or a high-performance filesystem (Linux)
• Use multiple disks, spread your mail directories across them
• Use pairs of mirrored disks (RAID1), not RAID5
• Distribute the load across multiple boxes - clustering
• Enforce quotas to limit disk space and inodes used by each customer

Separate SMTP (smarthost) and POP3/IMAP services separate

Separating SMTP and POP3/IMAP on separate machines is an easy way to scale your mail service. Plan for it by using separate hostnames (e.g. pop3 and smtp) that may initially point to the same machine.

There is an additional advantage: mail routing works correctly even if one of your customers leaves (moves their domain's MX records to point somewhere else) without telling you.