AfNOG 2012 Workshop on Network Technology

Track SS-E

Scalable Internet Services


This course on Scalable Internet Services is part of the AfNOG 2012 Workshop on Network Technology, held in conjunction with the AfNOG meeting in Serrekunda, Gambia, in May 2012.

We use hands-on training in a well-equipped classroom over a five-day period to teach skills required for the configuration and operation of large scale Internet services.

Who should attend

Technical staff who are now providing Internet Services, or those who will be involved in the establishment and/or provisioning of a basic national Internet Services in the country.


Experience using and administering *NIX Servers, Name Servers, Web Servers and Mail Servers.


Joe Abley JA Canada
Brian Candler BC UK
Joel Jaeggli JJ US
Frank Kuse FK Ghana
Michuki Mwangi MM Kenya
Chris Wilson CW UK
Kevin Chege KC Kenya
Chimwemwe Frederick CF Malawi


o Pa Ali Nyang				-	KENYA	   
o Ghislain Nkeramugaba			-	Rwanda	   
o Kwabena Gyekye-Koranteng		-	GHANA	   
o Jeng	Njie				-	Gambia	   
o Nicholas Wambui Muchiri		-	KENYA	   
o Jainaba Sissoho			-	Gambia	   
o Seidu	Njie				-	Gambia	   
o Njemmeh Jeng Njie			-	Gambia	   
o Ousainou Darboe			-	Gambia	   
o Momodou Alieu	Jallow			-	Gambia	   
o Fanta	Bouba				-	TCHAD	   
o Momodou Lamin	Sanyang			-	Gambia	   
o Aminayanasam	Omineokuma		-	Nigeria
o Kumba	kuyateh				-	Gambia	   
o Greystone Chakhumata			-	Malawi	   
o Pa Lamin Jobarteh			-	Gambia	   
o Abdoulie Faal				-	Gambia	   
o Lama	Jallow				-	Gambia	   
o Ayodeji Ojo				-	NIGERIA	   
o Malick JOBE				-	Gambia	   
o Christian Ahiauzu			-	NIGERIA	   
o Lamin K. Sonko			-	Gambia	   
o Yahaar Ndow				-	Gambia	   
o Ousainou Darboe			-	Gambia	   
o Samba	Njie				-	Gambia	 
o Tlali Mamothokoane			-	LESOTHO	 


Monday Tuesday Wednesday Thursday Friday
09:00-11:00 SSE Intro DNS DNS Exim Exim
Tea Break
11:30-13:00 DNS DNS DNS Monitoring DNSSEC
14:00-16:00 Apache Apache Virtualization
Load Balancing
DNSSEC Security and Availability
Tea Break
16:30-18:30 RADIUS Virtualization Backup Load Balancing Scaling mail services
Closing Survey
Evening Sessions (20:00-22:00) Helpdesk Closed Closed


### Introduction Frank Kuse ### DNS Joe Abley #### Fundamentals Goal: to understand overall purpose and structure of DNS + IP addresses vs. names + DNS as a distributed, hierarchical database + Domain names and resource records: - A, PTR, MX, CNAME, TXT, SOA/NS + Domain name lookup responses + Reverse DNS + DNS as client-server model - Resolver - Cache - Authoritative server + Testing DNS (dig) + Understanding output from dig + Practical Exercises: - Configure Unix resolver - Use dig { A, other (e.g. MX), non-existent answer, reverse lookup } - Use tcpdump to show queries being sent to cache #### DNS Caching and Debugging Goal: to understand operation of a recursive nameserver + Recap of previous session + DNS as a distributed database. + Resource record NS: referral of answer + Caching nameserver and root servers + Caching used to reduce load (esp. top level servers) + Issue of stale data in caches (problems with distributed systems). - TTL records on each record - Negative TTL in SOA + Recursion and caching (dig +norec) + Demo: + Practical Exercise: - Debugging DNS Worksheet (with dig +norec ): . Students work on their own examples + Configuring a caching nameserver - check /var/named/etc/namedb/named.conf - run tcpdump - rndc start - change /etc/resolv.conf to point to your nameserver - query two times - { Look at 'aa' flag, TTL, query time } - rndc flush - cache is authoritative for + What sort of hardware would you choosing when building a DNS cache? + Improving the configuration of a cache NS + Managing a caching nameserver + Practical Exercise: - Building your own cache nameserver - Improving the configuration of the cache NS + Question and Answer session + Summary #### Configuring Authoritative Name Servers Goal: to properly configure an authoritative nameserver + Recap of caching NS + DNS Replication + Outside world cannot tell the difference between master and slave + When does replication take place? + Two (2) Dangers with serial numbers + Configuration of Master & Slave NS - Format of Resource Records { SOA and NS } + Ten (10) Common DNS Operational and Configuration Errors (RFC1912) #### Exercises Setting up authoritative name services for a domain + Master & Slave nameserver exercises #### Delegation and Reverse DNS + How do you delegate a subdomain? + Glue records + Reverse DNS - Subnets smaller than /24 + DNS Landmarks - Key organisations and people + The Root Zone + Top-Level Domains - Generic and Country Code TLDs + Registries, Registrars, Registrants + Nameserver Vendors + Conferences, Industry Groups + Mailing Lists + DNS Summary + Further reading ### DNSSEC Joe Abley DNSSEC High Level Awareness ### RADIUS Frank Kuse ### Apache Kevin Chege + Installing Apache22 from FreeBSD ports + Configure Apache with basic configuration + Start Apache httpsd daemon and connect to local box + Verify local ssl certificate works + Configuring Apache with SSL + Example SSL Apache configuration file + Sample config for Virtual Hosting ### Virtualization Joel Jaeggli ### Load Balancing Joel Jaeggli ### Backup Chris Wilson + Why bother with backups? + User error + Major disaster + Different requirements + Complicating factors + The plan + Backing up everything + Types of backups - Full, Differential, Incremental + Backing up files and systems + Software options - FreeBSD UFS snapshots - rsync . local and remote - tar - dump - Amanda server - duplicity ### Monitoring Brian Candler ### Exim Chimwemwe Frederick + Email Introduction + How Email Appears to Work + How Email Really Works + Mail User Agent (MUA)‏ + Mail Delivery Agent (MDA) + Mail Transfer Agent (MTA)‏ + Email Queue + MTA to MTA Transfer + DNS resolution and transfer process + Delivery + Troubleshooting Email Issues + RFCs + Exim Introduction + What is Exim? + Who uses Exim? + Why use Exim? + Why not to use Exim? + Installing Exim + The Exim Game Chris Wilson Exim and Internet Mail + What is Exim? + Who uses Exim? + Why use Exim? + Why not to use Exim? + Root and sudo + Exim Overview + SMTP commands and Access Control + Queueing and bouncing + Backup/secondary mail exchangers + The Exim configuration file + Global settings + Relaying for domains and hosts + Sending your first email (by hand) ### Mail Michuki Mwangi #### Dovecot with Virtual Users + Dovecot - Server for POP and IMAP - What is Dovecot? - Installing dovecot from ports - Configuring Dovecot #### Webmail using Squirrelmail #### Scaling mail services + Mailserver scalability - Linear password files - Linear mbox files - Too many files in one directory - CPU limits - Disk performance - Keep your SMTP (smarthost) and POP3 services separate + Notes and Clustering and NFS - Using Network File System (NFS) - Using Proxies - Load balancing - Database backends - FreeBSD NFS ### Security and Availability Joel Jaeggli ### Closing Survey Joel Jaeggli