Network Management & Monitoring Using RANCID ============ Notes: ------ * Commands preceded with "$" imply that you should execute the command as a general user - not as root. * Commands preceded with "#" imply that you should be working as root. * Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") imply that you are executing commands on remote equipment, or within another program. Exercises --------- 1. Connect to your PC using ssh 2. Become root, and install postfix (mail system we are using) $ sudo bash # apt-get install postfix You might get a prompt on Postfix install, select "Internet Site" At the next prompt, enter your hostname (e.g. pcxx.ws.nsrc.org) # service postfix status Also check that the 'mutt' mail client is installed, as well as telnet and subversion: # apt-get install mutt telnet subversion 3. Add an alias for the rancid user in /etc/aliases file # vi /etc/aliases rancid-all: sysadm rancid-admin-all: sysadm Save the file, then run: # newaliases 4. Install Rancid itself # apt-get install rancid It will prompt for warning, and Really continue? Select Yes 5. Edit /etc/rancid/rancid.conf # vi /etc/rancid/rancid.conf Find this line in rancid.conf: #LIST_OF_GROUPS="sl joebobisp" And, underneath it add the following line: LIST_OF_GROUPS="all" (with no '#' at the front of line) Also, we want to use Subversion, and not CVS: Find the line with the parameter RCSSYS, and change it to: RCSSYS=svn; export RCSSYS and the line with CVSROOT: CVSROOT=$BASEDIR/SVN; export CSVROOT 6. Change to the rancid user From a root prompt, switch identity to become the 'rancid' user: # su -s /bin/bash rancid Check that you ARE the rancid user: $ id You should see something similar (numbers may be different): uid=104(rancid) gid=109(rancid) groups=109(rancid) ***** IF YOU ARE NOT USER RANCID NOW, do NOT continue ****** 7. Create /var/lib/rancid/.cloginrc $ vi /var/lib/rancid/.cloginrc add user 10.10.* cisco add password 10.10.* cisco cisco (The first 'cisco' is the username, the second and third 'cisco' are the password and enable password used to login to your router. The star in the IP means that it will try to use this username and password for all routers in 10.10. network) Now protect this file so that it cannot be read by other users: $ chmod 600 /var/lib/rancid/.cloginrc In reality you want to create an extra user for RANCID on the Cisco with limited rights instead. 8. Test login to the router of your group Login to your router with clogin. You might have to type yes to the first warning, but should not need to enter a password, this should be automatic. $ /var/lib/rancid/bin/clogin 10.10.X.254 (replace x with your group no.) You should get something like: spawn ssh -c 3des -x -l cisco 10.10.2.254 The authenticity of host '10.10.2.254 (10.10.2.254)' can't be established. RSA key fingerprint is 73:f3:f0:e8:78:ab:49:1c:d9:5d:49:01:a4:e1:2a:83. Are you sure you want to continue connecting (yes/no)? Host 10.10.2.254 added to the list of known hosts. yes Warning: Permanently added '10.10.2.254' (RSA) to the list of known hosts. Password: rtr2>enable Password: rtr2# Exit the router rtr2#exit 9. Initialize the SVN repository for rancid: $ /usr/lib/rancid/bin/rancid-cvs You should see something similar to this: Committed revision 1. Checked out revision 1. At revision 1. A configs Adding configs Committed revision 2. A router.db Adding router.db Transmitting file data . Committed revision 3. ***** If you have problems ****** If this does not work, then either you are missing the subversion package, or something was not properly configured during the previous steps. You should verify that subversion is installed and then before running the rancid-cvs command again do the following: $ exit # apt-get install subversion # su - /bin/bash rancid $ cd /var/lib/rancid $ rm -rf all $ rm -rf SVN Now try running the rancid-cvs command again: $ /usr/lib/rancid/bin/rancid-cvs ******************************** 10. Create the router.db $ vi /var/lib/rancid/all/router.db Add this line: 10.10.xxx.254:cisco:up (remember to replace xxx as appropriate) 11. Let's run rancid! $ /usr/lib/rancid/bin/rancid-run (Should take about half a minute) Run it again, since the first time it might not commit correctly: $ /usr/lib/rancid/bin/rancid-run 12. Check out the logs: $ cd /var/lib/rancid/logs $ ls -l ... View the contents of the file(s): $ less all.* 13. Look at the configs $ cd /var/lib/rancid/all/configs $ less 10.10.x.254 If all went well, you can see the config of the router. 14. Let's change an interface Description on the router $ /usr/lib/rancid/bin/clogin 10.10.x.254 At the "rtrX#" prompt, enter the command: rtrX# conf term You should see: Enter configuration commands, one per line. End with CNTL/Z. rtrX(config)# Enter: rtrX(config)# interface LoopbackXX (replace XX with your PC no) for example rtr5(config)# interface Loopback17 (if your PC is number 17) You should get this prompt: rtr2(config-if)# Enter: rtr2(config-if)# description rtr2(config-if)# end You should now have this prompt: rtr2# To save the config to memory: rtr2# write memory You should see: Building configuration... [OK] To exit type: exit 15. Let's run rancid again: $ /usr/lib/rancid/bin/rancid-run Look at the config and logs $ ls /var/lib/rancid/logs/ 16. Let's see the differences $ cd /var/lib/rancid/all/configs $ ls -l You should see all the router config files $ svn log 10.10.x.254 (where xxx is the IP of your router, .1 or .2 or 3.) Notice the revisions. Let's view the difference between two versions: $ svn diff -r 5:6 10.10.x.254 | less ... can you find your changes ? Or those of others in the same time? 17. Check your mail Now we will exit from these shells to get back to being the "sysadm" user, and run 'mutt' to see the mails which rancid has sent: $ exit # exit $ id ... check that you are now the 'sysadm' user again; ... if not, log out and in again $ mutt (When asked to create the Mail directory, say Yes) If everything goes as planned, you should be able to read the mails sent by Rancid. (use 'q' return to mail index, and 'q' again to quit mutt) 18. Let's make rancid run automatically every 30 minutes from cron $ sudo bash # crontab -e -u rancid It will ask you for your favorite editor. Add this line: */30 * * * * /usr/lib/rancid/bin/rancid-run ... then save and quit 19. Now add all the other routers Note the IP addresses for the routers 10.10.x.254 where x goes from 1 to 8 Update the router.db # su -s /bin/bash rancid $ vi /var/lib/rancid/all/router.db Add some other classroom routers to the file. You should end up with something like: 10.10.1.254:cisco:up 10.10.2.254:cisco:up 10.10.3.254:cisco:up 10.10.4.254:cisco:up 10.10.5.254:cisco:up 10.10.6.254:cisco:up 10.10.7.254:cisco:up 10.10.8.254:cisco:up (Note that "cisco" means this is Cisco equipment -- it tells Rancid that we are expecting to talk to a Cisco device here. You can also talk to Juniper, HP, ...) 20. Run rancid again: $ /usr/lib/rancid/bin/rancid-run (Should take a minute or more now, be patient) 21. Check out the logs: $ cd /var/lib/rancid/logs $ ls -l ... Pick the latest file and view it $ less all.YYYYMMDD.HHMMSS 22. Look at the configs $ cd /var/lib/rancid/all/configs $ more 10.10.* If all went well, you can see the configs of ALL routers 23. Run RANCID again just in case someone changed some configuration on the router $ /usr/lib/rancid/bin/rancid-run (patienceÉ) 23. Play with clogin: $ /usr/lib/rancid/bin/clogin -c "show clock" 10.10.x.254 What do you notice ? 24. Add the RANCID SVN repository in to SVNWeb If you are still logged in as user rancid, get back to root $ exit # Install SVNWeb: # apt-get install websvn * Say Yes to the question if you want to configure it now * Hit Ok for the next question about supporting various web servers * As directory add /var/lib/rancid/SVN both times when asked on the next screens. Be sure that "SVN" is uppercase. * Hit Ok for the next screen talking about permissions 25. Fix permissions. The web server must be able to read the SVN archive # chgrp -R www-data /var/lib/rancid/SVN # chmod g+w -R /var/lib/rancid/SVN 26. Browse files from your webbrowser http://pcXXX.ws.nsrc.org/websvn Browse the files under the 'all/configs' directory. You can see all your router configuration files here. 27. Review revisions If you want to see different revisions side by side, you can do that with WebSVN too. However diffs between revisions are broken in the package which comes with Ubuntu 10.04 LTS solved in the latest version Patch that bug with a simple patchfile: # apt-get install wget patch # cd /usr/share/websvn/include # wget http://noc.ws.nsrc.org/downloads/patch-svn # patch < patch-svn You can now also browse difference between file revisions with WebSVN. Browse to http://pcXXX.ws.nsrc.org/websvn again, go to all, configs Select your router file (10.10.x.254) and click "Compare with Previous" You should now see the latest changes side by side. 28. How to secure the WebSVN directory You can use the Apache2 default authentication mechanism to add a password to the WebSVN. This is the htpasswd/htaccess mechanism. Create the htpasswd file # cd /usr/share/websvn # htpasswd -c .htpasswd admin (enter the password when asked two times) # chmod 600 .htpasswd # chown www-data:www-data .htpasswd Edit the file /etc/apache2/conf.d/websvn - under the line enter: AuthType Basic AuthName "Restricted Access" AuthUserFile /usr/share/websvn/.htpasswd Require valid-user Save and exit your editor. Restart the web server # /etc/init.d/apache2 restart Now if you login to http://pcXX/svnweb it should be password protected. +---- Rev. 31 May 2011