Lab Notes The purpose of this module is to introduce the concept of an Internet Exchange Point, how to peer at IXPs, and look at some of the recommended configuration practices. It has four ASNs participating at the IX, with four routers being assigned to each autonomous system. One router peers at the IX, the other router is internal to the autonomous system (so will have an OSPF and iBGP session with the IX facing router). It also has two routers external transit providers. Lab Exercises 1. A More Advanced IXP. This IXP example has four participating ASNs with each ASN having two routers. One router peers at the IXP; the other router is internal to the ISP's network and connects to an upstream ISP. The two routers in each ASN present at the IXP use OSPF and iBGP to communicate routing information with the each other. The three upstream ASNs are also interconnected with each other, to represent the high speed Internet core. What we have done here is to simulate an Internet Exchange Point and its participants's connections to upstream ISPs. 2. Basic Configuration. Each router team should configure their router to fit into the network layout depicted in Figure 1. Check all connections. Note that all links to the IX switch are by Ethernet. The links between each router in each AS is by also a point-to-point Ethernet connection (to represent the WAN connection from the IX back to the ISP's core network). The links from each AS to their upstream is by Ethernet (to represent the WAN link to the upstream). 3. Addressing Plan. These address ranges should be used throughout this section of the module. AS1234 196.200.220.0/26 AS5678 196.200.220.64/26 AS9101112 196.200.220.128/26 AS13141516 196.200.220.192/26 4. Basic Router Setup. Set up the routers as you would have done in previous modules. That is, basic security, IP addressing etc. The addressing for the point-to-point link between the two routers in each AS is left as an exercise to the Router Teams. But remember, a point-to-point link requires a /30 address block. Don't forget to configure the loopback interfaces too. 5. IXP LAN. The address range used for the IXP LAN is 196.200.221.128/25 the route collector has an IP address of 196.200.221.250. Each of the ASes is assigned addresses sequentially for use on the exchange point LAN. So, for example, AS1234 has 196.200.221.129; AS5678 has 196.200.221.130, etc. 6. Configure the ethernet of each router at the IXP. The ethernet interfaces connected to the IXP should be configured appropriately for a public connection. Review the earlier IXP presentation. The configuration for AS1234 might be: ! interface GigabitEthernet 1/0 description Exchange Point LAN ip address 196.200.221.129 255.255.255.128 no ip directed-broadcast no ip proxy-arp no ip redirects ! If you are unclear as to what any of the configuration lines do, please ask the lab instructor. Checkpoint #1: When you have properly configured your router, and the other routers at the IXP are reachable (i.e. you can ping the other routers), please let the instructor know. 7. Configuring OSPF (part 1). Obviously each team will need to configure OSPF between the two routers in their AS. The router which is internal to the ASN's network should be straightforward to configure. It has one loopback interface, one interface connecting to the router at the IXP, and one interface connecting to the upstream. Therefore OSPF should have two network statements, and one active interface configured. Note we do not configure the interface pointing to the upstream as we will be using next-hop-self for our iBGP sessions. Example for Router 1 might be: ! router ospf 105 log-adjacency-changes passive-interface default no passive-interface GigabitEthernet 1/0 ! interface GigabitEthernet 1/0 ip ospf 105 area 0 ! interface loopback 0 ip ospf 105 area 0 ! 8. Configuring OSPF (part 2). The OSPF configuration on the router connecting to the IXP needs more care. It is highly important that the IXP LAN network does not ever appear in the ISP's routing table, either in OSPF or in iBGP. If it does, there is the possibility of leaking the IX LAN network by the ISP's BGP to other ASNs, providing transit to the IXP. This might not be a good thing, and many IXPs have rules against this behaviour. The OSPF configuration for the IXP router looks something like this, for example for Router 1 ! router ospf 105 log-adjacency-changes passive-interface default no passive-interface GigabitEthernet 1/0 ! interface GigabitEthernet 1/0 ip ospf 105 area 0 ! interface loopback 0 ip ospf 105 area 0 ! NB. There is NO configuration for the Ethernet interface facing the IXP. Do NOT put any in, repeat, do NOT! It is NOT a mistake. 9. Configuring iBGP between upstream facing routers and the IXP facing router. Any router which is connected to other ASNs (the ISP border routers) such as the ISP's upstreams or ISP's peers needs to have a modified iBGP configuration when peering with the IXP facing router. If all the prefixes learned from the upstream ISP are passed onto the IXP facing router, then there is the possibility that the IXP peers (and non-peers) can point static routes for those destinations at the IXP facing router, thereby gaining outbound transit across the local ASN. This is undesirable, and is basically a security risk. Think about how you could solve this. A solution was explained during the presentation. 10. Configuring iBGP on IXP facing routers (Part 1). The iBGP configuration on the IXP facing router needs much more care. We do NOT want the IXP LAN to appear in our iBGP. And because the IXP LAN is not in OSPF, we can't use this net for valid next-hops. So that BGP has a valid next-hop, we use the next-hop-self BGP configuration as we have done for our other iBGP sessions earlier on. For example, for Router 1: router bgp 1 no synchronization neighbor ibgp-peers remote-as 1 neighbor ibgp-peers update-source loopback0 neighbor ibgp-peers next-hop-self ! set external next-hops to us neighbor ibgp-peers password cisco neighbor ibgp-peers descr iBGP peers neighbor ibgp-peers send-community no auto-summary ! This configuration is also valid for other iBGP peers in the network if there were more than two routers in the ASN. ISPs try and keep configuration differences to the absolute minimum, and having two different peer-groups defined for iBGP is usually sufficient! 11. Configuring iBGP on IXP facing routers (Part 2). The second important difference required for the iBGP at the IXP router is that the local address block must NOT be originated here. If the IX router is disconnected from the core backbone for whatever reason, if it originated the ISP's address block, it would end up blackholing all traffic from the IXP participants to the local network. This is highly undesirable, so the correct procedure is to originate the ISP address block only in the core of the network, not the edge, so that normal BGP failover rules can apply. Q. What do you think the normal BGP failover rules might be in this case? A. If the IXP facing router is disconnected from the core network, the iBGP fails, the OSPF fails. So the router no longer hears the announcement of the ISP's address block from the core, so no longer announces it to the IXP peers, its eBGP neighbours. When this happen, the other ASNs will use fall back paths (alternative to the IXP) to reach the local ASN. These are the normal BGP failover rules. 12. Configuring iBGP on the IXP facing routers (Part 3). The second very important requirement at an Internet exchange point is that a default route (or the routes not originated by the local ASN) should not be made available at the peering LAN. If this requirement is not followed, then it is possible for the Exchange Point participants to use the local network to get transit to the rest of the Internet. In this Module, each ASN will be receiving a default route from their upstream provider (a common scenario in the Internet today). This default route must NOT be made available at the ISP's Exchange Point router. There are two ways to do this. Either block the announcement of the default route by iBGP at the borders router which connect to the upstreams, or set up a static default route to the Null interface at the IXP router (or indeed do both!). For example: ip route 0.0.0.0 0.0.0.0 null0 Then if any of the IXP participants point a default route to the local network, the traffic will simply be dumped in the Null interface of their IXP router. Only traffic for specific destinations which available in the routing table on the IXP router will be forwarded to the rest of the network. This is a very important network security requirement. 13. Configuring eBGP on the IXP facing routers. Next, eBGP needs to be set up on the IXP routers. Create a peer-group and apply that peer-group to each eBGP neighbour. A sample configuration for Router1 might be: ip prefix-list myprefixes permit 10.4.48.0/20 ip prefix-list peer100 permit 10.1.0.0/20 .. ip prefix-list peer106 permit 10.7.96.0/20 ! router bgp 1 no synchronization bgp log-neighbor-changes neighbor ixp-peers peer-group neighbor ixp-peers remove-private-AS neighbor ixp-peers prefix-list myprefixes out neighbor ixp-peers route-map set-local-pref in neighbor remote-as 100 neighbor description Peering with AS123400 neighbor peer-group ixp-peers neighbor prefix-list peer100 in .. neighbor remote-as 106 neighbor description Peering with AS123406 neighbor peer-group ixp-peers neighbor prefix-list peer106 in no auto-summary ! route-map set-local-pref permit 10 set local-preference 150 ! The configurations for the other routers will be similar to this one. All router teams will have done sufficient BGP configuration throughout this workshop to extrapolate from the above examples. If in any doubt, ask the lab demonstrator for assistance. Note the prefix-lists. There is a per-peer inbound prefix-list. Some service providers only filter ASes that have inherent dangers, and do not prevent against inbound leaking of prefixes incorrectly originated by the peer AS. But only filtering on prefixes doesn't scale, especially in larger IXPs with large participating service providers as they are frequently adding to the prefixes they announce. The Internet Routing Registry is usually used to solve this problem. (see or example http://www.ripe.net/data-tools/db/the-internet-routing-registry-history-and-purpose) 14. Set up passwords on the eBGP sessions at the IXP. Negotiate with each ASN a password which you can use on your BGP session with them. And then agree to cut the eBGP session over to using passwords such that the eBGP session does not fall over due to password mismatches (as in Module 2). An excerpt from Router 8's configuration might be: router bgp 103 ... neighbor password peer104 ... ! 15. Configuring eBGP between the routers in AS123400, AS200 and AS300. The instructors will operate the three transit ASNs; they will now configure eBGP between each other. No filtering is expected or required between the three ASNs, so the eBGP configuration is quite straightforward, 16. Configuring eBGP with the routers in AS123400, AS200 and AS300. All IXP members must now configure eBGP sessions with the routers in the three transit ASNs. They must expect to receive the default route from their upstreams, and only send their address block to the upstream. An example configuration for Router 2 might be: ip prefix-list myprefixes permit 196.200.223.0/28 ! router bgp 1 no synchronization bgp log-neighbor-changes neighbor remote-as 100 neighbor description Peering with AS123400 neighbor password as16384-password neighbor prefix-list default in neighbor prefix-list myprefixes out no auto-summary 17. Connectivity Test. Check connectivity throughout the IXP network. Each router team should be able to see all the other routers at the IXP. When you are satisfied that BGP is working correctly, try running traceroutes to check the paths being followed. Q. Why do the traceroutes from one ASN to another ASN across the IXP show the middle hop as being star'ed out? Checkpoint #2: Once the BGP configuration has been completed, check the routing table and ensure that you have complete reachability over the entire network. If there are any problems, work with the other router teams to resolve those. 18. (OPTIONAL) Set up eBGP session with Router 18. The lab instructors will have configured Router 18 to be a route collector. This is a router which collects all the routes available at the IX. It serves purpose other than to be an information repository showing how many routes are available at the IX; quite often the IXP management will operate such a router, connected to a Looking Glass web interface, to increase the marketing value of the IX. The more peers who get attracted by the routes available at the IX, the greater the value proposition the IX is to the rest of the members. It's in everyone's interest to peer with the router collector: router bgp 103 ... neighbor 196.200.221.254 remote-as 65534 neighbor 196.200.221.254 description eBGP with the IX Route Collector neighbor 196.200.221254 password ixp-collector neighbor 196.200.221.254 remove-private-AS neighbor 196.200.221.254 prefix-list deny-all in neighbor 196.200.221.254 prefix-list myprefixes out ... ! ip prefix-list deny-all deny 0.0.0.0/0 le 32 Notice that the route collector is running in a private AS -- there isn't really any need for it to use a public AS as the Collector does not need to be directly visible outside of the IXP. Note also the inbound prefix filter blocking all prefixes on the eBGP session with the Route Collector. The Collector will not advertise any prefixes, by design. However, ISPs should never trust any other AS or its operator, so the inbound prefix filter is provided for safety, just in case of problems at the Route Collector. 19. Connectivity Test. Check connectivity throughout the IXP network. Each router team should be able to see all the other routers at the IXP. When you are satisfied that BGP is working correctly, try running traceroutes to check the paths being followed. 20. Completed! The IXP is now complete, up and running. The lab instructors will log into the route collector and show the prefixes visible. All 6 announcements should be clearly seen in the output of sh ip bgp on the route collector. Checkpoint #3: Compare your BGP routing table with that you see on the route collector. If you have missing prefixes, or some other problems, ask the lab demonstrators. 21. Summary. This module has given example configurations as used by Internet Service Providers at Internet Exchange Points. It has concentrated on using prefix-lists only -- more sophisticated configurations are possible by using as-path filters and BGP communities. These examples are left to the reader to consider. If there is time at the end of the workshop, ask the Instructor to test out some other scenarios.