Policy Based Routing example: route one subnet via ISP A and another via ISP B

Goal

Setup a network with a Cisco router that routes one local subnet via internet connection A and another local subnet via internet connection B.

In this example we’ll use one LAN side, which has two subnets:

  • 192.168.1.0/24
  • 192.168.2.0/24

We want traffic destined for the internet, originating from the 192.168.1.0/24 network, to be sent to ISP A, which is connected to FastEthernet0

We want traffic destined for the internet, originating from the 192.168.2.0/24 network, to be sent to ISP B, which is connected to FastEthernet1

Both internet connections get their IP address via DHCP.

Prerequisites

For this  configuration you’ll need:

  • One dual WAN router, such as a 1811
  • Two internet connections (or simulated ones)
  • 2 nodes, one for the 192.168.1.0/24 subnet and one for the 192.168.2.0/24 subnet. These can be two different computers, or two virtual machines, etc…
    • ClientA, Windows 7, will connect to 192.168.1.0/24 and surf the net using ISP A
    • ClientB, Windows XP, will connect to 192.168.2.0/24 and surf the net using ISP B

Configure the router

Let’s start by defining our WAN interfaces first, which are FastEthernet0 and FastEthernet1 for a Cisco 1811.  Ports FastEthernet2 to 9 are the integrated switch. Both WAN interfaces use DHCP to obtain an IP address.

Setting up the WAN interfaces

r3(config)#interface FastEthernet0
r3(config-if)#ip address dhcp
r3(config-if)#ip nat outside
r3(config-if)#no shutdown
r3(config-if)#interface FastEthernet1
r3(config-if)#ip address dhcp
r3(config-if)#ip nat outside
r3(config-if)#no shutdown
r3(config-if)#exit
r3(config)#

Verifying WAN operation

We can verify our WAN connection by running the following commands and observing their output.

! Display a list of interfaces and their IPv4 addresses
r3#show ip interface brief
Interface     IP-Address    OK? Method Status                Protocol
Async1        unassigned    YES unset  down                  down
FastEthernet0 213.193.228.1 YES DHCP   up                    up
FastEthernet1 88.9.5.1      YES DHCP   up                    up
FastEthernet2 unassigned    YES unset  up                    up
FastEthernet3 unassigned    YES unset  up                    down
FastEthernet4 unassigned    YES unset  up                    down
FastEthernet5 unassigned    YES unset  up                    down
FastEthernet6 unassigned    YES unset  up                    down
FastEthernet7 unassigned    YES unset  up                    down
FastEthernet8 unassigned    YES unset  up                    down
FastEthernet9 unassigned    YES unset  up                    down
NVI0          unassigned    YES unset  administratively down down
Vlan1         192.168.1.254 YES NVRAM  up                    up
r3#! Display IPv4 routes
r3#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 213.193.228.254 to network 0.0.0.0

S*    0.0.0.0/0 [254/0] via 213.193.228.254
                [254/0] via 88.9.5.254
      88.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        88.9.5.0/24 is directly connected, FastEthernet1
L        88.9.5.1/32 is directly connected, FastEthernet1
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Vlan1
L        192.168.1.254/32 is directly connected, Vlan1
      192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.2.0/24 is directly connected, Vlan1
L        192.168.2.254/32 is directly connected, Vlan1
      213.193.228.0/24 is variably subnetted, 2 subnets, 2 masks
C        213.193.228.0/24 is directly connected, FastEthernet0
L        213.193.228.1/32 is directly connected, FastEthernet0
r3#! Ping ISP A's gateway
r3#ping 213.193.228.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 213.193.228.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
r3#! Ping ISP B's gateway 
r3#ping 88.9.5.254
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 88.9.5.254, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
r3#! Ping known-to-reply-to-ping address on the internet 
r3#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 20/21/24 ms
r3#

Setting up the local network

We’re going to keep this simple. As described we have two internal IP ranges, we’ll assign an IP address to Vlan1 for each of the IP ranges. Of course this scenario is of little use in the real world, you would probably have two VLAN’s, etc… However, this is just a theory of operations exercise, we’ll just configure the computers connecting to the router manually.

r3(config)#interface Vlan1
r3(config-if)#ip address 192.168.1.254 255.255.255.0
r3(config-if)#ip address 192.168.2.254 255.255.255.0 secondary
r3(config-if)#ip nat inside
r3(config-if)#no shutdown
r3(config-if)#exit
r3(config)#

Verifying local network operation

Once we’ve configured the Vlan1 interface, we can check again with show ip interface that we have correctly configured the respective interface:

r3#show ip interface Vlan1

Now we configure our client machines. We will statically configure:

  • ClientA with IP address 192.168.1.5
  • ClientB with IP address 192.168.2.5

After applying the  network configuration, verify that we can ping the routers.

Configuring NAT

We configure two access-lists:

  • One for the 192.168.1.0/24 that should get translated to FastEthernet0’s IP address
  • One for the 192.168.2.0/24 that should get translated to FastEthernet1’s IP address
r3(config)#ip access-list standard 1
r3(config-std-nacl)#permit 192.168.1.0 0.0.0.255
r3(config-std-nacl)#exit
r3(config)#ip access-list standard 2
r3(config-std-nacl)#permit 192.168.2.0 0.0.0.255
r3(config-std-nacl)#exit
r3(config)#
r3(config)#ip nat inside source list 1 interface FastEthernet0 overload
r3(config)#ip nat inside source list 2 interface FastEthernet1 overload
r3(config)#

Basically what we’re telling here is that, when the packet’s source address gets translated, it should get the source address of the interface as specified. Thus, when packets from 192.168.1.0/24 get NAT’ed, they get the source IP of FastEthernet0. When packets from 192.168.2.0/24 get NAT’ed, they get the source IP of FastEthernet1.

Now, when the packets have their correct source address, that doesn’t mean they automatically fly out the right interface. We need to explicitly define that any packets for the interface from 192.168.1.0/24 must go via FastEthernet0 and from 192.168.2.0/24 via FastEthernet1. We do this with a route map.

Building the route-map

Using a route-map we can build policy based  routing. A route map can have several entries, and each entry can have any number of match & set statements. If an entry doesn’t match, the router continues evaluating the route map until it finds an entry that does. When the router finds an entry that does match, it does whatever you say it should do with the set statements.

Matching

The match statements describe on which traffic we should act. In this scenario we have two distinct types of traffic: that for ISP A and that for ISP B. So we’ll need to define two route map entries: one matching the traffic destined for ISP A and one matching the traffic destined for ISP B.

We will do the matching with an access-list. We have two different subnets we want to match, 192.168.1.0/24 and 192.168.2.0/24. But, if you remember correctly, we already have two access-lists matching those exacts subnets (ip access-list 1 & 2). So we’re going to re-use those.

Setting

The set statements describe what should happen on the traffic that matches. Again, we have two distinct types of traffic, that for ISP A and that for ISP B. When we’ve correctly matched the traffic we want, we have to set something, namely, what to do with it. The first route-map entry (matching traffic for ISP A) should make the router send the traffic to FastEthernet0. The second entry (matching traffic for ISP B) should make the router send the traffic to FastEthernet1.

Show me the money

We will call our route-map “ISPSelect”. (IOS calls it a tag).

! Create the first entry in our new route-map called "ISPSelect"
r3(config)#route-map ISPSelect permit 10
! Match the traffic from 192.168.1.0/24 with access-list 1
r3(config-route-map)#match ip address 1
! Set the interface the traffic should go to, to FastEthernet0
r3(config-route-map)#set interface FastEthernet0
r3(config-route-map)#exit

! Create the second entry in our route-map called "ISPSelect"
r3(config)#route-map ISPSelect permit 20
! Match the traffic from 192.168.2.0/24 with access-list 2
r3(config-route-map)#match ip address 2
! Set the interface the traffic should go to, to FastEthernet1
r3(config-route-map)#set interface FastEthernet1
r3(config-route-map)#exit
r3(config)#

Verifying route-map configuration

r3#show route-map ISPSelect
route-map ISPSelect, permit, sequence 10
 Match clauses:
 ip address (access-lists): 1
 Set clauses:
 interface FastEthernet0
 Policy routing matches: 0 packets, 0 bytes
route-map ISPSelect, permit, sequence 20
 Match clauses:
 ip address (access-lists): 2
 Set clauses:
 interface FastEthernet1
 Policy routing matches: 0 packets, 0 bytes
r3#

Applying the route-map

Now where do we apply this route-map? Let’s take the following rule into account:

Packets are first routed, then NAT’ed.

Since we’re talking about a route-map, the route-map influences what happens when the router routes the packets. So we have to apply this map to the interface where the packets are coming in. In this case, this is Vlan1. Using the ip policy route-map statement in the interface configuration, we can define which route-map we want to use there:

r3(config)#interface Vlan 1
r3(config-if)#ip policy route-map ISPSelect
r3(config-if)#exit
r3(config)#

Verifying route-map application

r3#show ip interface Vlan 1
Vlan1 is up, line protocol is up
 Internet address is 192.168.1.254/24
 Broadcast address is 255.255.255.255
 Address determined by non-volatile memory
 MTU is 1500 bytes
 Helper address is not set
 Directed broadcast forwarding is disabled
 Secondary address 192.168.2.254/24
 Outgoing access list is not set
 Inbound access list is not set
 Proxy ARP is disabled
 Local Proxy ARP is disabled
 Security level is default
 Split horizon is enabled
 ICMP redirects are always sent
 ICMP unreachables are always sent
 ICMP mask replies are never sent
 IP fast switching is enabled
 IP fast switching on the same interface is disabled
 IP Flow switching is disabled
 IP CEF switching is enabled
 IP CEF switching turbo vector
 IP Null turbo vector
 IP multicast fast switching is enabled
 IP multicast distributed fast switching is disabled
 IP route-cache flags are Fast, CEF
 Router Discovery is disabled
 IP output packet accounting is disabled
 IP access violation accounting is disabled
 TCP/IP header compression is disabled
 RTP/IP header compression is disabled
 Policy routing is enabled, using route map ISPSelect
 Network address translation is enabled, interface in domain inside
 BGP Policy Mapping is disabled
 Input features: Stateful Inspection, Virtual Fragment Reassembly, Virtual Fragment Reassembly After IPSec Decryption, Policy Routing, MCI Check
 Output features: NAT Inside, Stateful Inspection, NAT ALG proxy
 WCCP Redirect outbound is disabled
 WCCP Redirect inbound is disabled
 WCCP Redirect exclude is disabled
r3#

Testing

Right now you should be able to surf the internet from both subnets. Let’s verify traffic from our subnets are really following the right path. Let’s start by doing a traceroute. I’m going to trace the path to 8.8.8.8 and I’ll be using the -d switch to tracert to not do a reverse lookup on the IP addresses.

Doing a traceroute

Note the second hop.

Result on Windows 7, 192.168.1.0/24 subnet:

Result on Windows XP, 192.168.2.0/24 subnet:

A successful result is when packets from the Windows 7 computer get routed via 213.193.228.254, which is ISP A’s gateway and packets from the Windows XP computer get routed via 88.9.5.254, which is ISP B’s gateway.

Checking the NAT translation table

Further verification, check the NAT translation table on the router using show ip nat translations command. I’ll be pinging 8.8.8.8 from both the Windows 7 and the Windows XP workstation, which will create exactly two NAT entries in the translation table: one from the Windows 7 PC to 8.8.8.8 using ISP A and one from the Windows XP PC to 8.8.8.8 using ISP B.

r3#show ip nat translations
Pro  Inside global   Inside local    Outside local  Outside global
icmp 213.193.228.1:1 192.168.1.5:1   8.8.8.8:1      8.8.8.8:1
icmp 88.9.5.1:512  192.168.2.5:512 8.8.8.8:512    8.8.8.8:512
r3#

Further testing

You can do a packet capture between the router and the ISP’s modems.

You can pull the cables for the internet connections and check which subnets can still reach the internet.

Full configuration

!
! Last configuration change at 11:52:02 UTC Sat Jan 28 2012
!
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname r3
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
crypto pki token default removal timeout 0
!
!
dot11 syslog
ip source-route
!
!
!
!
!
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
license udi pid CISCO1811/K9 sn CENSORED
!
!
!
!
!
!
!
!
!
interface Multilink1
 no ip address
 ppp multilink
 ppp multilink group 1
!
interface FastEthernet0
 ip address dhcp
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface FastEthernet1
 ip address dhcp
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface FastEthernet2
!
interface FastEthernet3
!
interface FastEthernet4
!
interface FastEthernet5
!
interface FastEthernet6
!
interface FastEthernet7
!
interface FastEthernet8
!
interface FastEthernet9
!
interface Vlan1
 ip address 192.168.2.254 255.255.255.0 secondary
 ip address 192.168.1.254 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ip policy route-map ISPSelect
!
interface Async1
 no ip address
 encapsulation slip
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat inside source list 1 interface FastEthernet0 overload
ip nat inside source list 2 interface FastEthernet1 overload
!
logging esm config
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 2 permit 192.168.2.0 0.0.0.255
!
!
!
!
route-map ISPSelect permit 10
 match ip address 1
 set interface FastEthernet0
!
route-map ISPSelect permit 20
 match ip address 2
 set interface FastEthernet1
!
!
!
control-plane
!
!
!
line con 0
 speed 115200
line 1
 modem InOut
 stopbits 1
 speed 115200
 flowcontrol hardware
line aux 0
line vty 0 4
 login
 transport input all
!
end

Setting up an aggregated link between a Cisco switch and a VMware ESXi server

Goal

Increase bandwidth between the network and an ESXi host by setting up an aggregated link between the two.

These aggregated links are commonly refered to as an etherchannel, trunk, portchannel or teamed NIC’s.

Prerequisites

  • A recent Cisco switch that supports load balancing over an Etherchannel based on source and destination IP addresses. (For example, the popular Catalyst 2950 series do not support this loadbalancing method). You can check if your switch supports this load balancing by checking if the exec command
    port-channel load-balance src-dst-ip
    is available. If it’s not, you can’t use this switch for the purpose described in this post.

Caveats

  • ESXi does not support dynamic aggregated links with protocols like LACP. One must manually configure the link on both ends.
  • Available bandwidth to 1 single host will not increase, as this is the nature of the aggregation link technologies being used. If you have two 100 MBit links, the maximum attainable speed between a virtual machine and a single host on the network will still be 100 MBit. However, if two hosts were to connect to the virtual machine, chances are pretty good one host’s traffic will go via one physical link and the other host’s via the second physical link.
  • This example aggregates two physical links into one, you can use more. You can mix different port speeds, but recommended configuration is all links having the same speed.

Configure ESXi

Edit the network settings by going to Configuration -> Networking. Edit the virtual network properties which you want to create an aggregated link for, in this example this is vSwitch0.

Next, add the second network interface to the vSwitch in the Network Adapters tab:

Now we need to configure ESXi to bond the links on these to adapters together. Go back to the Ports tab and edit the vSwitch properties:

On the vSwitch properties window, go to the last tab NIC Teaming and set Load Balancing to “Route based on IP hash”:

That’s it for vmware. Now we need to configure the switch to create the aggregated link.

Configure the switch

In this example FastEthernet 0/23 and FastEthernet 0/24 are connected to my VMware ESXi server, so I’m going to use the interface range commands to apply the necessary configuration to both switchports.

It’s important you match the loadbalancing method ESXi uses to the one the switch uses. This is done using the port-channel load-balance command.

s2(config)#interface range FastEthernet 0/23 – 24
s2(config-if-range)#
s2(config-if-range)#channel-group 1 mode on
Creating a port-channel interface Port-channel 1
00:25:49: %LINK-3-UPDOWN: Interface Port-channel1, changed state to up
00:25:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
s2(config-if-range)#exit
s2(config)#port-channel load-balance src-dst-ip

Verify port-channel operation:

s2#show interface port-channel 1
Port-channel1 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 64d9.89ee.1234 (bia 64d9.89ee.1234)
  Description: To vwmare for VMs
  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported
  Members in this channel: Fa0/23 Fa0/24
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 6000 bits/sec, 2 packets/sec
  5 minute output rate 3000 bits/sec, 3 packets/sec
     32312407 packets input, 33220875322 bytes, 0 no buffer
     Received 135526 broadcasts (71135 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 71135 multicast, 0 pause input
     0 input packets with dribble condition detected
     55382934 packets output, 67979911754 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
s2#

References