Configuring Multiple Default Routes in Linux

SOURCE: http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/
Assume you have a Linux system with more than one network interface card (NIC) — say eth0 and eth1. By default, administrators can define a single, default route (on eth0). However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.

This can be a bit of a problem — especially when the two NICs share the same parent network and you’re trying to preserve sane traffic flows. In a nutshell, this post will explain how you can ensure traffic going into eth0 goes out only on eth0, as well as enforce all traffic going into eth1 goes out only on eth1.
You’ve found the one post that actually explains this issue; your googling has paid off. You wouldn’t believe how many advanced Linux routing websites out there explain how to route everything including your kitchen sink — yet fail to clearly explain something as simple as this.
As always, we’ll explain by example. Assume the following:

• eth0 - 10.10.70.38 netmask 255.255.255.0
• eth0's gateway is: 10.10.70.254
• eth1 - 192.168.7.126 netmask 255.255.255.0
• eth1's gateway is: 192.168.7.1

First, you’ll need to make sure your Linux kernel has support for “policy routing” enabled. (As a reference, I’m using a v2.6.13-gentoo-r5 kernel.)
During the kernel compilation process, you’ll want to:

cd /usr/src/linux
make menuconfig
Select "Networking --->"
Select "Networking options --->"
[*] TCP/IP networking
[*] IP: advanced router
Choose IP: FIB lookup algorithm (FIB_HASH)
[*] IP: policy routing
[*] IP: use netfilter MARK value as routing key

Next, you’ll want to download, compile, and install the iproute2 [1] utilities. (Most Linux distributions have binary packages for this utility.) Once installed, typing ip route show should bring up your system’s routing table. Type man ip for more information about this utility, in general.
Speaking of which, assume the system’s initial route configuration looks like this:

# netstat -anr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
10.10.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.7.1 0.0.0.0 UG 0 0 0 eth1

So, basically, the system is using eth1 as the default route. If anyone pings 192.168.7.126, then the response packets will properly go out eth1 to the upstream gateway of 192.168.7.1. But what about pinging 10.10.70.38? Sure, the incoming ICMP packets will properly arrive on eth0, but the outgoing response packets will be sent out via eth1! That’s bad.
Here’s how to fix this issue. Borrowing the method from a really sketchy website [2], you’ll first need to create a new policy routing table entry within the /etc/iproute2/rt_tables. Let’s call it table #1, named “admin” (for routing administrative traffic onto eth0).

# echo "1 admin" >> /etc/iproute2/rt_tables

Next, we’re going to set a couple of new entries within this “admin” table. Specifically, we’ll provide information about eth0‘s local /24 subnet, along with eth0‘s default gateway.

ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
ip route add default via 10.10.70.254 dev eth0 table admin

At this point, you’ve created a new, isolated routing table named “admin” that really isn’t used by the OS just yet. Why? Because we still need to create a rule referencing how the OS should use this table. For starters, type ip rule show to see your current policy routing ruleset. Here’s what an empty ruleset looks like:

0: from all lookup local
32766: from all lookup main
32767: from all lookup default

Without going into all the boring details, each rule entry is evaluated in ascending order. The main gist is that your normal main routing table appears as entry 32766 in this list. (This would be the normal route table you’d see when you type netstat -anr.)
We’re now going to create two new rule entries, that will be evaluated before the main rule entry.

ip rule add from 10.10.70.38/32 table admin
ip rule add to 10.10.70.38/32 table admin

Typing ip rule show now shows the following policy routing rulesets:

0: from all lookup local
32764: from all to 10.10.70.38 lookup admin
32765: from 10.10.70.38 lookup admin
32766: from all lookup main
32767: from all lookup default

Rule 32764 specifies that for all traffic going to eth0‘s IP, make sure to use the “admin” routing table, instead of the “main” one. Likewise, rule 32765 indicates that for all traffic originating from eth0‘s IP, make sure to use the “admin” routing table as well. For all other packets, use the “main” routing table. In order to commit these changes, it’s a good idea to type ip route flush cache.
Congratulations! You’re system should now properly route traffic to these two different default gateways. For more than 2 NICs, repeat the table/rule creation process as necessary.
Please provide comments, if you find any errors or have corrections to this post. I don’t claim that this method will work for everyone; this information is designed primarily to preserve my sanity, when configuring routing on future multi-NIC Linux systems.
References:
[1] http://www.policyrouting.org
[2] http://www.linuxhorizon.ro/iproute2.html
Update: Here are some additional resources, that I have found useful.
http://lartc.org/howto/lartc.rpdb.multiple-links.html
http://linux-ip.net/html/routing-tables.html
Update: Apparently, OpenBSD also now supports multiple default routes through a new feature called the Virtual Routing Table:
http://www.packetmischief.ca/2011/09/20/virtualizing-the-openbsd-routing-table/