Wright This Way

Linux rp_filter on multihomed system

I was recently working on a Linux system that was providing a NAT service for an environment. Initially, everything seemed to be working fine, but after some additional testing problems were discovered…

What was happening was that network traffic was coming into one interface, but this interface didn’t have a route back to the source system, and so the reply traffic needed to go out the default network interface, as expected. The problem (feature!) though is that newer Linux versions have a feature called Strict Reverse Path Forwarding, which checks to be sure that the interface receiving traffic is able to also able respond to this traffic. For a local subnet, this is usually fine, but for traffic outside of the local subnet, this requires routing, quite possibly by another network interface. And with the Strict Reverse Path Forwarding enabled, if traffic needs to bounce to another interface, then the packet gets dropped. For more info on this, see this link: https://access.redhat.com/solutions/53031

In my case, some testing revealed that I was able to ping the primary network connection, but not the secondary one.

As mentioned, my server has more than one network interface, and in fact has more than one NIC installed. The following command will show what interfaces your OS has configured, in my case, the two that are active are the ens2f0np0 and ens2f1np1. Newer linux versions have moved away from the older eth0, eth1, etc naming. eno here indicates EtherNet Onboard, which is a 4 port card, giving eno1-4. In PCI Slot 2, I have a 2 port card, so that registers as ens2, f0/f1 (function #), and I’m assuming that ‘np’ is Network Port.

[admin@localhost Documents]$ ls /proc/sys/net/ipv4/conf/
all  default  eno1  eno2  eno3  eno4 
 
ens2f0np0  ens2f1np1  lo
[admin@localhost Documents]$

I didn’t need to worry about the ‘lo’ (Loopback) interface, but I pasted the others into this block of code to check the filter status on each.

for i in all default eno1 eno2 eno3 eno4 ens2f0np0 ens2f1np1
do
cat /proc/sys/net/ipv4/conf/$i/rp_filter
done

Executing the script above, led me to this:

[admin@localhost Documents]$ for i in all default eno1 eno2 eno3 eno4 ens2f0np0 ens2f1np1
> do
> cat /proc/sys/net/ipv4/conf/$i/rp_filter
> done
1
0
0
0
0
0
0
0
[admin@localhost Documents]$

Now, reading up on rp_filter (link at top of article), it seemed that this filtering should have been disabled for all of my interfaces, but that first one for ‘all’ interfaces was set to 1, which was actually taking precedence over the individual settings, as its value was higher. Now, I could either set the ‘all’ setting to 0, which would turn this off for the entire system, or I could set my problem interface, ens2f1np1, to a value of 2. For testing, I used the following command:

sysctl -w net.ipv4.conf.ens2f1np1.rp_filter=2

This cleared my issue, and let my system correctly respond to pings on both interfaces, and reply back on that same interface. But, this wouldn’t survive a reboot, and creating a new config file in /etc/sysctl.d was required. Using a text editor, I created the file /etc/sysctl.d/50-rp_filter.conf with the following contents:

net.ipv4.conf.ens2f1np1.rp_filter = 2

Rebooting the system, this setting was successfully applied, and again both interfaces were pinging correctly.

UPDATE: The above was from a RedHat 8.6 system. I’ve just done a fresh install of RedHat 9.0, and this now has 0 for the ‘all’ group, and 1 for all other interfaces. Setting the problem interface to 2 is still a better option that setting it to 0, disabling this, in most cases.