Sometimes you think a problem is going to be really hard to solve, but it turns out to be pretty easy. A few weeks ago we started implementing a new feature for Hypernodes: Dedicated IP’s. This would make migrating and upgrading a lot easier for our customers because their node IP won’t change anymore. One problem we had to tackle whilst implementing this feature was that pretty much all cloud providers handled dedicated IP management in a different way. For example AWS calls them Elastic IP’s, while DigitalOcean calls them Floating IP’s. Of course this can all be coded into the Hypernode logic, so no problem, you’d say.
However, it didn’t take long for us to find out that mail traffic over the Dedicated IP didn’t work properly with DigitalOcean’s Floating IP’s. It turned out that this cloud provider was blocking port 25 and 587 outbound for all floating IP’s. DigitalOcean responded that this was an acknowledged new feature on their product roadmap, but with low priority due to the perceived “very rare instances where a customer deliberately configures their route to use the floating IP address in this way”. So now we were faced with the problem that we could only fully implement Floating IP’s without standard mail connectivity. Not an ideal situation to say the least. A solution was needed.
The concept of Floating IP’s is that the actual public IP is bound on a gateway somewhere and that the traffic is translated to the internal node IP. Some cloud providers always use this setup even without Floating IP’s and some bind the original public node IP directly on the node. The latter was also the case for the DigitalOcean Floating IP’s we were having problems with. So a possible solution was to route outbound mail traffic over the original node gateway. Only problem was that the internal IP (used for the floating IP) and the public node IP are bound on the same interface. So how do you route traffic destined for ports 25 and 587 to another gateway, while not affecting all other outbound traffic on the same interface?
We thought that this was next to impossible to achieve. But after some extensive googling it turned out to be pretty easy. I found this dusty stackexchange article which looked very promising. And it turned out this approach actually works. But what are we actually doing here? We have to go deeper.
This is the part we were particularly interested in:
# Populate secondary routing table ip route add default via 192.168.0.1 dev wlan0 table wlan-route # Anything with this fwmark will use the secondary routing table ip rule add fwmark 0x1 table wlan-route # Mark these packets so that iproute can route it through wlan-route iptables -A OUTPUT -t mangle -o eth1 -p tcp --dport 443 -j MARK --set-mark 1 # now rewrite the src-addr iptables -A POSTROUTING -t nat -o wlan0 -p tcp --dport 443 -j SNAT --to 192.168.0.2
- First we would have to create a extra routing table, turns out the linux kernel support this by default (link).
- Then add a default route to this table that uses the original node gateway IP.
- Then add the IP rule specified above to route packets with an (hexadecimal) fwmark of 1 to this routing table.
- Then add an fwmark of 1 to all packets to destination ports 25 and 587 using iptables.
- Finally add the SNAT to give this traffic the appropriate source IP using iptables. (The original node IP.)
This will result in the following bash commands:
# Add the routing table echo "200 mail-route" >> /etc/iproute2/rt_tables # Add the default route to the new table (original node gateway=188.8.131.52) ip route add default via 184.108.40.206 dev eth0 table mail-route # Add the ip rule to use this route with fwmark 0x1 ip rule add fwmark 0x1 table mail-route # Set fwmark for destination ports 25 and 587 iptables -A OUTPUT -t mangle -o eth0 -p tcp --dport 25 -j MARK --set-mark 1 iptables -A OUTPUT -t mangle -o eth0 -p tcp --dport 587 -j MARK --set-mark 1 # Rewrite the source address for these packets (original node IP=220.127.116.11) iptables -A POSTROUTING -t nat -o eth0 -p tcp --dport 25 -j SNAT --to 18.104.22.168 iptables -A POSTROUTING -t nat -o eth0 -p tcp --dport 587 -j SNAT --to 22.214.171.124
Pretty similar to the solution from the stackexchange article and guess what: it works. Now only these two ports will be routed over the original gateway instead of the floating IP. The visual representation of what we just did looks something like this:
Conclusion: Linux is awesome 🙂
Since we had a hard time believing that our case is “a rare instance”, we decided to share this information with everybody who would ever want to use DigitalOcean’s Floating IP’s for outbound mail. Hoping that one day we can save somebody from the headache we had.