Or How I learned to stop worrying and love my dual WAN with routed subnet and policy based routing pf firewall
One of the reasons I continue to use FreeBSD for my gateway/router/firewall is the pf firewall. My routing needs are way more elaborate than anything a pre-cooked router package like DD-WRT or even pfSense can do, so I have to roll my own firewall. Currently I have two DSL connections, through two different companies. I have a dynamic IP connection from my employer, Primus Canda, and I have a static IP connection with a routed /30 subnet from TekSavvy.
My setup needs to satisfy the following requirements:
- The Primus connection is used for all local LAN traffic
- The TekSavvy connection is used for all traffic to and from the routed subnet
- Connections from the local LAN to the routed subnet do not traverse the internet
- Traffic shaping on both connections, with different rulesets for each connection
- UPnP support, using miniupnpd
For the connection for the LAN, the traffic shaping needs to do the following:
- SSH and DNS traffic need high priority
- Traffic to and from my workplace VPN needs priority
- VoIP traffic needs high priority
- HTTP and regular web traffic should feel fast and responsive
- Anything left over goes to P2P and other uncategorized traffic
For the connection with the routed subnet, the requirements are a little different:
- SSH and DNS need high priority
- Inbound FTP control traffic (ie, not the actual data but just the control connection) needs priority
- Traffic originating from the routed subnet needs priority (this mostly just amounts to DNS requests and package updates
- FTP data traffic needs to fill in whatever is left over
This is all possible with pf, and I find the pf.conf format to be far more readable and thus less prone to errors than an iptables config file. To accomplish this I use packet tagging to label packets, and then use policy based routing to direct and control the traffic.
This article assumes some understanding of firewall/routing concepts and some familiarity with pf already as explaining the basics of pf are beyond the scope of this article. Lots of resources for getting comfortable with pf can be found here. Without further adieu, here is my pf.conf:
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
############
## Macros ##
############
# WAN wan interface macros
wan_if = tun0
wan_gw = tun0:peer
tek_if = tun1
tek_gw = tun1:peer
# DMZ interface macros
dmz_if = re1
dmz_gw = re1
table const { 2.2.2.240/30 }
# LAN interface macros
lan_if = re0
table <lan_net> const { $lan_if:network }
# A couple quick helper macros
icmp_types = “{ echoreq, unreach }”
blockbrutes = “flags S/SA keep state (max-src-conn 15, max-src-conn-rate 4/30, overload flush global)”
# Internal Host macros
table const { 192.168.1.25 }
table const { 192.168.1.30 }
table const { 192.168.1.50 }
table const { 2.2.2.242 }
# Port macros
mythbackend01_web = “{ 80 }”
macpro_p2p = “{ 15001 }”
jobs01_p2p = “{ 49160:49300 }”
lan_defq = “{ 21, 25, 80, 443, 110, \
143, 465, 587, 993, 995, \
1723, 1863, 3389, 5060, 5190, \
5222, 5223, 5269, 6665:6669, 8001, 8080 }”
# External Host macros
table const { 4.4.4.120 }
table const { \
6.6.6.6, 6.6.6.8, \
6.6.6.10, 6.6.6.12, \
6.6.6.14, 6.6.6.16, \
6.6.5.13 }
table const { \
0.0.0.0/8, 10.0.0.0/8, \
127.0.0.0/8, 169.254.0.0/16, \
172.16.0.0/12, 192.168.0.0/24, \
240.0.0.0/4, 255.255.255.255 }
table persist { 224.0.0.0/4, 239.0.0.0/4 }
table persist[/cc]
This is just a set of macros that I define to ease working with the rules below. Note that the dmz_if is actually the routed subnet from TekSavvy. The blockbrutes macro is used in my rules below, particularly for inbound FTP and SSH connections. Like the name suggests, this is to keep out the brute force attackers who are running dictionary attacks and don’t have the good sense to be subtle about it. I would still recommend running something like denyhosts. I also define the VPN gateway used by my workplace and the various VoIP gateways used by my workplace as well (I work in the VoIP department, so I do a lot of testing with these gateways).
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
######################
## Firewall Options ##
######################
set skip on lo
set block-policy return
set state-policy if-bound
set loginterface $wan_if
set limit states 30000[/cc]
Make sure the state policy is bound to the interface, as otherwise state can be matched on any interface which makes policy based routing difficult if not impossible. This does mean that any single connection now has two states though, on the WAN and the LAN interfaces, so I bumped up the number of states to 30,000 from the default of 10000. My router has plenty of RAM (512M) so keeping 30000 states is easy.
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
##################
## Scrub Options #
##################
# These scrub as per normal on the other interfaces
scrub in on $wan_if all fragment reassemble
scrub in on $tek_if all fragment reassemble
scrub in on $lan_if all random-id
scrub in on $dmz_if all random-id[/cc]
These scrub options prevent packet fragmentation, particularly with the TekSavvy link with is irregular in that it is a Multilink PPP link (see this post about MLPPP).
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
#############
## Queuing ##
#############
# Set queues on wan_if
# upstream: 882000 bits, downstream: 8666300 bits
altq on $wan_if bandwidth 760Kb hfsc (linkshare 760Kb upperlimit 760Kb) queue { wan_ackq, wan_dnsq, wan_vipq, wan_sshq, wan_webq, wan_defq, wan_p2pq }
queue wan_ackq bandwidth 5% priority 7 qlimit 200 hfsc (realtime 15%)
queue wan_dnsq bandwidth 5% priority 6 qlimit? 50 hfsc (realtime 5%)
queue wan_vipq bandwidth 5% priority 5 qlimit? 50 hfsc (realtime 88Kb)
queue wan_sshq bandwidth 10% priority 4 qlimit? 50 hfsc (realtime 10%)
queue wan_webq bandwidth 10% priority 3 qlimit? 50 hfsc (realtime 10%)
queue wan_defq bandwidth 40% priority 2 qlimit? 50 hfsc (default realtime 25%)
queue wan_p2pq bandwidth 5% priority 0 qlimit 25 hfsc (ecn red upperlimit 600Kb)
# Set queues on tek_if
# upstream: 416000 bits, downstream: 5056000 bits
altq on $tek_if bandwidth 325Kb hfsc (linkshare 325Kb upperlimit 325Kb) queue { tek_ackq, tek_dnsq, tek_sshq, tek_defq, tek_ftpq }
queue tek_ackq bandwidth 5% priority 7 qlimit 200 hfsc (realtime 15%)
queue tek_dnsq bandwidth 5% priority 6 qlimit? 50 hfsc (realtime 5%)
queue tek_sshq bandwidth 10% priority 4 qlimit? 50 hfsc (realtime 10%)
queue tek_defq bandwidth 50% priority 3 qlimit? 50 hfsc (default realtime 35%)
queue tek_ftpq bandwidth 5% priority 0 qlimit 25 hfsc (ecn red)[/cc]
These are the two ALTQ traffic shapers. I use the HFSC packet scheduler (though not to it’s full potential, I used to have a p2p queue and an ftp queue as child queues of a bulk queue, which was useful in a previous network incarnation). I define the maximum throughput of the link (typically for DSL take your sync rate, subtract about 13% for PPP overhead, then subtract 5% again for good measure, and use that as your upper limit). Some good info on HFSC and ALTQ can be found here.
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
######################
## NAT Translations ##
######################
# NAT traffic from the LAN to the tubes
nat on $wan_if from to any -> ($wan_if)
nat on $dmz_if from to -> ($dmz_if)
# Forward ports as requested
rdr on $wan_if inet proto { tcp, udp } from any to ($wan_if) port $mythbackend01_web ->
rdr on $wan_if inet proto { tcp, udp } from any to ($wan_if) port $jobs01_p2p ->
rdr on $wan_if inet proto { tcp, udp } from any to ($wan_if) port $macpro_p2p ->
# UPnPd rdr anchor
rdr-anchor “miniupnpd”[/cc]
I NAT traffic from the LAN to both the Internet and to the routed subnet. Same inbound port forwards, and an anchor rule for miniupnpd. Nothing particularly exceptional here.
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
##################
## Filter Rules ##
##################
###############
# Block Rules #
###############
# Block everything.
block log all
# Block logically impossible packets
block drop in log quick on $wan_if from to any
block drop out log quick on $wan_if from any to
# A little more pre-emptive removal
antispoof log for $wan_if
antispoof log for $lan_if
antispoof log for $tek_if
antispoof log for $dmz_if
# Block the bruteforcers
block drop in log quick from to any[/cc]
Because pf runs a packet through the entire ruleset and the last rule to match a packet is the one that is used you can create a default-deny policy by starting your ruleset by blocking everything and adding in pass rules for traffic you do want. This is by far the best method for building a firewall. The only exception to this is if a rule has the quick keyword, then the rest of the ruleset is ignored (which is by the blockbrutes and martian rules are not redundant).
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
##########################################
# Tag Traffic To/From The Gateway Itself #
##########################################
# Let SSH and ICMP connect to the gateway
pass in log on $wan_if inet proto icmp from any to ($wan_if) icmp-type $icmp_types queue wan_dnsq
pass in log on $wan_if inet proto tcp from any to ($wan_if) port ssh $blockbrutes queue wan_sshq
pass in log on $lan_if inet proto tcp from to ($lan_if) port ssh tag lan
pass in log on $dmz_if inet proto tcp from to ($dmz_if) port ssh tag dmz_gw
# Allow gateway to connect to the internet
pass out log on $wan_if inet proto icmp from ($wan_if) to any icmp-type $icmp_types queue wan_dnsq
pass out log on $wan_if inet proto { tcp, udp } from ($wan_if) to any queue wan_defq
pass out log on $wan_if inet proto udp from ($wan_if) to any port domain queue wan_dnsq
pass out log on $wan_if inet proto tcp from ($wan_if) to any port ssh queue wan_sshq
# Allow gateway to pass traffic out the teksavvy interface
pass out log on $tek_if inet proto icmp from ($tek_if) to any icmp-type $icmp_types queue tek_dnsq
pass out log on $tek_if inet proto { tcp, udp } from ($tek_if) to any queue tek_defq
# It would suck for the router to not be able to talk to the LAN or DMZ
pass out log on $lan_if inet from ($lan_if) to tag lan
pass out log on $dmz_if inet from ($dmz_if) to tag dmz_gw[/cc]
The gateway itself is an exception in the policy based routing setup, because traffic originating or terminating on the gateway itself only ever passes through one interface. So the rules here consist of both pass in and pass out rules. The rest of the ruleset is broken into two sections; the ‘pass in’ rules that tag all the traffic and let the traffic into the firewall but don’t actually let the traffic exit the firewall. Then we have a policy enforcement section section that uses the tags on the packets to determine if a packet may exit the firewall. This makes the rules particularly easy to read.
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
#######################
# Redirect Pass Rules #
#######################
pass in log on $wan_if inet proto { tcp, udp } from any to port $mythbackend01_web tag lan queue wan_webq
pass in log on $wan_if inet proto { tcp, udp } from any to port $jobs01_p2p tag lan queue wan_p2pq
pass in log on $wan_if inet proto { tcp, udp } from any to port $macpro_p2p tag lan queue wan_p2pq[/cc]
These are just the companion pass rules to the redirect rules in the NAT section.
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
########################################
# Tag Traffic To/From The TekSavvy DMZ #
########################################
# Let anything access the DMZ
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto icmp from any to tag dmz queue tek_dnsq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to tag dmz queue (tek_defq, tek_ackq)
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto udp from any to tag dmz queue tek_defq
# Queue traffic destined for the first server
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to port ftp $blockbrutes tag dmz queue tek_ftpq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to port ssh $blockbrutes tag dmz queue tek_sshq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to port domain tag dmz queue tek_dnsq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto udp from any to port domain tag dmz queue tek_dnsq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to port http tag dmz queue tek_defq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to port rsync tag dmz queue tek_ftpq
pass in log on $tek_if reply-to ($tek_if $tek_gw) inet proto tcp from any to port 55500:55600 tag dmz queue tek_ftpq
# Outbound traffic originating in the DMZ
pass in log on $dmz_if route-to ($tek_if $tek_gw) inet proto icmp from to any icmp-type $icmp_types tag tek_dns
pass in log on $dmz_if route-to ($tek_if $tek_gw) inet proto { tcp, udp } from to any tag tek_def
pass in log on $dmz_if route-to ($tek_if $tek_gw) inet proto tcp from to any port ssh tag tek_ssh
pass in log on $dmz_if route-to ($tek_if $tek_gw) inet proto { tcp, udp } from to any port domain tag tek_dns
pass in log on $dmz_if route-to ($tek_if $tek_gw) inet proto { tcp, udp } from to any port ntp tag tek_dns
pass in log on $dmz_if inet from to ($dmz_if) tag dmz[/cc]
Because the TekSavvy connection is not the default route for the gateway I need to explicitly specify that traffic to and from the routed subnet should be routed to the TekSavvy connection; this is accomplished with the reply-to and route-to flags in the rules. Inbound connections need to be both queued and tagged as traffic destined for the routed subnet, while traffic originating in the routed subnet is merely tagged to mark its destination interface and traffic shaper queue.
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
###############################
# Tag Traffic To/From The LAN #
###############################
# Tag traffic from the LAN to the tubes
pass in log on $lan_if inet proto icmp from ; to any tag wan_dns
pass in log on $lan_if inet proto { tcp, udp } from to any tag wan_p2p
pass in log on $lan_if inet proto tcp from to any port ssh tag wan_ssh
pass in log on $lan_if inet proto { tcp, udp } from to any port domain tag wan_dns
pass in log on $lan_if inet proto { tcp, udp } from to any port ntp tag wan_dns
pass in log on $lan_if inet proto udp from to any port 5060 tag wan_vip
pass in log on $lan_if inet proto udp from to any port 5062 tag wan_vip
pass in log on $lan_if inet proto udp from to any port 33433 >< 33626 tag wan_dns
pass in log on $lan_if inet proto tcp from to any port $lan_defq tag wan_def
pass in log on $lan_if inet proto { tcp, udp } from to any port 3283 tag wan_def
pass in log on $lan_if inet proto { tcp, udp } from to any tag wan_p2p
pass in log on $lan_if inet proto { tcp, udp } from to tag wan_web
pass in log on $lan_if inet proto { tcp, udp } from to tag wan_vip
# uPnPd rule anchor
pass in log on $lan_if from any to tag wan_defq
anchor “miniupnpd”
# Tag traffic from the LAN to the router and the DMZ
pass in log on $lan_if inet from to ($lan_if) tag lan
pass in log on $lan_if inet from to tag dmz_lan[/cc]
Traffic to and from the LAN doesn’t need to have the route-to and reply-to flags because the default gateway of the gateway itself is the correct interface for this traffic. It is however possible to specify this explicitly if you desire. None of the rules here actually queue any of the connections, just tag them (the only inbound connection to the LAN are already queued above in the redirect pass rules).
[cc lang=”pf” line_numbers=”off” nowrap=”off” ]
######################
# Policy Enforcement #
######################
# Make sure traffic can pass to the LAN and DMZ
pass out log on $lan_if tagged lan
pass out log on $lan_if from any to tag lan
pass out log on $dmz_if tagged dmz
pass out log on $dmz_if tagged dmz_lan
pass out log on $dmz_if tagged dmz_gw
pass out log on $dmz_if reply-to ($tek_if $tek_gw) tagged dmz
# Pass traffic out the WAN interfaces
pass out log on $wan_if inet proto icmp tagged wan_dns queue wan_dnsq
pass out log on $wan_if inet proto tcp tagged wan_dns queue (wan_dnsq, wan_ackq)
pass out log on $wan_if inet proto udp tagged wan_dns queue wan_dnsq
pass out log on $wan_if inet proto tcp tagged wan_vip queue (wan_vip, wan_ackq)
pass out log on $wan_if inet proto udp tagged wan_vip queue wan_vipq
pass out log on $wan_if inet proto tcp tagged wan_ssh queue (wan_sshq, wan_ackq)
pass out log on $wan_if inet proto tcp tagged wan_web queue (wan_webq, wan_ackq)
pass out log on $wan_if inet proto udp tagged wan_web queue wan_webq
pass out log on $wan_if inet proto tcp tagged wan_def queue (wan_defq, wan_ackq)
pass out log on $wan_if inet proto udp tagged wan_def queue wan_defq
pass out log on $wan_if inet proto tcp tagged wan_p2p queue (wan_p2pq, wan_ackq)
pass out log on $wan_if inet proto udp tagged wan_p2p queue wan_p2pq
# Pass traffic out the dmz interfaces
pass out log on $tek_if inet proto icmp tagged tek_dns queue tek_dnsq
pass out log on $tek_if inet proto tcp tagged tek_dns queue (tek_dnsq, tek_ackq)
pass out log on $tek_if inet proto udp tagged tek_dns queue tek_dnsq
pass out log on $tek_if inet proto tcp tagged tek_ssh queue (tek_sshq, tek_ackq)
pass out log on $tek_if inet proto tcp tagged tek_def queue (tek_defq, tek_ackq)
pass out log on $tek_if inet proto udp tagged tek_def queue tek_defq
pass out log on $tek_if inet proto tcp tagged tek_ftp queue (tek_ftpq, tek_ackq)
pass out log on $tek_if inet proto udp tagged tek_ftp queue tek_ftpq
pass out log on $tek_if inet proto tcp tagged tek_p2p queue (tek_p2pq, tek_ackq)
pass out log on $tek_if inet proto udp tagged tek_p2p queue tek_p2pq[/cc]
This is the policy enforcement section and is what makes this a ‘policy based’ firewall. Until this section we’ve allowed packets into the firewall, but haven’t let them exit. In this section we inspect the tags on the packets and then pass and queue them as appropriate. Each connection must have a matching ‘pass in’ and ‘pass out’ pair of rules in order to function. For traffic between the routed subnet and the LAN and the gateway itself I found it was necessary to have tags specifically for this, otherwise the return traffic on a connection from the LAN to the routed subnet would be routed out the public TekSavvy interface, and just generally would not work.
I’ve tried using WRT54G’s with DD-WRT and pfSense to handle traffic shaping on just a single link and I’ve never been satisfied with the results. This pf.conf configuration works like a charm and achieves all of my aims. I’ve never had audio quality issues with my VoIP calls, even when punishing my internet connection with tons of traffic and connections.
Sorry, the comment form is closed at this time.