Table of Contents

How to use multiple WAN connections using the mwan3 package

Latest release

The mwan3 packages current as of 2014-06-29 are:

  • mwan3_1.4-20_all.ipk
  • luci-app-mwan3_1.2-19_all.ipk

See below for the download procedure.

Description

Capabilities

The mwan3 package provides the following capabilities:

  • provides outbound WAN traffic load balancing over multiple WAN interfaces based on a numeric weight assignment
  • monitors each WAN connection using repeated ping tests and can automatically route outbound traffic to another WAN interface if the first WAN interface loses connectivity
  • provides specific outbound traffic rules to customize which outbound connections should use which WAN interface
    • this can be customized based on source IP, destination IP, source port(s), destination port(s), type of IP protocol
  • administration and configuration is through a LuCI configuration module
  • up to 250 physical and/or logical WAN interfaces are supported

Creators

  • Forum member Adze wrote mwan3
  • Forum member Arfett wrote the LuCI web interface

Many thanks!

Why should I use mwan3?

  • If you have multiple internet connections, you want to control which traffic goes through which WANs
  • Mwan3 can handle multiple levels of primary and backup interfaces, load-balanced or not. Different sources can have different primary or backup WANs.
  • Mwan3 uses netfilter mark mask to be compatible with other packages (such as OpenVPN, PPTP VPN, QoS-script, Tunnels, etc) as you can configure traffic to use the default routing table.
  • Mwan3 can also load-balance traffic originating from the router itself

How mwan3 load-balancing works

  • mwan3 uses normal Linux policy routing to balance outgoing traffic over multiple WAN connections
  • Linux outgoing network traffic load-balancing is performed on a per-IP connection basis – it is not channel-bonding, where a single connection (e.g. a single download) will use multiple WAN connections simultaneously
  • As such load-balancing will help speed multiple separate downloads or traffic generated from a group of source PCs all accessing different sites but it will not speed up a single download from one PC (unless the download is spread across multiple IP streams such as by using a download manager)

Mwan3 architecture

  • Mwan3 is triggered by hotplug-events. When an interface comes up, it creates a custom routing table and iptables rules. A new routing table is created for each interface. It then sets up iptables rules and uses iptables MARK to mark certain traffic. Based on these rules, the kernel determines which routing table to use. When an interface goes down, mwan3 deletes all the rules and routes to that interface.
  • Once all the routes and rules are initially set up, mwan3 exits. The kernel takes care of all the routing decisions. If a new interface hotplug event occurs, mwan3 will run again to adjust route and tables as needed.
  • A monitoring script (mwan3track) runs in the background checking if each WAN interface is up using a ping test. If an interface goes down, the script issues a hotplug event to cause mwan3 to adjust routing tables to the interface failure.

Mwan3 routing

The following steps are taken to route a packet with mwan3:

Every incoming packet (this includes router originated traffic) is handled by the iptables mwan3_hook. This hook takes 5 steps:

  1. Restore mark if previous set. If successful marked, goto step 5.
  2. Check if the packet arrives on a wan interface. If originated from a local connected ip network, then mark packet with iface_id 255 (default). If the packet is from another (non-local) network and arrives on wan interface, then mark it with iface_id. If successful marked, goto step 5.
  3. Check if packet destined for an ip connected (local) network. If so then mark packet with iface_id 255 (default) and goto step 5.
  4. Apply user rules and mark with configured iface_id. If no match leave unmarked.
  5. If marked then save mark.

Remember that iptables only marks the packet, it does not make routing decisions. Next in line are the ip rules. In following order they are:

  1. Ip rules 1001 till 1250 are for wan interface 1 till 250 respectively. This rule says: If packet is incoming from wan interface use main routing table, regardless of mark.
  2. Ip rules 2001 till 2250 are for wan interface 1 till 250 respectively. This rule says: If packet is marked with iface_id [1-250], use the corresponding wan interface routing table.
  3. Ip rule 2254 is a blackhole/unreachable rule. This rule says: If packet is marked with iface_id 254 (unreachable), drop packet and return icmp unreachable.

Next come the routing tables. These are really simple. There is just the standard main routing table and one routing table containing one gateway for each wan interface. Hopes this make troubleshooting easier.

Mwan3 command-line options

There are now some cli commands to help you troubleshoot or show status:

root@OpenWrt:~# mwan3
Syntax: /usr/sbin/mwan3 [command]

Available commands:
 start Start the service
 stop Stop the service
 restart Restart the service
 reload Reload configuration files (or restart if that fails)
 enable Enable service autostart
 disable Disable service autostart

 ifup <iface> Start service on interface
 ifdown <iface> Stop service on interface
 interfaces Show interfaces status
 policies Show policies status
 rules Show rules status
 status Show all status

  • Example:

root@OpenWrt:~# mwan3 status
Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)

Prerequisites

OpenWrt version

  • OpenWrt 12.09 or later is needed

Hardware

Any router configured with multiple WAN interfaces running OpenWrt 12.09 or later should work. Just pick a device with good OpenWrt support, preferably one with VLAN support for the additional interface flexibility VLAN support provides.

At least three interfaces need to exist for the minimal configuration: inside LAN, WAN1 and WAN2. The simplest way to do this is use VLANs to put individual switch ports into their own VLANs, thus each becoming separate interfaces.

  • As examples, the following specific devices are working well with mwan3:
    • A TP-LINK TL-WR1043ND hardware version 1.10 router (ar71xx platform) (tl-wr1043nd) using OpenWrt 12.09
    • A NetGear WNDR3800 router (ar71xx platform) (wndr3800) using OpenWrt 12.09

Package dependencies

  • The following packages are required, but they should be installed automatically if missing when mwan3 is installed so there is no need to manually install them beforehand
    • libc, ip, iptables, iptables-mod-conntrack-extra, iptables-mod-ipopt, kmod-ipt-conntrack-extra, kmod-ipt-ipopt

Package conflicts

  • Ensure no other multiple WAN package is installed such as multiwan – having multiwan installed at the same time as mwan3 is known not to work

Test external DNS/mail/etc. servers for access from each WAN interface

  • Users in the forum have reported problems with DNS resolution or being unable to send e-mail after implementing WAN load-balancing or failover using mwan3
  • The usual cause is they are using the DNS servers or a mail (SMTP/POP/IMAP) server provided by the ISP of the wan1 (original WAN) interface and when the router starts sending traffic out the wan2 interface, the ISP blocks access to its servers because the traffic is now coming from an address that is not in their own network. This is a common security configuration by ISPs and has nothing to do with mwan3 specifically.
  • Option 1: Before implementing any multiple WAN configuration, test any ISP-provided services to see if they are reachable from "foreign" IP addresses and ensure that they can still be used from source IPs not on the ISPs network.
  • Option 2: Change settings to switch to using servers that are known to be accessible from anywhere
    • For DNS servers, using Google Public DNS (at IPs 8.8.8.8 and 8.8.4.4) is a good choice
  • Option 3: Create user rules for traffic destined to "private" DNS servers to only exit the correct interface.

Multiple WAN interface and routing table preparation

The first WAN interface is named "wan"

The mwan3 default configuration file assumes two WAN interfaces are named "wan" and "wan2". If this is not the case, edit the file /etc/config/mwan3 to configure the "interface" definitions to have the same WAN names as defined in network configuration.

Create and configure a second WAN interface

The simplest way to create more WAN interfaces is to have a VLAN-capable router. This allows each individual port to become its own separate interface in OpenWrt.

  • Here is the basic procedure to create a new VLAN and assign a single port to it so as to create a second WAN interface
    • go to Network > Switch
      • remove a single physical port from the default VLAN 1; this port will be the new physical WAN2 port
      • assign the port to a new VLAN number such as 3 and set the port to be untagged in this single new VLAN and off in all other VLANs (note this VLAN, as with all VLANs, should also include the built-in CPU port as a tagged member, so there are a total of two ports in the new VLAN)
      • reboot the router for the new VLAN interface to become active (e.g. eth0.3 for what will be the new WAN2 interface)
  • Go to Network > Interfaces and add a new interface name for the new eth0.x adapter
    • name the new VLAN physical interface "wan2"
    • don't create a bridge over the specified interface
    • configure the new wan2 interface IP details
    • assign the new wan2 interface to the wan firewall zone

Create additional WAN interfaces (e.g. wan3, …) as desired if more than two WAN connections will be used.

Prepare and the check the default OS routing table for WAN interfaces and test

  • IMPORTANT: Before doing anything with mwan3, ensure that each WAN interface is working and that the default OS routing table is correctly configured for multiple WAN connections. Test each interface with a manual ping test before installing mwan3. There have been multiple reports of mwan3 problems on the forum when the problem is actually at the OS level and visible before mwan3 is even installed.

Step 1: Configure a different metric for each WAN interface

  • Instead of the default route metric setting of 0, specifically configure each WAN interface to use a different routing metric. This metric will only have an effect on the default routing table, not on the mwan3 routing tables.
  • The default (primary) WAN interface should have the lowest metric (e.g. 10) and each additional WAN interface a higher metric (e.g. 20, 30, etc.)
  • Every WAN interface should have "Use default gateway" enabled if this option is present

Note: PPPoE connections only show the "Use gateway metric" option if "Use default gateway" is enabled

WAN setting

WAN is the default WAN interface in this example, and so will get the lowest metric of 10.

  • Network > Interfaces
    • WAN > Edit
      • Advanced Settings
        • Use default gateway: enabled
        • Use gateway metric: 10
      • Save & Apply
WAN2 setting

WAN2 is the alternate WAN interface in this example, and so will get the a higher metric of 20.

  • Network > Interfaces
    • WAN2 > Edit
      • Advanced Settings
        • Use default gateway: enabled
        • Use gateway metric: 20
      • Save & Apply

Verify the routing table

If it is configured correctly, you should have a default gateway (the lines with a target address of 0.0.0.0/0) with a different metric set for each WAN interface. For example:

  • Active IPv4-Routes
    • Status > Routes

Network Target    IPv4-Gateway Metric
wan     0.0.0.0/0 ...          10
wan2    0.0.0.0/0 ...          20
lan ...
...

  • Ensure that every WAN interface has a gateway IP defined and has metric defined

Troubleshooting

Interfaces are missing a metric value
  • There was a report of some wireless interfaces missing a metric value and a gateway. The mwan3 syslog message error was "user.warn mwan3: Could not find gateway for interface wan1 (wlan0)"

Verify outbound traffic on each WAN interface

Check that each WAN interfaces works by trying to ping www.google.com out from each interface. Ensure all interfaces are correctly sending and receiving traffic before proceeding.

Test the wan (first WAN) connection

  • wan is hardware interface eth0.1 in this example:

root@openwrt:~# ping -c 1 -I eth0.1 www.google.com
PING www.google.com (209.85.148.103): 56 data bytes
64 bytes from 209.85.148.103: seq=0 ttl=54 time=19.637 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.637/19.637/19.637 ms

  • Ensure the single ping is successful on this interface ("1 packets transmitted, 1 packets received, 0% packet loss" should be displayed)

Test the wan2 connection

  • wan2 is hardware interface eth0.2 in this example:

root@openwrt:~# ping -c 1 -I eth0.2 www.google.com
PING www.google.com (209.85.148.99): 56 data bytes
64 bytes from 209.85.148.99: seq=0 ttl=56 time=25.552 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 25.552/25.552/25.552 ms

  • Ensure the single ping is successful on this interface ("1 packets transmitted, 1 packets received, 0% packet loss" should be displayed)

Test the wan3 connection

  • Repeat as above to ensure every WAN connection is working

Ensure the CONNTRACK module is enabled in OpenWrt

mwan3 requires that the CONNTRACK module is enabled and active on its WAN interfaces.

  • If the interfaces are in the "wan" firewall zone, and the "Masquerading" option is enabled for the firewall zone, the CONNTRACK module is enabled by default already (this is the usual case)
  • If masquerading/NAT is not enabled for the WAN interface (for example, if just routing without NAT is being using between the LAN and your different WAN interfaces), you need to add the following rule to the LAN and WAN zone configurations in your /etc/config/firewall:

option 'conntrack' '1'

Download packages

The mwan3 packages aren't in the OpenWrt standard package repository. The two packages need to be separately downloaded and installed.

The recommended download method involves connecting to your router's command line via telnet or SSH and downloading the installation files to your /tmp directory.

  • Here is a sample of the router command line method of downloading the files to the /tmp directory with the wget program.

root@OpenWrt: cd /tmp
root@OpenWrt:/tmp# rm mwan3_latest_all.ipk
root@OpenWrt:/tmp# wget http://213.136.13.52/mwan3_latest_all.ipk
root@OpenWrt:/tmp# rm luci-app-mwan3_latest_all.ipk
root@OpenWrt:/tmp# wget http://213.136.13.52/luci-app-mwan3_latest_all.ipk

The suggested alternative is to download each using a web browser on a PC and then transfer each package using scp.

  • Here is a sample PuTTY pscp command to copy both files from the current directory to the /tmp directory on the OpenWrt router using SCP (SSH secure copy). Enter the root password for the router when prompted to do so.

"C:\Program Files (x86)\PuTTY\pscp.exe" -scp *.ipk root@192.168.1.1:/tmp

Installation

cd /tmp

# update package list to prepare for package dependency downloads
opkg update

# back up the current mwan3 configuration file just in case the automatic backup doesn't work
cp -a /etc/config/mwan3 /etc/config/mwan3-tempbackup

# install mwan3 and all required package dependencies
opkg install mwan3_*.ipk

# install luci-app-mwan3
opkg install luci-app-mwan3_*.ipk

Reboot if needed

  • Check that there is a new tab in LuCI, Network > Load Balancing
  • Reboot the router if this tab is not present

Upgrades

The upgrade path is almost the same as the new install path.

  • Install mwan3 as per above, in the same way as in a new installation
  • The configuration file /etc/config/mwan3 will be the new, default version. The previous mwan3 file will be renamed as "mwan3-opkg.backup" but otherwise left intact in /etc/config as well
  • Manually update the new version of /etc/config/mwan3 to ensure interface names are correct and that previously configured interface, member, policy and rule settings are re-entered. Note that some keywords have been dropped over time (e.g. "option reroute") so follow the lead of how the default mwan3 file is configured.
  • Restart mwan3
  • Check its status in LuCI or from the command line (see below) to confirm all expected interfaces are up and testing OK

MWAN3 configuration

A mwan3 configuration consists of four section elements, namely:

  • interface
  • member
  • policy
  • rule

Interface configuration

For each wan interface configure an interface section and define how each WAN interface is tested for up/down status. Each interface section must have a name that corresponds with the interface name in your network config. The settings are described below:

Name Type Required Default Description
interface name string yes (none) The OpenWrt interface name as shown in OpenWrt's Network > Interfaces list (if using a PPPoE interface, the interface name specified here should be the underlying OpenWrt interface name, not the "pppoe-…" interface name)
enabled boolean yes (none) Should mwan3 run on this interface
track_ip list of ip addresses no (none) The hosts to test if interface is still alive. If this value is missing the interface is always considered up
reliability number no 1 Number of track_ip hosts that must reply for the test to be considered as successful. Ensure there are at least this many track_ip hosts defined or the interface will always be considered down
count number no 1 Number of pings to send to each host with each test
timeout seconds no 4 Number of seconds to wait for an echo-reply after an echo-request
interval seconds no 10 Number of seconds between each test
up number no 5 Number of successful tests to considered link as alive
down number no 5 Number of failed tests to considered link as dead

The primary reason to change the default settings is to shorten the time before an interface is failed-over (by reducing the ping interval and number of pings before the interface is down) or lengthen the time to avoid a false link failure report. Please note that if you change the timeout value on low bandwidth interfaces (e.g. 3g) or busy interfaces, that false time-outs can occur. A timeout value of less then 2 seconds is not recommended.

A typical interface section looks like this:

config interface 'wan'
        option enabled '1'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

  • The default configuration has wan2 disabled – enable the wan2 interface in the mwan3 configuration

Member configuration

Each member represents an interface with a metric and a weight value. Members are referenced in policies to define a pool of interfaces with corresponding metric and load-balancing weight. Members can't be used for rules directly. The default settings are described below:

Name Type Required Default Description
interface string yes (none) Member applies to this interface (use the same interface name as used in the mwan3 interface section, above)
metric number no 1 Members within one policy with a lower metric have precedence over higher metric members
weight number no 1 Members with same metric will distribute load based on this weight value

A typical member section looks like this:

config 'member' 'wan1_m1_w3'
	option 'interface' 'wan1'
	option 'metric' '1'
	option 'weight' '3'

  • A working mwan3 config has at least 2 members configured.

Policy configuration

Policies define how traffic is routed through the different WAN interface(s). Every policy has to have one or more members assigned to it, which defines the policy's traffic behavior. If a policy has a single member, traffic will only go out that member. If a policy has more than one member, it will either load-balance among members or use one member but fail-over to another, depending on how the members are configured.

If there is more than one member assigned to a policy, members within the policy with a lower metric have precedence over higher metric members. Members with the same metric will load-balance. Load-balancing members (with same metric) will distribute load based on assigned weights values.

A typical policy section looks like this:

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m1_w2'

  • If a policy is not referenced by a specific traffic rule, the policy will not do anything, so it is fine to leave unused policies in place in case they are desired in the future.
  • If you have a traffic rule that matches a policy, but all the members (interfaces) for that policy are down, the exit strategy for that policy is "unreachable".
  • A working mwan3 config has at least 1 policy configured.
  • Ensure no policy name is longer than 15 characters

Rule configuration

A rule describes what traffic to match and what policy to assign for that traffic. These are the available options:

Name Type Required Default Description
use_policy string yes (none) Use this policy for traffic that matches or set to default to use the default routing table to lookup
src_ip ip address no any Match traffic from the specified source ip address
src_port port or range no any Match traffic from the specified source port or port range, if relevant proto is specified
proto protocol no all Match traffic using the given protocol. Can be one of tcp, udp, icmp or all or it can be a numeric value, representing one of these protocols or a different one
dest_ip ip address no any Match traffic directed to the specified destination ip address
dest_port port or range no any Match traffic directed at the given destination port or port range, if relevant proto is specified
  • There are a number of sample rules defined to show how they work. Edit the rules as desired and delete all the rest of the default rules.

A typical rule section looks like this:

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option use_policy 'wan_wan2_wan3'

  • Order is important. Rules are evaluated in top-to-bottom order, with the first matching rule applying. The rule name is just descriptive and has no operational impact. If no match is found, routing lookup is done via the default routing table.
  • A working mwan3 config has at least 1 rule configured.

Example configuration

config interface 'wan'
        option enabled '1'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wan2'
        option enabled '1'
        list track_ip '8.8.8.8'
        list track_ip '208.67.220.220'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'wan2_m1_w2'
        option interface 'wan2'
        option metric '1'
        option weight '2'

config member 'wan2_m2_w2'
        option interface 'wan2'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'

config policy 'wan2_only'
        list use_member 'wan2_m1_w2'

config policy 'balanced'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m1_w2'

config policy 'wan_wan2'
        list use_member 'wan_m1_w3'
        list use_member 'wan2_m2_w2'

config policy 'wan2_wan'
        list use_member 'wan_m2_w3'
        list use_member 'wan2_m1_w2'

config rule 'sticky_even'
        option src_ip '0.0.0.0/0.0.0.1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wan_wan2'

config rule 'sticky_odd'
        option src_ip '0.0.0.1/0.0.0.1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wan2_wan'

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

Further configuration tips

OpenWrt hotplug script fix

  • Forum member tcherenato found that adding a 1 second pause to the OpenWrt hotplug launch script helps prevent occasional segmentation faults when mwan3 performs hotplug operations. It is not known currently what the root issue is (or even if it is in mwan3 at all) but the change is recommended.

vi /sbin/hotplug-call

  • add "sleep 1" at the indicated location

...

[ \! -z "$1" -a -d /etc/hotplug.d/$1 ] && {
        for script in $(ls /etc/hotplug.d/$1/* 2>&-); do (
                [ -f $script ] && . $script
                ## customization: add a 1 second delay to prevent segmentation faults
                sleep 1
        ); done
}

Start mwan3

  • Mwan3 automatically will start after each reboot but if a reboot has not occurred yet, the package can be manually started.
  • see the "Administration" section below

Preserve configuration files in an OpenWrt upgrade

  • the configuration file "/etc/config/mwan3" is already preserved by default during an OpenWrt upgrade

Verification of basic operation

Check MWAN3 status in cli

root@OpenWrt:~# mwan3 status
Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)

Policy balanced:
 wan2 (40%)
 wan (60%)

Policy wan1_only:
 wan (100%)

Policy wan2_only:
 wan2 (100%)

Policy wan2_wan:
 wan2 (100%)

Policy wan_wan2:
 wan (100%)

Local connected networks:
destination        policy             hits     
------------------------------------------------
127.0.0.0/8        default            22       
224.0.0.0/3        default            0        
192.168.1.0/24     default            0        
192.168.33.0/24    default            0        
213.154.232.8/29   default            0        

Active rules:
source             destination        proto  src-port      dest-port     policy          hits     
---------------------------------------------------------------------------------------------------
0.0.0.0/0          213.136.223.128/25 tcp    0:65535       80            wan_wan2        0        
1.2.3.4            5.6.7.8            udp    12345:54321   12345:54321   wan2_wan        0        
0.0.0.0/0          0.0.0.0/0          all                                balanced        2862     

Check status in the MWAN3 overview page

  • Network > Load Balancing
    • Overview
      • MWAN3 Multi-WAN Interface Live Status
        • this area should show all WAN interfaces as "ONLINE"
      • MWAN3 Multi-WAN Interface Systemlog
        • this area will show recent mwan3 log messages

Check kernel routing tables

  • "ip route show table x" (where x is interface ID) should show a routing table specifically for that interface – these tables are generated by mwan3

Verification of WAN interface load-balancing

  • Go to Network > Interfaces
    • Send traffic from a test inside PC
      • Note: Load-balancing is connection-based (not channel bonding), so use multiple programs accessing different servers to generate traffic (such as two downloads, each from a separate site)
    • Observe the interface packet counts (counters are updated automatically)
    • Verify that traffic is going out all expected WAN interfaces

Verification of WAN interface failover

Test interface failover

  • Go to Network > Load Balancing > Overview
    • Manually disconnect a WAN connection
    • Wait for interface failure detection to happen – the mwan3 status display should update
  • Go to Network > Interfaces
    • Send traffic from a test inside PC and observe the interface packet counts to ensure traffic is now going out the alternate WAN port (counters are updated automatically)
    • Check that the external IP address has changed to the wan2 interface (such as by going to http://whatismyip.com)

Test interface fail-back

  • Restore the primary WAN connection
  • Wait for detection that the WAN link is back up
  • Repeat the same tests as above to ensure traffic has moved back to the now-working WAN interface

Administration

Stop mwan3

  • This will remove all mwan3 created ip rules, ip route tables and iptables rules and revert back to the default routing configuration, which will have all WAN traffic going out the WAN interface with the lowest metric
  • It will kill all running mwan3track instances
  • This would normally only be done as a troubleshooting measure

LuCI

  • System > Startup
    • mwan3 > click "Stop" in the "Stop" column to stop the service

SSH

mwan3 stop

Start mwan3

  • This will restore normal mwan3 routing tables

LuCI

  • System > Startup
    • mwan3 > click "Start" in the "Start" column to start the service

SSH

mwan3 start

Monitoring mwan3

Manual status check

  • Network > Load Balancing
    • Overview > MWAN3 Multi-WAN Interface Live Status
      • verify all interfaces show "ONLINE" status

Automated status check

It would be good to be able to automatically monitor interface status through mwan3 using a monitoring system (for example, OpenNMS or monit).

  • FIXME
  • How to do this? Ideas:
    • send the local syslog to a remote monitoring station and do syslog scraping?
    • do LuCI mwan3 web status screen scraping? need to log in somehow and the interface ONLINE string is actually generated using JavaScript so simple HTML string matching won't work…

Alerting

It would be good for mwan3 to send some kind of alert (e.g. e-mail, SNMP trap) if it detects a failed interface and performs a failover, or if it performs a fail-back.

  • FIXME

Controlling the mapping between internal IP sources and external IPs and interfaces

When using multiple WAN connections, there will be multiple external IPs which can be used as the external IP for outgoing NATed traffic. In particular, an external interface might have a block of external IPs that should be mapped in a particular way to specified internal servers. For example, the internal mail server should send out traffic on the same external IP identified in its MX record. This is the procedure to do this.

Step 1: Set mwan3 rules to send traffic out the right interface

Add an mwan3 traffic rule that directs the specific desired source IP out the correct WAN interface. Rules are processed in top-down order, so be sure this specific rule is higher in the list (thus higher priority) than more general rules below that implement load-balancing or failover in the default case.

  • Define a mwan3 interface member setting for the desired external interface (called "wan3" in the example below)
  • Create a mwan3 policy that only sends traffic out the external interface that has the desired external IP

config policy 'wan3_only'
        list use_member 'wan3_m1_w10'

  • Create a mwan3 rule to have traffic from the internal IP 172.16.1.20 always go out the interface named wan3 using the policy "wan3_only"

config rule 'Mailserver_uses_wan3_only'
        option src_ip '172.16.1.20'
        option proto 'all'
        option use_policy 'wan3_only'

If the external WAN interface only has a single external IP, this is all that is needed. If the interface has multiple external IPs, both the next two steps are also needed.

Step 2: Assign multiple external IP addresses selected interface (optional)

This step is only needed if the desired external interface needs to have multiple external IP addresses assigned to it.

The specified external interface may have multiple IPs assigned to it. For OpenWrt 12.09, the preferred way to do this is using multiple interface definitions – see reference.

  • Network > Interfaces > Add new interface…
    • Create Interface
      • Name of the new interface: e.g. "wan3_2", "wan3_3", …
      • Protocol of the new interface: Static address
      • Create a bridge over multiple interfaces: do not enable
      • Cover the following interface: select the physical interface that will have this (additional) IP address, e.g. eth0.2
    • Submit
  • Network > Interfaces > Interfaces - (new interface name)
    • Common Configuration > General Setup
      • Protocol: Static address
      • IPv4 address: (enter the desired additional external IP)
      • IPv4 netmask: select or enter the correct netmask
      • IPv4 gateway: (leave blank as the already defined default gateway will be used)
      • IPv4 broadcast: (leave blank to auto-set this)
      • Use custom DNS servers: (leave blank as DNS servers should be set through the WAN interface settings)
    • Common Configuration > Firewall Settings
      • Create / Assign firewall-zone: select the desired firewall zone, usually "wan" for an additional external IP
    • Save & Apply

Step 3: Set OpenWrt NAT rules to send traffic out the right IP on the selected interface (optional)

This step is only needed if the desired external interface has multiple external IP addresses assigned to it.

With multiple external IP addresses, we want to control which address is used when sending out traffic from particular servers. This is configured using a source NAT rule in OpenWrt.

Note that both a mwan3 rule to select the interface and an SNAT rule to select the specific IP on that interface are needed to correctly send traffic out a specific external IP.

  • Network > Firewall > Traffic Rules
    • Source NAT
      • add a source NAT rule and edit details to specify the desired inside source IP and the desired external IP – the following code block is an example of the resulting configuration in /etc/config/firewall

config redirect
        option target 'SNAT'
        option name 'Mail server goes out 170.53.100.25'
        option src 'lan'
        option dest 'wan'
        option src_ip '172.16.1.20'
        option src_dip '170.53.100.25'
        option proto 'all'

mwan3 and other programs

ddns-scripts

Example 1: Register the external IP of the active WAN interface

This is the case where you want external clients using a DDNS name to automatically reconnect to the alternate WAN interface if the primary WAN interface fails.

  • Configure ddns-scripts to use the "web" update mechanism as this will reflect the current active external IP

Example 2: Register the external IP of a specific WAN interface using the "interface" source

This is the case where you want each specific WAN interface to register its own DDNS name and the WAN interface in question has an external IP directly assigned to it.

  • Configure ddns-scripts to use the "interface" source and specify the desired WAN physical interface name (e.g. eth0.50)

Example 3: Register the external IP of a specific WAN interface using the "web" source

This is the case where you want each specific WAN interface to register its own DDNS name but the WAN interface in question is behind a NAT device and so does not directly have the correct external IP.

This is tricky when the WAN interface is not the default WAN interface, as ddns-scripts cannot be configured to use a specific interface to check its IP.

Option 1: use a static route
  • Looking up the dyndns.org checkip.dyndns.org hostname shows there are four valid IPs for this DNS name
  • Choose one of them and create a static route to that specific IP through the desired (non-default) WAN interface
    • Do a traceroute to the IP to verify traffic is going out the desired WAN interface
  • Change the ddns-scripts ip_url to be this specific IP, e.g.
    http://91.198.22.70
  • Ensure any other web update ddns-scripts configurations don't use the hostname checkip.dyndns.org, as this may be forced out the specified WAN interface using the static route without realizing
Idea for option 2: use curl
  • ddns-scripts has the option of using curl instead of wget to check a web site to retrieve an IP address
  • curl has an option to force it to use a specified interface
  • This option would involve installing curl and modifying the ddns-scripts to specify on the curl command line which interface to use
  • UNTESTED

OpenVPN

Example 1: Have OpenVPN Server be accessible through multiple WAN interfaces

If load-balancing between multiple WAN interfaces, it is desirable to have OpenVPN clients be able to connect through all active WAN interfaces.

In a multiple WAN interface failover scenario, OpenVPN will not accept client connections on the secondary WAN interface after a failover, as it started listening on the primary WAN interface when it was started.

The following configuration will allow multiple WAN interface to be used with OpenVPN Server.

Step 1: Listen only on the internal LAN interface
  • Configure OpenVPN Server to listen on the internal LAN interface only, not on any WAN interface. The internal LAN interface will not go down or change, and so it provides a stable listening interface for OpenVPN.

vi /etc/openvpn/my-vpn.conf

...
# Which local IP address should OpenVPN
# listen on? (optional)
;local a.b.c.d
## Customization: have OpenVPN listen on the internal LAN interface IP only to allow client re-connections after a WAN interface failover
local 192.168.1.1

...

Step 2: Set up port-forward(s)
  • Configure a port-forward on the "wan" source zone to OpenVPN Server listening on the internal LAN interface. The port-forward will be active on every WAN interface and work the same way regardless of what WAN interface is active.
  • Create a firewall rule like the following:
    • Network > Firewall > Port Forwards
      • Name: OpenVPN forward to unchanging inside IP
      • Protocol: UDP
      • Source zone: wan
      • Source IP address: any
      • External IP address: any
      • External port: 1194 (the default OpenVPN UDP port)
      • Internal zone: lan
      • Internal IP address: (the internal LAN interface IP address)
      • Internal port: 1194
      • Enable NAT Loopback: enabled (the default)
Step 3: OpenVPN client and DNS configuration
  • If load-balancing between multiple active WAN interfaces, the suggested approach is to register multiple DNS A records for the same DNS name. Clients will use just one of the IPs. As per the OpenVPN man page description of the –remote client parameter, "If host is a DNS name which resolves to multiple IP addresses, one will be randomly chosen, providing a sort of basic load-balancing and failover capability."
  • If failing over from a primary to a secondary WAN interface, one approach is to use ddns-scripts to update the IP of the DNS name used by OpenVPN clients

privoxy transparent HTTP proxy

Transparent HTTP proxying relies on using iptables rules to transparently redirect outgoing traffic to port 80 first through the local proxy at another port number.

For example, here is a OpenWrt redirect rule to redirect outgoing traffic to TCP 80 port and re-send it to the local proxy listening on TCP port 8118. This will go into iptables NAT table rules.

config redirect
    option target 'DNAT'
    option dest 'lan'
    option proto 'tcp'
    option src 'lan'
    option src_dip '!10.0.2.1'
    option src_dport '80'
    option dest_ip '10.0.2.1'
    option dest_port '8118'
    option name 'Transparent Proxy [privoxy]'
    option enabled '1'

The problem is that mwan3 adds rules to the iptables's MANGLE table, and this is handled before the NAT table. So when a client makes a request to fetch a web page, it is first marked by mwan3. Mwan3 decides based on your mwan3 rules which wan interface to exit and marks the session accordingly.

Next, iptable nat rule handling takes place and diverts the web page request to privoxy. The reply from privoxy however is part of the same session and is already marked to leave a wan interface. The reply from privoxy is then send over the internet, which is obviously incorrect.

To fix this add the following rules to your mwan3 config:

config 'rule' 'rule1'
    option 'proto' 'tcp'
    option 'dest_port' '80'
    option 'src_ip' '10.0.2.1'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'wan1_wan2_loadbalanced'

config 'rule' 'rule2'
    option 'proto' 'tcp'
    option 'dest_port' '80'
    option 'src_ip' '10.0.2.0/24'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'default'

config 'rule' 'rule3'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'wan1_wan2_loadbalanced'

The policy "wan1_wan2_loadbalanced" is just an example. Change it to whatever policy you like.

Back to top

doc/howto/mwan3.txt · Last modified: 2014/07/29 01:13 by timmillerdyck