Configure SureLink active recovery to detect WAN/WWAN failures

Problems can occur beyond the immediate WAN/WWAN connection that prevent some IP traffic from reaching its destination. Normally this kind of problem does not cause the EX12 device to detect that the WAN has failed, because the connection continues to work while the core problem exists somewhere else in the network.

Using Digi SureLink, you can configure the EX12 device to regularly probe connections through the WAN to determine if the WAN has failed, and to perform recovery actions, such as changing the interface metric to use a new default gateway.

Required configuration items

Additional configuration items

Order of precedence for SureLink actions

SureLink recovery actions are preformed in the order that they are configured. As a result, if you include the Reboot Device with other SureLink recovery actions, it should be the last action in the recovery action list. Otherwise, the device will reboot and all recovery actions listed after the Reboot Device action will be ignored.


  1. Log into Digi Remote Manager, or log into the local Web UI as a user with full Admin access rights.
  2. Access the device configuration:

    1. Locate your device as described in Use Digi Remote Manager to view and manage your device.
    2. Click the Device ID.
    3. Click Settings.
    4. Click to expand Config.

    1. On the menu, click System. Under Configuration, click Device Configuration.

      The Configuration window is displayed.

  3. Click Network > Interfaces.
  4. Create a new WAN or WWAN or select an existing one:
  5. After creating or selecting the WAN or WWAN, click SureLink.

    By default, SureLink is enabled for the preconfigured WAN (ETH2) and WWAN (Modem). The default configuration tests the DNS servers configured for the interface.

    When SureLink is configured for Wireless WANs, SureLink tests are only run if the cellular modem is connected and has an IP address. Use the SIM failover options to configure the EX12 device to automatically recover the modem in the event that it cannot obtain an IP address. See Configure a Wireless Wide Area Network (WWAN) for details about SIM failover.

  6. (Optional) Change the Test interval between connectivity tests.

    Allowed values are any number of weeks, days, hours, minutes, or seconds, and take the format number{w|d|h|m|s}.

    For example, to set Interval to ten minutes, enter 10m or 600s.

    The default is 15 minutes.

  7. (Optional) If more than one test target is configured, for Success condition, select either:
    • One test passes: Only one test needs to pass for Surelink to consider an interface to be up.
    • All test pass: All tests need to pass for SureLink to consider the interface to be up.
  8. (Optional) For Pass threshold, type or select the number of times that the test must pass after failure, before the interface is determined to be working and is reinstated.
  9. (Optional) For Response timeout, type the amount of time that the device should wait for a response to a test failure before considering it to have failed.

    Allowed values are any number of weeks, days, hours, minutes, or seconds, and take the format number{w|d|h|m|s}.

    For example, to set Response timeout to ten minutes, enter 10m or 600s.

    The default is 15 seconds.

  10. Click to expand Tests.

    By default, Test DNS servers configured for this interface is automatically configured and enabled. This test communication with DNS servers that are either provided by DHCP, or statically configured for this interface.

    1. Click .

      New tests are enabled by default. To disable, click to toggle off Enable.

    2. Type a Label for the test.

    3. Click to toggle on IPv6 if the test should apply to both IPv6 rather than IPv4.

    4. Select the Test type.

      • Ping test: Uses ICMP to determine connectivity.

        • Ping target: The type of target for the ping, one of:
          • Hostname or IP address of an external server.
            • Ping host: hostname or IP address of the server.
          • The Interface gateway. If Interface gateway is selected, an initial traceroute is sent to the hostname or IP address configured in the SureLink advanced settings, and then the first hop in that route is used for the ping test.
          • The Interface address.
          • The Interface DNS server.
        • Ping payload size: The number of bytes to send as part of the ping payload.

      • DNS test: Performs a DNS query to the named DNS server.

        • DNS server: The IP address of the DNS server.

      • HTTP test: Uses HTTP(s) GET requests to determine connectivity to the configured web server.

        • Web server: The URL of the web server.

      • Test DNS servers configured for this interface: Tests communication with DNS servers that are either provided by DHCP, or statically configured for this interface.

      • Test the interface status: Tests the current status of the interface. The test fails if the interface is down. Failing this test infers that all other tests fail.

        • Down time: The amount of time that the interface is down before the test can be considered to have failed.

          Allowed values are any number of weeks, days, hours, minutes, or seconds, and take the format number{w|d|h|m|s}.

          For example, to set Down time to ten minutes, enter 10m or 600s.

        • Initial connection time: The amount of time to wait for the interface to connect for the first time before the test is considered to have failed.

          • Allowed values are any number of weeks, days, hours, minutes, or seconds, and take the format number{w|d|h|m|s}.

          For example, to set Initial connection time to ten minutes, enter 10m or 600s.

      • Custom test: Tests the interface with custom commands.

        • The Commands to run to test.

      • TCP connection test: Tests that the interface can reach a destination port on the configured host.

        • TCP connect host: The hostname or IP address of the host to create a TCP connection to.
        • TCP connect port: The TCP port to create a TCP connection to.

      • Test another interface's status: Tests the status of another interface.

        • Test interface: The interface to test.
        • IP version: The type of IP connection, one of:
          • Any: Either the IPv4 or IPv6 connection must be up.
          • Both: Both the IPv4 or IPv6 connection must be up.
          • IPv4: The IPv4 connection must be up.
          • IPv6: The IPv6 connection must be up.
        • Expected status: The status required for the test to past.
          • Up: The test will pass only if the referenced interface is up and passing its own SureLink tests (if applicable).
          • Down: The test will pass only if the referenced interface is down or failing its own SureLink tests (if applicable).

    5. Repeat for each additional test.
  11. Add recovery actions:
    1. Click to expand Recovery actions.

      By default, there are two preconfigured recovery actions:

      • Update routing: Uses the Change default gateway action, which increases the interface's metric by 100 to change the default gateway.

      • Restart interface.

    2. Click .

      New recovery actions are enabled by default. To disable, click to toggle off Enable.

    3. Type a Label for the recovery action.

    4. For Recovery type, select Reboot device.

    5. For Recovery type, select the type of recovery action. If multiple recovery actions are configured, they are performed in the order that they are listed.

      • Change default gateway: Increases the interface's metric to change the default gateway.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • Increase metric to change active default gateway: Increase the interface's metric by this amount. This should be set to a number large enough to change the routing table to use another default gateway. The default is 100.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

      • Restart interface.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

      • Reset modem: This recovery action is available for WWAN interfaces only.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

      • Switch to alternate SIM: Switches to an alternate SIM. This recovery action is available for WWAN interfaces only.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

      • Reboot device.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

      • Execute custom Recovery commands.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • The Commands to run to recovery connectivity.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

      • Powercycle the modem. This recovery action is available for WWAN interfaces only.

        • SureLink test failures: The number of failures for this recovery action to perform, before moving to the next recovery action.
        • Override wait interval before performing the next recovery action: The time to wait before the next test is run. If set to the default value of 0s, the Test interval is used.

    6. Repeat for each additional recovery action.

    1. Click to expand Advanced settings.

    2. For Delayed Start, type the amount of time to wait while the device is starting before SureLink testing begins. This setting is bypassed when the interface is determined to be up.

      Allowed values are any number of weeks, days, hours, minutes, or seconds, and take the format number{w|d|h|m|s}.

      For example, to set Delayed start to ten minutes, enter 10m or 600s.

      The default is 300 seconds.

    3. For Backoff interval, type the time to add to the test interval when restarting the list of actions. This option is capped at 15 minutes.

      Allowed values are any number of weeks, days, hours, minutes, or seconds, and take the format number{w|d|h|m|s}.

      For example, to set Backoff interval to ten minutes, enter 10m or 600s.

      The default is 300 seconds.

    4. Test interface gateway by pinging is used by the Interface gateway Ping test as the endpoint for traceroute to use to determine the interface gateway. The default is 8.8.8.8, and should only be changed if this IP address is not accessible due to networking issues.
  13. Click Apply to save the configuration and apply the change.
  1. Select the device in Remote Manager and click Actions > Open Console, or log into the EX12 local command line as a user with full Admin access rights.

    Depending on your device configuration, you may be presented with an Access selection menu. Type admin to access the Admin CLI.

  2. At the command line, type config to enter configuration mode:

    > config
(config)>

  3. Create a new WAN or WWAN, or edit an existing one:
    (config)> network interface my_wan
(config network interface my_wan)>
  4. Enable SureLink.

    By default, SureLink is enabled for the preconfigured WAN (eth2) and WWAN (modemwwan2). The default configuration tests the DNS servers configured for the interface.

    When SureLink is configured for Wireless WANs, SureLink tests are only run if the cellular modem is connected and has an IP address. Use the SIM failover options to configure the EX12 device to automatically recover the modem in the event that it cannot obtain an IP address. See Configure a Wireless Wide Area Network (WWAN) for details about SIM failover. 

    (config network interface my_wan)> surelink enable true
(config network interface my_wan)>

  5. By default, the Test DNS servers configured for this interface test is automatically configured and enabled. This tests communication with DNS servers that are either provided by DHCP, or statically configured for this interface.

    To add additional tests:

    1. Add a test:

      (config network interface my_wan)> add surelink tests end
(config network interface my_wan surelink tests 1)>

    2. New tests are enabled by default. To disable: 

      (config network interface my_wan surelink tests 1)> enable false
(config network interface my_wan surelink tests 1)>
    3. Create a label for the test:
      (config network interface my_wan surelink tests 1)> label string
(config network interface my_wan surelink tests 1)>
    4. if the test should apply to both IPv6 rather than IPv4, enable IPv6:
      (config network interface my_wan surelink tests 1)> ipv6 true
(config network interface my_wan surelink tests 1)>

    5. Set the test type:

      (config network interface my_wan surelink tests 1)> test value
(config network interface my_wan surelink tests 1)>

      • ping: Uses ICMP to determine connectivity.

        • Set the ping_method: 

          (config network interface my_wan surelink tests 1)> ping_method value
(config network interface my_wan surelink tests 1)>

          where value is one of:

          • hostname: The hostname or IP address of an external server.

            • Set ping_host to the hostname or IP address of the server:

              (config network interface my_wan surelink tests 1)> ping_host hostname/IP_address
(config network interface my_wan surelink tests 1)>

          • interface_gateway. If set, an initial traceroute is sent to the hostname or IP address configured in the SureLink advanced settings, and then the first hop in that route is used for the ping test.

          • interface_address.

          • interface_dns: The interface's DNS server.

        • Set the number of bytes to send as part of the ping payload:

          (config network interface my_wan ipsec tunnel ipsec_example surelink tests 1)> ping_size int
(config network interface my_wan surelink tests 1)>

      • dns: Performs a DNS query to the named DNS server. 

        (config network interface my_wan surelink tests 1)> dns_server IP_address
(config network interface my_wan surelink tests 1)>

      • http: Uses HTTP(s) GET requests to determine connectivity to the configured web server. 

        • (config network interface my_wan surelink tests 1)> http url
(config network interface my_wan surelink tests 1)>

      • dns_configured: Tests communication with DNS servers that are either provided by DHCP, or statically configured for this interface.

      • interface_up: Tests the current status of the interface. The test fails if the interface is down. Failing this test infers that all other tests fail.

        • Set the amount of time that the interface is down before the test can be considered to have failed.

          (config network interface my_wan surelink tests 1)> interface_down_time value
(config network interface my_wan surelink tests 1)>

          where value is any number of weeks, days, hours, minutes, or seconds, and takes the format number{w|d|h|m|s}.

          For example, to set interface_down_time to ten minutes, enter either 10m or 600s:

          (config network interface my_wan surelink tests 1)> interface_down_time 600s
(config)>

        • Set the amount of time to wait for the interface to connect for the first time before the test is considered to have failed.

          (config network interface my_wan surelink tests 1)> interface_timeout value
(config network interface my_wan surelink tests 1)>

          where value is any number of weeks, days, hours, minutes, or seconds, and takes the format number{w|d|h|m|s}.

          For example, to set interface_timeout to ten minutes, enter either 10m or 600s:

          (config network interface my_wan surelink tests 1)> interface_timeout 600s
(config)>

      • custom_test: Tests the interface with custom commands. 

        • (config network interface my_wan  surelink tests 1)> custom_test_commands "string"
(config network interface my_wan surelink tests 1)>

      • tcp_connection: Tests that the interface can reach a destination port on the configured host.

        • Set the hostname or IP address of the host to create a TCP connection to:
          (config network interface my_wan surelink tests 1)> tcp_host hostname/IP_address
(config network interface my_wan surelink tests 1)>
        • Set the TCP port to create a TCP connection to.
          (config network interface my_wan surelink tests 1)> tcp_port port
(config network interface my_wan surelink tests 1)>

      • other: Tests the status of another interface.

        • Set the interface to test.
          1. (config network interface my_wan surelink tests 1)> other_interface ?

Test interface: Test the status of this other interface.
Format:
  /network/interface/setupip
  /network/interface/setuplinklocalip
  /network/interface/eth1
  /network/interface/eth2
  /network/interface/loopback
Current value: 

(config network interface my_wan surelink tests 1)> other_interface
          2. Set the interface. For example:
            (config network interface my_wan surelink tests 1)> other_interface /network/interface/eth2
(config network interface my_wan surelink tests 1)>
        • Set the type of IP connection:
          (config network interface my_wan surelink tests 1)> other_ip_version value
(config network interface my_wan surelink tests 1)>

          where value is one of:

          • any: Either the IPv4 or IPv6 connection must be up.
          • both: Both the IPv4 or IPv6 connection must be up.
          • ipv4 The IPv4 connection must be up.
          • ipv6: The IPv6 connection must be up.
        • The status required for the test to past.
          (config network interface my_wan surelink tests 1)> other_status value
(config network interface my_wan surelink tests 1)>

          where value is one of:

          • up: The test will pass only if the referenced interface is up and passing its own SureLink tests (if applicable).
          • down: The test will pass only if the referenced interface is down or failing its own SureLink tests (if applicable).

    6. Repeat for each additional test.

  6. Add recovery actions:
    1. Type ... to return to the root of the configuration:
      (config network interface my_wan surelink tests 1)> ...
(config)>
    2. Add a recovery action:
      (config)> add network interface my_wan surelink actions end
(config network interface my_wan surelink actions 0)>

    3. New actions are enabled by default. To disable: 

      (config network interface my_wan surelink actions 0)> enable false
(config network interface my_wan surelink actions 0)>
    4. Create a label for the action:
      (config network interface my_wan surelink actions 0)> label string
(config network interface my_wan surelink actions 0)>

    5. Set the type of recovery action. If multiple recovery actions are configured, they are performed in the order that they are listed. The command varies depending on whether the interface is a WAN or WWAN:

      • WAN interfaces:

        (config network interface my_wan surelink actions 0)> action value
(config network interface my_wan surelink actions 0)>

      • WWAN interfaces:

        (config network interface my_wan surelink actions 0)> modem_action value
(config network interface my_wan surelink actions 0)>

      where value is one of:

      • update_routing_table: Increases the interface's metric to change the default gateway.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the amount that the interface's metric should be increased. This should be set to a number large enough to change the routing table to use another default gateway.
          (config network interface my_wan surelink actions 0)> metric_adjustment_modem int
(config network interface my_wan surelink actions 0)>

          The default is 100.

        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

      • restart_interface.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

      • reset_modem: This recovery action is available for WWAN interfaces only.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

      • switch_sim: Switches to an alternate SIM. This recovery action is available for WWAN interfaces only.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

      • modem_power_cycle: This recovery action is available for WWAN interfaces only.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

      • reboot_device.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

      • custom_action: Execute custom recovery commands.

        • Set the number of failures for this recovery action to perform, before moving to the next recovery action:
          (config network interface my_wan surelink actions 0)> test_failures int
(config network interface my_wan surelink actions 0)>

          The default is 3.

        • Set the commands to run to attempt to recovery connectivity.
          (config network interface my_wan surelink actions 0)> custom_action_commands_modem "string"
(config network interface my_wan surelink actions 0)>
        • Set the time to wait before the next test is run. If set to the default value of 0s, the test interval is used.
          (config network interface my_wan surelink actions 0)> override_interval int
(config network interface my_wan surelink actions 0)>

    6. Repeat for each additional recovery action.
    1. Type ... to return to the root of the configuration:
      (config network interface my_wan surelink actions 0)> ...
(config)>
    2. Set the test interval between connectivity tests:
      (config)> network interface my_wan surelink interval value
(config)>

      where value is any number of weeks, days, hours, minutes, or seconds, and takes the format number{w|d|h|m|s}.

      For example, to set interval to ten minutes, enter either 10m or 600s:

      (config)> network interface my_wan surelink interval 600s
(config)>

      The default is 15m.

    3. If more than one test target is configured, set the success condition:
      (config)> network interface my_wan surelink success_condition value
(config)>

      where value is either:

      • one: Only one test needs to pass for Surelink to consider an interface to be up.
      • all: All tests need to pass for SureLink to consider the interface to be up.
    4. Set the number of times that the test must pass after failure, before the interface is determined to be working and is reinstated.
      (config)> network interface my_wan surelink pass_threshold int
(config)>

      The default is 1.

    5. Set the amount of time that the device should wait for a response to a test failure before considering it to have failed:
      (config)> network interface my_wan surelink timeout value
(config)>

      where value is any number of weeks, days, hours, minutes, or seconds, and takes the format number{w|d|h|m|s}.

      For example, to set timeout to ten minutes, enter either 10m or 600s:

      (config)> network interface my_wan surelink timeout 600s
(config)>

      The default is 15s.

    6. Set the amount of time to wait while the device is starting before SureLink testing begins. This setting is bypassed when the interface is determined to be up.
      (config)> network interface my_wan surelink advanced delayed_start value
(config)>

      where value is any number of weeks, days, hours, minutes, or seconds, and takes the format number{w|d|h|m|s}.

      For example, to set delayed_start to ten minutes, enter either 10m or 600s:

      (config)> network interface my_wan surelink advanced delayed_start 600s
(config)>

      The default is 300s.

    7. Set the time to add to the test interval when restarting the list of actions. This option is capped at 15 minutes.
      (config)> network interface my_wan surelink advanced backoff_interval value
(config)>

      where value is any number of weeks, days, hours, minutes, or seconds, and takes the format number{w|d|h|m|s}.

      For example, to set backoff_interval to ten minutes, enter either 10m or 600s:

      (config)> network interface my_wan surelink advanced backoff_interval 600s
(config)>

      The default is 300 seconds.

    8. The interface_gateway parameter is used by the Interface gateway Ping test as the endpoint for traceroute to use to determine the interface gateway. The default is 8.8.8.8, and should only be changed if this IP address is not accessible due to networking issues. To set to an alternate host:
      (config)> network interface my_wan surelink advanced interface_gateway hostname/IP_address
(config)>
  8. Save the configuration and apply the change.
    (config network interface my_wan ipv4 surelink)> save
Configuration saved.
>
  9. Type exit to exit the Admin CLI.

    Depending on your device configuration, you may be presented with an Access selection menu. Type quit to disconnect from the device.