Configuring Non Forwarding Interfaces in 4.2.0+

By Tyler Carroll posted 02-13-2019 13:45

  
In this blog post I will describe how to configure non forwarding interfaces, a new feature in release 4.2.0. Non forwarding interfaces are used for several purposes including HA peer node control traffic, reaching a 128T Conductor from a 128T Router, reaching managed 128T Routers from a 128T Conductor, management traffic and reaching the public internet.

The goal of this feature is to allow the user to configure all non forwarding interfaces through the 128T configuration and eliminate the need for the user to have to drop into the Linux shell to manually tweak ifcfg scripts. This feature also aims to make deployment of 128T Routers easier by eliminating the need to setup interfaces before installing the product. This feature leverages the device-interface and network-interface configuration options and re-purposes some of the existing configuration fields for non forwarding interfaces. This feature is dynamically reconfigurable so the user can create/modify/delete interfaces without having to restart 128T. In release 4.2.0 the only part that is not dynamically reconfigurable is changing a forwarding interface to non forwarding and vice versa.

When upgrading a 128T Conductor or Router to version 4.2.0 you will notice a new validation warning if you are running in an HA configuration:
*admin@T127_DUT1.Conductor# validate
Validating...
Warnings:
1. Router 'Conductor' is missing a non-forwarding fabric or shared interface on each node. The router may not have inter node
connectivity unless the interfaces were setup manually.
reported by router 'Conductor'

    config
        authority
            router Conductor

Candidate configuration is valid​

If you have an existing deployment and just upgraded to version 4.2.0 then your non forwarding interfaces were setup manually at some point. If you are deploying a new HA router for the first time then you will need to configure your non forwarding interfaces in order to establish connectivity to your HA peer node. Even if your non forwarding interfaces were setup manually I recommend using 128T to manage the interfaces from now on. Before we get started I want to highlight some important terminology.

Forwarding: An interface used to pass session traffic.

Non Forwarding: An interface that is not used to pass session traffic and used for management or control traffic.

Non forwarding external interface: An interface that is externally facing such as a management interface or an interface used to reach your 128T Conductor, your managed 128T Routers or the public internet. There is no limit to the number of external interfaces that a user can configure per node. External interfaces can be Ethernet, LTE, PPPoE or T1 devices. They can also be configured with static IP addresses or to use DHCPv4.

Non forwarding fabric interface: An interface that is used to connect directly to the node's HA peer. Since a direct connection from one box to the other is assumed in this case, the interface is configured in Linux as a network team. The network team configuration ensures that if a node's HA peer is rebooted that the node's interface does not get a link down event and the IP address does not disappear from Linux, which causes internal connectivity issues within the node itself. The user may configure one fabric or shared interface per node, and that interface is used for all HA control traffic. The fabric interface must be an Ethernet interface with exactly one address. This address maps to the IP address found in the node's global.init file and 128T will automatically update global.init when this address is added or modified.

Non forwarding shared interface: A combination of an external and fabric interface. A shared interface serves all the same purposes as an external interface while also providing connectivity to the HA peer. A shared interface is not directly connected to the HA peer so it is not configured as a Linux network team. The user may configure one fabric or shared interface per node, and that interface is used for all HA control traffic. The shared interface must be an Ethernet interface with exactly one address. This address maps to the IP address found in the node's global.init file and 128T will automatically update global.init when this address is added or modified.

A quick recap:

External interfaces are not used for any control traffic to the node's HA peer and the user has no limit to how many external interfaces they configure. A user may configure either one fabric or one shared interface per node, and that interface will be used for all HA control traffic.

Lets get into the actual configuration of these interfaces. To configure a non-forwarding interface, set the forwarding flag to false. In this example I am configuring an ethernet device-interface:
node    T116_DUT1
    name                      T116_DUT1

    device-interface          control
        name               control
        type               ethernet
        pci-address        0000:00:04.0
        forwarding         false​

The next step is to add a network-interface, lets start with a fabric interface to reach our HA peer node and to eliminate that warning stating that we may not have access to our HA peer node unless interfaces were setup manually. I am choosing fabric in this situation because my HA nodes are directly connected:
node    T127_DUT1
    name                      T127_DUT1

    device-interface          control
        name               control
        type               ethernet
        pci-address        0000:00:04.0
        forwarding         false

        network-interface  peer-fabric-intf
            name       peer-fabric-intf
            type       fabric

            address    172.16.1.1
                ip-address     172.16.1.1
                prefix-length  24
                gateway        172.16.1.201
            exit
        exit
    exit
exit​

This interface will allow us to establish connectivity to our HA peer node. However, you will notice that the warning has not gone away yet. We need to configure another fabric interface on our HA peer node so it can establish connectivity to us:
node    T127_DUT2
    name                      T127_DUT2

    device-interface          control
        name               control
        type               ethernet
        pci-address        0000:00:04.0
        forwarding         false

        network-interface  peer-fabric-intf
            name       peer-fabric-intf
            type       fabric

            address    172.16.1.2
                ip-address     172.16.1.2
                prefix-length  24
                gateway        172.16.1.201
            exit
        exit
    exit
exit​


Perfect! Now the validation warning is gone and I will commit my configuration.

If you have worked with a 128T Router then you are probably familiar with the file /etc/128technology/global.init. This file is used by a 128T node on startup to learn its own control IP address as well as its HA peer node's control IP address. Upon committing our new configuration the IP addresses in global.init will be updated automatically. 128T ensures that the IP addresses in global.init match the IP addresses in configuration that are being used for control traffic so please don't modify this file directly!

[root@t127-dut1 ~]# cat /etc/128technology/global.init
{
  "init" : {
    "control" : {
      "T127_DUT1" : {
        "host" : "172.16.1.1"
      },
      "T127_DUT2" : {
        "host" : "172.16.1.2"
      }
    },
    "conductor" : {
    },
    "routerName" : "Router"
  }
}​​


In addition to global.init being updated, the interfaces will be created in Linux and 128T will dynamically update all SSH tunnels which are used to send control traffic to our peer:

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::f816:3eff:fe4c:5ebe  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:4c:5e:be  txqueuelen 1000  (Ethernet)
        RX packets 5581689  bytes 1057148542 (1008.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5818989  bytes 1002306918 (955.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

team-eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.1.1  netmask 255.255.255.0  broadcast 172.16.1.255
        inet6 fe80::f816:3eff:fe4c:5ebe  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:4c:5e:be  txqueuelen 1000  (Ethernet)
        RX packets 5544219  bytes 977056448 (931.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5777931  bytes 979222766 (933.8 MiB)
        TX errors 0  dropped 2 overruns 0  carrier 0  collisions 0


Notice the interfaces are configured as network team because we chose type fabric. If we had chosen type shared then interface eth1 would have been configured directly with the static IP address.

Note: Upon committing configuration changing the control IP addresses, it can take up to two minutes for the node to reconnect internally or reconnect to its HA peer node. This delay happens because 128T is waiting for the TCP state machine to close existing connections so don't panic and just sit tight and wait for everything to reconnect:

admin@T127_DUT1.Conductor# show system connectivity
Wed 2019-02-13 16:09:06 UTC

===================== ===================== ==============
 Local Node            Remote Node           State
===================== ===================== ==============
 T127_DUT1.Conductor   T127_DUT2.Conductor   connected

Completed in 0.11 seconds
admin@T127_DUT1.Conductor# show system connectivity internal
Wed 2019-02-13 16:09:30 UTC

===================== ===================== ================= ================= ===========
 Local Node            Remote Node           Service           Address           Message
===================== ===================== ================= ================= ===========
 T127_DUT1.Conductor   T127_DUT1.Conductor   Zookeeper         127.0.0.1:4370    Connected
 T127_DUT1.Conductor   T127_DUT1.Conductor   ssc               127.0.0.2:12222   Connected
 T127_DUT1.Conductor   T127_DUT1.Conductor   step-repository   127.0.0.2:15555   Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   Internal SSH      127.0.0.1:932     Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   LeaderElect       127.0.0.1:2225    Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   Quorum            127.0.0.1:2224    Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   ZED               127.0.0.1:4392    Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   Zookeeper         127.0.0.1:4371    Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   influx-rpc        127.0.0.3:8088    Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   ssc               127.0.0.3:12222   Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   step-repository   127.0.0.3:15555   Connected
 T127_DUT1.Conductor   T127_DUT2.Conductor   tank              127.0.0.3:11011   Connected


Now that we have established full connectivity to our HA peer node, lets configure some external management interfaces. In this case I will be configuring an external Ethernet interface to use DHCPv4:

device-interface          mgmt
    name               mgmt
    type               ethernet
    pci-address        0000:00:03.0
    forwarding         false

    network-interface  ext-mgmt-intf
        name               ext-mgmt-intf
        type               external
        default-route      true

        management-vector
            name      ext-mgmt-vector
            priority  100
        exit
        dhcp               v4
    exit
exit


Please notice that we configured a few additional fields. We set the default-route to true to instruct Linux to set this external interface as the default route for all traffic. Secondly we configured a management-vector, which is required when setting the default-route to true. The user is allowed to define multiple interfaces as the default-route, so the management-vector is used to define the priority of all interfaces which are set as the default route.

Note: If you configure the interface that you are currently using to connect to 128T as an external interface you will notice your connection hang for a few moments while 128T takes over the interface after the config is committed. Once the commit is complete you will notice the interface's ifcfg script has been updated:

[root@t127-dut1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
DEFROUTE=yes
DEVICE=eth0
METRIC=100
MTU=1500
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
USERCTL=no


The goal of this feature is to eliminate the need for any users to have to drop to the Linux shell to manually configure interfaces. A configuration field name ifcfg-option was added to the network-interface to allow super users to add any config field directly to an ifcfg script that 128T currently does not support. 128T does validate that the user is not trying to configure any options that 128T already configures to avoid the user stomping on 128T's settings. One example is firewalld zones, perhaps this external interface needs to be configured as a trusted interface:

network-interface  ext-mgmt-intf
    name               ext-mgmt-intf
    global-id          5
    type               external
    default-route      true

    management-vector
        name      ext-mgmt-vector
        priority  100
    exit
    dhcp               v4

    ifcfg-option       ZONE
        name   ZONE
        value  trusted
    exit
exit


Upon committing this configuration you will see the ifcfg script get updated with the new value:

[root@t127-dut1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
DEFROUTE=yes
DEVICE=eth0
METRIC=100
MTU=1500
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
ZONE=trusted


This config field will allow the user to set any custom settings that are not support by 128T yet.

That wraps up all the information for non forwarding interfaces. I will write follow up blog posts any time new config fields or capabilities are support in further releases.

If you are seeing connectivity issues either between HA nodes or from a managed 128T Router and a 128T Conductor please check out these blog posts discussing debugging techniques for disconnected nodes:
https://community.128technology.com/blogs/tyler-carroll/2019/02/06/debugging-a-disconnected-peer-node
https://community.128technology.com/blogs/tyler-carroll/2019/02/05/debugging-a-disconnected-node-from-conductor

1 comment
76 views

Permalink

Comments

03-28-2019 12:19

Nicely written Tyler :)