Provisioning 4.2.0+ Routers with a 4.2.0+ Conductor

By Tyler Carroll posted 10-10-2019 14:36

  
In this blog post I will describe how to provision 4.2.0+ managed 128T Routers with a 4.2.0+ 128T Conductor. The main changes being discussed here are the addition of non forwarding interfaces and their integration with the provisioning process. This process is meant to enable communication between HA peer nodes within a router, so this process is not necessary for standalone managed Routers.

In release 4.2.0 came the addition of non forwarding interfaces and changed how 128T interacts with Linux interfaces. Before reading this blog post I strongly suggest getting up to speed with non forwarding interfaces:

https://community.128technology.com/blogs/tyler-carroll/2019/02/13/configuring-non-forwarding-interfaces-in-420

If you are interested in adding new 4.1.X 128T Routers under a 4.2.0 128T Conductor then the process is a bit different and I recommend reading this blog post instead:

https://community.128technology.com/blogs/tyler-carroll/2019/08/23/provisioning-41x-routers-with-a-420-conductor

In previous releases managed 128T Routers would need their Linux interfaces setup ahead of time in order to enable HA sync communication between their peer nodes. In release 4.2.0 both 128T Conductors and 128T Routers can configure non forwarding interfaces, which will result in Linux interfaces being setup automatically for HA sync between peer nodes. When configuring new 128T Routers on the Conductor, simply add a non forwarding interface of type fabric to each peer node, with the PCI address of the Linux interface that you want 128T to use for HA sync communication.

Here is an example of the configuration on one node:
admin@T124_DUT1.Conductor (node[name=T124_DUT3])# show
name                      T124_DUT3
role                      combo

device-interface          ha-sync
    name               ha-sync
    type               ethernet
    pci-address        0000:00:04.0
    enabled            true
    forwarding         false

    network-interface  peer-fabric
        name               peer-fabric
        global-id          3
        type               fabric
        default-route      false
        mtu                1500

        address            172.16.1.3
            ip-address     172.16.1.3
            prefix-length  24
        exit
    exit
exit​​


And here is the matching peer node:

admin@T124_DUT1.Conductor (node[name=T124_DUT4])# show
name                      T124_DUT4
role                      combo

device-interface          ha-sync
    name               ha-sync
    type               ethernet
    pci-address        0000:00:04.0
    enabled            true
    forwarding         false

    network-interface  peer-fabric
        name               peer-fabric
        global-id          4
        type               fabric
        default-route      false
        mtu                1500

        address            172.16.1.4
            ip-address     172.16.1.4
            prefix-length  24
        exit
    exit
exit​​

Now when these assets are connected to the Conductor and receive configuration, the Linux interfaces are created automatically:
dpdk1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::f816:3eff:fe9d:302  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:9d:03:02  txqueuelen 1000  (Ethernet)
        RX packets 7089415  bytes 1483100080 (1.3 GiB)
        RX errors 1898  dropped 0  overruns 0  frame 1898
        TX packets 6992490  bytes 1626590330 (1.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

team-dpdk1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.1.3  netmask 255.255.255.0  broadcast 172.16.1.255
        inet6 fe80::f816:3eff:fe9d:302  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:9d:03:02  txqueuelen 1000  (Ethernet)
        RX packets 6399914  bytes 1271569210 (1.1 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6276694  bytes 1433130548 (1.3 GiB)
        TX errors 0  dropped 3 overruns 0  carrier 0  collisions 0​

And the global.init is setup automatically to use those Linux interfaces:
{
  "init" : {
    "control" : {
      "T124_DUT4" : {
        "host" : "172.16.1.4"
      },
      "T124_DUT3" : {
        "host" : "172.16.1.3"
      }
    },
    "conductor" : {
      "conductor-node-one" : {
        "host" : "192.168.1.6"
      },
      "conductor-node-two" : {
        "host" : "192.168.1.13"
      }
    },
    "routerName" : "Router"
  }​

And we have full connectivity from the managed Router to its peer node:
admin@T124_DUT3.Router# show system connectivity
Fri 2019-08-23 15:13:17 UTC

============ ===================== ===========
 Local Node   Remote Node           State
============ ===================== ===========
 T124_DUT3    T124_DUT1.Conductor   connected
 T124_DUT3    T124_DUT2.Conductor   connected
 T124_DUT3    T124_DUT4.Router      connected

Completed in 0.11 seconds​


That's it!


If the non forwarding interfaces are not configured before adding the managed HA Router then the peer nodes will not have connectivity between them. The non forwarding interfaces can be configured after the managed Routers are connected to the Conductor, but it is recommended that they are configured before. The non forwarding interfaces are dynamically configurable so no 128T restart will be required if they are configured after the Routers connect to the Conductor. However, when sending a configuration update to a managed HA Router the Conductor only needs to send the configuration to one of the peer nodes. Normally the peers synchronize their configuration, but in this state the peers cannot communicate to each other. If you notice that only 1/2 peer nodes received the configuration update and created their Linux interfaces then simply restart 128T on the other node to force a configuration sync to the Conductor. Once the node comes up and receives config from the Conductor then it will create its Linux interfaces and establish connectivity to its peer node.

Quick Recap:

  1. When using a 128T 4.2.0+ Conductor and adding new 128T HA 4.2.0+ managed Routers, make sure you configure a non forwarding fabric interface on each node to enable peer connectivity.
  2. If you forgot to configure the interfaces ahead of time and you don't have peer connectivity, you can add them later but you may have to restart one of the peer nodes to force it to get the new configuration if it doesn't create its Linux interfaces.

Please comment if you have any questions!
0 comments
56 views

Permalink