Provisioning 4.1.x Routers with a 4.2.0+ Conductor

By Tyler Carroll posted 08-23-2019 11:19

In this blog post I will describe a small caveat when using a 4.2.0+ 128T Conductor and 4.1.x managed 128T Routers. This caveat only involves adding new 4.1.x managed Routers with a 4.2.0+ Conductor! When upgrading your Conductor to 4.2.0 any existing 4.1.x Routers will continue to function. This caveat only involves adding 4.1.x managed HA Routers because its directly related to the interface and IP address that the Router uses to communicate to its HA peer. Adding new 4.1.x managed standalone Routers are not affected by this caveat.

A quick recap:

This caveat only affects adding new 4.1.x 128T HA managed Routers under a 4.2.0+ 128T Conductor. It does not affect existing Routers or adding new 4.1.x standalone Routers.

In release 4.2.0 came the addition of non forwarding interfaces and changed how 128T interacts with Linux interfaces. Before reading this blog post I strongly suggest getting up to speed with non forwarding interfaces:

When using a 4.2.0 Conductor if the user has configured non forwarding interfaces in the 128T config, then when provisioning new 4.2.0 managed Routers those managed Routers will create the non forwarding interfaces in Linux to enable communication to their peer nodes. Once the interfaces are created the Routers dynamically create SSH tunnels to their peer nodes over the newly created interfaces. However, if the managed Router is on version 4.1.x or any version less than 4.2.0, the managed Router cannot create the Linux interfaces or dynamically create the SSH tunnels.

When provisioning a 4.1.x Router in the past, the method was to manually setup the Linux interfaces beforehand:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8950
        inet  netmask  broadcast
        inet6 fe80::f816:3eff:fedc:677a  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:dc:67:7a  txqueuelen 1000  (Ethernet)
        RX packets 195  bytes 29104 (28.4 KiB)
        RX errors 6  dropped 0  overruns 0  frame 6
        TX packets 210  bytes 32145 (31.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0​​

After installing 128T or using an ISO with 128T preinstalled, the user would run the initializer which would prompt the user for its HA IP address, which already existed in Linux:
IP address used to reach HA peer

The initializer would use the IP address for its HA peer to set a config value (called a grain in salt terms) in the salt-minion config called the node-ip:
[root@t124-dut3 ~]# cat /etc/salt/minion
auth_safemode: True
autoload_dynamic_modules: False
enable_legacy_startup_events: False
include: /usr/lib/128technology/python/salt/file_roots/_beacons/*.conf
log_level_logfile: debug
master_alive_interval: 30
master_tries: -1
ping_interval: 1
random_reauth_delay: 120
recon_default: 5000
recon_max: 30000
recon_randomize: True
tcp_authentication_retries: -1
tcp_keepalive_cnt: 3
tcp_keepalive_idle: 5
tcp_keepalive_intvl: 10
transport: tcp​

The node-ip is what a 4.1.x Conductor would use to know which IP address a 4.1.x Router was going to use to communicate to its HA peer. Nothing created this IP address for the user, the IP address had to exist on the system already for the node to have proper connectivity to its peer.

A 4.2.0 Conductor no longer uses the node-ip grain and replaces its functionality with non forwarding interfaces in config. In order to properly provision a managed 4.1.x Router with a 4.2.0 Conductor, the IP address and interface that would have been used to set the node-ip grain needs to be put in the Conductor's 128T config instead for both nodes. These IP addresses must be configured before adding the new managed Routers so the Conductor can provision them appropriately. Once again, this interface and IP address must exist in Linux already because a 4.1.x Router cannot create it:
admin@T124_DUT1.Conductor (node[name=T124_DUT3])# show
name                      T124_DUT3
role                      combo

device-interface          ha-sync
    name               ha-sync
    type               ethernet
    pci-address        0000:00:04.0
    enabled            true
    forwarding         false

    network-interface  peer-fabric
        name               peer-fabric
        global-id          3
        type               fabric
        default-route      false
        mtu                1500

            prefix-length  24

Now that we have added our non forwarding fabric interfaces to each node of our managed 4.1.x Router, we are ready to add the asset-id to configuration and associate the new Router with our Conductor! As you can see because the non forwarding interfaces were setup before we associated the new Router with the Conductor, the new managed Router was provisioned properly:
[root@t124-dut3 ~]# cat /etc/128technology/global.init
    "init": {
        "routerName": "Router",
        "control": {
            "T124_DUT4": {
                "host": ""
            "T124_DUT3": {
                "host": ""
        "conductor": {
            "conductor-node-one": {
                "host": ""
            "conductor-node-two": {
                "host": ""

All managed 4.1.x routers have transitioned to the running state:

admin@T124_DUT1.Conductor# show assets
Fri 2019-08-23 15:12:48 UTC

=========== =========== =========== ==================== ========= ========
 Router      Node        Asset Id    128T Version         Status    Errors
=========== =========== =========== ==================== ========= ========
 Conductor   T124_DUT1   T124_DUT1   4.2.0-1.debug.el7    running        0
             T124_DUT2   T124_DUT2   4.2.0-1.debug.el7    running        0
 Router      T124_DUT3   T124_DUT3   4.1.5-2.el7.centos   running        0
             T124_DUT4   T124_DUT4   4.1.5-2.el7.centos   running        0

Completed in 1.79 seconds

And we have full connectivity from the managed Router to its peer node:

admin@T124_DUT3.Router# show system connectivity
Fri 2019-08-23 15:13:17 UTC

============ ===================== ===========
 Local Node   Remote Node           State
============ ===================== ===========
 T124_DUT3    T124_DUT1.Conductor   connected
 T124_DUT3    T124_DUT2.Conductor   connected
 T124_DUT3    T124_DUT4.Router      connected

Completed in 0.11 seconds

Quick recap:

  1. This caveat only affects adding new 4.1.x 128T HA managed Routers under a 4.2.0+ 128T Conductor. It does not affect existing Routers or adding new 4.1.x standalone Routers.
  2. Instead of setting the HA peer IP address of the already created Linux interface through the initializer, add the interface and IP address to the 128T config as a non forwarding fabric interface before associating the new Router with the Conductor.

Please comment if you have any questions!



08-23-2019 13:22

@Jessie Bryan yes I would setup one interface in the 128T config as a non forwarding fabric interface. While the Router is on version 4.1.x the Conductor will simply use the IP address specified while provisioning new 4.1.x Routers. When the managed Router is upgraded to 4.2.0 it will actually create the team interface in Linux for you if you specify the non forwarding interface to be a fabric interface. We use fabric in this case to indicate a direct connection to the peer node. Unfortunately in 4.2.0 we only support one physical ethernet interface per team interface in Linux, but a workout for you would be:

1. Add one of the ethernet interfaces into the 128T config as a non forwarding fabric interface.
2. When 128T comes up you will see it take over the interfaces and modify the ifcfg scripts and create a team interface. A backup of the old ifcfg scripts will be created at /var/run/128technology/network-script-backups.tar.gz.
3. Manually add your 2nd ethernet interface to the exiting network team that 128T created.

I hope to support this use case in the near future and do this all automatically!

08-23-2019 13:06

We are in the process of squashing all bugs in the 4.2.0 candidate image.  The release has made its way onto our corporate router infrastructure and is starting to endure longevity and solutions tests.  There is still a ways to go in order to achieve the level of quality required for our customers.  We believe the release will be available before the end of September.  Any early feedback is greatly welcomed!

08-23-2019 11:53

Hey @Tyler Carroll, I was wondering if you could clarify a point.

For 128T < 4.2 version, we have Linux Land using a "Team" HA-Sync utilizing two physical ethernet ports. With that, we use a 128T fabric interface for handling dog-leg and in some cases in-band management.

With this in mind, do we continue to use two physical ethernet interfaces for ha-sync purposes but now in 128T as a fabric?

08-23-2019 11:45

@Jessie Bryan I don't think I'm qualified to answer that question! Maybe @Patrick A Timmons or @Michael Baj will have a better idea when 4.2.0 will GA.​​​​​

08-23-2019 11:33

Excellent write-up @Tyler Carroll !

I will be referring to this blog once we roll out 4.2 (we have a lot of HA).

Speaking of 4.2 ....any idea when it's going GA?​