NSM v1.1.1 is released via a set of example use cases.

Kubernetes Compatibility

NSM v1.1.1 has been successfully integration tested with Kubernetes versions:

• v1.22.1
• v1.21.1
• v1.20.7
• v1.19.11
• v1.18.15
• v1.17.11

Known Issues

GKE Compatibility

GKE has manifested an odd Netlink IP Neighbor bug with IPv6 in its latest node-version. This is expected to worked around automatically in NSM in the future. Meanwhile, there is a workaround:

• NSM Bug
• Workaround: Downgrade to GKE Node version 1.20.10-gke.301

Note on v1.1.1 being first version

Due to mechanical issues with the release machinery’s interaction with GOPROXY, the first release of v1.1 is v1.1.1, not v1.1.0

Changes since v1.0.0

P2MP

One of the most fundamental components of Network Service Mesh in Kubernetes is the Forwarder. It handles the mechanics of cross connecting whatever ‘local mechanism’ (like kernel interface, vfio, or memif) is injected into the Client Pod to whatever local (kernel,vfio, memif) or remote (vxlan, wireguard, etc) mechanism is needed to reach the Network Service Endpoint providing its requested Network Service.

Traditionally, the Forwarder has been Point to Point (P2P). P2P has the benefit of having a single simple contract: a packet(or frame) that goes in one side comes out the other.

As the number and variety of NSM use cases has grown, some use cases have emerged in which the Forwarder needs to do more complex cross connects than just P2P. Point to multipoint (P2MP), by contrast to P2P, have a large number of mutually incompatible implementations. Often P2MP will be spoken of in shorthand as if its is a single unitary thing: it is not.

In order to support these sorts of use cases, NSM made a slight architectual enhancement: have the forwarder select the NSE (or in the case of P2MP NSEs) to which it should be connecting. This change allows a Forwarder to support zero or more possible P2MP senarios.

Given a Network Service that requires some particular P2MP capabilty (or any other kind of capability) from its Forwarder for one of its logical links, it is necessary to have some capacity of ‘hinting’ that requirement to the NSMgr so that it selects the correct Forwarder in the event that more than on is present. NetworkService matches can now be decorated with ‘metadata’ with ‘labels’ for use by the NSMgr in selecting the correct Forwarder for each logical hop the match represents.

Interdomain improvements

• Network Service Mesh Interdomain now supports Spire Federation. Spire Federation allows workloads from different Trust Domains to connect to a Network Service securely and subject to authorization.
• Network Service Mesh Interdomain CI now tests interdomain simultaneously across:
  • GKE
  • EKS
  • AKS
  • Vanilla K8s on Bare Metal (using Equinix Metal) validating that, for example, a Pod in GKE can connect to a Network Service being provided by a Pod in AKS. ( See interdomain integration test results )
• Try NSM Interdomain

VPP Forwarder Improvements

• SRIOV support incorporated into the VPP Forwarder Pods can now request either a ‘kernel’ or ‘vfio’ interface backed by an SRIOV VF that connects it to its requested Network Service be injected into the Pod
  Try NSM SRIOV
• VLAN remote mechanism: the VPP Forwarder now supports use of VLAN as a remote mechanism. This allows the physical network to be partitioned into vWires distinguished by VLAN tags and introduces a method for cluster breakout. Please note this feature is alpha as CI driven examples are not yet available.
• Configurable VXLAN port: In some deployments, the default VXLAN port 4789 may already be used by the underlying infrastructure, thus making the routers/switches drop packets sent by NSM over the same port due to security policies. NSM_VXLAN_PORT env variable is now added to the VPP forwarder, allowing users to set a different port number.

OVS Forwarder

A new forwarder based on the OpenvSwitch (OVS) data plane has been added to NSM as alternative to the VPP forwarder. The OVS forwarder can install its own OVS instance or re-use an existing OVS instance on the host. For standard use cases, the OVS forwarder supports the local kernel mechanism (using veth pairs) and remote VXLAN mechanism with typical kernel datapath performance.

For high-performance use cases the OVS forwarder can off-load the NSM data plane to a Mellanox ConnectX SmartNIC, relying on standard OVS HW-offload capabilities. In this scenario, the OVS forwarder injects ConnectX (smart) Virtual Functions into NSC and NSE pods as local mechanism, where they can be consumed as DPDK or RDMA user-space interfaces to achieve full pass-through performance. The SmartNIC handles bonding and VXLAN encapsulation transparently for the applications.

The OVS forwarder is also planned to support upcoming forwarder use cases, such as remote VLAN mechanism for break-out to external fabric, VLAN-trunking of multiple client connections on a single NSE interface and P2MP forwarding.

Try the OVS Forwarder

Usability

• Examples in deployment-k8s are greatly simplified by using Spire K8s Registrar to create workload identities.
• Kustomize examples in deployment-k8s can now be run with urls rather than inline in a checked out copy of the repo as was required in v1.0.0

Stability

• Healing - simplified to improve stability and reduce reconvergence time
• Refresh and Timeout vastly simplified resulting in improved stability.

Upgraded depencencies

• Upgraded to VPP 21.06
• Upgraded to Spire v1.1