Skip to content

sonic-vs: mark the wire interfaces ARP-off and IPv6-off so only the port answers - #3390

Open
roc-ops wants to merge 1 commit into
srl-labs:mainfrom
roc-ops:fix/sonic-vs-wire-interfaces
Open

sonic-vs: mark the wire interfaces ARP-off and IPv6-off so only the port answers#3390
roc-ops wants to merge 1 commit into
srl-labs:mainfrom
roc-ops:fix/sonic-vs-wire-interfaces

Conversation

@roc-ops

@roc-ops roc-ops commented Sep 7, 2026

Copy link
Copy Markdown

A sonic-vs node ends up with two kernel interfaces per link: the veth containerlab creates
(ethN) and the tap syncd derives from it - eth1 becomes Ethernet0, eth2 becomes
Ethernet4, and so on. SONiC configures the port; the ethN interface is only the wire that
carries the port's frames.

Left alone, the kernel treats that wire as an ordinary interface. It answers ARP for the port's
address with the wire's MAC, and it brings up an IPv6 link-local address there. A neighbour can
therefore cache the wrong MAC for the port, and can discover a device on the link that isn't the
port at all. Both interfaces see the same ARP request, so which reply the neighbour keeps is a
race - the entry is unreliable rather than reliably wrong, which makes it easy to miss.

SONiC's own virtual-switch test harness turns ARP off on these interfaces for the same reason:
sonic-swss/tests/conftest.py#L245-L246,
class VirtualServer - "disable arp, so no neigh on physical interfaces".

This change does the same for every data endpoint of the node (eth0, the management interface,
is skipped), running inside the container:

ip link set arp off dev <ifname>
sysctl -w net.ipv6.conf.<ifname>.disable_ipv6=1

A failure is logged as a warning and the deploy continues - this is a hardening fixup, not
something a lab should die on. PostDeploy applies it before the SONiC agents start, and
PostDeployEndpoints applies it again for links added to an already running node -
clab tools veth create -t <topo>, and clab apply where the node is not recreated
(link-apply-mode: live or restart), which is the path deployLinks uses. Start re-applies it
after clab stop/clab start, because moving a netdev between network namespaces resets its IPv6
sysctls. The commands are idempotent, so running them more than once is harmless.

Scope: this affects only the sonic-vs kind. dell_sonic, plvision_sonic and sonic-vm are
VM-based kinds and do not have this container/tap duality. The one way an existing user could
notice a difference: a lab that addresses the ethN interface directly instead of the
Ethernet port would lose IPv4 and IPv6 on that interface. The documented workflow for this kind
configures the port (config interface ip add Ethernet0 ...), which is unaffected.

The kind documentation gains a short note explaining the ethN/Ethernet relationship, and
there is a unit test for the command list.

Verified on a two-node sonic-vs lab (s1:eth1 <-> s2:eth1, Ethernet0 addressed
10.1.1.1/30 + 2001:db8::1/64 and 10.1.1.2/30 + 2001:db8::2/64):

  • IPv4: eth1 shows NOARP; ping -c3 10.1.1.2 is 0% loss and ip neigh show 10.1.1.2
    resolves to s2's Ethernet0 MAC, not s2's eth1 MAC.
  • IPv6: net.ipv6.conf.eth1.disable_ipv6 = 1 and ip -6 addr show eth1 is empty, while
    Ethernet0 keeps its own link-local; ping6 -c3 2001:db8::2 is 0% loss and
    ip -6 neigh show 2001:db8::2 resolves to s2's Ethernet0 MAC.
  • Later-added link: clab tools veth create -t <topo> -a s1:eth2 -b s2:eth2 on the running
    lab leaves eth2 with NOARP and disable_ipv6=1 on both nodes.
  • Stop/start: without the Start override, clab stop + clab start leaves eth1 at
    net.ipv6.conf.eth1.disable_ipv6 = 0 with a fresh fe80:: address (NOARP survives); with it,
    eth1 comes back disable_ipv6 = 1, NOARP, and ip -6 addr show eth1 empty.
  • eth0 is untouched throughout: no NOARP, IPv6 still enabled.
  • Control on stock 0.78.2: the same test resolves 10.1.1.2 to s2's eth1 (wire) MAC on 2 of
    3 flush-and-ping rounds and to the port MAC on 1 of 3 - the race, observed.

…ort answers

A sonic-vs node ends up with two kernel interfaces per link: the veth
containerlab creates (ethN) and the tap syncd derives from it (eth1
becomes Ethernet0, eth2 becomes Ethernet4, and so on). SONiC configures
the port; the ethN interface is only the wire that carries the port's
frames.

Left alone, the kernel treats that wire as an ordinary interface. It
answers ARP for the port's address with the wire's MAC, and it brings up
an IPv6 link-local address there. A neighbour can therefore cache the
wrong MAC for the port, and can discover a device on the link that isn't
the port at all.

SONiC's own virtual-switch test harness turns ARP off on these interfaces
for the same reason (sonic-swss, tests/conftest.py, VirtualServer:
"disable arp, so no neigh on physical interfaces"). The node now does the
same for every data endpoint, disabling both ARP and IPv6 on the wire and
leaving eth0 untouched: PostDeploy quiets the wires before the SONiC
agents start, and PostDeployEndpoints quiets them again for links added
to an already running node. Start re-applies the fixups after a stop, because
stopping a node parks its endpoints in a separate network namespace and a
netdev that crosses a namespace has its IPv6 configuration rebuilt - so
disable_ipv6 falls back to 0 and a link-local returns on the wire. The
commands are idempotent, and a failure is logged as a warning rather than
failing the deploy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants