sonic-vs: mark the wire interfaces ARP-off and IPv6-off so only the port answers - #3390
Open
roc-ops wants to merge 1 commit into
Open
sonic-vs: mark the wire interfaces ARP-off and IPv6-off so only the port answers#3390roc-ops wants to merge 1 commit into
roc-ops wants to merge 1 commit into
Conversation
…ort answers A sonic-vs node ends up with two kernel interfaces per link: the veth containerlab creates (ethN) and the tap syncd derives from it (eth1 becomes Ethernet0, eth2 becomes Ethernet4, and so on). SONiC configures the port; the ethN interface is only the wire that carries the port's frames. Left alone, the kernel treats that wire as an ordinary interface. It answers ARP for the port's address with the wire's MAC, and it brings up an IPv6 link-local address there. A neighbour can therefore cache the wrong MAC for the port, and can discover a device on the link that isn't the port at all. SONiC's own virtual-switch test harness turns ARP off on these interfaces for the same reason (sonic-swss, tests/conftest.py, VirtualServer: "disable arp, so no neigh on physical interfaces"). The node now does the same for every data endpoint, disabling both ARP and IPv6 on the wire and leaving eth0 untouched: PostDeploy quiets the wires before the SONiC agents start, and PostDeployEndpoints quiets them again for links added to an already running node. Start re-applies the fixups after a stop, because stopping a node parks its endpoints in a separate network namespace and a netdev that crosses a namespace has its IPv6 configuration rebuilt - so disable_ipv6 falls back to 0 and a link-local returns on the wire. The commands are idempotent, and a failure is logged as a warning rather than failing the deploy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A
sonic-vsnode ends up with two kernel interfaces per link: the veth containerlab creates(
ethN) and the tapsyncdderives from it -eth1becomesEthernet0,eth2becomesEthernet4, and so on. SONiC configures the port; theethNinterface is only the wire thatcarries the port's frames.
Left alone, the kernel treats that wire as an ordinary interface. It answers ARP for the port's
address with the wire's MAC, and it brings up an IPv6 link-local address there. A neighbour can
therefore cache the wrong MAC for the port, and can discover a device on the link that isn't the
port at all. Both interfaces see the same ARP request, so which reply the neighbour keeps is a
race - the entry is unreliable rather than reliably wrong, which makes it easy to miss.
SONiC's own virtual-switch test harness turns ARP off on these interfaces for the same reason:
sonic-swss/tests/conftest.py#L245-L246,class
VirtualServer- "disable arp, so no neigh on physical interfaces".This change does the same for every data endpoint of the node (
eth0, the management interface,is skipped), running inside the container:
A failure is logged as a warning and the deploy continues - this is a hardening fixup, not
something a lab should die on.
PostDeployapplies it before the SONiC agents start, andPostDeployEndpointsapplies it again for links added to an already running node -clab tools veth create -t <topo>, andclab applywhere the node is not recreated(
link-apply-mode: liveorrestart), which is the pathdeployLinksuses.Startre-applies itafter
clab stop/clab start, because moving a netdev between network namespaces resets its IPv6sysctls. The commands are idempotent, so running them more than once is harmless.
Scope: this affects only the
sonic-vskind.dell_sonic,plvision_sonicandsonic-vmareVM-based kinds and do not have this container/tap duality. The one way an existing user could
notice a difference: a lab that addresses the
ethNinterface directly instead of theEthernetport would lose IPv4 and IPv6 on that interface. The documented workflow for this kindconfigures the port (
config interface ip add Ethernet0 ...), which is unaffected.The kind documentation gains a short note explaining the
ethN/Ethernetrelationship, andthere is a unit test for the command list.
Verified on a two-node
sonic-vslab (s1:eth1 <-> s2:eth1,Ethernet0addressed10.1.1.1/30+2001:db8::1/64and10.1.1.2/30+2001:db8::2/64):eth1showsNOARP;ping -c3 10.1.1.2is 0% loss andip neigh show 10.1.1.2resolves to s2's
Ethernet0MAC, not s2'seth1MAC.net.ipv6.conf.eth1.disable_ipv6 = 1andip -6 addr show eth1is empty, whileEthernet0keeps its own link-local;ping6 -c3 2001:db8::2is 0% loss andip -6 neigh show 2001:db8::2resolves to s2'sEthernet0MAC.clab tools veth create -t <topo> -a s1:eth2 -b s2:eth2on the runninglab leaves
eth2withNOARPanddisable_ipv6=1on both nodes.Startoverride,clab stop+clab startleaveseth1atnet.ipv6.conf.eth1.disable_ipv6 = 0with a freshfe80::address (NOARP survives); with it,eth1comes backdisable_ipv6 = 1,NOARP, andip -6 addr show eth1empty.eth0is untouched throughout: noNOARP, IPv6 still enabled.10.1.1.2to s2'seth1(wire) MAC on 2 of3 flush-and-ping rounds and to the port MAC on 1 of 3 - the race, observed.