Skip to content

[meshnet] Manage system-level kernel and link tunables for packet performance - #734

Merged
kraney merged 5 commits into
openconfig:mainfrom
kraney:meshnet-tunables
Sep 8, 2026
Merged

[meshnet] Manage system-level kernel and link tunables for packet performance#734
kraney merged 5 commits into
openconfig:mainfrom
kraney:meshnet-tunables

Conversation

@kraney

@kraney kraney commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Note: this PR stacks on top of #733 . Please merge that one first (which should make the shared commits disappear from this PR.) The incremental change of this PR starts with sha:b3c915457042d606dd0dd9bb659ed1ea619af81e.

    Implement kernel sysctl tuning in meshnetd for high-density, high-throughput network meshes.

    Reason for Change:
    - Default Linux link queue lengths, netdev backlogs, multicast membership limits, and
      strict rp_filter settings cause packet drops during netdev creation bursts and multi-link tests.
    - IPv6 DAD (Duplicate Address Detection) and Router Solicitations introduce 1-2s readiness
      delays per netdev when launching virtual topologies.

    Key Changes:
    - System & Kernel Tuning (`utils/wireutil/sys_tune.go`):
      - Sets `RLIMIT_NOFILE` to 1,048,576 open file descriptors.
      - Sets `netdev_max_backlog` to 10,000 for device queue bursts.
      - Sets OS send/recv max & default buffers (`rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) to 16 MB.
      - Sets ARP/neighbor GC thresholds (`gc_thresh1`=1024, `gc_thresh2`=4096, `gc_thresh3`=8192).
      - Sets `igmp_max_memberships` to 10,000 and `mld_max_msf` to 4,096 to prevent multicast group netdev drops.
      - Sets `rp_filter` to 2 (Loose mode) to allow asymmetric routing across multi-link meshes.
      - Disables IPv6 DAD (`accept_dad`=0) and RS floods (`router_solicitations`=0) for instant link readiness.
      - Sets `ipv6/route/max_size` to 1,048,576 for large routing tables.
      - Configured link `txqueuelen` (default 10,000) across TAP, vEth, and vxLAN interface creation.
      - All tunables are overridable via environment variables without recompilation.

@kraney

kraney commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

FWIW, I created a throughput test comparing before & after for grpcwire. ("Before" being before #731 , "after" being after #734 , including the full set of improvements.)

Before:
image

After:
image

Comment thread third_party/meshnet/daemon/main.go Outdated
@kraney
kraney marked this pull request as draft August 17, 2026 16:24
…mance

    Implement kernel sysctl tuning, gRPC stream multiplexing, and persistent TAP device
    packet I/O in meshnetd for high-density, high-throughput network meshes.

    Reason for Change:
    - Default Linux link queue lengths, netdev backlogs, multicast membership limits, and
      strict rp_filter settings cause packet drops during netdev creation bursts and multi-link tests.
    - IPv6 DAD (Duplicate Address Detection) and Router Solicitations introduce 1-2s readiness
      delays per netdev when launching virtual topologies.

    Key Changes:
    - System & Kernel Tuning (`utils/wireutil/sys_tune.go`):
      - Sets `RLIMIT_NOFILE` to 1,048,576 open file descriptors.
      - Sets `netdev_max_backlog` to 10,000 for device queue bursts.
      - Sets OS send/recv max & default buffers (`rmem_max`, `wmem_max`, `rmem_default`, `wmem_default`) to 16 MB.
      - Sets ARP/neighbor GC thresholds (`gc_thresh1`=1024, `gc_thresh2`=4096, `gc_thresh3`=8192).
      - Sets `igmp_max_memberships` to 10,000 and `mld_max_msf` to 4,096 to prevent multicast group netdev drops.
      - Sets `rp_filter` to 2 (Loose mode) to allow asymmetric routing across multi-link meshes.
      - Disables IPv6 DAD (`accept_dad`=0) and RS floods (`router_solicitations`=0) for instant link readiness.
      - Sets `ipv6/route/max_size` to 1,048,576 for large routing tables.
      - Configured link `txqueuelen` (default 10,000) across TAP, vEth, and vxLAN interface creation.
      - All tunables are overridable via environment variables without recompilation.

   Environment Variable          │ Default Value                 │ Configures
  ───────────────────────────────┼───────────────────────────────┼─────────────────────────────────────────────────────
   LINK_TXQUEUELEN               │ 10000                         │ Interface txqueuelen for TAP, vEth, and vxLAN links
   RLIMIT_NOFILE                 │ 1048576                       │ Process open file descriptor limit (unix.Setrlimit)
   NETDEV_MAX_BACKLOG            │ 10000                         │ /proc/sys/net/core/netdev_max_backlog
   RMEM_MAX                      │ 16777216                      │ /proc/sys/net/core/rmem_max (16 MB)
   WMEM_MAX                      │ 16777216                      │ /proc/sys/net/core/wmem_max (16 MB)
   RMEM_DEFAULT                  │ 16777216                      │ /proc/sys/net/core/rmem_default (16 MB)
   WMEM_DEFAULT                  │ 16777216                      │ /proc/sys/net/core/wmem_default (16 MB)
   ARP_GC_THRESH1                │ 1024                          │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh1
   ARP_GC_THRESH2                │ 4096                          │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh2
   ARP_GC_THRESH3                │ 8192                          │ /proc/sys/net/ipv4 & ipv6/neigh/default/gc_thresh3

   Sysctl / Limit                        │ Env Var Override                     │ High-Density Default
  ───────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────
   igmp_max_memberships                  │ IGMP_MAX_MEMBERSHIPS                 │ 10000
   mld_max_msf                           │ MLD_MAX_MSF                          │ 4096
   rp_filter                             │ RP_FILTER                            │ 2 (loose)
   accept_dad                            │ IPV6_ACCEPT_DAD                      │ 0 (disabled)
   router_solicitations                  │ IPV6_ROUTER_SOLICITATIONS            │ 0 (disabled)
   max_size (IPv6 route)                 │ IPV6_ROUTE_MAX_SIZE                  │ 1048576
   netdev_max_backlog                    │ NETDEV_MAX_BACKLOG                   │ 10000
   rmem_max / wmem_max                   │ RMEM_MAX / WMEM_MAX                  │ 16777216 (16 MB)
   gc_thresh1..3                         │ ARP_GC_THRESH1..3                    │ 1024 / 4096 / 8192
   RLIMIT_NOFILE                         │ RLIMIT_NOFILE                        │ 1048576

Defaults are chosen with the intent of making custom tuning unnecessary for all but the
most demanding topologies.
kraney added 2 commits August 31, 2026 21:39
This limits to much more defensible settings. It also moves
responsibility for broad host settings to the packer builds.

Document updated to explain the ones that remain.
@kraney
kraney marked this pull request as ready for review August 31, 2026 22:12
@kraney
kraney merged commit 5c858dc into openconfig:main Sep 8, 2026
15 of 16 checks passed
@kraney
kraney deleted the meshnet-tunables branch September 8, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants