i recently got to write the design and configuration for a client's new rack (AS64500). most network work is migrating and iterating on someone's old setup. this one was a blank page, nothing to inherit, so i did the opposite and configured the interior fabric with BGP unnumbered. no addresses on the internal links. peers find each other over IPv6 link-local, IPv4 routes ride an IPv6 next-hop, and the numbered links are the external ones, where the other side isn't ours to configure.

this is the long version of how it's built, node by node, plus the two things that didn't work the way i expected.

none of this went onto the hardware blind. i built the whole rack in containerlab first (full anonymized lab on github), cEOS standing in for the arista switches and FRR for the vyos edges. in that fabric lab FRR also stands in for the cilium leaves, so i can iterate on switch and edge configs without booting kubernetes for every change. a second lab runs the real thing, Cilium on a kind cluster with its own BGP control-plane, where the server side gets validated against the same switches. that's the pre-production stage: every change goes through a lab before it touches a real port. the diagram above is generated from the fabric topology, and it's where the two bugs further down turned up.

the rule

everything follows from one decision. the interior is IPv6-only, and IPv4 only appears on numbered external links.

inside the AS, switch to switch, switch to edge, edge to edge, there are no IPv4 addresses and no /31s. the loopbacks are IPv6 (fd00::1, fd00::2, fd00::11, fd00::12). the router-ids are still dotted-quad (10.0.0.1, 10.0.0.11, and so on) but that's a 32-bit identifier, not an address you can ping. IPv4 only shows up where it has to, on the transit and customer links, where the peer can't hand us an IPv6-only session.

the topology is two arista 7060cx switches (sw1 AS65001, sw2 AS65002), two vyos edges sharing one external face (vyos1 and vyos2, both AS64500), and two cilium speakers (cilium1 AS65301, cilium2 AS65302). transit comes in from two upstreams, and a customer hangs off the switches. it's eBGP everywhere except the single iBGP session between the two edges: per-switch and per-server private ASNs, no route reflector.

the switches run arista EOS. the edges run vyos, which is FRR under the hood. the servers run Cilium, and in the fabric lab FRR stands in for them. so the configs below are EOS on the switches, FRR on the edges, and Cilium on the servers.

in the diagram, every solid teal link is unnumbered and every dashed grey link is numbered. the dashed ones are the links that leave the building.

first, a little ipv6

most people never learn IPv6 properly. they assume it's just IPv4 with hexadecimals. but the protocol has real depth, and this whole trick leans on one piece of it that IPv4 never had: a link addresses itself, with no help from you. it's worth two minutes at the packet level. if you want more than two minutes, APNIC's week-long onsite IPv6 fundamentals course is where i actually learned this, and it was worth the week.

the moment an interface comes up it gives itself a link-local address in fe80::/10, with a stable, opaque interface id (RFC 7217). no DHCP, no assignment, and the address never leaves the link it's on.

before it uses that address it checks nobody else has it. that's duplicate address detection (DAD, RFC 4862): it sends one neighbor solicitation, an NS (ICMPv6 type 135), with the source set to ::, the unspecified address, because it has no valid address yet, and the destination set to the solicited-node multicast of the tentative address, ff02::1:ffXX:XXXX, formed from the low 24 bits of the target. if some other node already holds that address it answers with a neighbor advertisement (NA, type 136) and the tentative address is abandoned. if nothing answers, it flips from tentative to usable. the interface id is 64 bits of effectively random, so a collision is vanishingly unlikely and DAD passes on the first NS.

then the routers make themselves known. a host can ask with a router solicitation (RS, type 133) sent to ff02::2, the all-routers group, and routers answer, and also advertise on their own schedule, with router advertisements (RA, type 134) to ff02::1, all-nodes. the RA is sourced from the router's own link-local and carries its presence, the on-link prefixes, the MTU. resolving an address to a MAC uses the same NS/NA pair aimed at the solicited-node multicast, the IPv6 replacement for ARP: a targeted multicast that only the likely owner hears, not a broadcast to the whole segment.

tldr: on any IPv6 link, both ends come up with a stable, unique, self-assigned address and a built-in way to find and reach each other, before a single line of config. that is the property the rest of this rides on.

what unnumbered means here

we lean on exactly that. an interior link gets no /31. FRR peers over the fe80:: IPv6 already put on the interface, and the arista side just needs ipv6 enable. the only thing we switch on is RAs, so each end learns the other's link-local and knows who its neighbor is:

interface eth1
 no ipv6 nd suppress-ra
 ipv6 nd ra-interval 5

without that, neighbor eth1 interface has nothing to discover and the session stays down.

the last piece is carrying IPv4 over that IPv6 next-hop. that's RFC 5549 (updated by 8950): an IPv4 prefix advertised with an IPv6 next-hop, so 192.0.2.0/24 crosses a link with no IPv4 address on it. arista has to be told to do this; FRR does it on its own. both below.

the switch side

arista EOS doesn't implement remote-as external. you give it the AS per neighbor and peer the interface into a peer-group:

neighbor PG-INTERIOR peer group
neighbor PG-INTERIOR timers 10 30
neighbor PG-INTERIOR bfd
!
neighbor interface Et33 peer-group PG-INTERIOR remote-as 65002   ! peer-link to sw2
neighbor interface Et1  peer-group PG-INTERIOR remote-as 64500   ! vyos1
neighbor interface Et2  peer-group PG-INTERIOR remote-as 64500   ! vyos2

and the IPv4-over-IPv6 next-hop that FRR did for free has to be turned on by hand:

address-family ipv4
   neighbor PG-INTERIOR activate
   neighbor PG-INTERIOR next-hop address-family ipv6 originate

next-hop address-family ipv6 originate is the EOS knob for advertising IPv4 NLRI with an IPv6 next-hop. the interior interfaces themselves are just no switchport, no ip address, ipv6 enable, with RA on. the only interior neighbors missing here are the servers, which peer by address in a separate group, for reasons that come up below.

the edge

the vyos edges are where the two worlds meet. inside, the iBGP between the edges and the eBGP down to the switches is all unnumbered:

! ibgp to vyos1 (unnumbered)
neighbor eth3 interface remote-as 64500
neighbor eth3 timers 10 30
neighbor eth3 bfd
!
! ebgp to switches (unnumbered)
neighbor eth1 interface remote-as external
neighbor eth1 timers 10 30
neighbor eth1 bfd
neighbor eth2 interface remote-as external
neighbor eth2 timers 10 30
neighbor eth2 bfd

remote-as 64500 on the iBGP peer is the one place we give an explicit AS instead of external. the two edges are the same AS on purpose: AS64500 is the single face we show the world, one origin for the prefixes, not two. that makes the link between them iBGP, the only iBGP in the network. everything else has its own private ASN, so it's eBGP. with just two edges that one session is already a full mesh, so there's no route reflector to run.

interior timers are tightened to 10 30 from FRR's default 60 180, with BFD doing the sub-second work on top.

outside, the transit sessions stay numbered. real addresses on the upstream links, local-role customer (RFC 9234) and ttl-security hops 1 (GTSM) per peer, bgp ebgp-requires-policy left on for BCP 194. that's the normal ISP pattern and there's no reason to fight it. unnumbered is for links you own both ends of.

the result is less to manage, and it scales without planning. adding an interior link is ip link set up, no ipv6 nd suppress-ra, and one neighbor ethN interface line, the same on every box, so a new switch or edge is that recipe plus its ASN, nothing to look up. no subnet to allocate, no IPAM row, no /31 pool that grows with the fabric. that's why unnumbered is the default for large leaf-spine networks, where the interior links dwarf a two-switch rack and hand-numbering them would be a job in itself (RFC 7938).

the server side

the one interior link that doesn't get unnumbered is the server. the servers run Cilium, and Cilium's BGP control-plane only peers by explicit address, there's no interface peer the way FRR has one. so the cilium-to-switch links are the single interior exception: numbered, a /31 and a /127 per leg.

the speaker is declared in a CiliumBGPClusterConfig, one BGP instance per node with a peer entry per switch and family:

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
  name: cilium1
spec:
  nodeSelector:
    matchLabels:
      server: "1"
  bgpInstances:
    - name: "65301"
      localASN: 65301
      peers:
        - name: sw1-v4
          peerASN: 65001
          peerAddress: 10.10.110.0
          peerConfigRef: { name: cilium-peer }
        - name: sw1-v6
          peerASN: 65001
          peerAddress: "fd00:10:110::"
          peerConfigRef: { name: cilium-peer }
        - name: sw2-v4
          peerASN: 65002
          peerAddress: 10.10.111.0
          peerConfigRef: { name: cilium-peer }
        - name: sw2-v6
          peerASN: 65002
          peerAddress: "fd00:10:111::"
          peerConfigRef: { name: cilium-peer }

the parts that matter:

• every peer is keyed by a peerAddress, so a node brings up four sessions, a v4 and a v6 to each switch. the unnumbered side of the fabric gets one interface session that carries both families; here it's a session per family per neighbor, with an address to allocate for each.
• the matching switch side is plain numbered eBGP in its own PG-CILIUM group: neighbor 10.10.110.1 remote-as 65301, neighbor fd00:10:110::1 remote-as 65301, and so on. same peer-group and timers, just addresses instead of interfaces.
• what gets advertised isn't a network statement, it's a Cilium object. a CiliumLoadBalancerIPPool hands out the service IPs and a CiliumBGPAdvertisement picks which to announce. the anycast LB address comes out of both cilium1 and cilium2 and gets ECMP'd at the switches.

this is the boundary between the routed fabric and kubernetes, and it's the one spot in the interior that looks like an ordinary numbered eBGP session.

where it broke

unnumbered works well until something downstream doesn't speak it. two things caught us.

BFD is fine on unnumbered, broken on numbered eBGP, at least on this EOS. on ceos:4.36.0.1F, BFD on the unnumbered link-local peers comes up reliably, show bfd peers brief shows up. but add neighbor X bfd to a numbered eBGP peer, the switch-to-customer link, and that session wedges in Idle with Last reset: Address family activated (n/a) and never establishes. the unnumbered peers on the same switch stay up. FRR-to-FRR numbered eBGP does BFD fine, so it's specific to numbered eBGP on EOS. we left the numbered customer peers without BFD and let them fall back to BGP holdtime plus GTSM. the link you'd reach for BFD on first, the external one, is the one place it wouldn't go.

the second one is cilium, the place i wanted unnumbered most. the server links above are numbered because they have to be: cilium's BGP doesn't implement interface peering (cilium/cilium#22132, open since 2022), and it doesn't negotiate BFD either (#22394) in the OSS build. so failure detection on those links drops from BFD's sub-second to BGP holdtime, around 30 seconds. the FRR stand-ins in the fabric lab do speak unnumbered and BFD, so any demo that shows sub-second cilium failover is really showing the FRR stubs, not stock cilium.