Currently, the only way to attach a piece of information to an network packet (sk_buff) that will travel with it through the Linux network stack is the 32-bit mark field.

Once set, the mark can be read in firewall rules, used to drive routing, and accessed by BPF programs, among other uses. This versatility leads to fierce competition over the mark’s bits. Being just 32 bits wide, it often ends up limiting its practical applications.

That is why in 2024 we embarked on a journey to enable users to attach hundreds of bytes of metadata to network packets - a concept which we call "rich packet metadata" - which unblocks new and exciting applications such as: * Tracing packets through layers of the network stack, even when crossing the kernel-user space barrier. * Metadata-based packet redirection, routing, and socket steering with early packet classification in XDP. * Extraction of information from encapsulation headers and passing it to user space, or vice versa.

While we have made significant progress since the project has started. Our journey ([1], [2]) to let BPF programs and user-space apps attach rich metadata to packets is far from over. In this talk, we'll share what's been done, what's next, what we've learned, and where are the dragons we've yet to slay.

Part I: Upstream Progress and Roadmap

We'll cover:

• Why we shifted from the old "skb traits" idea [3] to reusing existing skb metadata.
• How bpf_dynptr came to the rescue, and why skb->data_meta still haunts us.
• The blockers that keep metadata from traveling cleanly through the Rx path and how we plan to fix them.
• Our roadmap for making metadata work on the Tx path.
• Ideas for producing and consuming metadata directly in the network stack.

Part II: Lessons from Production

Since our last update [2], we've built several features with packet metadata in Cloudflare's production environment. We'll share hard-earned lessons, including:

• Managing metadata contents and optimizing metadata area size.
• Using a TLV structure to encode metadata, and how we shuffle it between packet data, metadata area, and maps.
• Real-world challenges of reading and writing metadata efficiently.
• Passing metadata from packet to socket layers, with full access for TCP via socket options—and our creative hacks for UDP.

Finally, we'll discuss where things still hurt:

• Testing headaches and why BPF_PROG_RUN needs love.
• What an ideal user API would look like for us.

If you're curious about where packet metadata is headed, or want to help shape the future, this session is for you.

[1] https://lpc.events/event/18/contributions/1935/ [2] https://www.netdevconf.info/0x19/sessions/talk/traits-rich-packet-metadata.html [3] https://lore.kernel.org/all/20250422-afabre-traits-010-rfc2-v2-0-92bcc6b146c9@arthurfabre.com/