• Introducing uFowarder: The Consumer Proxy for Kafka Async Queuing 5 February / GlobalIntroduction Uber has one of the largest deployments of Apache Kafka® in the world, processing trillions of messages and multiple petabytes of data per day. • Three years ago, we built a push-based consumer proxy for Kafka’s Async Queue. • It’s become the primary option for reading data from Kafka in pub-sub use cases at Uber, with over 1,000 consumer services onboarded. • We open-sourced the solution under the name uForwarder. • This blog describes the challenges we faced when productionizing uForwarder and the solutions we implemented to optimize hardware efficiency, ensure consumer isolation, address head-of-line blocking, and support message delay processing. • By reading this blog, you can understand the thinking behind uForwarder before applying it to your use cases.
Article Summaries:
- Uber has open‑sourced uForwarder, a push‑based consumer proxy that simplifies reading from Apache Kafka’s async queue. Built to support Uber’s massive Kafka deployment, the proxy fetches messages via Kafka’s binary protocol and forwards each one to consumer services over gRPC, then aggregates results and commits offsets only when safe. The design hides Kafka’s partitioning and consumer‑group complexities, improving hardware efficiency, consumer isolation, and resilience to head‑of‑line blocking. Uber’s blog details how the team tackled challenges such as poison‑pill messages, message size limits, and instance failures, and how they added a dead‑letter queue and delay‑processing support to keep the system robust.
Sources: