• Husky layers over object storage, using fragments to batch trillions of daily events. • Metadata per fragment lets queries locate relevant data without scanning entire store. • Design trades off compaction frequency, target file size, sorting, and indexing for speed. • Write‑heavy workload with minimal updates; queries focus on newest data, occasional deep scans. • Workers fetch fragments, scan events, and merge results, keeping query cost proportional to fragment count. • Efficient compaction keeps fragment size optimal, reducing storage costs and improving read latency.
Article Summaries:
- Datadog’s Husky event‑store system has introduced a new storage‑layer design that improves data compaction at scale. Husky aggregates incoming observability events into “fragments” stored on object storage, with metadata kept in FoundationDB. To reduce query costs, the team implemented a streaming‑merge compaction that combines many small fragments into larger ones, keeping the number of files low while preserving atomic visibility for queries. This approach balances rapid ingestion-ensuring data is queryable shortly after arrival-with efficient scanning of large datasets, enabling both low‑latency edge queries and heavy analytical workloads across trillions of events daily.
Sources: