• EXPEDIA GROUP TECHNOLOGY - DATA A strategic shift toward data flows that are version-controlled, testable, and auditable Ensuring data quality in production environments is critical - but challenging - especially when test environments lack access to live data. • Traditional approaches often involve duplicating production tables for User Acceptance Testing (UAT) and validation, leading to resource overhead, governance challenges, and inconsistencies. • Photo by Yu Kato on Unsplash What is Write‑Audit‑Publish? • Write‑Audit‑Publish (WAP) is a robust data validation workflow that improves data quality by staging data changes through three steps: Source: Write-Audit-Publish Stages Write New data is written to an isolated , non-production environment, preventing disruption to production. • Audit The data is validated with quality checks such as, NULL detection, Duplicate checks or schema integrity. • Publish After passing audits, data is atomically committed to the production environment, ensuring consistency and accuracy.
Article Summaries:
- Expedia Group Technology has rolled out an Iceberg‑based Write‑Audit‑Publish (WAP) workflow to improve data quality in production. The approach writes new data to an isolated Iceberg branch, runs automated audits (NULL checks, duplicates, schema validation) without touching the live table, and then atomically publishes the validated data to the main branch. This eliminates the need to duplicate production tables for UAT, reduces storage and compute overhead, and cuts operational costs. By reusing the same ETL logic across production and testing, the method lowers risk, enhances auditability, and streamlines promotion of clean data.
Sources: