• Share this post Keep up with us Summary Learn how System Tables expose platform telemetry as queryable tables, including metadata and execution insights for Lakeflow jobs and pipelines.Use example queries to turn this telemetry into insights on reliability, cost and efficiency opportunities at scale for Lakeflow jobs.Centralize these insights into a shared, day-to-day operational view for data engineering teams with the Lakeflow dashboard template. • Learn how System Tables expose platform telemetry as queryable tables, including metadata and execution insights for Lakeflow jobs and pipelines. • Use example queries to turn this telemetry into insights on reliability, cost and efficiency opportunities at scale for Lakeflow jobs. • Centralize these insights into a shared, day-to-day operational view for data engineering teams with the Lakeflow dashboard template. • The 3 AM Problem It is 3 AM and something broke. • The dashboard is stale, an SLA slipped, and everyone is guessing which part of the platform drifted out of line.
Article Summaries:
- Databricks has expanded its System Tables to include Lakeflow Jobs, adding richer schemas that expose job configurations, task definitions, run timelines, and ownership data in a single, queryable interface. The new tables-such as system.lakeflow.jobs, job_tasks, job_run_timeline, and job_task_run_timeline-use SCD Type 2 semantics to preserve full change history, enabling configuration auditing and trend analysis. They support cross‑workspace, region‑wide analysis, helping teams monitor pipeline health, surface cost‑saving opportunities, and quickly identify failures. With 365 days of data and a 17‑fold YoY increase in daily queries, the feature is rapidly adopted for advanced observability and operational insight.
Sources: