Project Siphon

Earlier pipelines missed 30–40% of database changes, so analytics, product intelligence and AI were built on incomplete data.

Legacy pipelines tied up data and SRE engineers in maintenance.

GitLab runs three deployment models and was moving to a cell-based architecture, so CDC had to work everywhere.

PostgreSQL changes are published into NATS JetStream, capturing every insert, update and delete.

Go consumers write Parquet through buffered and direct write paths, with right-sized files and schema drift handled by versionHash keying.

Apache Iceberg tables on Nessie / Snowflake Open Catalog serve the same governed data to Snowflake and Spark, with Prometheus metrics at every stage.

Run as DRI from feasibility to production through a working council of seven functions: data engineering, analytics, data science, instrumentation, product engineering, database operations and SRE.

Mirror staging and production in a Dockerised Nessie/GCS dev environment, cutting onboarding from days to under 2 hours.

Design for the Cells migration from the start, so one platform serves all three deployment models.