Key takeaways:
- Most pipeline complexity lives in the data the pipeline carries, not in the structure of the pipeline itself
- Data pipelines are full of bugs with soft edges.
- Insights get left on the cutting room floor.
The solution to general technical debt isn’t a secret: it’s automated testing.
… data quality isn’t a purely technical problem.