Nate Rosidi demonstrates SQL workflow with testing, CI/CD and data quality automation
Updated
Updated · KDnuggets · May 4
Nate Rosidi demonstrates SQL workflow with testing, CI/CD and data quality automation
1 articles · Updated · KDnuggets · May 4
Using an Amazon daily-spending query, he shows PostgreSQL logic wrapped in Python unittest with an in-memory SQLite database, controlled customer and order data, and expected outputs.
He also outlines a GitHub Actions pipeline that runs on pushes and pull requests to main, using Ubuntu, Python 3.10 and automated unit-test discovery.
The workflow adds SQL-based data quality checks for duplicate names, negative costs, invalid dates and orphaned orders, aiming to catch regressions and bad data before analytics pipelines fail.
When does the engineering overhead of this SQL testing framework become more of a burden than a benefit for agile data teams?
If bad data costs millions, why isn't treating SQL like critical software already the universal standard for all data teams?
With AI automating analytics, does this rigorous discipline for SQL become obsolete or even more critical for ensuring reliable outcomes?