The problem
Data quality issues kept recurring across the organization, and every team was diagnosing them separately. I gathered requirements from 10 teams to understand the shape of the problem before writing any code.
The approach
The first version was a high-performance Spark tool that compares and verifies roughly 252 million JSON files, ranging from a few KB to 250MB each, and reports granular differences through bot notifications. Instead of stopping at a single-purpose verification utility, I iterated it into a reusable, generic validation platform: one tool serving both release verification and general data quality checks, with automated quality gates and alerting built in.
The outcome
- Processes 252 million JSON files in under 30 minutes.
- Downstream data quality incidents dropped 35% across 10 teams.
- Adopted by all teams in the organization as the standard quality gate.
Why it matters
This is the project where I learned that the product work (discovery across 10 teams, iterating from a utility into a platform, driving adoption) mattered as much as the engineering.