Work / Hybrid ML Data Remediation

Hybrid ML Data Remediation

Machine LearningRequirements DefinitionApache SparkData Quality
Role
Led, requirements to delivery
Timeline
Senior SWE period
Team
Data science & business stakeholders
Stack
Scala · Python · Spark · AWS EMR
TL;DR · Quality issues in key map data elements needed remediation at scale. I designed a hybrid system, a Scala heuristic algorithm feeding an ML classification model, running on Spark across millions of records. It reached 90% model accuracy with over 80% precision and recall on both classes, and set the precedent in our sub-organization for solving business problems with AI.

The problem

Key map data elements had persistent quality issues, and fixing them meant finding them first, across millions of records.

The approach

I led the remediation initiative and designed a hybrid solution: a heuristic algorithm in Scala running at scale on Apache Spark, paired with an ML classification model that produces the final predictions, with the heuristic output feeding the ML pipeline as an input feature layer. Requirements came out of joint definition sessions with data science and business stakeholders.

The outcome

  • 90% model accuracy, with over 80% precision and recall for both classes.
  • One of the first initiatives in the sub-organization to combine engineering and AI against a critical business problem, establishing a reusable approach.

Why it matters

Pure ML wasn't the answer; pure heuristics weren't either. The result came from engineering judgment about how to combine them.

← All work