Dynamic Data Ingestion Framework for HEDIS Schema Drift

Developed a dynamic Python-based ingestion framework to automatically adapt to unpredictable data layout changes, preventing ETL pipeline failures and automating Source-to-Target mapping documentation.

Python ETL Enterprise Data Warehouse Healthcare IT Data Architecture Excel Automation

The Challenge

While serving as Product Owner leading a team migrating legacy systems to a new Enterprise Data Warehouse, we encountered persistent ETL pipeline failures. HEDIS data was periodically sent to us with unannounced layout changes, and the healthcare system's IT department was unable to provide advance notice of these schema shifts. These unexpected modifications continuously broke our Extract, Transfer, Load (ETL) processes, causing data delays and requiring costly, time-consuming manual intervention from engineers to remap the data.

Architectural Approach

To resolve the pipeline bottleneck, I leveraged my Python expertise to design a dynamic data ingestion tool. Within a single week, I developed a working prototype that could dynamically parse and ingest incoming data regardless of structural layout changes. Furthermore, I engineered the script to automatically generate an Excel spreadsheet to serve as the required Source-to-Target Mapping documentation. By providing this prototype to my engineering team, we eliminated the downtime associated with manual re-mapping and significantly accelerated our EDW migration efforts.

Project Repository: https://github.com/MLAbram/python-dynamic-data-ingestion-framework
GitHub Profile: https://github.com/MLAbram

Outcomes & Impact

~90%

Reduction in Pipeline Maintenance

Eliminated the recurring manual engineering hours traditionally required to rewrite ETL scripts following unannounced HEDIS data layout changes.

40+ Hours

Saved Per Schema Shift

By dynamically auto-generating the Source-to-Target Mapping Excel documentation, engineers were spared from manually reverse-engineering layout changes.

1 Week

Rapid Prototype Delivery

Designed, developed, and delivered a fully functional dynamic ingestion prototype to the engineering team to accelerate the EDW migration.

Need resilient architecture for your next project?

Let's discuss how we can optimize your infrastructure and database performance.

Schedule a Call