In this role, you'll be managing end-to-end data engineering and automation, ensuring the highest standards of data quality, efficiency, scalability and security. Your day-to-day will include:
- Pipeline Management: Oversee and refine data pipelines, ensuring data is collected, transformed, and stored accurately across all systems. This includes monitoring and troubleshooting pipelines to prevent data loss, downtime and improve efficiency.
- ETL Development: Build, maintain, and improve ETL workflows that support data integration from multiple sources, handling various data types and frequencies. This includes event-driven solutions to sequence and automate Azure-centric data pipelines.
- Data Modeling & Transformation: Collaborate with a Data Management team to design data models that align with business needs and perform data transformations to create usable data sets for analysis and reporting.
- SQL Query Optimization: Write, test, and optimize SQL queries to ensure efficient data retrieval and processing.
- Collaboration with Analytics Teams: Work closely with data scientists, analysts, and stakeholders to support data requests, visualization needs, and other analytics functions.
- Cloud Infrastructure Management: Monitor and manage an Azure Data Platform and collaborate on other cloud platforms (Oracle Cloud, AWS) to maintain scalability and ensure secure data access.
- Data Quality Assurance: Conduct regular quality checks, clean data sets, and implement data governance practices to maintain accuracy and reliability.
- Documentation & Reporting: Keep thorough documentation of data processes, configurations, and workflows. Provide regular updates on system performance and data availability to downstream users and program director.
- Innovation & Continuous Improvement: Stay up to date on industry trends, experiment with new data tools and techniques, and apply best practices to drive continuous improvement.
- AI/ML Integration: Support AI/ML initiatives to enhance and support data processes, predictive modeling, and drive business value for business units.