The Data Engineer will support our data department on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects
Design & Develop Transformations aspects using ELT framework to modernize the Ingestion pipelines and build data transformations at scale
Experience in the areas of design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies such as S3, Hive, Databricks, Scala, Python and Distributed data processing framework
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources into SQL Server, MongoDB, and others
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc
Be in tune with emerging trends Big data and cloud technologies and participate in evaluation of new technologies
Ensure compliance through the adoption of enterprise standards and promotion of best practice / guiding principles aligned with organization standards