Although this role will be sitting on-site, they must also be able to use personal workstation or options are available for rent. They are a Microsoft shop.
5-8+ years of experience in data engineering or a related role.
Seeking an individual with the ability to learn fast, eager to be in a collaborative team environment and work with an offshore team.
Design, build, and manage ETL data pipelines using AWS services services, specifically Glue, Athena, and Lambda is a plus.
Hands-on experience with Apache Hudi for real-time data management.
Strong programming skills in Python and PySpark.
Familiarity with SQL and NoSQL databases.
Knowledge of data governance and data security best practices.
Experience with workflow scheduling tools like AWS Step Functions or Control M.
Collaborate with cross-functional teams to meet business objectives.
Monitor performance and advise any necessary infrastructure changes.
Experience with Hadoop ecosystem is a must.
Experience with RedShift is a must.
Develop technical documentation including data dictionaries, metadata, and pipeline architecture.
Troubleshoot data issues and provide ongoing operational support.