Primary responsibilities :
- Work with Data Science and business teams to understand data requirements
- Design data pipelines and frameworks
- Develop scripts for ETL, Data Preparation, scheduling jobs etc.
- Use APIs to ingest third party data and structure the data in DB as per business use case
Mandatory Skills:
- 3+ years of experience in Data Engineering
- Python, SQL
- Experience with any of the cloud technologies like AWS, Google cloud etc.
- Experience in ETL tasks like extracting data from multiple sources, join multiple tables/schemas, create custom defined variables, and loading the data in database.
- Experience in Linux, copying files, code commits to Git, executing python scripts from CLI etc.
- Experience working with APIs to get, post and extract data from online sources
- Basic knowledge of configuring servers in cloud and deploying ETL scripts for example scheduling a script in selected server to retrieve data from Twitter site using twitter APIs
Good to Have:
Hands on experience on some of AWS services like Dynamo DB, Lambda, Sagemaker etc. Working knowledge with Snowflake, snow pipe will be an added advantage Knowledge of Machine Learning models