Data pipeline development: Collaborate with the data engineering team to develop and maintain data pipelines, ETL processes, and data integration workflows using Azure Data Factory, Azure Synapse Analytics, Spark SQL, and other relevant technologies. Ensure the timely and accurate movement of data between systems.
Data processing and transformation: Develop & Assist in implementing / maintaining data processing and transformation logic using Azure Synapse Analytics, Spark SQL or Databricks. Extract insights from raw data and transform it into meaningful and structured formats in our data products. Strong DBT (Data Build Tool) experience is a plus, as you may be involved in the implementation and management of DBT workflows.
Azure service utilization: Work with Azure services such as SQL Server, Azure Data Factory, Azure Synapse and Azure Databricks & DBT to build and manage data infrastructure components.
Leverage the capabilities of these services to ensure efficient and scalable data processing.
Version control and collaboration: Utilize Azure DevOps and Git for version control and collaborate effectively with the team to manage code repositories and ensure proper documentation and knowledge sharing.
Azure DevOps integration: Assist in integrating data engineering workflows with Azure DevOps for continuous integration, continuous deployment, and automated testing. Contribute to the implementation of CI/CD pipelines for data engineering projects.
Data Modelling, visualization and reporting: Exposure to Power BI or similar reporting tools. Ability to collaborate with cross-functional teams to gather requirements and translate them into optimal data models (dimensional modelling experience required)
Troubleshooting and support: Assist in identifying and resolving data pipeline issues, bottlenecks, and data quality problems. Provide support in investigating and troubleshooting data-related incidents.