Develop and Maintain Data Pipelines: Design, implement, and optimize data pipelines using Azure Databricks and Azure Data Factory to ensure efficient data ingestion and transformation.
Data Processing: Utilize Scala to process and manipulate large datasets, ensuring data quality and reliability.
SQL Query Development: Write complex SQL queries for data extraction, transformation, and loading (ETL) from various data sources.
Integration with .NET Applications: Collaborate with development teams to integrate data solutions with .NET applications and services.
Performance Tuning: Monitor and troubleshoot data pipeline performance, implementing optimizations as necessary.
Documentation: Create and maintain comprehensive documentation of data architecture, processes, and workflows.
Collaboration: Work closely with data analysts, business stakeholders, and other engineering teams to understand data requirements and deliver actionable insights.
Qualifications:
Experience: 12+ years of experience in data engineering or related fields, with a focus on Azure technologies.
Technical Skills:
Proficiency in Azure Databricks for data processing and analytics.
Strong knowledge of Azure Data Factory for data integration and orchestration.
Experience with Scala for data processing tasks.
Solid SQL skills for database querying and data manipulation.
Familiarity with .NET frameworks and integration practices.
Problem-Solving Skills: Strong analytical and problem-solving abilities, with a focus on delivering high-quality data solutions.
Communication: Excellent communication and collaboration skills to work effectively within a team environment.