Design, develop, and maintain ETL/ELT pipelines using Spark SQL, T-SQL, and other relevant tools.
Implement data integration and transformation workflows for structured and unstructured data sources.
Optimize data pipelines for performance, reliability, and scalability.
Azure Ecosystem Expertise:
Utilize Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake, and Azure Blob Storage for data movement, transformation, and storage.
Leverage Azure SQL Database and related services to build efficient and scalable data solutions.
Explore and implement best practices for data management and orchestration in the Azure cloud.
Data Modeling and Development:
Design and implement data models to support business intelligence, analytics, and reporting.
Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
Develop reusable data assets such as scripts, templates, and frameworks for engineering workflows.
Collaboration:
Work closely with data analysts, data scientists, and business teams to support their data needs.
Collaborate with software engineers and architects to integrate data solutions into broader systems.
Performance and Optimization:
Conduct performance tuning for Spark SQL and T-SQL queries and workflows.
Optimize data pipelines and data structures for faster processing and minimal latency.
Innovation and Best Practices:
Stay updated on emerging technologies and trends in data engineering and cloud platforms.
Contribute to the continuous improvement of data engineering processes and frameworks.