Data Pipeline Development: Design, develop, and migrate robust ETL/ELT pipelines in Azure Data Factory and Databricks to support data ingestion, transformation, and storage.
Unity Catalog Implementation & Migration: Implement and manage Unity Catalog within the Databricks environment to organize, secure, and govern data assets. Migrate legacy pipelines, notebooks, and tables to a Unity Catalog compliant format.
Data Modeling: Develop and maintain scalable data models and architectures that support business intelligence and analytics needs.
Data Governance: Collaborate with data governance teams to ensure data is properly classified, secured, and audited within Unity Catalog.
Performance Optimization: Monitor and optimize the performance of data pipelines and Databricks clusters to ensure cost-effective and efficient data processing.
Data Security: Implement and manage role-based access controls (RBAC) and data encryption practices to safeguard sensitive data.
Collaboration: Work with cross-functional teams to understand data requirements and deliver high-quality data solutions that meet business needs.
Automation: Develop automation scripts and tools to streamline data engineering processes and improve productivity.
Troubleshooting: Identify and resolve issues related to data pipelines, data quality, and performance bottlenecks.