Lead the development, design, and optimization of the client’s Azure-based data lakehouse eco-system.
Assess and enhance the performance of the current data lakehouse infrastructure by identifying bottlenecks, improving data processing efficiencies, and optimizing resource utilization.
Work with Azure Synapse to create, manage, and optimize notebooks written in Python, ensuring seamless integration and performance within the lakehouse environment.
Integrate new data sources and ensure that all data pipelines are efficient, scalable, and secure.
Provide consultation to stakeholders on the most effective strategies for integrating data sources, optimizing queries, and improving overall performance.
Evaluate current data products, workflows, and processes to recommend improvements that maximize the value of the data lakehouse eco-system.
Offer insights into the best practices for leveraging Microsoft Fabric tool suite for scalable and secure data operations.
Design and implement data pipelines for loading structured and unstructured data from multiple sources into the Azure data lakehouse.
Ensure efficient data transformation processes, leveraging Azure SQL, Synapse, and Python scripts for ETL/ELT workflows.
Integrate Power Platform tools (Power BI, Power Automate, Power Apps) to enable business users with self-service analytics and seamless access to the data lakehouse.
Ensure the implementation of robust data governance policies and practices, including data security, auditing, and compliance with industry regulations.
Monitor and manage role-based access control (RBAC) and data encryption to ensure the security of the data lakehouse.