Design and Implement Data Architecture: Lead the design and development of scalable, cloud-based data architectures leveraging Azure Data Lakehouse and Medallion Architecture principles.
Azure Databricks and Delta Lake: Architect and implement data pipelines and ETL processes using Azure Databricks, ensuring seamless integration with Delta Lake to enable ACID transactions, time travel, and optimized data storage.
Medallion Architecture: Implement the Medallion Architecture pattern, building Bronze, Silver, and Gold layers to ensure efficient data processing, aggregation, and enrichment for analytics and reporting.
Customer Data Platform (CDP) Integration: Design and manage the data architecture to effectively handle customer data, ensuring the Customer Data Platform (CDP) is integrated with the Azure data ecosystem, enabling personalized and customer-centric analytics.
Data Lakehouse Management: Lead the implementation and optimization of Azure Data Lake Storage (ADLS) to support structured, semi-structured, and unstructured data storage, ensuring efficient querying and data retrieval.
Data Governance and Security: Define and implement robust data governance, security, and compliance practices using tools like Azure Data Catalog, Azure Purview, and Azure Key Vault. Ensure data is securely stored and accessed in compliance with data privacy regulations (GDPR, CCPA).
Performance Optimization: Optimize data pipelines for performance, scalability, and cost-efficiency, ensuring that data is processed efficiently and meets SLAs for downstream analytics and reporting systems.
Collaboration with Data Engineers and Analysts: Work closely with data engineers, business analysts, and data scientists to define data requirements, ensuring that data pipelines are designed to meet business needs.
Real-Time Data Integration: Architect real-time data ingestion pipelines, integrating with various data sources (APIs, event hubs, etc.), and streamlining real-time analytics using Azure Databricks and Delta Lake.
Documentation and Best Practices: Establish best practices for data architecture, provide documentation for data pipelines and architecture, and ensure knowledge sharing across the organization.