Data Platform Infrastructure (Data Infrastructure resources 1-2)
- Capacity planning, configure, deploy and maintain Databricks clusters, workspaces and Snowflake infrastructure on Azure cloud
- Use Terraform to automate provisioning and deploy Databricks clusters, workspaces, Snowflake and associated Azure resources. Ensure consistency and repeatability by treating infrastructure as code
- Monitor and optimize Databricks cluster performance and Snowflake resource utilization, troubleshoot issues to ensure optimal performance and cost-effectiveness.
- Implement and manage access controls and security policies to protect sensitive data.
- Develop environment strategies across the technology stack and governance based on best practices
- Provide technical support to Databricks and Snowflake users, including troubleshooting and issue resolution.
- Implement and enforce security policies, RBAC, access controls, and encryption mechanisms.
- Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
- Collaborate with cross-functional teams, including data scientists, data engineers, and business analysts to understand their requirements and provide technical solutions
- Data Governance and Quality Management: Create and enforce data governance standards, ensuring robust data quality and compliance through tools such as Databricks Unity Catalog, Collibra and Snowflake Polaris.
- Enforce data governance, data quality, and enterprise standards, supporting a robust production environment