- DevSecOps experience with clear understanding of related tools, structure, and processes in an Agile/ Scrum setting, 5+ years
- Azure Cloud- Data and Analytic Services, 4+ Years
- Platform Administration using Azure Databricks, 4+ Years
- Strong experience with Python and PowerShell
- Experience building CI/DC pipelines
- Databricks workspaces, users, and groups management including SSO, provisioning, access control, and workspace storage
- Databricks Notebooks, SQL Spark, Delta Lake performance optimization
- SQL, GitHub, Azure DevOps, Terraform
Duties & Responsibilities:
- Design and implement intelligent, predictable, reliable, scalable, cost-efficient, and easy-to-use platforms and frameworks for Data Science Machine Learning (DSML), Enterprise Data Lake/Warehouse (EDL/EDW), Master Data Management (MDM) functions
- Implement, setup, configure, monitor, and maintain Microsoft Azure Databricks
- Evaluate and automate the scaling and capacity requirements within Azure environments. Automate Databricks environment deployments
- Troubleshoot and resolve Databricks performance issues, connectivity issues, security issues
- Implement Azure cloud capabilities including but not limited to storage, key vaults, virtual networking, virtual machines, and automated deployments
- Make recommendations and execute cost saving strategies for Databricks on Azure Cloud
- Establish patterns, build CI/CD pipelines, and frameworks
- Self-directed and able to balance the priorities of multiple teams, systems, and products with a leadership mindset
- Streamline development and operation processes via continuous integration, deployment, automated testing while leveraging SRE principles
- Drive efficient resolution for system outages, performance and functional shortcomings