We are seeking a highly skilled and experienced Data Engineer to join our innovative team. The ideal candidate will have a strong background in Azure cloud services, Databricks, and AI/ML technologies. This role requires a professional who can design, build, and manage complex data pipelines, implement scalable data processing systems, and leverage AI/ML capabilities to drive insights and innovations.
Key Responsibilities
- Design, construct, install, test, and maintain highly scalable data management systems.
- Ensure systems meet business requirements and industry practices for data integrity, security, and availability.
- Implement data ingestion routines, both real-time and batch, from a wide variety of data sources using Azure and Databricks.
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
- Employ advanced analytics techniques, including AI and machine learning models, to solve complex business problems.
- Work closely with data scientists, business analysts, and other stakeholders to understand data needs and implement solutions.
- Optimize data processing pipelines for performance and scalability.
- Stay current with new technologies and techniques in data engineering, cloud services, and AI/ML to improve project outcomes.
Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or a related field.
- Proven experience as a Data Engineer or in a similar role with a focus on Azure cloud services, Databricks, and AI/ML.
- Expertise in SQL, Python, Scala, or Java, with a strong foundation in data structures and algorithms.
- Experience building and optimizing 'big data' data pipelines, architectures, and data sets.
- Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
- Knowledge of machine learning algorithms and artificial intelligence principles.
- Familiarity with Azure Data Factory, Azure Data Lake, Azure SQL Data Warehouse, and other Azure storage solutions.
- Excellent problem-solving skills and the ability to work under tight deadlines.
- Strong communication and teamwork skills.