Competent ML engineer, who is independent, results driven and is capable of taking business requirements and building out the technologies to take it to production.
Big Data ML Engineer with expert level experience in Hadoop ecosystem and real-time analytics tools including PySpark/Scala-Spark/Hive/Hadoop CLI/MapReduce/ Storm/Kafka/Lambda
Architecture/Mongo
expert with using the larger Hadoop eco system
Familiar with job scheduling challenges in Hadoop
Experienced in creating and submitting Spark jobs
Experience in high performance tuning and scalability
Experience in working on real time stream processing technologies like Spark structured streaming, Kafka, FLink + rocksDB
Expertise in Python/Spark and their related libraries and frameworks
Experienced with Spring Framework and Spring Boot
Experience in building training ML pipeline and efforts involved in ML Model deployment
Experience in other ML concepts – Real time distributed model inferencing pipeline, Champion/Challenger framework, A/B Testing, Model performance Scorecard and assessment, Retraining framework, etc.
Unix/Linux expertise; comfortable with Linux operating system and Shell Scripting
Working Experience in Redis DB – Writing to and from redis-- very critcal Also exp Azure cache
Experience in creating Kubernetes open-source container-orchestration system for automating application deployment, scaling, and management; (Experience in Azure is preferable)
Experience in tweaking/using Jenkins, deployment orchestration and/or Kubernetes for CI/CD pipeline
Creating/modifying Dockers, microservices and deploying them via Kubernetes
PL/SQL, RDBMS background with Oracle/MySQL
Familiarity with Cassandra, Mongo preferred.
Design, Development, Unit and Integration testing of complex data pipelines and to handle data volumes to derive insights
Ability to optimize code to be able to run efficiently with stipulated SLA
Excellent problem-solving skills, with attention to detail, focus on quality and timely delivery of assigned tasks
Good presentation and communication skills required