Competent Big Data/Databrick engineer, who is independent, results driven and is capable of taking business requirements and building out the technologies to take it to production.
Big Data Engineer with expert level experience in Hadoop ecosystem and real time analytics tools including PySpark/Scala Spark/Hive/Hadoop CLI/MapReduce/ Storm/Kafka/Lambda Architecture/Mongo
Familiar with job scheduling challenges in Hadoop
Experienced in creating and submitting Spark jobs
Experience in high performance tuning and scalability
Experience in working on real time stream processing technologies like Spark structured streaming, Kafka
Expertise in Python/Spark and their related libraries and frameworks ¿ Experience in building pipeline and efforts involved in deployment