Design and development pf Python and spark based back-end software modules, performance improvement and testing of these modules.
Scripting using Python and shell scripts for ETL workflow.
Design and development of back end big data frameworks that is built on top of Spark with features like Spark as a service, workflow and pipeline management, handling batch and streaming jobs.
Required skills:
Expert level programming in Spark SQL and Python
Some data migration experience from on-premise to cloud (any cloud OK, GCP preferred)
Some post data migration experience needed, like data validation
At least 8+ years of experience building and managing complex products/solutions. Good problem Solving/analytical skills.
4+ years of experience working in Linux/Unix environment.
A degree in Computer Science, Management Information Systems is preferred but not required
Most importantly be a good team player, willingness to learn and implement Big data technologies.