Engineer should be able to design, build, operationalize, secure, and monitor data processing systems with a particular emphasis on security and compliance; scalability and efficiency; reliability and fidelity
Proficient in building and consuming REST services.
Experience building big data pipelines with Scala/Spark
Design and development of data ingestion services that can ingest 10s of TB of data;
Experience with object-oriented scripting languages: Java, Python(required), Scala(required) etc.
Research, design, and implement well tested improvements to the existing codebase
Review pull requests and provide valuable feedback to members of the team
Strong fundamentals in distributed systems design and development
Experience with relational and NoSQL database and related technologies (e.g. Postgres, Redis, Cassandra, ELK)
Strong working knowledge of container-based infrastructure on GCP (e.g. Docker, Kubernetes, GKE, etc.)
Experience with configuration management tools and concepts (e.g. Chef, Ansible, Docker, Puppet, etc.)
Experience with CI/CD tools and concepts (e.g. Jenkins, Artifactory, etc.)
Automate CI and deployment processes and best practices for the production data pipelines
Experience and working knowledge with HashiCorp Terraform including experience writing Terraform Modules
Ability to communicate effectively (both verbal and written)