- Involved in the Study of the business logic and coordinate with the client to gather the requirements.
- Experience with Hive, Nifi, Airflow, Snowflake and Kafka
- Experience in Cloud environments, particularly AWS - EC2, S3, Lambda
- Experience in Unit testing, integration testing for Spark scripts
- Scripting with Bash, CSH, Shell, or Python
- Designing and implementing large-scale ingest systems in a Big Data environment
- Optimizing all stages of the data life cycle, from initial planning, to ingest, through final display and beyond
- Designing and implementing data extraction, cleansing, transformation, loading, and replication/distribution capabilities
- Developing custom solutions/code to ingest and exploit new and existing data sources
- Working with Sponsor development teams to improve application performance
- Providing support to maintain, optimize, troubleshoot, and configure the AWS/spark/Hadoop environment as needed
- Collaborating with team-mates, other service providers, vendors, and users to develop new and more efficient method
Primary Skill: Hive, Snowflake, Nifi, Jenkins, Airflow, AWS, Unix and Kafka
Secondary Skill: EMR, Spark & Scala
Good to Have - Java, Python