- Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in data engineering development tools and technologies
- Work with a team of developers with deep experience in spark, hive, machine learning, distributed microservices, and full stack systems
- Create and maintain overall optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Manage data migrations/conversions and troubleshooting data processing issues.
- Utilize programming languages like Python Java, Scala and RDBMS and NoSQL databases and Cloud based data warehousing services such as Snowflake
- Share your passion for staying on top of tech trends, experimenting with and learning new technologies, participating in internal & external technology communities, and mentoring other members of the engineering community
- Perform unit tests and conducting reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance
Required Qualifications (5 – 8 bullet points on must have skills)
Required Qualifications:
- Experience 5 to 12 years
- Experience working with distributed teams
- Familiar with Agile/Scrum development process.
Programming/languages:
- Python, SQL
- JavaScript, Java or Scala, a plus
Data engineering:
- Spark/Pyspark
- Kafka, queue/messaging paradigms, a plus
AWS:
- security/networking basics, as well as S3, Kafka, Kinesis, Glue, RDS, No SQL databases, Lambda and Step Functions
- Redshift, Athena a plus
Not required, but bonus if you have:
- Machine learning academic or work experiences.
- Kubernetes or docker knowledge.