Job title: Data Engineer
Primary Work Location: San Francisco , CA
Experience : 5-6+years
Duration: 6+ months
Direct Client and No Layers
JOB DESCRIPTION:
Data Engineer to support our client located in San Francisco, CA.
Location: Remote (SF post COVID)
Duration: 6 months (possible extensions up to 2 years)
Position Responsibilities
- Collaborate on a daily basis with the product team. This includes pairing for all aspects of software delivery.
- Create and maintain optimal data pipeline architecture.
- Assemble large, complex data sets that meet functional and non-functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, and re-designing infrastructure for greater scalability.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
Minimum Qualifications:
- Bachelor's Degree in Computer Science or job-related discipline or equivalent experience
- 5 years-experience with software delivery
- Experience delivering product with Agile / Scrum methodologies
- Experience with PY-Spark
- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
- Experience with ETL flows using Python
- Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
- Strong analytic skills related to working with unstructured datasets.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
- Proficiency with the following tools that enable the candidate to contribute autonomously: Glue, Kafka, Redshift (with a focus on infrastructure -as-code), Python.
Desired Qualifications
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- AWS DevOps skills
- Humble – is open to being coached, has high Emotional Quotient (EQ) and is self-aware
- Hungry – desires to get things done while honoring people, and seeks better ways to do the job, is highly motivated by the significant impact this work will have
- Collaborative – has strong interpersonal skills; demonstrates empathy with teammates and stakeholders, cares about and works well with teammates
- Willingness to impact beyond defined role
- Experience with data & analytics product development