Spark , Python , SQL , AWS, Pyspark, DW concepts
Responsibilities:
- (The primary tasks, functions and deliverables of the role)
- Design and build reusable components, frameworks and libraries at scale to support analytics products
- Design and implement product features in collaboration with business and Technology stakeholders
- Identify and solve issues concerning data management to improve data quality
- Clean, prepare and optimize data for ingestion and consumption
- Collaborate on the implementation of new data management projects and re-structure of the current data architecture
- Implement automated workflows and routines using workflow scheduling tools
- Build continuous integration, test-driven development and production deployment frameworks
- Collaboratively review design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards
- Analyze and profile data for designing scalable solutions
- Troubleshoot data issues and perform root cause analysis to proactively resolve product and operational issues
- Lead and mentor team of data engineers
- Trouble shoot key issues and ensure team member performance