Design, development and testing of data ingestion pipelines, perform end to end automation of ETL process and migration of various datasets that are being ingested into the platform.
Participate in discussion for technical architecture, Migration activities, collaborate with Product Managers and Architects to establish the physical application framework (e.g. libraries, modules, execution environments)
Develop transformation logic, interfaces and reports as need to meet project requirements.
Required Skills:
10+ years of experience building and managing complex products/solutions.
5+ years of experience supporting data heavy applications and databases leveraging Python and PySpark
Strong data handling and text log analysis skills
Expert level experience in Python, PySpark and SQL