Diagnose existing architecture and assess data utility; identify gaps
Lead feature engineering and DS pipeline creation, including ETL experience and support for DS feature selection efforts
Design and jointly develop the data architecture with data architect; ensure scalability, security and maintenance
Build data tools and products for effort automation and easy data accessibility
Responsibilities
Collaborate with Data Architect to execute tasks within agreed framework and with
Collaborates with Product Owner to deliver solutions that avoid unnecessary changes in the future
Gathers requirements, assesses gaps and builds roadmaps and architectures to help the analytics driven organization achieve its goals
Works closely with Data Analysts to ensure data quality and availability for analytical modelling
Explores suitable options, designs, and creates data pipeline (data lake / data warehouses) for specific analytical solutions
Defines ETL / ELT based on jointly defined requirements
Orchestrates pipeline and queries with sequence of imports
Identifies gaps and implements solutions for data security, quality and automation of processes
Support maintenance, bug fixing and performance analysis along data pipeline based on a TCO approach
Collaborates with business and understands UC application of data
Technical Skills
BS/MS degree in Computer Science, Engineering, background in Mathematics and Statistics is good to have
At least 10 years of experience in Data Engineering Domain.
Ability to write clean code and documentation, optimize code, and reduce infrastructure cost
Data manipulation and munging in Python, incl. at scale e.g., PySpark or a similar language
Experience on cloud infrastructure (e.g., Microsoft Azure, Databricks)
Experience using SQL, PL/SQL or T-SQL with RDBMSs like Teradata, MS SQL Server, Oracle etc in production environments is a plus.
Experience in data engineering data bases/warehouses
Experience on Big Data platforms (e.g. Hadoop, Map/Reduce, Spark, HBase, CouchDB, Hive) preferred
Experience with BI and data visualization packages is a plus
Comfort with shell scripting (e.g., Bash) and Python (or similar) scripting
Business acumen and ability to communicate technical matters to non-experts
Managerial Skills
Industry and domain expert coaching junior colleagues, demonstrates achieving business value by exceeding stakeholder expectations with delivered products, deep understanding of agile workflows.
Owns full responsibility and accountability for key decisions, e.g., about decisions with regards to product roadmap periodization.