Data Engineer who will primarily focus on building data pipelines. They will be expected to leverage a variety of advanced tools and technologies such as Kafka/Kinesis for real-time data processing/streaming, Relational/No-SQL databases for robust data storage and management, and other Integration tools for seamless data flow across various cloud and on-premises platforms. The Data Engineer will also utilize ETL processes to extract data from various sources, transform the data to fit operational and business needs, and load it into an end target. In addition to these, they will be expected to have familiarity with Pub-Sub messaging patterns or similar data dissemination models to ensure efficient data distribution and consumption. One of the primary goals is to create a real-time bidirectional data pipeline from Oracle transactional databases to a data lake in the cloud.
Work Products and Outcomes:
The Data Engineer shall meet the following key high-level work products and outcomes as identified by LADBS: (Note: this list is not exhaustive.)
- Develop, construct, test, and maintain data architectures and pipelines.
- Create best-practice ETL frameworks; repeatable and reliable data pipelines that convert data into powerful signals and features.
- Handle raw data (structured, unstructured, and semi-structured) and align it into a more usable, structured format that is better suited for reporting and analytics.
- Work with the cloud solutions architect to ensure data solutions are aligned with company platform architecture and all aspects related to infrastructure.
- Collaborate with business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.
- Ensure data pipeline architecture will support the requirements of the business.
- Document processes and perform periodic system reviews to ensure adherence to established standards and processes.
- Evaluate and advise on technical aspects of open work requests in the product backlog with the project lead.
- Define Cloud infrastructure Reference Architectures for common solution archetypes
- Real-time bidirectional data pipeline from Oracle transactional databases to a data lake in the cloud.
- Clear, comprehensive documentation related to the data pipeline.
- Regular reports on the status of data pipeline development.
- Other related tasks identified by LADBS
Performance Specifications:
The qualified candidate must possess the following skills and experience in the following areas:
- A bachelor's degree in Computer Science, Data Science, Software/Computer Engineering, or a related field.
- Proven experience as a data engineer or in a similar role, with a track record of manipulating, processing, and extracting value from large disconnected datasets.
- Demonstrated technical proficiency with data architecture, databases, and processing large data sets.
- Proficient in Oracle databases and comprehensive understanding of ETL processes, including creating and implementing custom ETL processes.
- Experience with cloud services (AWS, Azure), and understanding of distributed systems, such as Hadoop/MapReduce, Spark, or equivalent technologies.
- Knowledge of Kafka, Kinesis, OCI Data Integration, Azure Service Bus or similar technologies for real-time data processing and streaming.
- Experience designing, building, and maintaining data processing systems, as well as experience working with either a MapReduce or an MPP system.
- Strong organizational, critical thinking, and problem-solving skills, with clear understanding of high-performance algorithms and Python scripting.
- Experience with machine learning toolkits, data ingestion technologies, data preparation technologies, and data visualization tools is a plus.
- Excellent communication and collaboration abilities, with the ability to work in a dynamic, team-oriented environment and adapt to changes in a fast-paced work environment.
- Data-driven mindset, with the ability to translate business requirements into data solutions.
- Experience with version control systems e.g. Git, and with agile methodologies/scrum.
- Certifications in related field would be an added advantage (e.g. Google Certified Professional Data Engineer, AWS Certified Big Data, etc.).
Evaluation Criteria:
LADBS will review the TOS Responses received and select up to 5 of the most qualified candidates to be interviewed based on a review of the resumes provided and the criteria below.
- Education
- Relevant degree in Computer Science, Engineering, Information Technology, or related field
- Advanced degrees or certifications related to data engineering
- Experience
- Previous work experience with data migration and engineering
- Hands-on experience with data warehouses
- Demonstrated experience in managing and optimizing data pipelines and architectures
- Technical Knowledge
- Strong understanding of streaming data platforms and pub-sub models
- In-depth knowledge of data warehousing concepts, including data storage, retrieval, and pipeline optimization