Tagcor

Data Engineer

Design and implement data pipelines using Pyspark Notebooks.
Manage and optimize data storage and processing through the Medallion Architecture (Bronze, Silver, Gold layers).
Develop and maintain ETL processes using notebooks to ingest, transform, and load data from various sources.
Integrate APIs to facilitate data exchange between systems.
Create and manage table views to surface data to end users.
Ensure data quality, consistency, and security across all layers.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
Monitor and troubleshoot data pipelines to ensure reliability and performance.
Stay up to date with the latest Azure technologies and best practices.