Design and implement scalable data pipelines using Databricks or AWS services such as Glue, Redshift, S3, Lambda, EMR, Athena Develop and maintain ETL processes to transform and integrate data from multiple sources.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
Optimize and tune performance of data pipelines and queries
Ensure data quality and integrity through robust testing and validation processes.
Implement data security and compliance best practices.
Monitor and troubleshoot data pipeline issues and ensure timely resolution
Stay updated with the latest developments in AWS Data engineering technologies and best practices.