Tagcor

GCP SQL Python Data Pipeline Development

Data Pipeline Development: Design and implement ETL/ELT pipelines using Cloud Dataflow, Apache Beam, Cloud Composer (Airflow), and Pub/Sub.
Database & Data Warehousing: Develop and optimize queries for BigQuery, Cloud SQL (PostgreSQL, MySQL), and Firestore.
Data Processing & Transformation: Write efficient Python and SQL scripts to transform and clean data for analytics and reporting.
Workflow Orchestration: Automate and schedule data workflows using Cloud Composer (Apache Airflow).
Real-Time & Batch Processing: Implement real-time data streaming with Pub/Sub, Dataflow, and Kafka while supporting batch processing pipelines.
Performance Optimization: Tune SQL queries, optimize data storage, and improve pipeline efficiency to reduce latency and cost.
Security & Compliance: Ensure data governance, security, and compliance with best practices in IAM, VPC, encryption, and auditing.
Monitoring & Troubleshooting: Implement monitoring, logging, and alerting using Stackdriver, Cloud Logging, and Cloud Monitoring.
Collaboration: Work closely with data analysts, data scientists, and software engineers to integrate data solutions into business applications.