Support the standardization, customization, and ad-hoc data analysis, and will develop the mechanisms to ingest, analyze, validate, normalize, and clean data. Develop and maintain data engineering best practices and contribute to Insights on data analytics and visualization concepts, methods, and techniques.
- Design and develop data applications using selected tools and frameworks as required and requested.
- Collaborate with Engineers and Business to build intuitive products that address the needs of our customers.
- Read, extract, transform, stage and load data to selected tools and frameworks as required and requested.
- Gather and process raw data at scale.
- Perform tasks such as writing scripts, web scraping, calling-writing APIs, write SQL queries, etc.
- Work closely with the engineering team to integrate your work into our production systems.
- Process unstructured data into a form suitable for analysis. Analyze processed data.
- Participate in design and code reviews to ensure the quality of the products we deliver.
- Establish a good working rapport with Development group, System Engineers, Quality Assurance/Testers, and Business users.
- Develop scripts or workflows to facilitate complex ingress processes
Must have
- Python: Boto3, Multiprocessing, Classes
- Java Script (Snowflake SP)
- AWS: Lambda, Triggers, EC2, S3, CloudWatch, Security.
- Snowflake: SQL & Stored Procedures
- Airflow: Celery, MQRabbit, Postgress
- Unix-Linux: Shell Scripting, admin experience
- Experience building APIs
Nice to have