The Data Engineer and DataOps will contribute to the growth of our Enterprise Data Warehouse and our wider cloud-based data platform. They will deliver excellent data products and adhere to key design principles while following established best practices. The Data Engineer and DataOps will work closely with our architects and infrastructure teams to build secure, scalable solutions for operational, reporting, and analytical data needs. The Data Engineer may also collaborate directly with customers on their mission-critical data workloads and analytical needs.
Roles & Responsibilities :
- Drive data projects through to delivery removing obstacles as needed to ensure success.
- Gain a working understanding of Legal and General America as a business and collaborate with leaders to contribute to cohesive, end-to-end experiences for customers.
- Consult on data projects by analyzing complex end to end data product requirements and existing business processes. Contribute to the development and implementation of data solutions.
- Translate business data stories into a technical story breakdown with work estimates.
- Implement production processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
- Develop and maintain scalable data pipelines for both streaming and batch requirements and build out new API integrations to support continuing increases in data volume and complexity.
- Responsible for producing data building blocks, data models, and data flows for varying client demands such as dimensional data, standard and ad hoc reporting, data feeds, dashboard reporting, and data science research & exploration.
- Identify changes needed to improve customer/partner experience through a data driven approach.
- Maintain knowledge and understanding of corporate security procedures and policies for implementation, auditing, and documentation.
- Attending the DataOps forums (standards, modelling, and change boards) to achieve continued improvements into our delivery methods, standards and documentation.
- Providing mentorship & support to junior & mid-level data engineers in your team & be a role model to colleagues across the Data Ops function.
- Keep up to date with industry trends and apply to projects as appropriate.
- Prepare technical documentation and participate in code reviews to ensure that application code meets internal standards and follows industry best practices.
- Adhering to and promoting compliance with the company’s data management policies and procedures
- Ensuring reports are completed to the specified requirements and the agreed SLAs and ensure the requestor understands any limitations.
Required Skills :
- 10+ years of hands-on experience architecting and implementing complex data solutions.
- 7+ years of experience architecting and implementing large operational data stores.
- 5+ years of experience with Big Data technologies like Apache Spark, Python.
- 3+ years Linux Systems Administrator with expertise in Private, Public and Hybrid Cloud platforms along with using various tools like Docker, Kubernetes, ANSIBLE, Jenkins, GIT, Maven.
- Experience with DevOps for Data preferred.
- Experience with Machine Learning, Natural Language Processing, Deep Learning, and related areas would be beneficial.
- Experience in using SSIS tools like Import and Export Wizard, Package Installation, and SSIS Package Designer.
- Experience in importing/exporting data between different sources like Oracle/Access/Excel etc. using SSIS/DTS utility.
- Expert in analyzing, designing, developing, installing, configuring, and deploying MS SQL Server suite of products with Business Intelligence in SQL Server Reporting Services 2005/2008, SQL Server Analysis Services of 2005/2008, and SQL Server Integration Services.
- Strong verbal and written communication skills with excellent interpersonal communication and collaboration skills.
- Ability to communicate technical details clearly across organizational boundaries.
- Cloud and services experience, with Azure cloud experience highly desirable.
- Excellent analytical skills with systematic and structured approach to solution design
- Experience with or exposure to Agile and interactive development processes.
- Knowledge of Azure SQL database, Databricks, and Apache Spark
- Experience working with Azure Synapse, Azure Data Lake, Azure Analysis Services, Azure Data Factory desirable.
- Experience with Lakehouse architecture desirable.
- Experience with DataOps, Azure ML and Azure Purview desirable.
- Experience with Power BI or Tableau desirable.
- Preferred Data Engineering on Microsoft Azure DP-203