Sr. Data Platform Engineer (REMOTE) needed --- Data Platform Engineering, Master's degree, Data Platform, AZURE Infrastructure, importantly Data bricks, ADLS, PostgresDB, Event Hubs, AKS, Data bricks Admin, Databricks Certified, latest releases (Unity Catalog), Python. Python + SQL, MLOps, CI/CD GitHub, GitHub Actions, No. of Models / Pipelines, Large Language Model (LLM) Agents (for 3 use cases, IGS, HBT, PMT), Pipeline to create / improve evaluation framework for LLMs, Test Model Performance / Accuracy, Perform prompt engineering, Create the pipelines as reusable components, Create a pipeline to fine-tune an open-source model
Qualified candidates will have 6+ years of Data Platform Engineer experience and a college degree. Master's degree + 6 + years of experience.
6+ years of hands-on skills as a Data Platform Engineer.
AZURE Infrastructure experience – importantly Databricks, ADLS, PostgresDB, event hubs, and AKS
Databricks Admin – Certified candidates preferred – ideally in the latest releases that include the Unity Catalog.
Able to code in Python. Python + SQL.
High-level understanding of MLOps.
CI/CD GitHub, GitHub actions etc.
Excellent communication skills both verbal and written are required.
No. of models/pipelines - Large Language Model (LLM) Agents (for 3 use cases, IGS, HBT, PMT) with a set of tools developed for the individual use cases.
Pipeline to create and improve the evaluation framework for the Large Language Model (LLMs) to test model performance and accuracy.
Perform prompt engineering for the existing toolset to fit the use case and evaluate using the above framework.
Create the pipelines as reusable components for additional use cases. the pipeline includes the tools, agents, base prompts, evaluation framework, etc
Create a pipeline to fine-tune an open-source model based on the feedback received from the beta releases of the app.