Lead AI/ML Engineer

HIGH LEVEL DUTIES

• Own the design, development, and adoption of cloud-based tooling and frameworks for AI-driven workflows, focusing on LLMs and agentic frameworks, ensuring scalability, reproducibility, and usability.  
• Collaborate closely with the Lead Data Platform Engineer to integrate and deploy AI solutions into the DataHub platform, ensuring seamless operationalization.  
• Partner with the Cloud Data Architect to define and implement best practices, standards, and governance for AI model development, particularly for LLMs and agentic systems.  
• Drive the adoption of standardized tooling for model development, testing, and monitoring by data scientists across the organization.  
• Lead the implementation of MLOps and AIOps practices for AI model lifecycle management, working with the Lead Data Platform Engineer to automate deployment pipelines.  
• Promote best practices for AI development, including prompt engineering, model interpretability, responsible AI, and agentic framework design, in alignment with architectural standards.  
• Provide technical mentorship to data scientists, fostering expertise in AI tooling and adoption of standardized frameworks.  
• Ensure governance and compliance for AI models, collaborating with the Cloud Data Architect to address fairness, transparency, and regulatory requirements.  
• Monitor and optimize the adoption and performance of AI tooling, ensuring alignment with business needs and platform reliability.

SPECIFIC DUTIES

• Own the development of a cloud-native MLOps/AIOps framework for training, fine-tuning, validating, and monitoring AI models, including LLMs and agentic frameworks, ensuring ease of adoption by data scientists.  
• Build and maintain reusable AI libraries and templates for common use cases (e.g., prompt engineering, LLM fine-tuning, Retrieval-Augmented Generation (RAG), agentic task orchestration).  
• Drive adoption of best practices for LLM experimentation (e.g., prompt optimization, context window management) and agentic system design, in collaboration with the Cloud Data Architect.  
• Create tooling for explainability and interpretability of LLMs and agentic systems, aligning with architectural standards for stakeholder trust and compliance.  
• Collaborate with the Lead Data Platform Engineer to automate AI model deployment pipelines using cloud-native tools (e.g., Azure ML, Databricks MLflow, Hugging Face).  
• Develop monitoring solutions to track model performance, drift, and bias, working with the Lead Data Platform Engineer to integrate with DataHub monitoring systems.  
• Partner with the Cloud Data Architect to establish AI governance frameworks, including model documentation, versioning, and responsible AI policies for LLMs and agentic systems.  
• Facilitate integration of AI workflows with the DataHub, collaborating with the Lead Data Platform Engineer to ensure seamless access to curated data domains for LLM training and agentic system inputs.  
• Work with the Lead Data Platform Engineer to optimize data pipelines for AI use cases, including vector databases and embeddings for LLMs and RAG systems.  
• Own the development and adoption of agentic frameworks for autonomous task execution and multi-agent collaboration, ensuring alignment with platform deployment capabilities.  
• Document and promote best practices for cloud-based AI tools (e.g., Jupyter, PySpark,  LlamaIndex, MCP, A2A), in collaboration with the Cloud Data Architect.  
• Lead the adoption of advanced AI techniques (e.g., few-shot learning, chain-of-thought prompting, multi-modal LLMs), ensuring compatibility with platform infrastructure. 

ROLE REQUIREMENTS

• 7-10 years of experience in AI/ML engineering, with at least 3 years in cloud-based AI environments and hands-on work with LLMs and agentic frameworks.  
• 3+ years of experience with MLOps/AIOps practices, focusing on tooling for model development and monitoring.  
• 5+ years of proficiency in Python (e.g., Pandas, Scikit-learn, TensorFlow, PyTorch, Hugging Face Transformers) and SQL.  
• 3+ years of experience with cloud platforms (Azure Databricks or similar platform preferred) 
• Strong experience with PySpark for large-scale data processing, feature engineering, and embeddings for LLMs.  
• 3+ years of experience with CI/CD pipelines and version control (e.g., Git, Azure DevOps) for AI tooling integration.  
• Deep understanding of machine learning, generative AI, LLMs, and agentic frameworks, including techniques like fine-tuning, RAG, and multi-agent systems.  
• Hands-on experience with LLM frameworks (e.g., LangChain, LlamaIndex, AutoGen) and prompt engineering for tasks like text generation, summarization, and reasoning.  
• Experience building AI tooling for production environments, collaborating on containerization (e.g., Docker, Kubernetes) with platform engineers.  
• Strong knowledge of AI governance, responsible AI, and ethical considerations for LLMs (e.g., bias mitigation, hallucination control), aligned with architectural standards.  
• Excellent communication and collaboration skills to work with the Lead Data Platform Engineer, Cloud Data Architect, and cross-functional stakeholders.  
• Proven ability to mentor data scientists and drive adoption of AI tooling and frameworks.  
• Experience with big data tools (e.g., Hadoop, Spark, Kafka) and data orchestration platforms (e.g., Airflow, Databricks Workflows).  
• Familiarity with vector databases (e.g., Pinecone, Weaviate) and infrastructure as code for AI workloads is a plus.