Tests, evaluates, and optimizes enterprise generative AI applications, agents, workflows, prompts, models, and architectures. Builds automated evaluation pipelines, benchmarks, regression tests, monitoring, telemetry, and governance controls. Develops multi-agent workflows using LangGraph and LangChain, evaluates model versions, and supports documentation, traceability, model registries, and experiment tracking. Collaborates with engineering, data science, security, compliance, governance, and business stakeholders.
Care Lumen | Remote (US Only)
Employment Type: Contractor (3-6 months, with possibility of extension)
Salary Range: $129 to $150 an hour
Scope
The Senior Generative AI Engineer is an individual contributor responsible for testing, evaluating, optimizing, and monitoring enterprise Generative AI systems. The primary focus is improving the quality, reliability, and performance of existing GenAI solutions through systematic testing, evaluation of new model versions, prompt optimization, and application architecture improvements. The role will also help drive evaluation methodology, monitoring capabilities, and AI governance requirements for selected initiatives. Approximately 25% of the role is expected to support AI governance activities, including model documentation, versioning, registry management, experiment tracking, auditability, and related controls.
The role collaborates with engineering, data science, architecture, security, compliance, governance, and business stakeholders. It does not have direct reports or a formal leadership mandate.
What You’ll Do
- Test and evaluate GenAI applications, AI agents, and LLM-powered workflows for quality, reliability, accuracy, safety, and task effectiveness.
- Develop and execute repeatable GenAI evaluation methodologies, test datasets, benchmark suites, regression tests, and automated evaluation pipelines.
- Use DeepEval and related evaluation approaches to measure GenAI output quality and identify regressions across prompts, workflows, application changes, and model versions.
- Optimize GenAI solution architecture by decomposing complex prompts and workflows into appropriate subtasks, tools, and specialized agents.
- Build and improve multi-agent workflows using orchestration frameworks such as LangGraph and LangChain.
- Optimize prompts, system instructions, context management, agent interactions, and model selection / configurations using evaluation results (including latency and model performance) and production feedback (including user feedback).
- Evaluate new foundation models and model versions to determine quality, behavior, risks, performance, (groundedness, completeness, bias, etc) and suitability for specific use cases.
- Design and implement monitoring capabilities for GenAI systems, including quality metrics, operational telemetry, audit trails, alerts, and performance reporting.
- Help drive AI governance activities for selected initiatives, including model and application documentation, risk assessments, evaluation evidence, model versioning, registry management, experiment tracking, and change traceability.
- Collaborate across technical, governance, security, compliance, and business teams to improve GenAI solutions and support responsible production use.
Competencies
- GenAI testing and quality evaluation
- GenAI evaluation methodology and automated evaluation
- Prompt, context, and application architecture optimization
- Multi-agent architecture and orchestration
- Foundation model and model-version testing
- AI governance and model lifecycle management
- GenAI monitoring, observability, and traceability
- Traditional machine learning testing and validation
- Analytical problem-solving and cross-functional collaboration
What’s Required
- 10+ years of overall professional experience in data science, machine learning, AI engineering, or a related technical field.
- 2+ years of hands-on experience developing, testing, evaluating, or optimizing Generative AI systems.
- Demonstrated experience testing GenAI applications and establishing systematic methods for assessing LLM output quality.
- Hands-on experience with DeepEval, LangGraph, LangChain, Python, and modern AI and machine learning libraries.
- Experience designing or optimizing multi-agent systems, agent orchestration patterns, or complex LLM workflows.
- Strong hands-on experience with Databricks for data, AI, or machine learning workloads, including model versioning, model registries, and experiment tracking.
- Practical experience supporting AI governance, model lifecycle controls, documentation, traceability, and auditability.
- Prior experience on traditional machine learning projects as an ML engineer, data scientist, or similar contributor, with a strong understanding of model testing and validation methods.
- Ability to communicate technical findings, evaluation results, risks, and recommendations to technical and nontechnical stakeholders.
- Ability to maintain at least four hours of overlap with U.S. Eastern Time (EST/EDT) each workday.
- Bachelor's or advanced degree in computer science, data science, engineering, mathematics, statistics, physics, computational science, or another scientific, quantitative, or computational discipline.
What’s Preferred
- Experience with Databricks Mosaic AI.
- Experience working with PHI / PII data inside highly regulated domain like healthcare.
- Experience testing or optimizing multi-agent GenAI systems in production or production-like environments.
- Experience working with embedding models, semantic retrieval, vector search, or Retrieval-Augmented Generation architectures.
- Experience applying AI governance or Responsible AI requirements in a regulated enterprise environment.
Compensation & Benefits
The expected total cash compensation for this role is market-driven, with an expected compensation rate of $129 to $150 per hour. Hourly rate is ultimately determined based on a candidate's skills, expertise, and experience.
Offers are contingent upon successful completion of a background screening. Background screenings will be conducted in compliance with applicable laws and will not be initiated without your consent.
Care Lumen is an equal opportunity employer committed to building a diverse and inclusive workplace. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, disability status, age, or any other legally protected status.
We are committed to providing reasonable accommodations to qualified individuals with disabilities in our job application process. If you need assistance or an accommodation due to a disability, please contact us at [email protected].
About Care Lumen
Care Lumen is transforming healthcare by connecting people, care, and data—so that every decision leads to better outcomes. We empower care teams with a unified platform that delivers intelligent, real-time insights, streamlined workflows, and stronger collaboration across clinical, operational, and administrative domains.
Our vision is bold: to reimagine how care is delivered by unifying clinical operations and unlocking insights that improve lives. We believe in the power of great people and purposeful technology—and with all of us aligned around a common goal, our potential grows even stronger. We're building for impact today and evolving through innovation tomorrow.
We’re a remote-first company that values flexibility, autonomy, and results over hours logged. We're guided by values that push us to excel, stay curious, and take smart risks. The best solutions come from teams that listen, learn, and build together. And we're focused on outcomes that matter—not just for our clients, but for the people they serve.
Care Lumen’s LinkedIn
Care Lumen’s website
Ready to Apply?
We are committed to providing reasonable accommodations to qualified individuals with disabilities in our job application process. If you need assistance or an accommodation due to a disability, please contact us at [email protected].
About Care Lumen
Care Lumen is transforming healthcare by connecting people, care, and data—so that every decision leads to better outcomes. We empower care teams with a unified platform that delivers intelligent, real-time insights, streamlined workflows, and stronger collaboration across clinical, operational, and administrative domains.
Our vision is bold: to reimagine how care is delivered by unifying clinical operations and unlocking insights that improve lives. We believe in the power of great people and purposeful technology—and with all of us aligned around a common goal, our potential grows even stronger. We're building for impact today and evolving through innovation tomorrow.
We’re a remote-first company that values flexibility, autonomy, and results over hours logged. We're guided by values that push us to excel, stay curious, and take smart risks. The best solutions come from teams that listen, learn, and build together. And we're focused on outcomes that matter—not just for our clients, but for the people they serve.
Care Lumen’s LinkedIn
Care Lumen’s website
Ready to Apply?
About
Newfire Global Partners is a leading custom software development company that specializes in working on complex problems. With a strong focus on meeting the unique needs of each organization, they connect clients with local leaders and trusted remote teams to deliver exceptional IT services and software development solutions. Their range of expertise spans various domains, including Data & Analytics, AI/ML, Software Engineering, Product Management, and Marketing. By leveraging recent innovations in artificial intelligence, Newfire Global Partners ensures that their clients can transform the new normal into a competitive advantage. Through their advisory services, experienced leaders guide businesses through transformations and de-risk operations throughout the entire lifecycle. They also offer talent augmentation, bringing together highly skilled professionals from multi-disciplinary teams to support software, product, and marketing needs. Newfire Global Partners is committed to providing excellent service and driving client growth. Their team of industry experts works closely with organizations, whether it's driving transformation, building from scratch, or getting things done efficiently. Trusted by VC & PE firms, established businesses, and startups, Newfire Global Partners is proud to be a reliable partner in various industries, including healthcare, enterprise, and fintech. Partner with Newfire Global Partners today and unlock the expertise that accelerates success.
Similar Jobs
Beauty • Robotics • Design • Appliances • Manufacturing
Serves as the strategic HR partner for Direct-to-Consumer and Sales teams. Responsibilities include workforce planning, organizational design, talent management, hiring, onboarding, training, performance management, employee engagement, succession planning, change management, and HR metrics analysis. The role advises senior leaders, partners with HR Centers of Excellence, supports organizational transformation, and promotes an inclusive, high-performance culture.
Top Skills:
Ai ToolsClaude
Artificial Intelligence • Big Data • Healthtech • Software • Biotech
Own strategic client relationships across onboarding, adoption, renewals, and expansion. Lead business reviews, value realization plans, account strategies, and cross-functional delivery. Use usage and health metrics to improve retention, identify expansion opportunities, and build business cases. Partner with Sales on upsell opportunities and forecasting while developing executive-level client advocates and feeding customer insights into Product and leadership. Travel 30–40% across the USA.
Top Skills:
BioinformaticsGenomicsNgs WorkflowsSophia Ddm
Healthtech • Software • Analytics • Biotech • Pharmaceutical • Manufacturing
Promotes Takeda pharmaceutical products through scientific customer visits, communicates medical information to healthcare experts, develops target-customer relationships, identifies needs, supports market-share growth, and represents the company professionally. The role requires achievement of sales targets, strong key-account management, execution, learning, and teamwork skills. Oncology promotion experience, particularly in lung cancer, breast cancer, urology, or medical oncology, is preferred.
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
