Easy Apply
Easy Apply
Design and run evaluations for LLM-powered features: build datasets and rubrics, analyze production logs for failures, run experiments, track product/operational metrics, and partner with engineers to productionize improvements and monitoring dashboards.
Join Tech @ Ro to build the future of healthcare, from the ground up!
At Ro, we believe that when people achieve their health goals, they can achieve their life goals. The highest-leverage way to move society forward is to give people their health, and the current healthcare system isn’t built to do that. It was built to bill, not to serve patients.
We’re building a new system. One where the patient is in control. One designed from scratch for the digital age.
At Ro, technology isn’t just a function… It's core to how we deliver care. We’ve built a vertically integrated healthcare platform that connects telehealth, diagnostics, pharmacy, and logistics into a seamless, end-to-end experience for millions of patients.
…and we’re just getting started.
As part of Tech @ Ro, you’ll work on systems that operate at scale, with an opportunity to:
- Solve complex, high-concurrency problems across a full-stack platform
- Build and ship quickly with tight feedback loops and real-world impact
- Own systems end-to-end, from architecture to production performance
- Work alongside experienced operators, technical leaders, and clinicians
- Help define how modern healthcare should be delivered
We’re a performance-driven team with a strong sense of ownership and urgency. We move fast, learn quickly, and hold a high bar for what we build, and do so with a big heart — because patients depend on it.
If you’re motivated by impact, scale, and the chance to help lead the patient revolution, come build with us.
Ro is building a team focused on shipping LLM-powered products across the patient experience, clinical operations, and internal tooling.
We're hiring an Applied AI Scientist to help measure, evaluate, and improve our AI systems. You'll answer one of the most important questions in applied AI: "Is this actually working?" You'll design evaluations, analyze production behavior, run experiments, and partner closely with engineers to improve the quality of AI-powered features.
You'll work on real production systems while learning modern evaluation techniques from experienced teammates.
What You'll Do
- Build evaluation datasets, rubrics, and synthetic test cases for LLM-powered features across patient and internal workflows.
- Analyze production logs to identify model failures, hallucinations, quality issues, and operational bottlenecks.
- Design and run experiments end-to-end, including hypothesis development, dataset creation, evaluation, analysis, and recommendations.
- Track key product and operational metrics including resolution rate, handle time, touches to resolution, latency, and quality and identify opportunities for improvement.
- Partner with engineers to validate improvements and productionize successful experiments.
- Help build or integrate tooling and dashboards that make AI performance easy to understand and monitor.
Who You Are
- 1–4 years in data science, analytics, applied ML, or a closely adjacent role.
- Strong Python and SQL skills.
- Hands-on experience building or evaluating LLM-powered applications through work, research, school, hackathons, or side projects.
- You’re curious about how AI systems work beyond the model itself, including retrieval, prompting, evaluation, and production behavior.
- You’re comfortable communicating analytical findings clearly to engineers, product managers, and operational stakeholders.
- You are excited about learning rapidly in a field where best practices continue to evolve.
- Bonus: Experience with evaluation tooling, A/B testing frameworks, production model monitoring, healthcare, or other operations-heavy environments.
A note on reporting structure
This is a new function at Ro, and we're being deliberate about not over-defining it. Your manager and where you sit on the org chart will depend on the specific shape of the team we end up with. We'd rather find the right people and figure out the lines around them than pre-draw boxes and miss great candidates. If that ambiguity is a deal-breaker, this isn't the right role; if it sounds like an opportunity, we want to talk.
-
The target base salary for this position ranges from $149,600 - $184,000, in addition to a competitive equity and benefits package (as applicable). When determining compensation, we analyze and carefully consider several factors, including location, job-related knowledge, skills and experience. These considerations may cause your compensation to vary.
Ro is consistently recognized as a top workplace in Health Care, in New York, and for Women and Parents—earning more than 20 honors from Fortune, Great Place to Work, and PEOPLE since 2021. In 2025 alone, we ranked top 5 among medium workplaces in Health Care and New York, and top 50 nationwide.
At Ro, we believe that our diverse perspectives are our biggest strengths — and that embracing them will create real change in healthcare. As an equal opportunity employer, we provide equal opportunity in all aspects of employment, including recruiting, hiring, compensation, training and promotion, termination, and any other terms and conditions of employment without regard to race, ethnicity, color, religion, sex, sexual orientation, gender identity, gender expression, familial status, age, disability and/or any other legally protected classification protected by federal, state, or local law.
Ro is committed to providing reasonable accommodations for qualified individuals with disabilities in our application and interview process. If you require a reasonable accommodation in the application or interview process, please contact us at [email protected].
See our California Privacy Policy here.
Similar Jobs at Ro (Ro.co)
.png)
Healthtech • Pharmaceutical • Telehealth
Own evaluation, measurement, and optimization of production LLM-powered features. Design reproducible evaluation frameworks, run experiments and analyses to identify failure modes and regressions, build metrics and dashboards, partner with engineering to productionize improvements, and mentor teammates on experimental design and measurement best practices.
Top Skills:
BraintrustCausal InferenceExperimentation PlatformsLangsmithLlmsOpenai EvalsPythonSQL
Healthtech • Pharmaceutical • Telehealth
Build and ship LLM-powered features across patient and clinical workflows. Partner with clinical operators, prototype AI workflows (prompts, agents, RAG), write Python/SQL to connect AI systems, define success metrics, run evaluations, and iterate based on user feedback.
Top Skills:
Agent WorkflowsEhr SystemsLarge Language Models (Llms)Orchestration FrameworksPromptingPythonRetrieval-Augmented Generation (Rag)SQL
Healthtech • Pharmaceutical • Telehealth
Embed with clinical and operations teams to identify high-value AI opportunities; build LLM-powered features end-to-end including prompts, orchestration, Python services, evaluation suites, golden datasets, and metrics-driven experiments to measure product impact and prioritize work.
Top Skills:
APIsEvaluation SuitesLlmsModel OrchestrationPrompt EngineeringPython
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
