Perform structured evaluations of AI assistants using MCP and plugins: design realistic prompts, execute multi-step personal workflows (screen-recorded), assess outputs for usefulness and safety, identify failures, and write detailed reports and rubrics. Remote, US-based contractor role requiring 20+ hours/week and rapid task turnaround.
This role is for one of our clients
Compensation: $50-$190 per hour
Join a pioneering AI research initiative focused on evaluating and improving how advanced AI systems perform complex, real-world personal workflows. We are seeking experienced AI power users with extensive hands-on experience using Model Context Protocol (MCP) and AI plugins/connectors to complete multi-step personal tasks that require planning, judgment, context awareness, and tool integration.
This role is ideal for individuals who regularly rely on AI in their personal lives and understand how connected tools can enhance productivity across a wide range of everyday scenarios.
RequirementsKey Responsibilities
- Design realistic prompts that reflect complex, real-world personal workflows.
- Execute multi-step tasks using AI while recording your screen as part of the evaluation process.
- Utilize personal AI plugins and connectors throughout workflow execution.
- Assess AI-generated outputs for practicality, personalization, reasoning quality, and overall usefulness.
- Identify where AI systems fail to understand context, misuse connected tools, overlook constraints, or produce unrealistic recommendations.
- Write detailed, structured explanations highlighting AI strengths, weaknesses, and improvement opportunities.
- Develop and apply comprehensive evaluation rubrics to measure AI performance consistently across different task types.
Your evaluations may include workflows involving:
- Personal health and wellness
- Travel planning and itinerary management
- Food, dining, and activity planning
- Home services and maintenance
- Career exploration and job search
- Productivity and personal organization
- Other complex day-to-day life management tasks
- Must be based in the United States.
- Strong hands-on experience with Model Context Protocol (MCP) and AI plugin/connector ecosystems.
- Frequent use of AI plugins/connectors such as Google Drive, Expedia, Notion, and similar productivity platforms.
- Heavy personal use of Large Language Models with an established usage history of at least 6 months.
- Experience using AI for multi-step planning, research, decision-making, and everyday personal workflows.
- Strong analytical thinking, written communication, and attention to detail.
- Ability to evaluate AI outputs for accuracy, usefulness, personalization, safety, and reasoning quality.
- Willingness to sign a data-sharing consent agreement through DocuSign.
- Experience designing, applying, or reviewing structured evaluation rubrics.
- Extensive rubric development and evaluation experience, including 100+ hours working on AI evaluation or quality assessment projects.
- Deep familiarity with AI-assisted productivity workflows involving multiple connected applications.
- Experience identifying AI limitations and recommending improvements based on real-world usage scenarios.
- Help shape the future of intelligent AI assistants capable of supporting real-world personal workflows.
- Influence the development of AI systems that better understand context, preferences, trade-offs, and practical decision-making.
- Work on meaningful evaluations spanning health, travel, productivity, career planning, and everyday life management.
- Collaborate on cutting-edge AI initiatives with opportunities for continued engagement based on performance.
- Expected commitment: 20+ hours per week.
- Initial onboarding and ramp-up period of approximately 1–2 days.
- Ability to complete assigned work within 24 hours of task assignment.
- Desktop or laptop computer required (Chromebooks are not supported).
- Selected contributors will begin with a trial period to evaluate quality, consistency, and overall project fit.
- As this project is in its early stages, there may be a short delay before initial task assignments become available.
We are committed to providing equal opportunities to all qualified applicants. Reasonable accommodations are available throughout the application and engagement process.
Contract & Engagement Details- Independent contractor engagement.
- Fully remote with flexible working hours.
- Project timelines may be extended, shortened, or concluded based on project requirements and individual performance.
- Work does not require access to confidential or proprietary information from any current or former employer.
- Payments are issued weekly based on approved work completed through supported payment platforms.
- At this time, we are unable to support H1-B or STEM OPT candidates.
Similar Jobs
Artificial Intelligence • Fintech • Insurance • Marketing Tech • Software • Analytics
Handle inbound calls and warm leads to understand customers’ insurance needs, recommend appropriate coverages, and convert prospects into policyholders. The role includes paid training and licensing support, requires a Property and Casualty license after hire, and involves remote work on a fixed weekday and weekend schedule. Representatives must deliver persuasive, customer-focused service while meeting sales goals.
Top Skills:
PcWired High-Speed Internet
Beauty • Robotics • Design • Appliances • Manufacturing
Own global compensation cycle administration, system configuration, benchmarking, job architecture, salary structures, pay reporting, equity support, and compensation analysis across the Americas, EMEA, and APAC. Partner with HR, recruiting, payroll, legal, finance, and managers on offers, promotions, pay equity, and transparency requirements. Improve compensation operations through AI and automation, maintain data integrity, and deliver training and communications on compensation programs.
Top Skills:
Artificial IntelligenceAutomationBusiness IntelligenceFidelity PswGoogle SheetsMercerExcelOracle HcmPaveRadford
Beauty • Robotics • Design • Appliances • Manufacturing
Own end-to-end data products, including source onboarding, API and file-feed ingestion, orchestration, incremental and idempotent pipelines, Snowflake and dbt Cloud modeling, testing, documentation, alerting, and production reliability. Establish engineering standards, manage secrets and governance, use AI coding tools responsibly, collaborate with vendors and stakeholders, and mentor junior engineers.
Top Skills:
AirflowAmazon S3DagsterDbt CloudGitJinjaPythonRest ApisSftpSnowflakeSQL
What you need to know about the Los Angeles Tech Scene
Los Angeles is a global leader in entertainment, so it’s no surprise that many of the biggest players in streaming, digital media and game development call the city home. But the city boasts plenty of non-entertainment innovation as well, with tech companies spanning verticals like AI, fintech, e-commerce and biotech. With major universities like Caltech, UCLA, USC and the nearby UC Irvine, the city has a steady supply of top-flight tech and engineering talent — not counting the graduates flocking to Los Angeles from across the world to enjoy its beaches, culture and year-round temperate climate.
Key Facts About Los Angeles Tech
- Number of Tech Workers: 375,800; 5.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Snap, Netflix, SpaceX, Disney, Google
- Key Industries: Artificial intelligence, adtech, media, software, game development
- Funding Landscape: $11.6 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Strong Ventures, Fifth Wall, Upfront Ventures, Mucker Capital, Kittyhawk Ventures
- Research Centers and Universities: California Institute of Technology, UCLA, University of Southern California, UC Irvine, Pepperdine, California Institute for Immunology and Immunotherapy, Center for Quantum Science and Engineering
