Senior QA Automation Engineer – AI Applications

Senior QA Automation Engineer – AI Applications 

We are seeking an experienced QA Automation Engineer to ensure the quality, reliability, and performance of AI-powered applications. The ideal candidate has expertise in automated testing and software quality engineering, with experience building scalable testing frameworks and leveraging AI-powered testing tools to improve test development, maintenance, and coverage. They combine deep QA automation expertise with practical experience using AI to enhance testing workflows. Experience testing AI/ML applications or validating machine learning models is an asset. 

Responsibilities 

• Design and maintain automated test suites, evaluation scenarios, metrics, and test datasets for AI-powered features. 

• Automate QA workflows using Python and CI/CD (e.g., GitHub Actions, Jenkins), leveraging AI-powered testing tools to improve test development, maintenance, and coverage. 

• Create and maintain test harnesses for AI services, applications, and APIs. 

• Perform API testing for AI services and backend systems. 

• Define quality strategy and acceptance criteria for AI-powered applications across functional, API, integration, and AI/ML features. 

• Test AI/ML applications and machine learning models from a product and user perspective. 

• Evaluate AI/ML model outputs for correctness, consistency, and expected application behavior. 

• Define structured rubrics that translate qualitative findings into measurable evaluation gates. 

• Build quality gates for CI/CD and deployment pipelines, including automated regression and integration testing. 

• Mentor teams on QA automation best practices and consistent quality standards for AI-powered applications. 

• Contribute to multi-team projects and codebases, ensuring code quality and consistency. 

• Participate in code reviews and provide constructive feedback to peers. 

• Document and present findings and ideas across the company. 

Minimum Qualifications 

• Bachelor’s degree in Computer Science, Engineering, or equivalent experience. 

• 5+ years of professional experience in software quality assurance, test automation, or software engineering. 

• Strong programming skills in Python, with experience in test automation. 

• Hands-on experience using AI-powered testing tools or AI coding assistants to improve test automation, productivity, and quality. 

• Experience with CI/CD tools and processes. 

• Familiarity with popular CAD environments and tooling. 

• Proficient in automation and UAT test suites/frameworks. 

• Experience designing QA frameworks or platforms used by multiple teams. 

• Excellent problem-solving skills and attention to detail. 

• Strong communication and collaboration skills. 

• Understanding of software architecture and design patterns. 

• Ability to work in an Agile development environment. 

Preferred Qualifications 

• Experience testing AI/ML applications or validating machine learning models. 

• Experience with data validation tooling (e.g., Great Expectations) or labeling workflows. 

• Familiarity with ML frameworks (e.g., PyTorch, TensorFlow). 

• Experience with data pipelines and orchestration tools (e.g., Airflow, Metaflow). 

• Knowledge of MLOps practices (e.g., model monitoring, drift detection, deployment validation). 

• Knowledge of ML evaluation methods, metrics, and benchmarking. 

• Passion for learning new technologies and improving existing systems. 

• Experience with cloud providers (e.g., AWS, Azure, Google Cloud Platform). 

• Experience testing AI/ML services in production environments. 

• Knowledge of experiment tracking tools (e.g., Comet, MLflow, Weights & Biases). 

The Ideal Candidate 

• You demonstrate initiative to provide solutions and to learn and develop new technologies. 

• Comfortable building QA systems from scratch and writing maintainable automation. 

• You enjoy learning and collaborating across global locations. 

• You are comfortable working in newly forming ambiguous areas. 

• You are comfortable building scalable and maintainable QA automation frameworks that will be relied on by others. 

• You effectively leverage AI-powered tools to improve testing workflows and engineering productivity. 

• You can communicate well with others.