Senior Automation QA Engineer

Senior Automation QA Engineer (ASU Sunny) 

Location: Calgary, Canada (Remote-Friendly) 

Robots & Pencils is seeking a Senior Automation QA Engineer to lead automated quality assurance efforts for an AI-powered chat platform that helps students get instant, intuitive answers across a wide range of topics. 

Our client currently operates an existing chatbot that is experiencing critical issues. This project will rebuild and modernize the chat experience, leveraging large language model (LLM) capabilities to deliver natural, conversational interactions that meet modern user expectations. You’ll play a key role in ensuring this next-generation platform achieves exceptional reliability, performance, and user satisfaction. 

As a senior member of the engineering team, you’ll take ownership of the end-to-end automation strategy and execution, from framework design and test coverage planning to implementation, integration, and continuous improvement. You’ll collaborate closely with engineers, designers, and product managers to deliver intelligent, high-quality, and seamless user experiences. 

Key Responsibilities 

Automation Strategy & Framework Development 

• Define and implement comprehensive automation strategies, frameworks, and coding standards. 

• Design, develop, and maintain automated test suites for web, API, and backend components. 

• Translate complex user flows and LLM-based conversational logic into automated validation scripts. 

• Integrate automation into CI/CD pipelines to support continuous testing and fast feedback loops. 

Testing & Reporting 

• Execute automated test runs across multiple environments and platforms. 

• Monitor, debug, and maintain automated test results, ensuring stability and reliability. 

• Identify, document, and track software defects through resolution in collaboration with development teams. 

• Deliver clear automation metrics and quality reports to stakeholders, including test coverage and release readiness. 

Engineering Collaboration 

• Partner with developers and product managers to ensure quality is embedded at every stage of the SDLC. 

• Participate in sprint planning, backlog grooming, and code reviews to align automation efforts with release goals. 

• Collaborate on test data setup, mocking strategies, and CI/CD optimization for faster deployments. 

• Advocate for testability, maintainability, and performance in architectural discussions. 

Process & Tooling Excellence 

• Evaluate and adopt modern automation tools (e.g., Playwright, Cypress, Selenium, or equivalent). 

• Implement API testing using tools such as Postman, REST Assured, or similar frameworks. 

• Support performance testing (e.g., JMeter, k6) and monitor system behavior under load. 

• Ensure adherence to accessibility, security, and performance testing standards. 

Team Leadership & Knowledge Sharing 

• Mentor QA engineers in automation best practices, code quality, and test design principles. 

• Collaborate with distributed teams to maintain consistency in automation coverage and practices. 

• Champion a culture of continuous improvement, emphasizing reliability, observability, and proactive testing. 

Required Skills & Qualifications 

• 5+ years of experience in software quality assurance, with a strong focus on test automation. 

• Proficiency in JavaScript/TypeScript or Python for building and maintaining test automation frameworks. 

• Experience with UI and API automation tools such as Playwright, Cypress, Selenium, or REST Assured. 

• Solid understanding of CI/CD pipelines and integration of automated tests (GitHub Actions, GitLab CI, Jenkins, etc.). 

• Strong knowledge of QA methodologies, SDLC, and Agile processes. 

• Familiarity with web application architecture and REST/GraphQL API testing. 

• Upper-Intermediate English proficiency (written and spoken) for effective collaboration. 

Nice to Have 

• Experience testing AI-powered or LLM-based chat applications. 

• Familiarity with performance and load testing tools (k6, JMeter). 

• Experience working with cloud environments (AWS, GCP, Azure) and containerized systems (Docker, Kubernetes). 

• Knowledge of data validation, logging, and observability tools (Datadog, Kibana, Sentry). 

• Contributions to automation frameworks, open-source tools, or QA process documentation. 

Personal Competencies 

• Accountability – Owns the automation quality process from framework to release validation. 

• Technical Craftsmanship – Writes efficient, maintainable, and scalable automation code. 

• Collaboration – Builds trust and alignment with engineers and product teams. 

• Analytical Thinking – Solves complex test and infrastructure challenges through data-driven insights. 

• Adaptability – Excels in fast-paced, evolving projects that leverage cutting-edge AI technologies. 

• Quality Mindset – Balances speed, precision, and long-term maintainability.