Senior QA Automation
Summary
Senior QA Automation Engineer building and maintaining automated test frameworks for AI/LLM and supply-chain systems—validating model outputs, APIs, and ERP integrations across CI/CD using Python, Pytest, Playwright, Selenium, Locust, JMeter, K6, DeepEval, TruLens, and Great Expectations.
About:
This role sits at the intersection of advanced AI development and rigorous quality assurance
within a leading technology organization focused on delivering secure, reliable solutions for
mission-critical sectors. The Senior QA Automation Engineer will play a key part in ensuring that
innovative AI and supply chain systems meet the highest standards of reliability and
compliance, supporting the organization’s commitment to operational excellence and
technological advancement.
Responsibilities:
- Lead the design and implementation of automated testing frameworks for AI and LLM-based
systems
- Develop and maintain automated test suites to validate AI outputs for accuracy, consistency,
and alignment with ground truth datasets
- Build and execute integration tests to ensure seamless data flow between AI agents and
enterprise systems such as ERP platforms
- Create and run performance tests to benchmark AI model latency, throughput, and reliability in
real-time environments
- Automate API testing to verify secure data exchange, RBAC enforcement, and PII redaction
before data reaches AI models
- Map automated test cases to system requirements and user needs to support formal
verification and validation processes
- Prepare and deliver test readiness documentation and evidence for stage gate and production
release reviews
- Manage defect identification, tracking, and resolution, ensuring traceability to specific model
versions or datasets
- Collaborate with cross-functional teams, including data scientists and systems engineers, to
align testing with acceptance criteria
Skills:
- Expert proficiency in Python for test automation
- Strong experience with Pytest, Selenium, and Playwright
- Advanced skills in performance testing tools (Locust, JMeter, K6)
- Hands-on experience evaluating LLMs and AI models (DeepEval, TruLens, custom evaluators)
- Proficiency in SQL and data validation tools (Great Expectations)
- Experience integrating automated tests into CI/CD pipelines (GitLab)
- Familiarity with requirements management and traceability tools (Jira, Jama, Polarion)
- Strong knowledge of test management and reporting tools (TestRail, Allure)
- Solid understanding of version control systems (Git, GitLab)
- Deep knowledge of modern quality engineering practices (Shift Left, Test Pyramid, mono-repo
architecture)