Senior QA Engineer

This role sits within a fast-growing technology company developing advanced AI agents for the
construction industry, focused on automating the analysis of building models, drawings, and
regulatory documents to enhance compliance, planning, and design validation. As part of a
senior engineering team, this position plays a critical role in ensuring the reliability and
trustworthiness of complex, non-deterministic AI systems that support key decision-making
processes in construction projects.

Responsibilities:

• Design and execute comprehensive tests for agentic AI systems, focusing on tool selection,

multi-step reasoning, and recovery from failures

• Evaluate agent outputs for grounding, citation accuracy, and evidence-backed claims
• Develop and maintain automated frameworks to assess generative AI outputs for hallucination,

consistency, and factual accuracy

• Build and run guardrail and adversarial tests to identify vulnerabilities such as prompt injection

and instruction manipulation

• Validate multi-turn conversations and ensure correct handling of conversational state across

sessions

• Implement and monitor continuous evaluation of agent performance in production

environments

• Investigate and triage quality incidents, tracing failures to specific models or datasets and

collaborating on fixes

• Create and maintain robust integration and API tests to ensure seamless backend and

user-facing workflows

• Enforce quality gates and produce verification evidence to support confident releases
• Map test cases to system requirements, ensuring traceability and comprehensive coverage

Skills:

• Expert Python programming for test automation and custom evaluation tooling
• Hands-on experience with LLM/agent evaluation frameworks (e.g., DeepEval, TruLens,

RAGAS, or custom evaluators)

• Proficiency in designing and implementing non-deterministic and generative AI test strategies
• Strong knowledge of integration, API, and UI automation tools (e.g., Selenium, Playwright,

Requests)

• Experience with performance and load testing tools (e.g., Locust, JMeter, k6)
• Proficiency with SQL and data validation tools (e.g., Great Expectations)
• Familiarity with vector databases and retrieval corpora
• Experience integrating automated tests into CI/CD pipelines (e.g., GitLab)
• Ability to define and communicate pass/fail criteria for probabilistic systems
• Strong collaboration and communication skills with engineering and product teams