AI Quality Architect

We are looking for a senior hands-on architect who designs how software quality is assured when AI agents generate, execute, and maintain most of the testing. They own the architecture of the quality system: agent pipelines, context and oracle strategy, trust and risk boundaries, and cost and reliability of AI-driven testing. They are not a test manager and not a script writer. They decide what agents may do autonomously, how we know their results are correct, and where humans must stay in the loop.

Seniority: Staff / Principal level (typically 10+ years in software quality or engineering, at least 1–2 years of real production work with LLMs or agents).

1. Core responsibilities

• Design the end-to-end AI-driven test architecture (generation, execution, self-healing, triage, reporting).
• Define the oracle strategy: how correctness is specified so that agents can verify it.
• Build the evaluation framework for the agents themselves (evals, regression sets, drift detection, precision/recall of defects found).
• Define autonomy levels, guardrails, and human-in-the-loop checkpoints; own AI-in-QA governance and risk.
• Manage context architecture: specs, requirements, code, and test knowledge exposed to agents (RAG, MCP tools, knowledge graphs).
• Control cost, latency, and flakiness of AI-based pipelines; integrate into CI/CD.
• Set standards and mentor QA engineers, analysts, and developers across teams.

2. Must-have (hard requirements)

• Background in test architecture or quality engineering at scale: SDET, test automation architecture, or developer productivity.
• Production experience building or operating LLM/agent-based workflows (not only using Copilot or ChatGPT).
• Solid engineering skills: Python and/or TypeScript/Java, API and service architecture, CI/CD, containers.
• Understanding of LLM limitations: hallucination, non-determinism, prompt and context design, evaluation methods.
• Experience designing test strategy for distributed systems (microservices, event-driven, API-first).
• Ability to show measurable outcomes (escaped defects, cycle time, cost per test, automation reliability).

3. Nice-to-have

• Experience testing AI/ML products (LLM evals, red-teaming, safety and bias testing).
• MCP (Model Context Protocol) server or tool development; multi-agent orchestration.
• Observability and test analytics (OpenTelemetry, test impact analysis, flaky-test analytics).
• Domain with complex business logic (telecom, billing, fintech, enterprise B2B).
• Public footprint: talks, articles, open-source, patents.

4. We offer:

• Competitive salary;
• Medical insurance, covering dental services;
• More than 300 hard and soft-skills programs by the corporate career development center;
• Open environment and encouraging knowledge sharing culture;
• Free English courses and opportunity to practice foreign languages daily;
• Flexible working hours.