Lead Quality Engineer – AI Delivery @Marsh Tech

Company:

Description:

The Role

We are looking for an experienced Lead Quality Engineer – AI Delivery to lead Quality Engineering across multiple AI delivery pods building enterprise-scale AI solutions. This role will be based in Cluj and has a requirement of working at least three days a week in the office.

Our engineering teams already operate within an established shift-left Secure SDLC, with core engineering, security, code quality, and pipeline controls embedded into delivery. This role builds on that foundation, ensuring each AI solution has an appropriate risk-based quality strategy, with particular focus on GenAI evaluation, end-to-end quality, UAT readiness, and release confidence.

You will work closely with Product Managers, Product Owners, Engineering Leads, Architects, and developers, combining strategic quality leadership with hands-on technical capability.

We will count on you to:

• Lead Quality Engineering across multiple AI delivery pods and establish consistent, risk-based quality standards.
• Partner with Product and Engineering to define test strategy, acceptance criteria, UAT approach, quality risks, and release-readiness criteria.
• Ensure appropriate coverage across functional, integration, end-to-end, non-functional, AI-specific, and business acceptance testing.
• Define testing and evaluation approaches for LLM, RAG, agentic, and AI-enabled solutions, including grounding, hallucination risk, regression, tool calling, workflow behaviour, and non-deterministic outputs.
• Establish appropriate AI evaluation datasets, test harnesses, baselines, thresholds, and regression approaches.
• Guide and, where needed, implement reusable automation across web, APIs, backend services, integrations, asynchronous workflows, and AI components.
• Ensure products are ready for UAT, with appropriate test evidence, environments, test data, defect status, entry/exit criteria, and known risks.
• Provide Pod Leads and programme leadership with clear quality metrics, risks, defect trends, AI evaluation results, UAT readiness, and release-readiness reporting.
• Conduct quality reviews across pods and identify gaps in testing, testability, observability, or quality evidence.
• Use production incidents, escaped defects, telemetry, and user feedback to continuously improve quality approaches.
• Coach engineers and delivery teams to strengthen shared ownership of quality across the lifecycle.

What you need to have

• Proven experience leading Quality Engineering for enterprise-scale AI / GenAI solutions in production.
• Hands-on experience testing LLM, RAG, agentic, workflow-based, or AI-enabled applications.
• Strong experience defining end-to-end test strategies, UAT approaches, quality gates, and release-readiness criteria.
• Experience working across multiple delivery teams or pods in complex enterprise environments.
• Strong software engineering and automation skills in TypeScript / JavaScript and/or Python.
• Experience with modern automation frameworks such as Playwright, Cypress, Selenium, Jest, Mocha, or PyTest.
• Strong experience testing web applications, APIs, backend services, integrations, and distributed systems.
• Experience with AI evaluation frameworks, regression suites, test harnesses, and representative evaluation datasets.
• Good understanding of CI/CD, cloud-native systems, observability, and Secure SDLC practices.
• Strong stakeholder management skills and the ability to translate quality evidence into clear risks, metrics, and recommendations for leadership.

What makes you stand out

• Experience with LangChain, LangSmith, Mastra, or similar AI engineering and evaluation tooling.
• Experience testing complex RAG, multi-agent, or tool-calling solutions.
• Experience with performance, resilience, accessibility, or security testing.
• Experience using production telemetry and incidents to improve test and evaluation strategies.
• Ability to balance strong quality controls with pragmatic, fast-moving AI delivery.

Technology context

Typical technologies include:

React, TypeScript, Node.js / NestJS, Python / FastAPI, REST APIs, async workflows, AWS / Azure, GitHub Actions / GitLab CI / Azure DevOps, LLM APIs, RAG, agentic workflows, LangChain, LangSmith, and Mastra.

Why join our team:

• We help you be your best through professional development opportunities, interesting work, and supportive leaders.

• We foster a vibrant and inclusive culture where you can work with talented colleagues to create new solutions and have an impact for colleagues, clients, and communities.

• Our scale enables us to provide a range of career opportunities, as well as benefits and rewards to enhance your well-being.

• A yearly budget and the opportunity to build your flexible benefits package (up to 20 percent of your annual salary).

• 30+ days off, including legal days, birthday, public holiday replacements, and benefits options.

• Performance bonus scheme.

• Matching charity contributions, charity days off, and the Pay it Forward charity challenge.

• Core benefits: Pension, Life and Medical Insurance, Meal Vouchers, Travel Insurance.