AI Testing / LLM Testing
Your AI works…
until it doesn’t.
LLMs hallucinate. Agents break. Prompts get exploited. We test your AI on five pillars, with a measurement system you keep — so it behaves predictably, safely, and reliably in production.
The problem
AI does not fail like normal software.
What we test
Five pillars. Each one a package you can buy on its own.
The first pillar is what every QA company sells. The other four are built for AI: they score the model's behaviour, attack it, and keep watching it after launch.
Functional
The app around the AI
Login, permissions, the chat opens, a question sends, an answer renders, exports work, error and rate-limit states behave. The model is mocked, so it runs on every pull request.
Pass rule: 100% green on every pull request
AI security
Attack it before users do
Prompt injection through user input and through your own data, jailbreaks, system-prompt leakage, cross-customer data access, PII extraction, harmful content.
Pass rule: zero criticals, attack success under an agreed ceiling
AI quality
Is the answer actually good?
A golden set of real questions with expected answers. Grounding and relevance metrics, a calibrated judge model with a written rubric, agent tool-call trajectories, robustness on messy input, consistency across runs and model updates.
Pass rule: every metric above threshold, no drop against baseline
Performance & cost
Fast and affordable enough
Latency p50 and p95, time to first token, tokens and cost per conversation, behaviour under concurrent load and provider rate limits. Measured on every eval run.
Pass rule: under your stated budget
Monitoring & drift
Still good next month?
Nightly evals against a stored baseline and weekly scoring of sampled production traffic, so a silent model update or a prompt tweak shows up as a step in the chart, not a customer complaint.
Pass rule: two weeks under threshold raise an alert
The stack
Open source, self-hosted where your data is involved.
Every tool sits behind an adapter in our framework, so if you already run a different eval or tracing platform, you keep it.
How an engagement runs
Six to eight weeks to a running pipeline.
Discovery
We inventory every AI feature, classify it by risk, walk the architecture and agree the thresholds. Week 1.
Dataset build
Golden set from your real traffic and your experts, plus perturbation and adversarial sets. Judge rubrics calibrated against human labels. Weeks 2 to 3.
Baseline scorecard
Every applicable test runs against the current version. You get a number per quality dimension, with failing cases and traces. Week 4.
Automation
Functional suite, evals wired into your CI with gates, tracing and the drift job deployed. Weeks 5 to 7.
Handover or run
Your team takes the pipeline, or we run it: nightly evals, weekly drift report, re-baseline on every prompt or model change.
Every pull request
Functional suite and deterministic checks. Model calls replayed from recordings. Fast and free.
Every night
Full evals on the live model, every dataset, compared against the stored baseline.
Every week
A sample of production conversations scored for grounding, relevance and safety. Alert after two weeks under threshold.
What you receive
A pipeline and a scorecard you keep, not a one-off report. The same report shape every month, so you can compare your AI with itself over time.
Put a number on your AI before your customers do.
From a single LLM feature to a tool-calling agent: a baseline scorecard in four weeks, a regression pipeline in eight.
Start AI testing