Spec-Driven Engineering: Replacing Flaky Unit Tests with LLM-as-a-Judge Evals — How Does It Work in Production?

TL;DR: The era of assert(expected == actual) is dead in AI-native software engineering. Attempting to test probabilistic, generative systems with deterministic string matching or brittle regex leads to flaky CI/CD pipelines and developer burnout. The future of production AI testing relies on Spec-Driven Engineering—using tiered, LLM-as-a-Judge evaluations governed by strict behavioral rubrics and Conductor track gates to mathematically guarantee system behavior without bankrupting your FinOps budget.

It was 2:00 AM on a Thursday, and my Slack notifications were screaming.

Our core production deployment was blocked. The continuous integration (CI) pipeline had failed for the fourth time in a row. I pulled up the logs, expecting to find a misconfigured environment variable, a rogue database migration, or perhaps a sudden API rate limit. Instead, I found a unit test failure that perfectly encapsulated the existential crisis of modern software engineering.

The test was designed to verify that our customer support AI agent correctly extracted a user's intent and formatted it as a JSON payload. For months, the test had passed flawlessly. But earlier that day, we had updated our base model. The new model was objectively better—more empathetic, more accurate, and faster. Yet, it broke the build.

Why? Because instead of outputting {"intent": "refund", "confidence": 0.95}, the model had decided to be polite. It outputted: Here is the extracted information you requested: \n\n {"intent": "refund", "confidence": 0.95}.

Our test, a standard string assertion written by a senior engineer who had spent a decade mastering Test-Driven Development (TDD), looked exactly like this: assertEqual(response.text, expected_json_string).

The test failed. The pipeline halted. The deployment was blocked. And in that moment, staring at the red "FAILED" badge on my monitor, I realized a fundamental truth that the industry is still struggling to accept: the paradigms we built to test deterministic software are fundamentally broken when applied to probabilistic AI systems. We are trying to measure a cloud with a ruler.

The Illusion of Regex and Semantic Similarity

When you first realize that exact string matching is useless for Large Language Models (LLMs), your engineering instincts kick in. You look for workarounds. You try to bend the probabilistic nature of the model back into the deterministic box you are comfortable with.

My team’s first instinct was to use Regular Expressions. If the model insists on adding conversational filler, we thought, we will simply regex our way out of it. We wrote complex patterns to extract the JSON blocks from the markdown wrappers. For a week, the CI pipeline was green again. But then, the model started using single quotes instead of double quotes. Or it added a trailing comma. Or it decided to output XML because a user's prompt subtly biased it in that direction. The regex patterns became increasingly arcane, a fragile house of cards that required a PhD in string manipulation to maintain. We were no longer testing the AI; we were testing our ability to write regex.

When regex failed, we turned to the academic darling of 2024: semantic similarity.

The theory was elegant. Instead of comparing the exact text, we would convert both the expected output and the actual output into vector embeddings using a model like Gemini Embedding 2. We would then calculate the cosine similarity between the two vectors. If the similarity score was above 0.85, the test passed.

It felt like magic—until it didn't.

Semantic similarity is brilliant at determining if two pieces of text are talking about the same topic. It is catastrophically bad at determining if they are conveying the same fact. Consider these two sentences:

1. "The patient's blood pressure is dangerously high, administer medication immediately."
2. "The patient's blood pressure is completely normal, withhold all medication."

To a dense embedding model, these sentences are incredibly close in vector space. They share the same vocabulary, the same syntactic structure, and the exact same semantic domain (cardiology). A cosine similarity check will often score them at 0.92 or higher. But in a production healthcare application, treating these two outputs as "equivalent" is a literal matter of life and death.

Embeddings measure proximity of concept, not accuracy of reasoning. They cannot verify if a model hallucinated a metric, if it adhered to a strict brand tone, or if it successfully avoided generating personally identifiable information (PII). We realized that we couldn't rely on math to judge logic. We needed reasoning to judge reasoning.

The Evaluator's Dilemma

This realization inevitably leads every AI engineering team to the same conclusion: LLM-as-a-Judge. If only an LLM can understand the nuanced, probabilistic output of another LLM, then we must use an LLM in our CI/CD pipeline to evaluate the results.

We built a new testing framework. Instead of assertions, we wrote prompts. We took our test cases, fed the outputs into gemini-3.1-pro-preview, and asked it: "Did the original model answer the user's question correctly without hallucinating?"

Initially, the results were spectacular. The flaky tests disappeared. The judge model easily saw through conversational filler, ignored minor formatting quirks, and correctly identified when the core logic was sound. We patted ourselves on the back, convinced we had solved AI testing.

Then, the reality of production hit us.

First, there was the latency. A traditional unit test suite of 5,000 tests runs in about 12 seconds. Our new LLM-as-a-Judge suite took three and a half hours. Developers, accustomed to rapid feedback loops, began context-switching. Pull requests languished. The velocity of the entire engineering organization ground to a halt.

Second, there was the cost. Running thousands of evaluations through a frontier reasoning model on every single git commit is a FinOps nightmare. Our cloud bill for CI/CD compute skyrocketed. We were spending more money testing the application than we were serving it to actual users.

But the most insidious problem was the return of the flakiness, this time in a new, more dangerous form. We encountered what I call the "Evaluator's Dilemma": Who evaluates the evaluator?

We noticed that our judge model was inconsistent. On Tuesday, it would pass a test. On Wednesday, given the exact same inputs, it would fail it. We discovered that the judge was susceptible to its own biases. It suffered from "verbosity bias" (assuming longer answers were inherently better) and "positional bias" (favoring information presented at the beginning of a prompt).

We had replaced a brittle, deterministic testing system with a slow, expensive, and unpredictable one. We hadn't solved the problem; we had merely abstracted it behind an API call. Throwing a massive model at a test suite isn't an engineering strategy; it's a liability.

Enter Spec-Driven Engineering

The breakthrough came when we stopped looking at how we were testing and started looking at what we were testing.

In traditional software, you test the output. In AI-native software, you must test the behavior. This is the core philosophy of Spec-Driven Engineering. You do not ask the judge, "Is this output correct?" You ask the judge, "Does this output adhere to this specific, deterministic behavioral rubric?"

We began leveraging the Vertex AI Gen AI evaluation service. This service fundamentally changed our approach by forcing us to formalize our evaluation criteria. Instead of writing open-ended prompts for our judge, we defined strict, multi-dimensional rubrics.

A rubric is a behavioral contract. For example, instead of a generic "check for hallucinations" prompt, we defined a Spec:

• Score 1: The response contains factual claims not present in the provided context (Severe Hallucination).
• Score 2: The response relies on external knowledge outside the context, but the knowledge is generally accepted as true (Minor Hallucination).
• Score 3: The response strictly uses only the provided context, but misses key details (Incomplete).
• Score 4: The response strictly uses only the provided context and captures all key details (Perfect).

By forcing the LLM-as-a-Judge to grade against a highly specific, constrained rubric, we drastically reduced the non-determinism of the evaluator. The judge was no longer giving an opinion; it was executing a classification task based on a strict behavioral specification.

But a rubric alone doesn't solve the latency and cost issues. To build a truly production-ready CI/CD pipeline for AI, we had to rethink our architecture. We couldn't send every test to a massive reasoning model. We needed a system of tiered gates.

Architecting the Tiered Evaluation Pipeline

In physical manufacturing, quality assurance isn't done by sending every single bolt to the chief engineer for inspection. You use automated, fast sensors to check the basic dimensions (the fast gate), and you only send the complex, assembled engines to the senior inspectors (the deep gate).

We applied this exact principle to our AI CI/CD pipeline using a pattern I call Conductor Track Gates.

Instead of a single evaluation step, we built a multi-stage routing system. When a developer commits code, the test suite triggers a tiered evaluation process.

1. The Fast Gate (Syntax & Schema): The first line of defense is speed. We use gemini-3.8-flash (or gemini-2.5-flash for our stable LTS pipelines). Flash models are incredibly fast and cost a fraction of a cent. We use them to evaluate the deterministic aspects of the output: Did it return valid JSON? Did it adhere to the required schema? Is the tone generally polite? If the output fails the Fast Gate, the test fails immediately. No need to waste expensive compute on deep reasoning.
2. The Deep Gate (Reasoning & Grounding): Only if the output passes the Fast Gate does it move to the Deep Gate. Here, we deploy our heavy hitters, like gemini-3.1-pro-preview or gemini-2.5-pro. These models are tasked with the complex, rubric-driven evaluations: Did it hallucinate? Did it follow the complex multi-step reasoning instructions? Is it properly grounded in the provided enterprise data?

Here is the architectural topology of a Spec-Driven Conductor pipeline:

flowchart LR
    A[Developer Commit] --> B{Traditional Unit Tests}
    B -- Pass --> C[Generate AI Outputs]
    B -- Fail --> Z[Block Merge]
    
    C --> D{Fast Gate: Gemini 3.8 Flash}
    D -- "Check: Schema, JSON, Tone" --> E{Pass?}
    E -- No --> Z
    
    E -- Yes --> F{Deep Gate: Gemini 3.1 Pro}
    F -- "Check: Hallucination, Logic Rubric" --> G{Pass?}
    
    G -- No --> Z
    G -- Yes --> H[Approve Merge / Deploy]
    
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#bbf,stroke:#333,stroke-width:2px
    style F fill:#fbb,stroke:#333,stroke-width:2px
    style H fill:#bfb,stroke:#333,stroke-width:2px
    style Z fill:#f99,stroke:#333,stroke-width:2px

This architecture gives us the best of all worlds. We get the mathematical certainty of behavioral rubrics, the deep reasoning capabilities of frontier models, and the speed and cost-efficiency of lightweight models.

To make this concrete, let’s look at how this is actually implemented in production using the 2026 Gen AI SDK. We define a custom evaluator that uses a strict rubric, leveraging the Vertex AI evaluation framework to ensure structured, parseable outputs.

import os
from google import genai
from google.genai import types
from pydantic import BaseModel, Field

# Initialize the 2026 Gen AI SDK client
client = genai.Client(
    vertexai=True, 
    project=os.environ.get("GOOGLE_CLOUD_PROJECT"), 
    location="us-central1"
)

# 1. Define the deterministic output schema for the Judge
class EvaluationResult(BaseModel):
    score: int = Field(description="The integer score from 1 to 4 based on the rubric.")
    reasoning: str = Field(description="Step-by-step justification for the score.")
    passed: bool = Field(description="True if score is 3 or 4, False otherwise.")

# 2. Define the strict Behavioral Rubric (The Spec)
HALLUCINATION_RUBRIC = """
You are an impartial, strict quality assurance judge. Evaluate the ACTUAL_OUTPUT against the SOURCE_CONTEXT.
Do not evaluate based on general knowledge. Only use the provided rubric.

RUBRIC:
1: ACTUAL_OUTPUT contains factual claims not present in SOURCE_CONTEXT.
2: ACTUAL_OUTPUT relies on external knowledge outside SOURCE_CONTEXT, even if true.
3: ACTUAL_OUTPUT strictly uses only SOURCE_CONTEXT, but misses key details.
4: ACTUAL_OUTPUT strictly uses only SOURCE_CONTEXT and captures all key details.
"""

def run_deep_gate_eval(source_context: str, actual_output: str) -> EvaluationResult:
    """
    Executes the Deep Gate evaluation using a reasoning model (Pro).
    This is only called if the Fast Gate (Flash) has already verified schema/syntax.
    """
    prompt = f"""
    {HALLUCINATION_RUBRIC}
    
    SOURCE_CONTEXT:
    {source_context}
    
    ACTUAL_OUTPUT:
    {actual_output}
    """
    
    # We use Gemini 3.1 Pro Preview for deep reasoning tasks in the CI pipeline
    response = client.models.generate_content(
        model='gemini-3.1-pro-preview',
        contents=prompt,
        config=types.GenerateContentConfig(
            response_mime_type="application/json",
            response_schema=EvaluationResult,
            temperature=0.0, # Force maximum determinism from the judge
        ),
    )
    
    # The SDK automatically parses the structured output into our Pydantic model
    return response.parsed

# Example CI/CD Test Execution
def test_agent_grounding():
    context = "The company refund policy allows returns within 30 days of purchase with a receipt."
    agent_output = "You can return the item within 30 days if you have a receipt. We also offer store credit for up to 60 days."
    
    # In a real pipeline, this is preceded by a Fast Gate check using gemini-3.8-flash
    
    eval_result = run_deep_gate_eval(context, agent_output)
    
    # The assertion is no longer on the string, but on the behavioral spec
    assert eval_result.passed is True, f"Eval Failed. Score: {eval_result.score}. Reason: {eval_result.reasoning}"

# If we run this, it will FAIL because the agent hallucinated the "60 days store credit" rule.

Notice the critical shift in the code above. We are no longer asserting that agent_output == expected_string. We are asserting that eval_result.passed is True. The test itself is deterministic; the complexity of dealing with probability has been cleanly encapsulated within the rubric and the judge model. By setting the temperature=0.0 and forcing a structured JSON output via response_schema, we strip away the conversational fluff from the judge, turning it into a cold, calculating classification engine.

The Economics of Evaluation

Architecture without FinOps is just academic daydreaming. The most common pushback I get from engineering directors when I propose LLM-as-a-Judge is the cost. "Doddi," they say, "we run 100,000 unit tests a month. If we use a Pro model for every test, we'll blow our entire cloud budget on CI/CD."

They are absolutely right. If you implement a naive, single-tier evaluation strategy where every single test case is routed to a heavy reasoning model, you will bleed money. This is why the Conductor track gates are not just a latency optimization; they are a financial necessity.

Let's look at the actual math using verified Google Cloud SKUs for our stable LTS pipeline models (Gemini 2.5 Pro and Gemini 2.5 Flash).

If we assume a standard enterprise CI/CD pipeline running 100,000 evaluation test cases per month, where the average test requires 2,000 input tokens (the prompt, the rubric, the context, and the output to be evaluated) and generates 500 output tokens (the reasoning and the score).

Here is the deterministic FinOps breakdown comparing a naive approach versus our tiered Spec-Driven approach:

📊 Production FinOps & TCO Simulation: Monthly CI/CD Evaluation Costs: Naive vs. Tiered Spec-Driven Gates (Verified SKU Math)

Production Workload Assumptions (us-central1 / asia-southeast1):

• 100,000 evaluation test cases run per month in the CI/CD pipeline.
• Average test case requires 2,000 input tokens (prompt + context + output to evaluate).
• Average evaluation response generates 500 output tokens (reasoning + score).
• Option A uses Gemini 2.5 Pro for all 100,000 evaluations (Naive Approach).
• Option B uses a Tiered Spec-Driven approach: 80% of tests are gated by Gemini 2.5 Flash (syntax, schema, basic tone), and only 20% pass through to Gemini 2.5 Pro for deep reasoning and hallucination checks.

Architecture Option	Verified SKU Unit Price & Monthly Formula	Verified Monthly Cost
Naive Single-Tier Eval (All Pro)	Pro Judge Input Tokens (200M): $1.25/1M input tokens × 200 = $250.00 Pro Judge Output Tokens (50M): $10/1M output tokens × 50 = $500.00	$750.00 / mo
Tiered Spec-Driven Eval (80% Flash / 20% Pro)	Flash Fast-Gate Input (160M): $0.15/1M input tokens × 160 = $24.00 Flash Fast-Gate Output (40M): $0.6/1M output tokens × 40 = $24.00 Pro Deep-Eval Input (40M): $1.25/1M input tokens × 40 = $50.00 Pro Deep-Eval Output (10M): $10/1M output tokens × 10 = $100.00	$198.00 / mo
Net FinOps Impact (Monthly Savings)	Verified by the Python SKU engine	73.6% TCO Reduction ($552.00 / mo)

Official Google Cloud SKU Pricing Sources (2026.09): cloud.google.com

By implementing the Conductor track gates, we don't just reduce latency; we slash our evaluation costs by nearly 74%. The vast majority of regressions in AI development—broken JSON schemas, missing keys, sudden shifts in tone—are caught by the incredibly cheap Flash models at the Fast Gate. We reserve the expensive Pro models exclusively for the complex, nuanced reasoning tasks that actually require them. This is how you scale AI engineering without scaling your budget linearly.

The Future of AI-Native CI/CD

We are standing at the precipice of a massive shift in software engineering. The tools and methodologies we relied on for the last twenty years are proving inadequate for the next twenty.

Spec-Driven Engineering is not just a testing strategy; it is a fundamental realignment of how we define "correctness" in software. When you stop obsessing over the exact string of characters a system outputs and start focusing on the behavioral boundaries it must operate within, you free your engineering team to iterate faster. You allow the underlying models to improve and evolve without breaking your builds.

The Vertex AI Gen AI evaluation service and the broader Model Garden ecosystem are providing the primitives we need to build these systems. But it is up to us, the architects and engineers, to assemble them correctly.

We must abandon the psychological comfort of the simple assert(expected == actual). We must embrace the complexity of behavioral rubrics. We must design tiered, economically viable pipelines. Because in the AI-native future, the teams that win won't be the ones with the best models; they will be the ones with the best systems for evaluating those models in production.

The next time your CI pipeline fails at 2:00 AM because an LLM decided to be polite, don't write another regex. Write a rubric. Build a gate. And let the machines judge the machines.