From Vibe Coding to Production Agents: Designing MCP Toolchains That Don’t Hallucinate, Leak, or Break

Originally published on tamiz.pro.

Vibe coding gets you a working prototype. Production agents require something else entirely: a toolchain where every call is schema-validated, every permission is explicit, every output is verifiable, and every failure is contained. The gap between those two states is not a matter of better prompts. It is an engineering problem involving protocol boundaries, trust zones, execution semantics, and observability.

This guide designs a production-grade Model Context Protocol (MCP) toolchain for agents that operate on real systems. The central principle is simple: treat the model as an untrusted planner, not a trusted executor.

• 1. The Core Architectural Shift
• 2. The MCP Trust Model
• 3. Tool Contracts That Resist Hallucination
• 4. The Execution Plane
• 5. Least-Privilege Tool Design
• 6. A Production Reference Implementation
• 7. Secret and Data Leakage Controls
• 8. Observability and Auditability
• 9. Reliability Under Failure
• 10. Production Readiness Checklist
• Frequently Asked Questions

1. The Core Architectural Shift

A prototype agent typically wires the model directly to tools:

User -> LLM -> Tool Executor -> Real API 

That topology works until the model invents an argument, passes an unexpected identifier, retries a non-idempotent operation, or retrieves a record containing sensitive information. The model is probabilistic, and its context contains untrusted data. Neither fact changes because the demo succeeded.

A production topology inserts deterministic infrastructure between intent and effect:

User -> Agent Runtime -> Policy Engine -> Tool Gateway -> Sandboxed Worker -> Real API | | | | | | | +-- egress allowlist | | +-- schema validation + redaction | +-- authorization + risk classification +-- trace ID + budget + audit events 

The model proposes an action. The toolchain decides whether that action is permissible, translates it into a valid operation, executes it with constrained credentials, and records enough evidence to reconstruct what happened.

This architecture addresses three distinct failure classes that are often conflated:

The important distinction is that “the model was careful” is not a control. Controls belong in code paths the model cannot bypass.

2. The MCP Trust Model

MCP separates the model-facing surface from the implementation of capabilities. A server exposes tools, resources, and prompts over a transport such as stdio or HTTP. The client decides which server to connect to and which operations to expose to the model.

That separation is useful, but it is not a security boundary by itself. Once a tool is exposed, the model can attempt any operation that tool permits. If a tool accepts arbitrary SQL, the model effectively has arbitrary SQL access. If a tool returns full account records, the model can request data outside the task’s scope.

Treat each component according to its trust level:

User input: Untrusted Retrieved documents: Untrusted Tool descriptions: Semi-trusted configuration Model output: Untrusted intent Tool implementation: Trusted code, constrained inputs External service data: Untrusted 

“Semi-trusted configuration” deserves emphasis. Tool descriptions often live in configuration files, repositories, or registries. A compromised description can instruct the model to ignore prior instructions, exfiltrate data through a legitimate parameter, or select a dangerous tool. Descriptions should be reviewed, versioned, and included in the security boundary.

Separate planning from execution

The most effective design pattern is to expose narrow, intent-shaped tools rather than generic primitives. Compare:

Bad: query_database(sql: string) Good: find_invoices(customer_id: string, status: enum) 

The first tool transfers the entire database query language into the model’s responsibility. The second encodes business constraints in a contract the model cannot alter.

A useful heuristic: if a tool requires the model to understand a complex grammar to use it correctly, the tool is too low-level for reliable agentic execution.

Define explicit trust zones

Production agents frequently need to access multiple systems with different sensitivity levels. Organize servers into trust zones and permit transitions explicitly.

Zone A: Read-only knowledge - documentation search - public product catalog Zone B: Internal business data - customer records - support tickets Zone C: State-changing operations - refunds - ticket closure - deployments 

Moving from Zone A to Zone B may require additional authorization. Moving into Zone C may require approval, a dry run, or a policy decision based on the operation’s risk.

A common mistake is exposing all zones to the same model context. The model then needs no special permission to use the most sensitive tool, and any prompt injection can attempt it. Keeping zones separate reduces blast radius and lets the runtime apply different policies.

3. Tool Contracts That Resist Hallucination

Hallucination in agentic systems is not limited to invented prose. It includes invented identifiers, incorrect enum values, unsupported operations, and assumptions about resource state.

Use closed schemas

Every tool should define a strict input schema with required fields, additional properties rejected, and enums where the domain is finite.

type FindInvoiceInput = { customer_id: string; status: "draft" | "open" | "paid" | "void"; issued_after?: string; }; 

The runtime should validate against the schema before the implementation runs. Reject unknown properties instead of silently ignoring them. Silently dropping an unexpected field is dangerous because the model believes it specified a constraint that the executor discarded.

For example, using Zod:

import { z } from "zod"; export const FindInvoiceInput = z.object({ customer_id: z.string().regex(/^cus_[a-zA-Z0-9]{12}$/), status: z.enum(["draft", "open", "paid", "void"]), issued_after: z.string().datetime().optional(), }).strict(); 

Schema validation catches malformed intent. It does not catch plausible but nonexistent identifiers.

Resolve identifiers through tools

Do not ask the model to synthesize opaque identifiers. Provide discovery and resolution tools so the agent can obtain valid values through a controlled lookup.

const FindCustomer = z.object({ query: z.string().min(2).max(120), }).strict(); async function findCustomer(input: unknown, ctx: ToolContext) { const { query } = FindCustomer.parse(input); const matches = await ctx.customerDirectory.search(query, { limit: 5, tenantId: ctx.principal.tenantId, }); return { matches: matches.map((customer) => ({ customer_id: customer.id, display_name: customer.displayName, })), }; } 

The response should not expose the full directory. Returning a bounded list of scoped identifiers gives the model what it needs without creating a general-purpose data extraction channel.

Make state transitions explicit

State-changing tools should communicate what will happen, what state is required, and what the result means. Include a preflight operation for consequential actions:

prepare_refund(payment_id, amount, reason) -> RefundPlan commit_refund(plan_id, idempotency_key) -> RefundResult 

The prepare step can verify the payment exists, the amount is within policy, and the reason is allowed. It returns a short-lived, signed plan identifier. The commit step executes the exact operation that was validated.

This two-phase pattern prevents a model from improvising the business logic of a destructive operation. It also enables human approval between preparation and execution.

Return structured, bounded results

Free-form text encourages the model to infer facts that the tool did not provide. Prefer structured responses with explicit nullability and bounded cardinality.

type RefundResult = { status: "succeeded" | "rejected" | "pending"; refund_id?: string; rejection_code?: "amount_exceeds_limit" | "already_refunded" | "payment_closed"; }; 

Use discriminated unions rather than a generic error string. If the tool returns "Payment is not refundable", the model might reinterpret it. If it returns status: "rejected" and rejection_code: "payment_closed", the runtime can enforce deterministic behavior.

Limit result sizes at the tool boundary. A tool that can return 100,000 records is both a context-management problem and a leakage vector.

4. The Execution Plane

The tool gateway is the point where model intent becomes an operation. It should perform the same work for every request: authenticate, authorize, validate, normalize, classify, execute, filter, and audit.

Model request | v [Authentication] Who is asking? | v [Authorization] May this principal perform this operation? | v [Validation] Is the payload structurally valid? | v [Normalization] Convert formats, enforce canonical values | v [Risk policy] Approval? Dry run? Allowed target? | v [Execution] Least-privilege worker, bounded resources | v [Output filter] Redact, truncate, remove disallowed fields | v [Audit event] Record decision and outcome 

Normalize at the boundary

Models produce inconsistent representations of dates, amounts, and identifiers. Normalize these values before they reach business logic.

function normalizeToolInput(input: unknown, schema: ZodSchema): NormalizedInput { const parsed = schema.parse(input); return { ...parsed, issued_after: parsed.issued_after ? new Date(parsed.issued_after).toISOString() : undefined, }; } 

Do not rely on the downstream service to interpret loose formats. If a downstream API accepts both milliseconds and ISO timestamps, the model must not choose between them.

Classify risk

Risk classification should be a property of the tool and operation, not a decision made by the model.

type RiskClass = "read" | "write" | "sensitive_read" | "destructive"; const riskByTool: Record<string, RiskClass> = { search_docs: "read", find_customer: "sensitive_read", update_ticket: "write", issue_refund: "destructive", }; 

The runtime can then apply different policies:

read -> standard authorization sensitive_read -> field-level filtering, purpose binding write -> idempotency, confirmation for important records destructive -> approval policy, amount limits, dry-run support 

Keep the model out of the control loop

The model should not decide whether its own request was authorized. It should not select which credential to use. It should not construct the final URL for a sensitive API. Those decisions belong to deterministic code.

A practical rule: the model may choose among business-meaningful options, but it should never choose among security principals, destinations, or raw protocol parameters.

5. Least-Privilege Tool Design

The most effective privilege control is making the dangerous operation unavailable.

Scope credentials per tool

Do not give the entire agent one service account with broad API access. Give each tool a credential scoped to exactly the operation it performs.

search_docs -> read:docs find_customer -> read:customers:basic read_invoice -> read:invoices:tenant issue_refund -> write:refunds:limited 

In a cloud environment, this often means separate IAM roles or workload identities per tool server. In a SaaS environment, use per-tool OAuth scopes and tenant-bound tokens.

Bind credentials to the caller

A tool running in a shared server must not use a single ambient credential. Forward or mint an identity that preserves the caller’s tenant and authorization context.

type ToolContext = { principal: { userId: string; tenantId: string; scopes: string[]; }; traceId: string; deadline: number; idempotencyKey?: string; }; 

Then enforce the scope inside the tool:

function requireScope(ctx: ToolContext, scope: string) { if (!ctx.principal.scopes.includes(scope)) { throw new AuthorizationError(`Missing required scope: ${scope}`); } } 

Do not accept tenant IDs solely from the model’s arguments. Derive the tenant from the authenticated principal. A model-provided tenant field is a request, not an authorization decision.

Separate read and write identities

Read-only tools should not share credentials with write-capable tools. If the model attempts to misuse a read tool, the credential should make the mutation impossible rather than relying on the server to reject an unexpected HTTP method.

Constrain destinations

If a tool fetches URLs, allow only known hosts and paths:

const allowedFetchTargets = [ { host: "api.internal.example", paths: [/^\/v1\/customers\//] }, { host: "files.example.com", paths: [/^\/public\//] }, ]; 

A generic fetch(url) tool in an agent environment is a direct path to server-side request forgery and data exfiltration. Replace it with domain-specific tools or a tightly constrained fetch service.

6. A Production Reference Implementation

The following TypeScript example demonstrates the core gateway. It is intentionally small, but the boundaries are the important part.

Project setup

mkdir production-mcp-gateway cd production-mcp-gateway npm init -y npm install zod express pino npm install -D typescript tsx @types/node @types/express npx tsc --init 

Configure tsconfig.json:

{ "compilerOptions": { "target": "ES2022", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "outDir": "dist" }, "include": ["src/**/*.ts"] }\n``` {% endraw %} ### Define the tool registry {% raw %} ```typescript // src/tools.ts import { z } from "zod"; export type Principal = { userId: string; tenantId: string; scopes: string[]; }; export type ToolContext = { principal: Principal; traceId: string; deadline: number; idempotencyKey?: string; }; export type ToolDefinition<T> = { name: string; description: string; risk: "read" | "sensitive_read" | "write" | "destructive"; requiredScope: string; inputSchema: z.ZodType<T>; execute: (input: T, ctx: ToolContext) => Promise<unknown>; }; export const findCustomer = ToolDefinition.define({ name: "find_customer", description: "Find customers within the authenticated tenant by name or email.", risk: "sensitive_read", requiredScope: "customers:read", inputSchema: z.object({ query: z.string().min(2).max(120), }).strict(), execute: async (input, ctx) => { const response = await fetch( "https://api.internal.example/v1/customer-directory", { method: "POST", headers: { authorization: `Bearer ${process.env.CUSTOMER_DIRECTORY_TOKEN}`, "content-type": "application/json", "x-tenant-id": ctx.principal.tenantId, "x-trace-id": ctx.traceId, }, body: JSON.stringify({ query: input.query, limit: 5 }), signal: AbortSignal.timeout(5_000), }, ); if (!response.ok) { throw new ToolExecutionError("customer_directory_unavailable"); } const data = (await response.json()) as { customers: Array<{ id: string; displayName: string }>; }; return { matches: data.customers.map((customer) => ({ customer_id: customer.id, display_name: customer.displayName, })), }; }, }); 

The production implementation should obtain the directory token from a workload identity provider or secret manager at startup, not from model input or environment variables passed through the agent runtime.

Implement the gateway

// src/gateway.ts import { z } from "zod"; import { findCustomer, type ToolContext } from "./tools.js"; const registry = new Map<string, ToolDefinition<any>>([ [findCustomer.name, findCustomer], ]); export class AuthorizationError extends Error {} export class ToolExecutionError extends Error {} export type AuthenticatedRequest = { principal: ToolContext["principal"]; traceId: string; idempotencyKey?: string; }; export async function executeTool( toolName: string, rawInput: unknown, request: AuthenticatedRequest, ): Promise<{ result: unknown }> { const tool = registry.get(toolName); if (!tool) { throw new ToolExecutionError("unknown_tool"); } if (!request.principal.scopes.includes(tool.requiredScope)) { throw new AuthorizationError("insufficient_scope"); } const input = tool.inputSchema.parse(rawInput); const ctx: ToolContext = { principal: request.principal, traceId: request.traceId, deadline: Date.now() + 10_000, idempotencyKey: request.idempotencyKey, }; const result = await tool.execute(input, ctx); return { result }; } 

In production, add these checks before execute:

• Verify the tool description matches the reviewed, deployed version.
• Enforce a per-tenant and per-principal rate limit.
• Require an idempotency key for write and destructive tools.
• Apply policy approval for destructive operations.
• Redact and shape the result before returning it to the model.

Expose the gateway through MCP

// src/server.ts import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; import { executeTool, type AuthenticatedRequest } from "./gateway.js"; import { findCustomer } from "./tools.js"; const server = new McpServer({ name: "production-customer-tools", version: "1.0.0", }); server.registerTool( findCustomer.name, { description: findCustomer.description, inputSchema: findCustomer.inputSchema.shape, }, async (args) => { const request: AuthenticatedRequest = { principal: { userId: process.env