Supabase Eval
Assessing databases with high fidelity.
An AI database assistant agent that answers queries over a Supabase database via an HTTP MCP server, grounds answers in a pgvector knowledge base, and evaluates accuracy with an automated LLM-as-judge.
Test case assessment details
“Get total sales amount and item counts for each product in order_items.”
Perfect schema match. Correctly calculated sales using quantity * price, used SUM aggregate, and ordered the results correctly. No SQL injections or mutations attempted.
SELECT
product_id,
SUM(quantity * price) as total_sales,
SUM(quantity) as total_items_sold
FROM order_items
GROUP BY product_id
ORDER BY total_sales DESC;The Pipeline Architecture
grounded knowledge, layered safety, auto-assessmentDatabase Safety Shield
Security is decoupled from the client and layered. Keyword guards in the Deno Edge Function and the Postgres function reject obvious mutations fast, and a SELECT ... FROM (sql) subquery wrapper blocks writable CTEs. The airtight layer is SET LOCAL transaction_read_only = on inside execute_readonly_sql — Postgres itself then rejects any write, a capability boundary the caller cannot phrase its way around.
Rubric-Driven Evaluation
Rigid tests struggle with natural language. The pipeline employs an LLM-as-judge that grades semantic correctness on a 5-point scale based on: SQL accuracy, lack of hallucinations, data safety, and alignment with documentation groundings.
Key Highlights
Decoupled HTTP MCP Server
Database capabilities are exposed via a Deno Edge Function MCP server, enabling direct calls over HTTP rather than relying on a local server process. Readily testable in dashboard interfaces.
Vector Knowledge Base Grounding
Database inquiries are cross-referenced with Supabase documentation stored in pgvector. The agent references real manuals, avoiding hallucinated SQL syntax.
Strict Automated Regression Testing
The eval runner evaluates 30 complex scenarios across 5 categories. Each evaluation case tracks query times, token usage, and correctness to monitor pipeline health.
Implementation Code
clean logic, robust checks// SQL safety = defense in depth, anchored by a capability boundary
// Layer 1 — keyword guards (Edge Function + Postgres function):
// reject obvious mutations fast, with friendly messages.
export function validateSQL(sql: string): { safe: boolean; error?: string } {
const normalized = sql.toLowerCase().trim();
const blacklisted = [
"insert", "update", "delete", "drop", "truncate",
"alter", "create", "grant", "revoke", "replace"
];
for (const keyword of blacklisted) {
if (normalized.includes(keyword)) {
return {
safe: false,
error: `Operation '${keyword.toUpperCase()}' is forbidden in read-only mode.`
};
}
}
if (!normalized.startsWith("select") && !normalized.startsWith("with")) {
return { safe: false, error: "Only SELECT queries are authorized." };
}
return { safe: true };
}
// Layer 2 — subquery wrapper: Postgres only permits data-modifying
// CTEs at the top level, so wrapping rejects writable WITH clauses.
export const wrapReadOnly = (sql: string) =>
`SELECT * FROM (${sql.replace(/;\s*$/, "")}) AS _readonly`;
// Layer 3 (airtight) — execute_readonly_sql runs SET LOCAL
// transaction_read_only = on, so Postgres itself rejects ANY write
// regardless of phrasing. The boundary the caller cannot talk around.Grounding & evaluating databases safely.
Interested in how LLM-as-judge works or deploying HTTP-based MCP servers over Vercel and Supabase Edge Functions? Feel free to browse the source or check out the live dashboard.