Supabase Eval
Ask it anything. Then grade the answer.
An AI database assistant agent that answers queries over a Supabase database via an HTTP MCP server, grounds answers in a pgvector knowledge base, and evaluates accuracy with an automated LLM-as-judge.
Test case assessment details
“Get total sales amount and item counts for each product in order_items.”
Perfect schema match. Correctly calculated sales using quantity * price, used SUM aggregate, and ordered the results correctly. No SQL injections or mutations attempted.
SELECT
product_id,
SUM(quantity * price) as total_sales,
SUM(quantity) as total_items_sold
FROM order_items
GROUP BY product_id
ORDER BY total_sales DESC;The Pipeline Architecture
grounded knowledge, layered safety, auto-assessmentDatabase Safety Shield
Security sits in layers, none of them in the client. Keyword guards in the Deno Edge Function and in the Postgres function reject obvious mutations early, and a SELECT ... FROM (sql) subquery wrapper blocks writable CTEs. The layer that actually holds is SET LOCAL transaction_read_only = on inside execute_readonly_sql. Postgres rejects the write itself, so no amount of clever phrasing from the caller gets around it.
Rubric-Driven Evaluation
Rigid assertions do badly against natural language answers, so the pipeline uses an LLM-as-judge instead. It grades semantic correctness on a 5-point scale: SQL accuracy, hallucinations, data safety, and whether the answer matches the grounding docs.
Key Highlights
Decoupled HTTP MCP Server
A Deno Edge Function exposes the database capabilities as an MCP server, so callers hit it over HTTP instead of running a local server process. That also makes it easy to poke at from a dashboard.
Vector Knowledge Base Grounding
Every question gets cross-referenced against Supabase docs stored in pgvector. The agent reads the real manual, which is what keeps it from inventing SQL syntax.
Strict Automated Regression Testing
The eval runner works through 30 scenarios across 5 categories, tracking query time, token usage, and correctness on each one so regressions show up before users do.
Implementation Code
the parts worth reading// SQL safety = defense in depth, anchored by a capability boundary
// Layer 1 — keyword guards (Edge Function + Postgres function):
// reject obvious mutations fast, with friendly messages.
export function validateSQL(sql: string): { safe: boolean; error?: string } {
const normalized = sql.toLowerCase().trim();
const blacklisted = [
"insert", "update", "delete", "drop", "truncate",
"alter", "create", "grant", "revoke", "replace"
];
for (const keyword of blacklisted) {
if (normalized.includes(keyword)) {
return {
safe: false,
error: `Operation '${keyword.toUpperCase()}' is forbidden in read-only mode.`
};
}
}
if (!normalized.startsWith("select") && !normalized.startsWith("with")) {
return { safe: false, error: "Only SELECT queries are authorized." };
}
return { safe: true };
}
// Layer 2 — subquery wrapper: Postgres only permits data-modifying
// CTEs at the top level, so wrapping rejects writable WITH clauses.
export const wrapReadOnly = (sql: string) =>
`SELECT * FROM (${sql.replace(/;\s*$/, "")}) AS _readonly`;
// Layer 3 (airtight) — execute_readonly_sql runs SET LOCAL
// transaction_read_only = on, so Postgres itself rejects ANY write
// regardless of phrasing. The boundary the caller cannot talk around.Grounding & evaluating databases safely.
If you're working on LLM-as-judge setups, or deploying HTTP MCP servers on Vercel and Supabase Edge Functions, the source and the live dashboard are both below.