LLM Eval Framework

⚙️ Configuration

Eval Enabled

Loading…

When off, no LLM judge calls are made and scores stay null.

Sample Rate

100%

% of queries that get evaluated. 100% = every query.

Flag Threshold (Overall Score)

0.70

Queries scoring below this are marked flagged in the table.

Judge Model ID

Model used for judge calls. Should be a fast, cheap model.

Judge Provider

🦙

Ollama

☁️

AWS Bedrock

🤖

Claude API

🔷

Azure OpenAI

✓ Saved

📈 Summary

—

Total Queries

—

Evaluated

—

Avg Overall

—

Avg Relevance

—

Avg Faithfulness

—

Avg Context Prec.

—

Flagged

—

Hallucinations

🗂 Eval Scores

Flagged only

Question	Overall	Relevance	Faithfulness	Context	Halluc.	Status	Time
Loading…

🗃 Golden Dataset

Question	Expected Answer (excerpt)	Retrieval	Actions
Loading…

+ Add Q&A Pair

Question

Expected Answer

✓ Added

🏃 Eval Run History

Started	Triggered by	Status	Model	Questions	Avg Score	vs Baseline	Flagged
No eval runs yet. Click "Run Eval Now" to start.