Project 03
LLM Evaluation Harness
Golden sets, faithfulness scoring, and a CI gate that fails on regression
Runs a versioned golden-question set against the knowledge base on every commit, scoring faithfulness, answer relevance and retrieval hit-rate. Token spend and p95 latency are tracked per run. A regression beyond threshold fails the pipeline — the same discipline applied to prompts that you would apply to code. Results are written to S3 and rendered as a public dashboard.
Try it
How it works
- 01Golden set
Versioned question/answer pairs with expected source documents.
- 02Run
GitHub Actions triggers the eval Lambda on every push to main.
- 03Score
RAGAS-style faithfulness and relevance, plus deterministic retrieval hit-rate.
- 04Gate
Non-zero exit if any metric drops more than 5% against the last green run.
- 05Publish
Results appended to a JSON history in S3, read directly by this site.
Infrastructure & running cost
| Service | Role | Cost |
|---|---|---|
| GitHub Actions | CI trigger — free for public repos | $0 |
| Lambda | Eval execution | Free tier |
| S3 | Run history as static JSON | ~$0.05/mo |
Directly extends the CI evaluation work already on the CV at NatWest.