_Prompt Eval Harness
Evaluate and compare LLM prompts with multi-model A/B testing via Ollama on-prem. Measure latency, token count, and binary keyword match score for each prompt × model cell. 100% privacy — no data sent to the cloud.
Evaluation prompt
Separate multiple prompts with a line break between each one, up to the maximum of lines.A keyword per line (same order of prompts). Score 1.0 if the keyword is in the answer; otherwise, 0.0 for qualitative manual scoring. Leave blank for now.
Ollama On-Prem Models
Select one or more models. Each model will be evaluated on all prompts. If a model is not installed on the VPS, the cell will show a degrade message instead of an error. Run ollama pull <model> to install it.
How it works
Write the prompts
Insert up to 20 prompts separated by ---. You can add an expected keyword for each prompt for automated scoring.
Choose the models
Select the installed Ollama models on your Hetzner VPS, with each model evaluated across all prompts in an N x M matrix.
Analyze the results
Compare latency, token count and binary scores for each row. Export to CSV for off-line analysis.
Come utilizzare Prompt Eval Harness
Write your prompts
Write one or more prompts in the text area, separated by a line of three dashes (max 20 prompts).
Select models
Select one or more Ollama on-prem models to evaluate based on the inserted prompts.
Add expected keyword (optional)
Insert a keyword expected for prompts to enable binary scoring automation.
Start evaluation
Read the result matrix with latency, tokens and scores for each prompt cell for each model.
Suggerimenti
- Use brief and distinctive keywords for reliable binary scoring, such as technical terms.
- Compare latency between different models to choose the best model for your use case.
- Respect the 6 runs per minute limit: group multiple prompts into a single execution instead of running separate runs.
Domande frequenti
How many prompts and models can I evaluate together?
Up to 20 prompts, evaluated on all selected Ollama models. The tool generates a matrix with one cell for each prompt-model combination.
How does automatic scoring work?
Binary Input: If you insert a keyword expected for a prompt, the cell receives a score of 1.0 if the keyword is found in the model's response, otherwise 0.0. Without a keyword, scoring remains qualitative manual.
What does it mean when a cell shows "Model not available"?
The selected Ollama model is not installed on the VPS. The tool degrades the cell with a message instead of blocking an error; it needs to be installed using the command ollama pull.
Are my prompts sent to external cloud services?
No, inference occurs entirely on the on-prem installed Ollama on the VPS Hetzner. No data is sent to external cloud providers.
Can I export my evaluation results?
Yes, the CSV button exports the entire matrix (prompt, model, response, latency, token, score) to a downloadable file.