Bring your own custom-trained or pre-trained models and define the scenarios your product depends on. Omega runs several models on the same scenario at once, each in an isolated sandbox, and ranks them against each other.
| # | Model | Δ last run | |||||
|---|---|---|---|---|---|---|---|
| 1 | support-ft-v3yours | 0.94 | 0.91 | 0.78 | 0.88 | 0.88 | +0.04 |
| 2 | base-model-b | 0.89 | 0.84 | 0.86 | 0.85 | 0.86 | 0.00 |
| 3 | support-ft-v2yours | 0.90 | 0.87 | 0.71 | 0.84 | 0.83 | −0.03 · SQL |
| 4 | base-model-a | 0.81 | 0.79 | 0.80 | 0.77 | 0.79 | 0.00 |
A model that wins public benchmarks can still fail on the tasks your product actually runs.
Testing models one at a time, on different prompts and different days, gives results you can't compare.
A model update can improve one scenario and make another worse without anyone noticing.
Add custom-trained, fine-tuned or pre-trained models to your workspace.
Describe the use cases to test when the run starts, and what counts as a pass.
Every model runs on the same scenarios at the same time, each in its own sandbox.
See which model wins each scenario and which wins overall.
Run the same scenarios on every new version to catch regressions.
Describe the use cases that matter when the run starts, in your own terms, with the criteria that count as a pass. Every model is judged on the same scenarios, so the results are comparable.
Run several models on the same scenario at once. Each one gets its own isolated sandbox, so no run affects another and every model sees exactly the same test.
When a model changes, run the same scenarios again. Omega compares the new results with the last run and flags any scenario that got worse, even when the overall score went up.
Compare your fine-tuned models with pre-trained ones on the exact scenarios your product depends on.
Point to a ranking per scenario when you pick a model, and see straight away when an update helps or hurts.
Once you know which models work for your use cases, build the product itself with autonomous agents.
Explore Autonomous development →Red-teamingEverything built on Omega is red-teamed before release, by your team or ours.
Explore Red-teaming →Your own custom-trained or fine-tuned models, and any pre-trained models you want to compare them with.
Yes. That is what Evaluate & rank is built for: run several models on the same scenario at once, and Omega ranks them against each other.
You describe your use cases at the start of a run, along with what a good result looks like. Every model in the run is tested against the same scenarios.
Each model runs in its own isolated sandbox, on your organisation's dedicated cluster, or inside your own servers if you use Omega on-prem.
Re-run the same scenarios on each new model version. Omega compares the results with the previous run and flags any scenario that got worse.
We'll run them side by side and show you which one wins, scenario by scenario.