Evaligo builds benchmarks and RL environments for LLM agents, with a SWE-Race leaderboard and A/B testing. Free plan; Pro is $7/month. Features reviewed.
Researched and fact-checked by Bartu Cavusoglu · Updated October 26, 2026
Evaligo is a platform that builds benchmarks and reinforcement learning (RL) environments for LLM agents. Each benchmark is built from real work and paired with an environment containing tasks, simulated tools and a verifier. Model teams can train on the environments, evaluate on held-out tasks, and then publish the result or keep it private.
The site also describes a self-serve app where users test models on public tasks with their own API key, and a pricing page for a visual workflow builder with built-in A/B testing of agentic AI flows. This review covers both, using only what Evaligo publishes about itself.
How does Evaligo work?
Evaligo describes a three-step loop: build, train, measure. A benchmark and environment come with public and private test splits and a separate training split. Rollouts run on the training split only, and the environment uses Harbor format so any RL trainer can consume it. A sample command launches an agent such as mini-swe-agent against evaligo/swe-race.
For measurement, Pass@1 is computed on the private split that nobody trains on. In the app, users pick a benchmark, connect a model from any provider or their own endpoint, change the scaffold (system prompt, tools, step budget, reasoning effort, router) and compare runs on Pass@1, cost per solved task and trajectories.
Key features of Evaligo
The live benchmark is SWE-Race, which tests real concurrency bugs in repositories pinned to a commit, run in a sandbox with no network and verified by the project's hidden tests. Guardrails (prompt injection, data exfiltration, unauthorized tool calls) and Tool use (long-horizon multi-app tasks) are listed as in preparation.
Environments rely on stateful simulated tools, including chat, email, calendar, CRM, tickets, code host, spreadsheet, database, browser and payments. Verification options include hidden tests, state checks, policy checks and LLM-as-judge scoring. Users can also add their own mock by describing an API, or ask Evaligo to build one.
Key Features
SWE-Race coding benchmark — Live benchmark of real bugs such as race conditions, deadlocks and cancellation bugs, with a pinned commit per repository, a sandbox with no network, and verification by each project's own hidden tests.
Public, private and training splits — Each benchmark has public and private test splits plus a separate training split. Pass@1 on the private split gives the official held-out score.
RL environments in Harbor format — Environments run in Harbor format so any RL trainer can use them, with rollouts on the training split only. A harbor run command is shown for launching an agent against evaligo/swe-race.
Stateful simulated tools — Mocks of team chat, email, calendar, CRM, tickets, code host, docs and drive, spreadsheet, database, web browser, payments, and a sealed shell. Each keeps its own state and can be seeded with data and prompt injections.
Custom mocks and environments — Users can describe an API and have Evaligo build a stateful mock, or ask the team to build the environment with their own tasks, tools and verifier.
Multiple verification methods — Grading options include hidden tests in a clean verifier, state checks across every app's final state, policy checks for forbidden actions, and LLM-as-judge rubric scores from a multi-vendor panel.
Scaffold testing and run comparison — Users pick a benchmark, connect any provider or endpoint, change the system prompt, tools, step budget, reasoning effort or router, and compare Pass@1, cost and trajectories side by side.
Public model leaderboard — Evaligo runs public models on its benchmarks and publishes scores, including failures. SWE-Race v0.1 lists GLM-5.3 Flash at 81.9% and GPT-5.6 Luna at 81.3%.
A/B testing of agentic flows — The workflow builder includes built-in A/B testing that compares flow variants on quality, cost and latency, and flows can be deployed as API endpoints.
Web scraping and browser automation — Page scraping, HTTP requests, CSS selectors, link extraction and headless browser actions such as click, type and screenshot are available as nodes.
Pros & Cons
Pros
Benchmarks are built from real work, with SWE-Race using each project's own hidden tests as the verifier.
Separate public, private and training splits keep the held-out score free of training contamination.
Environments use Harbor format, so any RL trainer can consume them.
Stateful simulated tools can be seeded with data and prompt injections without touching real accounts.
Free plan with 500 credits and unlimited bring-your-own API keys, no credit card required.
Public leaderboard shows scores for named models, failures included.
Cons
Only SWE-Race is live; the Guardrails and Tool use benchmark families are marked in preparation.
Official held-out scores require submitting the scaffold to Evaligo, since the private split stays with the company.
Pricing for RL training and custom environment builds is not published.
The site describes both a benchmark platform and a workflow builder, and the published plans cover only the workflow builder; no team plan exists yet.
Pricing
Free $0 month
Pro $7 month
Frequently Asked Questions
What is Evaligo?
Evaligo builds benchmarks and RL environments for LLM agents. Each benchmark comes from real work and includes tasks, simulated tools and a verifier. Teams can train a model on the environments and evaluate it on held-out tasks.
How much does Evaligo cost?
The pricing page lists a Free plan at $0 per month with 500 credits and a Pro plan at $7 per month with 10,000 credits. Pricing for RL environment training or custom builds is not published.
Does Evaligo have a free plan?
Yes. The Free plan costs $0, requires no credit card, and includes 500 credits per month and unlimited use of your own API keys. The app's public-task testing is also described as free with your own model API key.
What is SWE-Race?
SWE-Race is Evaligo's live coding benchmark of race conditions, deadlocks and cancellation bugs in real repositories. It has 188 tasks from 106 projects, with 93 held out and 2,006 graded attempts, and uses each project's hidden tests for verification.
Which models lead the SWE-Race leaderboard?
In the SWE-Race v0.1 results dated 3 Oct 2026, GLM-5.3 Flash scored 81.9% and GPT-5.6 Luna 81.3%, sharing first place. DeepSeek V4 Flash scored 72.1%.
What does Evaligo integrate with?
The integrations page lists 12 connections: GitHub, Bitbucket, OpenAI, Anthropic, Google Gemini, Google Ads, Google Analytics, Google Search Console, Twitter/X, SMTP email, a web scraper and an AI Prompt node.
Can I keep my results private?
Yes. Evaligo states that results stay private unless you choose to publish them, and that training on its environments can be done privately.
Does Evaligo offer a team or enterprise plan?
Not yet. The pricing page says team features are in development and asks users who need collaboration, SSO or enterprise features to contact support@evaligo.com.