AI Agents
agent-memory-benchmark
A benchmark for comparing AI agent memory systems head-to-head.
★ 144 starsNOASSERTION
View guide →WebArena-Verified: the audited, deterministically scored release of the WebArena web-agent benchmark.
webarena-verified is an open-source ai agents project by ServiceNow (ServiceNow/webarena-verified on GitHub). It has 62 GitHub stars and 13 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 8 March 2026.
Decide whether to evaluate on the full dataset or the smaller, harder subset for cheaper runs.
Install webarena-verified with pip, run it directly with uvx, or use the published Docker image.
Export the dataset, start the site environments you need with the env commands, and configure your agent's environment URLs and credentials.
Run your agent, capture its responses and network traces, and score tasks with eval-tasks or the Python API.
Every task, reference answer, and evaluator has been manually reviewed and corrected relative to the original release.
Evaluation replaces LLM-as-a-judge and substring matching with type-aware normalisation and structural comparison.
Agent runs can be scored without live web environments by replaying captured network traces.
A difficulty-prioritised subset of the full task set supports faster, lower-cost evaluation.
The CLI starts and stops the benchmark's websites, with optimised Docker images and an environment-control API and dashboard.
Evaluate from the command line, from Python through the WebArenaVerified API, or entirely inside the published container image.
Most users install the package or use uvx/Docker instead of cloning.
git clone https://github.com/ServiceNow/webarena-verified.git
cd webarena-verifiedOr skip installation entirely and call it through uvx.
pip install webarena-verifiedFetch the dataset, then score an example agent log as the README demonstrates.
webarena-verified dataset-get --output webarena-verified.json
uvx webarena-verified eval-tasks --task-ids 108 --output-dir examples/agent_logs/demoIt is maintained by ServiceNow, with academic guidance acknowledged in the README.
Yes, it is Apache-2.0 licensed and published on PyPI and as a Docker image; running the site environments and your agent uses your own infrastructure and model access.
No API key is required by the benchmark, and scoring is deterministic rather than LLM-judged; the agent you evaluate uses its own model credentials.
The original WebArena contained tasks and evaluators with known errors and relied partly on fuzzy matching; the Verified release audits every task and evaluator and scores deterministically, including offline from traces.
Run uvx webarena-verified --help, export the dataset with dataset-get, start a site with env start, and score a run with eval-tasks.
A benchmark for comparing AI agent memory systems head-to-head.
Recursive self-improving physical agent OS — verified, evidence-based execution for robots and simulators
A benchmark for tool-agent-user interaction in real-world domains.
WorkArena: ServiceNow's benchmark for how well web agents handle everyday knowledge-work tasks.
Alibaba's in-page GUI agent: control any web interface with natural language, from one script tag
Rakazo — an open-source, self-hosted AI agent for persistent teammates: bring your own model and computer, run it on web, desktop, iPhone and Android
The most-starred other AI Agents projects in the LibreAI catalogue — the realistic shortlist if webarena-verified is not the right fit.
A personal AI that really does things — on any OS, with your data staying yours.
An agentic skills framework & software development methodology that works.
Agent harness optimization system with skills, memory and security for Claude Code, Codex and more
The agent that grows with you — a Python framework for adaptive AI agents.
DeepSeek's plugin-based agent harness where everything is a plugin.
Free and open — grab the source, star the repo, and start creating.