Dev Tools
ai-game-devtools
Resource hub for AI-powered game development — LLMs, world models, agents.
★ 1.4k starsMIT
View guide →A streamlined framework for evaluating and benchmarking LLMs, VLMs and AIGC models.
evalscope is an open-source dev tools project by modelscope (modelscope/evalscope on GitHub). It has 3,509 GitHub stars and 509 forks, is released under the Apache-2.0 license, and was last updated on GitHub on 8 October 2026.
Decide the first concrete outcome you want from it, so you can pick the right example, template or mode instead of exploring everything at once.
Clone the repository (or install the published package) and skim its README quick start for the requirements for your platform.
Run the documented install step and set any required configuration — such as provider keys, data sources or local services — before the first run.
Execute the smallest documented example end to end, confirm the output is what you expected, then adapt it to your real data or workflow.
Industry-recognised benchmarks including MMLU, C-Eval and GSM8K built in, with custom dataset support.
Evaluate LLMs, VLMs, embedding, reranker and AIGC models in one framework.
Native backend plus OpenCompass, VLMEvalKit and RAGEval integrations.
Drive benchmarks such as GSM8K, AIME and SWE-bench Agentic inside a controlled multi-turn AgentLoop with Docker sandbox and full traces.
Stress-test model services with TTFT, TPOT and throughput metrics, recordable to wandb/swanlab.
Interactive web dashboard for comparison, plus pairwise Arena mode to rank models head-to-head.
Get the source so you can inspect, configure and run it locally.
git clone https://github.com/modelscope/evalscope.git
cd evalscopeInstall the package from PyPI.
pip install evalscopeRun a first benchmark against an OpenAI-compatible endpoint.
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5It is maintained by modelscope on GitHub (modelscope/evalscope), where the source, issues and releases live.
Yes — it is free and open source under Apache-2.0, which permits use, modification and commercial redistribution, with licence/notice retention.
For API models you supply that provider's key; local models downloaded from ModelScope need no paid key, just compute.
Compared with OpenCompass or lm-eval used standalone, evalscope's distinguishing focus is: a streamlined framework for evaluating and benchmarking LLMs, VLMs and AIGC models. Pick it when that focus matches your need; the alternative may suit you better if its broader or different scope is the priority.
Clone https://github.com/modelscope/evalscope (or install its published package), follow the README quick start to install and configure it, then run the smallest documented example — the Run section on this page mirrors those steps.
Resource hub for AI-powered game development — LLMs, world models, agents.
Fine-tune and deploy 600+ LLMs and 300+ multimodal models with PEFT or full-parameter training.
Unified, efficient fine-tuning of 100+ LLMs and VLMs with a CLI and Web UI — ACL 2024
A coding-agent harness optimized for smaller local LLMs — small models, serious coding work
Backend.AI is a streamlined, container-based computing cluster platform.
An end-to-end framework for building custom AI applications and agents on top of your database.
The most-starred other Dev Tools projects in the LibreAI catalogue — the realistic shortlist if evalscope is not the right fit.
The open source coding agent
The model-definition framework behind modern AI — thousands of pretrained transformers for text, vision, and audio.
Anthropic's agentic coding tool that lives in your terminal and works across your whole codebase
Cross-platform desktop all-in-one manager for Claude Code, Codex, OpenCode and other coding agents
OpenAI's lightweight coding agent that runs locally in your terminal
Free and open — grab the source, star the repo, and start creating.