Opinion: Token Saving Plugins Fail Benchmarks in Coding Tasks

A developer published benchmarks assessing the effectiveness of token-saving plugins, RTK and Ponytail, against a custom agent harness called turaArm, finding the plugins offer limited real-world benefits. The author claims commonly used tools like RTK and Ponytail perform poorly on long-horizon coding tasks, with turaArm achieving an 80% success rate on modeled costs.

The benchmarks compared token usage and cost across different configurations, including a “Balanced” and “Direct” mode for turaArm, alongside Codex CLI. Results indicate turaArm configurations used significantly fewer tokens than Codex CLI, despite comparable pass rates.

The author questions why developers continue to adopt plugins that offer minimal savings when more effective solutions exist. The full report and code are available on GitHub, allowing others to replicate and validate the findings.