$ benchmarks

נסו את ה-benchmark בעצמכם

Harbor is a framework for evaluating and optimizing agents and models in container environments. This guide walks you through running Terminal-Bench 2.0 with Boost enabled, using ourharbor-boostfork.

01How Harbor works

Harbor runs agents against benchmark tasks inside isolated containers. Ajob fans out manytrials; each trial pairs anagent with onetask from adataset. The harness records tokens, cost, duration, and reward so configurations can be compared side by side.

Adapted from theHarbor core conceptsdocumentation.

Task: A single instruction, container environment, and test script used to evaluate an agent or model.
Dataset: A collection of tasks, usually corresponding to a public benchmark such as Terminal-Bench.
Agent: A program that completes tasks — for this study, unmodified Claude Code.
Container environment: An isolated container per task. Harbor talks to Docker and cloud runtimes through one environment interface.
Trial: One agent's attempt at one task — a rollout that produces a reward.
Job: A collection of trials run in parallel across agents, tasks, and models.

02Prerequisites

Python 3.12+ anduv
Docker running locally (default Harbor environment)
An agent API key —ANTHROPIC_API_KEYor Amazon Bedrock credentials

03Clone harbor-boost

The Performance Lab report used a Harbor fork that can layer Boost into each trial container. Clone it and sync dependencies:

git clone https://github.com/jfrog-fastci/harbor-boost
cd harbor-boost
uv sync --all-extras --dev

Confirm Harbor is available:

uv run harbor --help
uv run harbor datasets list

04Set credentials

Anthropic direct (simplest for a smoke test):

export ANTHROPIC_API_KEY=<your-key>

Or Amazon Bedrock (matches the published report):

export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-token>
export AWS_REGION=us-east-1

05Smoke test one task with Boost

Boost is off by default. Pass--with-boostso Harbor layers Boost into the trial image, then runsboost initafter agent setup. Start with a single Terminal-Bench task:

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -t terminal-bench/fix-git \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  --with-boost \
  -k 1 \
  -n 1 \
  -y

Flag cheat sheet:-d dataset,-t single task,-a agent,-m model,-k attempts,-n concurrency,-y auto-confirm. Optionally pin a release with--boost-version vX.Y.Z.

Bedrock variant (pass agent env with--ae):

uv run harbor run \
  --ae CLAUDE_CODE_USE_BEDROCK=1 \
  --ae AWS_REGION=us-east-1 \
  --ae AWS_BEARER_TOKEN_BEDROCK=$AWS_BEARER_TOKEN_BEDROCK \
  -d terminal-bench/terminal-bench-2 \
  -t terminal-bench/fix-git \
  -a claude-code \
  -m "global.anthropic.claude-haiku-4-5-20251001-v1:0" \
  --with-boost \
  -k 1 \
  -n 1 \
  -y

06Run the full 89-task suite

Drop-tto run all Terminal-Bench 2.0 tasks. Raise concurrency as your machine allows; the published lab used isolated Docker with one container per task.

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  --with-boost \
  --cpus ignore \
  -k 1 \
  -n 4 \
  -y

Compare against baseline (same agent and model, no compression) by omitting--with-boost:

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  -k 1 \
  -n 4 \
  -y

07What Harbor does with Boost

Image build — Harbor appends a Boost install layer to each task Dockerfile (or wraps a prebuilt image), downloading Boost into the container.
Agent setup — Claude Code is installed as usual; the agent binary itself is unmodified.
boost init — Harbor runsboost init --claude --cursor --accept-termsso hooks rewrite shell commands through Boost before they reach the context window.
Trial + telemetry — After the agent finishes, Harbor collectsboost reportoutput andhistory.dbalongside Harbor’s own token and cost metrics.

08Check the results

Job output lands underjobs/. Inside each trial directory, look for:

agent/boost-report-after-agent.txt— Boost savings report
agent/history.db— Boost command history
result.json— Harbor reward, tokens, timing, and cost

To publish the job toHarbor Huband get a shareable link, add--uploadto anyharbor runcommand. Runs are private by default; pass--publicif you want them visible on the Hub.

09Gotchas

Forgetting--with-boostruns baseline Harbor with no compression.
First builds are slow: each task image gets a Boost layer. You can optionally pin a release with--boost-version vX.Y.Zfor reproducibility.
Some Terminal-Bench tasks request high CPU counts; use--cpus ignoreif Docker rejects the resource request.

10Source

Full source, Boost wrappers, and further examples live in the fork:

Open harbor-boost on GitHub

Back to thePerformance Lab report.

גרסה חדשה של Boost זמינה— Smarter savings and cleaner output