Benchmark selbst ausführen

01How Harbor works

Harbor runs agents against benchmark tasks inside isolated containers. Ajob fans out manytrials; each trial pairs anagent with onetask from adataset. The harness records tokens, cost, duration, and reward so configurations can be compared side by side.

Adapted from theHarbor core conceptsdocumentation.

Task

A single instruction, container environment, and test script used to evaluate an agent or model.

Dataset

A collection of tasks, usually corresponding to a public benchmark such as Terminal-Bench.

Agent

A program that completes tasks — for this study, unmodified Claude Code.

Container environment

An isolated container per task. Harbor talks to Docker and cloud runtimes through one environment interface.

Trial

One agent's attempt at one task — a rollout that produces a reward.

Job

A collection of trials run in parallel across agents, tasks, and models.

02Prerequisites

• Python 3.12+ anduv
• Docker running locally (default Harbor environment)
• An agent API key —ANTHROPIC_API_KEYor Amazon Bedrock credentials

03Clone harbor-boost

The Performance Lab report used a Harbor fork that can layer Boost into each trial container. Clone it and sync dependencies:

git clone https://github.com/jfrog-fastci/harbor-boost
cd harbor-boost
uv sync --all-extras --dev

Confirm Harbor is available:

uv run harbor --help
uv run harbor datasets list

04Set credentials

Anthropic direct (simplest for a smoke test):

export ANTHROPIC_API_KEY=<your-key>

Or Amazon Bedrock (matches the published report):

export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-token>
export AWS_REGION=us-east-1

05Smoke test one task with Boost

Boost is off by default. Pass--with-boostso Harbor layers Boost into the trial image, then runsboost initafter agent setup. Start with a single Terminal-Bench task:

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -t terminal-bench/fix-git \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  --with-boost \
  -k 1 \
  -n 1 \
  -y

Flag cheat sheet:-d dataset,-t single task,-a agent,-m model,-k attempts,-n concurrency,-y auto-confirm. Optionally pin a release with--boost-version vX.Y.Z.

Bedrock variant (pass agent env with--ae):

uv run harbor run \
  --ae CLAUDE_CODE_USE_BEDROCK=1 \
  --ae AWS_REGION=us-east-1 \
  --ae AWS_BEARER_TOKEN_BEDROCK=$AWS_BEARER_TOKEN_BEDROCK \
  -d terminal-bench/terminal-bench-2 \
  -t terminal-bench/fix-git \
  -a claude-code \
  -m "global.anthropic.claude-haiku-4-5-20251001-v1:0" \
  --with-boost \
  -k 1 \
  -n 1 \
  -y

06Run the full 89-task suite

Drop-tto run all Terminal-Bench 2.0 tasks. Raise concurrency as your machine allows; the published lab used isolated Docker with one container per task.

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  --with-boost \
  --cpus ignore \
  -k 1 \
  -n 4 \
  -y

Compare against baseline (same agent and model, no compression) by omitting--with-boost:

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  -k 1 \
  -n 4 \
  -y

07What Harbor does with Boost

1. Image build — Harbor appends a Boost install layer to each task Dockerfile (or wraps a prebuilt image), downloading Boost into the container.
2. Agent setup — Claude Code is installed as usual; the agent binary itself is unmodified.
3. boost init — Harbor runsboost init --claude --cursor --accept-termsso hooks rewrite shell commands through Boost before they reach the context window.
4. Trial + telemetry — After the agent finishes, Harbor collectsboost reportoutput andhistory.dbalongside Harbor’s own token and cost metrics.

08Check the results

Job output lands underjobs/. Inside each trial directory, look for:

• agent/boost-report-after-agent.txt— Boost savings report
• agent/history.db— Boost command history
• result.json— Harbor reward, tokens, timing, and cost

To publish the job toHarbor Huband get a shareable link, add--uploadto anyharbor runcommand. Runs are private by default; pass--publicif you want them visible on the Hub.

09Gotchas

• Forgetting--with-boostruns baseline Harbor with no compression.
• First builds are slow: each task image gets a Boost layer. You can optionally pin a release with--boost-version vX.Y.Zfor reproducibility.
• Some Terminal-Bench tasks request high CPU counts; use--cpus ignoreif Docker rejects the resource request.

10Source

Full source, Boost wrappers, and further examples live in the fork:

Back to thePerformance Lab report.