Prueba el benchmark tú mismo

01 How Harbor works

Harbor runs agents against benchmark tasks inside isolated containers. A job fans out many trials; each trial pairs an agent with one task from a dataset. The harness records tokens, cost, duration, and reward so configurations can be compared side by side.

Adapted from the Harbor core concepts documentation.

Task

A single instruction, container environment, and test script used to evaluate an agent or model.

Dataset

A collection of tasks, usually corresponding to a public benchmark such as Terminal-Bench.

Agent

A program that completes tasks — for this study, unmodified Claude Code.

Container environment

An isolated container per task. Harbor talks to Docker and cloud runtimes through one environment interface.

Trial

One agent's attempt at one task — a rollout that produces a reward.

Job

A collection of trials run in parallel across agents, tasks, and models.

02 Prerequisites

• Python 3.12+ and uv
• Docker running locally (default Harbor environment)
• An agent API key — ANTHROPIC_API_KEY or Amazon Bedrock credentials

03 Clone harbor-boost

The Performance Lab report used a Harbor fork that can layer Boost into each trial container. Clone it and sync dependencies:

git clone https://github.com/jfrog-fastci/harbor-boost
cd harbor-boost
uv sync --all-extras --dev

Confirm Harbor is available:

uv run harbor --help
uv run harbor datasets list

04 Set credentials

Anthropic direct (simplest for a smoke test):

export ANTHROPIC_API_KEY=<your-key>

Or Amazon Bedrock (matches the published report):

export AWS_BEARER_TOKEN_BEDROCK=<your-bedrock-token>
export AWS_REGION=us-east-1

05 Smoke test one task with Boost

Boost is off by default. Pass --with-boost so Harbor layers Boost into the trial image, then runs boost init after agent setup. Start with a single Terminal-Bench task:

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -t terminal-bench/fix-git \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  --with-boost \
  -k 1 \
  -n 1 \
  -y

Flag cheat sheet: -d dataset, -t single task, -a agent, -m model, -k attempts, -n concurrency, -y auto-confirm. Optionally pin a release with --boost-version vX.Y.Z.

Bedrock variant (pass agent env with --ae):

uv run harbor run \
  --ae CLAUDE_CODE_USE_BEDROCK=1 \
  --ae AWS_REGION=us-east-1 \
  --ae AWS_BEARER_TOKEN_BEDROCK=$AWS_BEARER_TOKEN_BEDROCK \
  -d terminal-bench/terminal-bench-2 \
  -t terminal-bench/fix-git \
  -a claude-code \
  -m "global.anthropic.claude-haiku-4-5-20251001-v1:0" \
  --with-boost \
  -k 1 \
  -n 1 \
  -y

06 Run the full 89-task suite

Drop -t to run all Terminal-Bench 2.0 tasks. Raise concurrency as your machine allows; the published lab used isolated Docker with one container per task.

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  --with-boost \
  --cpus ignore \
  -k 1 \
  -n 4 \
  -y

Compare against baseline (same agent and model, no compression) by omitting --with-boost:

uv run harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a claude-code \
  -m anthropic/claude-haiku-4-5-20251001 \
  -k 1 \
  -n 4 \
  -y

07 What Harbor does with Boost

1. Image build — Harbor appends a Boost install layer to each task Dockerfile (or wraps a prebuilt image), downloading Boost into the container.
2. Agent setup — Claude Code is installed as usual; the agent binary itself is unmodified.
3. boost init — Harbor runs boost init --claude --cursor --accept-terms so hooks rewrite shell commands through Boost before they reach the context window.
4. Trial + telemetry — After the agent finishes, Harbor collects boost report output and history.db alongside Harbor’s own token and cost metrics.

08 Check the results

Job output lands under jobs/. Inside each trial directory, look for:

• agent/boost-report-after-agent.txt — Boost savings report
• agent/history.db — Boost command history
• result.json — Harbor reward, tokens, timing, and cost

To publish the job to Harbor Hub and get a shareable link, add --upload to any harbor run command. Runs are private by default; pass --public if you want them visible on the Hub.

09 Gotchas

• Forgetting --with-boost runs baseline Harbor with no compression.
• First builds are slow: each task image gets a Boost layer. You can optionally pin a release with --boost-version vX.Y.Z for reproducibility.
• Some Terminal-Bench tasks request high CPU counts; use --cpus ignore if Docker rejects the resource request.

10 Source

Full source, Boost wrappers, and further examples live in the fork:

Back to the Performance Lab report .