ttft — First-Token Latency Live in the Status Bar, Cache Rate in /ttft

Overview

Two of the most impactful signals in a coding agent are invisible: how long until the model starts typing (TTFT) and how fast it types (decode throughput). pi's built-in footer shows token usage, cost, and the latest message's cache-hit rate, but has no TTFT and no throughput. @piex-dev/ttft ports that capability from deepseek-harness's StatsLine to the pi status bar:

TTFT 1.2s · 45.3t/s

Fully automatic: TTFT is painted the instant the first token arrives, exact throughput follows when the stream completes (before tools run). Each turn is persisted via pi.appendEntry, so /resume rebuilds the full history.

Cache-hit statistics (session-cumulative + per-turn) are computed too, but they stay off the status bar: pi's built-in footer already shows the latest turn's CH%, and a second, differently-scoped cache number on the same bar would only confuse. The cumulative rate lives in /ttft, matching deepseek-harness's cache N% semantics.

How it works

Event sources: TTFT is the delta from turn_start.timestamp (emitted by pi before each LLM request, with a timestamp) to the first message_update (token-by-token stream events; the first is the first token), painted immediately. Throughput divides decode wall time (first token → last stream update) by usage.output, sampled only when all three exist, decode ≥ 200ms, and decode spans at least half the TTFT (double guard against buffered replay); message_end carries usage the moment the stream completes, before tools run, so the exact t/s hits the bar right away. The cache rate folds the four disjoint usage buckets (input/output/cacheRead/cacheWrite) with cacheRead / (input + cacheRead + cacheWrite), the same formula as deepseek-harness's TokenTotals, shown in /ttft detail.

One cache number per bar: pi 0.84+ already renders ↑input ↓output R W CH%, where CH is the latest assistant message's hit rate; deepseek-harness shows the session-cumulative rate (buckets summed first, then divided). @piex-dev/ttft fills the latter gap (in a long session it better reflects whether prefix reuse keeps paying off), but puts it in /ttft instead of the bar: two cache values with different scopes side by side would leave users unsure which one to read. The bar shows only what pi is missing entirely (TTFT and throughput); cache signals belong to /ttft.

Persistence & rebuild: each settled turn appends a ttft custom entry via pi.appendEntry("ttft", …) (never sent to the LLM); session_start scans sessionManager.getEntries() and rebuilds. Turn history comes only from ttft custom entries; token totals come only from assistant/toolResult-message and compaction/branch-summary usage (the same algorithm as pi's built-in footer, custom tools can report usage). The two sources never overlap, so rebuilding cannot double-count. The TTFT anchor is turn_start: pi emits it before the request is assembled, so this measures end-to-end first-token latency (including prompt assembly), the same step-level semantics deepseek-harness uses.

Usage

Install

pi install npm:@piex-dev/ttft

Source: extensions/ttft

Usage

Zero configuration. The segment appears automatically after the first turn:

TTFT <latency> · <tokens/s>t/s

TTFT is painted the moment the first token arrives and stays fixed for the turn; the exact t/s follows when the stream completes (message_end, before tools run); decode samples shorter than 200ms or shorter than half the TTFT show no t/s (gateway chunk-replay noise, marked buffered in /ttft); the whole segment renders dim, matching the rest of the footer. Cache rates stay off the bar to avoid colliding with pi's built-in CH%; see them here:

/ttft    # session avg TTFT, token totals, cumulative cache rate,
         # per-turn TTFT/throughput/output/rate table

Verify

pi -e ./extensions/ttft/src/ttft.ts -p "say hi" --no-session
bun test extensions/ttft/test/ttft.test.ts

Implementation

Package: extensions/ttft, a single src/ttft.ts (~300 lines) with pure metric logic (formatting, hit-rate formula, turn-record derivation) layered from event wiring, fully unit-tested.

turn_start        record the turn anchor (event.timestamp)
message_update    first assistant update → first token, paintStatus
message_end       stream completes with usage → exact t/s on the bar (before tools)
turn_end          sample usage, derive TurnRecord, appendEntry persistence
session_start     rebuild history + token totals from entries
session_shutdown  drop live references, prevent cross-session leaks

Key details: repaints happen only at first token, stream completion, and turn settle, never per token during streaming; the bar carries TTFT and t/s only, while cache totals are maintained continuously but rendered solely in /ttft; decode samples shorter than 200ms or shorter than half the TTFT count as gateway-buffered replay, so t/s is suppressed and marked buffered in /ttft while the output token count is still recorded; chunk replay is the classic disguise: after a 22s TTFT, 1000 tokens flushed in 0.9s would read as an impossible "1147 t/s"; pi runs several sessions in one process (/new, /resume, /fork), so module state is fully rebuilt on session_start and handlers verify the session id, keeping stale-session events off another session's bar; turn numbering resumes from the historical max + 1; appendEntry failures never break the status bar (best-effort persistence); turns without timing or a first token produce no record, matching deepseek-harness's "no sample, no display".

Design notes

Core tradeoffs: always-visible status beats a query command (waiting anxiety happens in the moment, not in retrospection); one signal, one home (cache already has a built-in display, so this package only adds the missing scope instead of a second bar segment); persistence beats memory (/resume is a frequent action, and stats that reset lose their meaning).

Changelog

Roadmap

Versions