Handoff-H1: An Orchestrated Vision-Agent System for Material Quantity Takeoff from Construction Blueprints

Abstract

Converting a set of architectural blueprints into a complete material quantity takeoff requires visual perception across drawing sheets, dimensional and multi-hop reasoning, and grounding in construction conventions that the drawings never state. We present Handoff-H1, a generally available takeoff system built from three layers that earn their results in combination: purpose-built computer-vision models that recover typed primitives from each sheet; vision agents—tool-using agents equipped with generic image operations and in-house visual-task tools, including CV-model-backed counting and detection and the decomposition of framing plans into individual members; and a persistent, hierarchically structured project foundation, grounded in a curated construction knowledge base and consumed by orchestrated trade-scoped estimation with an independent verification pass. We evaluate on the Construction Blueprint Takeoff Benchmark (TAKEOFFBENCH-V1): 10 real, permissioned, PII-stripped residential blueprint sets paired with consensus-validated expert takeoffs—2,009 verified line items, restricted for scoring to the 1,348 primary-tier materials that drive an estimate—scored per trade by an LLM judge on material coverage and quantity Precision@25% (P@.25) and combined into a precision-weighted composite. Under identical scoring from the raw PDF, seven frontier and open-weight models span composites of 35–61, and independent professional estimators—scored against the same reconciled gold standard—post 77.6% (65.5% coverage, 87.9% P@.25). Handoff-H1, working end-to-end from the raw PDF, reaches 81.6% (86.1% coverage, 78.8% P@.25): roughly 20 points above the strongest frontier agent, and above the independent estimators by pairing near-human quantity precision with coverage they do not reach. The evaluation harness is public for the open harbor framework; the blueprint sets and ground truth are available upon request for research use.

Figure 1: The problem in one sheet. A dimension floor plan from a residential drawing set (title block removed), the three demands it makes of its reader, and four of the line items it must yield (quantities from Handoff-H1's takeoff of this set). A set like this one produces on the order of a hundred primary line items across nine trades, and none of the quantities on the right is written anywhere on the left.

1. Introduction

Construction cost estimation is a discipline in which a single error in material quantities can cascade into tens of thousands of dollars of overrun on a modest residential project. The task appears deceptively simple—count doors, measure walls, calculate roof area—but it requires (i) dimensional extraction across multiple drawing sheets, (ii) multi-hop reasoning that combines floor-plan dimensions with general-notes specifications and trade-level conventions, and (iii) grounding in domain knowledge that is rarely fully specified in the drawings themselves (Figure 1). Framing lumber is spaced at 16 inches on center; drywall comes in 4×8 sheets; shingle coverage depends on roof pitch. None of these conventions appears in the dimension lines. The scale compounds the difficulty: each drawing set in our benchmark yields between 69 and 166 primary-material line items across nine trades, and every one of them is the end point of that same read–reconcile–apply chain.

This combination defeats general-purpose systems from two directions at once. Large language models given raw blueprint content produce estimates that read fluently but lack dimensional grounding: hallucinated quantities, confused units, and missing trade-specific logic. Flat retrieval over blueprint chunks improves factual grounding but mixes foundation measurements with roofing specifications in a single context window, destroying the trade-level coherence that human estimators maintain naturally. And the visual layer is its own obstacle: the elements a takeoff is billed from—framing members, wall runs, roof planes—must first be seen, counted, and measured across sheets drawn at different scales (Figure 2), a perception task that frontier vision–language models do not yet perform reliably on construction documents [15, 17]. These are three different failures—missed visual structure, missing domain convention, and unmanaged context—and Section 3 builds one layer for each.

Figure 2: A typical foundation detail—a spread-base sonotube footing, drawn at its own scale (3/4″ = 1′-0″). A single callout block fans out into quantities across three scopes, and none of them appears as a number anywhere on the sheet. Reading the words is not enough; a takeoff must attach each one to the right trade and assembly.

We present Handoff-H1, a takeoff system built on the position that no single capability closes this gap. Handoff-H1 combines purpose-built computer-vision models for construction drawings; vision agents that pair generic image operations with in-house visual-task tools; a persistent, hierarchically structured project foundation in the spirit of hierarchical retrieval-augmented generation [11], grounded in a curated construction knowledge base. The system powers the takeoff features of the Handoff product. On the Construction Blueprint Takeoff Benchmark (TAKEOFFBENCH-V1)—built on consensus-validated expert takeoffs whose construction we describe in full—Handoff-H1 reaches an 81.6% composite (86.1% material coverage, 78.8% Precision@25%), while seven frontier and open-weight models, run as tool-using agents from the same raw PDFs under the same evaluation harness, span composites of 35 to 61. Using the same harness, independent professional estimators were evaluated to devise a Human Performance score on all cases reaching 77.6% composite, in comparison we see that Handoff-H1 gives up a few points of quantity precision in exchange for far higher coverage, thus creating a more thorough takeoff on items and trades that humans tend to miss.

Concretely, this report contributes: (i) a categorical description of the Handoff-H1 architecture—perception, knowledge, and orchestration—with the design rationale for each layer; (ii) a self-contained account of TAKEOFFBENCH-V1, including the consensus process by which expert ground truth was constructed and audited; (iii) a three-way evaluation under the same per-trade LLM judge—seven frontier and open-weight models, independent professional estimators, and Handoff-H1—including a per-trade analysis of where each excels; and (iv) an access model for the benchmark—public evaluation harness, data upon request—chosen to keep its gold standard out of training corpora.

2. Related work

Retrieval-augmented generation and hierarchical knowledge

RAG [16] augments generation with retrieved context, typically via dense retrieval over an embedded chunk index; the Naive/Advanced/Modular taxonomy [7] places Handoff-H1 in the Modular family, with a structured synthesis layer between retrieval and generation. Hierarchical architectures organize retrieved knowledge into multi-granular structures: HiRAG [11] clusters entities into local, bridge, and global layers and attributes up to 16% of multi-hop performance to the bridge; RAPTOR [23] builds a recursive abstractive tree; GraphRAG [10] and LightRAG [9] retrieve over entity-relation graphs. Handoff-H1's project foundation occupies a related niche, with the bridge granularity fixed by a construction-trade schema rather than learned clustering. The foundation's persistence follows the LLM-Wiki pattern [14]—curated knowledge maintained across queries rather than re-derived per query—applied at the retrieval index rather than the model weights, in contrast to domain-specialized pretraining such as BloombergGPT [25]. Domain-adapted RAG [28] and self-reflective retrieval [3] address adjacent error modes.

Agentic systems and per-component optimization

Tool-using agents that perform reasoning and acting [27] and self-refinement architectures that critique and revise their own outputs [19] are standard components of production LLM pipelines; Handoff-H1 adopts both patterns to adapt in the case of conflicting information.

Construction-drawing datasets and the takeoff ground-truth gap

Public datasets for architectural drawings target recognition rather than estimation: CubiCasa5K [13] provides 5,000 densely annotated floor plans; FloorPlanCAD [6] offers roughly 16,000 vector drawings for panoptic symbol spotting; ArchCAD-400K [18] scales symbol spotting to hundreds of thousands of drawings; ResPlan [1] contributes 17,000 residential plans as vector graphs. None carries the material quantities a takeoff requires. On the estimation side, CEQuest [26] probes drawing literacy with 164 questions rather than full line-item takeoffs. Earlier automation splits between BIM-based takeoff, which assumes complex modeling alongside structured input [24], and understanding of the floor-plan through deep-learning symbol detection [21]. Spatial understanding from drawings remains unsolved, Petersson et al. [20] report leading systems at or below a random baseline on floor-plan reconstruction. The scarcity of ground truth for takeoff is structural: a complete takeoff is a heavy expert lift, and estimators genuinely disagree on continuous-measurement quantities, so a defensible gold standard demands reconciliation rather than a single annotator (Section 4.2).

Specialized AI systems for takeoff vs. general-purpose document AI

Multimodal large language models have enabled a new category of AI document assistants that work directly on construction drawings, integrating OCR, natural language understanding, document retrieval, and PDF interaction. Bluebeam Max, powered by Claude [4], is one of the first commercially available assistants embedded in a professional construction-document workflow, aimed at boosting estimator productivity through document comprehension and interactive assistance. We conducted experiments with Bluebeam Max and several other general-purpose assistants on projects covering the benchmark's trade range, but we do not analyze them in depth because tools in this category are not intended to execute a takeoff fully autonomously, but rather to respond to user actions as the operator performs a takeoff or navigates a set of drawings. In that capacity, Bluebeam Max performed reasonably well in our tests—delivering dependable text search and extraction, interpretation of schedules and callouts, scale-aware measurements, and markup generation when steered by the user—albeit with some shortcomings. Its understanding is constrained to the document's embedded text layer, so rasterized sheets and image-only schedules were reported as unreadable, and its page-scoped context did not reconcile quantities across the sheets of a set, leading to double-counting of doors shown on multiple sheets and missing quantities outside the active page. Importantly, when prompted to perform a full takeoff, the assistant either refused or asked the user to provide the geometry it lacked, instead of hallucinating quantities—behavior that is appropriate for a system of this type. Since such assistants do not autonomously generate the structured, per-trade line items on which the benchmark's metrics are computed, placing them on the same leaderboard would misrepresent both their intended role and the capabilities the benchmark is meant to assess.

Benchmark access models

Open release maximizes reproducibility but exposes ground truth to training-corpus contamination, which harms the validity of benchmarks over time [22]. Several evaluation efforts therefore withhold some or all of their data: FrontierMath keeps a private holdout set [8], ARC benchmarks maintain private evaluation splits [5], and credentialed-access corpora such as MIMIC gate data behind identity verification, thorough review and is restricted under usage agreements [12]. TAKEOFFBENCH-V1 adopts the same posture: the harness is public and the data is provided upon request after review (Section 4.5).

3. The Handoff-H1 system

3.1 Design overview

Handoff-H1 is organized as three layers over general-purpose base models (Figure 3). A perception layer recovers the typed visual structure from the raw drawing set. A knowledge layer—the project foundation—organizes what perception recovers, together with a curated construction knowledge base, into a persistent hierarchy. An orchestration layer runs vision agents against the foundation and audits their output. Each layer answers a distinct failure mode of general-purpose agents on this task: missed visual structure, missing domain conventions, and unmanaged context. None of the three is the differentiator on its own; the human-level performance on the benchmark (Section 5) is the product of their combination.

Figure 3: The Handoff-H1 architecture, at the level this report discusses it: Blueprint set (raw PDF) → Perception (purpose-built CV and VLMs) → Project foundation (hierarchical, consistency-resolved), grounded by a curated construction knowledge base and fed back by CV models as callable tools → Orchestrated estimation (vision agents) → Independent verification (source access) → Primary-material takeoff. Model identities, prompts, and orchestration specifics are withheld.

3.2 Perception: purpose-built vision models

Before any estimate is produced, a dedicated perception pass recovers typed primitives from each sheet: rooms classified by function; walls, doors, windows, and fixtures detected and labeled; project-level dimensions and scales; and region-level structure such as plan areas and drawing callouts (Figure 4). This layer is built from computer-vision and vision–language models specialized on construction drawings extraction. The same models serve multiple purposes in this pipeline: they are the front end of building the knowledge layer (Section 3.4), and they are exposed as callable tools to the estimation-stage agents (Section 3.3). Purely semantic retrieval reliably misses authoritative structural elements such as project-level dimensions or object counting; perception output is what makes those elements addressable and reliable for downstream tasks.

Figure 4: Reading a floor plan. The perception layer recovers typed primitives from each sheet: rooms classified by function (living, kitchen, garage, porch, bathroom), and walls, doors, windows, and fixtures detected and labeled with provenance. Discrete-count elements like these sit at the easy end of the difficulty spectrum; continuous-measurement trades (Section 6) are substantially harder.

3.3 Vision agents

The estimation stage is carried out by vision agents: tool-using agents [27] whose tool set spans two tiers. The first tier is generic image manipulation—cropping a region of a sheet, zooming to read a dimension string or a callout, panning across sheets—the operations any capable multimodal agent can be given. The second tier is task-specific: in-house tools that decompose a framing plan into its individual members, and tools that invoke the purpose-built computer-vision models of Section 3.2 to count, detect, and measure elements within a selected region. The agent decides when to trust its own reading of a drawing and when to delegate to a specialized tool; the second tier gives it capabilities—member-level framing decomposition, reliable counting over dense symbol fields—that prompting a general vision–language model does not provide.

3.4 Project foundation

The project foundation is a persistent, hierarchically structured knowledge representation built once per project and consumed on demand by everything downstream. Construction proceeds in three phases. Extraction performs dense semantic retrieval over the perception output, surfacing dimensional measurements, specification text, and visual structure with explicit provenance and authority tags. Synthesis partitions extracted content by construction clusters and grounds each partition against the curated knowledge base; the partition plays the role of the bridge layer in the hierarchical-RAG sense [11]—a domain-meaningful intermediate granularity at which retrieval and reasoning proceed without cross-cluster context contamination. Consistency resolution responsible for finding conflicts within partitions and solving them.

The curated knowledge base that grounds synthesis follows the LLM-Wiki pattern [14]: construction reference materials, estimating guides, material catalogs, and pricing data curated by domain experts into structured, searchable entries. The curation encodes logic that experienced estimators carry implicitly and it is continuously built and improved with information such as assembly conventions, default specifications, code-driven defaults—that no amount of open-domain nor in-context retrieval supplies. Its contents, sources, and curation methodology remain proprietary. Once constructed, the foundation is persisted: estimation runs consume it without re-processing the blueprint.

3.5 Orchestration and verification

Each vision agent works within its given scope with foundation queries and visual tools, rather than one agent attempting the whole project in a single context. An independent verification pass then audits the assembled takeoff with direct access to the source drawings—a self-refinement architecture [19] adapted to retrieval-grounded outputs—catching missed items and implausible quantities before the takeoff is finalized. The output is a structured set of primary-material line items: trade section, material description, quantity, and unit of measure. Stage decomposition, routing, and verification logic are not disclosed.

4. The Construction Blueprint Takeoff Benchmark

Handoff-H1 is evaluated on TAKEOFFBENCH-V1, a benchmark we built for exactly this task and describe here in full: the dataset, the consensus process behind its ground truth, the scoring protocol, and the access model.

4.1 Dataset

TAKEOFFBENCH-V1 comprises 10 residential blueprint sets drawn from real projects that Handoff is permissioned to use. Each set is a complete drawing package—floor plans, foundation plans, elevations, wall sections, and roof plans—representative of the segment that constitutes the majority of estimation volume. Every set is stripped of personally identifying information (owner and firm names, addresses, professional stamps, and project identifiers) by an automated detection pass followed by human review. The ground truth is a clean, line-item schema: trade section, material description, quantity, and unit of measure. Every set is evaluated across nine trades: Concrete, Framing, Roofing, Siding, Insulation, Drywall, Electrical, Plumbing, and Windows & Doors.

The ground truth is deliberately limited to primary materials—the framing members, sheets, and structural quantities that dominate cost—and excludes secondary items (minor fasteners, incidental accessories) and labor lines. This is a substantive design choice, not an oversight: secondary quantities are largely preference-driven and not worth reconciling to a single number, so including them would inject noise into both the ground truth and the scoring. Restricting the benchmark to primary materials keeps the gold standard verifiable and the comparison fair.

Ten sets understate the evaluation's size. Each drawing package decomposes into hundreds of images and accompanying text, and evaluation is item-level: the reconciled ground truth across the benchmark carries 2,009 verified line items, 1,348 of them primary-tier and in scope for scoring. A drawing set is therefore not one task but hundreds—every primary item is its own find, measure, and specify problem, judged individually.

4.2 Ground truth and human performance

A takeoff benchmark is only as trustworthy as its gold standard, and a single estimator's opinion is not a defensible one. Each ground-truth takeoff is therefore a reconciled, multi-reviewer artifact. Independent professional estimators, each with at least five years of experience, were engaged and paid to produce complete takeoffs from each drawing set according to a fixed instruction set. An internal team of construction experts then reviewed each submission in two stages. They first audited structure—that the trades, scopes, naming conventions, and overall organization matched the benchmark's guidance; agreement at this level defines what we call human performance for the task. They then audited coverage and precision, making corrections by hand, without AI assistance. A takeoff accepted by at least two construction experts and at least two independent estimators becomes the inter-estimator-approved takeoff, and that artifact is the gold standard for each drawing set. The independent submissions — with naming and structure convention —, scored by the benchmark's own judge system against the inter-estimator-approved takeoff, place human performance at 65.5% coverage and 87.9% Precision@25%—a realistic reference that sits below 100% on the continuous-measurement trades where estimators legitimately differ, and that Section 5.2 compares against Handoff-H1 directly.

4.3 Metrics

Predictions and ground truth are grouped by trade, and an LLM judge (OpenAI gpt-5.5)—now a standard, low-cost alternative to exact-match scoring for structured outputs—matches predicted items to ground-truth items within each trade, allowing many-to-many groupings so that several predicted rows can be reconciled against one ground-truth assembly. Only material line items participate; labor and non-material rows are excluded. Judging is stochastic, so no reported number rests on a single sample: every prediction set is judged ten times and the reported metrics are averages across judge runs. From the matches we compute two metrics. Coverage is the share of ground-truth primary items for which the model produced a matching prediction; it measures completeness. An uncovered item is not necessarily an outright omission: the judge matches on construction specification, so a prediction that names the wrong spec leaves the ground-truth line uncovered—5/8″ drywall does not match a 1/2″ ground-truth item, and plain 1/2″ does not match the moisture-resistant board a bathroom calls for. Quantity Precision@25% (P@.25) is the share of matched items whose predicted quantity falls within 25% of the approved quantity. A single composite combines them,

Precision and coverage are combined multiplicatively so that the composite score rewards estimates that are both comprehensive and accurate. Precision alone is trivially gamed: a model that reports a single correct line item and omits the remainder of the takeoff would post 100%. Scaling by coverage removes this degenerate case, since a score can only be high when most of the takeoff is recovered and the recovered items are correct.

The exponents weight precision more heavily than coverage. This asymmetry is deliberate. The ground truth is more reliable about the items it contains than about the items it omits, an uncovered item may reflect a gap in the gold standard rather than a genuine omission, whereas a wrong quantity on a matched item is an unambiguous failure. Coverage misses are therefore penalized more leniently than precision errors.

Precision is scored against a 25% tolerance rather than a tighter band. Differences within 10% are common even when a single professional remeasures their own prior work, particularly for items quantified by linear foot or square foot, where ordinary measurement variance can account for a discrepancy of that magnitude. A 10% threshold would consequently penalize measurement noise rather than genuine error.

The four measurement types—count, linear, area, and volumetric—are collapsed into a single score rather than reported individually. This simplifies comparison across runs, at the cost of overpenalizing counts: an off-by-one error typically exceeds the 25% tolerance, so counting rarely falls within the band that dimensional measurements comfortably occupy.

4.4 Evaluation protocol

Every system starts from the same input: the blueprint PDF and nothing else. Each baseline model runs as an agent that must emit a structured set of line items per trade. Because some language models are not natively multimodal, the harness provides a multimodality equalizer: for those models, regions of interest are supplied as text descriptions together with a visual-analysis tool (backed by Gemini 3.1 Pro) that answers targeted visual questions about a region on demand, so the comparison turns on reasoning rather than image support. Baseline agents are run up to three times per blueprint set and their scores averaged. Handoff-H1 takes no assistance from the harness: it runs end-to-end from the raw PDF, since the perception layer is part of the system being measured. The independent estimators' takeoffs (Section 4.2) are scored by the identical judge against the same approved ground truth, which is what makes the three-way comparison of Section 5 comparable.

4.5 Availability and access

The evaluation harness is public. TAKEOFFBENCH-V1 is packaged for the open harbor framework under Apache-2.0: each task is a (blueprint, trade) pair in which the agent writes a predictions file and a separate verifier runs the judge against the held-out ground truth and emits a per-task reward (Appendix A).

The blueprint sets and ground-truth takeoffs are available upon request. Researchers and practitioners can request access through the benchmark repository¹ or by writing to research@handoff.ai; requests are reviewed before the data is provided. We adopt this access model for one reason above others: a benchmark whose gold standard circulates in web crawls eventually measures memorization rather than capability [22], and expert-consensus takeoff ground truth is too expensive to burn that way. Withholding evaluation data behind a request step is by now standard practice [5, 8, 12]; it also lets us honor the terms under which the underlying projects were permissioned. Leaderboard submissions remain open—request the data, run the public harness, and submit results—and we welcome research partnerships and individual researchers interested in the problem.

What the evaluation captures—and what it does not. Two caveats bound the interpretation of TAKEOFFBENCH-V1 scores. First, matching is performed by an LLM judge: it tolerates differences in labeling conventions that exact-match scoring would miss, but it is not a substitute for a human adjudicator on genuinely ambiguous rows—and averaging ten judge runs controls sampling variance, not any systematic bias the judge may carry. Second, Precision@25% is measured against a single reconciled quantity, yet estimators themselves disagree on continuous-measurement trades—whether a dimension string includes adjacent wall thickness, how a sloped roof reconciles against its plan footprint—so the achievable ceiling there is below 100% even for careful humans (Section 4.2).

5. Results

This section reports the complete evaluation over the 10 benchmark sets: the main comparison against seven frontier and open-weight models (Section 5.1), the comparison against independent professional estimators (Section 5.2), and the per-trade analysis of where each excels (Section 5.3).

5.1 Main comparison

Table 1 reports the main comparison: seven frontier and open-weight models, each run as a tool-using agent from the raw PDF, scored under the identical per-trade judge against Handoff-H1 and against the independent professional estimators of Section 4.2. Figure 5 shows the composite ranking and the coverage and precision underlying it.

No frontier agent reaches the estimators. The seven models span composites of 34.7 to 61.4, all below the independent-estimator reference of 77.6. The strongest is claude-fable-5 (61.4), an Anthropic Mythos-class model positioned above the Opus tier [2]; gpt-5.6 follows at 58.6. The closed models hold the top of the field, but the edge is not categorical: the strongest open-weight model (Kimi-K2.6, 54.5) is level with claude-opus-4-8 (54.6). Base-model choice moves a takeoff agent by tens of points—and still leaves the best of them sixteen short of a professional estimator.

Handoff-H1 opens a 20-point gap from identical inputs. Handoff-H1 reaches 81.6 composite, roughly 20 points above the best frontier agent, from exactly the same starting point: the raw PDF. The gap is not bought with a stronger base model—the frontier field's internal spread shows how little that buys on this task; it is the perception, knowledge, and orchestration layers that the baselines do not have (Section 3).

Frontier agents see more than they can measure. The failure profile of the frontier field is one-sided (Figure 5): coverage exceeds precision for every one of the seven models—claude-fable-5 finds 77.3% of the primary items but quantifies only 52.8% of its matches within the 25% band. The bottleneck for a capable general agent is no longer locating the scope of a takeoff; it is the dimensional reasoning that turns a found item into a defensible quantity. The human estimators fail in exactly the opposite direction (65.5 coverage, 87.9 precision), which sets up the comparison of Section 5.2.

5.2 Comparison with professional estimators

The practical baseline for a takeoff system is not a language model; it is a professional estimator. We therefore compare Handoff-H1 with the held-out independent expert takeoff for each drawing set (Section 4.2), both scored by the same judge against the same approved ground truth.

Handoff-H1 posts the higher composite on six of the ten sets, the estimators on four (one of them a half-point wash). The mechanics behind the split are consistent. The estimators' losses are coverage. Their takeoffs often missed primary items that impacted coverage largely on Structural Framing members - such as joists, rafters and beams - and items that are only called-out on drawing details such as drywall sheets of project-specified thickness and rebar quantities, which require reading complex details and combining details with foundation plans. Handoff-H1 had the edge on this front largely due to completeness self-revision when performing takeoffs (Section 3.5). The estimators win on precision. On the sets they win, they measure what they take off at 89% P@.25 or better, and the single best takeoff in the pool is human work. But the two differ in consistency as much as in average score: Handoff-H1's per-set composite stays within a nine-point range, while the estimators' swings more than thirty. Humans produce both the best takeoffs and the worst; the system gives up the high peak in exchange for a floor the human pool never reaches. For takeoffs that feed downstream estimates, that steadiness matters more than an occasional high score.

5.3 Per-trade quality

Table 2 breaks the composite down by trade for all nine systems, and Figure 6 draws the three-way comparison—Handoff-H1, the independent estimators, and the strongest frontier agent—as a radar over the nine trades.

Handoff-H1 leads five of the nine trades; the independent estimators lead on Siding decisively and on Electrical and Windows & Doors by small margins; Concrete is tied; no frontier agent leads any trade.

Three patterns organize the field. First, counting-like trades are where everyone converges: Plumbing is the best or near-best trade for every system in the table—fixtures are enumerable—and the margins there are small (Handoff-H1 94, estimators 89, claude-fable-5 90); Windows & Doors behaves the same way, with the estimators (89) a point ahead of H1 (88). Second, derivation-heavy trades are where Handoff-H1 pulls away from both references: its four largest margins over the estimators are Drywall (+23), Insulation (+14), Framing (+11, the trade its member-level decomposition tooling of Section 3.3 targets directly), and Roofing (+10)—exactly the trades whose quantities and item specifications must be derived from areas, pitches and multi-sheet specification reasoning rather than read off single plans, and exactly where the estimators' skipped scopes concentrate. The frontier agents bottom out on the same trades (no model above 57 on Drywall or 54 on Framing), so the derivation problem is not solved by scale. Third, the estimators keep genuine territory: Siding is their decisive win (82 against 65, Handoff-H1's weakest trade), they edge the system on Electrical (85 against 81) and Windows & Doors, and with Concrete tied between H1 and estimators. Among the frontier agents the per-trade ranking reshuffles—Kimi-K2.6 overtakes both Claude models on Framing and Drywall, claude-sonnet-4-6 leads the models on Electrical—evidence that they fail in different places; with gpt-5.6 and fable-5 standing standouts primarily on precision amongst the other models tested.

6. Failure modes and limitations

Precision trails coverage. At 86.1% coverage and 78.8% P@.25, the system's remaining errors concentrate on quantities rather than omissions or spec-level errors: more than one matched quantity in five still falls outside the 25% band, where the human estimators hold 87.9%. The asymmetry is structural. Coverage failures are discrete—an item omitted outright, or taken off under the wrong specification and left unmatched (Section 4.3)—and respond to orchestration, verification, and knowledge grounding; precision depends on dimensional reasoning, unit conversions, and trade conventions that are intrinsically harder to fix from the estimation stage alone, this is why improvements to the curated knowledge base and to the vision components are a continuous task.

Continuous-measurement trades remain the hard core. Trades measured in linear or square feet (framing, siding, roofing) show consistently higher error rates than discrete-count trades (doors, windows, fixtures), in Handoff-H1 and in every baseline (Table 2). Wall lengths must be inferred from floor-plan dimensions, requiring resolution of ambiguities—whether a dimension line includes or excludes adjacent wall thickness—that even human estimators debate. Roofing compounds the problem: the area billed is the sloped area, which the plan view foreshortens, so the quantity must be recovered from pitch and framing information rather than read off the footprint (Figure 7). These are the trades where the human reference itself sits furthest below 100% (Section 4.2)—and Siding, the one trade where the estimators beat Handoff-H1 decisively (Section 5.3), is the system's clearest open problem in this family due to the possible variations in drawing convention - where materials do not follow a clear visual convention thus making it harder to treat it as a computer vision task.

Figure 7: A roof plan with surfaces broken out by plane and pitch. Roofing is sold in squares, and the true area depends on a slope the plan view hides: the billed quantity must be recovered from pitch and framing information rather than read off the flat footprint. Continuous-measurement trades like this are where estimators legitimately disagree, and where both Handoff-H1 and the human ceiling fall furthest from a single gold number.

Knowledge-base maintenance. The curated construction knowledge base requires ongoing maintenance as building codes evolve, material standards change, and pricing data ages. Regional variation in construction practice is a particular challenge.

Evaluation scope. Results are reported on residential blueprints. Commercial construction, multi-story buildings, and projects with partial or inconsistent drawing sets may behave differently and remain to be evaluated.

7. Discussion and next steps

Human comparison. The practical baseline for this task is a professional estimator, and Section 5.2 now measures against one directly: on this benchmark Handoff-H1 reaches human-level performance and surpasses it on several fronts—the composite itself (81.6 against 77.6 on average), six of the ten sets, and five of the nine trades. The overall quality of the takeoff is higher because the system covers more ground, and covering more ground is the harder road: every additional item quantified is another item scored, so Handoff-H1's 78.8% precision is earned over roughly a third more matched items than the estimators' 87.9%, which is computed only over the items and specs that are accurately detected and named.

The gap against frontier models. Every baseline starts from exactly what Handoff-H1 starts from—the raw PDF—and the 20-point gap to the strongest frontier agent persists. This locates the difference not in any single ingredient but in the combination the system is built around: perception the agents can call on demand, knowledge partitioned the way estimators actually work, and orchestration that keeps each trade's reasoning scoped and verified.

Next steps. We will keep the comparison current as new frontier models ship, extend evaluation beyond the residential segment, and broaden the estimator pool so the human reference reflects more than one professional takeoff per set. Requests for benchmark access, leaderboard submissions, research partnerships, and individual collaborations are welcome at research@handoff.ai.

8. Conclusion

We presented Handoff-H1, a generally available system for material quantity takeoff from construction blueprints, and the benchmark we built to measure it. The system's design commitment is combination over silver bullet: purpose-built vision models, vision agents with task-specific visual tools, a persistent partitioned project foundation grounded in a curated knowledge base, and orchestrated estimation with independent verification. On TAKEOFFBENCH-V1—10 permissioned residential drawing sets with consensus-validated, primary-only expert takeoffs—Handoff-H1 reaches an 81.6% composite (86.1% coverage, 78.8% P@.25) from the raw PDF, roughly 20 points above the strongest of seven frontier and open-weight agents run from the same input, and above the independent professional estimators scored against the same gold standard (77.6%), whose takeoffs are often more precise but far less thorough. The evaluation harness is public; the data is available upon request, an access model chosen so that the benchmark keeps measuring capability rather than memorization. We invite the community to request access, submit results, and pressure-test the gap.

Appendix A — Running the benchmark with harbor

The evaluation harness runs on the open harbor framework and is public under Apache-2.0; the blueprint sets and ground truth are delivered upon an approved access request (Section 4.5). Each task is a single (blueprint, trade) pair: an agent emits predicted line items and a verifier grades them against the held-out ground truth.

# clone the public harness
git clone https://github.com/handoffai/residential-takeoff-benchmark

# place the requested data under tasks/ once access is granted

# run one (blueprint, trade) task on the raw-PDF track
harbor run -p tasks/<case>-<trade> -a handoff -m <provider>/<model>

# aggregate per-task rewards into the leaderboard
harbor report

References

1. Mohamed Abouagour and Eleftherios Garyfallidis. Resplan: A large-scale vector-graph dataset of 17,000 residential floor plans, 2025. URL https://arxiv.org/abs/2508.14006.
2. Anthropic. Claude Fable 5 and Mythos 5. Anthropic news post, 2026. URL https://www.anthropic.com/news/claude-fable-5-mythos-5. Launched 2026-06-09.
3. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL https://arxiv.org/abs/2310.11511.
4. Bluebeam, Inc. Bluebeam Max. Product page, 2026. URL https://www.bluebeam.com/bluebeam-max/. Accessed 2026-07-14.
5. François Chollet. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911.01547.
6. Zhiwen Fan, Lingjie Zhu, Honghua Li, Xiaohao Chen, Siyu Zhu, and Ping Tan. Floorplancad: A large-scale cad drawing dataset for panoptic symbol spotting, 2021. URL https://arxiv.org/abs/2105.07147.
7. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024. URL https://arxiv.org/abs/2312.10997.
8. Elliot Glazer, Ege Erdil, Tamay Besiroglu, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2025. URL https://arxiv.org/abs/2411.04872.
9. Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. Lightrag: Simple and fast retrieval-augmented generation, 2025. URL https://arxiv.org/abs/2410.05779.
10. Haoyu Han, Yu Wang, Harry Shomer, et al. Retrieval-augmented generation with graphs (graphrag), 2025. URL https://arxiv.org/abs/2501.00309.
11. Haoyu Huang, Yongfeng Huang, Junjie Yang, Zhenyu Pan, Yongqiang Chen, Kaili Ma, Hongzhi Chen, and James Cheng. Retrieval-augmented generation with hierarchical knowledge, 2025. URL https://arxiv.org/abs/2503.10150.
12. Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, et al. Mimic-iii, a freely accessible critical care database. Scientific Data, 3(1):160035, May 2016. doi: 10.1038/sdata.2016.35.
13. Ahti Kalervo, Juha Ylioinas, Markus Häikiö, Antti Karhu, and Juho Kannala. Cubicasa5k: A dataset and an improved multi-task model for floorplan image analysis, 2019. URL https://arxiv.org/abs/1904.01920.
14. Andrej Karpathy. Llm wiki. GitHub Gist, April 2026. URL https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f. Accessed 2026-07-17.
15. Om Khangaonkar, Hadi J. Rad, and Hamed Pirsiavash. Multimodal language models cannot spot spatial inconsistencies, 2026. URL https://arxiv.org/abs/2604.00799.
16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://arxiv.org/abs/2005.11401.
17. Fenfen Lin, Yesheng Liu, Haiyu Xu, et al. Do vision-language models measure up? benchmarking visual measurement reading with measurebench, 2026. URL https://arxiv.org/abs/2510.26865.
18. Ruifeng Luo, Zhengjie Liu, Tianxiao Cheng, et al. Archcad-400k: A large-scale cad drawings dataset and new baseline for panoptic symbol spotting, 2025. URL https://arxiv.org/abs/2503.22346.
19. Aman Madaan, Niket Tandon, Prakhar Gupta, et al. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv.org/abs/2303.17651.
20. Lukas Petersson, Axel Backlund, Axel Wennstöm, Hanna Petersson, Callum Sharrock, and Arash Dabiri. Blueprint-bench: Comparing spatial intelligence of llms, agents and image models, 2025. URL https://arxiv.org/abs/2509.25229.
21. Alireza Rezvanifar, Melissa Cote, and Alexandra Branzan Albu. Symbol spotting on digital architectural floor plans using a deep learning-based framework, 2020. URL https://arxiv.org/abs/2006.00684.
22. Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark, 2023. URL https://arxiv.org/abs/2310.18018.
23. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. Raptor: Recursive abstractive processing for tree-organized retrieval, 2024. URL https://arxiv.org/abs/2401.18059.
24. Hosein Taghaddos, Ali Mashayekhi, and Behnam Sherafat. Automation of construction quantity take-off: Using building information modeling (bim). In Construction Research Congress 2016, page 2218–2227. ASCE, May 2016. doi: 10.1061/9780784479827.221.
25. Shijie Wu, Ozan Irsoy, Steven Lu, et al. Bloomberggpt: A large language model for finance, 2023. URL https://arxiv.org/abs/2303.17564.
26. Yanzhao Wu, Lufan Wang, and Rui Liu. Cequest: Benchmarking large language models for construction estimation, 2025. URL https://arxiv.org/abs/2508.16081.
27. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https://arxiv.org/abs/2210.03629.
28. Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. Raft: Adapting language model to domain specific rag, 2024. URL https://arxiv.org/abs/2403.10131.

¹ https://github.com/handoffai/residential-takeoff-benchmark

Handoff-H1 is what already powers takeoffs inside the Handoff product, and it's now available via API on the API plan.