How accurate is AI HS code classification — and how would you know?

The short answer

The only number that matters is the digit depth

The Harmonized System is hierarchical. A code is built by descending a tree:

• 2 digits — chapter (e.g. 61, apparel and clothing accessories, knitted)
• 4 digits — heading
• 6 digits — subheading, the level the WCO harmonises internationally
• 8 and 10 digits — national tariff lines, where duty rates and measures actually attach

Duty is calculated at the deepest level you file at. An answer that is right at chapter level and wrong at ten digits is not partly right — it is a different code, a different rate, and potentially a different set of regulatory controls. Which is why a single "accuracy" figure without a stated digit depth is close to meaningless.

The published results show exactly how much the depth matters. HSCodeComp, a benchmark published in the ACL 2026 long papers, evaluated 23 systems on 632 e-commerce products requiring unique ten-digit codes, annotated by 26 domain experts:

The shape is the finding. Almost every system clusters around 78–82% at chapter level, and then the field falls apart. That is why "over 80% accuracy" is a claim you should treat as incomplete until someone tells you the depth.

What the benchmarks actually measure

This is the part that is almost never stated, and it matters more than the numbers.

That gap is not pedantry. A code can match the expert label and still be queried, amended or rejected by the authority that processes it, because acceptance depends on the declaration, the supporting documentation, the origin rules applied and the officer's judgement. Agreement and acceptance are different things, and only one of them clears your goods.

Three sources establish the current state, with different strengths:

• HSCodeComp — peer-reviewed at ACL 2026, 23 systems, 26 expert annotators. Authored by Alibaba Group, a commercial party in this space, which is worth knowing even though the methodology is open.
• A deterministic agentic workflow (arXiv preprint, 2026) — affiliations include the General Administration of Customs of the P.R.C. Reports 75.0% top-1 and 91.5% top-3 at four digits, and 64.2% top-1 and 78.3% top-3 at six digits. Notable for the customs affiliation, but a preprint, not an independent audit.
• A port-operations study (arXiv preprint, Pusan National University and CyberLogitec) — on data from an operational container terminal, zero-shot small models reached roughly 14% mean accuracy at six digits; their constrained framework reached 62.65%, best model 71.61%.

The second number in those results is the one procurement teams should ask for more often: top-3 containment. If the correct code is in the shortlist even when it is not ranked first, a reviewer confirms rather than researches. A system at 64% top-1 and 78% top-3 is a materially different tool from one at 64% top-1 with no containment figure published.

Why accuracy collapses as you go deeper

The benchmarks are unusually candid about the mechanism. The HSCodeComp authors observe that agents "maintain competitive accuracy at the 2-digit Chapter level (∼82%)" but "fail to navigate" country-specific classification at ten digits, and that "excessive reasoning steps often induce 'reasoning drift' that degrades accuracy."

Anyone who has classified goods will recognise the second finding. Long chains of inference are where a classifier talks itself into a neighbouring heading — the same failure mode a junior classifier has when they reason from the product's marketing name instead of its material and function.

The General Interpretative Rules are the framework that resolves this, and they are ordered for a reason: the terms of the headings and any relative section or chapter notes come first, and classification is determined by those before anything else is considered. A classifier that reasons from product description to plausible category, without anchoring on the heading text and the notes, will produce answers that look reasonable and are wrong at the depth that matters.

What a confidence score is for

Given the published numbers, the design conclusion is straightforward: automated classification is a triage system, not an oracle. The confidence score is the component that makes triage possible, and it is the thing to interrogate hardest.

A confidence score is only useful if it is calibrated — that is, if items scored at 90% are actually correct about 90% of the time. A score that merely ranks outputs by internal certainty, without calibration against outcomes, will still let a large volume of wrong codes through at the top of the range, because there is no evidence the top of the range means anything.

So the operational design looks like this:

• A high band that files automatically. Set it from measured calibration, not from a vendor's suggested default.
• A middle band that files with a shortlist. The reviewer sees the top candidates and confirms, rather than starting from nothing.
• A low band that goes to a human with the product data attached. These are the genuinely hard items — composite goods, kits, novel materials, anything where "what is it" is itself a question.
• A feedback loop. Every amendment and every authority query is a labelled example. A classifier deployed without capturing those is not improving.

The threshold is not a technical parameter. It is the point where the expected cost of a wrong code — duty adjustment, penalty exposure, delayed release, and the labour of correcting it — crosses the cost of a person reviewing it. That crossover is different for a low-value parcel and a high-duty industrial component, which is why a single global threshold is usually wrong.

What this means at your volume

Accuracy is easier to reason about as a workload than as a percentage. The planner below applies the published figures to your own monthly volume.

Classification workload planner

Enter your monthly classification volume and the loaded cost of a manual review. The accuracy figures are the published benchmark results cited above, not any product's claim.

Items classified per month

Cost per manual review (EUR)

Scenario	Accuracy	Auto-cleared	Human review	Review cost / month

Arithmetic on published benchmark figures. It models top-1 exact match only and assumes every non-match requires a human, which is conservative — a shortlist-based workflow reviews fewer. It says nothing about any specific product.

How to benchmark it on your own declarations

A vendor's benchmark, however well constructed, is not your data. The only figure that predicts your experience is one measured on your own goods, filed at your own depth, in your own destination markets. This is a few weeks of work and it is worth it before a multi-year commitment.

1. Pull a stratified sample of your own declared items. A few hundred is enough to see the shape if you stratify by chapter and by value rather than taking the most recent 300, which will over-represent whatever you ship most.
2. Get an independent expert code for each item, blind. The classifier must not see the expert answer, and the expert must not see the classifier's — otherwise you are measuring agreement between them, not accuracy against a reference.
3. Score at the digit depth you actually file at. If you file ten-digit codes, a six-digit result is not your number.
4. Record top-3 containment alongside top-1. It is the figure that predicts whether a reviewer confirms or researches.
5. Then check what customs did. Amend the sample with the outcome: accepted, queried, amended, rejected. This is the only part of the exercise that measures acceptance rather than agreement, and it is the number no vendor can give you.
6. Test the calibration of the confidence score. Bucket the results by confidence band and check the hit rate within each band. If the 90% band is right 60% of the time, the score is a ranking, not a probability, and no threshold set from it will behave as intended.

Publish nothing from this internally except the number your team will act on. The deliverable is a threshold and a review queue, not a report.

What to ask a vendor

These questions separate a system that has been measured from one that has been demonstrated:

• At what digit depth is your accuracy figure, and on how many items? A figure quoted without a depth is not a figure.
• Was the benchmark run by you or by someone independent? Self-reported numbers are not disqualifying — unlabelled ones are.
• What is your top-3 or top-5 containment? If it is not measured, the review workflow has no shortlist and every miss costs full research time.
• What does the confidence score mean, and how is it calibrated? Ask for the hit rate per confidence band, not a description of the model.
• What happens to a low-confidence item? A review queue with the product data attached is a feature; a lower-ranked guess is not.
• Does it cite the legal basis? The heading text, section and chapter notes, and the General Interpretative Rules applied. A code without its reasoning cannot be audited.
• How are country-specific suffixes handled? The international six digits are the easy part. The national line is where the duty rate lives.
• How do corrections feed back? If amendments do not become training signal, the system's accuracy on your goods is frozen at deployment.

Sources

• Lan, Yang, Jia et al., HSCodeComp: A Realistic and Expert-level Agent Benchmark for Hierarchical Rule Application, ACL 2026 (Long Papers) — aclanthology.org/2026.acl-long.937.pdf
• Zhang, Zhuang, Zhou et al., A Deterministic Agentic Workflow for HS Tariff Classification, arXiv:2605.14857 — export.arxiv.org/pdf/2605.14857
• Kim et al., Trie Constraints and Hierarchy-Aware Semantic Alignment for HS Code Prediction with Small Language Models, arXiv:2608.04464 — browse-export.arxiv.org/pdf/2608.04464
• World Customs Organization, What is the Harmonized System?