Within the broader rights metadata mapping and licensing automation pipeline, this stage owns a single, high-consequence number: the confidence cutoff above which an asset publishes as public domain without a curator ever seeing it. Copyright status in a museum collection is rarely a clean flag at ingestion. It emerges from probabilistic scoring across creation dates, jurisdictional life-plus terms, author mortality data, and institutional provenance. Threshold tuning is the deterministic decision layer that turns that score into one of three destinations — open publication, manual review, or a fallback queue for ambiguous records — and it is the layer where a value set one point too low quietly exposes an in-copyright work to the world.

This page specifies the score contract that feeds the cutoff, the penalty arithmetic that keeps borderline assets out of the open tier, the dynamic US publication boundary that must never be hard-coded, and the calibration and audit machinery that lets legal counsel move the cutoff without a pipeline redeploy. The governing rule inherited from the parent gate holds here too: when the evidence is thin, the pipeline defers to a human, never to open access. A false “in copyright” verdict merely delays a scan; a false “public domain” verdict is an unauthorized disclosure the institution cannot take back.

Workflow Context

The threshold layer sits downstream of automating copyright status checks, which hands over a validated record and a raw confidence signal, and upstream of the IIIF manifest generation and delivery endpoints that publish the surviving assets. Its job is not to compute copyright from scratch — that arithmetic lives in the status engine — but to decide how much confidence is enough. Two institutions running identical rule code will legitimately choose different cutoffs: a university gallery digitizing pre-1900 prints tolerates a lower bar than a photography archive full of mid-century material with tangled authorship.

The role that owns this stage is a Python automation engineer encoding policy set by a collections manager and, often, legal counsel. The engineer’s obligation is a strict separation of two concerns. The scoring engine aggregates normalized date vectors and jurisdictional rules into a single confidence metric and knows nothing about routing. The routing engine applies the institution’s cutoff to that metric and knows nothing about how it was computed. Keeping these apart means counsel can raise the public-domain bar for the whole collection by editing one externalized parameter, and the change is auditable without reading a line of scoring logic.

Assets that clear the cutoff flow to open publication; those that fall short but stay plausible route to a review queue; those with genuinely insufficient data — no death date, contested authorship, a recent acquisition — drop into the fallback path examined under handling orphan works in digital collections. Records that score below the open bar for a licensable reason rather than an evidentiary one are handed sideways to routing Creative Commons licenses for an alternative assignment.

Prerequisites

Before wiring the threshold layer, confirm the following are in place:

  • Python 3.9+ for the PEP 604 | union syntax and the asyncio batch features used below.
  • pydantic (v2) for the score contract — this page uses the v2 API (Field, model_dump, ConfigDict); see the pydantic v2 documentation for constraint syntax.
  • Normalized input records. The scoring engine assumes dates and jurisdiction codes have already been resolved upstream; the field mapping that produces them is the subject of the parent taxonomy layer’s LIDO-to-database mapping.
  • An externalized cutoff configuration. Thresholds must live outside the deployment artifact — in a version-controlled config store legal counsel can edit — not as constants recompiled into the image.
  • A jurisdictional term source. Statutory durations must be referenced from an authoritative table, not frozen in code; the US durations are published by the U.S. Copyright Office.
  • An append-only audit sink reachable at decision time, so every input vector, score, cutoff hash, and routing outcome is recorded for compliance replay.
  • Interchange standards for the surviving payload. Public-domain verdicts write the LIDO schema lido:rights element and the IIIF Presentation API 3.0 rights property before manifest generation.

Score & Threshold Reference

The score contract is the single point of truth for what the cutoff may reason about. A strict Pydantic v2 model bounds the raw confidence to [0.0, 1.0], constrains the jurisdiction to a two-letter code, and carries the provenance flags that drive penalties. The threshold configuration is a separate, immutable object so that routing parameters can never be mutated at runtime.

Field Type Constraint Purpose
asset_id str required Idempotency and audit key
creation_year int | None optional US publication-term input
author_death_year int | None optional Life-plus decay input
jurisdiction str ^[A-Z]{2}$ Selects the rule branch
raw_confidence_score float 0.0 ≤ s ≤ 1.0 Upstream status signal
provenance_flags list[str] default empty Penalty triggers

The routing destinations form a closed, three-value set, each derived from the score’s position relative to the cutoff:

Destination Condition Downstream effect
public_domain score ≥ cutoff Open publication, permissive rights
manual_review cutoff × ratio ≤ score < cutoff Curator confirms before release
orphan_work_queue score < cutoff × ratio Fallback evaluation, never open

Step-by-Step Implementation

Production pipelines batch thousands of records asynchronously; blocking I/O or an exhausted connection pool collapses throughput. The stack below defines the immutable configuration, the score contract, the penalty arithmetic, and an async worker pool that routes each record, presented in turn.

1. Define the immutable configuration and score contract

The configuration is a frozen dataclass so a runaway task cannot mutate the cutoff mid-batch. The payload is a strict Pydantic v2 model that rejects an out-of-range confidence or a malformed jurisdiction code at construction, before any arithmetic runs.

python
from __future__ import annotations
import asyncio
import logging
from dataclasses import dataclass
from datetime import date
from enum import Enum
from pydantic import BaseModel, Field

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger(__name__)

class RoutingDestination(str, Enum):
    PUBLIC_DOMAIN = "public_domain"
    MANUAL_REVIEW = "manual_review"
    ORPHAN_WORK_QUEUE = "orphan_work_queue"

@dataclass(frozen=True)
class ThresholdConfig:
    pd_confidence_cutoff: float = 0.92
    manual_review_ratio: float = 0.75  # fraction of the cutoff that still warrants review
    batch_size: int = 250
    max_retries: int = 3
    retry_delay: float = 1.5

class AssetRightsPayload(BaseModel):
    asset_id: str
    creation_year: int | None = None
    author_death_year: int | None = None
    jurisdiction: str = Field(pattern=r"^[A-Z]{2}$")
    raw_confidence_score: float = Field(ge=0.0, le=1.0)
    provenance_flags: list[str] = Field(default_factory=list)

The frozen=True on ThresholdConfig is not cosmetic: it turns any accidental reassignment of the cutoff into a FrozenInstanceError at the moment of the bug rather than a silently drifted threshold three thousand records later.

2. Score the record — penalties applied to the raw confidence

This is the correctness heart of the layer. Provenance risk is subtracted from the raw confidence, so a borderline asset accumulates risk and slides toward review; it does not trip a separate binary check that ignores its score. The US publication boundary is derived from the current year, never hard-coded — the 95-year term advances every January 1, so a constant is wrong twelve months after it is written.

python
def compute_threshold_score(payload: AssetRightsPayload, config: ThresholdConfig) -> float:
    """Apply jurisdictional decay and provenance penalties to raw confidence."""
    base = payload.raw_confidence_score
    penalty = 0.0

    # The US publication cutoff advances every January 1 (95-year term), so
    # derive it from the current year rather than hard-coding a constant.
    us_pd_cutoff = date.today().year - 96  # latest publication year now in the US public domain

    if "unverified_author" in payload.provenance_flags:
        penalty += 0.15
    if payload.author_death_year is None and payload.creation_year and payload.creation_year > us_pd_cutoff:
        penalty += 0.10
    if payload.jurisdiction == "US" and payload.creation_year and payload.creation_year <= us_pd_cutoff:
        base = min(base + 0.05, 1.0)

    adjusted = max(base - penalty, 0.0)
    return round(adjusted, 3)

Note the asymmetry that keeps the layer fail-safe: penalties can only lower a score, and the sole bonus — a small lift for a US work already past the rolling cutoff — is clamped to 1.0. An unverified author strips fifteen points; a missing death date on a recent work strips ten. A work with both problems can never claw its way back above a well-set cutoff.

3. Route the record against the cutoff

Routing reads the score and nothing else. The three-way comparison places each asset in exactly one destination, with the review band defined as a fraction of the cutoff rather than a second free-floating constant — move the cutoff and the review band moves with it.

python
async def route_asset(payload: AssetRightsPayload, config: ThresholdConfig) -> RoutingDestination:
    score = compute_threshold_score(payload, config)

    if score >= config.pd_confidence_cutoff:
        return RoutingDestination.PUBLIC_DOMAIN
    elif score >= config.pd_confidence_cutoff * config.manual_review_ratio:
        return RoutingDestination.MANUAL_REVIEW
    else:
        return RoutingDestination.ORPHAN_WORK_QUEUE

4. Drive the batch through an async worker pool

The batch evaluates every record concurrently and collects results with asyncio.gather(..., return_exceptions=True) so a single poisoned record cannot cancel its siblings. Any record that raises falls back to manual_review — the safe direction — rather than vanishing or defaulting to open.

python
async def process_batch(
    payloads: list[AssetRightsPayload],
    config: ThresholdConfig,
) -> dict[str, RoutingDestination]:
    results: dict[str, RoutingDestination] = {}
    tasks = [route_asset(p, config) for p in payloads]
    routed = await asyncio.gather(*tasks, return_exceptions=True)

    for payload, outcome in zip(payloads, routed):
        if isinstance(outcome, Exception):
            logger.error("Routing failed for %s: %s", payload.asset_id, outcome)
            results[payload.asset_id] = RoutingDestination.MANUAL_REVIEW
        else:
            results[payload.asset_id] = outcome

    return results

Across input variants the contract holds: a CSV export with year columns as strings is coerced by Pydantic before scoring; an XML payload carrying a lowercase or three-letter jurisdiction is rejected at construction by the pattern constraint; and an API record missing both year fields scores low and lands in orphan_work_queue deterministically instead of raising.

Processing Logic

The score-then-route sequence is easiest to read as a branch diagram: one confidence value is computed once, then compared against the cutoff and the review band, with everything below the band deferring to the fallback path.

Score-then-route branch: one confidence value compared against the cutoff and the review band Left to right: an AssetRightsPayload feeds a compute_threshold_score box that subtracts penalties and applies the jurisdiction lift. Its output enters a diamond that compares the score to the cutoff, fanning out to three destinations — public_domain when the score is at or above the cutoff, manual_review when it is at or above the cutoff times the review ratio, and a dashed orphan_work_queue box (fail-safe, never open) when it falls below that band. AssetRights Payload compute_threshold_score penalties − · jurisdiction lift + → one value score vs cutoff public_domain open publication manual_review curator confirms first orphan_work_queue fail-safe · never open score ≥ cutoff ≥ cutoff × ratio < cutoff × ratio

Rights and Access Routing

The destination is not the end of the story — it is the access-tier signal every downstream consumer honours. A public_domain verdict injects http://creativecommons.org/publicdomain/mark/1.0/ into the record’s rights field and clears the asset for unrestricted IIIF tile serving. A manual_review verdict withholds the asset from anonymous access until a curator confirms it, and where that confirmation resolves to an open license rather than a public-domain mark, the record is handed to routing Creative Commons licenses. An orphan_work_queue verdict never becomes publicly resolvable; it follows the risk-tiered fallback under handling orphan works in digital collections. Records whose status is time-bound rather than confidence-bound — a term that lapses on a known future date — are scheduled through implementing embargo workflows instead of being forced through the cutoff.

Because the destination set is closed at three values, a downstream access-control layer can switch on exactly those three; a new tier cannot appear without a routing rule being added deliberately.

Calibration & Audit Trails

A static cutoff decays as legal interpretation evolves and collection composition shifts, so calibration runs as a continuous feedback loop rather than a one-time setting. Every curator override on a manual_review asset becomes a labelled example: an asset the curator opened teaches that the cutoff was slightly high for that jurisdiction, one they withheld teaches that it was low. The system tallies false positives and false negatives per jurisdiction and surfaces the drift so counsel can adjust the externalized cutoff on evidence rather than intuition.

Calibration feedback loop with config-hash-stamped audit records A clockwise loop across the top: scoring engine to routing to curator overrides to a per-jurisdiction false-positive and false-negative tally. The tally drops down to an externalized cutoff config box (versioned, effective date, config hash), which returns along the bottom to the scoring engine. Routing also records each decision into an audit record box, and a dashed arrow from the config box stamps the config hash onto that audit record. Scoring engine confidence metric Routing apply cutoff Curator overrides labelled examples FP / FN tally per jurisdiction Audit record vector · score · dest config-hash · UTC Externalized cutoff config versioned · effective date config hash records decision adjust cutoff stamp config hash reload — governs next batch

Every threshold change follows a documented change-management step: the version-controlled config records the old and new cutoff alongside an effective date, so a compliance audit can reconstruct exactly which value governed any historical decision and roll back if needed. The audit sink captures the full input vector, the computed score, the routing destination, the configuration hash, and a UTC timestamp for each record. When an asset transitions from restricted to open, that justification chain is what defends the decision if a rightsholder later objects.

Verification and Testing

The layer’s correctness rests on two properties that must be pinned by tests rather than trusted: penalties lower the score (they are not absolute gates), and the US cutoff is dynamic. The assert-based cases below exercise both, plus the fail-safe direction of the review band, and run under pytest -q or as a plain script.

python
from datetime import date

CONFIG = ThresholdConfig()

def test_penalty_lowers_score_not_gates_it():
    # An unverified author subtracts 0.15 from the raw score, it does not zero it.
    p = AssetRightsPayload(asset_id="T1", jurisdiction="FR",
                           raw_confidence_score=0.90, provenance_flags=["unverified_author"])
    assert compute_threshold_score(p, CONFIG) == 0.75

def test_us_cutoff_is_dynamic():
    # A US work published exactly at the rolling boundary gets the +0.05 lift.
    boundary = date.today().year - 96
    p = AssetRightsPayload(asset_id="T2", jurisdiction="US",
                           creation_year=boundary, raw_confidence_score=0.90)
    assert compute_threshold_score(p, CONFIG) == 0.95

def test_missing_death_year_penalizes_recent_work():
    recent = date.today().year - 96 + 5  # inside the US term, no death year
    p = AssetRightsPayload(asset_id="T3", jurisdiction="US",
                           creation_year=recent, raw_confidence_score=0.90)
    # 0.90 - 0.10 = 0.80, below the 0.92 cutoff but above 0.92*0.75 = 0.69 -> review
    assert compute_threshold_score(p, CONFIG) == 0.80

async def test_low_score_routes_to_fallback():
    p = AssetRightsPayload(asset_id="T4", jurisdiction="US",
                           raw_confidence_score=0.30)
    assert await route_asset(p, CONFIG) == RoutingDestination.ORPHAN_WORK_QUEUE

All four should pass. For a live dry run, call process_batch over a sample of real identifiers and tally the destinations without letting any dispatch reach a production endpoint — a pre-flight that catches a mistuned cutoff or a drifted export before it publishes anything.

In This Section

One focused guide goes deeper on the fallback path this layer feeds:

  • Handling Orphan Works in Digital Collections — the deterministic, memory-efficient fallback chain that isolates records whose rightsholder cannot be identified, applies jurisdictional risk thresholds, and keeps a bulk run uninterrupted instead of halting on the ambiguous records the cutoff defers.

FAQ

Why are penalties subtracted from the score instead of acting as their own pass/fail checks?

Because copyright confidence is cumulative, not categorical. A single provenance concern rarely settles a work’s status on its own, but two or three together should. Subtracting each penalty from the raw confidence lets those concerns accumulate, so a work with an unverified author and a missing death date slides below the cutoff even if either problem alone would not. A set of independent binary gates would either over-reject on any one flag or ignore how flags compound.

Why compute the US public domain cutoff at runtime rather than store a constant?

The cutoff is not a fixed year — it advances every January 1 as another year of works ages out of the 95-year publication term. Hard-coding “published before 1929” bakes in a value that is wrong twelve months later and silently withholds newly-public works from the open portal. Deriving date.today().year - 96 at evaluation time keeps the boundary correct without a redeploy; the full copyright arithmetic behind it lives in automating copyright status checks.

What cutoff value should an institution start with?

There is no universal number, which is why the cutoff is externalized rather than shipped. A collection dominated by clearly-old material tolerates a lower bar; one full of contested mid-century authorship needs a higher one. Start conservative — the 0.92 default errs toward review — then let curator overrides on the manual_review queue supply the labelled evidence to move it. Never lower the cutoff to clear a backlog; that trades an audit risk for throughput.

Where do records that fall below the review band go?

They route to orphan_work_queue and never become publicly resolvable on the strength of the score alone. That queue is not a rejection bin — it feeds the risk-tiered evaluation in handling orphan works in digital collections, which applies jurisdictional risk scoring and, where appropriate, a defensible diligent-search record before any release.

How is a threshold change made auditable?

Every cutoff lives in a version-controlled configuration with an effective date, and every decision record stamps the configuration hash that governed it. Together they let an audit replay any historical routing decision against the exact cutoff in force at the time, and roll a change back cleanly if a later legal interpretation demands it. The change itself follows a documented review step rather than an ad-hoc edit.