Operational Context

A rights manager inherits a legacy collection of 180,000 digitized objects whose copyright status lives in a free-text rights_note column and a scattering of internal codes, and the delivery portal now needs a machine-readable RightsStatements.org URI on every record before it can drive access. The task is a backfill: stamp the correct one of the twelve standard statements onto each object in one automated pass, without a registrar hand-typing 180,000 URIs and without a careless script flattening the handful of statements a curator already set by hand. This is the write stage that follows the read-side transformation in mapping RightsStatements.org URIs to collection fields: there, an incoming URI is canonicalized and classified; here, a URI is derived from the object’s copyright facts and written back. It sits inside the automating copyright status checks stage and feeds the same restricted-by-default contract the rest of the rights metadata and licensing automation pipeline enforces. The exact failure this page prevents: a blind UPDATE ... SET rights_uri = ... that runs against the whole table, overwrites curated values with a machine guess, and cannot be undone.

Root Cause Analysis

Three properties make a bulk stamp dangerous, and each maps to one guarantee the solution must provide.

First, the statement is not a single fact — it is a function of three. You cannot pick a URI from copyright status alone. An in-copyright work resolves to a different statement depending on whether a use restriction applies (InC-EDU, InC-NC) and whether the rightsholder is locatable at all (InC-RUU, or InC-OW-EU under the EU orphan-works register). A no-copyright work needs a reason: NoC-CR for contractual restrictions, NoC-OKLR for other legal ones, NoC-US only for United States works. And three statements — CNE, UND, NKC — encode grades of “not determined” that must never collapse into a positive public-domain claim. Deriving the right URI therefore requires copyright status, jurisdiction, and determinability together; drop any axis and the mapping is ambiguous.

Second, a blind write destroys curated values and is irreversible. Some records already carry a statement a curator set deliberately after a rights review. A bulk UPDATE that recomputes every row will silently replace that human decision with whatever the heuristic produces, and once the previous value is gone there is no diff to reconstruct it from. The write must treat an existing curated URI as read-only and must let the operator see every proposed change before a single byte is written.

Third, re-running must be safe. Backfills are never one-shot: a run is interrupted, a mapping rule is corrected, a subset is re-imported. If the job is not idempotent, the second run either double-applies or thrashes rows that are already correct. The remedy on all three fronts is the same shape used across the pipeline: a pure derivation function with no side effects, a plan phase that emits a diff, and an apply phase that writes only genuine changes and skips rows already at their target — the same fail-safe posture taken when handling orphan works in digital collections.

Canonical Solution

The design separates deciding from writing. derive_uri is a pure function mapping (status, jurisdiction, determinability) onto exactly one of the twelve URIs. plan_change compares that target against the record’s current value and classifies the row into one of four actions — assign, update, match, or skip-curated — without touching storage. apply_batch then performs the only side effect, writing the two actions that represent a real change and leaving everything else alone. Running the plan with dry_run=True (the default) yields a full diff and writes nothing.

python
from __future__ import annotations

from collections import Counter
from dataclasses import dataclass
from enum import Enum
from typing import Callable, Iterable, Iterator

from pydantic import BaseModel, ConfigDict, field_validator

# The complete RightsStatements.org 1.0 controlled vocabulary, keyed by token.
RS: dict[str, str] = {
    "InC":       "https://rightsstatements.org/vocab/InC/1.0/",
    "InC-OW-EU": "https://rightsstatements.org/vocab/InC-OW-EU/1.0/",
    "InC-EDU":   "https://rightsstatements.org/vocab/InC-EDU/1.0/",
    "InC-NC":    "https://rightsstatements.org/vocab/InC-NC/1.0/",
    "InC-RUU":   "https://rightsstatements.org/vocab/InC-RUU/1.0/",
    "NoC-CR":    "https://rightsstatements.org/vocab/NoC-CR/1.0/",
    "NoC-NC":    "https://rightsstatements.org/vocab/NoC-NC/1.0/",
    "NoC-OKLR":  "https://rightsstatements.org/vocab/NoC-OKLR/1.0/",
    "NoC-US":    "https://rightsstatements.org/vocab/NoC-US/1.0/",
    "CNE":       "https://rightsstatements.org/vocab/CNE/1.0/",
    "UND":       "https://rightsstatements.org/vocab/UND/1.0/",
    "NKC":       "https://rightsstatements.org/vocab/NKC/1.0/",
}
VALID_RS_URIS: frozenset[str] = frozenset(RS.values())


class Status(str, Enum):
    IN_COPYRIGHT = "in_copyright"
    NO_COPYRIGHT = "no_copyright"
    UNDETERMINED = "undetermined"            # assessed, evidence inconclusive
    NOT_EVALUATED = "not_evaluated"          # never assessed
    NO_KNOWN_COPYRIGHT = "no_known_copyright"  # concluded free, not certain


class Jurisdiction(str, Enum):
    US = "US"
    EU = "EU"
    OTHER = "OTHER"


class Restriction(str, Enum):
    NONE = "none"
    CONTRACTUAL = "contractual"
    NON_COMMERCIAL = "non_commercial"
    EDUCATIONAL = "educational"
    OTHER_LEGAL = "other_legal"


class Determination(BaseModel):
    # The three derivation axes, plus the guard that protects a human decision.
    model_config = ConfigDict(str_strip_whitespace=True)

    object_id: str
    status: Status
    jurisdiction: Jurisdiction = Jurisdiction.OTHER
    restriction: Restriction = Restriction.NONE
    rightsholder_locatable: bool = True
    rights_uri: str | None = None            # existing value; may be curated
    curated: bool = False                    # True once a human set it by hand

    @field_validator("rights_uri", mode="before")
    @classmethod
    def blank_is_none(cls, v: str | None) -> str | None:
        # Legacy exports store "", "n/a", or whitespace where a URI belongs.
        # Collapsing to None makes "has no statement yet" a single is-None test.
        if v is None:
            return None
        s = str(v).strip()
        return s or None


def derive_uri(rec: Determination) -> str:
    """Map (status, jurisdiction, determinability) onto exactly one of the 12 URIs."""
    match rec.status:
        case Status.NOT_EVALUATED:
            return RS["CNE"]                 # nobody has assessed the work yet
        case Status.NO_KNOWN_COPYRIGHT:
            return RS["NKC"]                 # reasonably concluded free, not certain
        case Status.UNDETERMINED:
            return RS["UND"]                 # assessed, but the evidence is silent
        case Status.IN_COPYRIGHT:
            if not rec.rightsholder_locatable:
                # An unlocatable rightsholder is an orphan work: the EU register
                # has a dedicated statement; every other jurisdiction uses RUU.
                if rec.jurisdiction is Jurisdiction.EU:
                    return RS["InC-OW-EU"]
                return RS["InC-RUU"]
            match rec.restriction:
                case Restriction.EDUCATIONAL:
                    return RS["InC-EDU"]
                case Restriction.NON_COMMERCIAL:
                    return RS["InC-NC"]
                case _:
                    return RS["InC"]
        case Status.NO_COPYRIGHT:
            match rec.restriction:
                case Restriction.CONTRACTUAL:
                    return RS["NoC-CR"]
                case Restriction.NON_COMMERCIAL:
                    return RS["NoC-NC"]
                case Restriction.OTHER_LEGAL:
                    return RS["NoC-OKLR"]
                case _:
                    # A bare "no copyright" is only assertable for US works.
                    # Elsewhere a reason is required, so fail safe to UND.
                    if rec.jurisdiction is Jurisdiction.US:
                        return RS["NoC-US"]
                    return RS["UND"]

The three “not determined” statuses map directly, the IN_COPYRIGHT branch refines on locatability then restriction, and the NO_COPYRIGHT branch refines on the legal reason then jurisdiction. Every path returns a member of VALID_RS_URIS, and the _ catch-alls guarantee the function is total — no input combination falls off the end without a URI. The decision tree below traces that selection.

Selecting one of the twelve RightsStatements.org statements from status, jurisdiction, and determinability Left to right: a record feeds a status decision diamond. Three branches lead straight to the not-determined statements CNE, UND, and NKC. The in-copyright branch reaches a refinement diamond that fans to five In Copyright statements. The no-copyright branch reaches a refinement diamond that fans to four No Copyright statements. The not-determined leaves are drawn dashed in the caution colour; the In Copyright leaves in the primary colour; the No Copyright leaves in the success colour. not evaluated undetermined no known © in copyright no copyright Record rights facts status? 5 outcomes CNE copyright not evaluated UND copyright undetermined NKC no known copyright InC use · locatable? InC default in copyright InC-EDU educational use permitted InC-NC non-commercial permitted InC-OW-EU EU orphan work InC-RUU rightsholder unlocatable NoC reason · US? NoC-CR contractual restrictions NoC-NC non-commercial only NoC-OKLR other legal restrictions NoC-US United States works

With the derivation isolated as a pure function, the write path can compare the target against what is already stored and decide — per row — whether a write is warranted at all. That comparison is the entire safety mechanism.

python
class Action(str, Enum):
    ASSIGN = "assign"          # no prior value -> write the derived URI
    UPDATE = "update"          # stale machine value -> overwrite and log it
    MATCH = "match"            # already correct -> write nothing (idempotent)
    SKIP_CURATED = "skip"      # a human set a different value -> never touch it


@dataclass(frozen=True)
class Change:
    object_id: str
    old: str | None
    new: str
    action: Action


def plan_change(rec: Determination) -> Change:
    derived = derive_uri(rec)
    current = rec.rights_uri
    if current == derived:
        action = Action.MATCH               # target already met; no-op on re-run
    elif current is None:
        action = Action.ASSIGN              # nothing to overwrite
    elif rec.curated:
        action = Action.SKIP_CURATED        # protect the curator's decision
    else:
        action = Action.UPDATE              # replace a stale derived value
    return Change(rec.object_id, current, derived, action)


WRITING = {Action.ASSIGN, Action.UPDATE}    # the only actions that touch storage


def plan_batch(rows: Iterable[dict]) -> Iterator[Change]:
    # Pure: yields the full diff for preview without writing anything.
    for raw in rows:
        yield plan_change(Determination.model_validate(raw))


def render_diff(changes: Iterable[Change]) -> str:
    lines = []
    for c in changes:
        if c.action is Action.MATCH:
            continue                        # unchanged rows stay out of the diff
        old = c.old if c.old is not None else "(empty)"
        lines.append(f"{c.action.value:6} {c.object_id}: {old} -> {c.new}")
    return "\n".join(lines)


def apply_batch(
    rows: Iterable[dict],
    write: Callable[[str, str], None],
    *,
    dry_run: bool = True,                    # preview by default; opt in to writes
) -> Counter:
    tally: Counter = Counter()
    for change in plan_batch(rows):
        tally[change.action.value] += 1
        if change.action in WRITING and not dry_run:
            write(change.object_id, change.new)   # the single side effect
    return tally

Because plan_change classifies a row already at its target as MATCH and apply_batch writes only ASSIGN and UPDATE, a second run over the same data writes nothing — the job is idempotent by construction. The Counter returned from a dry_run=True pass is the operator’s go/no-go signal: it reports exactly how many rows would be assigned, how many curated values would be protected, and how many are already correct, before any write is authorized.

Edge Cases and Variants

  • CSV vs. XML vs. API input. plan_batch and apply_batch take any iterable of dicts, so feed them a csv.DictReader, an lxml iterparse generator that calls elem.clear() per record, or a paged API cursor from polling museum APIs with Python Requests. Only the row source changes; derive/plan/apply is identical.
  • InC vs. InC-OW-EU vs. InC-RUU. All three are in-copyright. The split is locatability first: a locatable rightsholder never reaches an orphan statement. Only an unlocatable rightsholder does, and then jurisdiction decides — InC-OW-EU for a work registered under the EU orphan-works regime, InC-RUU everywhere else. Never emit InC-OW-EU for a non-EU record.
  • NoC-US vs. NKC. NoC-US is a positive assertion — the institution has determined a work is free of copyright in the United States. NKC (No Known Copyright) is weaker: no evidence of a subsisting copyright was found, but the institution stops short of asserting there is none. Map a determined US-public-domain verdict to NoC-US; map an inconclusive one to NKC. Collapsing the second into the first over-claims.
  • CNE for the undetermined backfill. In a legacy import where most rows were simply never assessed, the safe default status is not_evaluated, which yields CNE — an explicit “we have not looked yet” statement rather than a guess. Do not default unassessed rows to any NoC value.
  • Do-not-overwrite curated values. Any row with curated=True and a differing existing URI resolves to SKIP_CURATED and is never written, even in a live run. To re-derive a curated row deliberately, clear its curated flag first — an explicit, auditable action, never a side effect of the batch.
  • Strict vs. lenient validation. Determination.model_validate raises on an unknown status or jurisdiction token. In a backfill, wrap it in try/except ValidationError and route the offender to a quarantine table; in an interactive re-sync, surface the error to the operator rather than skipping the row silently.

Validation

Prove the two invariants that matter — every status resolves to the expected statement, and a completed run is idempotent — with an assert-based test that needs no database:

python
# test_bulk_uri.py  ->  run with:  pytest -q test_bulk_uri.py
def _rec(**over):
    base = {"object_id": "OBJ.1", "status": "in_copyright"}
    return Determination.model_validate({**base, **over})

def test_each_status_maps_to_its_statement():
    assert derive_uri(_rec(status="not_evaluated")) == RS["CNE"]
    assert derive_uri(_rec(status="undetermined")) == RS["UND"]
    assert derive_uri(_rec(status="no_known_copyright")) == RS["NKC"]
    assert derive_uri(_rec(status="in_copyright")) == RS["InC"]
    assert derive_uri(_rec(status="in_copyright", restriction="educational")) == RS["InC-EDU"]
    assert derive_uri(_rec(status="in_copyright", rightsholder_locatable=False)) == RS["InC-RUU"]
    assert derive_uri(
        _rec(status="in_copyright", rightsholder_locatable=False, jurisdiction="EU")
    ) == RS["InC-OW-EU"]
    assert derive_uri(_rec(status="no_copyright", jurisdiction="US")) == RS["NoC-US"]
    assert derive_uri(_rec(status="no_copyright", restriction="contractual")) == RS["NoC-CR"]

def test_every_derivation_is_a_valid_uri():
    assert derive_uri(_rec(status="no_copyright", jurisdiction="OTHER")) in VALID_RS_URIS

def test_bulk_apply_is_idempotent():
    rows = [
        {"object_id": "A", "status": "in_copyright"},
        {"object_id": "B", "status": "no_copyright", "jurisdiction": "US"},
    ]
    store: dict[str, str] = {}
    first = apply_batch(rows, store.__setitem__, dry_run=False)
    assert first["assign"] == 2
    # feed the written values back in: a second pass must write nothing
    rows2 = [{**r, "rights_uri": store[r["object_id"]]} for r in rows]
    second = apply_batch(rows2, store.__setitem__, dry_run=False)
    assert second["match"] == 2 and second["assign"] == 0

def test_curated_value_is_never_overwritten():
    rec = _rec(status="no_copyright", jurisdiction="US",
               rights_uri=RS["InC"], curated=True)
    assert plan_change(rec).action is Action.SKIP_CURATED

A green run confirms that each status lands on its expected statement, that a non-US no-copyright record with no stated reason still yields a valid vocabulary URI rather than an invented one, that a completed batch re-runs as all-match, and that a curated value is protected from the stamp.

Standards Alignment

The vocabulary here is RightsStatements.org 1.0 — the twelve-term controlled set jointly maintained for cultural-heritage rights, whose canonical URIs take the form https://rightsstatements.org/vocab/{TERM}/1.0/ with a required trailing slash. The three families encode a deliberate gradient: the InC statements assert subsisting copyright, the NoC statements assert its absence for a stated reason, and CNE/UND/NKC mark degrees of undetermined status, which is why none of them may be treated as a positive public-domain claim. Once a URI is stamped, the read-side contract in mapping RightsStatements.org URIs to collection fields canonicalizes and classifies it into the CMS enum, the jurisdictional term arithmetic that decides an in-copyright-versus-public-domain status runs through threshold tuning for public domain, and the stamped URI travels downstream into the IIIF Presentation API 3.0 rights property when adding rights and requiredStatement to manifests. The pure derivation function uses structural pattern matching from the standard match statement to keep the mapping exhaustive.

FAQ

How do I choose between InC-OW-EU and InC-RUU for an orphan work?

Both apply to in-copyright works whose rightsholder cannot be located, and jurisdiction is the deciding axis. InC-OW-EU is specific to works cleared through the European Union orphan-works register; InC-RUU (Rights-holder Unlocatable or Unidentifiable) is the general statement for every other case. The derivation picks InC-OW-EU only when rightsholder_locatable is false and the jurisdiction is EU; otherwise it returns InC-RUU. Never stamp InC-OW-EU on a non-EU record.

Why does the tool refuse to overwrite some existing URIs?

Because a curator may have set them deliberately. Any record flagged curated=True whose stored URI differs from the derived one resolves to the skip action and is left untouched even during a live write, so a bulk backfill can never flatten a human rights decision. To re-derive a curated record, clear its curated flag first — that makes the override an explicit, logged choice rather than a silent side effect of the batch.

What makes the batch update idempotent?

The plan phase compares each derived URI against the value already stored and marks a row that is already correct as match, and the apply phase writes only the assign and update actions. A row at its target produces no write, so re-running the job over data it already stamped changes nothing. That lets you resume an interrupted backfill or re-run after a mapping fix without double-applying or thrashing correct rows.

Should undetermined records get a NoC statement or CNE?

Neither NoC nor any public-domain claim. A record whose copyright was never assessed maps to CNE (Copyright Not Evaluated); one assessed with inconclusive evidence maps to UND; one where no copyright was found but none is asserted maps to NKC. These three exist precisely so an undetermined status is stated honestly instead of being rounded up to a free-to-use claim that the institution cannot defend.