Operational Context

A rights engineer is loading a backlog of scanned and re-typed deeds of gift into the collections system, and each one carries a paragraph of free text that governs how the donated objects may be published: an embargo until a named date, a bar on commercial reuse, a mandatory credit line, a request for donor anonymity, or an outright prohibition on publication. The digitization pipeline needs a machine-readable access decision for every object, but the only source of truth is a sentence a curator typed from a paper instrument decades ago. The exact failure this page removes is the one where a deed clause is read incorrectly — or not read at all — and an object the donor restricted is exposed on the public site. Getting a legal restriction wrong is not a data-quality blemish; it is a breach of the gift agreement the institution signed. This work sits inside the automating donor agreement terms stage and feeds the restricted-by-default contract that the wider rights metadata and licensing automation pipeline enforces before anything reaches delivery.

Root Cause Analysis

Deed parsing goes wrong for three connected reasons, and each pushes in the dangerous direction — toward accidental openness — unless the design explicitly resists it.

First, natural-language clauses are ambiguous. A single deed can say “the collection shall not be published during the lifetime of the donor,” “no part may be reproduced for commercial gain,” and “the gift is to be credited to an anonymous benefactor” — three different restrictions, each phrased in prose that no fixed field maps onto cleanly. The instrument was written for a lawyer, not a parser, so meaning is carried in verbs and negation rather than in a controlled vocabulary.

Second, naive keyword matching over- and under-flags. Grepping for the word “commercial” flags a clause that merely permits commercial licensing; missing the phrase “made public” lets a real embargo slip through. A flat keyword list has no notion of confidence, so a weak coincidental hit and a strong unambiguous match are treated identically, and there is no signal to tell an automated route from one that needs a human.

Third, and decisively, a legal restriction must never silently default to open. When extraction finds nothing it recognizes, the safe interpretation is not “unrestricted” — it is “not yet understood.” A pipeline that treats an unmatched clause as permission has inverted the burden of proof. The remedy across all three fronts is the same: normalize the clause text, extract candidates as scored patterns rather than bare keywords, model each result as a strict Pydantic v2 record, and gate every low-confidence or unrecognized clause into curator review instead of an access tier.

Canonical Solution

The solution is a single pass from raw clause to a structured Restriction. Text is normalized, a small table of scored rules extracts candidate restriction types, the highest-confidence candidate is chosen, and a confidence gate decides whether the result maps to an access tier automatically or is routed to a curator. Crucially, the parser has no way to emit an open tier: AccessTier reserves OPEN for a human decision made downstream, so no code path in this module can open an object. The annotated implementation runs against any clause string or list of clauses.

python
from __future__ import annotations

import re
from dataclasses import dataclass
from datetime import date
from enum import Enum

from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator

CONFIDENCE_GATE = 0.75  # below this, a clause is never mapped automatically


class RestrictionType(str, Enum):
    EMBARGO_UNTIL = "embargo_until"
    NO_COMMERCIAL_USE = "no_commercial_use"
    CREDIT_LINE_REQUIRED = "credit_line_required"
    DONOR_ANONYMITY = "donor_anonymity"
    PUBLICATION_PROHIBITED = "publication_prohibited"
    NEEDS_REVIEW = "needs_review"          # nothing recognized, or below the gate


class AccessTier(str, Enum):
    OPEN = "open"                          # assigned ONLY by a curator, never here
    GATED = "gated"                        # attribution / credit enforced
    EMBARGOED = "embargoed"                # time-bound release
    RESTRICTED = "restricted"              # closed by an explicit prohibition
    REVIEW = "review"                      # held for a human decision


@dataclass(frozen=True)
class Rule:
    kind: RestrictionType
    pattern: re.Pattern[str]
    confidence: float


# Order is irrelevant: the highest-confidence match wins, not the first. Each
# pattern runs over normalized (lower-cased, whitespace-collapsed) clause text.
_RULES: list[Rule] = [
    Rule(RestrictionType.EMBARGO_UNTIL,
         re.compile(r"(?:embargo(?:ed)?|not (?:be )?(?:published|displayed|made public)) "
                    r"until \d{4}-\d{2}-\d{2}"), 0.90),
    Rule(RestrictionType.PUBLICATION_PROHIBITED,
         re.compile(r"(?:shall not|may not|must not) be "
                    r"(?:published|reproduced|displayed|exhibited)"), 0.85),
    Rule(RestrictionType.DONOR_ANONYMITY,
         re.compile(r"anonymous (?:gift|donor|benefactor)|"
                    r"(?:remain|wish(?:es)? to remain) anonymous"), 0.85),
    Rule(RestrictionType.NO_COMMERCIAL_USE,
         re.compile(r"non[- ]?commercial|not for commercial|"
                    r"no commercial (?:use|exploitation|gain)"), 0.80),
    Rule(RestrictionType.CREDIT_LINE_REQUIRED,
         re.compile(r"credit line|must be credited|courtesy of|"
                    r"acknowledg(?:e|ment) of the donor"), 0.80),
]

_TIER = {
    RestrictionType.EMBARGO_UNTIL: AccessTier.EMBARGOED,
    RestrictionType.PUBLICATION_PROHIBITED: AccessTier.RESTRICTED,
    RestrictionType.DONOR_ANONYMITY: AccessTier.GATED,
    RestrictionType.NO_COMMERCIAL_USE: AccessTier.GATED,
    RestrictionType.CREDIT_LINE_REQUIRED: AccessTier.GATED,
}

_EMBARGO_DATE = re.compile(r"until (\d{4})-(\d{2})-(\d{2})")


class Restriction(BaseModel):
    # extra=forbid means a stray key is an error, not a silently accepted field.
    model_config = ConfigDict(extra="forbid")

    kind: RestrictionType
    confidence: float = Field(ge=0.0, le=1.0)
    evidence: str                          # the exact matched span, for audit
    tier: AccessTier
    embargo_until: date | None = None

    @field_validator("tier")
    @classmethod
    def uncertain_stays_closed(cls, v: AccessTier, info: ValidationInfo) -> AccessTier:
        # A needs-review result may ONLY carry the REVIEW tier. This makes an
        # accidental "review -> open" mapping impossible to construct at all.
        if info.data.get("kind") is RestrictionType.NEEDS_REVIEW and v is not AccessTier.REVIEW:
            raise ValueError("uncertain clauses must route to review, never to a tier")
        return v


def normalize(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip().lower()


def _parse_embargo(evidence: str) -> date | None:
    m = _EMBARGO_DATE.search(evidence)
    if not m:
        return None
    try:
        return date(int(m.group(1)), int(m.group(2)), int(m.group(3)))
    except ValueError:
        return None                        # 2040-13-40 is a review, not a crash


def extract(clause: str) -> list[tuple[RestrictionType, float, str]]:
    # Every rule that fires yields a scored candidate carrying its matched span.
    norm = normalize(clause)
    return [(r.kind, r.confidence, m.group(0))
            for r in _RULES if (m := r.pattern.search(norm))]


def _review(evidence: str, confidence: float) -> Restriction:
    return Restriction(kind=RestrictionType.NEEDS_REVIEW, confidence=confidence,
                       evidence=evidence[:160], tier=AccessTier.REVIEW)


def classify(clause: str, gate: float = CONFIDENCE_GATE) -> Restriction:
    candidates = extract(clause)
    if not candidates:
        return _review(clause, 0.0)                 # unrecognized != unrestricted
    kind, confidence, evidence = max(candidates, key=lambda c: c[1])
    if confidence < gate:
        return _review(evidence, confidence)        # weak signal -> a human decides
    embargo = _parse_embargo(evidence) if kind is RestrictionType.EMBARGO_UNTIL else None
    if kind is RestrictionType.EMBARGO_UNTIL and embargo is None:
        return _review(evidence, confidence)        # "embargo" without a date is unsafe
    return Restriction(kind=kind, confidence=confidence, evidence=evidence,
                       tier=_TIER[kind], embargo_until=embargo)


# Higher wins: a single review or prohibition in a multi-clause deed dominates
# any weaker, more permissive sibling clause.
_SEVERITY = {
    AccessTier.REVIEW: 4,
    AccessTier.RESTRICTED: 3,
    AccessTier.EMBARGOED: 2,
    AccessTier.GATED: 1,
    AccessTier.OPEN: 0,
}


def resolve_deed(clauses: list[str], gate: float = CONFIDENCE_GATE) -> Restriction:
    found = [classify(c, gate) for c in clauses if c.strip()]
    if not found:
        return _review("", 0.0)
    return max(found, key=lambda r: _SEVERITY[r.tier])

Because the highest-severity clause wins, a deed that mixes an open-sounding sentence with a single prohibition resolves to RESTRICTED, and any clause the extractor cannot confidently read pulls the whole deed to REVIEW. The parser is deterministic and side-effect free, so the same deed always yields the same record, and re-running the load is safe.

Parsing a deed clause into a routed restriction record through a confidence gate Left to right: a normalized deed clause feeds a scored pattern extractor, whose best candidate enters a confidence gate. High confidence exits up to an access-tier box (gated, embargoed, or restricted); low confidence or no match exits down to a dashed curator-review terminal that is never opened automatically. normalize score confident ambiguous Deed clause free text Extract candidates pattern + keyword confidence ≥ gate? Map to access tier gated · embargoed · restricted Curator review never opened automatically

The single decision that matters is the diamond: everything above the gate becomes a tier, everything below it — including a clause with no recognized restriction — becomes a review task, and the parser owns no path to OPEN.

Edge Cases and Variants

  • Embargo until a named date. The EMBARGO_UNTIL rule requires an ISO date in the matched span; a clause that says “embargoed until further notice” matches no date, so _parse_embargo returns None and the clause routes to review rather than becoming an open-ended embargo. Once a real date is present the record carries embargo_until, which the routing embargo dates to access tiers work turns into a scheduled release.
  • No-commercial-use vs. permission. A phrase like “may be licensed for commercial use” must not trip the non-commercial rule; anchoring the pattern on non-commercial, not for commercial, and no commercial use keeps a permissive clause from being read as a restriction. Where a clause maps to a Creative Commons NC term, hand it to automating CC BY-NC-ND tagging in Python.
  • Credit-line requirement. A mandatory credit line is a condition of access, not a bar to it, so it maps to GATED: the object publishes only where the delivery layer can enforce the attribution string, which becomes the manifest’s requiredStatement in IIIF delivery.
  • Donor anonymity. “An anonymous benefactor” restricts disclosure of the donor, not the object, so it also maps to GATED — the object may show, but the credit line must suppress the donor’s name. This is orthogonal to copyright and is handled separately from automating copyright status checks.
  • Multi-clause deeds. A single instrument often stacks several restrictions. resolve_deed classifies each clause and returns the most severe, so one prohibition or one unreadable clause governs the whole gift — the safe direction to round.
  • Strict vs. lenient input. The model is strict (extra="forbid"); when you feed rows from a bulk export, wrap classify in try/except ValidationError and route the offender to the same review tier rather than letting a malformed record skip the gate — the same discipline used when handling orphan works in digital collections.

Validation

The one invariant that protects the institution is that a restrictive clause can never resolve to an open tier. Prove it — plus the review-routing and multi-clause behaviours — with an assert-based test that needs no database:

python
# test_deed_restrictions.py  ->  run with:  pytest -q test_deed_restrictions.py
from datetime import date
import pytest
from pydantic import ValidationError

def test_prohibition_never_yields_open():
    r = classify("These works shall not be published or reproduced.")
    assert r.kind is RestrictionType.PUBLICATION_PROHIBITED
    assert r.tier is AccessTier.RESTRICTED
    assert r.tier is not AccessTier.OPEN

def test_embargo_extracts_a_date():
    r = classify("The gift shall not be made public until 2040-01-01.")
    assert r.tier is AccessTier.EMBARGOED
    assert r.embargo_until == date(2040, 1, 1)

def test_embargo_without_date_routes_to_review():
    r = classify("The collection is embargoed until further notice.")
    assert r.tier is AccessTier.REVIEW

def test_unrecognized_clause_is_review_not_open():
    r = classify("The donor conveys the objects to the museum in perpetuity.")
    assert r.kind is RestrictionType.NEEDS_REVIEW
    assert r.tier is AccessTier.REVIEW

def test_multi_clause_deed_takes_the_most_severe():
    deed = [
        "The gift may be credited to an anonymous benefactor.",
        "The works are licensed for non-commercial use only.",
        "These works shall not be exhibited.",
    ]
    assert resolve_deed(deed).tier is AccessTier.RESTRICTED

def test_review_tier_cannot_be_relabelled_open():
    with pytest.raises(ValidationError):
        Restriction(kind=RestrictionType.NEEDS_REVIEW, confidence=0.2,
                    evidence="…", tier=AccessTier.OPEN)

A green run confirms that a prohibition lands in RESTRICTED, that an embargo yields a real release date while a dateless “embargo” is held for review, that an unrecognized clause is never treated as permission, that the most severe clause governs a multi-clause deed, and that no Restriction can even be constructed that maps a review result to open.

Standards Alignment

A parsed restriction is an access-policy fact, and it maps cleanly onto the vocabularies the rest of the pipeline already speaks. The restriction record is orthogonal to copyright: a work can be fully in copyright yet openly licensed by its donor, or in the public domain yet restricted by the terms of gift, so a deed restriction is never a substitute for a RightsStatements.org copyright statement — the two travel together. In LIDO, the donor’s terms belong in <lido:rightsWork> with the credit line preserved in <lido:creditLine>, while the object’s copyright status carries its own RightsStatements.org URI. At delivery, a gated or restricted object surfaces in an IIIF Presentation API 3.0 manifest whose rights property still carries the copyright statement and whose requiredStatement carries the mandated credit line, with runtime access enforced separately rather than through the manifest. The AccessTier values map directly onto the tier model shared with implementing embargo workflows, and the immutable rule table uses the frozen-config pattern from the Python dataclasses documentation.

FAQ

Why not just treat a clause with no recognized restriction as unrestricted?

Because “we did not recognize a restriction” and “there is no restriction” are different facts, and only one of them is safe to act on. An unmatched clause routes to NEEDS_REVIEW and the REVIEW tier, so a curator confirms the object is genuinely open before it publishes. The parser has no code path to OPEN at all — that value is reserved for a human decision.

How does the confidence gate decide what a human sees?

Each rule carries a confidence score, and classify keeps the highest-scoring candidate. If that score is below CONFIDENCE_GATE (0.75 by default), or the extractor found nothing, the clause is routed to review instead of a tier. Raising the gate sends more borderline clauses to curators; lowering it automates more but accepts more risk. The gate is one number, tuned against a sample of hand-labelled deeds.

A deed has five clauses that disagree. Which one wins?

The most restrictive. resolve_deed classifies every clause and returns the result with the highest severity, where REVIEW outranks RESTRICTED, which outranks EMBARGOED and GATED. So a single prohibition — or a single clause the parser cannot read confidently — governs the entire gift, which is the safe direction to round when the alternative is over-exposure.

No, and conflating them is a common error. Copyright is a property of the work and its term; a deed restriction is a contractual condition the donor attached to the gift. A public-domain photograph can still be restricted by its deed, and an in-copyright manuscript can be freely licensed by its donor. The restriction record and the RightsStatements.org copyright statement are stored and evaluated independently, and both must clear before an object opens.