Operational Context

A Python automation engineer maintains the ingest that feeds a museum’s collection management system, and the accession number is the one field the whole run turns on: it is the idempotency key for the CSV to database sync upsert, the primary identifier in every downstream export, and the join key registrars use to reconcile physical objects against digital records. The failure this page resolves is subtle and expensive. The ingest accepts an accession number that is almost right — 7.1994 where the label reads 007.1994, 1994/7/1 where the house standard is 1994.7.1, or a value a spreadsheet already coerced to a float — and nothing breaks immediately. The malformed key writes cleanly, then two weeks later a re-sync of the same export computes a different key, ON CONFLICT never fires, and the object is duplicated. Because the corruption is silent at write time, it surfaces only as a slow accretion of near-duplicate records that a registrar eventually has to untangle by hand. The fix is to make the accession number a validated, normalized value at the ingest boundary — enforced by a strict Pydantic v2 model — so an ambiguous or lossy value is rejected or canonicalized before it can become a key.

Root Cause Analysis

Three distinct root causes converge on the same corrupted-key symptom, and none of them is a one-off edge case.

First, numeric coercion drops leading zeros. An accession number like 007.1994 is a structured string, not a decimal. The moment any layer treats it as a number — a pandas reader inferring the column as float64, a JSON producer emitting 7.1994, a spreadsheet auto-formatting the cell — the leading zero vanishes and the two dotted segments collapse into a single float. 007.1994 and 7.1994 are different objects that now share a key, and the loss is irreversible by the time the value reaches the model. The defense has to intercept the type, not just the string, which is exactly what a mode="before" validator can do.

Second, institutions use varied formats. There is no single accession grammar. The Metropolitan Museum uses year.lot.item (1979.206.1057); LACMA prefixes a department code (M.2005.70); older registers use two segments (007.1994); and part objects append a designator (1994.7.1a). A validator hard-coded to one museum’s shape rejects another’s valid numbers, so the pattern has to describe a family of institutional forms rather than a single template.

Third, a bare regex cannot normalize legacy variants. The same object is often written 1994.7.1, 1994/7/1, or 1994 7 1 across a donor spreadsheet, a legacy Text Management System (TMS) dump, and a CollectiveAccess extract. A regex can reject the off-form variants, but rejecting them strands real records in quarantine. What ingest actually needs is to fold the known legacy separators onto one canonical form first, then validate — normalization and validation are two steps, and collapsing them into one pattern match loses records that only needed reformatting. This is the same normalize-then-enforce discipline the designing museum object schemas stage applies to every identifier field.

Canonical Solution

The model does three things in a fixed order: a mode="before" validator guards the raw type and normalizes separators while the value is still a string; a mode="after" validator enforces the institutional pattern on the canonical form; and an optional check-digit rule is offered as a reusable, opt-in variant. Keeping the string a string is the load-bearing move — every rule below runs on characters, never on a coerced number.

python
from __future__ import annotations

import re
from typing import Annotated

from pydantic import AfterValidator, BaseModel, ConfigDict, field_validator

# One canonical shape: an optional short alpha prefix, then two to four
# dot-delimited numeric segments, then an optional single-letter part code.
# \d{1,6} keeps each segment's literal digits, so 007 never becomes 7.
_ACCESSION_RE = re.compile(
    r"^(?:[A-Z]{1,4}\.)?"         # optional institutional prefix: 'M.', 'TR.'
    r"\d{1,6}(?:\.\d{1,6}){1,3}"  # 2-4 numeric segments: year.lot.item[.sub]
    r"(?:[a-z])?$"                # optional part designator: trailing '1a' style
)

# Fold ONLY slashes and whitespace onto dots. Hyphens are left alone so a
# sub-number range like '1994.7.1-.3' is never silently mangled into a segment.
_INNER_SEP = re.compile(r"\s*[/\s]\s*")
_EDGE_SEP = re.compile(r"^[.\s]+|[.\s]+$")


def normalize_accession(raw: str) -> str:
    """Fold legacy separator variants onto the canonical dotted form."""
    text = _INNER_SEP.sub(".", raw.strip())   # '1994 / 7 / 1' -> '1994.7.1'
    text = _EDGE_SEP.sub("", text)            # trim stray edge dots/space
    # Uppercase only an alpha prefix; digits and the part code are never touched,
    # so leading zeros survive normalization intact.
    if "." in text:
        head, _, tail = text.partition(".")
        if head.isalpha():
            text = head.upper() + "." + tail
    return text


def _require_pattern(v: str) -> str:
    if not _ACCESSION_RE.match(v):
        raise ValueError(f"{v!r} does not match the institutional accession pattern")
    return v


class ObjectRecord(BaseModel):
    """Strict contract for one ingested record's accession number."""

    model_config = ConfigDict(str_strip_whitespace=True, extra="forbid")

    accession_number: str

    @field_validator("accession_number", mode="before")
    @classmethod
    def reject_numeric_input(cls, v: object) -> str:
        # mode="before" sees the RAW value. If a CSV or JSON loader already turned
        # 007.1994 into the float 7.1994, the leading zero and the second segment
        # are gone — so refuse any non-string instead of validating corruption.
        if not isinstance(v, str):
            raise ValueError(
                "accession_number must arrive as a string; a numeric value has "
                "already dropped leading zeros and segment structure"
            )
        return normalize_accession(v)

    @field_validator("accession_number", mode="after")
    @classmethod
    def match_institutional_pattern(cls, v: str) -> str:
        # v is the normalized canonical form; enforce the house pattern once.
        return _require_pattern(v)

Two design choices carry most of the safety. The mode="before" guard rejects a numeric value outright rather than stringifying it, because by the time Python holds 7.1994 the information is already gone — turning it back into a string would only launder the corruption. And normalization runs inside that same before-validator, so the mode="after" pattern check always sees one canonical form and never has to enumerate every legacy separator.

For a model with many identifier fields — object number, parent lot, related-object references — re-declaring two validators per field is noise. Pydantic v2’s Annotated type plus AfterValidator packages the same rules into a reusable type, and a check-digit rule layers on for the minority of institutions that stamp one:

python
# Reusable type: apply the same normalize + pattern rules to any field on any
# model without re-declaring the validators. AfterValidator runs the callables
# left to right on the already-parsed string.
AccessionNumber = Annotated[
    str,
    AfterValidator(normalize_accession),
    AfterValidator(_require_pattern),
]


def verify_check_digit(v: str) -> str:
    """Opt-in house rule: the final digit is a mod-11 check over the rest."""
    digits = [int(c) for c in v if c.isdigit()]
    if len(digits) < 2:
        raise ValueError("too few digits to carry a check digit")
    *body, check = digits
    # ISBN-style weighting: each body digit is weighted 2, 3, 4, ... from the
    # right. Most museums assign NO check digit — apply this only where they do.
    total = sum(d * w for d, w in zip(reversed(body), range(2, len(body) + 2)))
    if (11 - total % 11) % 11 != check:
        raise ValueError(f"check digit failed for {v!r}")
    return v


# Opt-in variant for institutions that stamp a check digit on barcoded labels.
CheckedAccessionNumber = Annotated[AccessionNumber, AfterValidator(verify_check_digit)]


class LinkedObject(BaseModel):
    """Reuses the annotated type for both an object and its parent lot."""

    model_config = ConfigDict(extra="forbid")

    accession_number: AccessionNumber
    parent_accession: AccessionNumber | None = None

Because a plain str field in Pydantic v2 already refuses an int or float in lax mode, the AccessionNumber annotated type inherits the numeric-rejection guarantee for free; the explicit before-validator on ObjectRecord exists only to attach a clearer, museum-specific error message to that same refusal.

Accession validation decision path — type guard, normalize, pattern gate, accept, with a reject branch A raw value from the loader flows left to right through a type guard that admits only strings, a normalize step that folds legacy separators onto the dotted form, and a house pattern gate with an optional check digit, ending in an accepted canonical accession number. A numeric value fails the type guard and an unmatched form fails the pattern gate; both branch down to a single rejected-and-quarantined state. Keep it a string, normalize, then enforce — a lossy value never becomes a key Raw value from loader Type guard string only Normalize fold separators Pattern gate house regex + opt. check digit Accepted canonical form Rejected quarantine for review not a string no match

Edge Cases and Variants

  • Multiple institutional formats. The _ACCESSION_RE family admits year.lot.item (1979.206.1057), a two-segment register (007.1994), a department-prefixed form (M.2005.70), and a part designator (1994.7.1a). To onboard a new institution, widen the prefix class or segment count in one place — never loosen the pattern to “anything with a dot,” which would let a corrupted key through.
  • Ranges and sub-numbers like .1-.3. A single field must hold a single object, so a range such as 1994.7.1-.3 is a set of three accession numbers, not one value. Hyphens are deliberately excluded from separator folding; split the range into 1994.7.1, 1994.7.2, 1994.7.3 in a pre-processing step and validate each expanded member, rather than storing the range string as a key.
  • TMS vs. CollectiveAccess formats. TMS typically exports the object number in a dedicated column already close to canonical, while CollectiveAccess idno values often carry the department prefix and locale-specific separators. Point both sources at the same normalize_accession front door so the canonical key is source-independent; the field-level differences are catalogued in TMS vs CollectiveAccess schema mapping.
  • Strict vs. lenient runs. The model is strict: an unmatched form raises and stops at the boundary. For an authoritative production sync, keep it strict and route rejects to quarantine. For a first-pass backfill, wrap ObjectRecord(**raw) in try/except ValidationError and send the offender to the same dead-letter table for a curator to reformat, rather than dropping it silently.
  • Check-digit institutions. Most museums assign no check digit; a few stamp one on barcoded labels to catch transcription errors. Apply CheckedAccessionNumber only where the house rules define the algorithm — enabling it against numbers that were never issued with a check digit would reject every valid record.

Validation

Prove the two invariants that matter — a leading zero is preserved and a numeric value is refused, not coerced — with an assert-based test that needs no database:

python
# test_accession_validators.py  ->  run with:  pytest -q test_accession_validators.py
import pytest
from pydantic import BaseModel, ValidationError


def test_leading_zero_is_preserved():
    rec = ObjectRecord(accession_number="007.1994")
    assert rec.accession_number == "007.1994"          # the zero is not dropped


def test_numeric_input_is_rejected():
    with pytest.raises(ValidationError):
        ObjectRecord(accession_number=7.1994)           # float already corrupted


def test_legacy_separators_normalize():
    rec = ObjectRecord(accession_number="1994 / 7 / 1")
    assert rec.accession_number == "1994.7.1"           # slashes folded to dots


def test_prefix_is_uppercased():
    assert ObjectRecord(accession_number="m.2005.70").accession_number == "M.2005.70"


def test_malformed_is_rejected():
    with pytest.raises(ValidationError):
        ObjectRecord(accession_number="not-an-accession")


def test_annotated_type_reuse():
    obj = LinkedObject(accession_number="1979.206.1057", parent_accession="1979.206")
    assert obj.parent_accession == "1979.206"


def test_check_digit_variant_rejects_bad_digit():
    class Barcoded(BaseModel):
        acc: CheckedAccessionNumber

    with pytest.raises(ValidationError):
        Barcoded(acc="1994.7.1")     # trailing digit is not the computed check

A green run confirms that 007.1994 survives intact, that a float is refused at the boundary instead of being laundered into a bad key, that legacy slash and space separators fold to the canonical dotted form, that a lowercase department prefix is uppercased, and that the reusable annotated type carries the same guarantees to a second field.

Standards Alignment

The canonical accession number is what makes a record addressable in every standard downstream. In LIDO, it belongs in <lido:workID lido:type="accession number"> inside <lido:repositorySet>, and validating it before serialization is what keeps LIDO’s objectID stable across re-harvests. In CIDOC-CRM terms the accession number is an E42 Identifier bound to the object by P1 is identified by, so preserving its exact characters — leading zeros and all — is the difference between one identity and two. The same value flows into the JSON-LD identifier property modeled in designing museum object schemas, and it is the natural key the automated record ingestion and sync workflows pipeline uses for idempotent upserts. When the object is later published, that identifier is what a IIIF Presentation API 3.0 manifest carries in its metadata, so a validator that fails open here corrupts every layer above it. See the Pydantic v2 documentation for the field_validator, AfterValidator, and Annotated APIs used above.

FAQ

Why does 007.1994 lose its leading zero on import?

Because something upstream treated it as a number. A dataframe reader infers the column as float64, or a JSON producer emits 7.1994, and the leading zero and the dotted segment structure are gone before the value ever reaches your model. Guard against it with a mode="before" validator that refuses any non-string input, and stream identifier columns as strings so no numeric inference ever touches them.

Why normalize before validating instead of writing one big regex?

Because normalization and validation answer different questions. A regex can only accept or reject; it cannot reformat 1994/7/1 into 1994.7.1. Folding known legacy separators onto the canonical form first, then matching the house pattern, salvages records that were merely written in an old style — a single strict pattern would strand every one of them in quarantine.

How do I support several institutions’ formats in one validator?

Describe a family of forms, not one template. The pattern here allows an optional short alpha prefix, two to four dot-delimited numeric segments, and an optional part-letter, which covers the Met, LACMA, and legacy two-segment registers. Add a new institution by widening the prefix class or segment count in that one regex, and keep the separator-folding front door shared so the canonical key stays source-independent.

Should every accession number carry a check digit?

No. Most museums assign none, and enabling a check-digit rule against numbers that were never issued with one rejects every valid record. Treat it as an opt-in variant — the CheckedAccessionNumber annotated type — applied only where an institution’s own rules define the algorithm.