Operational Context

A Python automation engineer is bulk-synchronizing a legacy collections export into a public discovery layer, and a batch of restricted images lands on the portal carrying no usable rights tag. The source rows describe assets licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 — the most restrictive of the free licenses, forbidding both commercial reuse and derivative works — but the legacy CMS stored that fact as free-text rights statements (“CC BY-NC-ND”, “by-nc-nd 4.0”, “https://creativecommons.org/licenses/by-nc-nd/4.0”) instead of a single canonical URI. This page resolves the exact task of turning those inconsistent strings into a deterministic dcterms:rights value that IIIF manifests, LIDO catalogs, and the public portal all read the same way, while guaranteeing that a malformed or ambiguous record never reaches publication. It is the no-reuse branch of the router described in routing Creative Commons licenses, and a sibling to the URI-resolution work in mapping RightsStatements.org URIs to collection fields.

Root Cause Analysis

Three failures account for nearly every misrouted CC BY-NC-ND asset, and each has a distinct fix.

First, URI drift. Legacy ETL concatenation strips trailing slashes, downgrades https:// to http://, changes case, or replaces the hyphenated component order. Creative Commons URIs are opaque identifiers whose only canonical form for this license is https://creativecommons.org/licenses/by-nc-nd/4.0/ — trailing slash included — so any tag that differs by a character will not dereference on a viewer that resolves the deed. Normalize before you validate, never after.

Second, and most dangerous, partial matching permits the wrong reuse. A lenient validator that accepts a substring or a prefix will happily let a by, by-nc, or by-nd URI through as if it were by-nc-nd. Publishing a work as CC BY when its rights require the -NC (non-commercial) and -ND (no-derivatives) restrictions silently authorizes commercial and derivative use the institution never granted. The invariant that prevents it: a record is either an exact, character-for-character match to the canonical URI after normalization, or it is quarantined for human review — there is no in-between.

Third, memory blows up at scale. Loading a whole export into a DataFrame for row-wise tagging makes peak memory grow linearly with file size and triggers OOM kills on constrained runners — the same trap covered for large CSV batches. Streaming records through a validation gate and writing fixed-size chunks bounds peak memory to one batch regardless of how many rows arrive.

Canonical Solution

The pipeline is three composable parts: a Pydantic v2 model that normalizes and exact-matches the URI per record, a generator that streams validated rows without holding the file in memory, and a chunked writer that emits idempotent IIIF Collections. The diagram traces the flow; any URI that fails the exact-match gate branches to quarantine rather than to the manifest.

Exact-match gate routing CC BY-NC-ND records to manifest or quarantine A left-to-right pipeline. Stream rows feeds Normalize URI, which feeds a diamond Exact match gate. The exact branch runs right through Map LIDO/IIIF into a Chunked writer that emits a Collection JSON file. The near-miss and missing branch drops down from the gate into a dashed Quarantine CSV box labelled embargo plus human review, and a caption notes that by, by-nc, by-nd, and 3.0 URIs are rejected there by design. exact near-miss / missing Stream rows csv · one at a time Normalize URI case · scheme · slash Exact match? Map LIDO/IIIF rights = canonical URI Chunked writer islice · idempotent Collection JSON by_nc_nd_0000.json Quarantine CSV embargo · human review Rejected by design: by · by-nc · by-nd · 3.0

Step 1 — Schema enforcement and URI canonicalization

The validator normalizes whitespace, case, separators, and trailing slash into the single canonical form, then rejects anything that is not an exact match. It never rewrites a near-miss into the canonical value — that would be the partial-match bug in disguise — it only accepts what already normalizes cleanly.

python
from typing import Optional
from pydantic import BaseModel, field_validator

CANONICAL_BY_NC_ND = "https://creativecommons.org/licenses/by-nc-nd/4.0/"

class CCByNcNdRecord(BaseModel):
    asset_id: str
    license_uri: str
    rights_statement: Optional[str] = None

    @field_validator("license_uri")
    @classmethod
    def canonicalize(cls, v: str) -> str:
        # Normalize scheme, case, separators, and trailing slash, THEN match.
        normalized = v.strip().lower().replace(" ", "-").replace("_", "-")
        normalized = normalized.replace("http://", "https://").rstrip("/") + "/"
        if not normalized.startswith("https://"):
            raise ValueError(f"License URI must use HTTPS: {v!r}")
        if normalized != CANONICAL_BY_NC_ND:
            # A by / by-nc / by-nd URI reaches here and is rejected on purpose.
            raise ValueError(f"Not canonical CC BY-NC-ND 4.0: {v!r}")
        return CANONICAL_BY_NC_ND

Step 2 — Streaming ingestion and LIDO/IIIF mapping

Read the legacy CSV with the standard-library csv module and yield one validated record at a time. A ValidationError on a single row is caught and re-emitted as an error record so the batch never halts — failed rows are separated downstream into a quarantine file. This is the same per-record validation discipline used in schema validation with Pydantic.

python
import csv
from pathlib import Path
from typing import Iterator
from pydantic import ValidationError

def stream_records(input_path: Path) -> Iterator[dict]:
    with open(input_path, mode="r", encoding="utf-8", newline="") as f:
        for row in csv.DictReader(f):
            try:
                rec = CCByNcNdRecord(
                    asset_id=row["asset_id"],
                    license_uri=row.get("license_uri", ""),
                    rights_statement=row.get("rights_statement"),
                )
                yield rec.model_dump()
            except ValidationError as e:
                # Keep the original id so curators can trace the source row.
                yield {"asset_id": row.get("asset_id"), "error": str(e)}

Step 3 — Chunked output and idempotent writes

islice pulls fixed-size chunks off the generator so exactly one batch is reachable at a time. Each chunk becomes one IIIF Presentation API 3.0 Collection whose members carry the canonical URI in the rights property; the deterministic file name lets a restarted run overwrite the same batch instead of duplicating it. Error records are dropped from the manifest here — Step 4 routes them to their own file.

python
import json
from itertools import islice
from pathlib import Path
from typing import Iterator

CHUNK_SIZE = 1000

def write_manifest_chunks(records: Iterator[dict], output_dir: Path) -> None:
    output_dir.mkdir(parents=True, exist_ok=True)
    it = iter(records)
    for chunk_idx in range(10**9):  # bounded only by input length
        chunk = list(islice(it, CHUNK_SIZE))
        if not chunk:
            break
        collection = {
            "@context": "http://iiif.io/api/presentation/3/context.json",
            "id": f"https://example.org/iiif/collection/by_nc_nd_{chunk_idx:04d}",
            "type": "Collection",
            "label": {"en": ["CC BY-NC-ND 4.0 batch"]},
            "items": [
                {
                    "id": f"https://example.org/iiif/{rec['asset_id']}/manifest",
                    "type": "Manifest",
                    "label": {"en": [rec["asset_id"]]},
                    "rights": rec["license_uri"],
                }
                for rec in chunk
                if "error" not in rec
            ],
        }
        out_file = output_dir / f"by_nc_nd_{chunk_idx:04d}.json"
        # Deterministic name = idempotent rewrite on restart, no duplicates.
        with open(out_file, "w", encoding="utf-8") as f:
            json.dump(collection, f, indent=2, ensure_ascii=False)

Step 4 — Error routing and conservative fallback

Split the stream so valid records feed the manifest writer and rejects land in a quarantine CSV with their original identifier and error. When license_uri is absent entirely, apply an embargo state rather than defaulting to open access — the same fail-safe posture used when handling orphan works in digital collections and when date-based embargo triggers gate release.

python
import csv
import logging
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

def run_pipeline(input_csv: Path, output_dir: Path, quarantine_csv: Path) -> None:
    logging.info("Starting CC BY-NC-ND tagging pipeline")
    valid, rejected = [], []
    for rec in stream_records(input_csv):
        (rejected if "error" in rec else valid).append(rec)

    write_manifest_chunks(iter(valid), output_dir)

    with open(quarantine_csv, "w", encoding="utf-8", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["asset_id", "error"])
        writer.writeheader()
        writer.writerows(rejected)

    logging.info("Tagged %d assets; quarantined %d for review",
                 len(valid), len(rejected))

For a truly large export where holding both valid and rejected lists is itself too much, keep Step 3’s streaming shape end to end: tee the generator or write rejects incrementally so peak memory still tracks one chunk, not the file.

Edge Cases and Variants

  • CSV vs. XML input. For LIDO XML exports, replace csv.DictReader with an iterparse loop over xml.etree.ElementTree, clear()-ing each element after read so the tree does not accumulate; feed the same CCByNcNdRecord model. For JSON-API payloads, iterate the paginated response and validate each item identically — the validator is the invariant, the reader is swappable.
  • CC 3.0 vs. 4.0. A by-nc-nd/3.0/ URI is a different license, not a stale form of 4.0. Do not normalize the version — quarantine 3.0 records and route them to a version-specific policy decision.
  • Ported / jurisdiction URIs. Older assets may carry .../by-nc-nd/3.0/us/ style ported URIs. These fail the exact-match gate by design; handle them with an explicit allowlist only after a rights review, never by loosening the validator.
  • Strict vs. lenient mode. The model above is strict on purpose. If a migration genuinely needs to report near-misses rather than reject them, add a separate audit pass that records the normalized-but-unmatched value — do not relax the gate that feeds publication.
  • Deed URL vs. legal-code URL. The rights property should carry the license deed URI (.../4.0/), not a legalcode path; keep the canonical constant pointing at the deed.

Validation

Confirm the two behaviours that matter: exact matches pass and canonicalize identically, and every partial or downgraded variant is rejected rather than absorbed.

python
import pytest
from pydantic import ValidationError

def test_variants_canonicalize_to_the_same_uri():
    for raw in [
        "https://creativecommons.org/licenses/by-nc-nd/4.0/",
        "http://creativecommons.org/licenses/by-nc-nd/4.0",   # scheme + slash drift
        "  CC ... /licenses/by-nc-nd/4.0/  ".replace("...", "creativecommons.org")
            .strip(),
    ]:
        rec = CCByNcNdRecord(asset_id="a1", license_uri=raw)
        assert rec.license_uri == CANONICAL_BY_NC_ND

def test_less_restrictive_licenses_are_rejected():
    for wrong in [
        "https://creativecommons.org/licenses/by/4.0/",
        "https://creativecommons.org/licenses/by-nc/4.0/",
        "https://creativecommons.org/licenses/by-nd/4.0/",
        "https://creativecommons.org/licenses/by-nc-nd/3.0/",
    ]:
        with pytest.raises(ValidationError):
            CCByNcNdRecord(asset_id="a2", license_uri=wrong)

A green run proves scheme and slash drift are repaired, and — critically — that a by, by-nc, by-nd, or 3.0 URI never slips through as by-nc-nd/4.0/.

Standards Alignment

The output conforms to three specifications at once. Under LIDO, the canonical URI populates lido:rightsWorkSet/lido:rightsType/lido:conceptID with lido:source naming Creative Commons, so the record the LIDO-to-database mapping layer persists round-trips back to XML. For delivery, the IIIF Presentation API 3.0 rights property requires exactly one dereferenceable URI from an approved set that Creative Commons licenses satisfy — the manifest carries the canonical deed URI straight through. Because the validator keys on the stable license identifier rather than free text, it composes cleanly with authority reconciliation such as Getty AAT and TGN, and the strict Pydantic v2 contract is the same discipline the rest of the rights metadata and licensing automation pipeline enforces before any record is published.

Frequently Asked Questions

Why reject a near-miss URI instead of correcting it to the canonical form?

Because “correcting” is guessing. Normalizing whitespace, case, scheme, and trailing slash is safe — those never change which license is named. But rewriting by-nc into by-nc-nd would invent a restriction the source never asserted, and rewriting by-nc-nd/3.0/ into 4.0/ would change the license terms. The validator normalizes only the cosmetic forms and quarantines anything whose meaning is ambiguous.

Is CC BY-NC-ND treated as reusable or restricted in routing?

Restricted for reuse. It permits redistribution with attribution but forbids commercial use and derivatives, so it routes like a no-reuse license: the asset can be surfaced with its rights statement, but downstream systems must not offer it for remix or commercial licensing. That is the branch this page owns within routing Creative Commons licenses.

What happens to a record with no license_uri at all?

It is quarantined and treated as embargoed, never as open access. A missing rights value is the highest-risk case, so the pipeline fails safe and routes it to human review rather than publishing an untagged asset.

Does chunked streaming change the tags a record receives?

No. Chunking only bounds memory. Each row is normalized, validated, and mapped identically whether it arrives in a 1,000-row or a 100,000-row file; keep one chunk reachable at a time so peak memory tracks batch size, not row count.