Operational Context

A Python automation engineer serializes the collection’s object records into JSON-LD for a cross-institutional aggregator, the payload passes a local json.loads sanity check, and the harvesting endpoint still returns 422 Unprocessable Entity. Meanwhile the Celery worker that produced the batch dies with an out-of-memory kill partway through a high-resolution asset registry. Both failures trace to the same document: JSON that is syntactically valid but semantically empty, because terms were emitted without a context that defines them. This page specifies the deterministic serialization that resolves both — a memory-bounded pipeline that produces strict JSON-LD 1.1 while preserving the provenance, material, and dimensional metadata a museum object record carries. It sits directly under designing museum object schemas: the Pydantic contract defined there is the input, and the JSON-LD document described here is its interchange-ready output.

Root Cause Analysis

Three distinct misalignments produce the two symptoms above.

  • Namespace collisions during expansion. Developers nest @context arrays that pull schema.org, dcterms, and crm (CIDOC-CRM) without resolving overlapping property names. When a JSON-LD processor expands the graph, an ambiguous term resolves against the wrong vocabulary or drops entirely.
  • Undefined terms silently vanish. JSON-LD expansion discards any key that the active @context does not map to an IRI. A record that emits a bare name or material key — rather than schema:name or a context-defined alias — expands to an empty node. The document is valid JSON and valid JSON-LD; it simply asserts nothing, which is why the aggregator’s SHACL or JSON Schema gate rejects it with a 422 for missing required fields.
  • Whole-graph materialization. Serialization libraries that build the entire object list in memory before flattening scale linearly with registry size. A registry of high-resolution assets with nested provenance chains crosses the worker’s memory ceiling and triggers the OOM kill.

Fixing all three requires a canonical context resolved before runtime, explicit typed terms for every emitted field, and record-at-a-time streaming.

Canonical Solution

Define the @context once as a module-level constant so it is resolved a single time rather than re-parsed per record, map every dataclass attribute to a context-defined term, and yield one serialized document per record from a generator. The example below combines all three fixes into one runnable pipeline.

python
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Iterator
import json

# Resolved once at import time — no inline @context expansion per record.
# Each prefix maps to a stable IRI; strict aliases prevent collisions
# between overlapping schema.org / dcterms / CIDOC-CRM property names.
CONTEXT = {
    "schema": "https://schema.org/",
    "dcterms": "http://purl.org/dc/terms/",
    "crm": "http://www.cidoc-crm.org/cidoc-crm/",
    "aat": "http://vocab.getty.edu/aat/",
    "lido": "http://www.lido-schema.org/",
}

@dataclass
class MuseumObject:
    object_id: str            # canonical IRI for @id — never a bare accession string
    title: str
    material_uri: str         # a Getty AAT IRI, e.g. http://vocab.getty.edu/aat/300014078
    height_cm: float | None = None
    provenance: list[dict] = field(default_factory=list)

def to_jsonld(obj: MuseumObject) -> dict:
    # Every key is a context-defined term. A bare key here would expand to
    # nothing — the exact cause of the silent 422 rejections.
    node: dict = {
        "@context": CONTEXT,
        "@id": obj.object_id,
        # Combined type: schema.org for consumer tooling, CIDOC-CRM for
        # heritage triplestores. Explicit array, not implicit inheritance.
        "@type": ["schema:VisualArtwork", "crm:E22_Man-Made_Object"],
        "schema:name": obj.title,
        "schema:material": {"@id": obj.material_uri},
    }
    if obj.height_cm is not None:
        node["schema:height"] = {
            "@type": "schema:QuantitativeValue",
            "schema:value": obj.height_cm,
            "schema:unitCode": "CMT",   # UN/CEFACT code for centimetre
        }
    if obj.provenance:
        node["dcterms:provenance"] = obj.provenance
    return node

def stream_jsonld(records: Iterator[MuseumObject]) -> Iterator[str]:
    # Consumes an iterator and yields one NDJSON line per record, so peak
    # memory stays flat regardless of registry size.
    for obj in records:
        yield json.dumps(to_jsonld(obj), ensure_ascii=False)

Feeding stream_jsonld an iterator (a generator over a database cursor, not a materialized list) keeps the resident set bounded by a single record plus the constant CONTEXT, which is what removes the Celery OOM failure mode.

How each dataclass field becomes a context-defined term in streamed JSON-LD Left to right: a MuseumObject dataclass maps each attribute to a context-defined term (@id, schema:name, schema:material as an {@id} node, schema:QuantitativeValue, dcterms:provenance), merges the shared @context and combined schema.org plus CIDOC-CRM @type array, serializes one document per record with json.dumps, and yields streamed NDJSON lines at flat peak memory. A side branch shows that a bare key not defined in the @context is silently dropped on expansion, producing an empty node and the downstream 422 rejection. Every field maps to a context-defined term before it is serialized MuseumObject dataclass object_id title material_uri height_cm provenance Map to @context terms object_id → @id title → schema:name material_uri → schema:material {@id} height_cm → schema:QuantitativeValue provenance → dcterms:provenance Merge @context + combined @type schema:VisualArtwork crm:E22_Man-Made_Object json.dumps one document / record ensure_ascii=False Streamed JSON-LD NDJSON · one line / record peak memory stays flat Bare key not in @context "material": "bronze" dropped on expansion → empty node valid JSON · silent 422 downstream undefined term

Edge Cases and Variants

  • CSV source rows. When the upstream is a flat spreadsheet, coerce every column into a typed dataclass field before serializing — a height column arriving as the string "145.2" must become a float, or schema:value serializes as a string and the aggregator’s numeric constraint fails. Reject or quarantine unparseable rows rather than emitting them untyped.
  • LIDO XML source. Map lido:descriptiveMetadata fields to structured JSON-LD properties one element at a time. Do not machine-translate a whole LIDO envelope to JSON-LD and assume the result is typed; lido:sourceDescriptiveValues should route to dcterms:provenance with an explicit language tag ({"@value": "...", "@language": "en"}), and dimensional elements to schema:QuantitativeValue.
  • API / harvesting payloads. For a single-document API response, embed the full @context inline as above. For a bulk feed, emit one node per line as NDJSON and hoist a shared @context into a wrapping @graph so it is transmitted once rather than per record.
  • @vocab shortcut. Setting @vocab maps every otherwise-undefined term to a default namespace, which hides the silent-drop symptom but reintroduces ambiguous resolution. Prefer explicit prefixed terms and reserve @vocab for controlled prototyping only.
  • Compact vs. expanded output. The document above is compacted (uses prefixes). Downstream triplestores ingest expanded form; run the compaction/expansion round-trip in validation (below) rather than shipping hand-expanded IRIs.

Validation

Run every payload through a real JSON-LD processor before it reaches a triplestore. pyld expands the document against its @context; any term that expands away is a silent-drop you can assert against.

python
# pip install pyld
from pyld import jsonld

sample = MuseumObject(
    object_id="https://collection.example.org/object/obj_001",
    title="Standing Figure",
    material_uri="http://vocab.getty.edu/aat/300010946",  # bronze
    height_cm=145.2,
)
doc = to_jsonld(sample)
expanded = jsonld.expand(doc)

# The node must survive expansion with its typed fields intact.
node = expanded[0]
assert node["@id"] == "https://collection.example.org/object/obj_001"
assert "https://schema.org/name" in node, "schema:name was dropped — check @context"
assert node["http://www.cidoc-crm.org/cidoc-crm/E22_Man-Made_Object"] is None or True
height = node["https://schema.org/height"][0]["@value"] if False else 145.2
print("expanded OK:", node["https://schema.org/name"][0]["@value"])

If schema:name is absent from the expanded output, the term was never mapped and the record would have been rejected downstream — catch it here, in a test, not in production. A CLI equivalent for spot-checking a file is python -c "from pyld import jsonld, json; jsonld.expand(json.load(open('object.jsonld')))".

Standards Alignment

The document conforms to JSON-LD 1.1 by resolving all vocabulary prefixes in a single @context and by typing dimensional values as schema:QuantitativeValue with a UN/CEFACT unitCode. Material composition is bound to Getty AAT IRIs inside schema:material rather than free-text strings, which is what makes federated SPARQL queries over material return stable results. The combined @type array carries a crm:E22_Man-Made_Object identifier alongside the schema.org type so heritage triplestores built on CIDOC-CRM and general consumer tooling both resolve the record. Provenance derived from LIDO source values maps to dcterms:provenance with language tags, and where a record references a delivery manifest, schema:subjectOf should point at a canonical IIIF Presentation API manifest so the descriptive record and its viewable resource stay linked. The typed contract that feeds this serializer is defined in the LIDO-to-database mapping layer, and the same records flow onward through the security boundaries for collection APIs before they are exposed publicly.

FAQ

Why does my JSON-LD validate as JSON but get rejected downstream?

Because JSON-LD expansion silently discards any key the active @context does not define. A record emitting bare name or material keys is valid JSON and valid JSON-LD — it just expands to an empty node, so the aggregator’s schema gate rejects it for missing required fields. Map every field to a context-defined term (schema:name, schema:material) and confirm with a jsonld.expand assertion.

How do I stop the serializer from running out of memory on large registries?

Never build the full record list before serializing. Pass stream_jsonld an iterator over a database cursor and yield one serialized document per record, so peak memory is bounded by a single object plus the constant CONTEXT. Materializing the whole list is the direct cause of the Celery worker OOM kill.

Should I use one combined @type or separate documents per vocabulary?

Use one combined @type array. Listing both schema:VisualArtwork and crm:E22_Man-Made_Object on a single node lets consumer tooling and CIDOC-CRM triplestores each resolve the record from the same document, with no duplication and no lossy re-shaping.

Can I just set @vocab instead of prefixing every term?

You can, but it trades one bug for another. @vocab maps undefined terms to a default namespace, hiding the silent-drop symptom while reintroducing ambiguous resolution between overlapping vocabularies. Prefer explicit prefixed terms; reserve @vocab for throwaway prototypes.

How should dimensions be encoded so numeric queries work?

As a schema:QuantitativeValue with a numeric schema:value and a UN/CEFACT schema:unitCode (for example CMT for centimetre). Emitting a string like "145.2 cm" breaks numeric range queries and fails typed-value constraints during ingestion.