Operational Context

A taxonomy developer inherits a legacy export where every object carries a free-text medium and place string — “etching”, “oil on canvas”, “Firenze”, “N. Italy” — and none of them carry an identifier. Before those records can enter the Getty AAT and TGN resolution stage, which takes a URI as input and turns it into a typed vocabulary record, something has to convert the string into that URI. This page is that missing step: programmatic term→URI reconciliation run at ingest or normalization time, when thousands of distinct free-text terms have to be matched against the Art & Architecture Thesaurus (AAT) and the Thesaurus of Geographic Names (TGN) in one bounded pass.

The task looks trivial for a single term and turns hostile at scale. A curator can eyeball “etching” and pick the right AAT concept in seconds; a batch job cannot, because the same idea reaches the pipeline as “etching”, “etchings”, “etched print”, and “gravure” across ten thousand rows, and each unmatched string either blocks a structural write or silently reaches a public facet under the wrong concept. The engineer who owns this code needs a resolver that queries the Getty vocabularies SPARQL endpoint, ranks candidate concepts by how well their labels match the input, commits only the confident matches automatically, and routes everything ambiguous to human review — never guessing when the evidence is thin.

Root Cause Analysis

Naive term resolution fails along three concrete axes, each with one root cause rather than a spray of unrelated bugs.

First, exact-label lookups miss variants. Matching an input string against skos:prefLabel with string equality resolves “etchings” and nothing else. Real collection data is full of singular/plural drift, spelling variants, punctuation, and near-synonyms that live in Getty as skos:altLabel rather than the preferred term. A resolver that only compares preferred labels reports a miss on terms that are, in fact, present in the vocabulary — and a false miss is just as expensive as a false match, because it dumps a resolvable term into the review queue and buries the genuinely unmatched ones.

Second, unbounded live queries are slow and rate-limited. Issuing one SPARQL request per row against vocab.getty.edu for a ten-thousand-row table is both slow and antisocial: the endpoint throttles sustained traffic, and most of those requests are redundant because the same free-text term repeats hundreds of times across the batch. Without caching, the resolver re-asks Getty the same question all day and still finishes late.

Third, ambiguous matches must not auto-commit. “Drawings” maps to several AAT concepts — visual works, technical drawings, the act of drawing — and a fuzzy matcher that always returns its single best guess will confidently attach the wrong URI. The correct behavior is a confidence gate: a term resolves automatically only when its top candidate is both good (above an acceptance score) and decisive (clearly ahead of the runner-up). Everything else is a review item, exactly as the parent Getty AAT and TGN guide requires — auto-accepting a fuzzy match without human confirmation is how the wrong objectWorkType reaches a public index.

Canonical Solution

The resolver builds one full-text SPARQL query per distinct term, scoped to a single vocabulary scheme, using the Getty Lucene index (luc:term), which searches across both preferred and alternate labels. It POSTs the query to the Getty JSON endpoint over a reused requests.Session, parses the standard SPARQL-results JSON bindings into candidate concepts, scores each candidate’s label against the input, and applies an acceptance-plus-margin gate. Confident, decisive matches commit their URI; ambiguous, weak, or empty results carry a suggested label but no committed URI and land in review. Results are cached on the normalized (scheme, term) key so a term that repeats a thousand times issues exactly one HTTP request.

The query is the load-bearing part. The Getty full-text index is exposed through the luc:term predicate; scoping with skos:inScheme keeps an AAT run from returning TGN places and vice versa, and gvp:prefLabelGVP gives back the canonical display label even when the match came from an alternate label.

sparql
PREFIX luc:  <http://www.ontotext.com/owlim/lucene#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX gvp:  <http://vocab.getty.edu/ontology#>
PREFIX xl:   <http://www.w3.org/2008/05/skos-xl#>

SELECT ?subject ?label ?parent WHERE {
  ?subject luc:term "etching" ;
           skos:inScheme <http://vocab.getty.edu/aat/> ;
           gvp:prefLabelGVP/xl:literalForm ?label .
  OPTIONAL {
    ?subject gvp:broaderPreferred/gvp:prefLabelGVP/xl:literalForm ?parent .
  }
}
LIMIT 10

The annotated Python below builds that query for any term and scheme, runs it, and applies the confidence gate. It resolves against any iterable of raw strings and shares one session and one cache across the whole batch.

python
from __future__ import annotations

import json
from dataclasses import dataclass
from difflib import SequenceMatcher
from enum import Enum
from typing import Iterable, Iterator

import requests
from pydantic import BaseModel, ConfigDict

GETTY_SPARQL = "https://vocab.getty.edu/sparql.json"
AAT_SCHEME = "http://vocab.getty.edu/aat/"
TGN_SCHEME = "http://vocab.getty.edu/tgn/"

# %(term)s is filled with a JSON-quoted literal so an embedded quote in the
# curator string cannot break out of the Lucene string and rewrite the query.
_QUERY = """
PREFIX luc:  <http://www.ontotext.com/owlim/lucene#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX gvp:  <http://vocab.getty.edu/ontology#>
PREFIX xl:   <http://www.w3.org/2008/05/skos-xl#>

SELECT ?subject ?label ?parent WHERE {
  ?subject luc:term %(term)s ;
           skos:inScheme <%(scheme)s> ;
           gvp:prefLabelGVP/xl:literalForm ?label .
  OPTIONAL {
    ?subject gvp:broaderPreferred/gvp:prefLabelGVP/xl:literalForm ?parent .
  }
}
LIMIT %(limit)d
"""


class MatchState(str, Enum):
    RESOLVED = "resolved"      # confident, decisive match — URI committed
    REVIEW = "review"          # candidates exist but not decisive — held for a curator
    UNMATCHED = "unmatched"    # no candidate at all — held for a curator


@dataclass(frozen=True)
class Candidate:
    uri: str
    label: str            # canonical prefLabel, even if the match hit an altLabel
    parent: str | None    # immediate broader concept, for disambiguation


class ResolvedTerm(BaseModel):
    model_config = ConfigDict(frozen=True)

    query_term: str
    uri: str | None            # None unless the match is auto-committable
    preferred_label: str | None
    scheme: str
    confidence: float
    state: MatchState


def _normalize(text: str) -> str:
    # Case-fold and collapse whitespace so "  Etchings " and "etchings" score as
    # identical; scoring on the normalized form keeps trivial drift from leaking
    # a real match into the review queue.
    return " ".join(text.strip().casefold().split())


def build_query(term: str, scheme: str, limit: int = 10) -> str:
    return _QUERY % {"term": json.dumps(term), "scheme": scheme, "limit": limit}


def run_query(
    term: str, scheme: str, session: requests.Session, limit: int = 10
) -> list[Candidate]:
    resp = session.post(
        GETTY_SPARQL,
        data={"query": build_query(term, scheme, limit)},
        headers={"Accept": "application/sparql-results+json"},
        timeout=15,
    )
    resp.raise_for_status()
    bindings = resp.json()["results"]["bindings"]
    return [
        Candidate(
            uri=b["subject"]["value"],
            label=b["label"]["value"],
            parent=b["parent"]["value"] if "parent" in b else None,
        )
        for b in bindings
    ]


def _score(term: str, candidate: Candidate) -> float:
    # SequenceMatcher gives 1.0 for an exact normalized label and a graded ratio
    # for near matches, so "etching" scores high against "etchings" but low
    # against an unrelated concept that the Lucene index happened to surface.
    return SequenceMatcher(None, _normalize(term), _normalize(candidate.label)).ratio()


def resolve_term(
    term: str,
    scheme: str,
    session: requests.Session,
    cache: dict[tuple[str, str], ResolvedTerm],
    *,
    accept: float = 0.92,
    margin: float = 0.08,
) -> ResolvedTerm:
    key = (scheme, _normalize(term))
    if key in cache:
        return cache[key]                       # a repeated term costs zero requests

    ranked = sorted(
        ((round(_score(term, c), 4), c) for c in run_query(term, scheme, session)),
        key=lambda pair: pair[0],
        reverse=True,
    )

    if not ranked:
        result = ResolvedTerm(
            query_term=term, uri=None, preferred_label=None,
            scheme=scheme, confidence=0.0, state=MatchState.UNMATCHED,
        )
    else:
        top_score, top = ranked[0]
        runner_up = ranked[1][0] if len(ranked) > 1 else 0.0
        # Auto-commit only when the best candidate is BOTH good and clearly ahead
        # of the second — a strong-but-tied result is ambiguous, not resolved.
        decisive = top_score >= accept and (top_score - runner_up) >= margin
        result = ResolvedTerm(
            query_term=term,
            uri=top.uri if decisive else None,      # suggestion only until reviewed
            preferred_label=top.label,
            scheme=scheme,
            confidence=top_score,
            state=MatchState.RESOLVED if decisive else MatchState.REVIEW,
        )

    cache[key] = result
    return result


def resolve_all(
    terms: Iterable[str],
    scheme: str,
    cache: dict[tuple[str, str], ResolvedTerm] | None = None,
) -> Iterator[ResolvedTerm]:
    cache = {} if cache is None else cache
    with requests.Session() as session:
        # Getty's LOD guidance asks for an identifying User-Agent so throttling
        # can be attributed; one session reuses the connection across the batch.
        session.headers["User-Agent"] = "MuseumPipeline/1.0 (collections@example.org)"
        for term in terms:
            yield resolve_term(term, scheme, session, cache)

The gate is the whole point. A ResolvedTerm only carries a non-null uri when its top candidate clears the acceptance score and leads the runner-up by the margin, so “drawings” — which returns several plausible AAT concepts of similar label distance — produces a REVIEW record with a suggested label and no committed URI, while “etchings” produces a RESOLVED record whose URI flows straight into the downstream resolution stage. Feeding resolve_all a lazy source and reusing the cache keeps a ten-thousand-row batch to one request per distinct term.

A free-text term resolves to a committed URI only when its best match is good and decisive; everything else is held for review resolve_term first tests the normalized term against the local cache. A hit returns the cached ResolvedTerm with no HTTP request. A miss POSTs a luc:term Lucene SPARQL query scoped by scheme to the Getty endpoint, scores each returned candidate's label against the input with SequenceMatcher, and applies a decision gate: is the top score at or above the acceptance threshold and ahead of the runner-up by the margin? On yes, the term resolves and commits its Getty URI. On no — weak, tied, or empty results — it routes to the curator review queue with a suggested label but no committed URI. Confident and decisive commits a URI; anything else is held for review Free-text term medium · place string In cache? Cached result zero requests SPARQL query luc:term · Getty LOD Score labels SequenceMatcher accept & margin? RESOLVED commit Getty URI REVIEW QUEUE curator · never auto-commit hit miss yes · decisive no · weak / tied

Edge Cases and Variants

  • AAT vs. TGN endpoints. Both vocabularies share one SPARQL service; the only difference is the skos:inScheme scope — http://vocab.getty.edu/aat/ for object and material terms, http://vocab.getty.edu/tgn/ for places. Pass the scheme per field so a medium string is never matched against a geographic name. TGN also carries a parent-place hierarchy worth surfacing (via gvp:broaderPreferred) because “Cambridge” is ambiguous until you see whether the parent resolves under England or Massachusetts.
  • Multiple candidates. When two concepts score within the margin of each other, the gate deliberately declines to choose and emits a REVIEW record. Widen the LIMIT, keep the top few candidates, and present them to the curator with their parent labels — the parent field exists precisely to disambiguate “drawings (visual works)” from “drawings (documents)”.
  • Deprecated terms. Getty retires concepts but keeps their URIs resolvable. If a matched subject is flagged obsolete, treat it as a review item rather than a clean resolve — add ?subject gvp:obsolete ?flag to the query or filter it out — so you never commit a term the vocabulary itself has withdrawn.
  • Strict vs. lenient thresholds. The accept/margin pair is the tuning surface. A conservative run (higher accept, wider margin) sends more terms to review and commits fewer wrong URIs; a lenient run commits more but risks the wrong concept reaching a facet. For a first pass over an unfamiliar collection, start strict and relax only after auditing the review queue.
  • Offline cache. For repeatable, endpoint-independent runs, back the in-process dict with a persistent store (a JSON sidecar or Redis) keyed on (scheme, normalized_term). Once a term is resolved, subsequent batches never touch Getty for it, which both survives rate limiting and makes the pipeline deterministic in CI where the endpoint is unreachable.
  • Alternate vocabularies. The same Lucene-plus-score-plus-gate shape works against ULAN (agents) or ICONCLASS by swapping the scheme URI and predicate; only the confidence gate stays constant, because “do not auto-commit an ambiguous match” is vocabulary-independent.

Validation

Prove the two invariants that matter — a clean exact match resolves to the expected URI, and an ambiguous or empty result never commits one — with an assert-based test that mocks the SPARQL response and touches no network:

python
# test_getty_sparql.py  ->  run with:  pytest -q test_getty_sparql.py
from enum import Enum


class _FakeResp:
    def __init__(self, payload: dict):
        self._payload = payload

    def raise_for_status(self) -> None:
        pass

    def json(self) -> dict:
        return self._payload


class _FakeSession:
    """Stands in for requests.Session; returns a fixed SPARQL-results payload."""

    def __init__(self, payload: dict):
        self._payload = payload

    def post(self, *args, **kwargs) -> _FakeResp:
        return _FakeResp(self._payload)


def _binding(uri: str, label: str, parent: str | None = None) -> dict:
    b = {"subject": {"value": uri}, "label": {"value": label}}
    if parent is not None:
        b["parent"] = {"value": parent}
    return b


def test_exact_term_resolves_to_expected_uri():
    payload = {"results": {"bindings": [
        _binding("http://vocab.getty.edu/aat/300041365", "etchings", "prints"),
        _binding("http://vocab.getty.edu/aat/300053241", "etching (process)"),
    ]}}
    result = resolve_term("etchings", AAT_SCHEME, _FakeSession(payload), {})
    assert result.state is MatchState.RESOLVED
    assert result.uri == "http://vocab.getty.edu/aat/300041365"
    assert result.confidence == 1.0


def test_ambiguous_match_routes_to_review_without_a_uri():
    payload = {"results": {"bindings": [
        _binding("http://vocab.getty.edu/aat/300033644", "drawings (visual works)"),
        _binding("http://vocab.getty.edu/aat/300027695", "drawings (documents)"),
    ]}}
    result = resolve_term("drawings", AAT_SCHEME, _FakeSession(payload), {})
    assert result.state is MatchState.REVIEW
    assert result.uri is None
    assert result.preferred_label is not None      # a suggestion is still offered


def test_no_candidate_is_unmatched():
    payload = {"results": {"bindings": []}}
    result = resolve_term("zzq-not-a-term", AAT_SCHEME, _FakeSession(payload), {})
    assert result.state is MatchState.UNMATCHED
    assert result.uri is None


def test_cache_prevents_a_second_query():
    calls = {"n": 0}

    class _CountingSession(_FakeSession):
        def post(self, *args, **kwargs):
            calls["n"] += 1
            return super().post(*args, **kwargs)

    payload = {"results": {"bindings": [
        _binding("http://vocab.getty.edu/aat/300041365", "etchings"),
    ]}}
    session, cache = _CountingSession(payload), {}
    resolve_term("etchings", AAT_SCHEME, session, cache)
    resolve_term("  Etchings ", AAT_SCHEME, session, cache)   # normalizes to same key
    assert calls["n"] == 1

A green run confirms the four behaviors the resolver exists to guarantee: an exact label resolves to the URI you expect at full confidence, a near-tie is declined and held for review with a suggestion rather than a committed URI, an empty result is flagged unmatched instead of silently dropped, and the cache collapses a repeated term to a single request. Once these pass, a live smoke test is one call to resolve_all(["etching"], AAT_SCHEME) against the real endpoint to confirm headers, POST body, and JSON shape end to end.

Standards Alignment

This step consumes the Getty vocabularies as Linked Open Data: the AAT and TGN are published as SKOS/GVP graphs with a full-text index over their preferred and alternate labels, queryable through the SPARQL service at vocab.getty.edu, as documented in Getty Vocabularies as Linked Open Data. The committed output — a stable AAT or TGN URI — is exactly the identifier the Getty AAT and TGN resolution stage then expands into a typed record and places into the correct LIDO element on its way to the internal database mapping layer. Treating the URI as the durable key and the label as a refreshable attribute is what keeps this reconciliation from reintroducing vocabulary drift, and validating each ResolvedTerm through a strict Pydantic v2 model means a malformed match fails at the boundary rather than leaking into a public facet. The lookup tables that map resolved concepts to your internal fields are worked through in the Getty AAT and TGN vocabulary mapping tables guide, and the string-distance scoring relies on the standard-library difflib module.

FAQ

Why use the Lucene luc:term index instead of matching skos:prefLabel directly?

Because a plain skos:prefLabel equality match only finds the preferred term and misses every variant. The Getty Lucene index searches across preferred and alternate labels and tolerates the singular/plural and spelling drift that fills real collection data, so “etching” surfaces the “etchings” concept even though the preferred label differs. You still score the candidate’s canonical gvp:prefLabelGVP afterward — the index widens the net, the score decides what to keep.

How do I stop the resolver from committing the wrong URI on an ambiguous term?

That is exactly what the acceptance-plus-margin gate does. A term auto-commits only when its top candidate clears the accept score and leads the runner-up by margin; a strong-but-tied result fails the margin test and becomes a review item with a suggested label but no committed URI. Raise accept and widen margin to be more conservative — the resolver’s job is to be confidently right or explicitly unsure, never quietly wrong.

How do I keep a large batch inside Getty’s rate limits?

Cache on the normalized (scheme, term) key and reuse one requests.Session for the whole run, so a term that appears a thousand times issues one request. For repeatability across runs, persist the cache to Redis or a JSON sidecar so a restart does not re-query resolved terms, and set an identifying User-Agent as the Getty LOD guidance asks so any throttling can be attributed to your institution.

Can I resolve AAT and TGN terms in the same pass?

Yes, but scope each query to one scheme. Route medium and objectWorkType strings with the AAT scheme and place strings with the TGN scheme, so a material is never matched against a geographic name. The scoring, gating, and caching are identical for both; only the skos:inScheme URI changes per field.