Operational Context

A Python automation engineer receives a full LIDO export from a partner museum — a single .xml file that unpacks to several gigabytes, holding tens of thousands of <lido:lido> records inside one <lido:lidoWrap> root. The obvious first line, tree = ElementTree.parse("export.xml"), works fine against a fixture of ten records and then, against the real dump, climbs to the machine’s memory ceiling and dies with MemoryError — or, worse, drives the box into swap and the nightly ingestion job never returns. This is the flat-file XML equivalent of the problem the CSV adapter solves for tabular exports: the source does not fit in memory, so it must be streamed, one record built and released at a time, never the whole document at once. This page resolves that exact failure. It replaces the full-tree parse with an annotated lxml.etree.iterparse loop that fires on the end event of each record element, extracts fields through namespace-qualified paths, clears each element the instant its record is emitted, and yields a validated Pydantic v2 model — so peak memory tracks one record, not the file. It sits inside the XML and OAI-PMH harvest adapters stage and feeds the wider automated record ingestion and sync pipeline the same validated-record contract every other adapter honors.

Root Cause Analysis

The OOM is not a bug in ElementTree; it is the documented behavior of a document parser meeting a document that is too large for the strategy. Three concrete causes stack up.

First, the full-tree parse holds everything at once. ElementTree.parse and lxml.etree.parse are tree builders: they read the entire file and return a fully materialized DOM in which every element, attribute, and text node is a live Python object retained until the tree is garbage-collected. For a multi-gigabyte LIDO wrap that is tens of thousands of records inflated into millions of interlinked objects, and the resident set is a multiple of the file size on disk. No amount of chunked reading helps while the parser’s contract is “return the whole tree.”

Second, iterating without clearing still leaks. Switching to iterparse is necessary but not sufficient. iterparse yields elements as their end tags are seen, but by default it keeps building the tree behind you: every record you have already processed remains attached to the root as a preceding sibling, so memory still grows linearly toward the same ceiling. The element must be explicitly cleared with elem.clear() after use, and the emptied preceding siblings must be deleted off the root, or the streaming parse leaks exactly as badly as the tree parse it replaced.

Third, namespace prefixes break naive tag matching. LIDO is a namespaced vocabulary. In the parsed tree an element’s .tag is not "lido" or "titleSet" — it is the James-Clark-qualified name "{http://www.lido-schema.org}lido". Code that tests elem.tag == "lido" or calls elem.find("titleSet") silently matches nothing and returns None, so the record comes out blank and the failure is invisible until a downstream NoneType error. The prefix lido: that appears in the file is a local alias bound by an xmlns declaration; the parser resolves it to the full namespace URI, and every lookup must use that URI, not the prefix text. The remedy on all three fronts is a single disciplined loop: filter iterparse to the record tag by its qualified name, read fields through a namespace map, validate into a strict model, then clear and prune.

Canonical Solution

The loop below streams any LIDO wrap in constant memory. iterparse is filtered to the record element by its namespace-qualified tag, so the loop body only ever sees a complete <lido:lido> subtree. Each field is pulled with findtext and an explicit namespace map; the raw subtree is mapped into a strict Pydantic model that rejects a record missing its mandatory identifier or title; and the moment the record is yielded, elem.clear() empties the subtree while the while elem.getprevious() is not None prune deletes the already-processed siblings the root would otherwise retain. Peak memory is therefore a function of the single largest record, not the file.

python
from __future__ import annotations

from pathlib import Path
from typing import Iterator

from lxml import etree
from pydantic import BaseModel, field_validator

LIDO_NS = "http://www.lido-schema.org"
NS = {"lido": LIDO_NS}                      # prefix -> URI map for every lookup
RECORD_TAG = f"{{{LIDO_NS}}}lido"          # James-Clark qualified name of one record


class LidoRecord(BaseModel):
    """Strict contract for one harvested LIDO record."""

    lido_rec_id: str
    title: str
    work_type: str | None = None

    @field_validator("lido_rec_id", "title", mode="before")
    @classmethod
    def _required_text(cls, v: str | None) -> str:
        # findtext returns None when the node is absent; an empty string when it
        # is present but blank. Both are invalid for a mandatory field, so we
        # normalize here and let the record raise rather than emit a hollow row.
        if v is None or not v.strip():
            raise ValueError("required LIDO text node is missing or empty")
        return v.strip()


def _to_record(elem: etree._Element) -> LidoRecord:
    # Every path is namespace-qualified via NS — matching on the bare prefix
    # "lido:titleSet" without the namespaces= map would silently find nothing.
    return LidoRecord(
        lido_rec_id=elem.findtext(".//lido:lidoRecID", namespaces=NS),
        title=elem.findtext(
            ".//lido:titleSet/lido:appellationValue", namespaces=NS
        ),
        work_type=elem.findtext(
            ".//lido:objectWorkType/lido:term", namespaces=NS
        ),
    )


def stream_lido(path: str | Path) -> Iterator[LidoRecord]:
    """Yield one validated LidoRecord per <lido:lido>, in constant memory."""
    # tag= filters the stream to record end-events; huge_tree guards against the
    # libxml2 node-count limit that a very large wrap would otherwise trip.
    context = etree.iterparse(
        str(path), events=("end",), tag=RECORD_TAG, huge_tree=True
    )
    for _event, elem in context:
        yield _to_record(elem)              # build + validate this record
        elem.clear()                        # 1. empty the subtree just emitted
        while elem.getprevious() is not None:
            del elem.getparent()[0]         # 2. drop processed siblings off root
    del context                             # 3. release the parser context

The two-step cleanup is the load-bearing detail. elem.clear() alone frees the record’s own children but leaves the emptied husk attached to <lido:lidoWrap>; after ten thousand records that is ten thousand husks the root still owns. Deleting the preceding siblings via del elem.getparent()[0] on each pass keeps the root’s child list at length one, so the tree never grows. Because stream_lido is a generator, a consumer can drive it straight into the Pydantic validation stage or an upsert without ever materializing the list of records.

Constant-memory iterparse loop over a LIDO wrap Left to right, five stages process one record: iterparse fires the end event for a lido:lido element; findtext extracts fields via namespace-qualified paths; a strict Pydantic v2 model validates them; the validated record is yielded to the consumer; then elem.clear() empties the subtree and preceding siblings are deleted off the root. A dashed loop-back arrow returns to the next record end-event. Because each record is cleared as soon as it is emitted, resident memory tracks a single record rather than the whole file. One record built, validated, and released before the next is read iterparse end-event · lido:lido findtext namespace-qualified Pydantic validate v2 yield record to consumer clear() + del preceding siblings memory stays flat valid next lido:lido — root child list stays length one

Edge Cases and Variants

  • LIDO vs. EAD vs. generic XML. The pattern is source-agnostic; only the record boundary and namespace change. For an EAD finding aid, filter iterparse on {urn:isbn:1-931666-22-9}c (component) instead of the LIDO record tag and map the EAD namespace; for an untyped generic export, drop the tag= filter and branch on elem.tag inside the loop. The clear-and-prune tail is identical in every case.
  • Default vs. prefixed namespaces. A file may declare xmlns="http://www.lido-schema.org" with no prefix rather than xmlns:lido="...". The resolved element names are byte-for-byte identical either way — the URI is what matters — so RECORD_TAG and the NS map work unchanged. Never bind the empty-prefix key in an lxml namespace map; give the default namespace a synthetic prefix like lido and query with it.
  • Prefix that is not lido. Some exporters bind lido: to a URI with a trailing slash, or use a different prefix entirely. Match on the URI, never the literal prefix text in the file, and the loop is immune to whatever alias the exporter chose.
  • Huge single record. Constant memory means constant in the size of one record — a LIDO object with thousands of <lido:event> blocks is still built whole before it is cleared. If a single record dwarfs the rest, stream a level deeper by filtering iterparse on the repeating child and reconstructing the parent key from a running context, at the cost of a more stateful loop.
  • Malformed or truncated wrap. A dump cut off mid-record raises lxml.etree.XMLSyntaxError at the point of damage. Wrap the for loop in try/except XMLSyntaxError to emit every record parsed before the break and route the tail to the same quarantine path the CSV batch handling guide uses, rather than losing the whole harvest to one bad byte.

Validation

Two properties need proof: fields are read correctly through the namespace, and memory does not grow with record count. A single pytest module covers both against a synthetic wrap written to a temp file, so it needs no gigabyte fixture — it drives many records through the loop and asserts the peak allocation stays bounded via tracemalloc.

python
# test_stream_lido.py  ->  run with:  pytest -q test_stream_lido.py
import tracemalloc
from pathlib import Path

import pytest
from pydantic import ValidationError

HEADER = '<lido:lidoWrap xmlns:lido="http://www.lido-schema.org">'
FOOTER = "</lido:lidoWrap>"


def _record(n: int) -> str:
    return (
        "<lido:lido>"
        f"<lido:lidoRecID>OBJ.{n}</lido:lidoRecID>"
        "<lido:descriptiveMetadata>"
        "<lido:objectClassificationWrap><lido:objectWorkTypeWrap>"
        "<lido:objectWorkType><lido:term>painting</lido:term></lido:objectWorkType>"
        "</lido:objectWorkTypeWrap></lido:objectClassificationWrap>"
        "<lido:objectIdentificationWrap><lido:titleWrap>"
        f"<lido:titleSet><lido:appellationValue>Work {n}</lido:appellationValue></lido:titleSet>"
        "</lido:titleWrap></lido:objectIdentificationWrap>"
        "</lido:descriptiveMetadata>"
        "</lido:lido>"
    )


def _write_wrap(path: Path, count: int) -> Path:
    path.write_text(HEADER + "".join(_record(i) for i in range(count)) + FOOTER)
    return path


def test_fields_read_through_namespace(tmp_path: Path):
    wrap = _write_wrap(tmp_path / "small.xml", 3)
    records = list(stream_lido(wrap))
    assert [r.lido_rec_id for r in records] == ["OBJ.0", "OBJ.1", "OBJ.2"]
    assert records[0].title == "Work 0"
    assert records[0].work_type == "painting"


def test_missing_title_is_rejected(tmp_path: Path):
    bad = HEADER + "<lido:lido><lido:lidoRecID>OBJ.X</lido:lidoRecID></lido:lido>" + FOOTER
    path = tmp_path / "bad.xml"
    path.write_text(bad)
    with pytest.raises(ValidationError):
        list(stream_lido(path))


def test_memory_is_flat_across_record_count(tmp_path: Path):
    wrap = _write_wrap(tmp_path / "big.xml", 20_000)
    tracemalloc.start()
    count = 0
    for _ in stream_lido(wrap):        # consume without retaining any record
        count += 1
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    assert count == 20_000
    assert peak < 8 * 1024 * 1024      # 20k records held in well under 8 MB

A green run confirms the namespace-qualified paths resolve, that a record missing its mandatory title raises at the boundary instead of emitting a hollow row, and that driving twenty thousand records through the loop holds peak memory to a few megabytes — the signature of a genuinely streaming parse rather than a tree build in disguise.

Standards Alignment

This loop targets LIDO 1.1, the Lightweight Information Describing Objects harvesting schema published by ICOM-CIDOC and expressed against the http://www.lido-schema.org namespace; the <lido:lidoRecID>, <lido:objectWorkType>, and <lido:titleSet>/<lido:appellationValue> paths used above are the 1.1 element names, and the same qualified-name discipline applies verbatim to a 1.0 file since the namespace URI is stable across those releases. LIDO is deliberately a release format built for harvesting and interchange, not the internal store of record — its event-centric model derives from CIDOC-CRM, so the flattened LidoRecord here is only the extraction surface. Turning these streamed records into the collection’s own persistent shape is the job of mapping LIDO to internal databases, and pulling a LIDO feed incrementally over the wire rather than from a single file is covered in harvesting OAI-PMH feeds in Python, where each OAI record payload is itself a LIDO element this exact parser consumes. See the lxml iterparse documentation and the Python xml.etree reference for the underlying event-parsing contract.

FAQ

Why does elem.clear() alone still leak memory on a big LIDO file?

Because clear() empties the element’s own children but leaves the emptied element attached to its parent, <lido:lidoWrap>. After thousands of records the root still owns thousands of husks, so the resident set keeps climbing. You must also delete the preceding siblings — while elem.getprevious() is not None: del elem.getparent()[0] — on each pass so the root’s child list never grows beyond the record currently in hand. Clear plus prune together is what makes the parse constant-memory.

My findtext("lido:titleSet/...") returns None even though the element is right there. Why?

The lido: in the file is a local alias, not the element’s real name. In the parsed tree the element is {http://www.lido-schema.org}titleSet, so a lookup must pass the namespace map: findtext(".//lido:titleSet/lido:appellationValue", namespaces=NS). Without namespaces=, lxml has no binding for the lido prefix in your query and matches nothing, returning None silently. Always resolve against the namespace URI, never the prefix text.

When should I use iterparse instead of lxml.etree.parse?

Whenever the document will not comfortably fit in a few times its on-disk size in RAM, or whenever record count is unbounded — nightly partner exports, OAI harvests, any multi-gigabyte wrap. parse returns the whole tree and is fine for small, bounded files you need random access into. iterparse trades random access for a streaming, record-at-a-time pass that holds one record’s worth of memory regardless of file size.

Does filtering iterparse with tag= make the clean-up unnecessary?

No. The tag= filter only changes which end events surface in your loop; lxml still builds the full tree behind the cursor, including every record you have already seen. The filter saves you a branch, not memory. You still need clear() and the sibling prune inside the loop, or the parse leaks exactly as a tree build would.