List Item Serialization Policy: Data Model & Serialization Rules

[ceberam]

There has been several Git issues and PRs related to the serialization of lists from a DoclingDocument recently. I thought it would be useful to share this summary of the serialization policy.

A. The DoclingDocument Tree for List Items

In DoclingDocument, a ListItem owns both its primary text and any continuation blocks (loose list content, nested sub-lists, tables, code blocks):

ListItem (holds .marker, .text)
  ├── [Optional] InlineGroup  <-- Role A: item's leading text (when formatted / hyperlinked)
  ├── TextItem / InlineGroup  <-- Role B: continuation paragraph(s)
  ├── CodeItem                <-- Continuation code block
  ├── TableItem               <-- Continuation table
  └── ListGroup               <-- Nested sub-list

B. The Dual Role of InlineGroup under a ListItem

An InlineGroup child of a ListItem serves two distinct purposes:

1. Role A (Item Leading Text): When the list item's primary text has formatting or links (for example, <li><b>Bold</b> text</li>), backends set ListItem.text = "" and place the InlineGroup as the first child (item.children[0]).
2. Role B (Continuation Block): When a subsequent paragraph has formatting/links (or trailing text following a sub-list/table), it is also created as an InlineGroup under ListItem.children.

The Definitive Disambiguation Rule

An InlineGroup represents the item's primary line text if and only if item.text == "" and item.children[0].cref == group.self_ref (it is the first child).

Any other InlineGroup child is a block-level continuation (like a TextItem).

C. HTML vs Markdown Serializers

• HTMLTextSerializer handles this correctly:
  • If has_inline_repr (item.text == "" and len(children) == 1 and isinstance(children[0], InlineGroup)), it renders the InlineGroup as item text.
  • It then calls doc_serializer.get_parts(item=item, visited=my_visited), serializing all block children inside <li>...</li>. Because my_visited is updated, the outer walk never duplicates them.
• MarkdownTextSerializer historically did not recurse into item.children:
  • ListItem.children were yielded to MarkdownListSerializer, which attempted to append every InlineGroup to the previous text line (my_parts[-1].text = f"{my_parts[-1].text}{p.text}"), corrupting trailing text and dropping loose list indentation.

D. The Markdown Serialization Specification

1. Concepts: Continuation Blocks and Loose Lists

In CommonMark and standard Markdown:

• Tight List: A list where each item consists solely of a single inline paragraph without blank lines separating the elements.
• Loose List: A list where at least one item contains multiple block-level elements or is separated from its siblings by blank lines.
• Continuation Block: Any secondary block-level element (such as an additional paragraph, a fenced code block, a block quote, or a table) that belongs to an existing list item and follows its opening header line.

To maintain list membership in CommonMark, every line of a continuation block must be indented to the item's content column (the indentation level immediately following the list marker and its trailing space), separated from previous blocks by blank lines.

Content Column Calculation

The content column indentation width is defined as:

content_column = len(marker) + 1

• For - or * : len("-") + 1 = 2 spaces.
• For 1. : len("1.") + 1 = 3 spaces.
• For 10. : len("10.") + 1 = 4 spaces.

2. Concrete Example

Input DoclingDocument Hierarchy:

ListGroup
  ├── ListItem(marker="1.", text="Install the package.")
  ├── ListItem(marker="2.", text="Set the API key.")
  │     ├── TextItem("Use the key from the dashboard.")
  │     └── CodeItem("export API_KEY=...")
  └── ListItem(marker="3.", text="Run the import.")

Serialized Markdown Output:

1. Install the package.
2. Set the API key.

   Use the key from the dashboard.

export API_KEY=...

3. Run the import.

If the continuation blocks (TextItem and CodeItem) were emitted without the 3-space indentation, a Markdown parser would interpret the code block as exiting the list to the document body, breaking the list into two disconnected lists.

3. Serialization Algorithm in MarkdownTextSerializer

When serializing a ListItem in MarkdownTextSerializer.serialize():

1. Primary Header Line:
   • If has_inline_repr (or empty item.text with first-child InlineGroup), serialize that InlineGroup as the item text line.
   • Otherwise, serialize item.text with the marker (- , 1. , etc.).
2. Continuation Blocks (_serialize_list_continuations):
   • Call doc_serializer.get_parts(item=item, visited=my_visited). (If the first child was used for has_inline_repr, mark it visited so it is not re-serialized).
   • Re-indent every continuation block line by content_column spaces.
   • Separate continuation blocks with blank lines (\n\n) to form valid CommonMark loose lists.
   • Emit a trailing blank line before the next sibling list item.
3. MarkdownListSerializer Simplification:
   • Because MarkdownTextSerializer now consumes and visits all child blocks, MarkdownListSerializer only receives top-level ListItem items. The fragile InlineGroup append heuristic in MarkdownListSerializer can be safely removed or guarded.