PubTabNet_benchmarks.md

PubTabNet Benchmarks

Create PubTabNet evaluation datasets. This downloads from Huggingface the PubTabNet_OTSL dataset and runs the evaluations for TableFormer using the first 1000 samples.

# Make the ground-truth
docling-eval create-gt --benchmark PubTabNet --split val --output-dir ./benchmarks/PubTabNet/ 

# Make predictions for tables.
docling-eval create-eval \
  --benchmark PubTabNet \
  --split val \
  --output-dir ./benchmarks/PubTabNet/ \
  --end-index 1000 \
  --prediction-provider TableFormer # use tableformer predictions only

Tableformer Evaluation

Create the evaluation report:

docling-eval evaluate \
  --modality table_structure \
  --benchmark PubTabNet \
  --split val \
  --output-dir ./benchmarks/PubTabNet/ 

Tableformer evaluation json

Visualize the report:

docling-eval visualize \
  --modality table_structure \
  --benchmark PubTabNet \
  --split val \
  --output-dir ./benchmarks/PubTabNet/ 

TEDS struct only report

TEDS struct with text report