ETPlot/docs/METADATA_IMPLEMENTATION.md

Metadata System Implementation Summary

What Was Implemented

1. Core Metadata Module (metadata.py)

• load_metadata_file(): Loads YAML/JSON metadata files with error handling
• load_folder_metadata(): Discovers and loads folder-level metadata (meta.yaml/meta.json)
• merge_metadata(): Merges parent and child metadata with proper override behavior
• resolve_metadata_for_plot(): Resolves final metadata for individual plots
• save_metadata_cache(): Saves resolved metadata to cache files for performance

2. Updated Gallery Generator (generate_gallery.py)

• Hierarchical inheritance: Folder metadata is inherited by subfolders and plots
• Plot-specific overrides: Individual plots can have their own metadata files
• Template integration: Metadata is passed to HTML templates for rendering
• Cache generation: meta_cache.json files are created in each output directory

3. Configuration Updates (config.py and config.yaml)

• Added MetadataConfig class with caching and inheritance options
• Updated main Config class to include metadata settings
• Added metadata section to config.yaml

4. Documentation and Examples

• METADATA_USAGE.md: Comprehensive documentation on using the metadata system
• examples/meta.yaml: Example folder metadata file
• examples/specific_plot.json: Example plot-specific metadata file
• validate_metadata.py: Utility script for validating metadata files

Key Features

Hierarchical Metadata Inheritance

root_folder/
├── meta.yaml              # Base metadata for all plots
├── subfolder/
│   ├── meta.yaml          # Inherits from parent, can override
│   ├── plot1.pdf
│   ├── plot1.yaml         # Plot-specific metadata
│   └── plot2.pdf          # Uses folder metadata

Flexible Format Support

• YAML files: .yaml, .yml
• JSON files: .json
• Automatic format detection based on file extension

Template Integration

• folder_metadata: Available in templates for folder-level metadata
• item.metadata: Available for each plot in the items loop
• Clean separation of concerns between data and presentation

Performance Optimization

• Metadata caching in meta_cache.json files
• Only reload when source files are newer than cache
• Efficient hierarchical resolution

Usage Examples

Basic Folder Metadata

# meta.yaml
title: "Physics Analysis Results"
experiment: "CMS"
author:
  name: "Researcher Name"
  institution: "University"
tags: ["analysis", "physics"]

Plot-specific Metadata

# my_plot.yaml (for my_plot.pdf)
title: "Signal Region Analysis"
plot_type: "histogram"
variables:
  x_axis: "mass"
  y_axis: "events"
highlight: true

Template Usage

<h1>{{ folder_metadata.title }}</h1>
{% for item in items %}
  <div class="plot">
    <h3>{{ item.metadata.title or item.name }}</h3>
    {% if item.metadata.plot_type %}
      <span class="type">{{ item.metadata.plot_type }}</span>
    {% endif %}
  </div>
{% endfor %}

Benefits

1. Flexibility: Support any metadata structure using YAML/JSON
2. Inheritance: Avoid repetition by inheriting from parent folders
3. Override capability: Fine-tune metadata for specific plots
4. Performance: Caching system for efficient repeated builds
5. Validation: Built-in error handling and validation utilities
6. Documentation: Comprehensive usage documentation and examples

The metadata system is now fully integrated and ready for use in your scientific plot gallery generator!