+{% endfor %}
+```
+
+## Benefits
+
+1. **Flexibility**: Support any metadata structure using YAML/JSON
+2. **Inheritance**: Avoid repetition by inheriting from parent folders
+3. **Override capability**: Fine-tune metadata for specific plots
+4. **Performance**: Caching system for efficient repeated builds
+5. **Validation**: Built-in error handling and validation utilities
+6. **Documentation**: Comprehensive usage documentation and examples
+
+The metadata system is now fully integrated and ready for use in your scientific plot gallery generator!
diff --git a/docs/METADATA_USAGE.md b/docs/METADATA_USAGE.md
new file mode 100644
index 0000000..86a55f3
--- /dev/null
+++ b/docs/METADATA_USAGE.md
@@ -0,0 +1,114 @@
+# Metadata System Documentation
+
+## Overview
+
+The metadata system allows you to add flexible metadata to your plots and folders using YAML or JSON files. Metadata is inherited hierarchically from parent folders and can be overridden at any level.
+
+## File Structure
+
+### Folder Metadata
+- **File names**: `meta.yaml`, `meta.yml`, or `meta.json`
+- **Location**: Place in any folder containing plots
+- **Scope**: Applies to all plots in the folder and subfolders (unless overridden)
+
+### Plot-specific Metadata
+- **File names**: `{plot_name}.yaml`, `{plot_name}.yml`, or `{plot_name}.json`
+- **Location**: Place in the same folder as the plot PDF file
+- **Scope**: Applies only to the specific plot with the same name
+
+## Hierarchy and Inheritance
+
+1. **Root folder**: Start with folder metadata in your source directory
+2. **Subfolders**: Each subfolder can have its own `meta.yaml` that merges with parent metadata
+3. **Plot-specific**: Individual plots can have their own metadata files that override folder metadata
+
+## Example Usage
+
+### Folder Structure
+```
+analysis_results/
+├── meta.yaml # Root folder metadata
+├── signal/
+│ ├── meta.yaml # Signal-specific metadata
+│ ├── mass_plot.pdf
+│ └── mass_plot.yaml # Plot-specific metadata
+└── background/
+ ├── meta.yaml # Background-specific metadata
+ └── qcd_plot.pdf
+```
+
+### Example Metadata Fields
+
+**Common fields for folder metadata:**
+- `title`: Folder title
+- `description`: Folder description
+- `experiment`: Experiment name (CMS, ATLAS, etc.)
+- `dataset`: Dataset identifier
+- `analysis_type`: Type of analysis
+- `author`: Author information
+- `parameters`: Analysis parameters
+- `tags`: Categorization tags
+
+**Common fields for plot metadata:**
+- `plot_type`: Type of plot (histogram, scatter, etc.)
+- `variables`: Variable information (x_axis, y_axis, units)
+- `selection`: Selection criteria
+- `statistics`: Statistical information
+- `display`: Display options (highlight, featured, order_priority)
+
+## Configuration
+
+The metadata system can be configured in `config.yaml`:
+
+```yaml
+metadata:
+ cache_enabled: true # Enable metadata caching
+ inherit_from_parent: true # Enable hierarchical inheritance
+```
+
+## Output
+
+### HTML Template
+Metadata is available in the HTML template as:
+- `folder_metadata`: Current folder's resolved metadata
+- `item.metadata`: Individual plot metadata (in items loop)
+
+### Cache Files
+- `meta_cache.json`: Generated in each web directory
+- Contains resolved metadata for all plots in that directory
+- Used for performance optimization and debugging
+
+## Usage Tips
+
+1. **Start simple**: Begin with basic folder metadata and add complexity as needed
+2. **Use inheritance**: Put common metadata in parent folders to avoid repetition
+3. **Override selectively**: Use plot-specific metadata only when needed
+4. **Consistent naming**: Use consistent field names across your metadata files
+5. **Validate format**: Ensure YAML/JSON files are valid before running the generator
+
+## Integration with Templates
+
+In your HTML templates, you can access metadata like:
+
+```html
+
+
{{ folder_metadata.title }}
+
{{ folder_metadata.description }}
+
+
+{% for item in items %}
+
+
{{ item.name }}
+ {% if item.metadata.plot_type %}
+ {{ item.metadata.plot_type }}
+ {% endif %}
+ {% if item.metadata.tags %}
+
+ {% for tag in item.metadata.tags %}
+ {{ tag }}
+ {% endfor %}
+
+ {% endif %}
+
+{% endfor %}
+```
diff --git a/examples/meta.yaml b/examples/meta.yaml
new file mode 100644
index 0000000..fec7514
--- /dev/null
+++ b/examples/meta.yaml
@@ -0,0 +1,34 @@
+# Example folder metadata file
+# This file should be named "meta.yaml" in a folder containing plots
+
+# Folder-level metadata that applies to all plots in this folder
+title: "Analysis Results"
+description: "Comprehensive analysis of ttbar events in Run 2 data"
+experiment: "CMS"
+dataset: "Run2_2016_nano_v9"
+analysis_type: "ttbar_analysis"
+
+# Author information
+author:
+ name: "Klaus Schmidt"
+ email: "kschmidt@example.com"
+ institution: "Example University"
+
+# Analysis parameters that apply to all plots in this folder
+parameters:
+ luminosity: "35.9 fb^-1"
+ center_of_mass_energy: "13 TeV"
+ selection: "baseline"
+
+# Tags for categorization
+tags:
+ - "ttbar"
+ - "run2"
+ - "cms"
+ - "analysis"
+
+# Custom styling or display options
+display:
+ highlight: true
+ category: "primary_results"
+ order_priority: 1
diff --git a/examples/specific_plot.json b/examples/specific_plot.json
new file mode 100644
index 0000000..4257fa9
--- /dev/null
+++ b/examples/specific_plot.json
@@ -0,0 +1,22 @@
+{
+ "title": "Specific Plot Analysis",
+ "description": "This metadata overrides folder metadata for this specific plot",
+ "plot_type": "histogram",
+ "variables": {
+ "x_axis": "invariant_mass",
+ "y_axis": "events",
+ "units": "GeV"
+ },
+ "selection": "signal_region",
+ "statistics": {
+ "entries": 125000,
+ "mean": 125.3,
+ "std_dev": 2.1
+ },
+ "tags": ["signal", "mass_peak", "important"],
+ "display": {
+ "highlight": true,
+ "featured": true,
+ "order_priority": 0
+ }
+}
diff --git a/generate_gallery.py b/generate_gallery.py
index 6b8dee4..cd640de 100644
--- a/generate_gallery.py
+++ b/generate_gallery.py
@@ -16,14 +16,22 @@ Features:
import subprocess
import shutil
from pathlib import Path
+from typing import Dict, Any, Optional
from jinja2 import Environment, FileSystemLoader
-from config import Config
+
+from orchestration.config import Config
+from orchestration.metadata import (
+ load_folder_metadata,
+ merge_metadata,
+ resolve_metadata_for_plot,
+ save_metadata_cache
+)
CONFIG = Config.from_yaml("config.yaml")
env = Environment(loader=FileSystemLoader("."))
-template = env.get_template("template.html")
+template = env.get_template("templates/gallery.html")
def convert_pdf_to_png(pdf_path: Path) -> None:
@@ -80,26 +88,36 @@ def needs_update(source_file: Path, target_file: Path) -> bool:
def build_gallery(source_dir: Path, web_dir: Path,
- relative_path: Path = None) -> None:
+ relative_path: Path = None,
+ inherited_metadata: Optional[Dict[str, Any]] = None) -> None:
"""
Recursively build gallery structure from source directory.
Processes all PDF files in the source directory, converts them to PNG,
copies both to the web directory, and generates index.html files with
- navigation and thumbnails.
+ navigation and thumbnails. Now includes metadata support.
Args:
source_dir: Source directory containing PDF files
web_dir: Target web directory for gallery output
relative_path: Relative path from gallery root (for navigation)
+ inherited_metadata: Metadata inherited from parent directories
"""
if relative_path is None:
relative_path = Path(".")
+
+ if inherited_metadata is None:
+ inherited_metadata = {}
+
+ # Load folder-level metadata and merge with inherited metadata
+ folder_metadata = load_folder_metadata(source_dir)
+ current_metadata = merge_metadata(inherited_metadata, folder_metadata)
pdf_files = list(source_dir.glob("*.pdf"))
subdirs = [d for d in source_dir.iterdir() if d.is_dir()]
items = []
+ plot_metadata_cache = {}
for pdf_file in pdf_files:
png_file = pdf_file.with_suffix(".png")
@@ -122,18 +140,27 @@ def build_gallery(source_dir: Path, web_dir: Path,
else:
print(f"Skipping {png_file.name} (up to date)")
+ # Resolve metadata for this specific plot
+ plot_metadata = resolve_metadata_for_plot(pdf_file, current_metadata)
+ plot_metadata_cache[pdf_file.stem] = plot_metadata
+
items.append({
"name": pdf_file.stem,
"pdf_href": pdf_file.name,
- "png_href": png_file.name
+ "png_href": png_file.name,
+ "metadata": plot_metadata
})
+ # Save metadata cache for this directory
+ save_metadata_cache(web_dir, plot_metadata_cache)
+
subdir_names = []
for subdir in subdirs:
subdir_web = web_dir / subdir.name
subdir_web.mkdir(exist_ok=True)
subdir_relative = relative_path / subdir.name
- build_gallery(subdir, subdir_web, subdir_relative)
+ # Pass current metadata to subdirectories
+ build_gallery(subdir, subdir_web, subdir_relative, current_metadata)
subdir_names.append(subdir.name)
output_html = web_dir / "index.html"
@@ -152,6 +179,14 @@ def build_gallery(source_dir: Path, web_dir: Path,
"total_size_bytes": current_stats["total_size"]
}
+ # Calculate relative path to assets based on directory depth
+ if relative_path == Path("."):
+ assets_path = "../assets"
+ else:
+ # Count the number of directory levels to go back
+ depth = len(relative_path.parts)
+ assets_path = "../" * (depth + 1) + "assets"
+
with output_html.open("w") as f:
f.write(template.render(
title=title,
@@ -159,7 +194,9 @@ def build_gallery(source_dir: Path, web_dir: Path,
subdirs=subdir_names,
relpath=str(relative_path),
ui=CONFIG.ui,
- stats=stats
+ stats=stats,
+ folder_metadata=current_metadata,
+ assets_path=assets_path
))
print(f"Generated {output_html}")
@@ -241,6 +278,20 @@ def main() -> None:
gallery_root.mkdir(parents=True, exist_ok=True)
+ # Copy assets folder to the gallery root
+ assets_src = Path("assets")
+ assets_dst = gallery_root.parent / "assets"
+
+ if assets_src.exists():
+ if assets_dst.exists():
+ shutil.rmtree(assets_dst)
+ shutil.copytree(assets_src, assets_dst)
+ print(f"Copied assets from {assets_src} to {assets_dst}")
+ else:
+ print(f"Warning: Assets directory {assets_src} not found")
+
+ plot_metadata_cache = {}
+
for source in CONFIG.sources:
source_path = Path(source.path)
@@ -270,12 +321,21 @@ def main() -> None:
else:
print(f"Skipping {source_png_path.name} (up to date)")
+ # Resolve metadata for this single plot
+ plot_metadata = resolve_metadata_for_plot(source_path, {})
+ plot_metadata_cache[source_path.stem] = plot_metadata
+
items = [{
"name": source_path.stem,
"pdf_href": pdf_name,
- "png_href": png_name
+ "png_href": png_name,
+ "metadata": plot_metadata
}]
+ # Save metadata cache for this directory
+ plot_cache = {source_path.stem: plot_metadata}
+ save_metadata_cache(source_web_dir, plot_cache)
+
# Calculate statistics for single file
current_stats = calculate_directory_stats(source_web_dir)
stats = {
@@ -293,7 +353,9 @@ def main() -> None:
subdirs=[],
relpath=source.name,
ui=CONFIG.ui,
- stats=stats
+ stats=stats,
+ folder_metadata={},
+ assets_path="../../assets"
))
print(f"Generated {output_html}")
diff --git a/backup.py b/orchestration/backup.py
similarity index 100%
rename from backup.py
rename to orchestration/backup.py
diff --git a/config.py b/orchestration/config.py
similarity index 85%
rename from config.py
rename to orchestration/config.py
index fc798e4..9672315 100644
--- a/config.py
+++ b/orchestration/config.py
@@ -35,6 +35,16 @@ class UIConfig:
search_debounce_ms: int
+@dataclass
+class MetadataConfig:
+ """Configuration for metadata handling."""
+ cache_enabled: bool = True
+ inherit_from_parent: bool = True
+ supported_formats: list[str] = field(
+ default_factory=lambda: ['.yaml', '.yml', '.json']
+ )
+
+
@dataclass
class GalleryItem:
"""Represents a single data source for the gallery."""
@@ -53,6 +63,7 @@ class Config:
paths: PathConfig
gallery: GalleryConfig
ui: UIConfig
+ metadata: MetadataConfig
sources: list[GalleryItem] = field(default_factory=list)
@property
@@ -96,18 +107,26 @@ class Config:
paths_data = data.get('paths', {})
gallery_data = data.get('gallery', {})
ui_data = data.get('ui', {})
+ metadata_data = data.get('metadata', {})
sources_data = data.get('sources', [])
paths = PathConfig(**paths_data)
gallery = GalleryConfig(**gallery_data)
ui = UIConfig(**ui_data)
+ metadata = MetadataConfig(**metadata_data)
sources = [
GalleryItem(name=source["name"], path=Path(source["path"]))
for source in sources_data
]
- return cls(paths=paths, gallery=gallery, ui=ui, sources=sources)
+ return cls(
+ paths=paths,
+ gallery=gallery,
+ ui=ui,
+ metadata=metadata,
+ sources=sources
+ )
def to_yaml(self, yaml_file: str) -> None:
"""
diff --git a/orchestration/metadata.py b/orchestration/metadata.py
new file mode 100644
index 0000000..a8b64d3
--- /dev/null
+++ b/orchestration/metadata.py
@@ -0,0 +1,132 @@
+"""
+Metadata Management for Scientific Gallery Generator
+
+This module handles loading, parsing, and caching of metadata for plots
+and folders in the gallery system. Supports YAML and JSON formats with
+hierarchical inheritance.
+
+Features:
+- Load metadata from YAML/JSON files
+- Hierarchical metadata inheritance from parent folders
+- Plot-specific metadata overrides
+- Metadata caching for performance
+"""
+
+import json
+import yaml
+from pathlib import Path
+from typing import Dict, Any
+
+
+def load_metadata_file(metadata_path: Path) -> Dict[str, Any]:
+ """
+ Load metadata from a YAML or JSON file.
+
+ Args:
+ metadata_path: Path to the metadata file
+
+ Returns:
+ Dictionary containing the metadata, empty dict if file doesn't exist
+ or can't be parsed
+ """
+ if not metadata_path.exists():
+ return {}
+
+ try:
+ with metadata_path.open('r', encoding='utf-8') as f:
+ suffix_lower = metadata_path.suffix.lower()
+ if suffix_lower == '.yaml' or suffix_lower == '.yml':
+ return yaml.safe_load(f) or {}
+ elif metadata_path.suffix.lower() == '.json':
+ return json.load(f) or {}
+ else:
+ print(f"Warning: Unknown metadata file format: "
+ f"{metadata_path}")
+ return {}
+ except (yaml.YAMLError, json.JSONDecodeError, IOError) as e:
+ print(f"Warning: Could not parse metadata file {metadata_path}: {e}")
+ return {}
+
+
+def load_folder_metadata(folder_path: Path) -> Dict[str, Any]:
+ """
+ Load folder-level metadata from meta.yaml or meta.json.
+
+ Args:
+ folder_path: Path to the folder to check for metadata
+
+ Returns:
+ Dictionary containing the folder metadata
+ """
+ # Try YAML first, then JSON
+ for filename in ['meta.yaml', 'meta.yml', 'meta.json']:
+ metadata_path = folder_path / filename
+ if metadata_path.exists():
+ return load_metadata_file(metadata_path)
+
+ return {}
+
+
+def merge_metadata(parent_metadata: Dict[str, Any],
+ child_metadata: Dict[str, Any]) -> Dict[str, Any]:
+ """
+ Merge parent and child metadata, with child values overriding parent
+ values.
+
+ Args:
+ parent_metadata: Metadata from parent folder
+ child_metadata: Metadata from current folder
+
+ Returns:
+ Merged metadata dictionary
+ """
+ merged = parent_metadata.copy()
+ merged.update(child_metadata)
+ return merged
+
+
+def resolve_metadata_for_plot(plot_path: Path,
+ inherited_metadata: Dict[str, Any]
+ ) -> Dict[str, Any]:
+ """
+ Resolve metadata for a specific plot by merging inherited metadata
+ with plot-specific metadata.
+
+ Args:
+ plot_path: Path to the plot file (PDF)
+ inherited_metadata: Metadata inherited from folder hierarchy
+
+ Returns:
+ Final merged metadata for the plot
+ """
+ plot_stem = plot_path.stem
+ plot_dir = plot_path.parent
+
+ # Check for plot-specific metadata files
+ for suffix in ['.yaml', '.yml', '.json']:
+ plot_metadata_path = plot_dir / f"{plot_stem}{suffix}"
+ if plot_metadata_path.exists():
+ plot_metadata = load_metadata_file(plot_metadata_path)
+ return merge_metadata(inherited_metadata, plot_metadata)
+
+ # No plot-specific metadata found, return inherited metadata
+ return inherited_metadata.copy()
+
+
+def save_metadata_cache(web_dir: Path,
+ plot_metadata_cache: Dict[str, Dict[str, Any]]
+ ) -> None:
+ """
+ Save plot metadata cache to meta_cache.json in the web directory.
+
+ Args:
+ web_dir: Web directory where the cache file should be saved
+ plot_metadata_cache: Dictionary mapping plot names to their metadata
+ """
+ cache_path = web_dir / "meta_cache.json"
+ try:
+ with cache_path.open('w', encoding='utf-8') as f:
+ json.dump(plot_metadata_cache, f, indent=2, ensure_ascii=False)
+ print(f"Saved metadata cache: {cache_path}")
+ except IOError as e:
+ print(f"Warning: Could not save metadata cache {cache_path}: {e}")
diff --git a/template.html b/template.html
deleted file mode 100644
index 75a9ef5..0000000
--- a/template.html
+++ /dev/null
@@ -1,2005 +0,0 @@
-
-
-
-
- {{ title }}
-
-
-
-
-
-
-