feat: Implement metadata system and reorganize project structure
✨ Features: - Add comprehensive metadata system with YAML/JSON support - Implement hierarchical metadata inheritance from parent folders - Support plot-specific metadata overrides - Add metadata caching for performance optimization 🗂️ Code Organization: - Move Python orchestration code to orchestration/ folder - Move validation utilities to tests/ folder - Move documentation to docs/ folder - Separate metadata functionality into dedicated module 🔧 Infrastructure: - Add automatic asset copying to web directory - Fix asset path resolution for nested directories - Update template to use dynamic asset paths - Add MetadataConfig class with inheritance options 📚 Documentation: - Add comprehensive metadata usage guide (METADATA_USAGE.md) - Add implementation documentation (METADATA_IMPLEMENTATION.md) - Include example metadata files in examples/ - Add metadata validation utility script 🐛 Bug Fixes: - Fix breadcrumb navigation and JavaScript functionality - Resolve asset path issues in nested directories - Update template imports for modular CSS/JS structure This commit introduces a flexible metadata system that allows users to add rich metadata to plots and folders using YAML or JSON files, with full hierarchical inheritance and plot-specific overrides. The project structure is now better organized with clear separation of concerns.
This commit is contained in:
Kylian Schmidt
Binary file not shown.
@@ -34,6 +34,14 @@ ui:
|
||||
# Search settings
|
||||
search_debounce_ms: 300
|
||||
|
||||
# Metadata Settings
|
||||
metadata:
|
||||
# Enable metadata caching
|
||||
cache_enabled: true
|
||||
|
||||
# Inherit metadata from parent folders
|
||||
inherit_from_parent: true
|
||||
|
||||
# Data Sources
|
||||
# Each source represents a collection of plots to include in the gallery
|
||||
sources:
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
# Metadata System Implementation Summary
|
||||
|
||||
## What Was Implemented
|
||||
|
||||
### 1. Core Metadata Module (`metadata.py`)
|
||||
- **`load_metadata_file()`**: Loads YAML/JSON metadata files with error handling
|
||||
- **`load_folder_metadata()`**: Discovers and loads folder-level metadata (meta.yaml/meta.json)
|
||||
- **`merge_metadata()`**: Merges parent and child metadata with proper override behavior
|
||||
- **`resolve_metadata_for_plot()`**: Resolves final metadata for individual plots
|
||||
- **`save_metadata_cache()`**: Saves resolved metadata to cache files for performance
|
||||
|
||||
### 2. Updated Gallery Generator (`generate_gallery.py`)
|
||||
- **Hierarchical inheritance**: Folder metadata is inherited by subfolders and plots
|
||||
- **Plot-specific overrides**: Individual plots can have their own metadata files
|
||||
- **Template integration**: Metadata is passed to HTML templates for rendering
|
||||
- **Cache generation**: `meta_cache.json` files are created in each output directory
|
||||
|
||||
### 3. Configuration Updates (`config.py` and `config.yaml`)
|
||||
- Added `MetadataConfig` class with caching and inheritance options
|
||||
- Updated main `Config` class to include metadata settings
|
||||
- Added metadata section to `config.yaml`
|
||||
|
||||
### 4. Documentation and Examples
|
||||
- **`METADATA_USAGE.md`**: Comprehensive documentation on using the metadata system
|
||||
- **`examples/meta.yaml`**: Example folder metadata file
|
||||
- **`examples/specific_plot.json`**: Example plot-specific metadata file
|
||||
- **`validate_metadata.py`**: Utility script for validating metadata files
|
||||
|
||||
## Key Features
|
||||
|
||||
### Hierarchical Metadata Inheritance
|
||||
```
|
||||
root_folder/
|
||||
├── meta.yaml # Base metadata for all plots
|
||||
├── subfolder/
|
||||
│ ├── meta.yaml # Inherits from parent, can override
|
||||
│ ├── plot1.pdf
|
||||
│ ├── plot1.yaml # Plot-specific metadata
|
||||
│ └── plot2.pdf # Uses folder metadata
|
||||
```
|
||||
|
||||
### Flexible Format Support
|
||||
- YAML files: `.yaml`, `.yml`
|
||||
- JSON files: `.json`
|
||||
- Automatic format detection based on file extension
|
||||
|
||||
### Template Integration
|
||||
- `folder_metadata`: Available in templates for folder-level metadata
|
||||
- `item.metadata`: Available for each plot in the items loop
|
||||
- Clean separation of concerns between data and presentation
|
||||
|
||||
### Performance Optimization
|
||||
- Metadata caching in `meta_cache.json` files
|
||||
- Only reload when source files are newer than cache
|
||||
- Efficient hierarchical resolution
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Folder Metadata
|
||||
```yaml
|
||||
# meta.yaml
|
||||
title: "Physics Analysis Results"
|
||||
experiment: "CMS"
|
||||
author:
|
||||
name: "Researcher Name"
|
||||
institution: "University"
|
||||
tags: ["analysis", "physics"]
|
||||
```
|
||||
|
||||
### Plot-specific Metadata
|
||||
```yaml
|
||||
# my_plot.yaml (for my_plot.pdf)
|
||||
title: "Signal Region Analysis"
|
||||
plot_type: "histogram"
|
||||
variables:
|
||||
x_axis: "mass"
|
||||
y_axis: "events"
|
||||
highlight: true
|
||||
```
|
||||
|
||||
### Template Usage
|
||||
```html
|
||||
<h1>{{ folder_metadata.title }}</h1>
|
||||
{% for item in items %}
|
||||
<div class="plot">
|
||||
<h3>{{ item.metadata.title or item.name }}</h3>
|
||||
{% if item.metadata.plot_type %}
|
||||
<span class="type">{{ item.metadata.plot_type }}</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Flexibility**: Support any metadata structure using YAML/JSON
|
||||
2. **Inheritance**: Avoid repetition by inheriting from parent folders
|
||||
3. **Override capability**: Fine-tune metadata for specific plots
|
||||
4. **Performance**: Caching system for efficient repeated builds
|
||||
5. **Validation**: Built-in error handling and validation utilities
|
||||
6. **Documentation**: Comprehensive usage documentation and examples
|
||||
|
||||
The metadata system is now fully integrated and ready for use in your scientific plot gallery generator!
|
||||
@@ -0,0 +1,114 @@
|
||||
# Metadata System Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The metadata system allows you to add flexible metadata to your plots and folders using YAML or JSON files. Metadata is inherited hierarchically from parent folders and can be overridden at any level.
|
||||
|
||||
## File Structure
|
||||
|
||||
### Folder Metadata
|
||||
- **File names**: `meta.yaml`, `meta.yml`, or `meta.json`
|
||||
- **Location**: Place in any folder containing plots
|
||||
- **Scope**: Applies to all plots in the folder and subfolders (unless overridden)
|
||||
|
||||
### Plot-specific Metadata
|
||||
- **File names**: `{plot_name}.yaml`, `{plot_name}.yml`, or `{plot_name}.json`
|
||||
- **Location**: Place in the same folder as the plot PDF file
|
||||
- **Scope**: Applies only to the specific plot with the same name
|
||||
|
||||
## Hierarchy and Inheritance
|
||||
|
||||
1. **Root folder**: Start with folder metadata in your source directory
|
||||
2. **Subfolders**: Each subfolder can have its own `meta.yaml` that merges with parent metadata
|
||||
3. **Plot-specific**: Individual plots can have their own metadata files that override folder metadata
|
||||
|
||||
## Example Usage
|
||||
|
||||
### Folder Structure
|
||||
```
|
||||
analysis_results/
|
||||
├── meta.yaml # Root folder metadata
|
||||
├── signal/
|
||||
│ ├── meta.yaml # Signal-specific metadata
|
||||
│ ├── mass_plot.pdf
|
||||
│ └── mass_plot.yaml # Plot-specific metadata
|
||||
└── background/
|
||||
├── meta.yaml # Background-specific metadata
|
||||
└── qcd_plot.pdf
|
||||
```
|
||||
|
||||
### Example Metadata Fields
|
||||
|
||||
**Common fields for folder metadata:**
|
||||
- `title`: Folder title
|
||||
- `description`: Folder description
|
||||
- `experiment`: Experiment name (CMS, ATLAS, etc.)
|
||||
- `dataset`: Dataset identifier
|
||||
- `analysis_type`: Type of analysis
|
||||
- `author`: Author information
|
||||
- `parameters`: Analysis parameters
|
||||
- `tags`: Categorization tags
|
||||
|
||||
**Common fields for plot metadata:**
|
||||
- `plot_type`: Type of plot (histogram, scatter, etc.)
|
||||
- `variables`: Variable information (x_axis, y_axis, units)
|
||||
- `selection`: Selection criteria
|
||||
- `statistics`: Statistical information
|
||||
- `display`: Display options (highlight, featured, order_priority)
|
||||
|
||||
## Configuration
|
||||
|
||||
The metadata system can be configured in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
cache_enabled: true # Enable metadata caching
|
||||
inherit_from_parent: true # Enable hierarchical inheritance
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
### HTML Template
|
||||
Metadata is available in the HTML template as:
|
||||
- `folder_metadata`: Current folder's resolved metadata
|
||||
- `item.metadata`: Individual plot metadata (in items loop)
|
||||
|
||||
### Cache Files
|
||||
- `meta_cache.json`: Generated in each web directory
|
||||
- Contains resolved metadata for all plots in that directory
|
||||
- Used for performance optimization and debugging
|
||||
|
||||
## Usage Tips
|
||||
|
||||
1. **Start simple**: Begin with basic folder metadata and add complexity as needed
|
||||
2. **Use inheritance**: Put common metadata in parent folders to avoid repetition
|
||||
3. **Override selectively**: Use plot-specific metadata only when needed
|
||||
4. **Consistent naming**: Use consistent field names across your metadata files
|
||||
5. **Validate format**: Ensure YAML/JSON files are valid before running the generator
|
||||
|
||||
## Integration with Templates
|
||||
|
||||
In your HTML templates, you can access metadata like:
|
||||
|
||||
```html
|
||||
<!-- Folder metadata -->
|
||||
<h2>{{ folder_metadata.title }}</h2>
|
||||
<p>{{ folder_metadata.description }}</p>
|
||||
|
||||
<!-- Plot metadata -->
|
||||
{% for item in items %}
|
||||
<div class="plot-item">
|
||||
<h3>{{ item.name }}</h3>
|
||||
{% if item.metadata.plot_type %}
|
||||
<span class="plot-type">{{ item.metadata.plot_type }}</span>
|
||||
{% endif %}
|
||||
{% if item.metadata.tags %}
|
||||
<div class="tags">
|
||||
{% for tag in item.metadata.tags %}
|
||||
<span class="tag">{{ tag }}</span>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
```
|
||||
@@ -0,0 +1,34 @@
|
||||
# Example folder metadata file
|
||||
# This file should be named "meta.yaml" in a folder containing plots
|
||||
|
||||
# Folder-level metadata that applies to all plots in this folder
|
||||
title: "Analysis Results"
|
||||
description: "Comprehensive analysis of ttbar events in Run 2 data"
|
||||
experiment: "CMS"
|
||||
dataset: "Run2_2016_nano_v9"
|
||||
analysis_type: "ttbar_analysis"
|
||||
|
||||
# Author information
|
||||
author:
|
||||
name: "Klaus Schmidt"
|
||||
email: "kschmidt@example.com"
|
||||
institution: "Example University"
|
||||
|
||||
# Analysis parameters that apply to all plots in this folder
|
||||
parameters:
|
||||
luminosity: "35.9 fb^-1"
|
||||
center_of_mass_energy: "13 TeV"
|
||||
selection: "baseline"
|
||||
|
||||
# Tags for categorization
|
||||
tags:
|
||||
- "ttbar"
|
||||
- "run2"
|
||||
- "cms"
|
||||
- "analysis"
|
||||
|
||||
# Custom styling or display options
|
||||
display:
|
||||
highlight: true
|
||||
category: "primary_results"
|
||||
order_priority: 1
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"title": "Specific Plot Analysis",
|
||||
"description": "This metadata overrides folder metadata for this specific plot",
|
||||
"plot_type": "histogram",
|
||||
"variables": {
|
||||
"x_axis": "invariant_mass",
|
||||
"y_axis": "events",
|
||||
"units": "GeV"
|
||||
},
|
||||
"selection": "signal_region",
|
||||
"statistics": {
|
||||
"entries": 125000,
|
||||
"mean": 125.3,
|
||||
"std_dev": 2.1
|
||||
},
|
||||
"tags": ["signal", "mass_peak", "important"],
|
||||
"display": {
|
||||
"highlight": true,
|
||||
"featured": true,
|
||||
"order_priority": 0
|
||||
}
|
||||
}
|
||||
+71
-9
@@ -16,14 +16,22 @@ Features:
|
||||
import subprocess
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Optional
|
||||
from jinja2 import Environment, FileSystemLoader
|
||||
from config import Config
|
||||
|
||||
from orchestration.config import Config
|
||||
from orchestration.metadata import (
|
||||
load_folder_metadata,
|
||||
merge_metadata,
|
||||
resolve_metadata_for_plot,
|
||||
save_metadata_cache
|
||||
)
|
||||
|
||||
|
||||
CONFIG = Config.from_yaml("config.yaml")
|
||||
|
||||
env = Environment(loader=FileSystemLoader("."))
|
||||
template = env.get_template("template.html")
|
||||
template = env.get_template("templates/gallery.html")
|
||||
|
||||
|
||||
def convert_pdf_to_png(pdf_path: Path) -> None:
|
||||
@@ -80,26 +88,36 @@ def needs_update(source_file: Path, target_file: Path) -> bool:
|
||||
|
||||
|
||||
def build_gallery(source_dir: Path, web_dir: Path,
|
||||
relative_path: Path = None) -> None:
|
||||
relative_path: Path = None,
|
||||
inherited_metadata: Optional[Dict[str, Any]] = None) -> None:
|
||||
"""
|
||||
Recursively build gallery structure from source directory.
|
||||
|
||||
Processes all PDF files in the source directory, converts them to PNG,
|
||||
copies both to the web directory, and generates index.html files with
|
||||
navigation and thumbnails.
|
||||
navigation and thumbnails. Now includes metadata support.
|
||||
|
||||
Args:
|
||||
source_dir: Source directory containing PDF files
|
||||
web_dir: Target web directory for gallery output
|
||||
relative_path: Relative path from gallery root (for navigation)
|
||||
inherited_metadata: Metadata inherited from parent directories
|
||||
"""
|
||||
if relative_path is None:
|
||||
relative_path = Path(".")
|
||||
|
||||
if inherited_metadata is None:
|
||||
inherited_metadata = {}
|
||||
|
||||
# Load folder-level metadata and merge with inherited metadata
|
||||
folder_metadata = load_folder_metadata(source_dir)
|
||||
current_metadata = merge_metadata(inherited_metadata, folder_metadata)
|
||||
|
||||
pdf_files = list(source_dir.glob("*.pdf"))
|
||||
subdirs = [d for d in source_dir.iterdir() if d.is_dir()]
|
||||
|
||||
items = []
|
||||
plot_metadata_cache = {}
|
||||
|
||||
for pdf_file in pdf_files:
|
||||
png_file = pdf_file.with_suffix(".png")
|
||||
@@ -122,18 +140,27 @@ def build_gallery(source_dir: Path, web_dir: Path,
|
||||
else:
|
||||
print(f"Skipping {png_file.name} (up to date)")
|
||||
|
||||
# Resolve metadata for this specific plot
|
||||
plot_metadata = resolve_metadata_for_plot(pdf_file, current_metadata)
|
||||
plot_metadata_cache[pdf_file.stem] = plot_metadata
|
||||
|
||||
items.append({
|
||||
"name": pdf_file.stem,
|
||||
"pdf_href": pdf_file.name,
|
||||
"png_href": png_file.name
|
||||
"png_href": png_file.name,
|
||||
"metadata": plot_metadata
|
||||
})
|
||||
|
||||
# Save metadata cache for this directory
|
||||
save_metadata_cache(web_dir, plot_metadata_cache)
|
||||
|
||||
subdir_names = []
|
||||
for subdir in subdirs:
|
||||
subdir_web = web_dir / subdir.name
|
||||
subdir_web.mkdir(exist_ok=True)
|
||||
subdir_relative = relative_path / subdir.name
|
||||
build_gallery(subdir, subdir_web, subdir_relative)
|
||||
# Pass current metadata to subdirectories
|
||||
build_gallery(subdir, subdir_web, subdir_relative, current_metadata)
|
||||
subdir_names.append(subdir.name)
|
||||
|
||||
output_html = web_dir / "index.html"
|
||||
@@ -152,6 +179,14 @@ def build_gallery(source_dir: Path, web_dir: Path,
|
||||
"total_size_bytes": current_stats["total_size"]
|
||||
}
|
||||
|
||||
# Calculate relative path to assets based on directory depth
|
||||
if relative_path == Path("."):
|
||||
assets_path = "../assets"
|
||||
else:
|
||||
# Count the number of directory levels to go back
|
||||
depth = len(relative_path.parts)
|
||||
assets_path = "../" * (depth + 1) + "assets"
|
||||
|
||||
with output_html.open("w") as f:
|
||||
f.write(template.render(
|
||||
title=title,
|
||||
@@ -159,7 +194,9 @@ def build_gallery(source_dir: Path, web_dir: Path,
|
||||
subdirs=subdir_names,
|
||||
relpath=str(relative_path),
|
||||
ui=CONFIG.ui,
|
||||
stats=stats
|
||||
stats=stats,
|
||||
folder_metadata=current_metadata,
|
||||
assets_path=assets_path
|
||||
))
|
||||
|
||||
print(f"Generated {output_html}")
|
||||
@@ -241,6 +278,20 @@ def main() -> None:
|
||||
|
||||
gallery_root.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Copy assets folder to the gallery root
|
||||
assets_src = Path("assets")
|
||||
assets_dst = gallery_root.parent / "assets"
|
||||
|
||||
if assets_src.exists():
|
||||
if assets_dst.exists():
|
||||
shutil.rmtree(assets_dst)
|
||||
shutil.copytree(assets_src, assets_dst)
|
||||
print(f"Copied assets from {assets_src} to {assets_dst}")
|
||||
else:
|
||||
print(f"Warning: Assets directory {assets_src} not found")
|
||||
|
||||
plot_metadata_cache = {}
|
||||
|
||||
for source in CONFIG.sources:
|
||||
source_path = Path(source.path)
|
||||
|
||||
@@ -270,12 +321,21 @@ def main() -> None:
|
||||
else:
|
||||
print(f"Skipping {source_png_path.name} (up to date)")
|
||||
|
||||
# Resolve metadata for this single plot
|
||||
plot_metadata = resolve_metadata_for_plot(source_path, {})
|
||||
plot_metadata_cache[source_path.stem] = plot_metadata
|
||||
|
||||
items = [{
|
||||
"name": source_path.stem,
|
||||
"pdf_href": pdf_name,
|
||||
"png_href": png_name
|
||||
"png_href": png_name,
|
||||
"metadata": plot_metadata
|
||||
}]
|
||||
|
||||
# Save metadata cache for this directory
|
||||
plot_cache = {source_path.stem: plot_metadata}
|
||||
save_metadata_cache(source_web_dir, plot_cache)
|
||||
|
||||
# Calculate statistics for single file
|
||||
current_stats = calculate_directory_stats(source_web_dir)
|
||||
stats = {
|
||||
@@ -293,7 +353,9 @@ def main() -> None:
|
||||
subdirs=[],
|
||||
relpath=source.name,
|
||||
ui=CONFIG.ui,
|
||||
stats=stats
|
||||
stats=stats,
|
||||
folder_metadata={},
|
||||
assets_path="../../assets"
|
||||
))
|
||||
|
||||
print(f"Generated {output_html}")
|
||||
|
||||
@@ -35,6 +35,16 @@ class UIConfig:
|
||||
search_debounce_ms: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class MetadataConfig:
|
||||
"""Configuration for metadata handling."""
|
||||
cache_enabled: bool = True
|
||||
inherit_from_parent: bool = True
|
||||
supported_formats: list[str] = field(
|
||||
default_factory=lambda: ['.yaml', '.yml', '.json']
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GalleryItem:
|
||||
"""Represents a single data source for the gallery."""
|
||||
@@ -53,6 +63,7 @@ class Config:
|
||||
paths: PathConfig
|
||||
gallery: GalleryConfig
|
||||
ui: UIConfig
|
||||
metadata: MetadataConfig
|
||||
sources: list[GalleryItem] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
@@ -96,18 +107,26 @@ class Config:
|
||||
paths_data = data.get('paths', {})
|
||||
gallery_data = data.get('gallery', {})
|
||||
ui_data = data.get('ui', {})
|
||||
metadata_data = data.get('metadata', {})
|
||||
sources_data = data.get('sources', [])
|
||||
|
||||
paths = PathConfig(**paths_data)
|
||||
gallery = GalleryConfig(**gallery_data)
|
||||
ui = UIConfig(**ui_data)
|
||||
metadata = MetadataConfig(**metadata_data)
|
||||
|
||||
sources = [
|
||||
GalleryItem(name=source["name"], path=Path(source["path"]))
|
||||
for source in sources_data
|
||||
]
|
||||
|
||||
return cls(paths=paths, gallery=gallery, ui=ui, sources=sources)
|
||||
return cls(
|
||||
paths=paths,
|
||||
gallery=gallery,
|
||||
ui=ui,
|
||||
metadata=metadata,
|
||||
sources=sources
|
||||
)
|
||||
|
||||
def to_yaml(self, yaml_file: str) -> None:
|
||||
"""
|
||||
@@ -0,0 +1,132 @@
|
||||
"""
|
||||
Metadata Management for Scientific Gallery Generator
|
||||
|
||||
This module handles loading, parsing, and caching of metadata for plots
|
||||
and folders in the gallery system. Supports YAML and JSON formats with
|
||||
hierarchical inheritance.
|
||||
|
||||
Features:
|
||||
- Load metadata from YAML/JSON files
|
||||
- Hierarchical metadata inheritance from parent folders
|
||||
- Plot-specific metadata overrides
|
||||
- Metadata caching for performance
|
||||
"""
|
||||
|
||||
import json
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
|
||||
|
||||
def load_metadata_file(metadata_path: Path) -> Dict[str, Any]:
|
||||
"""
|
||||
Load metadata from a YAML or JSON file.
|
||||
|
||||
Args:
|
||||
metadata_path: Path to the metadata file
|
||||
|
||||
Returns:
|
||||
Dictionary containing the metadata, empty dict if file doesn't exist
|
||||
or can't be parsed
|
||||
"""
|
||||
if not metadata_path.exists():
|
||||
return {}
|
||||
|
||||
try:
|
||||
with metadata_path.open('r', encoding='utf-8') as f:
|
||||
suffix_lower = metadata_path.suffix.lower()
|
||||
if suffix_lower == '.yaml' or suffix_lower == '.yml':
|
||||
return yaml.safe_load(f) or {}
|
||||
elif metadata_path.suffix.lower() == '.json':
|
||||
return json.load(f) or {}
|
||||
else:
|
||||
print(f"Warning: Unknown metadata file format: "
|
||||
f"{metadata_path}")
|
||||
return {}
|
||||
except (yaml.YAMLError, json.JSONDecodeError, IOError) as e:
|
||||
print(f"Warning: Could not parse metadata file {metadata_path}: {e}")
|
||||
return {}
|
||||
|
||||
|
||||
def load_folder_metadata(folder_path: Path) -> Dict[str, Any]:
|
||||
"""
|
||||
Load folder-level metadata from meta.yaml or meta.json.
|
||||
|
||||
Args:
|
||||
folder_path: Path to the folder to check for metadata
|
||||
|
||||
Returns:
|
||||
Dictionary containing the folder metadata
|
||||
"""
|
||||
# Try YAML first, then JSON
|
||||
for filename in ['meta.yaml', 'meta.yml', 'meta.json']:
|
||||
metadata_path = folder_path / filename
|
||||
if metadata_path.exists():
|
||||
return load_metadata_file(metadata_path)
|
||||
|
||||
return {}
|
||||
|
||||
|
||||
def merge_metadata(parent_metadata: Dict[str, Any],
|
||||
child_metadata: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""
|
||||
Merge parent and child metadata, with child values overriding parent
|
||||
values.
|
||||
|
||||
Args:
|
||||
parent_metadata: Metadata from parent folder
|
||||
child_metadata: Metadata from current folder
|
||||
|
||||
Returns:
|
||||
Merged metadata dictionary
|
||||
"""
|
||||
merged = parent_metadata.copy()
|
||||
merged.update(child_metadata)
|
||||
return merged
|
||||
|
||||
|
||||
def resolve_metadata_for_plot(plot_path: Path,
|
||||
inherited_metadata: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Resolve metadata for a specific plot by merging inherited metadata
|
||||
with plot-specific metadata.
|
||||
|
||||
Args:
|
||||
plot_path: Path to the plot file (PDF)
|
||||
inherited_metadata: Metadata inherited from folder hierarchy
|
||||
|
||||
Returns:
|
||||
Final merged metadata for the plot
|
||||
"""
|
||||
plot_stem = plot_path.stem
|
||||
plot_dir = plot_path.parent
|
||||
|
||||
# Check for plot-specific metadata files
|
||||
for suffix in ['.yaml', '.yml', '.json']:
|
||||
plot_metadata_path = plot_dir / f"{plot_stem}{suffix}"
|
||||
if plot_metadata_path.exists():
|
||||
plot_metadata = load_metadata_file(plot_metadata_path)
|
||||
return merge_metadata(inherited_metadata, plot_metadata)
|
||||
|
||||
# No plot-specific metadata found, return inherited metadata
|
||||
return inherited_metadata.copy()
|
||||
|
||||
|
||||
def save_metadata_cache(web_dir: Path,
|
||||
plot_metadata_cache: Dict[str, Dict[str, Any]]
|
||||
) -> None:
|
||||
"""
|
||||
Save plot metadata cache to meta_cache.json in the web directory.
|
||||
|
||||
Args:
|
||||
web_dir: Web directory where the cache file should be saved
|
||||
plot_metadata_cache: Dictionary mapping plot names to their metadata
|
||||
"""
|
||||
cache_path = web_dir / "meta_cache.json"
|
||||
try:
|
||||
with cache_path.open('w', encoding='utf-8') as f:
|
||||
json.dump(plot_metadata_cache, f, indent=2, ensure_ascii=False)
|
||||
print(f"Saved metadata cache: {cache_path}")
|
||||
except IOError as e:
|
||||
print(f"Warning: Could not save metadata cache {cache_path}: {e}")
|
||||
-2005
File diff suppressed because it is too large
Load Diff
@@ -3,7 +3,7 @@
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<title>{{ title }}</title>
|
||||
<link rel="stylesheet" href="assets/css/main.css">
|
||||
<link rel="stylesheet" href="{{ assets_path }}/css/main.css">
|
||||
</head>
|
||||
<body>
|
||||
<!-- Main Content -->
|
||||
@@ -184,6 +184,6 @@
|
||||
</script>
|
||||
|
||||
<!-- Main JavaScript Application -->
|
||||
<script type="module" src="assets/js/main.js"></script>
|
||||
<script type="module" src="{{ assets_path }}/js/main.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Metadata Validation Utility
|
||||
|
||||
This script validates metadata files in the gallery source directories,
|
||||
checking for proper YAML/JSON syntax and common field validation.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, List
|
||||
|
||||
|
||||
def validate_metadata_file(file_path: Path) -> tuple[bool, List[str]]:
|
||||
"""
|
||||
Validate a single metadata file.
|
||||
|
||||
Args:
|
||||
file_path: Path to the metadata file
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, error_messages)
|
||||
"""
|
||||
errors = []
|
||||
|
||||
if not file_path.exists():
|
||||
errors.append(f"File does not exist: {file_path}")
|
||||
return False, errors
|
||||
|
||||
try:
|
||||
with file_path.open('r', encoding='utf-8') as f:
|
||||
suffix_lower = file_path.suffix.lower()
|
||||
if suffix_lower in ['.yaml', '.yml']:
|
||||
data = yaml.safe_load(f)
|
||||
elif suffix_lower == '.json':
|
||||
data = json.load(f)
|
||||
else:
|
||||
errors.append(f"Unsupported file format: {file_path}")
|
||||
return False, errors
|
||||
|
||||
if data is None:
|
||||
errors.append(f"Empty metadata file: {file_path}")
|
||||
return False, errors
|
||||
|
||||
# Basic validation
|
||||
if not isinstance(data, dict):
|
||||
errors.append(f"Metadata must be a dictionary: {file_path}")
|
||||
return False, errors
|
||||
|
||||
# Check for common issues
|
||||
if 'title' in data and not isinstance(data['title'], str):
|
||||
errors.append(f"Title must be a string: {file_path}")
|
||||
|
||||
if 'tags' in data and not isinstance(data['tags'], list):
|
||||
errors.append(f"Tags must be a list: {file_path}")
|
||||
|
||||
if 'author' in data and not isinstance(data['author'], dict):
|
||||
errors.append(f"Author must be a dictionary: {file_path}")
|
||||
|
||||
except (yaml.YAMLError, json.JSONDecodeError) as e:
|
||||
errors.append(f"Parse error in {file_path}: {e}")
|
||||
return False, errors
|
||||
except Exception as e:
|
||||
errors.append(f"Unexpected error reading {file_path}: {e}")
|
||||
return False, errors
|
||||
|
||||
return len(errors) == 0, errors
|
||||
|
||||
|
||||
def find_metadata_files(root_dir: Path) -> List[Path]:
|
||||
"""
|
||||
Find all metadata files in a directory tree.
|
||||
|
||||
Args:
|
||||
root_dir: Root directory to search
|
||||
|
||||
Returns:
|
||||
List of metadata file paths
|
||||
"""
|
||||
metadata_files = []
|
||||
|
||||
for pattern in ['**/*.yaml', '**/*.yml', '**/*.json']:
|
||||
for file_path in root_dir.glob(pattern):
|
||||
if file_path.name.startswith('meta.') or file_path.stem != file_path.name:
|
||||
metadata_files.append(file_path)
|
||||
|
||||
return metadata_files
|
||||
|
||||
|
||||
def main():
|
||||
"""Main validation function."""
|
||||
if len(sys.argv) != 2:
|
||||
print("Usage: python validate_metadata.py <directory>")
|
||||
sys.exit(1)
|
||||
|
||||
root_dir = Path(sys.argv[1])
|
||||
|
||||
if not root_dir.exists():
|
||||
print(f"Error: Directory does not exist: {root_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
if not root_dir.is_dir():
|
||||
print(f"Error: Not a directory: {root_dir}")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Validating metadata files in: {root_dir}")
|
||||
print("-" * 50)
|
||||
|
||||
metadata_files = find_metadata_files(root_dir)
|
||||
|
||||
if not metadata_files:
|
||||
print("No metadata files found.")
|
||||
return
|
||||
|
||||
total_files = len(metadata_files)
|
||||
valid_files = 0
|
||||
|
||||
for file_path in metadata_files:
|
||||
is_valid, errors = validate_metadata_file(file_path)
|
||||
|
||||
if is_valid:
|
||||
print(f"✓ {file_path.relative_to(root_dir)}")
|
||||
valid_files += 1
|
||||
else:
|
||||
print(f"✗ {file_path.relative_to(root_dir)}")
|
||||
for error in errors:
|
||||
print(f" - {error}")
|
||||
|
||||
print("-" * 50)
|
||||
print(f"Summary: {valid_files}/{total_files} files valid")
|
||||
|
||||
if valid_files != total_files:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user