feat: Implement metadata system and reorganize project structure
✨ Features: - Add comprehensive metadata system with YAML/JSON support - Implement hierarchical metadata inheritance from parent folders - Support plot-specific metadata overrides - Add metadata caching for performance optimization 🗂️ Code Organization: - Move Python orchestration code to orchestration/ folder - Move validation utilities to tests/ folder - Move documentation to docs/ folder - Separate metadata functionality into dedicated module 🔧 Infrastructure: - Add automatic asset copying to web directory - Fix asset path resolution for nested directories - Update template to use dynamic asset paths - Add MetadataConfig class with inheritance options 📚 Documentation: - Add comprehensive metadata usage guide (METADATA_USAGE.md) - Add implementation documentation (METADATA_IMPLEMENTATION.md) - Include example metadata files in examples/ - Add metadata validation utility script 🐛 Bug Fixes: - Fix breadcrumb navigation and JavaScript functionality - Resolve asset path issues in nested directories - Update template imports for modular CSS/JS structure This commit introduces a flexible metadata system that allows users to add rich metadata to plots and folders using YAML or JSON files, with full hierarchical inheritance and plot-specific overrides. The project structure is now better organized with clear separation of concerns.
This commit is contained in:
Kylian Schmidt
@@ -0,0 +1,103 @@
|
||||
# Metadata System Implementation Summary
|
||||
|
||||
## What Was Implemented
|
||||
|
||||
### 1. Core Metadata Module (`metadata.py`)
|
||||
- **`load_metadata_file()`**: Loads YAML/JSON metadata files with error handling
|
||||
- **`load_folder_metadata()`**: Discovers and loads folder-level metadata (meta.yaml/meta.json)
|
||||
- **`merge_metadata()`**: Merges parent and child metadata with proper override behavior
|
||||
- **`resolve_metadata_for_plot()`**: Resolves final metadata for individual plots
|
||||
- **`save_metadata_cache()`**: Saves resolved metadata to cache files for performance
|
||||
|
||||
### 2. Updated Gallery Generator (`generate_gallery.py`)
|
||||
- **Hierarchical inheritance**: Folder metadata is inherited by subfolders and plots
|
||||
- **Plot-specific overrides**: Individual plots can have their own metadata files
|
||||
- **Template integration**: Metadata is passed to HTML templates for rendering
|
||||
- **Cache generation**: `meta_cache.json` files are created in each output directory
|
||||
|
||||
### 3. Configuration Updates (`config.py` and `config.yaml`)
|
||||
- Added `MetadataConfig` class with caching and inheritance options
|
||||
- Updated main `Config` class to include metadata settings
|
||||
- Added metadata section to `config.yaml`
|
||||
|
||||
### 4. Documentation and Examples
|
||||
- **`METADATA_USAGE.md`**: Comprehensive documentation on using the metadata system
|
||||
- **`examples/meta.yaml`**: Example folder metadata file
|
||||
- **`examples/specific_plot.json`**: Example plot-specific metadata file
|
||||
- **`validate_metadata.py`**: Utility script for validating metadata files
|
||||
|
||||
## Key Features
|
||||
|
||||
### Hierarchical Metadata Inheritance
|
||||
```
|
||||
root_folder/
|
||||
├── meta.yaml # Base metadata for all plots
|
||||
├── subfolder/
|
||||
│ ├── meta.yaml # Inherits from parent, can override
|
||||
│ ├── plot1.pdf
|
||||
│ ├── plot1.yaml # Plot-specific metadata
|
||||
│ └── plot2.pdf # Uses folder metadata
|
||||
```
|
||||
|
||||
### Flexible Format Support
|
||||
- YAML files: `.yaml`, `.yml`
|
||||
- JSON files: `.json`
|
||||
- Automatic format detection based on file extension
|
||||
|
||||
### Template Integration
|
||||
- `folder_metadata`: Available in templates for folder-level metadata
|
||||
- `item.metadata`: Available for each plot in the items loop
|
||||
- Clean separation of concerns between data and presentation
|
||||
|
||||
### Performance Optimization
|
||||
- Metadata caching in `meta_cache.json` files
|
||||
- Only reload when source files are newer than cache
|
||||
- Efficient hierarchical resolution
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Folder Metadata
|
||||
```yaml
|
||||
# meta.yaml
|
||||
title: "Physics Analysis Results"
|
||||
experiment: "CMS"
|
||||
author:
|
||||
name: "Researcher Name"
|
||||
institution: "University"
|
||||
tags: ["analysis", "physics"]
|
||||
```
|
||||
|
||||
### Plot-specific Metadata
|
||||
```yaml
|
||||
# my_plot.yaml (for my_plot.pdf)
|
||||
title: "Signal Region Analysis"
|
||||
plot_type: "histogram"
|
||||
variables:
|
||||
x_axis: "mass"
|
||||
y_axis: "events"
|
||||
highlight: true
|
||||
```
|
||||
|
||||
### Template Usage
|
||||
```html
|
||||
<h1>{{ folder_metadata.title }}</h1>
|
||||
{% for item in items %}
|
||||
<div class="plot">
|
||||
<h3>{{ item.metadata.title or item.name }}</h3>
|
||||
{% if item.metadata.plot_type %}
|
||||
<span class="type">{{ item.metadata.plot_type }}</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Flexibility**: Support any metadata structure using YAML/JSON
|
||||
2. **Inheritance**: Avoid repetition by inheriting from parent folders
|
||||
3. **Override capability**: Fine-tune metadata for specific plots
|
||||
4. **Performance**: Caching system for efficient repeated builds
|
||||
5. **Validation**: Built-in error handling and validation utilities
|
||||
6. **Documentation**: Comprehensive usage documentation and examples
|
||||
|
||||
The metadata system is now fully integrated and ready for use in your scientific plot gallery generator!
|
||||
@@ -0,0 +1,114 @@
|
||||
# Metadata System Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The metadata system allows you to add flexible metadata to your plots and folders using YAML or JSON files. Metadata is inherited hierarchically from parent folders and can be overridden at any level.
|
||||
|
||||
## File Structure
|
||||
|
||||
### Folder Metadata
|
||||
- **File names**: `meta.yaml`, `meta.yml`, or `meta.json`
|
||||
- **Location**: Place in any folder containing plots
|
||||
- **Scope**: Applies to all plots in the folder and subfolders (unless overridden)
|
||||
|
||||
### Plot-specific Metadata
|
||||
- **File names**: `{plot_name}.yaml`, `{plot_name}.yml`, or `{plot_name}.json`
|
||||
- **Location**: Place in the same folder as the plot PDF file
|
||||
- **Scope**: Applies only to the specific plot with the same name
|
||||
|
||||
## Hierarchy and Inheritance
|
||||
|
||||
1. **Root folder**: Start with folder metadata in your source directory
|
||||
2. **Subfolders**: Each subfolder can have its own `meta.yaml` that merges with parent metadata
|
||||
3. **Plot-specific**: Individual plots can have their own metadata files that override folder metadata
|
||||
|
||||
## Example Usage
|
||||
|
||||
### Folder Structure
|
||||
```
|
||||
analysis_results/
|
||||
├── meta.yaml # Root folder metadata
|
||||
├── signal/
|
||||
│ ├── meta.yaml # Signal-specific metadata
|
||||
│ ├── mass_plot.pdf
|
||||
│ └── mass_plot.yaml # Plot-specific metadata
|
||||
└── background/
|
||||
├── meta.yaml # Background-specific metadata
|
||||
└── qcd_plot.pdf
|
||||
```
|
||||
|
||||
### Example Metadata Fields
|
||||
|
||||
**Common fields for folder metadata:**
|
||||
- `title`: Folder title
|
||||
- `description`: Folder description
|
||||
- `experiment`: Experiment name (CMS, ATLAS, etc.)
|
||||
- `dataset`: Dataset identifier
|
||||
- `analysis_type`: Type of analysis
|
||||
- `author`: Author information
|
||||
- `parameters`: Analysis parameters
|
||||
- `tags`: Categorization tags
|
||||
|
||||
**Common fields for plot metadata:**
|
||||
- `plot_type`: Type of plot (histogram, scatter, etc.)
|
||||
- `variables`: Variable information (x_axis, y_axis, units)
|
||||
- `selection`: Selection criteria
|
||||
- `statistics`: Statistical information
|
||||
- `display`: Display options (highlight, featured, order_priority)
|
||||
|
||||
## Configuration
|
||||
|
||||
The metadata system can be configured in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
cache_enabled: true # Enable metadata caching
|
||||
inherit_from_parent: true # Enable hierarchical inheritance
|
||||
```
|
||||
|
||||
## Output
|
||||
|
||||
### HTML Template
|
||||
Metadata is available in the HTML template as:
|
||||
- `folder_metadata`: Current folder's resolved metadata
|
||||
- `item.metadata`: Individual plot metadata (in items loop)
|
||||
|
||||
### Cache Files
|
||||
- `meta_cache.json`: Generated in each web directory
|
||||
- Contains resolved metadata for all plots in that directory
|
||||
- Used for performance optimization and debugging
|
||||
|
||||
## Usage Tips
|
||||
|
||||
1. **Start simple**: Begin with basic folder metadata and add complexity as needed
|
||||
2. **Use inheritance**: Put common metadata in parent folders to avoid repetition
|
||||
3. **Override selectively**: Use plot-specific metadata only when needed
|
||||
4. **Consistent naming**: Use consistent field names across your metadata files
|
||||
5. **Validate format**: Ensure YAML/JSON files are valid before running the generator
|
||||
|
||||
## Integration with Templates
|
||||
|
||||
In your HTML templates, you can access metadata like:
|
||||
|
||||
```html
|
||||
<!-- Folder metadata -->
|
||||
<h2>{{ folder_metadata.title }}</h2>
|
||||
<p>{{ folder_metadata.description }}</p>
|
||||
|
||||
<!-- Plot metadata -->
|
||||
{% for item in items %}
|
||||
<div class="plot-item">
|
||||
<h3>{{ item.name }}</h3>
|
||||
{% if item.metadata.plot_type %}
|
||||
<span class="plot-type">{{ item.metadata.plot_type }}</span>
|
||||
{% endif %}
|
||||
{% if item.metadata.tags %}
|
||||
<div class="tags">
|
||||
{% for tag in item.metadata.tags %}
|
||||
<span class="tag">{{ tag }}</span>
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
```
|
||||
Reference in New Issue
Block a user