feat: Implement metadata system and reorganize project structure
✨ Features: - Add comprehensive metadata system with YAML/JSON support - Implement hierarchical metadata inheritance from parent folders - Support plot-specific metadata overrides - Add metadata caching for performance optimization 🗂️ Code Organization: - Move Python orchestration code to orchestration/ folder - Move validation utilities to tests/ folder - Move documentation to docs/ folder - Separate metadata functionality into dedicated module 🔧 Infrastructure: - Add automatic asset copying to web directory - Fix asset path resolution for nested directories - Update template to use dynamic asset paths - Add MetadataConfig class with inheritance options 📚 Documentation: - Add comprehensive metadata usage guide (METADATA_USAGE.md) - Add implementation documentation (METADATA_IMPLEMENTATION.md) - Include example metadata files in examples/ - Add metadata validation utility script 🐛 Bug Fixes: - Fix breadcrumb navigation and JavaScript functionality - Resolve asset path issues in nested directories - Update template imports for modular CSS/JS structure This commit introduces a flexible metadata system that allows users to add rich metadata to plots and folders using YAML or JSON files, with full hierarchical inheritance and plot-specific overrides. The project structure is now better organized with clear separation of concerns.
This commit is contained in:
@@ -0,0 +1,103 @@
|
||||
# Metadata System Implementation Summary
|
||||
|
||||
## What Was Implemented
|
||||
|
||||
### 1. Core Metadata Module (`metadata.py`)
|
||||
- **`load_metadata_file()`**: Loads YAML/JSON metadata files with error handling
|
||||
- **`load_folder_metadata()`**: Discovers and loads folder-level metadata (meta.yaml/meta.json)
|
||||
- **`merge_metadata()`**: Merges parent and child metadata with proper override behavior
|
||||
- **`resolve_metadata_for_plot()`**: Resolves final metadata for individual plots
|
||||
- **`save_metadata_cache()`**: Saves resolved metadata to cache files for performance
|
||||
|
||||
### 2. Updated Gallery Generator (`generate_gallery.py`)
|
||||
- **Hierarchical inheritance**: Folder metadata is inherited by subfolders and plots
|
||||
- **Plot-specific overrides**: Individual plots can have their own metadata files
|
||||
- **Template integration**: Metadata is passed to HTML templates for rendering
|
||||
- **Cache generation**: `meta_cache.json` files are created in each output directory
|
||||
|
||||
### 3. Configuration Updates (`config.py` and `config.yaml`)
|
||||
- Added `MetadataConfig` class with caching and inheritance options
|
||||
- Updated main `Config` class to include metadata settings
|
||||
- Added metadata section to `config.yaml`
|
||||
|
||||
### 4. Documentation and Examples
|
||||
- **`METADATA_USAGE.md`**: Comprehensive documentation on using the metadata system
|
||||
- **`examples/meta.yaml`**: Example folder metadata file
|
||||
- **`examples/specific_plot.json`**: Example plot-specific metadata file
|
||||
- **`validate_metadata.py`**: Utility script for validating metadata files
|
||||
|
||||
## Key Features
|
||||
|
||||
### Hierarchical Metadata Inheritance
|
||||
```
|
||||
root_folder/
|
||||
├── meta.yaml # Base metadata for all plots
|
||||
├── subfolder/
|
||||
│ ├── meta.yaml # Inherits from parent, can override
|
||||
│ ├── plot1.pdf
|
||||
│ ├── plot1.yaml # Plot-specific metadata
|
||||
│ └── plot2.pdf # Uses folder metadata
|
||||
```
|
||||
|
||||
### Flexible Format Support
|
||||
- YAML files: `.yaml`, `.yml`
|
||||
- JSON files: `.json`
|
||||
- Automatic format detection based on file extension
|
||||
|
||||
### Template Integration
|
||||
- `folder_metadata`: Available in templates for folder-level metadata
|
||||
- `item.metadata`: Available for each plot in the items loop
|
||||
- Clean separation of concerns between data and presentation
|
||||
|
||||
### Performance Optimization
|
||||
- Metadata caching in `meta_cache.json` files
|
||||
- Only reload when source files are newer than cache
|
||||
- Efficient hierarchical resolution
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Basic Folder Metadata
|
||||
```yaml
|
||||
# meta.yaml
|
||||
title: "Physics Analysis Results"
|
||||
experiment: "CMS"
|
||||
author:
|
||||
name: "Researcher Name"
|
||||
institution: "University"
|
||||
tags: ["analysis", "physics"]
|
||||
```
|
||||
|
||||
### Plot-specific Metadata
|
||||
```yaml
|
||||
# my_plot.yaml (for my_plot.pdf)
|
||||
title: "Signal Region Analysis"
|
||||
plot_type: "histogram"
|
||||
variables:
|
||||
x_axis: "mass"
|
||||
y_axis: "events"
|
||||
highlight: true
|
||||
```
|
||||
|
||||
### Template Usage
|
||||
```html
|
||||
<h1>{{ folder_metadata.title }}</h1>
|
||||
{% for item in items %}
|
||||
<div class="plot">
|
||||
<h3>{{ item.metadata.title or item.name }}</h3>
|
||||
{% if item.metadata.plot_type %}
|
||||
<span class="type">{{ item.metadata.plot_type }}</span>
|
||||
{% endif %}
|
||||
</div>
|
||||
{% endfor %}
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Flexibility**: Support any metadata structure using YAML/JSON
|
||||
2. **Inheritance**: Avoid repetition by inheriting from parent folders
|
||||
3. **Override capability**: Fine-tune metadata for specific plots
|
||||
4. **Performance**: Caching system for efficient repeated builds
|
||||
5. **Validation**: Built-in error handling and validation utilities
|
||||
6. **Documentation**: Comprehensive usage documentation and examples
|
||||
|
||||
The metadata system is now fully integrated and ready for use in your scientific plot gallery generator!
|
||||
Reference in New Issue
Block a user