lars/ETPlot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add fallback if ImageMagick is not available

Browse Source
This commit is contained in:
Kylian Schmidt
2026-05-15 09:07:48 +02:00
parent 73916268a2
commit 3c399d5733
4 changed files with 265 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+12 -8
View File
@@ -21,17 +21,20 @@ pytest tests/test_generate_gallery.py -v
# Run a single test by name
pytest tests/test_generate_gallery.py::test_needs_update_missing_target -v
# Generate gallery (CLI entry point after install)
gallery --config config.yaml --verbose
# Generate gallery
gallery generate --verbose
# Generate gallery (legacy script)
python generate_gallery.py --verbose
# Generate with a non-default config
gallery --config config.yaml generate --verbose
# Incremental update for one source only
gallery --source /path/to/plots --verbose
gallery generate --source /path/to/plots --verbose
# Clean regeneration
gallery --clean --verbose
gallery generate --clean --verbose
# Launch TUI
gallery tui
# Serve output locally
python -m http.server 8000 -d /web/kschmidt/public_html/
@@ -62,8 +65,9 @@ generate() [api.py]
|------|------|
| `gallery/api.py` | `generate()` — primary public entry point; orchestrates everything |
| `gallery/builder.py` | `build_gallery()` — recursive traversal; `get_template()`, `copy_assets()` |
| `gallery/config.py` | `GalleryConfig`, `GallerySource`, `GalleryDefaults` dataclasses; YAML loading |
| `gallery/cli.py` | CLI (`gallery` command) wrapping `generate()` |
| `gallery/config/__init__.py` | `GalleryConfig`, `GallerySource`, `ConfigManager` dataclasses; YAML loading |
| `gallery/cli.py` | CLI (`gallery` command) with subcommands: `generate`, `config`, `tui`, `install-completion` |
| `gallery/tui.py` | TUI (`gallery tui`) built with Textual; interactive config editor + generation |
| `gallery/utils/processing.py` | PDF→PNG via ImageMagick subprocess; `needs_update()` timestamp check; `render_gallery_page()` |
| `gallery/utils/metadata.py` | Load/merge/cache YAML+JSON metadata; per-plot metadata resolution |
| `gallery/utils/stats.py` | Directory size/count statistics |
README.md
+214 -231
View File
@@ -23,7 +23,7 @@
### Developer-Friendly
- **Python API**: Import and use programmatically in other projects
- **CLI Interface**: Command-line command `gallery`
- **CLI Interface**: Command-line command `gallery` with subcommands
- **TUI Interface**: Textual-based terminal UI `gallery tui`
- **Configuration Flexibility**: Config file stored under user `$HOME/.config/gallery`
- **Source Override**: Process single directories without full regeneration
@@ -43,38 +43,171 @@
<img src="docs/images/search.png" width="500px">
## 🚀 Installation
### From GitLab
Pip install:
```bash
pip install git+https://gitlab.com/kschmidt/web.git
```
Or git clone and `pip install .`. After installation the `gallery` command is available in your shell. Verify with:
```bash
gallery --help
```
### Dependencies
- Python 3.8+
- Python packages are installed automatically by pip (Jinja2, PyYAML, PyMuPDF, Textual, argcomplete, platformdirs)
- [ImageMagick](https://imagemagick.org/) is **optional** — used as a fallback if PyMuPDF is not available
### Shell Completion (optional)
Install tab-completion for bash/zsh/fish:
```bash
gallery install-completion
```
Then restart your shell or follow the printed instructions to activate it.
## 🖥️ CLI Usage
The `gallery` command has four subcommands: `generate`, `config`, `tui`, and `install-completion`.
### Quick start
```bash
# 1. Set your web output directory
gallery config set paths.web_folder /path/to/your/public_html
# 2. Add one or more plot source directories
gallery config add-source --path /path/to/plots --name my_analysis
# 3. Generate the gallery
gallery generate
```
### `gallery generate`
```bash
# Full regeneration
gallery generate
# Verbose output
gallery generate --verbose
# Clean rebuild (delete output before generating)
gallery generate --clean
# Regenerate a single source only
gallery generate --source /path/to/plots
# Use a non-default config file
gallery --config /path/to/config.yaml generate
```
### `gallery config`
```bash
# Show all configuration values
gallery config list
# Show the resolved config file path
gallery config path
# List configured sources
gallery config sources
# Get a single value
gallery config get gallery.png_dpi
# Set a value (all standard YAML types accepted)
gallery config set gallery.png_dpi 300
gallery config set metadata.cache_enabled true
# Add a source directory
gallery config add-source --path /path/to/plots --name my_plots
# Remove a source
gallery config remove-source my_plots
```
### Serving locally
```bash
python -m http.server 8000 -d /path/to/public_html
```
Then open `http://localhost:8000/gallery/` in your browser.
## 🖱️ TUI Usage
The TUI is a terminal user interface built with [Textual](https://textual.textualize.io/). It lets you edit the configuration and trigger generation without leaving your terminal.
```bash
gallery tui
```
### TUI Overview
The TUI is divided into collapsible sections and a persistent footer:
| Section | Contents |
|---------|----------|
| **Paths** | Web output folder |
| **Gallery Settings** | Plot root, PNG DPI, backup folder, cache and metadata options |
| **Sources** | Editable list of source directories — add or remove rows inline |
| **Generation** | Status indicator, scrollable log output |
The **footer bar** is always visible at the bottom:
| Button | Keyboard | Action |
|--------|----------|--------|
| Save Config | `Ctrl+S` | Write current values to config file |
| Generate | — | Run `gallery generate` in background |
| Quit | `Ctrl+Q` | Exit the TUI |
An **unsaved changes indicator** (`● unsaved changes`) appears in the title bar whenever a field has been modified but not yet saved.
Generation runs in a background thread and streams output line-by-line into the log widget. The status button changes colour: blue (idle) → yellow (running) → green (success) / red (error).
## ⚙️ Configuration
The config file lives at `$HOME/.config/gallery/config.yaml` by default. You can point to a different file with `gallery --config /path/to/config.yaml <subcommand>`.
### config.yaml Structure
```yaml
# Gallery configuration
web_folder: "/web/user/public_html"
plot_root: "gallery"
png_dpi: 150 # just the thumbnails
paths:
web_folder: "/web/user/public_html" # required
gallery:
plot_root: "gallery" # subdirectory inside web_folder
png_dpi: 400 # thumbnail resolution
backup_folder: "" # optional backup path
# Source directories
sources:
- name: "analysis_results"
path: "/path/to/plots/directory"
- name: "specific_plot"
path: "/path/to/single/plot.pdf"
path: "/path/to/another/directory"
# UI settings
ui:
search_debounce_ms: 300
max_recent_plots: 20
# Path settings
paths:
work_dir: "/work/directory"
metadata:
cache_enabled: true
inherit_from_parent: true
```
### Metadata Files
Create `metadata.yaml` files in your source directories. The fields are all arbitrary and rendered using
yaml object interpretation (dict, list...). You can have different metadata.yaml files in each folder,
with lower-level fields overriding the parent values. Great for labelling specific experiments.
Create `metadata.yaml` files in your source directories. Fields are arbitrary and rendered via YAML object interpretation (dict, list, …). Lower-level files override parent values — useful for labelling specific experiments.
```yaml
# metadata.yaml
@@ -98,6 +231,13 @@ authors:
- "Researcher B"
```
LaTeX formulas are supported in metadata values and rendered with MathJax:
```yaml
formula: "$$E = mc^2$$"
```
## 🎮 User Interface Guide
### Navigation Controls
@@ -109,268 +249,111 @@ authors:
| ⚖️ Compare | Side-by-side comparison | `Ctrl+C` |
| ☀️ Theme | Toggle dark/light theme | `Ctrl+T` |
### Metadata Section
![metadata](docs/images/metadata.png)
- **Toggle Button**: Show/hide folder information
- **Copy Path**: Quick access to metadata file location
- **Hover Tip**: Information about metadata file formats
- **YAML Structure**: Preserves original formatting and indentation
### View Modes
<img src="docs/images/grid_view.png" width="200px">
- **Grid View**: Thumbnail grid with metadata overlay
- **Large List**: Detailed list with larger previews
- **Compact List**: Dense list for quick scanning
## 🛠️ Development Guide
### Project Structure
```
scientific-gallery-generator/
├── generate_gallery.py # Main gallery generator
├── config.yaml # Configuration file
gallery/
├── api.py # generate() — primary public entry point
├── builder.py # build_gallery() recursive traversal; template/assets
├── cli.py # gallery CLI command (argparse + argcomplete)
├── tui.py # gallery tui (Textual TUI)
├── config/
│ └── __init__.py # GalleryConfig, GallerySource, ConfigManager
├── utils/
│ ├── metadata.py # YAML/JSON loading, inheritance, caching
│ ├── processing.py # PDF→PNG conversion; needs_update(); render_gallery_page()
│ ├── stats.py # Directory size/file count statistics
│ ├── backup.py # Backup functionality
│ └── datetime_utils.py # Date/time utilities for templates
├── templates/
│ └── gallery.html # Jinja2 template
├── assets/
│ ├── css/ # Stylesheets
│ │ ├── main.css # Main stylesheet
│ │ ├── variables.css # CSS variables
│ │ ├── metadata-section.css # Metadata styling
│ │ └── ...
│ └── js/ # JavaScript modules
│ ├── main.js # Application entry point
│ ├── gallery-app.js # Core application
│ ├── metadata-section.js # Metadata functionality
│ └── ...
├── orchestration/
│ ├── config.py # Configuration management
│ ├── metadata.py # Metadata handling
│ └── ...
└── tests/
└── validate_metadata.py # Metadata validation
│ └── gallery.html # Single Jinja2 template for all pages
└── assets/
├── css/ # Modular CSS; main.css imports all via @import
└── js/ # Vanilla ES modules; GalleryApp orchestrates all managers
```
### Running Tests
```bash
pytest tests/
pytest tests/test_generate_gallery.py -v
pytest tests/test_generate_gallery.py::test_needs_update_missing_target -v
```
### Adding New Features
#### 1. CSS Components
#### CSS component
```css
/* assets/css/new-feature.css */
.new-feature {
/* Your styles here */
}
/* gallery/assets/css/new-feature.css */
.new-feature { }
```
Add `@import url('./new-feature.css');` to `main.css`.
Add to `assets/css/main.css`:
```css
@import url('./new-feature.css');
```
#### 2. JavaScript Modules
#### JavaScript module
```javascript
// assets/js/new-feature.js
export class NewFeature {
constructor() {
this.init();
}
init() {
// Initialize your feature
}
}
// gallery/assets/js/new-feature.js
export class NewFeature { ... }
```
Import in `gallery-app.js` and instantiate in `GalleryApp`.
Import in `assets/js/gallery-app.js`:
```javascript
import { NewFeature } from './new-feature.js';
```
#### 3. Template Extensions
```html
<!-- templates/gallery.html -->
{% if new_feature_enabled %}
<div class="new-feature">
<!-- Your HTML here -->
</div>
{% endif %}
```
### Metadata System Extension
#### Custom Metadata Fields
```yaml
# metadata.yaml
custom_field: "value"
nested_data:
subfield: "nested value"
list_data:
- "item 1"
- "item 2"
```
#### LaTeX Support
```yaml
formula: "$$E = mc^2$$"
equation: "$$\\sum_{i=1}^{n} x_i = \\bar{x} \\cdot n$$"
```
## 🔧 Advanced Configuration
### ImageMagick Settings
```bash
# Increase memory limits for large PDFs
export MAGICK_MEMORY_LIMIT=2GB
export MAGICK_MAP_LIMIT=2GB
```
### Performance Optimization
```yaml
# config.yaml
png_dpi: 150 # Balance quality vs. file size
parallel_processing: true # Enable multi-threading
cache_enabled: true # Enable metadata caching
```
### Custom Styling
```css
/* Override theme colors */
:root {
--primary-color: #your-color;
--background-color: #your-bg;
}
```
## 🐛 Troubleshooting
## 🔧 Troubleshooting
### Common Issues
| Issue | Solution |
|-------|----------|
| ImageMagick not found | Install: `sudo apt-get install imagemagick` |
| Permission denied | Check file permissions and web directory access |
| PDF conversion fails | Verify PDF is not corrupted, increase memory limits |
| Metadata not showing | Check YAML syntax and file permissions |
| JavaScript errors | Clear browser cache and check console |
| `gallery` command not found | Run `pip install -e .` from the repo root |
| PDF conversion fails — no renderer | Install PyMuPDF: `pip install pymupdf` (or `apt-get install imagemagick` as fallback) |
| PDF conversion fails — corrupt file | Verify the PDF opens in a viewer; try `pip install --upgrade pymupdf` |
| Permission denied on web dir | Check file permissions and web directory access |
| Metadata not showing | Check YAML syntax with `python -c "import yaml; yaml.safe_load(open('metadata.yaml'))"` |
### ImageMagick memory limits (fallback backend only)
### Debug Mode
```bash
# Enable verbose logging
python generate_gallery.py --verbose
# Clean regeneration
python generate_gallery.py --clean
export MAGICK_MEMORY_LIMIT=2GB
export MAGICK_MAP_LIMIT=2GB
gallery generate --verbose
```
## 🤝 Contributing
### Debug / clean rebuild
**IMPORTANT**: Since this is in part a project made for fun made to test the limits of AI coding agents, all code pushed to this repository must be AI generated. Use your favorite agent of choice, mine was Github Copilot for VSCode. Bonus points if you let the agent write the commit messages too.
1. **Fork the repository**
2. **Create a feature branch**
```bash
git checkout -b feature/amazing-feature
gallery generate --verbose
gallery generate --clean --verbose
```
3. **Commit your changes**
```bash
git commit -m 'Add amazing feature'
```
4. **Push to the branch**
```bash
git push origin feature/amazing-feature
```
5. **Open a Pull Request**
### Development Setup
```bash
# Install development dependencies
pip install -r requirements-dev.txt
# Run tests
python -m pytest tests/
# Validate metadata
python tests/validate_metadata.py
```
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. See the Disclaimer section below for more information.
This project is licensed under the MIT License — see the [LICENSE](LICENSE) file for details.
## 🙏 Acknowledgments
### Disclaimer
This project was made *entirely* using Claude Sonnet 4.0 and GPT 4.1. 100% of the code was AI generated an no human hand was involved other than prompting the Agent. I do not claim any part of this project as my own work.
This project was made *entirely* using Claude Sonnet 4.0 and GPT 4.1. 100% of the code was AI generated and no human hand was involved other than prompting the Agent. I do not claim any part of this project as my own work.
### Packages:
### Packages
- **ImageMagick** for PDF to PNG conversion
- **Jinja2** for templating engine
- **PyMuPDF** for PDF to PNG conversion (pure Python, no system dependencies)
- **ImageMagick** as optional fallback PDF renderer
- **Jinja2** for templating
- **Textual** for the terminal UI
- **MathJax** for LaTeX rendering
- **PyYAML** for YAML processing
## 📚 Documentation
Currently empty
- **argcomplete** for shell tab-completion
## 🔗 Links
- [Example Gallery](https://etpwww.etp.kit.edu/~kschmidt/gallery/index.html)
### 3. **Maintainability**
- Each JavaScript module handles a specific feature area
- Modules can be independently maintained and tested
- Clear dependencies and interfaces between modules
- Individual files are much smaller and focused
- Easy to locate and modify specific functionality
- Reduced cognitive load when working on features
### 4. **Development Benefits**
- Better IDE support with syntax highlighting and intellisense
- Easier debugging with source maps
- Ability to add build tools if needed
## Module Responsibilities
### CSS Modules
- **variables.css**: Theme colors and CSS custom properties
- **base.css**: Typography, basic layout, list styles
- **navigation.css**: Breadcrumb and navigation button styles
- **search.css**: Search box, results, and highlighting
- **folder-tree.css**: Collapsible folder tree display
- **grid.css**: Plot thumbnails grid and selection states
- **sidebar.css**: Recent plots sidebar and overlay
- **floating-elements.css**: Action buttons and keyboard help
- **stats.css**: Gallery statistics display
- **comparison.css**: Plot comparison overlay
- **responsive.css**: Mobile and tablet adaptations
### JavaScript Modules
- **ThemeManager**: Light/dark theme switching and persistence
- **NavigationManager**: Breadcrumb building and folder tree construction
- **SearchManager**: Plot search with debouncing and results display
- **RecentPlotsManager**: Recent plots tracking and sidebar management
- **ComparisonManager**: Plot comparison functionality
- **StatsManager**: Gallery statistics calculation and display
- **KeyboardManager**: Keyboard shortcuts and escape handling
- **Utils**: File size formatting, thumbnail highlighting, gallery refresh
### Main Application
- **GalleryApp**: Orchestrates all managers and provides unified interface
- **main.js**: Entry point that initializes the application
## Usage
The restructured application maintains **full backward compatibility** with the original template. All existing functionality works exactly the same way.
### For Python Backend
Update your template reference to use the new template:
```python
# Instead of template.html, use:
template_path = 'templates/gallery.html'
```
### CSS Asset Path
The template expects CSS/JS assets to be served from `/assets/` relative to the gallery pages. Update your web server configuration to serve these static files.
gallery/utils/processing.py
+36 -8
View File
@@ -5,6 +5,14 @@ import subprocess
from pathlib import Path
from typing import Any, Dict
try:
import fitz # PyMuPDF
_PYMUPDF_AVAILABLE = True
except ImportError:
_PYMUPDF_AVAILABLE = False
_IMAGEMAGICK_AVAILABLE = shutil.which("convert") is not None
from jinja2 import Template
from gallery.utils.metadata import (
@@ -184,17 +192,15 @@ def render_gallery_page(
def convert_pdf_to_png(pdf_path: Path, config: GalleryConfig) -> None:
"""
Convert a PDF file to PNG format using ImageMagick.
Convert a PDF file to PNG format.
Uses PyMuPDF (fitz) when available; falls back to ImageMagick otherwise.
Only converts if the PNG doesn't exist or if the PDF is newer than
the PNG (with a 30-second buffer to handle filesystem timing issues).
Args:
pdf_path: Path to the source PDF file
config: Gallery configuration object
Raises:
subprocess.CalledProcessError: If ImageMagick conversion fails
RuntimeError: If neither PyMuPDF nor ImageMagick is available.
subprocess.CalledProcessError: If the ImageMagick fallback fails.
"""
png_path = pdf_path.with_suffix(".png")
@@ -204,12 +210,34 @@ def convert_pdf_to_png(pdf_path: Path, config: GalleryConfig) -> None:
if png_mtime >= (pdf_mtime + 30):
return
if _PYMUPDF_AVAILABLE:
_convert_pdf_pymupdf(pdf_path, png_path, config.png_dpi)
elif _IMAGEMAGICK_AVAILABLE:
_convert_pdf_imagemagick(pdf_path, png_path, config.png_dpi)
else:
raise RuntimeError(
"No PDF renderer found. Install PyMuPDF (`pip install pymupdf`) "
"or ImageMagick (`apt-get install imagemagick`)."
)
def _convert_pdf_pymupdf(pdf_path: Path, png_path: Path, dpi: int) -> None:
zoom = dpi / 72 # PDF coordinate space is 72 pt/inch
doc = fitz.open(str(pdf_path))
try:
pix = doc[0].get_pixmap(matrix=fitz.Matrix(zoom, zoom))
pix.save(str(png_path))
finally:
doc.close()
def _convert_pdf_imagemagick(pdf_path: Path, png_path: Path, dpi: int) -> None:
subprocess.run([
"convert",
"-density", str(config.png_dpi),
"-density", str(dpi),
str(pdf_path),
"-quality", "95",
str(png_path)
str(png_path),
], check=True)
pyproject.toml
+2 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "gallery"
version = "0.1.2"
version = "0.1.3"
description = "Scientific Gallery Generator - Create responsive HTML galleries from plot collections"
readme = "README.md"
requires-python = ">=3.8"
@@ -38,6 +38,7 @@ dependencies = [
"PyYAML>=5.0",
"argcomplete>=3.0",
"platformdirs>=3.0",
"pymupdf>=1.23",
"pytest",
"textual>=0.50",
]