Rewrite gallery as an installable package, add plotstyle, migrate build/CI to uv #1

Merged

lars merged 36 commits from dev into main 2026-07-24 10:49:38 +02:00

Conversation 0 Commits 36 Files Changed 54

+5608 -2919

lars commented 2026-07-24 10:45:36 +02:00

Owner

Copy Link

Copy Source

Summary

This brings main up to the current state of dev: a full package rewrite
of the gallery generator, a new companion plotting toolkit (plotstyle),
a production deployment setup, and a build/CI system migrated fully to uv.

Gallery: features and UI

• Metadata system: YAML/JSON metadata per folder, with hierarchical
  inheritance from parent folders, per-plot overrides, and an expandable
  metadata grid in the UI (long-line wrapping, inline LaTeX rendering via
  MathJax, copyable file paths).
• Grid / list / compact view toggle, plot comparison between two plots in
  the same folder, breadcrumb navigation, search.
• Incremental generation: PDF→PNG conversion and file copies are skipped
  when the target is already up to date (30s mtime buffer), so repeat runs
  only touch changed plots.
• Structured, tree-formatted logging throughout the pipeline (replacing
  scattered prints), and various UI fixes (spacing, long filename
  handling, subdirectory navigation reload).
• Project reorganized into an installable gallery package (CLI, TUI,
  config manager, Jinja2 template, modular CSS/JS) with a proper pytest
  suite.

New: plotstyle

A standalone matplotlib styling toolkit for producing the plots a gallery
source directory displays (no code dependency on gallery — the two only
meet on disk via PDFs + metadata.yaml). KIT corporate-design color
palette (categorical/sequential/diverging/status), a theme applied via
use(), new_figure() with size presets and title/params subtitles, a
same-size colorbar(), style_legend(), panel_label(), no_spines()
for pixel/bin-indexed plots, and savefig(). Ships with a full test suite
and a rendered example notebook (examples/plotstyle_showcase.ipynb).

Deployment

• Replaced the old Singularity container with Docker: multi-stage
  Dockerfile, docker-compose.yml (generator + nginx services sharing a
  volume), deploy/nginx.conf, deploy/entrypoint.sh.
• Added a plain-venv + systemd --user timer path for HPC login nodes
  without a Docker daemon (deploy/systemd/), for environments where
  public_html is already served by existing infrastructure.

CI/CD and build system

• Migrated CI from GitLab to Gitea Actions, then rewrote it to a
  lint/format/typecheck/audit/test pipeline (ruff, ty, pip-audit, pytest)
  decoupled from the Docker image build — Docker is now deploy-only.
• Moved off setuptools entirely. Both pyproject.toml files
  (gallery and plotstyle) now build with uv's own uv_build backend,
  and the repo is a proper uv workspace (plotstyle is a workspace
  member with its own src-layout package, matplotlib as its own direct
  dependency rather than a gallery extra; dev tooling lives in a
  dependency-group).
• CI and the Dockerfile now use uv end-to-end (astral-sh/setup-uv,
  uv sync, uv run, uv build, uv publish) instead of pip/build/
  twine. Lint/format/typecheck run as separate per-package steps
  (gallery, plotstyle, tests) so a failure points at exactly which
  package broke.
• Added .gitea/workflows/publish-plotstyle.yml: pushing a plotstyle-v*
  tag builds and publishes plotstyle to this repo's Gitea package
  registry.

Cleanup

Removed code superseded by the package restructuring: the old top-level
generate_gallery.py/orchestration//root templates/assets, the
never-wired backup_folder feature, the unfinished export-to-LaTeX
JS/CSS, obsolete Docker-container smoke tests, and stale docs describing
workflows that no longer exist.

## Summary This brings `main` up to the current state of `dev`: a full package rewrite of the gallery generator, a new companion plotting toolkit (`plotstyle`), a production deployment setup, and a build/CI system migrated fully to `uv`. ## Gallery: features and UI - Metadata system: YAML/JSON metadata per folder, with hierarchical inheritance from parent folders, per-plot overrides, and an expandable metadata grid in the UI (long-line wrapping, inline LaTeX rendering via MathJax, copyable file paths). - Grid / list / compact view toggle, plot comparison between two plots in the same folder, breadcrumb navigation, search. - Incremental generation: PDF→PNG conversion and file copies are skipped when the target is already up to date (30s mtime buffer), so repeat runs only touch changed plots. - Structured, tree-formatted logging throughout the pipeline (replacing scattered `print`s), and various UI fixes (spacing, long filename handling, subdirectory navigation reload). - Project reorganized into an installable `gallery` package (CLI, TUI, config manager, Jinja2 template, modular CSS/JS) with a proper pytest suite. ## New: `plotstyle` A standalone matplotlib styling toolkit for producing the plots a gallery source directory displays (no code dependency on `gallery` — the two only meet on disk via PDFs + `metadata.yaml`). KIT corporate-design color palette (categorical/sequential/diverging/status), a theme applied via `use()`, `new_figure()` with size presets and title/params subtitles, a same-size `colorbar()`, `style_legend()`, `panel_label()`, `no_spines()` for pixel/bin-indexed plots, and `savefig()`. Ships with a full test suite and a rendered example notebook (`examples/plotstyle_showcase.ipynb`). ## Deployment - Replaced the old Singularity container with Docker: multi-stage `Dockerfile`, `docker-compose.yml` (generator + nginx services sharing a volume), `deploy/nginx.conf`, `deploy/entrypoint.sh`. - Added a plain-venv + `systemd --user` timer path for HPC login nodes without a Docker daemon (`deploy/systemd/`), for environments where `public_html` is already served by existing infrastructure. ## CI/CD and build system - Migrated CI from GitLab to Gitea Actions, then rewrote it to a lint/format/typecheck/audit/test pipeline (ruff, ty, pip-audit, pytest) decoupled from the Docker image build — Docker is now deploy-only. - **Moved off setuptools entirely.** Both `pyproject.toml` files (`gallery` and `plotstyle`) now build with uv's own `uv_build` backend, and the repo is a proper **uv workspace** (`plotstyle` is a workspace member with its own src-layout package, matplotlib as its own direct dependency rather than a `gallery` extra; dev tooling lives in a `dependency-group`). - CI and the `Dockerfile` now use `uv` end-to-end (`astral-sh/setup-uv`, `uv sync`, `uv run`, `uv build`, `uv publish`) instead of `pip`/`build`/ `twine`. Lint/format/typecheck run as separate per-package steps (`gallery`, `plotstyle`, `tests`) so a failure points at exactly which package broke. - Added `.gitea/workflows/publish-plotstyle.yml`: pushing a `plotstyle-v*` tag builds and publishes `plotstyle` to this repo's Gitea package registry. ## Cleanup Removed code superseded by the package restructuring: the old top-level `generate_gallery.py`/`orchestration/`/root `templates`/`assets`, the never-wired `backup_folder` feature, the unfinished export-to-LaTeX JS/CSS, obsolete Docker-container smoke tests, and stale docs describing workflows that no longer exist.

lars added 36 commits 2026-07-24 10:45:36 +02:00

Refactor gallery generator with smart file handling and improved UI ... 976d90f26b

Key improvements:
- Add intelligent file update checking with 30-second buffer to avoid unnecessary operations
- Implement efficient PDF to PNG conversion (only when source is newer)
- Add smart file copying (skip if target is up-to-date)
- Create proper folder structure using plot_root/source_name pattern
- Improve template with better text handling for long plot names
- Add text wrapping, truncation, and hover tooltips for plot names
- Remove unnecessary directory cleaning for true incremental updates
- Fix title display issue (was showing '.' instead of 'Gallery')
- Add comprehensive logging showing what's processed vs skipped

Performance benefits:
- Subsequent runs are significantly faster (only processes changed files)
- Reduces ImageMagick conversions and file I/O operations
- Maintains file system timing robustness with buffer delays

UI improvements:
- Better handling of long filenames with word wrapping
- Constrained text areas prevent overlap between thumbnails
- Hover tooltips show full names when truncated
- Responsive grid layout maintained

Update and refactor 9041abb72f

Update b5b2c676b3

Add comparison between plots 71cf2031e0

Move jv and css to own folder 716e1c95fe

feat: Implement metadata system and reorganize project structure ... 72ebc5e103

✨ Features:
- Add comprehensive metadata system with YAML/JSON support
- Implement hierarchical metadata inheritance from parent folders
- Support plot-specific metadata overrides
- Add metadata caching for performance optimization

🗂️ Code Organization:
- Move Python orchestration code to orchestration/ folder
- Move validation utilities to tests/ folder
- Move documentation to docs/ folder
- Separate metadata functionality into dedicated module

🔧 Infrastructure:
- Add automatic asset copying to web directory
- Fix asset path resolution for nested directories
- Update template to use dynamic asset paths
- Add MetadataConfig class with inheritance options

📚 Documentation:
- Add comprehensive metadata usage guide (METADATA_USAGE.md)
- Add implementation documentation (METADATA_IMPLEMENTATION.md)
- Include example metadata files in examples/
- Add metadata validation utility script

🐛 Bug Fixes:
- Fix breadcrumb navigation and JavaScript functionality
- Resolve asset path issues in nested directories
- Update template imports for modular CSS/JS structure

This commit introduces a flexible metadata system that allows users to add
rich metadata to plots and folders using YAML or JSON files, with full
hierarchical inheritance and plot-specific overrides. The project structure
is now better organized with clear separation of concerns.

Add metadata button (WIP) 51a74b23f8

Add metadata (WIP) 8173d7618a

Add export to latex button (i think) 2d98ed0e7a

Fix template rendering issue and improve export functionality ... dbd67f6039

- Fix template rendering by storing rendered HTML in variable before writing
  This resolves subdirectories not appearing in navigation despite being
  correctly detected and included in template variables
- Add missing template variables (paths.work_dir) to gallery config
- Add CGI refresh handler for web-based gallery regeneration
- Improve PDF export functionality with better error handling
- Add ESC key shortcut documentation for exiting selection mode
- Enhanced export manager with proper dependency checking

Fixes issue where new subdirectories were detected but not displayed
in browser navigation due to incomplete template rendering.

Fix index not being reloaded if a subfolder was added 7a9275819a

Try to fix metadata stuff again (does not work) 465f9b6dc6

Add gallery index.html to website 491a4a9fd2

Add pyproject.toml 6eeacdc9b7

Remove features that were not working: Metadata, export button and reloading the website from the website itself dced82a56f

Add toggle for grid, list and compact list views 3e2aaa4be8

Added so much stuff I cant keep up. I think some sorting capability and a way to add metadata to each folder (WIP) f3adbe49fa

Add an expandable metadata grid bcb083bbb9

Add the metadata file path for easy copy+paste fc4928fef1

Fix metadata display not wrapping long lines. Add LaTeX support c6882b2a48

Fix metadata lists being shown in a single line 752cee2d06

Add proper logging eb770428b9

Fix UI. Reduce excessive spacing between plot grid and subdirectories ... 0a345a0f22

- Reduce plot-container margin-bottom from 450px to 2rem in view-controls.css
- Reduce body padding-bottom from 400px to 2rem in base.css
- Fix unnecessary spacing since stats box uses fixed positioning
- Enhance logger with professional tree-structured output and colors
- Replace print statements with structured logging throughout codebase

Merge main into dev: bring dev up to date with restructured package ... 8e3b928bb0

Resolves add/add conflicts in .gitignore, config.yaml, pyproject.toml and
content conflict in README.md — taking main's version in all cases, as it
reflects the complete package rewrite (gallery/ package, TUI, CI pipeline).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace Singularity with Docker; add production deployment setup ... 8cae6533d5

- Remove Singularity.def; add Dockerfile (python:3.11-slim + ImageMagick,
  non-root user, installs gallery package with dev extras)
- Add .dockerignore to keep image lean
- Rewrite .gitlab-ci.yml: build→test→publish stages using Docker-in-Docker;
  push per-commit SHA tag and promote to :latest on main
- Add docker-compose.yml: gallery-generator + nginx services sharing a
  named volume; configurable GENERATE_INTERVAL env var
- Add deploy/nginx.conf: gzip, security headers, correct caching policy
  (immutable for assets, no-store for HTML)
- Add deploy/entrypoint.sh: runs gallery generate on startup then loops
  on GENERATE_INTERVAL; exits cleanly when interval is 0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add plotstyle: reusable KIT-branded matplotlib styling toolkit ... 36ae18e880

Introduces a new plotstyle package providing consistent, presentation/thesis-ready
matplotlib figures: a KIT corporate-design color palette (categorical, sequential,
diverging, status), a theme applied via use() (KIT-black bottom spine only, left/bottom
ticks, horizontal gridlines, left-aligned titles, LaTeX text in Latin Modern Sans),
new_figure() with size presets and figure-level title/params subtitles, a
same-size colorbar() helper, style_legend() and panel_label() building blocks, and
savefig(). Ships as an optional "plotting" extra so core gallery installs stay
lightweight, with a full test suite and a runnable example notebook.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Add ps.no_spines() for imshow/pcolormesh/2D-histogram plots ... eeb483a4fa

Image- and bin-indexed plots have no meaningful x baseline, so the
themed bottom spine kept by use() doesn't apply to them. Adds an
explicit opt-in to disable all spines on such Axes, documents it in
plotstyle/CLAUDE.md, and updates the showcase notebook's colormap and
multi-panel examples to use it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Document plotstyle and the Docker deployment switch in CLAUDE.md/README ... 4ba321b327

Root-level docs hadn't caught up with the plotstyle package addition
(KIT styling toolkit, plotting extra, plotstyle/CLAUDE.md pointer) or
the earlier Singularity-to-Docker deployment switch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Rewrite CI to lint/typecheck/audit/test only; add HPC deployment path ... 3b9d1ef1a8

- Replace the Docker build/publish CI stages with ruff (lint + format
  check), ty (type check), pip-audit, and pytest run directly against
  python:3.11-slim; Docker remains for manual/server deployment only.
- Swap black/pylint/mypy for ruff/ty across pyproject.toml, and fix
  every resulting lint, format, and type diagnostic in gallery/ and
  plotstyle/.
- Fix tests broken/stale from before the package restructuring: wrong
  `utils.*` import paths, mock patch targets pointed at the wrong
  module, and PDF-conversion tests still assuming ImageMagick instead
  of the current PyMuPDF-first path. Drop test_container.py (obsolete
  Docker-container smoke tests, fully superseded elsewhere).
- Add a plain-venv + systemd --user timer deployment path
  (deploy/systemd/) for HPC login nodes without a Docker daemon, where
  public_html is already served by existing infrastructure.
- Document both in CLAUDE.md, including running CI's checks locally
  before committing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Migrate CI from GitLab to Gitea Actions ... ea5ec680c5

Replace .gitlab-ci.yml with .gitea/workflows/ci.yml, keeping the same
lint/format/typecheck/pip-audit/pytest jobs against python:3.11-slim,
and update docs that referenced the old GitLab pipeline file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Fix Gitea Actions: drop custom job container, use setup-python ... f7294909a2

The python:3.11-slim job container lacked a Node.js runtime, so the
runner couldn't exec actions/checkout inside it. Run jobs on the
default runner image instead and pin Python via actions/setup-python.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Skip editable package in pip-audit; require setuptools>=83.0.0 ... 2a7d3d244a

pip-audit can't look up the local editable "gallery" package on PyPI,
so skip it explicitly. Also bump the setuptools build requirement to
pull in recent security fixes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Pin setuptools>=83.0.0 as a dev dependency ... 067465503a

The build-system requirement only pins the isolated build backend, not
the runtime venv setuptools that pip-audit scans, so CI still saw the
vulnerable 79.0.1. Add it to the dev extra so pip actually installs
the patched version.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Remove pre-package-restructure cruft and dead features ... 2020ac4883

Delete the top-level implementation superseded by the gallery/ package
conversion (generate_gallery.py, orchestration/, root templates/ and
assets/, python/ scripts), stray scratch files, and docs describing a
container/GitLab-CI coverage workflow that no longer exists. Also drop
two half-wired, never-invoked features: the backup_folder config/TUI
option (create_backup() was never called from the pipeline) and the
unfinished export-to-LaTeX JS/CSS. Update README's install instructions
to the current Gitea remote.

Move build system from setuptools to uv, wire uv into CI/CD ... 5109b6505c

Convert both pyproject.toml files to the uv_build backend and turn the
repo into a proper uv workspace (plotstyle moved to src layout, since
uv_build requires the module in its own subdirectory). Dev tooling
moves from an optional-dependencies extra to a dependency-group, and
plotstyle's matplotlib dependency is now its own rather than a
gallery "plotting" extra.

Gitea Actions workflows and the Dockerfile now use astral-sh/setup-uv,
uv sync, uv run, uv build, and uv publish throughout instead of pip,
build, and twine. Also fixes .dockerignore, which was excluding
uv.lock, deploy/, and README.md and would have broken even the
previous Dockerfile's COPY of deploy/entrypoint.sh.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Split CI lint/format/typecheck steps per package ... 5064aec831

Run ruff check, ruff format --check, and ty check as separate steps
per package (gallery, plotstyle, tests) instead of one combined
command, so a failure points at which package broke.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars merged commit f3c03a0d0c into main 2026-07-24 10:49:38 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lars (Lars Bogner)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/ETPlot#1