lars/ETPlot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor gallery generator with smart file handling and improved UI

Browse Source 
Key improvements:
- Add intelligent file update checking with 30-second buffer to avoid unnecessary operations
- Implement efficient PDF to PNG conversion (only when source is newer)
- Add smart file copying (skip if target is up-to-date)
- Create proper folder structure using plot_root/source_name pattern
- Improve template with better text handling for long plot names
- Add text wrapping, truncation, and hover tooltips for plot names
- Remove unnecessary directory cleaning for true incremental updates
- Fix title display issue (was showing '.' instead of 'Gallery')
- Add comprehensive logging showing what's processed vs skipped

Performance benefits:
- Subsequent runs are significantly faster (only processes changed files)
- Reduces ImageMagick conversions and file I/O operations
- Maintains file system timing robustness with buffer delays

UI improvements:
- Better handling of long filenames with word wrapping
- Constrained text areas prevent overlap between thumbnails
- Hover tooltips show full names when truncated
- Responsive grid layout maintained
This commit is contained in:
Kylian Schmidt
2025-07-07 13:27:25 +02:00
parent c2943bbe7b
commit 976d90f26b
8 changed files with 576 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -88
View File
@@ -1,93 +1,9 @@
# web
## Getting started
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/topics/git/add_files/#add-files-to-a-git-repository) or push an existing Git repository with the following command:
```
cd existing_repo
git remote add origin https://gitlab.etp.kit.edu/kschmidt/web.git
git branch -M main
git push -uf origin main
```
## Integrate with your tools
- [ ] [Set up project integrations](https://gitlab.etp.kit.edu/kschmidt/web/-/settings/integrations)
## Collaborate with your team
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Set auto-merge](https://docs.gitlab.com/user/project/merge_requests/auto_merge/)
## Test and Deploy
Use the built-in continuous integration in GitLab.
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
# Editing this README
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
# Automated web-based plotting tool
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
Requires `Jinja2`
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
## Config
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
## License
For open source projects, say how it is licensed.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
## Usage
__pycache__/config.cpython-312.pyc
BIN
View File
Binary file not shown.
backup.py
+23
View File
@@ -0,0 +1,23 @@
import zipfile
import datetime
from pathlib import Path
WEB_FOLDER = Path("plots")
BACKUP_FOLDER = Path("backups")
today = datetime.date.today().strftime("%Y%m%d")
backup_name = f"backup-{today}.zip"
backup_path = BACKUP_FOLDER / backup_name
BACKUP_FOLDER.mkdir(parents=True, exist_ok=True)
if backup_path.exists():
print(f"Backup already exists: {backup_path}")
else:
print(f"Creating backup: {backup_path}")
with zipfile.ZipFile(backup_path, "w", zipfile.ZIP_DEFLATED) as zipf:
for path in WEB_FOLDER.rglob("*"):
if path.is_file():
arcname = path.relative_to(WEB_FOLDER.parent)
zipf.write(path, arcname)
print("✅ Backup complete.")
config.py
+64
View File
@@ -0,0 +1,64 @@
from dataclasses import dataclass, field, asdict
from pathlib import Path
import yaml
@dataclass
class GalleryItem:
name: str
path: Path
@dataclass
class Config:
web_folder: str = ""
backup_folder: str = ""
png_dpi: int = 400
plot_root: str = "gallery"
sources: list[GalleryItem] = field(default_factory=list)
@classmethod
def from_yaml(cls, yaml_file: str) -> "Config":
"""
Load configuration from a YAML file.
Args:
yaml_file (str): Path to the YAML file.
strict (bool):
If True, raises an error if a key in the YAML file does not exist in the Config class.
If False (default), adds all keys as attributes
Returns:
Config: Instance of this class
"""
with open(yaml_file, "r") as f:
new_config: dict = yaml.safe_load(f)
new_config["sources"] = [
GalleryItem(name=src["name"], path=Path(src["path"]))
for src in new_config.get("sources", False)
]
instance = cls(**{
k: v
for k, v in new_config.items()
})
return instance
def to_yaml(self, yaml_file: str) -> None:
"""
Save the current configuration to a YAML file.
Args:
yaml_file (str): Path to the YAML file.
"""
with open(yaml_file, "w") as f:
yaml.dump(asdict(self), f, default_flow_style=False)
if __name__ == "__main__":
config = Config()
config.to_yaml("config.yaml")
config.yaml
+7
View File
@@ -0,0 +1,7 @@
backup_folder: ''
plot_root: gallery
png_dpi: 400
sources:
- name: "ttbar_analysis"
path: "/work/kschmidt/NEEDLE/test_analysis/data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/"
web_folder: '/web/kschmidt/public_html/'
generate_gallery.py
+184
View File
@@ -0,0 +1,184 @@
import subprocess
import shutil
from pathlib import Path
from jinja2 import Environment, FileSystemLoader
from config import Config
CONFIG = Config.from_yaml("config.yaml")
env = Environment(loader=FileSystemLoader("."))
template = env.get_template("template.html")
def convert_pdf_to_png(pdf_path: Path):
png_path = pdf_path.with_suffix(".png")
# Check if PNG exists and is newer than PDF (with 30 second buffer)
if png_path.exists():
pdf_mtime = pdf_path.stat().st_mtime
png_mtime = png_path.stat().st_mtime
if png_mtime >= (pdf_mtime + 30): # 30 second buffer
# PNG is up to date, no need to convert
return
else:
print(f"PDF {pdf_path.name} is newer than PNG, reconverting...")
print(f"Converting {pdf_path} → {png_path}")
subprocess.run([
"convert",
"-density", str(CONFIG.png_dpi),
str(pdf_path),
"-quality", "95",
str(png_path)
], check=True)
def needs_update(source_file: Path, target_file: Path) -> bool:
"""Check if target file needs to be updated based on source file modification time"""
if not target_file.exists():
return True
source_mtime = source_file.stat().st_mtime
target_mtime = target_file.stat().st_mtime
# Return True if source is newer than target (with 30 second buffer)
return source_mtime > (target_mtime + 30)
def build_gallery(source_dir: Path, web_dir: Path, relative_path: Path = None):
"""
Build gallery for a directory, copying files to web directory and generating index.html
Args:
source_dir: Source directory containing PDFs
web_dir: Target web directory
relative_path: Relative path for navigation (None for root)
"""
if relative_path is None:
relative_path = Path(".")
# Ensure web directory exists
target_dir = web_dir / relative_path
target_dir.mkdir(parents=True, exist_ok=True)
# Find all PDFs in source directory
pdfs = sorted(source_dir.glob("*.pdf"))
items = []
for pdf in pdfs:
# Convert PDF to PNG in source directory
convert_pdf_to_png(pdf)
png_path = pdf.with_suffix(".png")
# Copy both PDF and PNG to target directory only if needed
target_pdf = target_dir / pdf.name
target_png = target_dir / png_path.name
if needs_update(pdf, target_pdf):
print(f"Copying {pdf.name} to {target_pdf}")
shutil.copy2(pdf, target_pdf)
else:
print(f"Skipping {pdf.name} (up to date)")
if png_path.exists() and needs_update(png_path, target_png):
print(f"Copying {png_path.name} to {target_png}")
shutil.copy2(png_path, target_png)
elif png_path.exists():
print(f"Skipping {png_path.name} (up to date)")
items.append({
"name": pdf.name,
"pdf_href": pdf.name,
"png_href": png_path.name
})
# Find subdirectories in source
subdirs = sorted([d.name for d in source_dir.iterdir() if d.is_dir()])
# Generate index.html in target directory
output_html = target_dir / "index.html"
# Create a meaningful title
if str(relative_path) == ".":
title = "Gallery"
else:
title = f"Gallery: {relative_path}"
with output_html.open("w") as f:
f.write(template.render(
title=title,
items=items,
subdirs=subdirs,
relpath=str(relative_path)
))
print(f"Generated {output_html}")
# Recursively process subdirectories
for subdir in subdirs:
source_subdir = source_dir / subdir
new_relative_path = relative_path / subdir
build_gallery(source_subdir, web_dir, new_relative_path)
def loop_over_sources():
web_root = Path(CONFIG.web_folder)
for source in CONFIG.sources:
print(f"Processing {source.name}: {source.path}")
if source.path.is_dir():
# Create a subdirectory based on plot_root and source name
source_web_dir = web_root / CONFIG.plot_root / source.name
build_gallery(source.path, source_web_dir)
print(f"Processed {source.name}: {source.path}")
elif source.path.suffix == ".pdf":
# For single PDF files, copy to web root with plot_root and source name as directory
source_web_dir = web_root / CONFIG.plot_root / source.name
convert_pdf_to_png(source.path)
png_path = source.path.with_suffix(".png")
# Ensure web directory exists
source_web_dir.mkdir(parents=True, exist_ok=True)
# Copy PDF and PNG to web directory only if needed
target_pdf = source_web_dir / source.path.name
target_png = source_web_dir / png_path.name
if needs_update(source.path, target_pdf):
print(f"Copying {source.path.name} to {target_pdf}")
shutil.copy2(source.path, target_pdf)
else:
print(f"Skipping {source.path.name} (up to date)")
if png_path.exists() and needs_update(png_path, target_png):
print(f"Copying {png_path.name} to {target_png}")
shutil.copy2(png_path, target_png)
elif png_path.exists():
print(f"Skipping {png_path.name} (up to date)")
# Create items list for template
items = [{
"name": source.path.name,
"pdf_href": source.path.name,
"png_href": png_path.name
}]
# Generate index.html for single PDF
output_html = source_web_dir / "index.html"
with output_html.open("w") as f:
f.write(template.render(
title=source.name,
items=items,
subdirs=[],
relpath=source.name
))
print(f"Generated {output_html}")
print(f"Processed {source.name}: {source.path}")
if __name__ == "__main__":
loop_over_sources()
print("Done")
template.html
+59
View File
@@ -0,0 +1,59 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>{{ title }}</title>
<style>
body { font-family: sans-serif; margin: 1em; }
h1 { font-size: 1.5em; }
.grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
gap: 1rem;
padding: 1rem 0;
}
.grid-item img {
max-width: 100%;
border: 1px solid #ccc;
border-radius: 4px;
}
.grid-item {
text-align: center;
max-width: 200px; /* Constrain to grid column width */
}
.plot-name {
margin-top: 0.5rem;
word-wrap: break-word;
word-break: break-word;
hyphens: auto;
font-size: 0.9em;
line-height: 1.2;
max-height: 3.6em; /* Limit to ~3 lines */
overflow: hidden;
padding: 0 0.2rem;
}
</style>
</head>
<body>
<h1>{{ relpath or 'Root' }}</h1>
<div class="grid">
{% for item in items %}
<div class="grid-item">
<a href="{{ item.pdf_href }}" target="_blank">
<img src="{{ item.png_href }}" alt="{{ item.name }}">
</a>
<div class="plot-name" title="{{ item.name }}">{{ item.name }}</div>
</div>
{% endfor %}
</div>
{% if subdirs %}
<h2>Subdirectories</h2>
<ul>
{% for sub in subdirs %}
<li><a href="{{ sub }}/index.html">{{ sub }}</a></li>
{% endfor %}
</ul>
{% endif %}
</body>
</html>
test.ipynb
+235
View File
@@ -0,0 +1,235 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 2,
"id": "aedbb9f5",
"metadata": {},
"outputs": [],
"source": [
"from dataclasses import dataclass, field, asdict\n",
"from pathlib import Path\n",
"\n",
"import yaml\n",
"\n",
"\n",
"@dataclass\n",
"class GalleryItem:\n",
" name: str\n",
" path: Path\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "66624faf",
"metadata": {},
"outputs": [],
"source": [
"yaml_file = Path(\"config.yaml\")\n",
"\n",
"with open(yaml_file, \"r\") as f:\n",
" new_config: dict = yaml.safe_load(f)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "40a472f8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'name': 'test_1_plot',\n",
" 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/plot__proc_3_5606769559__cat_incl__var_jet1_pt.pdf'}"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"new_config[\"sources\"][0]"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "745c97ad",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[GalleryItem(name='test_1_plot', path=PosixPath('data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/plot__proc_3_5606769559__cat_incl__var_jet1_pt.pdf')),\n",
" GalleryItem(name='test_2_dir', path=PosixPath('data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1'))]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sources = [\n",
" GalleryItem(name=src[\"name\"], path=Path(src[\"path\"]))\n",
" for src in new_config.get(\"sources\", False)\n",
"]\n",
"sources"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "a0102f9d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Config(web_folder='', backup_folder='', png_dpi=400, plot_root='gallery', sources=[{'name': 'test_1_plot', 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/plot__proc_3_5606769559__cat_incl__var_jet1_pt.pdf'}, {'name': 'test_2_dir', 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/'}])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"\n",
"@dataclass\n",
"class Config:\n",
" web_folder: str = \"\"\n",
" backup_folder: str = \"\"\n",
" png_dpi: int = 400\n",
" plot_root: str = \"gallery\"\n",
" sources: list[GalleryItem] = field(default_factory=list)\n",
"\n",
" @classmethod\n",
" def from_yaml(cls, yaml_file: str, strict: bool = False) -> \"Config\":\n",
" \"\"\"\n",
" Load configuration from a YAML file.\n",
"\n",
" Args:\n",
" yaml_file (str): Path to the YAML file.\n",
" strict (bool):\n",
" If True, raises an error if a key in the YAML file does not exist in the Config class.\n",
" If False (default), adds all keys as attributes\n",
"\n",
" Returns:\n",
" Config: Instance of this class\n",
" \"\"\"\n",
"\n",
" with open(yaml_file, \"r\") as f:\n",
" new_config: dict = yaml.safe_load(f)\n",
"\n",
" new_config[\"sources\"] = [\n",
" GalleryItem(name=src[\"name\"], path=Path(src[\"path\"]))\n",
" for src in new_config.get(\"sources\", False)\n",
" ]\n",
"\n",
" instance = cls(**{\n",
" k: v\n",
" for k, v in new_config.items()\n",
" if hasattr(cls, k) or not strict\n",
" })\n",
"\n",
" for key in new_config.keys():\n",
" if strict and not hasattr(instance, key):\n",
" raise KeyError(f\"Key '{key}' not found in Config class\")\n",
"\n",
" return instance\n",
"\n",
" def to_yaml(self, yaml_file: str) -> None:\n",
" \"\"\"\n",
" Save the current configuration to a YAML file.\n",
"\n",
" Args:\n",
" yaml_file (str): Path to the YAML file.\n",
" \"\"\"\n",
" with open(yaml_file, \"w\") as f:\n",
" yaml.dump(asdict(self), f, default_flow_style=False)\n",
"\n",
"\n",
"instance = Config(**{\n",
" k: v\n",
" for k, v in new_config.items()\n",
" #if hasattr(Config, k)\n",
"})\n",
"instance"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "0b516ae9",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'test_1_plot',\n",
" 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/plot__proc_3_5606769559__cat_incl__var_jet1_pt.pdf'},\n",
" {'name': 'test_2_dir',\n",
" 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/'}]"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"new_config[\"sources\"]"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "b047f0c5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'test_1_plot',\n",
" 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/plot__proc_3_5606769559__cat_incl__var_jet1_pt.pdf'},\n",
" {'name': 'test_2_dir',\n",
" 'path': 'data/cf_store/test_analysis/cf.PlotVariables1D/run2_2016_nano_v9/calib__test/sel__test/prod__test/weight__test/nominal/datasets_ttbar_dl/dev1/'}]"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"instance.sources"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}