[WIP] Add tests and coverage
This commit is contained in:
Kylian Schmidt
@@ -0,0 +1,275 @@
|
||||
# Automated Coverage Testing Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
This repository now includes automated code coverage testing using the `coverage.py` package. Coverage testing helps ensure that your tests adequately exercise your codebase and identifies untested code paths.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Container-based Coverage (Recommended)
|
||||
```bash
|
||||
# Build and test with coverage in container
|
||||
./tests/test_container.sh
|
||||
|
||||
# Or run coverage directly in container
|
||||
apptainer exec gallery-generator.sif python3 /src/tests/run_coverage.py
|
||||
```
|
||||
|
||||
### Local Coverage Testing
|
||||
```bash
|
||||
# Run coverage tests locally
|
||||
./tests/run_coverage_local.sh
|
||||
|
||||
# Or manually
|
||||
pip install coverage
|
||||
coverage run -m unittest tests.test_container
|
||||
coverage report
|
||||
coverage html
|
||||
```
|
||||
|
||||
## 📁 Coverage Files
|
||||
|
||||
### Core Coverage Files
|
||||
- **`.coveragerc`** - Coverage configuration file
|
||||
- **`tests/run_coverage.py`** - Automated coverage script for containers
|
||||
- **`tests/run_coverage_local.sh`** - Local coverage testing script
|
||||
|
||||
### Generated Reports
|
||||
- **`coverage.xml`** - XML format for CI/CD integration
|
||||
- **`coverage_html_report/`** - Interactive HTML reports
|
||||
- **`.coverage`** - Coverage data file
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Coverage Settings (`.coveragerc`)
|
||||
```ini
|
||||
[run]
|
||||
source = .
|
||||
omit =
|
||||
tests/* # Exclude test files
|
||||
__pycache__/* # Exclude cache
|
||||
assets/* # Exclude static assets
|
||||
docs/* # Exclude documentation
|
||||
templates/* # Exclude templates
|
||||
|
||||
[report]
|
||||
precision = 2 # 2 decimal places
|
||||
show_missing = True # Show missing line numbers
|
||||
skip_covered = False # Show all files
|
||||
|
||||
[html]
|
||||
directory = coverage_html_report
|
||||
title = Gallery Generator Coverage Report
|
||||
```
|
||||
|
||||
### Singularity Container Integration
|
||||
The coverage package is automatically installed in the container:
|
||||
```bash
|
||||
pip install --no-cache-dir jinja2 pyyaml coverage
|
||||
```
|
||||
|
||||
## 📊 Coverage Reports
|
||||
|
||||
### Console Report
|
||||
Shows coverage percentage and missing lines:
|
||||
```
|
||||
Name Stmts Miss Cover Missing
|
||||
-----------------------------------------------------
|
||||
generate_gallery.py 190 45 76.32% 156-167, 234-245
|
||||
orchestration/config.py 45 8 82.22% 78-82
|
||||
orchestration/logger.py 67 12 82.09% 45-48, 89-94
|
||||
-----------------------------------------------------
|
||||
TOTAL 302 65 78.48%
|
||||
```
|
||||
|
||||
### HTML Report
|
||||
Interactive report with:
|
||||
- Line-by-line coverage highlighting
|
||||
- Branch coverage details
|
||||
- Sortable file listings
|
||||
- Coverage trends
|
||||
|
||||
### XML Report
|
||||
Machine-readable format for CI/CD:
|
||||
- GitLab CI coverage visualization
|
||||
- External tool integration
|
||||
- Coverage badges
|
||||
|
||||
## 🎯 Coverage Targets
|
||||
|
||||
### Current Thresholds
|
||||
- **Minimum Target**: 80% overall coverage
|
||||
- **Warning Level**: Below 70% coverage
|
||||
- **Exclusions**: Test files, static assets, documentation
|
||||
|
||||
### Best Practices
|
||||
- **Focus on Core Logic**: Prioritize business logic coverage
|
||||
- **Test Edge Cases**: Include error handling and boundary conditions
|
||||
- **Regular Monitoring**: Run coverage with every commit
|
||||
- **Incremental Improvement**: Gradually increase coverage over time
|
||||
|
||||
## 🔄 CI/CD Integration
|
||||
|
||||
### GitLab CI Pipeline
|
||||
The coverage testing is integrated into the GitLab CI pipeline:
|
||||
|
||||
```yaml
|
||||
test:coverage:
|
||||
stage: test
|
||||
script:
|
||||
- apptainer exec $CONTAINER_IMAGE python3 /src/tests/run_coverage.py
|
||||
coverage: '/TOTAL.+?(\d+\.\d+)%/'
|
||||
artifacts:
|
||||
reports:
|
||||
coverage_report:
|
||||
coverage_format: cobertura
|
||||
path: coverage.xml
|
||||
```
|
||||
|
||||
### Features
|
||||
- **Automatic Reports**: Coverage reports in merge requests
|
||||
- **Badge Integration**: Coverage badges in README
|
||||
- **Trend Tracking**: Historical coverage data
|
||||
- **Failure Thresholds**: Fail builds below minimum coverage
|
||||
|
||||
## 🛠️ Advanced Usage
|
||||
|
||||
### Custom Coverage Runs
|
||||
```bash
|
||||
# Test specific modules
|
||||
coverage run --source=orchestration -m unittest tests.test_metadata
|
||||
|
||||
# Include/exclude patterns
|
||||
coverage run --omit="*/tests/*" -m unittest discover
|
||||
|
||||
# Branch coverage (more detailed)
|
||||
coverage run --branch -m unittest tests.test_container
|
||||
```
|
||||
|
||||
### Coverage Analysis
|
||||
```bash
|
||||
# Show missing lines
|
||||
coverage report --show-missing
|
||||
|
||||
# Generate detailed HTML
|
||||
coverage html --show-contexts
|
||||
|
||||
# Export data
|
||||
coverage json
|
||||
coverage xml
|
||||
```
|
||||
|
||||
### Integration with IDEs
|
||||
- **VS Code**: Coverage Gutters extension
|
||||
- **PyCharm**: Built-in coverage runner
|
||||
- **Vim**: Coverage highlighting plugins
|
||||
|
||||
## 📈 Coverage Metrics
|
||||
|
||||
### What Coverage Measures
|
||||
- **Statement Coverage**: Lines of code executed
|
||||
- **Branch Coverage**: Decision paths taken
|
||||
- **Function Coverage**: Functions called
|
||||
- **Class Coverage**: Classes instantiated
|
||||
|
||||
### What Coverage Doesn't Measure
|
||||
- **Code Quality**: Coverage ≠ good tests
|
||||
- **Logic Correctness**: 100% coverage ≠ bug-free
|
||||
- **Performance**: Execution speed not measured
|
||||
- **Security**: Vulnerabilities not detected
|
||||
|
||||
## 🧪 Testing Strategy
|
||||
|
||||
### Container Test Suite Coverage
|
||||
Current test files and their focus:
|
||||
|
||||
#### `tests/test_container.py`
|
||||
- **Environment validation** - Container setup
|
||||
- **Utility functions** - Helper functions
|
||||
- **Metadata system** - YAML processing
|
||||
- **PDF processing** - ImageMagick integration
|
||||
- **Gallery generation** - End-to-end workflow
|
||||
|
||||
#### `tests/test_build_container.py`
|
||||
- **Container building** - Singularity build process
|
||||
- **Dependency validation** - Package installation
|
||||
- **Application functionality** - Script execution
|
||||
|
||||
### Coverage Gaps Analysis
|
||||
Use `tests/test_coverage.py` to analyze:
|
||||
- Missing function coverage
|
||||
- Untested code paths
|
||||
- Critical functionality gaps
|
||||
- Integration test needs
|
||||
|
||||
## 🚨 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### No Coverage Data
|
||||
```bash
|
||||
# Ensure coverage is running tests
|
||||
coverage run --debug=trace -m unittest tests.test_container
|
||||
```
|
||||
|
||||
#### Import Errors
|
||||
```bash
|
||||
# Check PYTHONPATH
|
||||
export PYTHONPATH=/src:$PYTHONPATH
|
||||
```
|
||||
|
||||
#### Permission Issues
|
||||
```bash
|
||||
# Container write permissions
|
||||
apptainer exec --writable-tmpfs container.sif python3 tests/run_coverage.py
|
||||
```
|
||||
|
||||
### Debug Commands
|
||||
```bash
|
||||
# Check coverage configuration
|
||||
coverage debug config
|
||||
|
||||
# Verify data collection
|
||||
coverage debug data
|
||||
|
||||
# Test discovery
|
||||
coverage debug sys
|
||||
```
|
||||
|
||||
## 📚 References
|
||||
|
||||
- **Coverage.py Documentation**: https://coverage.readthedocs.io/
|
||||
- **GitLab CI Coverage**: https://docs.gitlab.com/ee/ci/testing/code_coverage.html
|
||||
- **Testing Best Practices**: Python Testing 101
|
||||
- **Container Testing**: Singularity/Apptainer Documentation
|
||||
|
||||
## 🔄 Maintenance
|
||||
|
||||
### Regular Tasks
|
||||
- **Weekly**: Review coverage reports
|
||||
- **Monthly**: Update coverage targets
|
||||
- **Release**: Ensure minimum coverage met
|
||||
- **Quarterly**: Review exclusion patterns
|
||||
|
||||
### Cleanup
|
||||
```bash
|
||||
# Remove coverage files
|
||||
./tests/cleanup.sh
|
||||
|
||||
# Manual cleanup
|
||||
rm -f .coverage coverage.xml
|
||||
rm -rf coverage_html_report/
|
||||
```
|
||||
|
||||
### Updates
|
||||
```bash
|
||||
# Update coverage package
|
||||
pip install --upgrade coverage
|
||||
|
||||
# Update container
|
||||
apptainer build --force container.sif Singularity.def
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*This automated coverage system provides comprehensive testing insights while maintaining the containerized, dependency-free approach of the gallery generator project.*
|
||||
Reference in New Issue
Block a user