Kylian Schmidt
276 lines
6.9 KiB
Markdown
276 lines
6.9 KiB
Markdown
# Automated Coverage Testing Documentation
|
|
|
|
## Overview
|
|
|
|
This repository now includes automated code coverage testing using the `coverage.py` package. Coverage testing helps ensure that your tests adequately exercise your codebase and identifies untested code paths.
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Container-based Coverage (Recommended)
|
|
```bash
|
|
# Build and test with coverage in container
|
|
./tests/test_container.sh
|
|
|
|
# Or run coverage directly in container
|
|
apptainer exec gallery-generator.sif python3 /src/tests/run_coverage.py
|
|
```
|
|
|
|
### Local Coverage Testing
|
|
```bash
|
|
# Run coverage tests locally
|
|
./tests/run_coverage_local.sh
|
|
|
|
# Or manually
|
|
pip install coverage
|
|
coverage run -m unittest tests.test_container
|
|
coverage report
|
|
coverage html
|
|
```
|
|
|
|
## 📁 Coverage Files
|
|
|
|
### Core Coverage Files
|
|
- **`.coveragerc`** - Coverage configuration file
|
|
- **`tests/run_coverage.py`** - Automated coverage script for containers
|
|
- **`tests/run_coverage_local.sh`** - Local coverage testing script
|
|
|
|
### Generated Reports
|
|
- **`coverage.xml`** - XML format for CI/CD integration
|
|
- **`coverage_html_report/`** - Interactive HTML reports
|
|
- **`.coverage`** - Coverage data file
|
|
|
|
## 🔧 Configuration
|
|
|
|
### Coverage Settings (`.coveragerc`)
|
|
```ini
|
|
[run]
|
|
source = .
|
|
omit =
|
|
tests/* # Exclude test files
|
|
__pycache__/* # Exclude cache
|
|
assets/* # Exclude static assets
|
|
docs/* # Exclude documentation
|
|
templates/* # Exclude templates
|
|
|
|
[report]
|
|
precision = 2 # 2 decimal places
|
|
show_missing = True # Show missing line numbers
|
|
skip_covered = False # Show all files
|
|
|
|
[html]
|
|
directory = coverage_html_report
|
|
title = Gallery Generator Coverage Report
|
|
```
|
|
|
|
### Singularity Container Integration
|
|
The coverage package is automatically installed in the container:
|
|
```bash
|
|
pip install --no-cache-dir jinja2 pyyaml coverage
|
|
```
|
|
|
|
## 📊 Coverage Reports
|
|
|
|
### Console Report
|
|
Shows coverage percentage and missing lines:
|
|
```
|
|
Name Stmts Miss Cover Missing
|
|
-----------------------------------------------------
|
|
generate_gallery.py 190 45 76.32% 156-167, 234-245
|
|
orchestration/config.py 45 8 82.22% 78-82
|
|
orchestration/logger.py 67 12 82.09% 45-48, 89-94
|
|
-----------------------------------------------------
|
|
TOTAL 302 65 78.48%
|
|
```
|
|
|
|
### HTML Report
|
|
Interactive report with:
|
|
- Line-by-line coverage highlighting
|
|
- Branch coverage details
|
|
- Sortable file listings
|
|
- Coverage trends
|
|
|
|
### XML Report
|
|
Machine-readable format for CI/CD:
|
|
- GitLab CI coverage visualization
|
|
- External tool integration
|
|
- Coverage badges
|
|
|
|
## 🎯 Coverage Targets
|
|
|
|
### Current Thresholds
|
|
- **Minimum Target**: 80% overall coverage
|
|
- **Warning Level**: Below 70% coverage
|
|
- **Exclusions**: Test files, static assets, documentation
|
|
|
|
### Best Practices
|
|
- **Focus on Core Logic**: Prioritize business logic coverage
|
|
- **Test Edge Cases**: Include error handling and boundary conditions
|
|
- **Regular Monitoring**: Run coverage with every commit
|
|
- **Incremental Improvement**: Gradually increase coverage over time
|
|
|
|
## 🔄 CI/CD Integration
|
|
|
|
### GitLab CI Pipeline
|
|
The coverage testing is integrated into the GitLab CI pipeline:
|
|
|
|
```yaml
|
|
test:coverage:
|
|
stage: test
|
|
script:
|
|
- apptainer exec $CONTAINER_IMAGE python3 /src/tests/run_coverage.py
|
|
coverage: '/TOTAL.+?(\d+\.\d+)%/'
|
|
artifacts:
|
|
reports:
|
|
coverage_report:
|
|
coverage_format: cobertura
|
|
path: coverage.xml
|
|
```
|
|
|
|
### Features
|
|
- **Automatic Reports**: Coverage reports in merge requests
|
|
- **Badge Integration**: Coverage badges in README
|
|
- **Trend Tracking**: Historical coverage data
|
|
- **Failure Thresholds**: Fail builds below minimum coverage
|
|
|
|
## 🛠️ Advanced Usage
|
|
|
|
### Custom Coverage Runs
|
|
```bash
|
|
# Test specific modules
|
|
coverage run --source=orchestration -m unittest tests.test_metadata
|
|
|
|
# Include/exclude patterns
|
|
coverage run --omit="*/tests/*" -m unittest discover
|
|
|
|
# Branch coverage (more detailed)
|
|
coverage run --branch -m unittest tests.test_container
|
|
```
|
|
|
|
### Coverage Analysis
|
|
```bash
|
|
# Show missing lines
|
|
coverage report --show-missing
|
|
|
|
# Generate detailed HTML
|
|
coverage html --show-contexts
|
|
|
|
# Export data
|
|
coverage json
|
|
coverage xml
|
|
```
|
|
|
|
### Integration with IDEs
|
|
- **VS Code**: Coverage Gutters extension
|
|
- **PyCharm**: Built-in coverage runner
|
|
- **Vim**: Coverage highlighting plugins
|
|
|
|
## 📈 Coverage Metrics
|
|
|
|
### What Coverage Measures
|
|
- **Statement Coverage**: Lines of code executed
|
|
- **Branch Coverage**: Decision paths taken
|
|
- **Function Coverage**: Functions called
|
|
- **Class Coverage**: Classes instantiated
|
|
|
|
### What Coverage Doesn't Measure
|
|
- **Code Quality**: Coverage ≠ good tests
|
|
- **Logic Correctness**: 100% coverage ≠ bug-free
|
|
- **Performance**: Execution speed not measured
|
|
- **Security**: Vulnerabilities not detected
|
|
|
|
## 🧪 Testing Strategy
|
|
|
|
### Container Test Suite Coverage
|
|
Current test files and their focus:
|
|
|
|
#### `tests/test_container.py`
|
|
- **Environment validation** - Container setup
|
|
- **Utility functions** - Helper functions
|
|
- **Metadata system** - YAML processing
|
|
- **PDF processing** - ImageMagick integration
|
|
- **Gallery generation** - End-to-end workflow
|
|
|
|
#### `tests/test_build_container.py`
|
|
- **Container building** - Singularity build process
|
|
- **Dependency validation** - Package installation
|
|
- **Application functionality** - Script execution
|
|
|
|
### Coverage Gaps Analysis
|
|
Use `tests/test_coverage.py` to analyze:
|
|
- Missing function coverage
|
|
- Untested code paths
|
|
- Critical functionality gaps
|
|
- Integration test needs
|
|
|
|
## 🚨 Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### No Coverage Data
|
|
```bash
|
|
# Ensure coverage is running tests
|
|
coverage run --debug=trace -m unittest tests.test_container
|
|
```
|
|
|
|
#### Import Errors
|
|
```bash
|
|
# Check PYTHONPATH
|
|
export PYTHONPATH=/src:$PYTHONPATH
|
|
```
|
|
|
|
#### Permission Issues
|
|
```bash
|
|
# Container write permissions
|
|
apptainer exec --writable-tmpfs container.sif python3 tests/run_coverage.py
|
|
```
|
|
|
|
### Debug Commands
|
|
```bash
|
|
# Check coverage configuration
|
|
coverage debug config
|
|
|
|
# Verify data collection
|
|
coverage debug data
|
|
|
|
# Test discovery
|
|
coverage debug sys
|
|
```
|
|
|
|
## 📚 References
|
|
|
|
- **Coverage.py Documentation**: https://coverage.readthedocs.io/
|
|
- **GitLab CI Coverage**: https://docs.gitlab.com/ee/ci/testing/code_coverage.html
|
|
- **Testing Best Practices**: Python Testing 101
|
|
- **Container Testing**: Singularity/Apptainer Documentation
|
|
|
|
## 🔄 Maintenance
|
|
|
|
### Regular Tasks
|
|
- **Weekly**: Review coverage reports
|
|
- **Monthly**: Update coverage targets
|
|
- **Release**: Ensure minimum coverage met
|
|
- **Quarterly**: Review exclusion patterns
|
|
|
|
### Cleanup
|
|
```bash
|
|
# Remove coverage files
|
|
./tests/cleanup.sh
|
|
|
|
# Manual cleanup
|
|
rm -f .coverage coverage.xml
|
|
rm -rf coverage_html_report/
|
|
```
|
|
|
|
### Updates
|
|
```bash
|
|
# Update coverage package
|
|
pip install --upgrade coverage
|
|
|
|
# Update container
|
|
apptainer build --force container.sif Singularity.def
|
|
```
|
|
|
|
---
|
|
|
|
*This automated coverage system provides comprehensive testing insights while maintaining the containerized, dependency-free approach of the gallery generator project.*
|