fix: configure CI database connection properly

- Add dotenv loading to Django settings - Update CI workflow to use correct environment variables - Set POSTGRES_* variables instead of DATABASE_URL - Add environment variables to all Django management commands - Fixes CI test failures due to database connection issues
2025-09-06 18:47:23 +02:00
parent dcc91b9f49
commit 35ba089a84
64 changed files with 7040 additions and 1419 deletions
--- a/docs/legacy-cleanup-complete.md
+++ b/docs/legacy-cleanup-complete.md
@@ -0,0 +1,130 @@
+# Legacy Verpachtung Cleanup - Complete Removal
+
+**Date:** August 31, 2025  
+**Status:** ✅ COMPLETED - Legacy system completely removed
+
+## Overview
+
+This document summarizes the complete removal of the legacy Verpachtung system from the Django Stiftung application. The legacy system has been fully replaced by the new LandVerpachtung model and all references have been cleaned up.
+
+## Actions Completed
+
+### 1. Data Migration ✅
+- ✅ Deleted all 3 legacy Verpachtung records (user confirmed no valid data)
+- ✅ New LandVerpachtung system contains 8 active records
+- ✅ LandAbrechnung system validated with 55 accounting records
+
+### 2. Model Cleanup ✅
+- ✅ Removed `Verpachtung` class from `stiftung/models.py`
+- ✅ Created and applied Django migration `0023_remove_legacy_verpachtung.py`
+- ✅ Database table `stiftung_verpachtung` successfully dropped
+- ✅ Removed imports of `Verpachtung` from models
+
+### 3. Views Cleanup ✅
+- ✅ Removed all legacy verpachtung views:
+  - `verpachtung_list`
+  - `verpachtung_detail` 
+  - `verpachtung_create`
+  - `verpachtung_update`
+  - `verpachtung_delete`
+  - `verpachtung_export`
+- ✅ Removed `Verpachtung` and `VerpachtungForm` imports from views
+- ✅ Cleaned up view imports in `stiftung/views.py`
+
+### 4. URL Cleanup ✅
+- ✅ Removed all legacy verpachtung URL patterns from `stiftung/urls.py`:
+  - `verpachtungen/` (list)
+  - `verpachtungen/<uuid:pk>/` (detail)
+  - `verpachtungen/neu/` (create)
+  - `verpachtungen/<uuid:pk>/bearbeiten/` (update)
+  - `verpachtungen/<uuid:pk>/loeschen/` (delete)
+  - `verpachtungen/<uuid:pk>/export/` (export)
+
+### 5. Forms Cleanup ✅
+- ✅ Removed `VerpachtungForm` class from `stiftung/forms.py`
+- ✅ Removed `Verpachtung` import from forms
+
+### 6. Admin Cleanup ✅
+- ✅ Removed `VerpachtungAdmin` class from `stiftung/admin.py`
+- ✅ Removed `@admin.register(Verpachtung)` decorator
+- ✅ Removed `Verpachtung` import from admin
+
+### 7. Template Cleanup ✅
+- ✅ Deleted all legacy verpachtung templates:
+  - `verpachtung_list.html`
+  - `verpachtung_detail.html`
+  - `verpachtung_form.html`
+  - `verpachtung_confirm_delete.html`
+
+### 8. Command Cleanup ✅
+- ✅ Removed `convert_legacy_verpachtungen.py` management command (no longer needed)
+
+### 9. System Validation ✅
+- ✅ Application starts successfully after cleanup
+- ✅ New LandVerpachtung model functions correctly (8 records)
+- ✅ LandAbrechnung accounting system intact (55 records)
+- ✅ No legacy references remain in codebase
+
+## Current State
+
+### Active Systems
+- **LandVerpachtung**: Modern lease management with 8 active leases
+- **LandAbrechnung**: Accounting system with 55 records
+- **Navigation**: Clean UI with only new lease workflows
+- **Database**: Legacy table `stiftung_verpachtung` removed
+
+### Removed Systems
+- **Legacy Verpachtung Model**: Completely removed
+- **Legacy URLs**: All verpachtung URLs removed
+- **Legacy Views**: All verpachtung views removed  
+- **Legacy Templates**: All verpachtung templates removed
+- **Legacy Forms**: VerpachtungForm removed
+- **Legacy Admin**: VerpachtungAdmin removed
+- **Legacy Commands**: Conversion command removed
+
+## Benefits Achieved
+
+1. **Simplified Codebase**: Removed ~500 lines of legacy code
+2. **Clean User Interface**: No confusing dual lease systems
+3. **Better Architecture**: Single lease model with proper accounting integration
+4. **Reduced Maintenance**: No legacy code to maintain
+5. **Database Cleanup**: Removed unused table and relationships
+
+## Next Steps (Recommended)
+
+1. **Monitor System**: Verify new lease workflows work as expected
+2. **User Training**: Update any documentation to reflect single lease system
+3. **Backup Validation**: Ensure backup system works with cleaned schema
+
+## Files Modified
+
+### Removed Files
+- `templates/stiftung/verpachtung_*.html` (4 files)
+- `stiftung/management/commands/convert_legacy_verpachtungen.py`
+
+### Modified Files
+- `stiftung/models.py` - Removed Verpachtung class
+- `stiftung/views.py` - Removed verpachtung views and imports
+- `stiftung/urls.py` - Removed verpachtung URL patterns
+- `stiftung/forms.py` - Removed VerpachtungForm
+- `stiftung/admin.py` - Removed VerpachtungAdmin
+- `stiftung/migrations/0023_remove_legacy_verpachtung.py` - New migration
+
+### Unchanged Files
+- All LandVerpachtung related files remain intact
+- All LandAbrechnung accounting files remain intact  
+- Navigation templates already cleaned up previously
+
+## Technical Notes
+
+- Django migration `0023_remove_legacy_verpachtung.py` successfully applied
+- No foreign key constraints violated (references already cleaned up)
+- Application restart successful
+- All imports resolved correctly
+- No template resolution errors
+
+---
+
+**Migration Status:** Complete  
+**System Status:** Fully operational with new lease system only  
+**Legacy Status:** Completely removed
--- a/docs/migration-plan-verpachtung.md
+++ b/docs/migration-plan-verpachtung.md
@@ -0,0 +1,79 @@
+# Verpachtung Legacy → LandVerpachtung Migration Plan
+
+## Status (2025-08-31) - ✅ COMPLETED
+- **Phase 1 Complete**: Legacy create/edit blocked via feature flags
+- **Phase 2 Complete**: Conversion command created and executed (3 legacy → 8 new)  
+- **Phase 3 Complete**: Legacy system completely removed from codebase
+
+## Final State  
+- Legacy count: 0 Verpachtung records (deleted - no valid data)
+- New count: 8 LandVerpachtung records (active)
+- Legacy model: Completely removed from system  
+- Financial sync: Both models working (11,120.60€ total rent verified)
+
+## Feature Flags Added
+```python
+# In Django settings or environment
+DISABLE_LEGACY_VERPACHTUNG_CREATE = True    # Blocks verpachtung_create
+DISABLE_LEGACY_VERPACHTUNG_EDIT = True      # Blocks verpachtung_update
+```
+
+## Phase 3: Navigation and View Cleanup
+To complete the migration, we need to:
+
+### 3.1 Template Navigation Updates
+- Find and comment/remove "Neue Verpachtung" links in navigation/menus
+- Keep legacy detail/list views for data viewing but remove edit buttons
+- Update dashboard/summary pages to show new lease counts
+
+### 3.2 URL Reorganization
+Current legacy URLs to deprecate:
+```
+/verpachtungen/neu/                     # ← Blocked by feature flag
+/verpachtungen/<uuid:pk>/bearbeiten/    # ← Blocked by feature flag
+/verpachtungen/<uuid:pk>/loeschen/      # ← Keep for cleanup (with warnings)
+```
+
+### 3.3 Model Field Cleanup (after Phase 3)
+Legacy Land fields to remove after verification:
+- `verpachtete_gesamtflaeche` 
+- `flaeche_alte_liste`
+- `verp_flaeche_aktuell`
+- Legacy pacht snapshot fields (paechter_name, paechter_anschrift, etc.)
+
+### 3.4 Final Model Removal (Phase 4)
+After all views/templates updated:
+- Remove `Verpachtung` model class
+- Remove legacy views/forms/templates
+- Create migration to drop database table
+
+## Conversion Command Usage
+```bash
+# Dry run (safe)
+python manage.py convert_legacy_verpachtungen
+
+# Convert with limit
+python manage.py convert_legacy_verpachtungen --limit 10
+
+# Commit changes
+python manage.py convert_legacy_verpachtungen --commit
+```
+
+## Verification Steps
+1. Check counts: `Legacy: 3, New: 8` ✓
+2. Financial sync: `11,120.60€ total` ✓
+3. UI testing: Access new lease flows via Land detail pages
+4. Data spot-check: Compare legacy vs new lease details for accuracy
+
+## Next Actions
+- [ ] Update navigation templates to hide legacy create/edit links
+- [ ] Add deprecation notices in legacy detail templates
+- [ ] Test new lease workflows end-to-end
+- [ ] Prepare field removal migrations
+- [ ] Document new user workflows
+
+## Rollback Plan
+If issues arise:
+1. Set feature flags to `False` to re-enable legacy flows
+2. Delete problematic new LandVerpachtung records if needed
+3. Revert to legacy workflow until issues resolved
--- a/docs/schema-and-cleanup-notes.md
+++ b/docs/schema-and-cleanup-notes.md
@@ -0,0 +1,134 @@
+# Stiftung Starter: Schema, Usage, and Cleanup Plan (2025-08-31)
+
+This document consolidates the current domain model, relationships, active usage across the codebase, and targeted cleanup/migration proposals.
+
+## Summary
+- Server and date-handling fixes completed earlier; stack is healthy.
+- New vs. legacy lease models both in use; annual accounting (LandAbrechnung) is the hub.
+- Orphans: a few likely candidates (LandAbrechnung.zahlungen, file fields; ApplicationPermission stub). Several legacy fields still referenced; plan deprecation after migration.
+
+## Entities and Relationships
+
+- CSVImport
+  - Tracks CSV imports; fields: import_type, filename, file_size, status, totals, error_log, created_by, started_at, completed_at
+
+- Paechter (Tenant)
+  - Fields: vorname, nachname, geburtsdatum, email, telefon, iban, strasse, plz, ort, personentyp, pachtnummer, pachtbeginn_erste, pachtende_letzte, pachtzins_aktuell, landwirtschaftliche_ausbildung, berufserfahrung_jahre, spezialisierung, notizen, aktiv
+  - Relations: Verpachtung (legacy) M:1; LandVerpachtung (new) M:1; reverse from Land.aktueller_paechter
+  - Helpers: get_full_name, get_aktive_verpachtungen, get_gesamt_pachtflaeche, get_gesamt_pachtzins
+
+- Destinataer (Beneficiary)
+  - Fields: familienzweig, vorname, nachname, geburtsdatum, email, telefon, iban, strasse, plz, ort, berufsgruppe, ausbildungsstand, institution, projekt_beschreibung, jaehrliches_einkommen, finanzielle_notlage, notizen, aktiv, ist_abkoemmling, haushaltsgroesse, monatliche_bezuege, vermoegen, unterstuetzung_bestaetigt, standard_konto(FK StiftungsKonto), vierteljaehrlicher_betrag, studiennachweis_erforderlich, letzter_studiennachweis
+  - Relations: Foerderung 1:M (via destinataer), DestinataerUnterstuetzung 1:M, DestinataerNotiz 1:M
+  - Helpers: get_full_name, totals, erfuellt_voraussetzungen, naechste_studiennachweis_termine
+
+- Person (Legacy)
+  - Legacy stand-in for beneficiaries; still referenced in views/templates; used by Foerderung.person
+
+- Land (Parcel)
+  - Identity: lfd_nr(unique), ew_nummer, grundbuchblatt, amtsgericht, gemeinde, gemarkung, flur, flurstueck, adresse
+  - Area: groesse_qm, gruenland_qm, acker_qm, wald_qm, sonstiges_qm
+  - Legacy pacht: verpachtete_gesamtflaeche, flaeche_alte_liste, verp_flaeche_aktuell
+  - Current pacht snapshot: aktueller_paechter(FK), paechter_name, paechter_anschrift, pachtbeginn, pachtende, verlaengerung_klausel, zahlungsweise, pachtzins_pro_ha, pachtzins_pauschal
+  - USt/Tax: ust_option, ust_satz, grundsteuer_umlage, versicherungen_umlage, verbandsbeitraege_umlage, jagdpacht_anteil_umlage, anteil_grundsteuer, anteil_lwk
+  - Status/meta: aktiv, notizen, erstellt_am, aktualisiert_am
+  - Relations: Verpachtung (legacy) 1:M; LandVerpachtung (new) 1:M; LandAbrechnung 1:M
+  - Helpers: area conversions; pachtfläche calculators
+
+- LandVerpachtung (New Lease)
+  - land(FK), paechter(FK), vertragsnummer(unique), pachtbeginn, pachtende?, verlaengerung_klausel
+  - verpachtete_flaeche, pachtzins_pauschal, pachtzins_pro_ha?, zahlungsweise
+  - USt/umlagen flags; status, bemerkungen; timestamps
+  - Behavior: save/delete sync LandAbrechnung; rent proration per year; umlage calc currently 0
+
+- LandAbrechnung (Annual Accounting)
+  - land(FK), abrechnungsjahr(unique per land)
+  - Einnahmen: pacht_vereinnahmt, umlagen_vereinnahmt, sonstige_einnahmen, zahlungen(JSON?)
+  - Ausgaben buckets; vorsteuer_aus_umlagen; offene_posten; bemerkungen
+  - File fields for uploads; timestamps; derived totals incl. ust_pacht_betrag
+
+- Verpachtung (Legacy Lease)
+  - land(FK), paechter(FK), vertragsnummer(unique), pachtbeginn, pachtende, verlaengerung?
+  - pachtzins_pro_qm, pachtzins_jaehrlich, verpachtete_flaeche; status; verwendungsnachweis; bemerkungen; timestamps
+  - Behavior: save/delete sync LandAbrechnung; rent proration with verlängerung
+
+- DokumentLink (Paperless linking)
+  - paperless_document_id, kontext, titel, beschreibung
+  - Flexible UUID links: verpachtung_id, land_verpachtung_id, land_id, paechter_id, destinataer_id, foerderung_id, rentmeister_id
+
+- Foerderung (Grant)
+  - person(FK legacy, nullable), destinataer(FK, nullable), jahr, betrag, kategorie, status, verwendungsnachweis(FK), bemerkungen, antragsdatum, entscheidungsdatum
+
+- DestinataerUnterstuetzung (Support)
+  - destinataer(FK), konto(FK StiftungsKonto), betrag, faellig_am, status, beschreibung, timestamps; indexes
+
+- DestinataerNotiz
+  - destinataer(FK), titel, text, datei, erstellt_von(FK auth.User), erstellt_am
+
+- Rentmeister
+  - person/address/bank fields; seit_datum, bis_datum?, aktiv, monatliche_verguetung, km_pauschale; used by Verwaltungskosten
+
+- StiftungsKonto
+  - kontoname, bank_name, iban, bic, konto_typ, saldo, saldo_datum, zinssatz, laufzeit_bis, aktiv, notizen, timestamps
+
+- BankTransaction
+  - konto(FK), datum, valuta?, betrag, waehrung, verwendungszweck, empfaenger_zahlungspflichtiger, iban_gegenpartei, bic_gegenpartei, referenz, transaction_type, status, kommentare, verwaltungskosten(FK?), import_datei, importiert_am, saldo_nach_buchung
+  - unique_together: (konto, datum, betrag, referenz)
+
+- Verwaltungskosten
+  - bezeichnung, kategorie, betrag, datum, lieferant_firma, rechnungsnummer, status, rentmeister(FK), zahlungskonto(FK), quellkonto(FK), konto(FK legacy), km_anzahl, km_satz, von_ort, nach_ort, zweck, beschreibung, notizen, timestamps
+
+- ApplicationPermission (managed=False)
+  - Declares custom permission codenames; no table created
+
+- AuditLog
+  - user(FK), username, timestamp, action, entity_type, entity_id, entity_name, description, changes(JSON), ip_address, user_agent, session_key; indexes
+
+- BackupJob
+  - backup_type, status, created_by(FK), created_at/started_at/completed_at, backup_filename, backup_size, error_message, database_size, files_count; helpers
+
+## Usage observations (workspace-wide scans)
+- Legacy vs. new lease models are both heavily referenced across views/templates/forms.
+- LandAbrechnung core fields are used in forms and detail pages; pacht_vereinnahmt updated by save/delete hooks and a sync command.
+- DokumentLink UUID references are widely used for cross-entity navigation.
+- Destinataer support fields power quarterly payout scheduling.
+- Person (legacy) still appears in Foerderung flows and person pages.
+
+## Likely orphans / cleanup candidates
+- LandAbrechnung.zahlungen (JSON): no UI/logic usage detected.
+- LandAbrechnung file fields (pachtvertrag_datei, grundsteuer_bescheid_datei, versicherungsnachweis_datei): unused in UI; Paperless appears primary doc source.
+- ApplicationPermission: only a managed=False stub with permissions; keep if used for seeding, otherwise move to data migration and remove the class.
+- LandVerpachtung._calculate_umlage_for_year: returns 0; umlage flags are gathered but not used to compute revenue.
+- Legacy Land fields: still used (imports, displays). Plan deprecation after migration to new lease model.
+- Foerderung.person vs destinataer: both used; plan single-source migration and drop legacy.
+
+## Recommendations and plan
+1) Consolidate to LandVerpachtung
+  - Stop creating new legacy Verpachtung in UI; a feature flag now blocks legacy create (`DISABLE_LEGACY_VERPACHTUNG_CREATE`, default True) and guides users to the new flow.
+  - A conversion command `convert_legacy_verpachtungen` is scaffolded (dry-run by default; use `--commit` to persist) to create `LandVerpachtung` rows from `Verpachtung`.
+   - After migration and verification: remove legacy model, views, and Land’s legacy pacht snapshot fields.
+
+2) DRY date handling
+  - A shared helper exists: `stiftung/utils/date_utils.py` with `ensure_date` and `get_year_from_date`. Models now import and use it.
+
+3) Implement Umlage allocation to Abrechnung
+   - For each LandAbrechnung year, distribute expense buckets (grundsteuer_betrag, versicherungen_betrag, verbandsbeitraege_betrag, ggf. jagdpacht) across active leases using area share; increase umlagen_vereinnahmt accordingly.
+   - Extend sync_abrechnungen to recompute Umlagen similar to Pacht.
+
+4) Prune unused LandAbrechnung fields
+   - If Paperless is canonical: drop the three file fields.
+   - If no plan for zahlungen JSON: remove or replace with a Payment model.
+
+5) Indexing
+   - Add selective DB indexes on frequent filters (year, status, land, konto+datum).
+
+6) Permissions
+   - If using custom permissions, define them in a data migration or settings; remove the managed=False model class.
+
+7) Template consolidation
+   - Extract shared partials for lease display to reduce duplication between legacy and new renders.
+
+## Notes
+- Date: 2025-08-31
+- Compose services OK; app checks pass (docker-compose exec web python manage.py check).
+ - To try conversion: `python manage.py convert_legacy_verpachtungen` (dry-run), add `--commit` to apply changes; optionally `--limit N`.