diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-24 08:02:12 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-24 11:54:18 +0000 |
| commit | 70d0197e85ae4ef85202781f6d2dc9e76bd508b3 (patch) | |
| tree | 45efb6565e81ba755acc5955e68d5b7119d1e122 /docs/explanation | |
| parent | f8c3e3920ed2a1bdaab30be912276993449a5476 (diff) | |
feat(purgatory): add broken purgatory implementation
Diffstat (limited to 'docs/explanation')
| -rw-r--r-- | docs/explanation/architecture.md | 71 | ||||
| -rw-r--r-- | docs/explanation/decisions.md | 29 | ||||
| -rw-r--r-- | docs/explanation/purgatory-design.md | 112 |
3 files changed, 211 insertions, 1 deletions
diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md index efa3423..6da2295 100644 --- a/docs/explanation/architecture.md +++ b/docs/explanation/architecture.md | |||
| @@ -239,7 +239,76 @@ pub struct RepositoryAnnouncement { ... } | |||
| 239 | pub struct RepositoryState { ... } | 239 | pub struct RepositoryState { ... } |
| 240 | ``` | 240 | ``` |
| 241 | 241 | ||
| 242 | ### 5. Configuration ([`src/config.rs`](src/config.rs)) | 242 | ### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/)) |
| 243 | |||
| 244 | The purgatory system solves the "which arrives first?" problem where either nostr events or git pushes can arrive in any order. It provides an in-memory holding area for events and git data awaiting their counterparts. | ||
| 245 | |||
| 246 | **Design Document**: See [`purgatory-design.md`](purgatory-design.md) for complete design specifications. | ||
| 247 | |||
| 248 | #### Architecture | ||
| 249 | |||
| 250 | ```rust | ||
| 251 | /// Main purgatory structure with two separate stores | ||
| 252 | pub struct Purgatory { | ||
| 253 | /// State events (kind 30618) indexed by repository identifier | ||
| 254 | state_events: Arc<DashMap<String, Vec<StatePurgatoryEntry>>>, | ||
| 255 | |||
| 256 | /// PR events (kind 1617/1618) or placeholders indexed by event ID | ||
| 257 | pr_events: Arc<DashMap<String, PrPurgatoryEntry>>, | ||
| 258 | } | ||
| 259 | ``` | ||
| 260 | |||
| 261 | **Key Design Principles:** | ||
| 262 | |||
| 263 | 1. **Separate Storage**: State events and PR events use different indexing strategies | ||
| 264 | - State events: Indexed by `identifier` (multiple events can wait for same repo) | ||
| 265 | - PR events: Indexed by `event_id` (one-to-one mapping) | ||
| 266 | |||
| 267 | 2. **Late Binding**: State event refs are extracted at git push time, not event arrival | ||
| 268 | - Enables flexible matching when pushes arrive out-of-order | ||
| 269 | - Helper functions in [`helpers.rs`](../../src/purgatory/helpers.rs) handle ref extraction | ||
| 270 | |||
| 271 | 3. **Bidirectional Waiting**: Either side can arrive first | ||
| 272 | - **Event-first**: Event waits for git push | ||
| 273 | - **Git-first**: Placeholder created, waits for event | ||
| 274 | |||
| 275 | 4. **Automatic Expiry**: 30-minute default expiry, extensible during processing | ||
| 276 | - Background cleanup task runs every 60 seconds | ||
| 277 | - Removes expired entries from both stores | ||
| 278 | |||
| 279 | #### Data Types | ||
| 280 | |||
| 281 | See [`types.rs`](../../src/purgatory/types.rs) for complete definitions: | ||
| 282 | |||
| 283 | - **[`RefPair`](../../src/purgatory/types.rs:16)**: Ref name + object SHA pair | ||
| 284 | - **[`StatePurgatoryEntry`](../../src/purgatory/types.rs:29)**: State event with metadata | ||
| 285 | - **[`PrPurgatoryEntry`](../../src/purgatory/types.rs:52)**: PR event or placeholder with metadata | ||
| 286 | |||
| 287 | #### Integration Points | ||
| 288 | |||
| 289 | **Write Policy** ([`src/nostr/policy/`](../../src/nostr/policy/)): | ||
| 290 | - State policy checks git data existence before adding to purgatory | ||
| 291 | - PR policy checks for placeholders before adding to purgatory | ||
| 292 | - Events return "purgatory: will not be served until git data arrives" message | ||
| 293 | |||
| 294 | **Git Handlers** ([`src/git/handlers.rs`](../../src/git/handlers.rs)): | ||
| 295 | - On git push: Check purgatory for matching state events | ||
| 296 | - On refs/nostr/* push: Check purgatory for PR events or create placeholders | ||
| 297 | - Release events from purgatory when git data arrives | ||
| 298 | - Save released events to database | ||
| 299 | |||
| 300 | **Main.rs** ([`src/main.rs`](../../src/main.rs)): | ||
| 301 | - Creates `Arc<Purgatory>` at startup | ||
| 302 | - Passes to both write policy and git handlers | ||
| 303 | - Spawns background cleanup task (60-second interval) | ||
| 304 | |||
| 305 | #### Thread Safety | ||
| 306 | |||
| 307 | - Uses `Arc<DashMap>` for lock-free concurrent access | ||
| 308 | - Safe to share between HTTP handlers, WebSocket handlers, and background tasks | ||
| 309 | - No blocking locks in hot paths | ||
| 310 | |||
| 311 | ### 6. Configuration ([`src/config.rs`](src/config.rs)) | ||
| 243 | 312 | ||
| 244 | ```rust | 313 | ```rust |
| 245 | pub struct Config { | 314 | pub struct Config { |
diff --git a/docs/explanation/decisions.md b/docs/explanation/decisions.md index e9b7422..c72112d 100644 --- a/docs/explanation/decisions.md +++ b/docs/explanation/decisions.md | |||
| @@ -172,3 +172,32 @@ The additional complexity of parsing the Git protocol is minimal compared to the | |||
| 172 | 6. ⏭️ Implement push validation logic | 172 | 6. ⏭️ Implement push validation logic |
| 173 | 7. ⏭️ Integration tests | 173 | 7. ⏭️ Integration tests |
| 174 | 8. ⏭️ GRASP-01 compliance testing | 174 | 8. ⏭️ GRASP-01 compliance testing |
| 175 | |||
| 176 | ## Purgatory Implementation (2025-12-23) | ||
| 177 | |||
| 178 | Implemented according to design specification in [`purgatory-design.md`](purgatory-design.md). No significant deviations from original design. | ||
| 179 | |||
| 180 | **Implementation approach:** | ||
| 181 | - Phases 1-7 completed sequentially as planned | ||
| 182 | - All data structures match design specifications | ||
| 183 | - Integration points implemented as designed | ||
| 184 | |||
| 185 | **Key technical choices:** | ||
| 186 | |||
| 187 | 1. **Optional Purgatory in Git Handlers**: Used `Option<Arc<Purgatory>>` in git handler signatures for backward compatibility. This allows git handlers to function even when no purgatory is provided (e.g., in minimal test setups). | ||
| 188 | |||
| 189 | 2. **Cleanup Interval**: Background cleanup task runs every 60 seconds as designed, removing expired entries from both state and PR stores. | ||
| 190 | |||
| 191 | 3. **Thread-Safe Storage**: Used `Arc<DashMap>` for lock-free concurrent access, enabling safe sharing between HTTP handlers, WebSocket handlers, and background tasks. | ||
| 192 | |||
| 193 | 4. **Late Binding Implementation**: Ref extraction logic in [`helpers.rs`](../../src/purgatory/helpers.rs) extracts refs at git push time, not event arrival time, as specified in the design. | ||
| 194 | |||
| 195 | **Test integration:** | ||
| 196 | - Existing test code in `grasp-audit` was uncommented (Phases 7) | ||
| 197 | - No new integration tests added (as instructed) | ||
| 198 | - Test verification enabled in [`grasp-audit/src/client.rs`](../../grasp-audit/src/client.rs) and [`grasp-audit/src/specs/grasp01/push_authorization.rs`](../../grasp-audit/src/specs/grasp01/push_authorization.rs) | ||
| 199 | |||
| 200 | **Related Documentation:** | ||
| 201 | - Design: [`purgatory-design.md`](purgatory-design.md) | ||
| 202 | - Architecture: [`architecture.md`](architecture.md#5-purgatory-system-srcpurgatory) | ||
| 203 | - Implementation Plan: [`../purgatory-implementation-plan.md`](../purgatory-implementation-plan.md) | ||
diff --git a/docs/explanation/purgatory-design.md b/docs/explanation/purgatory-design.md index bf867ad..5ee4e06 100644 --- a/docs/explanation/purgatory-design.md +++ b/docs/explanation/purgatory-design.md | |||
| @@ -1,5 +1,10 @@ | |||
| 1 | # Purgatory Implementation Design | 1 | # Purgatory Implementation Design |
| 2 | 2 | ||
| 3 | **Status**: ✅ Implemented (2025-12-23) | ||
| 4 | **Implementation**: Phases 1-7 complete | ||
| 5 | **Source Code**: [`src/purgatory/`](../../src/purgatory/) | ||
| 6 | **Related**: [`docs/explanation/architecture.md`](architecture.md) - System architecture overview | ||
| 7 | |||
| 3 | ## Overview | 8 | ## Overview |
| 4 | 9 | ||
| 5 | Purgatory is an in-memory holding area for nostr events that depend on git data that hasn't arrived yet, **and** for git data that arrived before its corresponding nostr event. Events/placeholders are held until the other half arrives, at which point they are processed and saved to the database. | 10 | Purgatory is an in-memory holding area for nostr events that depend on git data that hasn't arrived yet, **and** for git data that arrived before its corresponding nostr event. Events/placeholders are held until the other half arrives, at which point they are processed and saved to the database. |
| @@ -899,3 +904,110 @@ WritePolicyResult::Accept | |||
| 899 | 2. **Same event submitted twice** - Deduplicated by event ID | 904 | 2. **Same event submitted twice** - Deduplicated by event ID |
| 900 | 3. **Push timeout during processing** - Entry expiry extended to 15 min minimum | 905 | 3. **Push timeout during processing** - Entry expiry extended to 15 min minimum |
| 901 | 4. **Race between event and git push** - Whichever completes the pair triggers release | 906 | 4. **Race between event and git push** - Whichever completes the pair triggers release |
| 907 | |||
| 908 | |||
| 909 | ## Purgatory Authorization Fix (2025-12-24) | ||
| 910 | |||
| 911 | **Critical Implementation Note**: The original purgatory design placed purgatory checking AFTER git push execution. This created a deadlock where pushes were rejected because the authorizing state event was in purgatory, not the database. | ||
| 912 | |||
| 913 | ### The Deadlock Problem | ||
| 914 | |||
| 915 | **Original broken flow:** | ||
| 916 | 1. State event arrives → No git data exists → Event stored in PURGATORY (not database) | ||
| 917 | 2. Git push arrives → Authorization checks DATABASE only → No state found → **PUSH REJECTED** ❌ | ||
| 918 | 3. Purgatory check runs → But push already failed, so this never helps | ||
| 919 | |||
| 920 | ### The Fix: Authorization-Time Purgatory Check | ||
| 921 | |||
| 922 | **Correct flow (implemented):** | ||
| 923 | 1. State event arrives → No git data exists → Event stored in purgatory | ||
| 924 | 2. Git push arrives → Authorization checks **DATABASE + PURGATORY** → State found in purgatory → **PUSH AUTHORIZED** ✅ | ||
| 925 | 3. After successful push → Save purgatory event to database → Remove from purgatory | ||
| 926 | |||
| 927 | ### Implementation Details | ||
| 928 | |||
| 929 | #### 1. Modified [`AuthorizationResult`](../../src/git/authorization.rs:397) | ||
| 930 | |||
| 931 | Added `from_purgatory: bool` field to track whether the authorizing state came from purgatory: | ||
| 932 | |||
| 933 | ```rust | ||
| 934 | pub struct AuthorizationResult { | ||
| 935 | pub authorized: bool, | ||
| 936 | pub reason: String, | ||
| 937 | pub state: Option<RepositoryState>, | ||
| 938 | pub maintainers: Vec<String>, | ||
| 939 | pub from_purgatory: bool, // NEW: Track event source | ||
| 940 | } | ||
| 941 | ``` | ||
| 942 | |||
| 943 | #### 2. Enhanced [`get_authorization_for_owner()`](../../src/git/authorization.rs:342) | ||
| 944 | |||
| 945 | Added purgatory checking when no state found in database: | ||
| 946 | |||
| 947 | ```rust | ||
| 948 | pub async fn get_authorization_for_owner( | ||
| 949 | database: &SharedDatabase, | ||
| 950 | identifier: &str, | ||
| 951 | owner_pubkey: &str, | ||
| 952 | purgatory: Option<&Arc<Purgatory>>, | ||
| 953 | pushed_refs: &[(String, String, String)], | ||
| 954 | repo_path: &Path, | ||
| 955 | ) -> Result<AuthorizationResult> | ||
| 956 | ``` | ||
| 957 | |||
| 958 | **Logic**: | ||
| 959 | 1. Check database for state events (existing behavior) | ||
| 960 | 2. If no state in database AND purgatory available: | ||
| 961 | - Parse pushed refs to RefPairs | ||
| 962 | - Get local refs from repository | ||
| 963 | - Call [`find_matching_states()`](../../src/purgatory/mod.rs:203) | ||
| 964 | - Filter to latest event from authorized authors | ||
| 965 | - Return authorization with `from_purgatory: true` | ||
| 966 | |||
| 967 | #### 3. Post-Push Purgatory Event Save | ||
| 968 | |||
| 969 | In [`handle_receive_pack()`](../../src/git/handlers.rs:187), after successful push: | ||
| 970 | |||
| 971 | ```rust | ||
| 972 | if from_purgatory { | ||
| 973 | if let (Some(db), Some(purg)) = (&database, &purgatory) { | ||
| 974 | // Save state event to database | ||
| 975 | db.save_event(&state.event).await?; | ||
| 976 | |||
| 977 | // Remove from purgatory | ||
| 978 | purg.remove_state_event(identifier, &state.event.id); | ||
| 979 | } | ||
| 980 | } | ||
| 981 | ``` | ||
| 982 | |||
| 983 | ### Files Modified | ||
| 984 | |||
| 985 | 1. **[`src/git/authorization.rs`](../../src/git/authorization.rs)** | ||
| 986 | - Added `from_purgatory` field to `AuthorizationResult` | ||
| 987 | - Modified `get_authorization_for_owner()` signature and logic | ||
| 988 | - Added purgatory checking when database has no state | ||
| 989 | |||
| 990 | 2. **[`src/git/handlers.rs`](../../src/git/handlers.rs)** | ||
| 991 | - Modified `authorize_push()` to accept purgatory and repo_path parameters | ||
| 992 | - Added tracking of `from_purgatory` flag | ||
| 993 | - Added post-push database save for purgatory events | ||
| 994 | |||
| 995 | ### Why This Order Matters | ||
| 996 | |||
| 997 | Checking purgatory DURING authorization (before push execution) is critical: | ||
| 998 | |||
| 999 | - **Prevents deadlock**: Push is authorized by purgatory state before execution | ||
| 1000 | - **Maintains atomicity**: Only saves to database after successful push | ||
| 1001 | - **Race condition safe**: First successful push claims the purgatory event | ||
| 1002 | |||
| 1003 | The alternative (checking purgatory after push) creates an insurmountable deadlock where valid pushes are rejected because their authorizing state is in purgatory instead of the database. | ||
| 1004 | |||
| 1005 | ### Testing | ||
| 1006 | |||
| 1007 | The fix enables the `test_push_authorized_by_owner_state` integration test scenario where: | ||
| 1008 | 1. State event is sent to relay (goes to purgatory - no git data yet) | ||
| 1009 | 2. Git push is sent (uses purgatory state for authorization) | ||
| 1010 | 3. State event is released from purgatory to database | ||
| 1011 | |||
| 1012 | --- | ||
| 1013 | |||