upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2025-12-24 08:02:12 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2025-12-24 11:54:18 +0000
commit70d0197e85ae4ef85202781f6d2dc9e76bd508b3 (patch)
tree45efb6565e81ba755acc5955e68d5b7119d1e122 /docs/explanation
parentf8c3e3920ed2a1bdaab30be912276993449a5476 (diff)
feat(purgatory): add broken purgatory implementation
Diffstat (limited to 'docs/explanation')
-rw-r--r--docs/explanation/architecture.md71
-rw-r--r--docs/explanation/decisions.md29
-rw-r--r--docs/explanation/purgatory-design.md112
3 files changed, 211 insertions, 1 deletions
diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md
index efa3423..6da2295 100644
--- a/docs/explanation/architecture.md
+++ b/docs/explanation/architecture.md
@@ -239,7 +239,76 @@ pub struct RepositoryAnnouncement { ... }
239pub struct RepositoryState { ... } 239pub struct RepositoryState { ... }
240``` 240```
241 241
242### 5. Configuration ([`src/config.rs`](src/config.rs)) 242### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/))
243
244The purgatory system solves the "which arrives first?" problem where either nostr events or git pushes can arrive in any order. It provides an in-memory holding area for events and git data awaiting their counterparts.
245
246**Design Document**: See [`purgatory-design.md`](purgatory-design.md) for complete design specifications.
247
248#### Architecture
249
250```rust
251/// Main purgatory structure with two separate stores
252pub struct Purgatory {
253 /// State events (kind 30618) indexed by repository identifier
254 state_events: Arc<DashMap<String, Vec<StatePurgatoryEntry>>>,
255
256 /// PR events (kind 1617/1618) or placeholders indexed by event ID
257 pr_events: Arc<DashMap<String, PrPurgatoryEntry>>,
258}
259```
260
261**Key Design Principles:**
262
2631. **Separate Storage**: State events and PR events use different indexing strategies
264 - State events: Indexed by `identifier` (multiple events can wait for same repo)
265 - PR events: Indexed by `event_id` (one-to-one mapping)
266
2672. **Late Binding**: State event refs are extracted at git push time, not event arrival
268 - Enables flexible matching when pushes arrive out-of-order
269 - Helper functions in [`helpers.rs`](../../src/purgatory/helpers.rs) handle ref extraction
270
2713. **Bidirectional Waiting**: Either side can arrive first
272 - **Event-first**: Event waits for git push
273 - **Git-first**: Placeholder created, waits for event
274
2754. **Automatic Expiry**: 30-minute default expiry, extensible during processing
276 - Background cleanup task runs every 60 seconds
277 - Removes expired entries from both stores
278
279#### Data Types
280
281See [`types.rs`](../../src/purgatory/types.rs) for complete definitions:
282
283- **[`RefPair`](../../src/purgatory/types.rs:16)**: Ref name + object SHA pair
284- **[`StatePurgatoryEntry`](../../src/purgatory/types.rs:29)**: State event with metadata
285- **[`PrPurgatoryEntry`](../../src/purgatory/types.rs:52)**: PR event or placeholder with metadata
286
287#### Integration Points
288
289**Write Policy** ([`src/nostr/policy/`](../../src/nostr/policy/)):
290- State policy checks git data existence before adding to purgatory
291- PR policy checks for placeholders before adding to purgatory
292- Events return "purgatory: will not be served until git data arrives" message
293
294**Git Handlers** ([`src/git/handlers.rs`](../../src/git/handlers.rs)):
295- On git push: Check purgatory for matching state events
296- On refs/nostr/* push: Check purgatory for PR events or create placeholders
297- Release events from purgatory when git data arrives
298- Save released events to database
299
300**Main.rs** ([`src/main.rs`](../../src/main.rs)):
301- Creates `Arc<Purgatory>` at startup
302- Passes to both write policy and git handlers
303- Spawns background cleanup task (60-second interval)
304
305#### Thread Safety
306
307- Uses `Arc<DashMap>` for lock-free concurrent access
308- Safe to share between HTTP handlers, WebSocket handlers, and background tasks
309- No blocking locks in hot paths
310
311### 6. Configuration ([`src/config.rs`](src/config.rs))
243 312
244```rust 313```rust
245pub struct Config { 314pub struct Config {
diff --git a/docs/explanation/decisions.md b/docs/explanation/decisions.md
index e9b7422..c72112d 100644
--- a/docs/explanation/decisions.md
+++ b/docs/explanation/decisions.md
@@ -172,3 +172,32 @@ The additional complexity of parsing the Git protocol is minimal compared to the
1726. ⏭️ Implement push validation logic 1726. ⏭️ Implement push validation logic
1737. ⏭️ Integration tests 1737. ⏭️ Integration tests
1748. ⏭️ GRASP-01 compliance testing 1748. ⏭️ GRASP-01 compliance testing
175
176## Purgatory Implementation (2025-12-23)
177
178Implemented according to design specification in [`purgatory-design.md`](purgatory-design.md). No significant deviations from original design.
179
180**Implementation approach:**
181- Phases 1-7 completed sequentially as planned
182- All data structures match design specifications
183- Integration points implemented as designed
184
185**Key technical choices:**
186
1871. **Optional Purgatory in Git Handlers**: Used `Option<Arc<Purgatory>>` in git handler signatures for backward compatibility. This allows git handlers to function even when no purgatory is provided (e.g., in minimal test setups).
188
1892. **Cleanup Interval**: Background cleanup task runs every 60 seconds as designed, removing expired entries from both state and PR stores.
190
1913. **Thread-Safe Storage**: Used `Arc<DashMap>` for lock-free concurrent access, enabling safe sharing between HTTP handlers, WebSocket handlers, and background tasks.
192
1934. **Late Binding Implementation**: Ref extraction logic in [`helpers.rs`](../../src/purgatory/helpers.rs) extracts refs at git push time, not event arrival time, as specified in the design.
194
195**Test integration:**
196- Existing test code in `grasp-audit` was uncommented (Phases 7)
197- No new integration tests added (as instructed)
198- Test verification enabled in [`grasp-audit/src/client.rs`](../../grasp-audit/src/client.rs) and [`grasp-audit/src/specs/grasp01/push_authorization.rs`](../../grasp-audit/src/specs/grasp01/push_authorization.rs)
199
200**Related Documentation:**
201- Design: [`purgatory-design.md`](purgatory-design.md)
202- Architecture: [`architecture.md`](architecture.md#5-purgatory-system-srcpurgatory)
203- Implementation Plan: [`../purgatory-implementation-plan.md`](../purgatory-implementation-plan.md)
diff --git a/docs/explanation/purgatory-design.md b/docs/explanation/purgatory-design.md
index bf867ad..5ee4e06 100644
--- a/docs/explanation/purgatory-design.md
+++ b/docs/explanation/purgatory-design.md
@@ -1,5 +1,10 @@
1# Purgatory Implementation Design 1# Purgatory Implementation Design
2 2
3**Status**: ✅ Implemented (2025-12-23)
4**Implementation**: Phases 1-7 complete
5**Source Code**: [`src/purgatory/`](../../src/purgatory/)
6**Related**: [`docs/explanation/architecture.md`](architecture.md) - System architecture overview
7
3## Overview 8## Overview
4 9
5Purgatory is an in-memory holding area for nostr events that depend on git data that hasn't arrived yet, **and** for git data that arrived before its corresponding nostr event. Events/placeholders are held until the other half arrives, at which point they are processed and saved to the database. 10Purgatory is an in-memory holding area for nostr events that depend on git data that hasn't arrived yet, **and** for git data that arrived before its corresponding nostr event. Events/placeholders are held until the other half arrives, at which point they are processed and saved to the database.
@@ -899,3 +904,110 @@ WritePolicyResult::Accept
8992. **Same event submitted twice** - Deduplicated by event ID 9042. **Same event submitted twice** - Deduplicated by event ID
9003. **Push timeout during processing** - Entry expiry extended to 15 min minimum 9053. **Push timeout during processing** - Entry expiry extended to 15 min minimum
9014. **Race between event and git push** - Whichever completes the pair triggers release 9064. **Race between event and git push** - Whichever completes the pair triggers release
907
908
909## Purgatory Authorization Fix (2025-12-24)
910
911**Critical Implementation Note**: The original purgatory design placed purgatory checking AFTER git push execution. This created a deadlock where pushes were rejected because the authorizing state event was in purgatory, not the database.
912
913### The Deadlock Problem
914
915**Original broken flow:**
9161. State event arrives → No git data exists → Event stored in PURGATORY (not database)
9172. Git push arrives → Authorization checks DATABASE only → No state found → **PUSH REJECTED** ❌
9183. Purgatory check runs → But push already failed, so this never helps
919
920### The Fix: Authorization-Time Purgatory Check
921
922**Correct flow (implemented):**
9231. State event arrives → No git data exists → Event stored in purgatory
9242. Git push arrives → Authorization checks **DATABASE + PURGATORY** → State found in purgatory → **PUSH AUTHORIZED** ✅
9253. After successful push → Save purgatory event to database → Remove from purgatory
926
927### Implementation Details
928
929#### 1. Modified [`AuthorizationResult`](../../src/git/authorization.rs:397)
930
931Added `from_purgatory: bool` field to track whether the authorizing state came from purgatory:
932
933```rust
934pub struct AuthorizationResult {
935 pub authorized: bool,
936 pub reason: String,
937 pub state: Option<RepositoryState>,
938 pub maintainers: Vec<String>,
939 pub from_purgatory: bool, // NEW: Track event source
940}
941```
942
943#### 2. Enhanced [`get_authorization_for_owner()`](../../src/git/authorization.rs:342)
944
945Added purgatory checking when no state found in database:
946
947```rust
948pub async fn get_authorization_for_owner(
949 database: &SharedDatabase,
950 identifier: &str,
951 owner_pubkey: &str,
952 purgatory: Option<&Arc<Purgatory>>,
953 pushed_refs: &[(String, String, String)],
954 repo_path: &Path,
955) -> Result<AuthorizationResult>
956```
957
958**Logic**:
9591. Check database for state events (existing behavior)
9602. If no state in database AND purgatory available:
961 - Parse pushed refs to RefPairs
962 - Get local refs from repository
963 - Call [`find_matching_states()`](../../src/purgatory/mod.rs:203)
964 - Filter to latest event from authorized authors
965 - Return authorization with `from_purgatory: true`
966
967#### 3. Post-Push Purgatory Event Save
968
969In [`handle_receive_pack()`](../../src/git/handlers.rs:187), after successful push:
970
971```rust
972if from_purgatory {
973 if let (Some(db), Some(purg)) = (&database, &purgatory) {
974 // Save state event to database
975 db.save_event(&state.event).await?;
976
977 // Remove from purgatory
978 purg.remove_state_event(identifier, &state.event.id);
979 }
980}
981```
982
983### Files Modified
984
9851. **[`src/git/authorization.rs`](../../src/git/authorization.rs)**
986 - Added `from_purgatory` field to `AuthorizationResult`
987 - Modified `get_authorization_for_owner()` signature and logic
988 - Added purgatory checking when database has no state
989
9902. **[`src/git/handlers.rs`](../../src/git/handlers.rs)**
991 - Modified `authorize_push()` to accept purgatory and repo_path parameters
992 - Added tracking of `from_purgatory` flag
993 - Added post-push database save for purgatory events
994
995### Why This Order Matters
996
997Checking purgatory DURING authorization (before push execution) is critical:
998
999- **Prevents deadlock**: Push is authorized by purgatory state before execution
1000- **Maintains atomicity**: Only saves to database after successful push
1001- **Race condition safe**: First successful push claims the purgatory event
1002
1003The alternative (checking purgatory after push) creates an insurmountable deadlock where valid pushes are rejected because their authorizing state is in purgatory instead of the database.
1004
1005### Testing
1006
1007The fix enables the `test_push_authorized_by_owner_state` integration test scenario where:
10081. State event is sent to relay (goes to purgatory - no git data yet)
10092. Git push is sent (uses purgatory state for authorization)
10103. State event is released from purgatory to database
1011
1012---
1013