upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/purgatory-implementation-plan.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/purgatory-implementation-plan.md')
-rw-r--r--docs/purgatory-implementation-plan.md566
1 files changed, 0 insertions, 566 deletions
diff --git a/docs/purgatory-implementation-plan.md b/docs/purgatory-implementation-plan.md
deleted file mode 100644
index 4ba453c..0000000
--- a/docs/purgatory-implementation-plan.md
+++ /dev/null
@@ -1,566 +0,0 @@
1# Purgatory Feature Implementation Plan
2
3**Status**: Ready for implementation
4**Created**: 2025-12-23
5**Design Doc**: [`docs/explanation/purgatory-design.md`](explanation/purgatory-design.md)
6
7## Overview
8
9Implement the purgatory feature for ngit-grasp relay according to GRASP-01 specification. Purgatory is an in-memory holding area for nostr events that depend on git data that hasn't arrived yet, and for git data that arrived before its corresponding nostr event.
10
11## Context for AI Agents
12
13### What is Purgatory?
14
15Purgatory solves the "which arrives first?" problem:
16
17- **Nostr event first**: Event waits for git push containing the data
18- **Git data first**: Git data waits for the nostr event to be published
19
20Events are held in memory (30 min expiry) until the other half arrives, then processed atomically.
21
22### Key Design Principles
23
241. **Separate storage**: State events (kind 30618) and PR events (kind 1617/1618) use different indices
252. **Late binding**: State event refs are extracted at git push time, not event arrival
263. **Bidirectional waiting**: Either side can arrive first
274. **Single expiry timer**: 30 min expiry, extended to 15 min minimum when processing starts
28
29### Existing Test Coverage
30
31Tests are ALREADY written and passing (with purgatory checks commented out):
32
33- [`tests/push_authorization.rs`](../tests/push_authorization.rs) - ngit-grasp integration tests
34- [`grasp-audit/src/specs/grasp01/push_authorization.rs`](../grasp-audit/src/specs/grasp01/push_authorization.rs) - detailed test implementations
35
36**DO NOT write new integration tests**. Only uncomment existing test code.
37
38### Critical Rules for All Agents
39
401. **Ask before deviating** from this plan
412. **Never add new integration tests** - only uncomment existing ones
423. **Always commit** changes before reporting completion
434. **Use nostr-sdk 0.43+ API**: Direct field access (`event.id`, `event.tags`), not method calls
445. **Test after each phase**: Run `cargo test --test push_authorization` to verify
456. **Update architecture docs** if implementation differs from design
46
47## Implementation Phases
48
49Each phase is sized for a single AI agent session with fresh context.
50
51---
52
53## Phase 1: Core Purgatory Data Structures
54
55**Goal**: Create the foundational purgatory module with all data structures and basic API.
56
57**Files to Create**:
58
59- `src/purgatory/mod.rs` - Public API and main Purgatory struct
60- `src/purgatory/types.rs` - Data structures (RefPair, Entry types)
61- Update `src/lib.rs` - Add `pub mod purgatory;`
62
63### Data Structures
64
65See design doc lines 63-126 for specifications.
66
67Key types:
68
69- `RefPair` - ref name + commit/tag SHA pair
70- `StatePurgatoryEntry` - State event with metadata
71- `PrPurgatoryEntry` - PR event or placeholder with metadata
72- `Purgatory` - Main struct with DashMap stores
73
74### Success Criteria
75
76- [x] All files created and compile successfully
77- [x] `cargo build` passes
78- [x] Data structures match design spec
79- [x] Basic method stubs present
80- [x] Commit: `feat(purgatory): add core data structures`
81
82### Agent Instructions
83
841. Create `src/purgatory/` directory with `mod.rs` and `types.rs`
852. Implement data structures per design doc
863. Add `pub mod purgatory;` to `src/lib.rs`
874. Implement method stubs (can return hardcoded values)
885. Verify `cargo build` passes
896. Commit changes
90
91---
92
93## Phase 2: Purgatory State Event Logic
94
95**Goal**: Implement state event purgatory methods with ref parsing and matching.
96
97**Files to Modify**:
98
99- `src/purgatory/mod.rs` - Implement state event methods
100- `src/purgatory/helpers.rs` (create) - Ref extraction utilities
101
102### Key Methods
103
104See design doc lines 383-403 for API details.
105
106- `add_state()` - Add state event to purgatory
107- `find_matching_states()` - Find events that match pushed refs
108- `extend_expiry()` - Extend timer for events being processed
109- `remove_state()` - Remove after successful processing
110
111### Helper Functions
112
113See design doc lines 443-471 for specifications.
114
115- `extract_refs_from_state()` - Parse ref tags from event
116- `can_satisfy_state()` - Check if push satisfies state event
117- `get_unpushed_refs()` - Get refs not in push
118
119### Success Criteria
120
121- [x] Ref extraction from tags works correctly
122- [x] Matching logic implements design spec
123- [x] Unit tests for helpers pass
124- [x] `cargo build` and `cargo test --lib` pass
125- [x] Commit: `feat(purgatory): implement state event logic`
126
127### Agent Instructions
128
1291. Create `helpers.rs` with ref parsing functions
1302. Implement state event methods in `mod.rs`
1313. Add unit tests for helper functions
1324. Verify all tests pass
1335. Commit changes
134
135---
136
137## Phase 3: Purgatory PR Event Logic
138
139**Goal**: Implement PR event purgatory methods and placeholder handling.
140
141**Files to Modify**:
142
143- `src/purgatory/mod.rs` - Implement PR event methods
144
145### Key Methods
146
147See design doc lines 406-434 for API details.
148
149- `add_pr()` - Add PR event to purgatory
150- `add_pr_placeholder()` - Create placeholder for git-first scenario
151- `find_pr()` - Find PR entry (event or placeholder)
152- `find_pr_placeholder()` - Find placeholder specifically
153- `remove_pr()` - Remove after processing
154- `cleanup()` - Remove expired entries (60s interval)
155
156### Success Criteria
157
158- [x] PR methods handle event and placeholder scenarios
159- [x] Cleanup removes expired entries from both stores
160- [x] Unit tests for PR logic pass
161- [x] `cargo build` and `cargo test --lib` pass
162- [x] Commit: `feat(purgatory): implement PR event logic and cleanup`
163
164### Agent Instructions
165
1661. Implement all PR event methods
1672. Ensure placeholder handling works correctly
1683. Implement cleanup with expiry checking
1694. Write unit tests
1705. Commit changes
171
172---
173
174## Phase 4: Integration with Write Policy (Nostr Events)
175
176**Goal**: Integrate purgatory into `Nip34WritePolicy` for event handling.
177
178**Files to Modify**:
179
180- `src/nostr/policy/mod.rs` - Add purgatory to PolicyContext
181- `src/nostr/builder.rs` - Pass purgatory to WritePolicy
182- `src/nostr/policy/state.rs` - Use purgatory for state events
183- `src/nostr/policy/pr_event.rs` - Use purgatory for PR events
184
185### Integration Points
186
187See design doc lines 477-573 for detailed integration logic.
188
189**State events**: Check if git data exists, if not add to purgatory with status=true message.
190
191**PR events**: Check for placeholders first, add to purgatory if no git data.
192
193### Success Criteria
194
195- [x] PolicyContext includes purgatory
196- [x] State/PR policies use purgatory when git data missing
197- [x] Events return "purgatory:" messages
198- [x] `cargo build` passes (expected errors in create_relay for Phase 6)
199- [x] Commit: `feat(purgatory): integrate with write policy`
200
201### Agent Instructions
202
2031. Add purgatory field to PolicyContext
2042. Update state policy to check/add to purgatory
2053. Update PR policy to check placeholders
2064. Update WritePolicy constructor signature
2075. Commit changes
208
209**Note**: Don't modify main.rs yet - that's Phase 6.
210
211---
212
213## Phase 5: Integration with Git Handlers (Git Pushes)
214
215**Goal**: Integrate purgatory into git push handlers to release events when git data arrives.
216
217**Files to Modify**:
218
219- `src/git/handlers.rs` - Check purgatory on push, release events
220
221### Integration Points
222
223See design doc lines 580-692 for detailed push handling logic.
224
225**Normal refs (state events)**:
226
227- Convert pushed refs to RefPairs
228- Get local refs
229- Find matching states in purgatory
230- Use for authorization
231- Release and save to database on success
232
233**refs/nostr/\* (PR events)**:
234
235- Extract event_id from ref name
236- Check purgatory for matching PR event
237- Verify commit match
238- Release from purgatory and save
239- Create placeholder if no event exists yet
240
241### Success Criteria
242
243- [x] Git pushes check purgatory for matching events
244- [x] State events released when git data pushed
245- [x] PR events released when refs/nostr/\* pushed
246- [x] Placeholders created for git-data-first
247- [x] Events saved to database when released
248- [x] `cargo build` passes
249- [x] Commit: `feat(purgatory): integrate with git handlers`
250
251### Agent Instructions
252
2531. Modify `handle_receive_pack()` to check purgatory
2542. Add logic for refs/nostr/\* detection
2553. Implement PR event matching and release
2564. Implement placeholder creation
2575. Add helper to extract commit from PR event
2586. Commit changes
259
260---
261
262## Phase 6: Main.rs Integration and Cleanup Task
263
264**Goal**: Wire purgatory into main.rs startup and add background cleanup task.
265
266**Files to Modify**:
267
268- `src/main.rs` - Create purgatory, pass to components, spawn cleanup
269
270### Main.rs Changes
271
272See design doc lines 696-727 for startup integration.
273
2741. Create `Arc<Purgatory>` at startup
2752. Pass to WritePolicy constructor
2763. Pass to git handlers (via app state or parameter)
2774. Spawn background task running `cleanup()` every 60 seconds
278
279### Success Criteria
280
281- [x] Purgatory created at startup
282- [x] Passed to all required components
283- [x] Cleanup task spawned and logs removals
284- [x] `cargo build` passes
285- [x] `cargo run` starts successfully
286- [x] Commit: `feat(purgatory): wire into main.rs with cleanup task`
287
288### Agent Instructions
289
2901. Review current main.rs structure
2912. Create purgatory early in startup
2923. Pass to WritePolicy
2934. Pass to git handlers
2945. Spawn cleanup task (60s interval)
2956. Test relay startup
2967. Commit changes
297
298---
299
300## Phase 7: Enable Test Code and Verification
301
302**Goal**: Uncomment purgatory test code and verify all tests pass.
303
304**Files to Modify**:
305
306- `grasp-audit/src/client.rs` - Lines 207-213
307- `grasp-audit/src/specs/grasp01/push_authorization.rs` - Lines 1356-1370
308
309### Uncomment Locations
310
311#### Location 1: grasp-audit/src/client.rs:207-213
312
313```rust
314// UNCOMMENT these lines in send_event_expect_purgatory_not_served():
315if !self.is_event_on_relay(event.id).await? {
316 return Err(anyhow!(
317 "event sent to relay was served instead of being put in purgatory"
318 ));
319}
320```
321
322#### Location 2: grasp-audit/src/specs/grasp01/push_authorization.rs:1356-1370
323
324```rust
325// UNCOMMENT entire block checking event not served before git push:
326// Check event is not yet served by relay (still in purgatory)
327match client.is_event_on_relay(pr_event.id).await {
328 Ok(on_relay) => {
329 if !on_relay {
330 return TestResult::new(...)
331 .fail("PR event not in purgatory...");
332 }
333 }
334 Err(_) => {
335 return TestResult::new(...).fail("failed to query relay");
336 }
337}
338```
339
340### Test Commands
341
342```bash
343# Run purgatory-related integration tests
344cargo test --test push_authorization
345
346# Run all tests
347cargo test
348```
349
350### Success Criteria
351
352- [x] Code uncommenting compiles without errors
353- [~] `cargo test --test push_authorization` runs (has fixture creation failures needing investigation)
354- [~] Purgatory functionality verified by tests (partial - 18 passed, 9 failed with fixture issues)
355- [x] No new tests added (only uncommented existing)
356- [~] `cargo test` (all tests) has 1 failure in nip34_announcements (pre-existing fixture issue)
357- [x] Commit: `feat(purgatory): enable test verification`
358
359**Status**: Code uncommenting complete. Test failures appear to be pre-existing fixture creation issues (OwnerStateDataPushed, MaintainerStateDataPushed, PR commit hash mismatches), not caused by uncommenting purgatory verification code. These failures need debugging in a separate session.
360
361### Agent Instructions
362
3631. Uncomment blocks in client.rs:207-213
3642. Uncomment blocks in push_authorization.rs:1356-1370
3653. Search for other TODO comments about purgatory
3664. Run `cargo test --test push_authorization -- --nocapture`
3675. Verify tests pass
3686. Run full `cargo test`
3697. Commit changes
370
371**Important**: If tests fail, debug and fix before marking phase complete.
372
373---
374
375## Phase 8: Documentation Updates
376
377**Goal**: Update architecture docs to reflect implementation.
378
379**Files to Modify**:
380
381- `docs/explanation/purgatory-design.md` - Add implementation status
382- `docs/explanation/architecture.md` - Add purgatory section
383- `docs/explanation/decisions.md` - Document decisions/deviations
384
385### Documentation Updates
386
3871. Mark purgatory-design.md as implemented
3882. Add purgatory system overview to architecture.md
3893. Document any implementation decisions that differ from design
390
391### Success Criteria
392
393- [x] purgatory-design.md marked as implemented
394- [x] Architecture doc updated
395- [x] Decisions documented if any deviations
396- [x] Documentation accurate to implementation
397- [x] Commit: `docs: update for purgatory implementation`
398
399### Agent Instructions
400
4011. Read implementation to understand what was built
4022. Update purgatory-design.md status banner
4033. Add purgatory section to architecture.md
4044. Document decisions/deviations if any
4055. Commit documentation
406
407---
408
409## Phase 9: Final Verification and Cleanup
410
411**Goal**: Run comprehensive tests, verify everything works.
412
413### Verification Steps
414
415```bash
416# 1. All tests
417cargo test
418
419# 2. Integration tests
420cargo test --test push_authorization -- --nocapture
421
422# 3. Clippy
423cargo clippy
424
425# 4. Format
426cargo fmt
427
428# 5. Release build
429cargo build --release
430
431# 6. Test startup
432cargo run &
433sleep 5
434curl http://localhost:3000
435pkill ngit-grasp
436```
437
438### Final Commit
439
440```
441feat(purgatory): complete implementation
442
443- Core data structures (RefPair, Entry types)
444- State event purgatory with late binding
445- PR event purgatory with bidirectional waiting
446- Write policy integration
447- Git handler integration
448- Background cleanup task (60s interval)
449- Test verification enabled
450
451All tests passing. Ready for review.
452```
453
454### Success Criteria
455
456- [x] All tests pass
457- [x] No clippy warnings
458- [x] Code formatted
459- [x] Relay starts without errors
460- [x] Cleanup logs visible
461- [x] No TODOs remaining
462- [x] Comprehensive final commit
463
464### Agent Instructions
465
4661. Run full test suite
4672. Run clippy and fix warnings
4683. Format with cargo fmt
4694. Test relay startup
4705. Review for TODOs/FIXMEs
4716. Create final commit with summary
4727. Report completion
473
474---
475
476## Reference Information
477
478### Key Design Doc Sections
479
480| Lines | Section | Description |
481| ------- | ------------------------ | -------------------------------------- |
482| 63-126 | Data Structures | RefPair, Entry types, Purgatory struct |
483| 131-191 | Event Flows | State/PR event arrival diagrams |
484| 193-263 | Git Push Flows | State matching, PR ref handling |
485| 375-438 | API Methods | Complete purgatory API specification |
486| 443-471 | Helper Functions | Ref extraction and matching |
487| 477-573 | Write Policy Integration | Event handler changes |
488| 580-692 | Git Handler Integration | Push handler changes |
489| 695-727 | Startup Integration | main.rs changes |
490
491### Test Files
492
493- `tests/push_authorization.rs` - Integration test runner
494- `grasp-audit/src/specs/grasp01/push_authorization.rs` - Test implementations
495- Lines 1356-1370 contain commented purgatory checks
496- `grasp-audit/src/client.rs:207-213` contains purgatory verification
497
498### nostr-sdk 0.43 Patterns
499
500```rust
501// ✅ CORRECT
502event.id // Direct field
503event.tags // Direct field
504event.pubkey // Direct field
505tag.kind() // Method on tag
506tag.content() // Method on tag
507
508// ❌ WRONG
509event.id() // No method call
510event.tags() // No method call
511```
512
513### Important Design Rules
514
5151. **Separate stores**: state_events and pr_events use different indices
5162. **Late binding**: Extract state refs at push time, not event arrival
5173. **Bidirectional**: Either event or git can arrive first
5184. **Expiry**: 30 min default, extend to 15 min when processing starts
5195. **Cleanup**: Background task runs every 60 seconds
5206. **In-memory**: Purgatory data lost on restart (acceptable per spec)
521
522### Phase Dependencies
523
524```mermaid
525graph TD
526 P1[Phase 1: Data Structures] --> P2[Phase 2: State Logic]
527 P1 --> P3[Phase 3: PR Logic]
528 P2 --> P4[Phase 4: Write Policy]
529 P3 --> P4
530 P2 --> P5[Phase 5: Git Handlers]
531 P3 --> P5
532 P4 --> P6[Phase 6: Main.rs]
533 P5 --> P6
534 P6 --> P7[Phase 7: Enable Tests]
535 P7 --> P8[Phase 8: Docs]
536 P8 --> P9[Phase 9: Verification]
537```
538
539---
540
541## Usage for Orchestrator
542
543To implement this plan with code agents:
544
545```
546Phase 1: "Implement Phase 1 from docs/purgatory-implementation-plan.md - create core purgatory data structures. Follow success criteria exactly."
547
548Phase 2: "Implement Phase 2 from docs/purgatory-implementation-plan.md - add state event logic with ref helpers. Build on Phase 1."
549
550[Continue for each phase...]
551```
552
553Each phase is independent enough for fresh context, but builds on previous phases. Always reference the plan document for complete details.
554
555---
556
557## Notes
558
559- **Do not deviate** from this plan without asking
560- **Never add integration tests** - only uncomment existing ones
561- **Always commit** before reporting phase completion
562- **Test frequently** - run cargo test after each significant change
563- **Update docs** if implementation differs from design
564- **Ask questions** if anything is unclear
565
566This plan is tracked in version control to prevent scope creep and ensure systematic implementation.