upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation/announcements-purgatory-design.md
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2026-02-23 14:49:30 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2026-02-23 14:49:30 +0000
commit4848c4029fc58f6f310a2babeae1ee82a7e41656 (patch)
treeccdfdaae41dd2907794a47bbeff562824dd3915b /docs/explanation/announcements-purgatory-design.md
parentf19b424e01fc5a682778c5e2bb194d242efd6987 (diff)
docs: update purgatory docs to reflect announcements purgatory implementation
Remove the pre-implementation planning docs (announcements-purgatory-design.md and announcements-purgatory-implementation.md) now that the feature is built. Update the three living docs to reflect what was actually implemented: - purgatory-design.md: expanded to cover all three purgatory stores (announcement, state, PR), including AnnouncementPurgatoryEntry structure, two-phase soft expiry lifecycle, expiry extension triggers, promotion flow, and updated integration points and file structure - grasp-02-proactive-sync.md: added SyncLevel enum (Full/StateOnly) to RepoSyncNeeds, documented the purgatory announcement sync timer as the registration path for purgatory announcements, updated filter building to describe build_sync_level_aware_filters() and StateOnly behaviour - grasp-02-proactive-sync-purgatory-git-data.md: expanded to cover announcement purgatory as a third entry type, added Timeline E showing soft-expiry and revival, replaced the single expiry section with separate hard-expiry (state/PR) and two-phase soft-expiry (announcements) sections with full justification for the 24-hour extended retention window
Diffstat (limited to 'docs/explanation/announcements-purgatory-design.md')
-rw-r--r--docs/explanation/announcements-purgatory-design.md254
1 files changed, 0 insertions, 254 deletions
diff --git a/docs/explanation/announcements-purgatory-design.md b/docs/explanation/announcements-purgatory-design.md
deleted file mode 100644
index 009547b..0000000
--- a/docs/explanation/announcements-purgatory-design.md
+++ /dev/null
@@ -1,254 +0,0 @@
1# Announcements Purgatory Design
2
3## Problem Statement
4
5**Primary problem:** Serving announcement events alongside empty bare git repos misleads clients into thinking we host content.
6
7When an announcement arrives, we must create the bare repo immediately (so git pushes can succeed). But if no git data ever arrives, we serve an empty repo and its announcement indefinitely. Clients see the announcement, try to clone, and get nothing. This is misleading.
8
9**Secondary problem:** Sync downloads events for repos that may never have content.
10
11Without purgatory, sync would fetch all L2/L3 events (patches, issues, etc.) for announcements that may never receive git data. This wastes bandwidth and creates orphaned events.
12
13## Solution Overview
14
15New announcements go to **purgatory** instead of being immediately accepted:
16
171. **Announcement arrives** - Create bare repo immediately, add announcement to purgatory
182. **Git data arrives** - Promote announcement from purgatory to active (now served to clients)
193. **No git data before expiry** - Delete bare repo, discard announcement (never served)
20
21This ensures we only serve announcements for repos that actually have content.
22
23## Key Design Decisions
24
25### 1. Bare Repo Created Immediately
26
27**Decision:** Create the bare git repo when announcement enters purgatory.
28
29**Why:** Git pushes may arrive at any time. Without a repo, pushes fail.
30
31**Consequence:** We allocate disk space for repos that may expire unused. Must delete repos on expiry.
32
33### 2. Git Data Triggers Promotion
34
35**Decision:** Git data arrival promotes the announcement to active status.
36
37**Why:** Git data proves the repository has content. State events alone don't prove content exists - they could reference empty repos.
38
39**Where:** Promotion happens in the git receive path after successful push/fetch with data.
40
41### 3. Replacement Announcements Skip Purgatory
42
43**Decision:** Announcements replacing an existing active announcement are accepted immediately.
44
45**Why:** The repository is already proven active with content.
46
47**How:** Check if active announcement exists for `(pubkey, identifier)` before routing to purgatory.
48
49### 4. Expiry Extension (Two Places)
50
51**Decision:** Extend purgatory announcement expiry (reset the 30-minute protocol timer) in two scenarios:
52
53| Trigger | Location | Why |
54| ---------------------------- | ------------------------------------ | ----------------------------------- |
55| State event arrives | `StatePolicy::process_state_event()` | Repo is actively receiving metadata |
56| Git auth extends state event | `src/git/auth.rs` | Repo is actively receiving git data |
57
58**Why:** Prevents premature expiry during slow sync operations or multi-step pushes. The protocol's 30-minute expiry is intended for abandoned repositories, not active ones receiving data.
59
60### 5. Authorization Must Check Purgatory Announcements
61
62**Decision:** When validating state events or git operations, check purgatory announcements in addition to the database.
63
64**Why:** State events and git pushes may arrive before git data promotes the announcement. They still need authorization from the announcement's maintainer set.
65
66**Where:** `fetch_repository_data()` and related authorization functions must query both DB and purgatory.
67
68### 6. Sync Only State Events for Purgatory Announcements
69
70**Decision:** Purgatory announcements trigger sync for state events only, not other L2/L3 events (patches, issues, PRs, etc.).
71
72**Why:** Other L2/L3 events would be rejected anyway (no promoted announcement in DB). Syncing them wastes bandwidth and creates work for announcements that may never promote.
73
74**How:** Sync uses a `SyncLevel` concept - `Full` for promoted repos, `StateOnly` for purgatory. On promotion, upgrade to `Full`.
75
76### 7. Soft Expiry Preserves Event Without Bare Repo
77
78**Decision:** When a purgatory announcement expires (30 minutes per protocol spec), delete the bare repo but retain the announcement event for an extended period (e.g., 24h).
79
80**Why the protocol specifies 30 minutes:** The grasp protocol defines a 30-minute expiry for announcement events to ensure clients don't indefinitely cache stale repository information.
81
82**Why we implement soft expiry:** The protocol's 30-minute expiry creates a sync/storage problem. Without soft expiry, we'd either:
83
84- Add expired announcements to `failed_events` and permanently reject future state events (losing potential revival when state events arrive late)
85- Re-fetch the announcement event repeatedly on every sync cycle (wasting bandwidth and creating unnecessary sync traffic)
86
87**Behavior during soft expiry:**
88
89- Bare repo is deleted (saves disk space, respects protocol expiry)
90- Announcement event retained in purgatory with `soft_expired` flag
91- Sync continues requesting state events (same as active purgatory)
92- If state event arrives: recreate bare repo, clear `soft_expired`, extend expiry
93- If announcement republished directly to us: treat as fresh arrival
94- After extended expiry: fully remove from purgatory
95
96**In summary:** Soft expiry is an implementation optimization that prevents us from constantly re-syncing announcement events or permanently blocking repositories that receive delayed state events.
97
98## Data Structure
99
100```rust
101// Key: (owner pubkey, identifier) - identifier alone is NOT unique
102announcement_purgatory: Arc<DashMap<(PublicKey, String), AnnouncementPurgatoryEntry>>
103
104pub struct AnnouncementPurgatoryEntry {
105 pub event: Event,
106 pub identifier: String,
107 pub owner: PublicKey,
108 pub repo_path: PathBuf,
109 pub relays: HashSet<String>, // For sync registration
110 pub created_at: Instant,
111 pub expires_at: Instant,
112 pub soft_expired: bool, // Bare repo deleted, event retained
113}
114```
115
116**Indexed by `(pubkey, identifier)`** because identifier is not unique across different owners. Lookups are primarily from nostr events which have pubkey and identifier readily available.
117
118## Flows
119
120### New Announcement Flow
121
122```
123Announcement arrives
124 |
125 v
126Is there an active announcement for (pubkey, identifier)?
127 |
128 +-- YES --> Accept immediately (replacement)
129 |
130 +-- NO --> Create bare repo
131 Add to purgatory
132 Return OK to client (but don't serve)
133```
134
135### Git Data Arrival Flow
136
137```
138Git push/fetch completes with data
139 |
140 v
141Is there a purgatory announcement for (pubkey, identifier)?
142 |
143 +-- YES --> Promote to active (move to database)
144 | Now served to clients
145 |
146 +-- NO --> Normal processing
147```
148
149### State Event Arrival Flow
150
151```
152State event arrives
153 |
154 v
155Is there an active announcement?
156 |
157 +-- YES --> Normal validation
158 |
159 +-- NO --> Check purgatory for announcement
160 |
161 +-- Found --> Validate against purgatory announcement
162 | Extend purgatory expiry
163 | State event goes to state purgatory
164 |
165 +-- Not found --> Reject or state purgatory
166```
167
168## Edge Cases
169
170| Scenario | Behavior |
171| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
172| Git data before announcement | Push fails (no repo exists) |
173| Announcement expires, no git data | Delete bare repo, set `soft_expired` flag, retain event for extended period |
174| Soft-expired announcement fully expires | Remove from purgatory entirely |
175| State event arrives for soft-expired announcement | Recreate bare repo, clear `soft_expired`, extend expiry |
176| State expires, announcement in purgatory | Announcement keeps its own expiry |
177| Multiple owners, same identifier | Each tracked separately by `(pubkey, identifier)` |
178| **Newer announcement replaces older (same pubkey)** | Replace purgatory entry, extend expiry, and state event expiry |
179| **Newer announcement changes services (unacceptable)** | Clear older announcement from purgatory, delete bare repo, remove state events from purgatory if exists |
180| Deletion event for purgatory announcement | Remove from purgatory, delete bare repo |
181
182## Purgatory Lifecycle
183
184An announcement progresses through purgatory states:
185
186```
187 ┌─────────────────────────────────────┐
188 │ │
189 v │
190Announcement ──> ACTIVE ──────────────────────────────────┤
191 arrives (bare repo exists) │
192 │ │
193 ├── Git data ──> PROMOTED (exit) │
194 │ │
195 ├── Deletion ──> REMOVED (exit) │
196 │ │
197 v │
198 SOFT_EXPIRED ──────────────────────────────┘
199 (bare repo deleted, ^
200 event retained) │
201 │ │
202 ├── State event arrives (revival)
203
204 └── Extended expiry ──> REMOVED (exit)
205```
206
207| Exit | Trigger | Action |
208| ------------------ | -------------------------------------------- | --------------------------------------------- |
209| **Promotion** | Git data arrives | Move to database, upgrade sync to Full |
210| **Soft expiry** | Initial timeout | Delete bare repo, retain event, continue sync |
211| **Full expiry** | Extended timeout (soft-expired) | Remove from purgatory entirely |
212| **Deletion** | Kind 5 event | Delete bare repo, remove from purgatory |
213| **Replacement** | Newer announcement (same pubkey, identifier) | Replace entry |
214| **Service change** | Newer announcement removes our service | Remove from purgatory |
215
216## Integration Points
217
218| File | Change |
219| ---------------------------------- | ---------------------------------------------------------- |
220| `src/purgatory/mod.rs` | Add `announcement_purgatory` store |
221| `src/purgatory/types.rs` | Add `AnnouncementPurgatoryEntry` |
222| `src/nostr/policy/announcement.rs` | Route new announcements to purgatory |
223| `src/git/receive.rs` | Promote on git data arrival |
224| `src/git/auth.rs` | Extend purgatory expiry when extending state event expiry |
225| `src/git/authorization.rs` | Check purgatory announcements for maintainer authorization |
226| `src/nostr/policy/state.rs` | Check purgatory for authorization |
227| `src/sync/mod.rs` | Add `SyncLevel` to `RepoSyncNeeds` |
228| `src/sync/filters.rs` | Respect sync level when building filters |
229| `src/sync/self_subscriber.rs` | Register purgatory announcements with `StateOnly` level |
230
231See [announcements-purgatory-implementation.md](./announcements-purgatory-implementation.md) for detailed implementation notes.
232
233## Testing
234
235- Announcement to purgatory, git data promotes it
236- Announcement soft-expires without git data (repo deleted, event retained)
237- State event revives soft-expired announcement (repo recreated)
238- Soft-expired announcement fully expires after extended period
239- State event extends purgatory expiry
240- Git auth extends purgatory expiry
241- Newer announcement replaces older in purgatory
242- Service change clears purgatory entry
243- `(pubkey, identifier)` indexing with multiple owners
244- Sync requests only state events for purgatory announcements
245- Sync upgrades to full on promotion
246
247## Risks
248
249| Risk | Mitigation |
250| ------------------------------------ | ------------------------------------------------------ |
251| Disk exhaustion from purgatory repos | Short expiry, soft expiry deletes repo early |
252| Race between promotion and expiry | Atomic operations |
253| Sync re-fetching expired events | Soft expiry retains event; no need for `failed_events` |
254| Filter explosion from many purgatory | Existing consolidation handles this (threshold at 70) |