upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation/purgatory-design.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/explanation/purgatory-design.md')
-rw-r--r--docs/explanation/purgatory-design.md1230
1 files changed, 506 insertions, 724 deletions
diff --git a/docs/explanation/purgatory-design.md b/docs/explanation/purgatory-design.md
index 5ee4e06..b984745 100644
--- a/docs/explanation/purgatory-design.md
+++ b/docs/explanation/purgatory-design.md
@@ -1,1013 +1,795 @@
1# Purgatory Implementation Design 1# Purgatory: In-Memory Holding Area for Events Awaiting Git Data
2 2
3**Status**: ✅ Implemented (2025-12-23) 3**Status**: ✅ Implemented
4**Implementation**: Phases 1-7 complete 4**Implementation**: [`src/purgatory/`](../../src/purgatory/)
5**Source Code**: [`src/purgatory/`](../../src/purgatory/)
6**Related**: [`docs/explanation/architecture.md`](architecture.md) - System architecture overview 5**Related**: [`docs/explanation/architecture.md`](architecture.md) - System architecture overview
7 6
7---
8
8## Overview 9## Overview
9 10
10Purgatory is an in-memory holding area for nostr events that depend on git data that hasn't arrived yet, **and** for git data that arrived before its corresponding nostr event. Events/placeholders are held until the other half arrives, at which point they are processed and saved to the database. 11Purgatory is an in-memory holding area that solves the **"which arrives first?"** problem in GRASP. Either nostr events or git pushes can arrive in any order:
12
13- **Event first**: Event waits in purgatory until git data arrives
14- **Git first**: Placeholder waits in purgatory until event arrives
15
16When both halves arrive, they are processed together and saved to the database.
11 17
12**Spec Reference**: [GRASP-01 Purgatory Section](../grasp/01.md:20-22) 18**Spec Reference**: [GRASP-01 Purgatory Section](https://github.com/DanConwayDev/grasp/blob/main/01.md#purgatory)
13 19
14> Accepted repo state announcements, PRs and PR Updates SHOULD be accepted with message "purgatory: won't be served until git data arrives" and kept in purgatory (not served) until the related git data arrives and otherwise discarded after 30 minutes. 20> Accepted repo state announcements, PRs and PR Updates SHOULD be accepted with message "purgatory: won't be served until git data arrives" and kept in purgatory (not served) until the related git data arrives and otherwise discarded after 30 minutes.
15 21
22---
23
16## Key Design Principles 24## Key Design Principles
17 25
18### 1. Separate Storage for State vs PR Events 26### 1. In-Memory Only
27
28Purgatory data is **not persisted** to disk. On restart, all purgatory entries are lost. This is acceptable because:
29
30- Events are still on other relays (can be re-submitted)
31- Git data can be re-pushed
32- 30-minute expiry means data is transient anyway
19 33
20State events (kind 30618) and PR events (kind 1617/1618) have fundamentally different matching and authorization patterns. They are stored in **separate purgatory stores** with different indices: 34### 2. Separate Storage for State vs PR Events
21 35
22- **State Events**: Indexed by `identifier` (d tag), matched via ref comparison 36State events (kind 30618) and PR events (kind 1617/1618) have fundamentally different matching patterns:
23- **PR Events**: Indexed by `event_id`, matched directly via `refs/nostr/<event-id>`
24 37
25### 2. Late Binding of Refs (State Events) 38| Event Type | Index | Matching Strategy |
39|------------|-------|-------------------|
40| **State Events** | `identifier` (d tag) | Compare refs at push time |
41| **PR Events** | `event_id` (hex string) | Direct match via `refs/nostr/<event-id>` |
26 42
27**Do NOT extract refs at event arrival time.** Extract and match refs at git push time. 43They use **separate DashMap stores** for efficient concurrent access.
28 44
29**Why?** Multiple state events might be in purgatory with different target states. An older state event's git data might arrive after a newer one is received. By waiting until push time, we: 45### 3. Late Binding for State Events
30 46
31- Compare the pushed refs against each purgatory state event's expected state 47**Critical:** Do NOT extract refs from state events at arrival time. Extract and match refs **at git push time**.
48
49**Why?** Multiple state events might be in purgatory with different target states. An older state event's git data might arrive after a newer one is received. By waiting until push time:
50
51- Compare pushed refs against each purgatory state event's expected state
32- Handle out-of-order git data arrival correctly 52- Handle out-of-order git data arrival correctly
33- Only release events when their specific target state is achieved 53- Only release events when their specific target state is achieved
34 54
35### 3. Bidirectional Waiting (PR Events) 55See [`src/purgatory/helpers.rs:can_satisfy_state`](../../src/purgatory/helpers.rs) for implementation.
56
57### 4. Bidirectional Waiting for PR Events
36 58
37For PR events, **either side can arrive first**: 59For PR events, **either side can arrive first**:
38 60
39- **Event first**: PR event waits in purgatory for git push to `refs/nostr/<event-id>` 61| Scenario | What Happens |
40- **Git first**: Push creates placeholder, waits for PR event to arrive 62|----------|--------------|
63| **Event first** | PR event waits in purgatory for git push to `refs/nostr/<event-id>` |
64| **Git first** | Push creates placeholder entry waiting for PR event |
41 65
42### 4. Ref Pairs, Not Just Commits 66Placeholders are identified by `PrPurgatoryEntry.event == None`.
43 67
44State events define a target state: specific refs pointing to specific objects (commits or annotated tags). We match **ref name + object SHA** pairs, not just raw commits. 68### 5. Authorization During Push (Not After)
45 69
46### 5. No Separate PurgatoryState Tracking 70**Critical for avoiding deadlock:** Authorization checks **both database and purgatory** during push validation.
47 71
48All entries use a single 30-minute expiry timer. When git push activity begins, we simply ensure at least 15 minutes remain on the timer (extend if needed). This eliminates complexity of tracking Secured vs Pending states. 72Without this, we'd have a deadlock:
731. State event arrives → No git data → Goes to **purgatory** (not database)
742. Git push arrives → Authorization checks **database only** → No state found → **REJECTED** ❌
49 75
50## Event Lifecycle 76With purgatory checking during authorization:
771. State event arrives → No git data → Goes to purgatory
782. Git push arrives → Checks **database + purgatory** → State found → **AUTHORIZED** ✅
793. After push succeeds → Save event to database → Remove from purgatory
51 80
52```mermaid 81See [`src/git/authorization.rs:51-162`](../../src/git/authorization.rs) for implementation.
53stateDiagram-v2 82
54 [*] --> Waiting: Event or git data arrives 83---
55 Waiting --> Processing: Other half arrives
56 Waiting --> Discarded: 30 min expiry
57 Processing --> Released: Match verified + authorized
58 Processing --> Rejected: Mismatch or unauthorized
59 Released --> [*]: Event saved to database
60 Rejected --> Waiting: Return to waiting - different match expected
61 Discarded --> [*]: Entry dropped
62```
63 84
64## Data Structures 85## Data Structures
65 86
66### RefPair - A single ref target 87### Core Types
67 88
68```rust 89```rust
69/// A reference name and its target object 90/// A reference name and its target object
70#[derive(Debug, Clone, Hash, Eq, PartialEq)] 91#[derive(Debug, Clone, Hash, Eq, PartialEq)]
71pub struct RefPair { 92pub struct RefPair {
72 /// Full ref name, e.g., "refs/heads/main" or "refs/tags/v1.0" 93 pub ref_name: String, // e.g., "refs/heads/main"
94 pub object_sha: String, // commit or annotated tag SHA
95}
96
97/// A ref update in a git push
98#[derive(Debug, Clone)]
99pub struct RefUpdate {
100 pub old_oid: String,
101 pub new_oid: String,
73 pub ref_name: String, 102 pub ref_name: String,
74 /// Target object SHA (commit or annotated tag)
75 pub object_sha: String,
76} 103}
77``` 104```
78 105
79### StatePurgatoryEntry 106### State Purgatory Entry
80 107
81```rust 108```rust
82pub struct StatePurgatoryEntry { 109pub struct StatePurgatoryEntry {
83 /// The nostr state event (kind 30618) awaiting git data 110 /// The nostr state event (kind 30618) awaiting git data
84 pub event: Event, 111 pub event: Event,
85 112
86 /// The repository identifier from the event's 'd' tag 113 /// Repository identifier from 'd' tag
87 pub identifier: String, 114 pub identifier: String,
88 115
89 /// Event author pubkey 116 /// Event author pubkey
90 pub author: PublicKey, 117 pub author: PublicKey,
91 118
92 /// When this entry was added to purgatory 119 /// When added to purgatory
93 pub created_at: Instant, 120 pub created_at: Instant,
94 121
95 /// Expiry deadline (30 min from creation, may be extended) 122 /// Expiry deadline (30 min from creation, may be extended)
96 pub expires_at: Instant, 123 pub expires_at: Instant,
97} 124}
98``` 125```
99 126
100### PrPurgatoryEntry 127**Note:** Refs are NOT extracted at creation time. They're extracted at push time for late binding.
128
129### PR Purgatory Entry
101 130
102```rust 131```rust
103pub struct PrPurgatoryEntry { 132pub struct PrPurgatoryEntry {
104 /// The nostr PR event, if received (None = git data arrived first) 133 /// The nostr PR event, if received (None = git data arrived first)
105 pub event: Option<Event>, 134 pub event: Option<Event>,
106 135
107 /// The expected commit SHA from 'c' tag (if event exists) 136 /// Expected commit SHA from 'c' tag (if event exists)
108 /// or the actual commit pushed (if git arrived first) 137 /// or actual commit pushed (if git arrived first)
109 pub commit: String, 138 pub commit: String,
110 139
111 /// When this entry was added to purgatory 140 /// When added to purgatory
112 pub created_at: Instant, 141 pub created_at: Instant,
113 142
114 /// Expiry deadline (30 min from creation, may be extended) 143 /// Expiry deadline (30 min from creation)
115 pub expires_at: Instant, 144 pub expires_at: Instant,
116} 145}
117``` 146```
118 147
119**Note:** `PrPurgatoryEntry.event` being `None` indicates the "git data first" scenario - we have a placeholder waiting for the PR event. 148**Key:** `event: None` indicates a placeholder (git-data-first scenario).
120 149
121### Purgatory Stores 150### Purgatory Stores
122 151
123```rust 152```rust
124pub struct Purgatory { 153pub struct Purgatory {
125 /// State events indexed by identifier 154 /// State events indexed by identifier (d tag)
126 state_events: DashMap<String, Vec<StatePurgatoryEntry>>, 155 /// Multiple state events per identifier allowed (different authors)
127 156 state_events: Arc<DashMap<String, Vec<StatePurgatoryEntry>>>,
157
128 /// PR events indexed by event_id (hex string) 158 /// PR events indexed by event_id (hex string)
129 pr_events: DashMap<String, PrPurgatoryEntry>, 159 /// Single entry per event ID
160 pr_events: Arc<DashMap<String, PrPurgatoryEntry>>,
161
162 /// Sync queue for background git data fetching
163 sync_queue: Arc<DashMap<String, SyncQueueEntry>>,
164
165 _git_data_path: PathBuf,
130} 166}
131``` 167```
132 168
169---
170
133## Event Flows 171## Event Flows
134 172
135### State Event (Kind 30618) Arrival 173### State Event Arrival (Kind 30618)
136 174
137```mermaid 175```mermaid
138sequenceDiagram 176sequenceDiagram
139 participant Client 177 participant Client
140 participant WritePolicy 178 participant WritePolicy
141 participant Purgatory 179 participant Purgatory
180 participant Database
142 participant GitRepos 181 participant GitRepos
143 182
144 Client->>WritePolicy: EVENT kind:30618 183 Client->>WritePolicy: EVENT kind:30618
145 WritePolicy->>WritePolicy: Validate structure 184 WritePolicy->>WritePolicy: Validate structure
146 WritePolicy->>WritePolicy: Parse identifier and author 185 WritePolicy->>WritePolicy: Parse identifier and author
147 186
148 Note right of WritePolicy: Check if we already have git data 187 Note right of WritePolicy: Check if git data exists
149 WritePolicy->>GitRepos: Check if any authorized repo has matching refs 188 WritePolicy->>GitRepos: Check if any authorized repo has matching refs
150 189
151 alt Git data already exists 190 alt Git data exists
152 GitRepos-->>WritePolicy: Refs match in repo X 191 GitRepos-->>WritePolicy: Refs match in repo X
153 WritePolicy->>WritePolicy: Process immediately 192 WritePolicy->>Database: Save event
154 WritePolicy->>Client: OK true - event saved 193 WritePolicy->>Client: OK true - event saved
155 else Git data not available yet 194 else Git data not available yet
156 WritePolicy->>Purgatory: add_state - event, identifier 195 WritePolicy->>Purgatory: add_state(event, identifier, author)
157 Purgatory->>Purgatory: Add entry indexed by identifier 196 Purgatory->>Purgatory: Store in state_events[identifier]
158 WritePolicy->>Client: OK true purgatory: will not be served until git data arrives 197 Purgatory->>Purgatory: Enqueue for sync (3min delay)
198 WritePolicy->>Client: OK true "purgatory: awaiting git data"
159 end 199 end
160``` 200```
161 201
162### PR Event (Kind 1617/1618) Arrival 202### PR Event Arrival (Kind 1617/1618)
163 203
164```mermaid 204```mermaid
165sequenceDiagram 205sequenceDiagram
166 participant Client 206 participant Client
167 participant WritePolicy 207 participant WritePolicy
168 participant Purgatory 208 participant Purgatory
209 participant Database
169 participant GitRepos 210 participant GitRepos
170 211
171 Client->>WritePolicy: EVENT kind:1617 212 Client->>WritePolicy: EVENT kind:1617/1618
172 WritePolicy->>WritePolicy: Validate structure 213 WritePolicy->>WritePolicy: Extract event_id and commit from 'c' tag
173 WritePolicy->>WritePolicy: Extract event_id and commit from c tag
174 214
175 Note right of WritePolicy: Check if git data already exists 215 Note right of WritePolicy: Check if git data exists
176 WritePolicy->>GitRepos: Check refs/nostr/event-id in authorized repos 216 WritePolicy->>GitRepos: Check refs/nostr/<event-id> in repos
177 217
178 alt Git data already exists 218 alt Git data exists in database
179 GitRepos-->>WritePolicy: Found matching commit 219 GitRepos-->>WritePolicy: Found with matching commit
180 WritePolicy->>WritePolicy: Process immediately 220 WritePolicy->>Database: Save event
181 WritePolicy->>Client: OK true - event saved 221 WritePolicy->>Client: OK true - event saved
182 else Git data arrived first - placeholder exists 222 else Placeholder exists in purgatory
183 WritePolicy->>Purgatory: find_pr_placeholder - event_id 223 WritePolicy->>Purgatory: find_pr_placeholder(event_id)
184 alt Placeholder found with matching commit 224 alt Placeholder has matching commit
185 Purgatory-->>WritePolicy: Placeholder entry 225 Purgatory-->>WritePolicy: Placeholder entry
186 WritePolicy->>WritePolicy: Process - save event 226 WritePolicy->>Database: Save event
187 WritePolicy->>Purgatory: remove - event_id 227 WritePolicy->>Purgatory: remove_pr(event_id)
188 WritePolicy->>Client: OK true - event saved 228 WritePolicy->>Client: OK true - event saved
189 else Placeholder found with different commit 229 else Placeholder has different commit
190 WritePolicy->>Client: OK false - commit mismatch 230 WritePolicy->>Client: OK false - commit mismatch
191 else No placeholder
192 WritePolicy->>Purgatory: add_pr - event, commit
193 WritePolicy->>Client: OK true purgatory: will not be served until git data arrives
194 end 231 end
232 else No git data yet
233 WritePolicy->>Purgatory: add_pr(event, event_id, commit)
234 Purgatory->>Purgatory: Store in pr_events[event_id]
235 Purgatory->>Purgatory: Enqueue for sync (3min delay)
236 WritePolicy->>Client: OK true "purgatory: awaiting git data"
195 end 237 end
196``` 238```
197 239
198### Git Push - State Event Matching 240### Git Push - State Refs
199 241
200When a push arrives to normal refs (branches/tags), we check for matching state events: 242**Critical:** Authorization happens BEFORE git-receive-pack execution, checking both database and purgatory.
201
202**Key Rule**: For a given `pubkey/identifier` repo, there can only be **one authoritative state event** - the one with the largest `created_at` among all authorized maintainers. State events in purgatory are only processed if they aren't superseded by existing state events on the relay.
203 243
204```mermaid 244```mermaid
205sequenceDiagram 245sequenceDiagram
206 participant GitClient 246 participant GitClient
207 participant GitHandler 247 participant GitHandler
248 participant Authorization
208 participant Purgatory 249 participant Purgatory
209 participant Database 250 participant Database
210 participant GitRepo 251 participant GitProcess
211 252
212 GitClient->>GitHandler: POST /npub/identifier/git-receive-pack 253 GitClient->>GitHandler: POST /npub/id/git-receive-pack
213 GitHandler->>GitHandler: Parse pushed refs into RefPairs 254 GitHandler->>Authorization: authorize_push(body, purgatory, database)
214 255
215 Note over GitHandler: Look up state events by identifier 256 Note over Authorization: Parse pushed refs from pkt-line format
216 GitHandler->>Purgatory: find_matching_states - identifier, pushed_refs 257 Authorization->>Authorization: parse_pushed_refs(body)
217 258 Authorization->>Authorization: Separate state refs from refs/nostr/*
218 loop For each state event in purgatory[identifier] 259
219 Purgatory->>Purgatory: Parse state event refs 260 Note over Authorization: Check database for state events
220 Purgatory->>Purgatory: Check: pushed_refs covers event refs that differ from local 261 Authorization->>Database: Query state events for identifier
221 Purgatory->>Purgatory: Check: event refs not in push already exist locally 262
222 alt All event refs can be satisfied 263 alt State found in database
223 Purgatory->>Purgatory: Add to candidates 264 Database-->>Authorization: State event
224 end 265 Authorization->>Authorization: Validate refs match
225 end 266 Authorization-->>GitHandler: Authorized (from_purgatory=false)
226 267 else No state in database - check purgatory
227 Purgatory-->>GitHandler: Vec of candidate state events 268 Authorization->>Purgatory: find_matching_states(identifier, pushed_refs, local_refs)
228 269
229 Note over GitHandler: Get all state events and announcements for identifier 270 alt Matching state in purgatory
230 GitHandler->>Database: Get announcements for identifier 271 Purgatory-->>Authorization: State event(s)
231 GitHandler->>Database: Get all state events for identifier from relay 272 Authorization->>Authorization: Filter to authorized authors
232 GitHandler->>GitHandler: collect_authorized_maintainers from announcements 273 Authorization->>Authorization: Find latest state
233 274 Authorization->>Authorization: Validate refs match
234 Note over GitHandler: Find authoritative state event 275 Authorization->>Purgatory: extend_expiry(15min)
235 loop For each candidate from purgatory 276 Authorization-->>GitHandler: Authorized (from_purgatory=true)
236 GitHandler->>GitHandler: Check if event.author is authorized for target repo 277 else No matching state anywhere
237 alt Author authorized 278 Authorization-->>GitHandler: Rejected - no authorized state
238 GitHandler->>GitHandler: Check relay state events for this identifier
239 GitHandler->>GitHandler: Compare created_at with all authorized state events
240 alt This is the largest created_at among authorized
241 GitHandler->>GitHandler: Verify OIDs for unpushed refs exist locally
242 alt All OIDs available
243 GitHandler->>GitHandler: Set as approved state event
244 end
245 else Superseded by existing state on relay
246 GitHandler->>Purgatory: Remove superseded event
247 end
248 end 279 end
249 end 280 end
250 281
251 Note over GitHandler: Only ONE approved state event per repo 282 alt Authorized
252 alt Approved state event exists 283 GitHandler->>GitProcess: Execute git-receive-pack
253 GitHandler->>Purgatory: extend_expiry - event_id - ensure 15 min 284
254 GitHandler->>GitRepo: Execute git-receive-pack 285 alt Push succeeds AND from_purgatory=true
255
256 alt Push successful
257 GitHandler->>Database: Save state event 286 GitHandler->>Database: Save state event
258 GitHandler->>GitRepo: Align ALL refs to state - including unpushed 287 GitHandler->>Purgatory: remove_state_event(identifier, event_id)
259 GitHandler->>GitHandler: Sync to other authorized maintainer repos
260 GitHandler->>Purgatory: remove - event_id
261 GitHandler->>GitClient: Push accepted 288 GitHandler->>GitClient: Push accepted
262 else Push failed 289 else Push succeeds AND from_purgatory=false
263 GitHandler->>GitClient: Push rejected 290 GitHandler->>GitClient: Push accepted
291 else Push fails
292 GitHandler->>GitClient: Push rejected - git error
264 end 293 end
265 else No approved state event 294 else Rejected
266 GitHandler->>GitClient: Push rejected - no authorized state 295 GitHandler->>GitClient: Push rejected - not authorized
267 end 296 end
268``` 297```
269 298
270### Git Push - PR Event (refs/nostr/event-id) 299### Git Push - PR Refs (refs/nostr/event-id)
271 300
272```mermaid 301```mermaid
273sequenceDiagram 302sequenceDiagram
274 participant GitClient 303 participant GitClient
275 participant GitHandler 304 participant GitHandler
276 participant Database
277 participant Purgatory 305 participant Purgatory
278 participant GitRepo 306 participant Database
279 307 participant GitProcess
280 GitClient->>GitHandler: POST /npub/identifier/git-receive-pack refs/nostr/abc123 308
281 GitHandler->>GitHandler: Extract event_id from ref 309 GitClient->>GitHandler: POST /npub/id/git-receive-pack refs/nostr/abc123
282 GitHandler->>GitHandler: Extract pushed commit SHA 310 GitHandler->>GitHandler: Extract event_id and commit from push
283 311
284 Note over GitHandler: First check relay database 312 Note over GitHandler: Check database first
285 GitHandler->>Database: Query for PR event with event_id 313 GitHandler->>Database: Query PR event with event_id
286 314
287 alt Event exists in database 315 alt Event exists in database
288 Database-->>GitHandler: PR event found 316 Database-->>GitHandler: PR event
289 GitHandler->>GitHandler: Compare commit tags 317 GitHandler->>GitHandler: Compare commit tags
290 alt Commit matches 318 alt Commit matches
291 GitHandler->>GitRepo: Execute push 319 GitHandler->>GitProcess: Execute push
292 GitHandler->>GitClient: Push accepted 320 GitHandler->>GitClient: Push accepted
293 else Commit mismatch 321 else Commit mismatch
294 GitHandler->>GitClient: Push rejected - commit mismatch with existing event 322 GitHandler->>GitClient: Push rejected - commit mismatch
295 end 323 end
296 else Event not in database 324 else Event not in database - check purgatory
297 GitHandler->>Purgatory: find_pr - event_id 325 GitHandler->>Purgatory: find_pr(event_id)
298 326
299 alt PR event in purgatory 327 alt PR event in purgatory
300 Purgatory-->>GitHandler: PR entry with event 328 Purgatory-->>GitHandler: PR entry with event
301 GitHandler->>GitHandler: Compare commit tags 329 GitHandler->>GitHandler: Compare commit tags
302 alt Commit matches 330 alt Commit matches
303 GitHandler->>GitRepo: Execute push 331 GitHandler->>GitProcess: Execute push
304 GitHandler->>Database: Save PR event 332 GitHandler->>Database: Save PR event
305 GitHandler->>GitHandler: Sync to other authorized repos 333 GitHandler->>Purgatory: remove_pr(event_id)
306 GitHandler->>Purgatory: remove - event_id
307 GitHandler->>GitClient: Push accepted 334 GitHandler->>GitClient: Push accepted
308 else Commit mismatch 335 else Commit mismatch
309 GitHandler->>GitClient: Push rejected - commit mismatch 336 GitHandler->>GitClient: Push rejected - commit mismatch
310 end 337 end
311 else No PR event anywhere 338 else No PR event anywhere (git-data-first)
312 Note over GitHandler: Git data first scenario 339 GitHandler->>GitProcess: Execute push - accept any commit
313 GitHandler->>GitRepo: Execute push - accept any data 340 GitHandler->>Purgatory: add_pr_placeholder(event_id, commit)
314 GitHandler->>Purgatory: add_pr_placeholder - event_id, commit
315 GitHandler->>GitClient: Push accepted - awaiting PR event 341 GitHandler->>GitClient: Push accepted - awaiting PR event
316 end 342 end
317 end 343 end
318``` 344```
319 345
320### Sync to Other Maintainer Repos 346---
321 347
322After successfully processing a state event or PR, we sync to other authorized repos: 348## Background Sync
323 349
324```mermaid 350Purgatory includes a background sync system that fetches git data from remote servers when events arrive before git data.
325sequenceDiagram
326 participant Handler
327 participant Database
328 participant Authorization
329 participant GitRepos
330 351
331 Handler->>Database: Get all announcements for identifier 352### Sync Architecture
332 Handler->>Authorization: collect_authorized_maintainers - announcements
333 Authorization-->>Handler: Map of owner -> authorized maintainers
334 353
335 loop For each owner in map 354```
336 alt Event author in owner's authorized set 355┌─────────────────────────────────────────────────────┐
337 Note right of Handler: This owner's repo should have this state/PR 356│ Sync Loop (1s) │
357│ - Checks sync_queue for ready identifiers │
358│ - Spawns tasks for each ready identifier │
359└─────────────────────────────────────────────────────┘
360 │
361 ▼
362┌─────────────────────────────────────────────────────┐
363│ sync_identifier(identifier) │
364│ 1. Try all non-throttled URLs sequentially │
365│ 2. Check if complete after each fetch │
366│ 3. Enqueue with throttled domains if incomplete │
367└─────────────────────────────────────────────────────┘
368 │
369 ▼
370┌─────────────────────────────────────────────────────┐
371│ sync_identifier_from_url(identifier, url) │
372│ 1. Collect needed OIDs from purgatory events │
373│ 2. Fetch OIDs from remote URL │
374│ 3. Process newly available git data │
375└─────────────────────────────────────────────────────┘
376 │
377 ▼
378┌─────────────────────────────────────────────────────┐
379│ process_newly_available_git_data(repo, oids) │
380│ 1. Find satisfiable state events in purgatory │
381│ 2. Find satisfiable PR events in purgatory │
382│ 3. Save events to database │
383│ 4. Sync git data to other owner repos │
384│ 5. Remove from purgatory │
385└─────────────────────────────────────────────────────┘
386```
338 387
339 alt State event 388### Sync Queue Entry
340 Handler->>GitRepos: Align owner's repo to state event refs 389
341 else PR event 390```rust
342 Handler->>GitRepos: Ensure refs/nostr/event-id exists 391pub struct SyncQueueEntry {
343 end 392 /// When to attempt next sync
344 end 393 pub next_attempt: Instant,
345 end 394
395 /// Number of sync attempts made
396 pub attempt_count: u32,
397
398 /// Whether a sync task is currently running
399 pub in_progress: bool,
400}
346``` 401```
347 402
348### Background Cleanup 403**Backoff strategy:**
404- First attempt: 20 seconds
405- Second attempt: 2 minutes
406- Subsequent attempts: 2 minutes
349 407
350```mermaid 408### Sync Delays
351sequenceDiagram
352 participant Timer
353 participant Purgatory
354 409
355 loop Every 60 seconds 410| Scenario | Delay | Reason |
356 Timer->>Purgatory: cleanup 411|----------|-------|--------|
412| User-submitted event | 3 minutes | Give time for git push to arrive |
413| Sync-triggered event | 500ms | Batch burst arrivals from negentropy |
357 414
358 Note over Purgatory: Clean state events 415### Domain Throttling
359 loop For each identifier in state_events
360 loop For each entry
361 alt now > expires_at
362 Purgatory->>Purgatory: Remove entry
363 end
364 end
365 end
366 416
367 Note over Purgatory: Clean PR events 417```rust
368 loop For each event_id in pr_events 418pub struct ThrottleManager {
369 alt now > entry.expires_at 419 /// Max requests per domain per minute
370 Purgatory->>Purgatory: Remove entry 420 max_requests_per_minute: usize,
371 end 421
372 end 422 /// Tracking window duration
373 end 423 window_duration: Duration,
424
425 /// Per-domain throttle state
426 domains: DashMap<String, DomainThrottle>,
427}
374``` 428```
375 429
376## API Methods 430**Rate limiting:**
431- Default: 5 requests per domain per 30 seconds
432- Tracks request timestamps in a sliding window
433- Queues identifiers when domain is throttled
434- Processes queue when capacity frees up
435
436See [`src/purgatory/sync/throttle.rs`](../../src/purgatory/sync/throttle.rs) for implementation.
437
438---
439
440## Purgatory API
377 441
378### Purgatory 442### Adding Entries
379 443
380```rust 444```rust
381impl Purgatory { 445impl Purgatory {
382 /// Create a new empty purgatory 446 /// Add a state event to purgatory
383 pub fn new() -> Self; 447 /// Automatically enqueues for sync with 3min delay
384 448 pub fn add_state(&self, event: Event, identifier: String, author: PublicKey);
385 // ==================== State Events ==================== 449
450 /// Add a PR event to purgatory
451 /// Automatically enqueues for sync with 3min delay
452 pub fn add_pr(&self, event: Event, event_id: String, commit: String);
453
454 /// Add a PR placeholder (git-data-first scenario)
455 pub fn add_pr_placeholder(&self, event_id: String, commit: String);
456}
457```
386 458
387 /// Add a state event (kind 30618) to purgatory 459### Finding Entries
388 /// Returns purgatory message for client response
389 pub fn add_state(&self, event: Event, identifier: String) -> String;
390 460
391 /// Find state events that could be satisfied by pushed refs 461```rust
392 /// Returns events where: 462impl Purgatory {
393 /// - All refs in event are either in pushed_refs OR already exist locally 463 /// Find state events waiting for an identifier
394 /// - At least one ref in event is in pushed_refs (something to update) 464 pub fn find_state(&self, identifier: &str) -> Vec<StatePurgatoryEntry>;
465
466 /// Find state events that match pushed refs (late binding)
395 pub fn find_matching_states( 467 pub fn find_matching_states(
396 &self, 468 &self,
397 identifier: &str, 469 identifier: &str,
398 pushed_refs: &[RefPair], 470 pushed_updates: &[RefUpdate],
399 local_refs: &HashMap<String, String>, 471 local_refs: &HashMap<String, String>,
400 ) -> Vec<Event>; 472 ) -> Vec<Event>;
473
474 /// Find a PR entry by event ID
475 pub fn find_pr(&self, event_id: &str) -> Option<PrPurgatoryEntry>;
476
477 /// Find a PR placeholder specifically (git-data-first)
478 pub fn find_pr_placeholder(&self, event_id: &str) -> Option<String>;
479}
480```
401 481
402 /// Extend expiry for entries about to be processed 482### Removing Entries
403 /// Ensures at least `duration` remaining on timer
404 pub fn extend_expiry(&self, event_ids: &[EventId], duration: Duration);
405
406 /// Remove state event after successful processing
407 pub fn remove_state(&self, event_id: &EventId);
408
409 // ==================== PR Events ====================
410
411 /// Add a PR event (kind 1617/1618) to purgatory
412 /// Returns purgatory message for client response
413 pub fn add_pr(&self, event: Event, commit: String) -> String;
414
415 /// Add a placeholder for git-data-first scenario
416 /// Called when push to refs/nostr/<event-id> arrives before the PR event
417 pub fn add_pr_placeholder(&self, event_id: EventId, commit: String);
418
419 /// Find PR entry by event ID
420 /// Returns the entry if found (may or may not have event)
421 pub fn find_pr(&self, event_id: &EventId) -> Option<PrPurgatoryEntry>;
422
423 /// Find PR placeholder (git-data-first entry without event)
424 pub fn find_pr_placeholder(&self, event_id: &EventId) -> Option<PrPurgatoryEntry>;
425 483
426 /// Remove PR entry after successful processing 484```rust
427 pub fn remove_pr(&self, event_id: &EventId); 485impl Purgatory {
486 /// Remove all state events for an identifier
487 pub fn remove_state(&self, identifier: &str);
488
489 /// Remove a specific state event by event ID
490 pub fn remove_state_event(&self, identifier: &str, event_id: &EventId);
491
492 /// Remove a PR entry
493 pub fn remove_pr(&self, event_id: &str);
494}
495```
428 496
429 // ==================== Maintenance ==================== 497### Maintenance
430 498
431 /// Remove expired entries (30 min) 499```rust
432 /// Returns count of removed entries 500impl Purgatory {
433 pub fn cleanup(&self) -> usize; 501 /// Remove expired entries (called every 60 seconds)
502 /// Returns (state_removed, pr_removed)
503 pub fn cleanup(&self) -> (usize, usize);
504
505 /// Extend expiry for entries about to be processed
506 /// Ensures at least `duration` remaining
507 pub fn extend_expiry(&self, identifier: &str, event_ids: &[EventId], duration: Duration);
508
509 /// Get current counts for metrics
510 pub fn count(&self) -> (usize, usize);
511}
512```
434 513
435 /// Get counts for metrics/debugging 514### Sync Queue Management
436 pub fn state_event_count(&self) -> usize;
437 pub fn pr_event_count(&self) -> usize;
438 pub fn pr_placeholder_count(&self) -> usize;
439 515
440 /// Check if an event is in purgatory (either store) 516```rust
441 pub fn contains(&self, event_id: &EventId) -> bool; 517impl Purgatory {
518 /// Enqueue identifier for sync with custom delay
519 pub fn enqueue_sync(&self, identifier: &str, delay: Duration);
520
521 /// Enqueue with default delay (3 minutes)
522 pub fn enqueue_sync_default(&self, identifier: &str);
523
524 /// Enqueue with immediate delay (500ms)
525 pub fn enqueue_sync_immediate(&self, identifier: &str);
526
527 /// Check if identifier has pending events
528 pub fn has_pending_events(&self, identifier: &str) -> bool;
529
530 /// Remove identifier from sync queue
531 pub fn remove_from_sync_queue(&self, identifier: &str);
442} 532}
443``` 533```
444 534
445### Helper: Extract and Match Refs 535---
536
537## Helper Functions
538
539### State Event Matching
446 540
447```rust 541```rust
448/// Extract ref pairs from a state event 542/// Extract ref pairs from a state event
449pub fn extract_refs_from_state(event: &Event) -> Vec<RefPair> { 543pub fn extract_refs_from_state(event: &Event) -> Vec<RefPair>;
450 // Parse refs/heads/* and refs/tags/* tags
451 // Return vec of RefPair { ref_name, object_sha }
452}
453 544
454/// Check if a state event can be satisfied by a push 545/// Check if a state event can be satisfied by a push
455///
456/// Returns true if: 546/// Returns true if:
457/// - Every ref in state_refs is either in pushed_refs (matching SHA) OR in local_refs (matching SHA) 547/// - Every ref in state is either in pushed_refs OR in local_refs
458/// - At least one ref in state_refs is actually being changed by the push 548/// - At least one ref in state is being changed by the push
459pub fn can_satisfy_state( 549pub fn can_satisfy_state(
460 state_refs: &[RefPair], 550 state_refs: &[RefPair],
461 pushed_refs: &[RefPair], 551 pushed_refs: &[RefPair],
462 local_refs: &HashMap<String, String>, 552 local_refs: &HashMap<String, String>,
463) -> bool; 553) -> bool;
464 554
465/// Get refs from state event that aren't being pushed but need updating 555/// Check if a state event can be applied to a repository
556/// Returns true if all required OIDs exist in the repo
557pub fn can_apply_state(
558 event: &Event,
559 repo_path: &Path,
560) -> Result<bool>;
561
562/// Get refs from state that aren't being pushed
466pub fn get_unpushed_refs( 563pub fn get_unpushed_refs(
467 state_refs: &[RefPair], 564 state_refs: &[RefPair],
468 pushed_refs: &[RefPair], 565 pushed_refs: &[RefPair],
469) -> Vec<RefPair>; 566) -> Vec<RefPair>;
470
471/// Verify all OIDs from refs exist in the local git repo
472pub fn verify_oids_exist(
473 repo_path: &Path,
474 refs: &[RefPair],
475) -> Result<bool, git::Error>;
476``` 567```
477 568
478## Integration Points 569See [`src/purgatory/helpers.rs`](../../src/purgatory/helpers.rs) for implementation.
479
480### 1. Nip34WritePolicy Changes
481 570
482```rust 571---
483pub struct Nip34WritePolicy {
484 ctx: PolicyContext,
485 purgatory: Arc<Purgatory>, // Shared with git handlers
486 // ... existing fields
487}
488```
489
490### 2. handle_state Changes
491 572
492On state event arrival: 573## Integration Points
493 574
494**Key Rules**: 575### 1. Event Policy (Nip34WritePolicy)
495 576
4961. Reject if we already have a state event from this author for this identifier with a larger `created_at` date (outdated event) 577State and PR events are added to purgatory when git data doesn't exist:
4972. If accepted, check if we need to sync repos with the same identifier
498 578
499```rust 579```rust
580// From src/nostr/policy/state.rs
500async fn handle_state(&self, event: &Event) -> WritePolicyResult { 581async fn handle_state(&self, event: &Event) -> WritePolicyResult {
501 let identifier = extract_identifier(&event)?; 582 let identifier = extract_identifier(event)?;
502 let author = event.pubkey; 583
503 let state_refs = extract_refs_from_state(&event); 584 // Check if we have matching git data
504 585 if self.has_matching_git_data(&identifier, event).await? {
505 // Check for existing state event from this author with larger created_at 586 return WritePolicyResult::Accept;
506 let existing_states = self.database.get_state_events_by_author_identifier(
507 &author,
508 &identifier
509 ).await?;
510
511 for existing in existing_states {
512 if existing.created_at > event.created_at {
513 // Reject - we have a newer state from this author
514 return WritePolicyResult::Reject {
515 status: false,
516 message: "rejected: newer state event exists for this author/identifier".into()
517 };
518 }
519 } 587 }
520 588
521 // Check if we already have matching git data
522 let repos = self.find_repos_for_identifier(&identifier).await?;
523 for repo in repos {
524 if self.refs_match_state(&repo, &state_refs).await? {
525 // Git data exists - process immediately
526 // Also trigger sync check for other repos with same identifier
527 // Pass the repo that has the git data so it can be used as source
528 self.check_and_sync_repos_for_identifier(&identifier, &event, &repo).await?;
529 return WritePolicyResult::Accept;
530 }
531 }
532
533 // Add to purgatory 589 // Add to purgatory
534 let msg = self.purgatory.add_state(event.clone(), identifier); 590 self.purgatory.add_state(
591 event.clone(),
592 identifier.clone(),
593 event.pubkey,
594 );
595
535 WritePolicyResult::Reject { 596 WritePolicyResult::Reject {
536 status: true, // Client sees OK 597 status: true, // Client sees OK
537 message: msg.into() 598 message: "purgatory: awaiting git data".into()
538 }
539}
540```
541
542### 3. handle_pr_event Changes
543
544On PR event arrival:
545
546**Key Rule**: Incoming PR events supersede existing refs. If the existing `refs/nostr/<event-id>` ref has a different commit_id, the ref should be removed and the event stored in purgatory to await new git data.
547
548```rust
549async fn handle_pr_event(&self, event: &Event) -> WritePolicyResult {
550 let commit = extract_c_tag_commit(&event)?;
551 let event_id = event.id.to_hex();
552
553 // Check if placeholder exists (git-data-first)
554 if let Some(placeholder) = self.purgatory.find_pr_placeholder(&event.id) {
555 if placeholder.commit == commit {
556 self.purgatory.remove_pr(&event.id); // Note this shouldnt remove the git data
557 return WritePolicyResult::Accept;
558 } else {
559 // Placeholder has different commit - incoming event supersedes
560 // Update placeholder with new expected commit
561 self.purgatory.remove_pr(&event.id);
562 // TODO also remove git data
563 let msg = self.purgatory.add_pr(event.clone(), commit);
564 return WritePolicyResult::Reject {
565 status: true, // Client sees OK - in purgatory awaiting correct git data
566 message: msg.into()
567 };
568 }
569 }
570
571 // Add to purgatory
572 let msg = self.purgatory.add_pr(event.clone(), commit);
573 WritePolicyResult::Reject {
574 status: true,
575 message: msg.into()
576 } 599 }
577} 600}
578``` 601```
579 602
580### 4. Git Handler Changes 603### 2. Git Push Authorization
581 604
582In `handle_receive_pack` for normal refs: 605Authorization checks both database and purgatory:
583
584**Key Rule**: Refs are sent to a specific repo (`pubkey/identifier`), and there can only be **one authorized state event** for that repo. Therefore, this function returns a single `Option<Event>` rather than `Vec<Event>`.
585 606
586```rust 607```rust
587async fn handle_state_refs_push( 608// From src/git/authorization.rs
588 &self, 609pub async fn authorize_push(
610 database: &SharedDatabase,
589 identifier: &str, 611 identifier: &str,
590 repo_owner: &str, 612 owner_pubkey: &str,
591 pushed_refs: &[RefPair], 613 request_body: &Bytes,
592) -> Result<Option<Event>, GitError> { 614 purgatory: &Arc<Purgatory>, // Critical!
593 let local_refs = git::list_refs(&self.repo_path)?; 615 repo_path: &std::path::Path,
594 616) -> anyhow::Result<AuthorizationResult> {
595 // Find matching state events 617 // Parse pushed refs
596 let candidates = self.purgatory.find_matching_states( 618 let pushed_refs = parse_pushed_refs(request_body);
597 identifier, 619
598 pushed_refs, 620 // Check database for state events
599 &local_refs, 621 let db_result = get_authorization_from_db(database, identifier).await?;
600 ); 622
601 623 if !db_result.authorized {
602 // Get all state events from relay for this identifier 624 // No state in database - check purgatory
603 let relay_states = self.database.get_state_events_for_identifier(identifier).await?; 625 let purgatory_result = get_state_authorization_for_specific_owner_repo(
604 626 database,
605 // Get announcements to determine authorization 627 identifier,
606 let announcements = self.database.get_announcements(identifier).await?; 628 owner_pubkey,
607 let auth_map = collect_authorized_maintainers(&announcements); 629 purgatory,
608 630 &pushed_refs,
609 // Find the authoritative state event (largest created_at among all authorized) 631 repo_path,
610 let mut best_candidate: Option<(Event, Timestamp)> = None; 632 ).await?;
611 633
612 // Check purgatory candidates 634 return purgatory_result;
613 for event in candidates {
614 if let Some(maintainers) = auth_map.get(repo_owner) {
615 if maintainers.contains(&event.pubkey.to_hex()) {
616 // Check if this event is superseded by any relay state
617 let superseded = relay_states.iter().any(|relay_state| {
618 let relay_authorized = auth_map.values()
619 .any(|m| m.contains(&relay_state.pubkey.to_hex()));
620 relay_authorized && relay_state.created_at > event.created_at
621 });
622
623 if superseded {
624 // Don't use this as best_candidate for THIS repo
625 // Note: Do NOT remove from purgatory - it may still be
626 // authoritative for a DIFFERENT repo with a different maintainer set
627 continue;
628 }
629
630 // Verify OIDs for unpushed refs exist
631 let state_refs = extract_refs_from_state(&event);
632 let unpushed = get_unpushed_refs(&state_refs, pushed_refs);
633 if verify_oids_exist(&self.repo_path, &unpushed)? {
634 // Track best candidate by created_at
635 if best_candidate.is_none() || event.created_at > best_candidate.as_ref().unwrap().1 {
636 best_candidate = Some((event, event.created_at));
637 }
638 }
639 }
640 }
641 }
642
643 // If we found an approved event, extend its expiry
644 if let Some((ref event, _)) = best_candidate {
645 self.purgatory.extend_expiry(&[event.id], Duration::from_secs(900));
646 } 635 }
647 636
648 Ok(best_candidate.map(|(e, _)| e)) 637 db_result
649} 638}
650``` 639```
651 640
652For `refs/nostr/<event-id>` pushes: 641### 3. Post-Push Processing
653 642
654**Key Rule**: If there is no event (neither in database nor in purgatory), a push of a different commit_id should be **accepted**. This is the "git-data-first" scenario where we're waiting for the PR event to arrive. 643After successful push, events from purgatory are saved to database:
655 644
656```rust 645```rust
657async fn handle_nostr_ref_push( 646// From src/git/handlers.rs
658 &self, 647if from_purgatory {
659 event_id: &str, 648 if let (Some(db), Some(purg)) = (&database, &purgatory) {
660 pushed_commit: &str, 649 // Save state event to database
661) -> Result<PushDecision, GitError> { 650 db.save_event(&state.event).await?;
662 // Check database first 651
663 if let Some(event) = self.database.get_event_by_id(event_id).await? { 652 // Remove from purgatory
664 let expected_commit = extract_c_tag_commit(&event)?; 653 purg.remove_state_event(identifier, &state.event.id);
665 if expected_commit == pushed_commit {
666 return Ok(PushDecision::Accept);
667 } else {
668 return Ok(PushDecision::Reject("commit mismatch with existing event"));
669 }
670 }
671
672 // Check purgatory for PR event
673 if let Some(entry) = self.purgatory.find_pr(&EventId::from_hex(event_id)?) {
674 if let Some(event) = entry.event {
675 // Event exists in purgatory - must match commit
676 let expected_commit = extract_c_tag_commit(&event)?;
677 if expected_commit == pushed_commit {
678 // Remove from purgatory before returning
679 self.purgatory.remove_pr(&EventId::from_hex(event_id)?); // note this shouldnt delete the git data
680 return Ok(PushDecision::AcceptAndRelease(event));
681 } else {
682 return Ok(PushDecision::Reject("commit mismatch with purgatory event"));
683 }
684 } else {
685 // Placeholder exists (previous push, no event yet)
686 // Accept and update placeholder with new commit
687 // This allows re-pushing with corrected data before event arrives
688 return Ok(PushDecision::AcceptAndUpdatePlaceholder(pushed_commit.to_string()));
689 }
690 } 654 }
691
692 // No event anywhere - git-data-first scenario
693 // Accept ANY commit and create placeholder awaiting the PR event
694 Ok(PushDecision::AcceptAndCreatePlaceholder(pushed_commit.to_string()))
695} 655}
696// TODO when AcceptAndUpdatePlaceholder gets called purgatory must get udpated with a new / updated entry either here of where AcceptAndCreatePlaceholder is handled
697``` 656```
698 657
699### 5. Main.rs Changes 658### 4. Background Sync Loop
700 659
701```rust 660Started during application initialization:
702// During startup
703let purgatory = Arc::new(Purgatory::new());
704 661
705// Pass to WritePolicy 662```rust
706let write_policy = Nip34WritePolicy::new( 663// From src/main.rs
707 &config.domain, 664let purgatory = Arc::new(Purgatory::new(git_data_path));
665let ctx = Arc::new(RealSyncContext::new(
708 database.clone(), 666 database.clone(),
709 &git_data_path,
710 purgatory.clone(), 667 purgatory.clone(),
711); 668 config.domain.clone(),
712 669 git_data_path.clone(),
713// Pass to git handlers (via shared state) 670));
714let git_state = GitState { 671let throttle_manager = Arc::new(ThrottleManager::new(5, 30));
715 purgatory: purgatory.clone(), 672throttle_manager.set_context(ctx.clone());
716 database: database.clone(), 673
717 // ... 674// Start sync loop
718}; 675let sync_handle = purgatory.clone().start_sync_loop(ctx, throttle_manager);
719 676
720// Spawn cleanup task 677// Start cleanup task
721let purgatory_cleanup = purgatory.clone(); 678let cleanup_handle = tokio::spawn(async move {
722tokio::spawn(async move {
723 let mut interval = tokio::time::interval(Duration::from_secs(60)); 679 let mut interval = tokio::time::interval(Duration::from_secs(60));
724 loop { 680 loop {
725 interval.tick().await; 681 interval.tick().await;
726 let removed = purgatory_cleanup.cleanup(); 682 let (state_removed, pr_removed) = purgatory.cleanup();
727 if removed > 0 { 683 if state_removed + pr_removed > 0 {
728 tracing::debug!("Purgatory cleanup removed {} expired entries", removed); 684 tracing::debug!(
685 "Purgatory cleanup removed {} state, {} PR entries",
686 state_removed, pr_removed
687 );
729 } 688 }
730 } 689 }
731}); 690});
732``` 691```
733 692
734## Post-Push Sync Flow 693---
735
736After successfully processing a state or PR event, sync to other maintainer repos:
737
738**Key Rules for State Events (30618)**:
739
7401. Fetch all state events matching the identifier from the database
7412. Check if another existing state event supersedes this one (the maintainer set may be different, so another maintainer may have a more recent state event)
7423. Only sync if this event is not superseded
743
744```rust
745async fn sync_to_other_repos(
746 &self,
747 identifier: &str,
748 event: &Event,
749 processed_repo_owner: &str,
750) -> Result<usize, SyncError> {
751 let announcements = self.database.get_announcements(identifier).await?;
752 let auth_map = collect_authorized_maintainers(&announcements);
753
754 // Fetch all state events for this identifier to check for superseding
755 let all_state_events = self.database.get_state_events_for_identifier(identifier).await?;
756
757 let mut synced = 0;
758 for (owner, maintainers) in auth_map {
759 // Skip the repo we just processed
760 if owner == processed_repo_owner {
761 continue;
762 }
763
764 // Check if event author is authorized for this owner's repo
765 if maintainers.contains(&event.pubkey.to_hex()) {
766 let repo_path = self.repo_path_for_owner(&owner);
767
768 match event.kind.as_u64() {
769 30618 => {
770 // State event - check if another state event supersedes this one
771 // for THIS owner's repo (maintainer sets may differ between owners)
772 let owner_maintainers = auth_map.get(&owner).cloned().unwrap_or_default();
773
774 // Find the authoritative state for this owner's repo
775 let superseding_state = all_state_events.iter()
776 .filter(|s| owner_maintainers.contains(&s.pubkey.to_hex()))
777 .filter(|s| s.created_at > event.created_at)
778 .max_by_key(|s| s.created_at);
779
780 if let Some(newer_state) = superseding_state {
781 // This event is superseded by a more recent state
782 // for this owner's repo - skip syncing this event
783 tracing::debug!(
784 "Skipping sync to {}: event {} superseded by {}",
785 owner,
786 event.id,
787 newer_state.id
788 );
789 continue;
790 }
791
792 // No superseding state - align refs
793 let state_refs = extract_refs_from_state(&event);
794 self.align_refs(&repo_path, &state_refs)?;
795 }
796 1617 | 1618 => {
797 // PR event - ensure nostr ref exists
798 let commit = extract_c_tag_commit(&event)?;
799 self.ensure_nostr_ref(&repo_path, &event.id, &commit)?;
800 }
801 _ => {}
802 }
803 synced += 1;
804 }
805 }
806
807 Ok(synced)
808}
809```
810 694
811## Implementation File Structure 695## File Structure
812 696
813``` 697```
814src/ 698src/
815├── purgatory/ 699├── purgatory/
816│ ├── mod.rs # Purgatory struct, public API 700│ ├── mod.rs # Main Purgatory struct and API
817│ ├── types.rs # RefPair, StatePurgatoryEntry, PrPurgatoryEntry 701│ ├── types.rs # RefPair, StatePurgatoryEntry, PrPurgatoryEntry
818│ ├── state_events.rs # State event purgatory logic 702│ ├── helpers.rs # Ref extraction and matching functions
819│ ── pr_events.rs # PR event purgatory logic 703│ ── sync/
820│ ── helpers.rs # extract_refs_from_state, can_satisfy_state, etc. 704│ ── mod.rs # Sync module exports
821├── nostr/ 705 ── loop.rs # Background sync loop
822│ ├── builder.rs # Modified: Nip34WritePolicy accepts Arc<Purgatory> 706│ ├── functions.rs # sync_identifier, sync_identifier_from_url
823│ ── policy/ 707│ ── context.rs # SyncContext trait and RealSyncContext
824│ ├── state.rs # Modified: handle_state uses purgatory 708│ ├── queue.rs # SyncQueueEntry
825│ └── pr_event.rs # Modified: handle_pr_event uses purgatory 709│ └── throttle.rs # ThrottleManager, DomainThrottle
826├── git/ 710├── git/
827│ └── handlers.rs # Modified: handle_receive_pack integrates purgatory 711│ ├── authorization.rs # authorize_push with purgatory checking
828└── main.rs # Modified: creates Purgatory, spawns cleanup task 712│ ├── handlers.rs # handle_receive_pack with post-push processing
713│ └── sync.rs # process_newly_available_git_data
714└── nostr/
715 └── policy/
716 ├── state.rs # State event policy with purgatory
717 └── pr_event.rs # PR event policy with purgatory
829``` 718```
830 719
831## Additional Type Definitions 720---
832
833### PushDecision Enum
834
835Used by `handle_nostr_ref_push` to communicate different outcomes to the caller:
836
837```rust
838/// Result of evaluating a push to refs/nostr/<event-id>
839pub enum PushDecision {
840 /// Push is valid - event exists in database and commit matches
841 Accept,
842
843 /// Push valid and event should be released from purgatory to database
844 AcceptAndRelease(Event),
845
846 /// Push valid - create new placeholder awaiting PR event
847 AcceptAndCreatePlaceholder(String), // commit SHA
848
849 /// Push valid - update existing placeholder with new commit
850 AcceptAndUpdatePlaceholder(String), // new commit SHA
851
852 /// Push rejected with reason
853 Reject(&'static str),
854}
855```
856
857### WritePolicyResult Clarification
858
859The design uses a pattern where purgatory events return `status: true` but the event is NOT saved:
860
861```rust
862// Event goes to purgatory - client sees OK but event not served until git data arrives
863WritePolicyResult::Reject {
864 status: true, // Nostr OK message to client
865 message: "purgatory: won't be served until git data arrives".into()
866}
867
868// Event rejected - client sees error
869WritePolicyResult::Reject {
870 status: false, // Nostr error message to client
871 message: "rejected: reason...".into()
872}
873
874// Event accepted and saved to database
875WritePolicyResult::Accept
876```
877
878**Note:** This is a quirk of using the `WritePolicyResult::Reject` variant for purgatory - the `status: true` ensures the client receives an OK response, but since we're in the Reject variant, the event is not automatically saved to the database. The purgatory mechanism holds it until git data arrives.
879
880## Test Scenarios
881 721
882### State Event Tests 722## Testing
883 723
8841. **Event arrives, git data exists** - Event processed immediately, saved to DB 724### Unit Tests
8852. **Event arrives, git data doesn't exist** - Event goes to purgatory, client sees OK
8863. **Git push arrives, matching event in purgatory** - Event released from purgatory, saved to DB
8874. **Git push arrives, no matching event** - Push rejected (no authorized state)
8885. **Event expires in purgatory** - Entry removed after 30 minutes (dont implement test due to 30m wait)
8896. **Multiple state events for same identifier** - Late binding at push time selects correct one
890 725
891### PR Event Tests 726Located in each module:
892 727
8931. **PR event arrives, git data exists** - Event processed immediately, saved to DB 728- **[`src/purgatory/mod.rs`](../../src/purgatory/mod.rs)** - Core purgatory operations
8942. **PR event arrives, no git data** - Event goes to purgatory awaiting git push 729- **[`src/purgatory/helpers.rs`](../../src/purgatory/helpers.rs)** - Ref matching logic
8953. **PR event arrives, placeholder exists with matching commit** - Event released, saved to DB 730- **[`src/purgatory/sync/functions.rs`](../../src/purgatory/sync/functions.rs)** - Sync functions with MockSyncContext
8964. **PR event arrives, placeholder exists with different commit** - Ref deleted, event to purgatory 731- **[`src/purgatory/sync/throttle.rs`](../../src/purgatory/sync/throttle.rs)** - Throttle manager
8975. **Git push to refs/nostr/ arrives, PR event exists in purgatory** - Event released, ref created
8986. **Git push to refs/nostr/ arrives, no PR event** - Placeholder created, awaiting event
8997. **Second git push updates placeholder** - Placeholder commit updated
900 732
901### Edge Cases (NOT TESTED) 733### Integration Tests
902 734
9031. **Relay restart** - All purgatory entries lost (acceptable per design) 735Located in [`tests/`](../../tests/):
9042. **Same event submitted twice** - Deduplicated by event ID
9053. **Push timeout during processing** - Entry expiry extended to 15 min minimum
9064. **Race between event and git push** - Whichever completes the pair triggers release
907 736
737- **State event purgatory flow** - Event arrives, git push releases it
738- **PR event purgatory flow** - Event arrives, git push releases it
739- **Git-data-first flow** - Git push creates placeholder, event completes it
740- **Authorization with purgatory** - Push authorized by purgatory state
741- **Background sync** - Sync fetches git data and releases events
908 742
909## Purgatory Authorization Fix (2025-12-24) 743---
910 744
911**Critical Implementation Note**: The original purgatory design placed purgatory checking AFTER git push execution. This created a deadlock where pushes were rejected because the authorizing state event was in purgatory, not the database. 745## Key Learnings
912 746
913### The Deadlock Problem 747### 1. Purgatory Authorization is Critical
914 748
915**Original broken flow:** 749Without checking purgatory during authorization, we have a deadlock:
9161. State event arrives → No git data exists → Event stored in PURGATORY (not database) 750- State event goes to purgatory (no git data)
9172. Git push arrives → Authorization checks DATABASE only → No state found → **PUSH REJECTED** ❌ 751- Push is rejected (no state in database)
9183. Purgatory check runs → But push already failed, so this never helps 752- Event never gets released
919 753
920### The Fix: Authorization-Time Purgatory Check 754**Solution:** `authorize_push()` checks both database and purgatory.
921 755
922**Correct flow (implemented):** 756### 2. Late Binding for State Events
9231. State event arrives → No git data exists → Event stored in purgatory
9242. Git push arrives → Authorization checks **DATABASE + PURGATORY** → State found in purgatory → **PUSH AUTHORIZED** ✅
9253. After successful push → Save purgatory event to database → Remove from purgatory
926 757
927### Implementation Details 758Extracting refs at event arrival time doesn't work when:
759- Multiple state events arrive for same identifier
760- Git data for older state arrives after newer state received
928 761
929#### 1. Modified [`AuthorizationResult`](../../src/git/authorization.rs:397) 762**Solution:** Extract and match refs at push time via `find_matching_states()`.
930 763
931Added `from_purgatory: bool` field to track whether the authorizing state came from purgatory: 764### 3. Bidirectional Waiting for PR Events
932 765
933```rust 766PR events can arrive before or after git data:
934pub struct AuthorizationResult { 767- Event first → Wait for git push
935 pub authorized: bool, 768- Git first → Create placeholder, wait for event
936 pub reason: String,
937 pub state: Option<RepositoryState>,
938 pub maintainers: Vec<String>,
939 pub from_purgatory: bool, // NEW: Track event source
940}
941```
942 769
943#### 2. Enhanced [`get_authorization_for_owner()`](../../src/git/authorization.rs:342) 770**Solution:** `PrPurgatoryEntry.event: Option<Event>` with `None` = placeholder.
944 771
945Added purgatory checking when no state found in database: 772### 4. Sync Queue Debouncing
946 773
947```rust 774When events arrive in bursts (e.g., negentropy sync), we don't want to spawn a sync task for each event.
948pub async fn get_authorization_for_owner(
949 database: &SharedDatabase,
950 identifier: &str,
951 owner_pubkey: &str,
952 purgatory: Option<&Arc<Purgatory>>,
953 pushed_refs: &[(String, String, String)],
954 repo_path: &Path,
955) -> Result<AuthorizationResult>
956```
957 775
958**Logic**: 776**Solution:** `enqueue_sync()` resets `attempt_count` and updates `next_attempt` if already queued.
9591. Check database for state events (existing behavior)
9602. If no state in database AND purgatory available:
961 - Parse pushed refs to RefPairs
962 - Get local refs from repository
963 - Call [`find_matching_states()`](../../src/purgatory/mod.rs:203)
964 - Filter to latest event from authorized authors
965 - Return authorization with `from_purgatory: true`
966 777
967#### 3. Post-Push Purgatory Event Save 778### 5. Domain Throttling with Queues
968 779
969In [`handle_receive_pack()`](../../src/git/handlers.rs:187), after successful push: 780When a domain is throttled, we still want to eventually sync from it.
970 781
971```rust 782**Solution:** `ThrottleManager` maintains per-domain queues and processes them when capacity frees.
972if from_purgatory {
973 if let (Some(db), Some(purg)) = (&database, &purgatory) {
974 // Save state event to database
975 db.save_event(&state.event).await?;
976
977 // Remove from purgatory
978 purg.remove_state_event(identifier, &state.event.id);
979 }
980}
981```
982 783
983### Files Modified 784---
984
9851. **[`src/git/authorization.rs`](../../src/git/authorization.rs)**
986 - Added `from_purgatory` field to `AuthorizationResult`
987 - Modified `get_authorization_for_owner()` signature and logic
988 - Added purgatory checking when database has no state
989
9902. **[`src/git/handlers.rs`](../../src/git/handlers.rs)**
991 - Modified `authorize_push()` to accept purgatory and repo_path parameters
992 - Added tracking of `from_purgatory` flag
993 - Added post-push database save for purgatory events
994
995### Why This Order Matters
996
997Checking purgatory DURING authorization (before push execution) is critical:
998
999- **Prevents deadlock**: Push is authorized by purgatory state before execution
1000- **Maintains atomicity**: Only saves to database after successful push
1001- **Race condition safe**: First successful push claims the purgatory event
1002
1003The alternative (checking purgatory after push) creates an insurmountable deadlock where valid pushes are rejected because their authorizing state is in purgatory instead of the database.
1004 785
1005### Testing 786## Related Documentation
1006 787
1007The fix enables the `test_push_authorized_by_owner_state` integration test scenario where: 788- [Inline Authorization](inline-authorization.md) - Why purgatory checking during authorization is essential
10081. State event is sent to relay (goes to purgatory - no git data yet) 789- [Architecture Overview](architecture.md) - Full system design
10092. Git push is sent (uses purgatory state for authorization) 790- [Background Sync](../how-to/purgatory-sync.md) - How to configure and monitor sync
10103. State event is released from purgatory to database 791- [Test Strategy](../reference/test-strategy.md) - How we test purgatory
1011 792
1012--- 793---
1013 794
795*Part of the [ngit-grasp explanation docs](./)*
generated by cgit v1.2.3 (git 2.39.1) at 2026-06-05 01:14:33 +0000