diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/explanation/announcements-purgatory-design.md | 72 | ||||
| -rw-r--r-- | docs/explanation/announcements-purgatory-implementation.md | 13 |
2 files changed, 47 insertions, 38 deletions
diff --git a/docs/explanation/announcements-purgatory-design.md b/docs/explanation/announcements-purgatory-design.md index a06a8b2..009547b 100644 --- a/docs/explanation/announcements-purgatory-design.md +++ b/docs/explanation/announcements-purgatory-design.md | |||
| @@ -48,14 +48,14 @@ This ensures we only serve announcements for repos that actually have content. | |||
| 48 | 48 | ||
| 49 | ### 4. Expiry Extension (Two Places) | 49 | ### 4. Expiry Extension (Two Places) |
| 50 | 50 | ||
| 51 | **Decision:** Extend purgatory announcement expiry in two scenarios: | 51 | **Decision:** Extend purgatory announcement expiry (reset the 30-minute protocol timer) in two scenarios: |
| 52 | 52 | ||
| 53 | | Trigger | Location | Why | | 53 | | Trigger | Location | Why | |
| 54 | | ---------------------------- | ------------------------------------ | ----------------------------------- | | 54 | | ---------------------------- | ------------------------------------ | ----------------------------------- | |
| 55 | | State event arrives | `StatePolicy::process_state_event()` | Repo is actively receiving metadata | | 55 | | State event arrives | `StatePolicy::process_state_event()` | Repo is actively receiving metadata | |
| 56 | | Git auth extends state event | `src/git/auth.rs` | Repo is actively receiving git data | | 56 | | Git auth extends state event | `src/git/auth.rs` | Repo is actively receiving git data | |
| 57 | 57 | ||
| 58 | **Why:** Prevents premature expiry during slow sync operations or multi-step pushes. | 58 | **Why:** Prevents premature expiry during slow sync operations or multi-step pushes. The protocol's 30-minute expiry is intended for abandoned repositories, not active ones receiving data. |
| 59 | 59 | ||
| 60 | ### 5. Authorization Must Check Purgatory Announcements | 60 | ### 5. Authorization Must Check Purgatory Announcements |
| 61 | 61 | ||
| @@ -75,20 +75,26 @@ This ensures we only serve announcements for repos that actually have content. | |||
| 75 | 75 | ||
| 76 | ### 7. Soft Expiry Preserves Event Without Bare Repo | 76 | ### 7. Soft Expiry Preserves Event Without Bare Repo |
| 77 | 77 | ||
| 78 | **Decision:** When a purgatory announcement expires, delete the bare repo but retain the announcement event for an extended period (e.g., 24h). | 78 | **Decision:** When a purgatory announcement expires (30 minutes per protocol spec), delete the bare repo but retain the announcement event for an extended period (e.g., 24h). |
| 79 | 79 | ||
| 80 | **Why:** This handles the case where a state event arrives after initial expiry. Without soft expiry, we'd either: | 80 | **Why the protocol specifies 30 minutes:** The grasp protocol defines a 30-minute expiry for announcement events to ensure clients don't indefinitely cache stale repository information. |
| 81 | - Add to `failed_events` and reject the state event (losing potential revival) | 81 | |
| 82 | - Re-fetch the announcement repeatedly (wasting sync bandwidth) | 82 | **Why we implement soft expiry:** The protocol's 30-minute expiry creates a sync/storage problem. Without soft expiry, we'd either: |
| 83 | |||
| 84 | - Add expired announcements to `failed_events` and permanently reject future state events (losing potential revival when state events arrive late) | ||
| 85 | - Re-fetch the announcement event repeatedly on every sync cycle (wasting bandwidth and creating unnecessary sync traffic) | ||
| 83 | 86 | ||
| 84 | **Behavior during soft expiry:** | 87 | **Behavior during soft expiry:** |
| 85 | - Bare repo is deleted (saves disk space) | 88 | |
| 89 | - Bare repo is deleted (saves disk space, respects protocol expiry) | ||
| 86 | - Announcement event retained in purgatory with `soft_expired` flag | 90 | - Announcement event retained in purgatory with `soft_expired` flag |
| 87 | - Sync continues requesting state events (same as active purgatory) | 91 | - Sync continues requesting state events (same as active purgatory) |
| 88 | - If state event arrives: recreate bare repo, clear `soft_expired`, extend expiry | 92 | - If state event arrives: recreate bare repo, clear `soft_expired`, extend expiry |
| 89 | - If announcement republished directly to us: treat as fresh arrival | 93 | - If announcement republished directly to us: treat as fresh arrival |
| 90 | - After extended expiry: fully remove from purgatory | 94 | - After extended expiry: fully remove from purgatory |
| 91 | 95 | ||
| 96 | **In summary:** Soft expiry is an implementation optimization that prevents us from constantly re-syncing announcement events or permanently blocking repositories that receive delayed state events. | ||
| 97 | |||
| 92 | ## Data Structure | 98 | ## Data Structure |
| 93 | 99 | ||
| 94 | ```rust | 100 | ```rust |
| @@ -198,29 +204,29 @@ Announcement ──> ACTIVE ─────────────────â | |||
| 198 | └── Extended expiry ──> REMOVED (exit) | 204 | └── Extended expiry ──> REMOVED (exit) |
| 199 | ``` | 205 | ``` |
| 200 | 206 | ||
| 201 | | Exit | Trigger | Action | | 207 | | Exit | Trigger | Action | |
| 202 | | -------------- | ---------------------------------------------- | -------------------------------------------- | | 208 | | ------------------ | -------------------------------------------- | --------------------------------------------- | |
| 203 | | **Promotion** | Git data arrives | Move to database, upgrade sync to Full | | 209 | | **Promotion** | Git data arrives | Move to database, upgrade sync to Full | |
| 204 | | **Soft expiry**| Initial timeout | Delete bare repo, retain event, continue sync| | 210 | | **Soft expiry** | Initial timeout | Delete bare repo, retain event, continue sync | |
| 205 | | **Full expiry**| Extended timeout (soft-expired) | Remove from purgatory entirely | | 211 | | **Full expiry** | Extended timeout (soft-expired) | Remove from purgatory entirely | |
| 206 | | **Deletion** | Kind 5 event | Delete bare repo, remove from purgatory | | 212 | | **Deletion** | Kind 5 event | Delete bare repo, remove from purgatory | |
| 207 | | **Replacement**| Newer announcement (same pubkey, identifier) | Replace entry | | 213 | | **Replacement** | Newer announcement (same pubkey, identifier) | Replace entry | |
| 208 | | **Service change** | Newer announcement removes our service | Remove from purgatory | | 214 | | **Service change** | Newer announcement removes our service | Remove from purgatory | |
| 209 | 215 | ||
| 210 | ## Integration Points | 216 | ## Integration Points |
| 211 | 217 | ||
| 212 | | File | Change | | 218 | | File | Change | |
| 213 | | ---------------------------------- | --------------------------------------------------------- | | 219 | | ---------------------------------- | ---------------------------------------------------------- | |
| 214 | | `src/purgatory/mod.rs` | Add `announcement_purgatory` store | | 220 | | `src/purgatory/mod.rs` | Add `announcement_purgatory` store | |
| 215 | | `src/purgatory/types.rs` | Add `AnnouncementPurgatoryEntry` | | 221 | | `src/purgatory/types.rs` | Add `AnnouncementPurgatoryEntry` | |
| 216 | | `src/nostr/policy/announcement.rs` | Route new announcements to purgatory | | 222 | | `src/nostr/policy/announcement.rs` | Route new announcements to purgatory | |
| 217 | | `src/git/receive.rs` | Promote on git data arrival | | 223 | | `src/git/receive.rs` | Promote on git data arrival | |
| 218 | | `src/git/auth.rs` | Extend purgatory expiry when extending state event expiry | | 224 | | `src/git/auth.rs` | Extend purgatory expiry when extending state event expiry | |
| 219 | | `src/git/authorization.rs` | Check purgatory announcements for maintainer authorization| | 225 | | `src/git/authorization.rs` | Check purgatory announcements for maintainer authorization | |
| 220 | | `src/nostr/policy/state.rs` | Check purgatory for authorization | | 226 | | `src/nostr/policy/state.rs` | Check purgatory for authorization | |
| 221 | | `src/sync/mod.rs` | Add `SyncLevel` to `RepoSyncNeeds` | | 227 | | `src/sync/mod.rs` | Add `SyncLevel` to `RepoSyncNeeds` | |
| 222 | | `src/sync/filters.rs` | Respect sync level when building filters | | 228 | | `src/sync/filters.rs` | Respect sync level when building filters | |
| 223 | | `src/sync/self_subscriber.rs` | Register purgatory announcements with `StateOnly` level | | 229 | | `src/sync/self_subscriber.rs` | Register purgatory announcements with `StateOnly` level | |
| 224 | 230 | ||
| 225 | See [announcements-purgatory-implementation.md](./announcements-purgatory-implementation.md) for detailed implementation notes. | 231 | See [announcements-purgatory-implementation.md](./announcements-purgatory-implementation.md) for detailed implementation notes. |
| 226 | 232 | ||
| @@ -240,9 +246,9 @@ See [announcements-purgatory-implementation.md](./announcements-purgatory-implem | |||
| 240 | 246 | ||
| 241 | ## Risks | 247 | ## Risks |
| 242 | 248 | ||
| 243 | | Risk | Mitigation | | 249 | | Risk | Mitigation | |
| 244 | | ------------------------------------ | ------------------------------------------------------- | | 250 | | ------------------------------------ | ------------------------------------------------------ | |
| 245 | | Disk exhaustion from purgatory repos | Short expiry, soft expiry deletes repo early | | 251 | | Disk exhaustion from purgatory repos | Short expiry, soft expiry deletes repo early | |
| 246 | | Race between promotion and expiry | Atomic operations | | 252 | | Race between promotion and expiry | Atomic operations | |
| 247 | | Sync re-fetching expired events | Soft expiry retains event; no need for `failed_events` | | 253 | | Sync re-fetching expired events | Soft expiry retains event; no need for `failed_events` | |
| 248 | | Filter explosion from many purgatory | Existing consolidation handles this (threshold at 70) | | 254 | | Filter explosion from many purgatory | Existing consolidation handles this (threshold at 70) | |
diff --git a/docs/explanation/announcements-purgatory-implementation.md b/docs/explanation/announcements-purgatory-implementation.md index d5b8698..263c253 100644 --- a/docs/explanation/announcements-purgatory-implementation.md +++ b/docs/explanation/announcements-purgatory-implementation.md | |||
| @@ -204,21 +204,22 @@ impl Purgatory { | |||
| 204 | .collect() | 204 | .collect() |
| 205 | } | 205 | } |
| 206 | 206 | ||
| 207 | /// Transition to soft-expired state | 207 | /// Transition to soft-expired state (protocol's 30min expiry reached) |
| 208 | pub fn soft_expire_announcement( | 208 | pub fn soft_expire_announcement( |
| 209 | &self, | 209 | &self, |
| 210 | key: &(PublicKey, String), | 210 | key: &(PublicKey, String), |
| 211 | ) -> Option<PathBuf> { | 211 | ) -> Option<PathBuf> { |
| 212 | if let Some(mut entry) = self.announcement_purgatory.get_mut(key) { | 212 | if let Some(mut entry) = self.announcement_purgatory.get_mut(key) { |
| 213 | entry.soft_expired = true; | 213 | entry.soft_expired = true; |
| 214 | entry.expires_at = Instant::now() + SOFT_EXPIRY_DURATION; // e.g., 24h | 214 | entry.expires_at = Instant::now() + SOFT_EXPIRY_DURATION; // e.g., 24h extended retention |
| 215 | Some(entry.repo_path.clone()) // Return path for bare repo deletion | 215 | Some(entry.repo_path.clone()) // Return path for bare repo deletion |
| 216 | } else { | 216 | } else { |
| 217 | None | 217 | None |
| 218 | } | 218 | } |
| 219 | } | 219 | } |
| 220 | 220 | ||
| 221 | /// Revive soft-expired announcement (caller must recreate bare repo) | 221 | /// Revive soft-expired announcement when state event arrives |
| 222 | /// (caller must recreate bare repo) | ||
| 222 | pub fn revive_announcement( | 223 | pub fn revive_announcement( |
| 223 | &self, | 224 | &self, |
| 224 | key: &(PublicKey, String), | 225 | key: &(PublicKey, String), |
| @@ -226,7 +227,7 @@ impl Purgatory { | |||
| 226 | if let Some(mut entry) = self.announcement_purgatory.get_mut(key) { | 227 | if let Some(mut entry) = self.announcement_purgatory.get_mut(key) { |
| 227 | if entry.soft_expired { | 228 | if entry.soft_expired { |
| 228 | entry.soft_expired = false; | 229 | entry.soft_expired = false; |
| 229 | entry.expires_at = Instant::now() + ACTIVE_EXPIRY_DURATION; | 230 | entry.expires_at = Instant::now() + ACTIVE_EXPIRY_DURATION; // Reset 30min protocol timer |
| 230 | return Some(entry.repo_path.clone()); // Caller recreates bare repo | 231 | return Some(entry.repo_path.clone()); // Caller recreates bare repo |
| 231 | } | 232 | } |
| 232 | } | 233 | } |
| @@ -270,9 +271,11 @@ When a state event arrives for a soft-expired announcement, the state policy mus | |||
| 270 | 1. Check purgatory for a matching announcement (in addition to DB) | 271 | 1. Check purgatory for a matching announcement (in addition to DB) |
| 271 | 2. Validate authorization against the purgatory announcement | 272 | 2. Validate authorization against the purgatory announcement |
| 272 | 3. If soft-expired, call `revive_announcement()` and recreate the bare repo | 273 | 3. If soft-expired, call `revive_announcement()` and recreate the bare repo |
| 273 | 4. Extend the announcement's expiry | 274 | 4. Extend the announcement's expiry (reset the 30-minute protocol timer) |
| 274 | 5. Route the state event to state purgatory | 275 | 5. Route the state event to state purgatory |
| 275 | 276 | ||
| 277 | **Why revival is necessary:** Without soft expiry + revival, late-arriving state events would either be permanently rejected (if we added the announcement to `failed_events`) or cause constant re-syncing of the announcement event. Revival allows us to respect the protocol's 30-minute expiry while still handling delayed state events gracefully. | ||
| 278 | |||
| 276 | The exact integration will depend on the current structure of `StatePolicy::process_state_event()` - see implementation phase for details. | 279 | The exact integration will depend on the current structure of `StatePolicy::process_state_event()` - see implementation phase for details. |
| 277 | 280 | ||
| 278 | ## File Change Summary | 281 | ## File Change Summary |