upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2026-02-23 15:08:37 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2026-02-23 15:08:37 +0000
commit26f608e5011b9d1ad6036da75b89272835e69695 (patch)
tree8b5dfe29f65abe80e59bddbcd3ee09c0a369dba8 /docs
parent4848c4029fc58f6f310a2babeae1ee82a7e41656 (diff)
persist and restore announcement events across graceful restarts
Extends purgatory persistence to include announcement purgatory entries. On graceful shutdown, non-soft-expired announcements are serialised to purgatory-state.json alongside state/PR/expired events; on startup they are restored, skipping any entry whose bare repo path no longer exists. Updates purgatory-design.md to reflect that purgatory persists through graceful shutdown and documents the new PurgatoryState disk format. Adds create_announcement_event helper to purgatory_helpers and three new integration tests in purgatory_persistence covering the full save/restore cycle, missing-repo skip, and the combined roundtrip with all entry types.
Diffstat (limited to 'docs')
-rw-r--r--docs/explanation/purgatory-design.md66
1 files changed, 62 insertions, 4 deletions
diff --git a/docs/explanation/purgatory-design.md b/docs/explanation/purgatory-design.md
index bd792d4..8e7d75c 100644
--- a/docs/explanation/purgatory-design.md
+++ b/docs/explanation/purgatory-design.md
@@ -39,14 +39,36 @@ This ensures we only serve announcements for repos that actually have content.
39 39
40## Key Design Principles 40## Key Design Principles
41 41
42### 1. In-Memory Only 42### 1. Graceful-Shutdown Persistence
43 43
44Purgatory data is **not persisted** to disk. On restart, all purgatory entries are lost. This is acceptable because: 44Purgatory state is **saved to disk on graceful shutdown** and **restored on startup**. This preserves in-flight work across planned restarts (deployments, reboots).
45
46On `SIGINT` / Ctrl-C, `main.rs` calls `purgatory.save_to_disk()` before exiting. On startup, if the state file exists, `purgatory.restore_from_disk()` is called before the server begins accepting connections.
47
48**What is persisted:**
49
50| Store | Persisted? | Notes |
51|-------|-----------|-------|
52| `announcement_purgatory` | ✅ Yes | Non-soft-expired entries only (bare repo must exist) |
53| `state_events` | ✅ Yes | All active entries |
54| `pr_events` | ✅ Yes | Both events and placeholders |
55| `expired_events` | ✅ Yes | Prevents re-sync loops after restart |
56| `sync_queue` | ❌ No | Rebuilt automatically after restore |
57
58**What is NOT persisted (unclean shutdown):**
59
60On a crash or `SIGKILL`, the state file is not written. In that case:
45 61
46- Events are still on other relays (can be re-submitted) 62- Events are still on other relays (can be re-submitted)
47- Git data can be re-pushed 63- Git data can be re-pushed
48- 30-minute expiry means data is transient anyway 64- 30-minute expiry means data is transient anyway
49 65
66**State file location:** `<git_data_path>/purgatory-state.json`
67
68**Downtime accounting:** Expiry deadlines are stored as duration offsets from the save timestamp. On restore, elapsed downtime is subtracted from each deadline. Entries that expired during downtime are immediately swept by the next cleanup tick.
69
70**Soft-expired announcements are excluded:** Their bare repos have already been deleted, so they cannot be meaningfully restored. They will be re-fetched via background sync if needed.
71
50### 2. Separate Storage for Each Event Type 72### 2. Separate Storage for Each Event Type
51 73
52| Store | Index | Purpose | 74| Store | Index | Purpose |
@@ -233,6 +255,31 @@ pub struct Purgatory {
233} 255}
234``` 256```
235 257
258### Persistence State (Disk Format)
259
260`Instant` fields cannot be serialized directly. Each entry type has a corresponding `Serializable*` wrapper that stores time fields as `u64` second offsets from a `saved_at: SystemTime` reference point. On restore, elapsed downtime is subtracted to produce the correct remaining TTL.
261
262```rust
263struct PurgatoryState {
264 version: u32, // currently 1
265 saved_at: SystemTime, // reference for offset math
266
267 /// Non-soft-expired announcements indexed by "owner_hex:identifier"
268 announcement_purgatory: HashMap<String, SerializableAnnouncementPurgatoryEntry>,
269
270 /// State events indexed by repository identifier
271 state_events: HashMap<String, Vec<SerializableStatePurgatoryEntry>>,
272
273 /// PR events (and placeholders) indexed by event ID hex
274 pr_events: HashMap<String, SerializablePrPurgatoryEntry>,
275
276 /// Expired event IDs → approximate expiry SystemTime
277 expired_events: HashMap<String, SystemTime>,
278}
279```
280
281The `announcement_purgatory` field uses `#[serde(default)]` so that state files written before announcement persistence was added (version 1 without the field) still deserialize correctly.
282
236--- 283---
237 284
238## Announcement Purgatory Flows 285## Announcement Purgatory Flows
@@ -806,8 +853,9 @@ A background timer (`run_purgatory_announcement_sync`, every 5 seconds) ensures
806``` 853```
807src/ 854src/
808├── purgatory/ 855├── purgatory/
809│ ├── mod.rs # Main Purgatory struct and API 856│ ├── mod.rs # Main Purgatory struct, API, save_to_disk, restore_from_disk
810│ ├── types.rs # RefPair, AnnouncementPurgatoryEntry, StatePurgatoryEntry, PrPurgatoryEntry 857│ ├── types.rs # RefPair, AnnouncementPurgatoryEntry, StatePurgatoryEntry, PrPurgatoryEntry
858│ ├── persistence.rs # instant_to_offset / offset_to_instant time conversion utilities
811│ ├── helpers.rs # Ref extraction and matching functions 859│ ├── helpers.rs # Ref extraction and matching functions
812│ └── sync/ 860│ └── sync/
813│ ├── mod.rs # Sync module exports 861│ ├── mod.rs # Sync module exports
@@ -835,7 +883,8 @@ src/
835 883
836Located in each module: 884Located in each module:
837 885
838- **[`src/purgatory/mod.rs`](../../src/purgatory/mod.rs)** - Core purgatory operations including announcement purgatory 886- **[`src/purgatory/mod.rs`](../../src/purgatory/mod.rs)** - Core purgatory operations including announcement purgatory; persistence round-trip tests for all entry types (state, PR, announcement, expired events, downtime calculation, soft-expired exclusion, missing-repo skip)
887- **[`src/purgatory/persistence.rs`](../../src/purgatory/persistence.rs)** - `instant_to_offset` / `offset_to_instant` round-trip tests
839- **[`src/purgatory/helpers.rs`](../../src/purgatory/helpers.rs)** - Ref matching logic 888- **[`src/purgatory/helpers.rs`](../../src/purgatory/helpers.rs)** - Ref matching logic
840- **[`src/purgatory/sync/functions.rs`](../../src/purgatory/sync/functions.rs)** - Sync functions with MockSyncContext 889- **[`src/purgatory/sync/functions.rs`](../../src/purgatory/sync/functions.rs)** - Sync functions with MockSyncContext
841- **[`src/purgatory/sync/throttle.rs`](../../src/purgatory/sync/throttle.rs)** - Throttle manager 890- **[`src/purgatory/sync/throttle.rs`](../../src/purgatory/sync/throttle.rs)** - Throttle manager
@@ -852,6 +901,7 @@ Located in [`tests/`](../../tests/):
852- **Git-data-first flow** - Git push creates placeholder, event completes it 901- **Git-data-first flow** - Git push creates placeholder, event completes it
853- **Authorization with purgatory** - Push authorized by purgatory state 902- **Authorization with purgatory** - Push authorized by purgatory state
854- **Background sync** - Sync fetches git data and releases events 903- **Background sync** - Sync fetches git data and releases events
904- **Persistence across restart** - Save/restore cycle preserves all entry types including announcements
855 905
856--- 906---
857 907
@@ -894,6 +944,14 @@ PR events can arrive before or after git data:
894 944
895**Solution:** `PrPurgatoryEntry.event: Option<Event>` with `None` = placeholder. 945**Solution:** `PrPurgatoryEntry.event: Option<Event>` with `None` = placeholder.
896 946
947### 6. Persistence Requires Instant → Duration Conversion
948
949`std::time::Instant` is not serializable and is not meaningful across process boundaries. Expiry deadlines must be converted to a portable form.
950
951**Solution:** Store each deadline as a `u64` second offset from a `saved_at: SystemTime` reference. On restore, subtract elapsed downtime from each offset to compute the new `Instant`. Entries whose deadline already passed during downtime get `expires_at = now` and are swept by the next cleanup tick.
952
953**Soft-expired announcements are excluded from persistence** because their bare repos have been deleted. Restoring them would leave purgatory entries pointing at non-existent repos. They are simply dropped; background sync will re-fetch the announcement event if needed.
954
897--- 955---
898 956
899## Related Documentation 957## Related Documentation