diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-02-23 15:08:37 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-02-23 15:08:37 +0000 |
| commit | 26f608e5011b9d1ad6036da75b89272835e69695 (patch) | |
| tree | 8b5dfe29f65abe80e59bddbcd3ee09c0a369dba8 /docs | |
| parent | 4848c4029fc58f6f310a2babeae1ee82a7e41656 (diff) | |
persist and restore announcement events across graceful restarts
Extends purgatory persistence to include announcement purgatory entries.
On graceful shutdown, non-soft-expired announcements are serialised to
purgatory-state.json alongside state/PR/expired events; on startup they
are restored, skipping any entry whose bare repo path no longer exists.
Updates purgatory-design.md to reflect that purgatory persists through
graceful shutdown and documents the new PurgatoryState disk format.
Adds create_announcement_event helper to purgatory_helpers and three new
integration tests in purgatory_persistence covering the full save/restore
cycle, missing-repo skip, and the combined roundtrip with all entry types.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/explanation/purgatory-design.md | 66 |
1 files changed, 62 insertions, 4 deletions
diff --git a/docs/explanation/purgatory-design.md b/docs/explanation/purgatory-design.md index bd792d4..8e7d75c 100644 --- a/docs/explanation/purgatory-design.md +++ b/docs/explanation/purgatory-design.md | |||
| @@ -39,14 +39,36 @@ This ensures we only serve announcements for repos that actually have content. | |||
| 39 | 39 | ||
| 40 | ## Key Design Principles | 40 | ## Key Design Principles |
| 41 | 41 | ||
| 42 | ### 1. In-Memory Only | 42 | ### 1. Graceful-Shutdown Persistence |
| 43 | 43 | ||
| 44 | Purgatory data is **not persisted** to disk. On restart, all purgatory entries are lost. This is acceptable because: | 44 | Purgatory state is **saved to disk on graceful shutdown** and **restored on startup**. This preserves in-flight work across planned restarts (deployments, reboots). |
| 45 | |||
| 46 | On `SIGINT` / Ctrl-C, `main.rs` calls `purgatory.save_to_disk()` before exiting. On startup, if the state file exists, `purgatory.restore_from_disk()` is called before the server begins accepting connections. | ||
| 47 | |||
| 48 | **What is persisted:** | ||
| 49 | |||
| 50 | | Store | Persisted? | Notes | | ||
| 51 | |-------|-----------|-------| | ||
| 52 | | `announcement_purgatory` | ✅ Yes | Non-soft-expired entries only (bare repo must exist) | | ||
| 53 | | `state_events` | ✅ Yes | All active entries | | ||
| 54 | | `pr_events` | ✅ Yes | Both events and placeholders | | ||
| 55 | | `expired_events` | ✅ Yes | Prevents re-sync loops after restart | | ||
| 56 | | `sync_queue` | ❌ No | Rebuilt automatically after restore | | ||
| 57 | |||
| 58 | **What is NOT persisted (unclean shutdown):** | ||
| 59 | |||
| 60 | On a crash or `SIGKILL`, the state file is not written. In that case: | ||
| 45 | 61 | ||
| 46 | - Events are still on other relays (can be re-submitted) | 62 | - Events are still on other relays (can be re-submitted) |
| 47 | - Git data can be re-pushed | 63 | - Git data can be re-pushed |
| 48 | - 30-minute expiry means data is transient anyway | 64 | - 30-minute expiry means data is transient anyway |
| 49 | 65 | ||
| 66 | **State file location:** `<git_data_path>/purgatory-state.json` | ||
| 67 | |||
| 68 | **Downtime accounting:** Expiry deadlines are stored as duration offsets from the save timestamp. On restore, elapsed downtime is subtracted from each deadline. Entries that expired during downtime are immediately swept by the next cleanup tick. | ||
| 69 | |||
| 70 | **Soft-expired announcements are excluded:** Their bare repos have already been deleted, so they cannot be meaningfully restored. They will be re-fetched via background sync if needed. | ||
| 71 | |||
| 50 | ### 2. Separate Storage for Each Event Type | 72 | ### 2. Separate Storage for Each Event Type |
| 51 | 73 | ||
| 52 | | Store | Index | Purpose | | 74 | | Store | Index | Purpose | |
| @@ -233,6 +255,31 @@ pub struct Purgatory { | |||
| 233 | } | 255 | } |
| 234 | ``` | 256 | ``` |
| 235 | 257 | ||
| 258 | ### Persistence State (Disk Format) | ||
| 259 | |||
| 260 | `Instant` fields cannot be serialized directly. Each entry type has a corresponding `Serializable*` wrapper that stores time fields as `u64` second offsets from a `saved_at: SystemTime` reference point. On restore, elapsed downtime is subtracted to produce the correct remaining TTL. | ||
| 261 | |||
| 262 | ```rust | ||
| 263 | struct PurgatoryState { | ||
| 264 | version: u32, // currently 1 | ||
| 265 | saved_at: SystemTime, // reference for offset math | ||
| 266 | |||
| 267 | /// Non-soft-expired announcements indexed by "owner_hex:identifier" | ||
| 268 | announcement_purgatory: HashMap<String, SerializableAnnouncementPurgatoryEntry>, | ||
| 269 | |||
| 270 | /// State events indexed by repository identifier | ||
| 271 | state_events: HashMap<String, Vec<SerializableStatePurgatoryEntry>>, | ||
| 272 | |||
| 273 | /// PR events (and placeholders) indexed by event ID hex | ||
| 274 | pr_events: HashMap<String, SerializablePrPurgatoryEntry>, | ||
| 275 | |||
| 276 | /// Expired event IDs → approximate expiry SystemTime | ||
| 277 | expired_events: HashMap<String, SystemTime>, | ||
| 278 | } | ||
| 279 | ``` | ||
| 280 | |||
| 281 | The `announcement_purgatory` field uses `#[serde(default)]` so that state files written before announcement persistence was added (version 1 without the field) still deserialize correctly. | ||
| 282 | |||
| 236 | --- | 283 | --- |
| 237 | 284 | ||
| 238 | ## Announcement Purgatory Flows | 285 | ## Announcement Purgatory Flows |
| @@ -806,8 +853,9 @@ A background timer (`run_purgatory_announcement_sync`, every 5 seconds) ensures | |||
| 806 | ``` | 853 | ``` |
| 807 | src/ | 854 | src/ |
| 808 | ├── purgatory/ | 855 | ├── purgatory/ |
| 809 | │ ├── mod.rs # Main Purgatory struct and API | 856 | │ ├── mod.rs # Main Purgatory struct, API, save_to_disk, restore_from_disk |
| 810 | │ ├── types.rs # RefPair, AnnouncementPurgatoryEntry, StatePurgatoryEntry, PrPurgatoryEntry | 857 | │ ├── types.rs # RefPair, AnnouncementPurgatoryEntry, StatePurgatoryEntry, PrPurgatoryEntry |
| 858 | │ ├── persistence.rs # instant_to_offset / offset_to_instant time conversion utilities | ||
| 811 | │ ├── helpers.rs # Ref extraction and matching functions | 859 | │ ├── helpers.rs # Ref extraction and matching functions |
| 812 | │ └── sync/ | 860 | │ └── sync/ |
| 813 | │ ├── mod.rs # Sync module exports | 861 | │ ├── mod.rs # Sync module exports |
| @@ -835,7 +883,8 @@ src/ | |||
| 835 | 883 | ||
| 836 | Located in each module: | 884 | Located in each module: |
| 837 | 885 | ||
| 838 | - **[`src/purgatory/mod.rs`](../../src/purgatory/mod.rs)** - Core purgatory operations including announcement purgatory | 886 | - **[`src/purgatory/mod.rs`](../../src/purgatory/mod.rs)** - Core purgatory operations including announcement purgatory; persistence round-trip tests for all entry types (state, PR, announcement, expired events, downtime calculation, soft-expired exclusion, missing-repo skip) |
| 887 | - **[`src/purgatory/persistence.rs`](../../src/purgatory/persistence.rs)** - `instant_to_offset` / `offset_to_instant` round-trip tests | ||
| 839 | - **[`src/purgatory/helpers.rs`](../../src/purgatory/helpers.rs)** - Ref matching logic | 888 | - **[`src/purgatory/helpers.rs`](../../src/purgatory/helpers.rs)** - Ref matching logic |
| 840 | - **[`src/purgatory/sync/functions.rs`](../../src/purgatory/sync/functions.rs)** - Sync functions with MockSyncContext | 889 | - **[`src/purgatory/sync/functions.rs`](../../src/purgatory/sync/functions.rs)** - Sync functions with MockSyncContext |
| 841 | - **[`src/purgatory/sync/throttle.rs`](../../src/purgatory/sync/throttle.rs)** - Throttle manager | 890 | - **[`src/purgatory/sync/throttle.rs`](../../src/purgatory/sync/throttle.rs)** - Throttle manager |
| @@ -852,6 +901,7 @@ Located in [`tests/`](../../tests/): | |||
| 852 | - **Git-data-first flow** - Git push creates placeholder, event completes it | 901 | - **Git-data-first flow** - Git push creates placeholder, event completes it |
| 853 | - **Authorization with purgatory** - Push authorized by purgatory state | 902 | - **Authorization with purgatory** - Push authorized by purgatory state |
| 854 | - **Background sync** - Sync fetches git data and releases events | 903 | - **Background sync** - Sync fetches git data and releases events |
| 904 | - **Persistence across restart** - Save/restore cycle preserves all entry types including announcements | ||
| 855 | 905 | ||
| 856 | --- | 906 | --- |
| 857 | 907 | ||
| @@ -894,6 +944,14 @@ PR events can arrive before or after git data: | |||
| 894 | 944 | ||
| 895 | **Solution:** `PrPurgatoryEntry.event: Option<Event>` with `None` = placeholder. | 945 | **Solution:** `PrPurgatoryEntry.event: Option<Event>` with `None` = placeholder. |
| 896 | 946 | ||
| 947 | ### 6. Persistence Requires Instant → Duration Conversion | ||
| 948 | |||
| 949 | `std::time::Instant` is not serializable and is not meaningful across process boundaries. Expiry deadlines must be converted to a portable form. | ||
| 950 | |||
| 951 | **Solution:** Store each deadline as a `u64` second offset from a `saved_at: SystemTime` reference. On restore, subtract elapsed downtime from each offset to compute the new `Instant`. Entries whose deadline already passed during downtime get `expires_at = now` and are swept by the next cleanup tick. | ||
| 952 | |||
| 953 | **Soft-expired announcements are excluded from persistence** because their bare repos have been deleted. Restoring them would leave purgatory entries pointing at non-existent repos. They are simply dropped; background sync will re-fetch the announcement event if needed. | ||
| 954 | |||
| 897 | --- | 955 | --- |
| 898 | 956 | ||
| 899 | ## Related Documentation | 957 | ## Related Documentation |