1 files changed, 92 insertions, 36 deletions
diff --git a/docs/explanation/announcements-purgatory-design.md b/docs/explanation/announcements-purgatory-design.md
index 4d0cc6d..a06a8b2 100644
--- a/docs/explanation/announcements-purgatory-design.md
+++ b/docs/explanation/announcements-purgatory-design.md
@@ -2,13 +2,13 @@
 ## Problem Statement
-**Primary problem:** serving an announcement event and also an empty bare git repos mislead clients into thinking we host content.
+**Primary problem:** Serving announcement events alongside empty bare git repos misleads clients into thinking we host content.
 When an announcement arrives, we must create the bare repo immediately (so git pushes can succeed). But if no git data ever arrives, we serve an empty repo and its announcement indefinitely. Clients see the announcement, try to clone, and get nothing. This is misleading.
-**Secondary problem:** Sync downloads refs to deleted repos.
+**Secondary problem:** Sync downloads events for repos that may never have content.
-When a repo expires or is cleaned up, sync may still try to download state event refs to it. We need announcements to remain in a holding state until git data proves the repo has content worth serving.
+Without purgatory, sync would fetch all L2/L3 events (patches, issues, etc.) for announcements that may never receive git data. This wastes bandwidth and creates orphaned events.
 ## Solution Overview
@@ -57,15 +57,37 @@ This ensures we only serve announcements for repos that actually have content.
 **Why:** Prevents premature expiry during slow sync operations or multi-step pushes.
-### 5. State Events Consider Purgatory Announcements
+### 5. Authorization Must Check Purgatory Announcements
-**Decision:** When validating state events, check purgatory announcements for authorization.
+**Decision:** When validating state events or git operations, check purgatory announcements in addition to the database.
-**Why:** State events may arrive before git data promotes the announcement. They still need authorization from the announcement's maintainer set.
+**Why:** State events and git pushes may arrive before git data promotes the announcement. They still need authorization from the announcement's maintainer set.
-### 6. We need to request State Events in sysc for announcement in purgatory but not other l2 or l3 events because they will be rejected.
+**Where:** `fetch_repository_data()` and related authorization functions must query both DB and purgatory.
-### 7. When creating the authorised maintainers for a repositoriy we need to also get relivant announcement events from purgatory as well as db.
+### 6. Sync Only State Events for Purgatory Announcements
+**Decision:** Purgatory announcements trigger sync for state events only, not other L2/L3 events (patches, issues, PRs, etc.).
+**Why:** Other L2/L3 events would be rejected anyway (no promoted announcement in DB). Syncing them wastes bandwidth and creates work for announcements that may never promote.
+**How:** Sync uses a `SyncLevel` concept - `Full` for promoted repos, `StateOnly` for purgatory. On promotion, upgrade to `Full`.
+### 7. Soft Expiry Preserves Event Without Bare Repo
+**Decision:** When a purgatory announcement expires, delete the bare repo but retain the announcement event for an extended period (e.g., 24h).
+**Why:** This handles the case where a state event arrives after initial expiry. Without soft expiry, we'd either:
+- Add to `failed_events` and reject the state event (losing potential revival)
+- Re-fetch the announcement repeatedly (wasting sync bandwidth)
+**Behavior during soft expiry:**
+- Bare repo is deleted (saves disk space)
+- Announcement event retained in purgatory with `soft_expired` flag
+- Sync continues requesting state events (same as active purgatory)
+- If state event arrives: recreate bare repo, clear `soft_expired`, extend expiry
+- If announcement republished directly to us: treat as fresh arrival
+- After extended expiry: fully remove from purgatory
 ## Data Structure
@@ -78,13 +100,14 @@ pub struct AnnouncementPurgatoryEntry {
    pub identifier: String,
    pub owner: PublicKey,
    pub repo_path: PathBuf,
+    pub relays: HashSet<String>,  // For sync registration
    pub created_at: Instant,
    pub expires_at: Instant,
+    pub soft_expired: bool,       // Bare repo deleted, event retained
 }
 ```
-**Indexed by `(pubkey, identifier)`** because identifier is not unique across different owners.
+**Indexed by `(pubkey, identifier)`** because identifier is not unique across different owners. Lookups are primarily from nostr events which have pubkey and identifier readily available.
-question: would it be more efficent to index by repo_path? this contains the pubkey and identifier data?
 ## Flows
@@ -138,27 +161,51 @@ Is there an active announcement?
 ## Edge Cases
-| Scenario                                               | Behavior                                                                                                                               |
+| Scenario                                               | Behavior                                                                                                |
-| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
+| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
-| Git data before announcement                           | Push fails (no repo exists)                                                                                                            |
+| Git data before announcement                           | Push fails (no repo exists)                                                                             |
-| Announcement expires, no git data                      | Delete bare repo, discard announcement                                                                                                 |
+| Announcement expires, no git data                      | Delete bare repo, set `soft_expired` flag, retain event for extended period                             |
-| State expires, announcement in purgatory               | Announcement keeps its own expiry                                                                                                      |
+| Soft-expired announcement fully expires                | Remove from purgatory entirely                                                                          |
-| Multiple owners, same identifier                       | Each tracked separately by `(pubkey, identifier)`                                                                                      |
+| State event arrives for soft-expired announcement      | Recreate bare repo, clear `soft_expired`, extend expiry                                                 |
-| **Newer announcement replaces older (same pubkey)**    | Replace purgatory entry, extend expiry, and state event expiry                                                                         |
+| State expires, announcement in purgatory               | Announcement keeps its own expiry                                                                       |
-| **Newer announcement changes services (unacceptable)** | Clear older announcement from purgatory for that `(pubkey, identifier)`, delete bare repo, remove state event for puragatory if exists |
+| Multiple owners, same identifier                       | Each tracked separately by `(pubkey, identifier)`                                                       |
-| Deletion event for purgatory announcement              | Remove from purgatory, delete bare repo                                                                                                |
+| **Newer announcement replaces older (same pubkey)**    | Replace purgatory entry, extend expiry, and state event expiry                                          |
+| **Newer announcement changes services (unacceptable)** | Clear older announcement from purgatory, delete bare repo, remove state events from purgatory if exists |
+| Deletion event for purgatory announcement              | Remove from purgatory, delete bare repo                                                                 |
+## Purgatory Lifecycle
-## Purgatory Exit Conditions
+An announcement progresses through purgatory states:
-An announcement leaves purgatory via:
+```
+                    ┌─────────────────────────────────────┐
+                    │                                     │
+                    v                                     │
+Announcement ──> ACTIVE ──────────────────────────────────┤
+  arrives        (bare repo exists)                       │
+                    │                                     │
+                    ├── Git data ──> PROMOTED (exit)      │
+                    │                                     │
+                    ├── Deletion ──> REMOVED (exit)       │
+                    │                                     │
+                    v                                     │
+               SOFT_EXPIRED ──────────────────────────────┘
+               (bare repo deleted,        ^
+                event retained)           │
+                    │                     │
+                    ├── State event arrives (revival)
+                    │
+                    └── Extended expiry ──> REMOVED (exit)
+```
-| Exit               | Trigger                                        | Action                             |
+| Exit           | Trigger                                        | Action                                       |
-| ------------------ | ---------------------------------------------- | ---------------------------------- |
+| -------------- | ---------------------------------------------- | -------------------------------------------- |
-| **Promotion**      | Git data arrives                               | Move to database, serve to clients |
+| **Promotion**  | Git data arrives                               | Move to database, upgrade sync to Full       |
-| **Expiry**         | Timeout                                        | Delete bare repo, discard          |
+| **Soft expiry**| Initial timeout                                | Delete bare repo, retain event, continue sync|
-| **Deletion**       | Kind 5 event                                   | Delete bare repo, discard          |
+| **Full expiry**| Extended timeout (soft-expired)                | Remove from purgatory entirely               |
-| **Replacement**    | Newer announcement (same pubkey, identifier)   | Replace entry                      |
+| **Deletion**   | Kind 5 event                                   | Delete bare repo, remove from purgatory      |
-| **Service change** | Newer announcement no longer lists our service | Discard old entry                  |
+| **Replacement**| Newer announcement (same pubkey, identifier)   | Replace entry                                |
+| **Service change** | Newer announcement removes our service     | Remove from purgatory                        |
 ## Integration Points
@@ -169,24 +216,33 @@ An announcement leaves purgatory via:
 | `src/nostr/policy/announcement.rs` | Route new announcements to purgatory                      |
 | `src/git/receive.rs`               | Promote on git data arrival                               |
 | `src/git/auth.rs`                  | Extend purgatory expiry when extending state event expiry |
+| `src/git/authorization.rs`         | Check purgatory announcements for maintainer authorization|
 | `src/nostr/policy/state.rs`        | Check purgatory for authorization                         |
+| `src/sync/mod.rs`                  | Add `SyncLevel` to `RepoSyncNeeds`                        |
+| `src/sync/filters.rs`              | Respect sync level when building filters                  |
+| `src/sync/self_subscriber.rs`      | Register purgatory announcements with `StateOnly` level   |
+See [announcements-purgatory-implementation.md](./announcements-purgatory-implementation.md) for detailed implementation notes.
 ## Testing
 - Announcement to purgatory, git data promotes it
- Announcement expires without git data (repo deleted)
+- Announcement soft-expires without git data (repo deleted, event retained)
+- State event revives soft-expired announcement (repo recreated)
+- Soft-expired announcement fully expires after extended period
 - State event extends purgatory expiry
 - Git auth extends purgatory expiry
 - Newer announcement replaces older in purgatory
 - Service change clears purgatory entry
 - `(pubkey, identifier)` indexing with multiple owners
+- Sync requests only state events for purgatory announcements
+- Sync upgrades to full on promotion
 ## Risks
-| Risk                                 | Mitigation                           |
+| Risk                                 | Mitigation                                              |
-| ------------------------------------ | ------------------------------------ |
+| ------------------------------------ | ------------------------------------------------------- |
-| Disk exhaustion from purgatory repos | Short expiry, monitor purgatory size |
+| Disk exhaustion from purgatory repos | Short expiry, soft expiry deletes repo early            |
-| Race between promotion and expiry    | Atomic operations                    |
+| Race between promotion and expiry    | Atomic operations                                       |
-| Sync re-fetching expired events      | Track expired event IDs              |
+| Sync re-fetching expired events      | Soft expiry retains event; no need for `failed_events`  |
+| Filter explosion from many purgatory | Existing consolidation handles this (threshold at 70)   |
-question: do expired annoucements go on failed_events list? what if a new state event comes in for it? surely then we want it again? but if not we dont want to keep donwloading it and havea a repo made for it. Should we have a longer period were we keep the event just in case, but delete the bare repo and only remake it when the state event arrives?

diff --git a/docs/explanation/announcements-purgatory-design.md b/docs/explanation/announcements-purgatory-design.md index 4d0cc6d..a06a8b2 100644 --- a/docs/explanation/announcements-purgatory-design.md +++ b/docs/explanation/announcements-purgatory-design.md
@@ -2,13 +2,13 @@
2		2
3	## Problem Statement	3	## Problem Statement
4		4
5	Primary problem: serving an announcement event and also an empty bare git repos mislead clients into thinking we host content.	5	Primary problem: Serving announcement events alongside empty bare git repos misleads clients into thinking we host content.
6		6
7	When an announcement arrives, we must create the bare repo immediately (so git pushes can succeed). But if no git data ever arrives, we serve an empty repo and its announcement indefinitely. Clients see the announcement, try to clone, and get nothing. This is misleading.	7	When an announcement arrives, we must create the bare repo immediately (so git pushes can succeed). But if no git data ever arrives, we serve an empty repo and its announcement indefinitely. Clients see the announcement, try to clone, and get nothing. This is misleading.
8		8
9	Secondary problem: Sync downloads refs to deleted repos.	9	Secondary problem: Sync downloads events for repos that may never have content.
10		10
11	When a repo expires or is cleaned up, sync may still try to download state event refs to it. We need announcements to remain in a holding state until git data proves the repo has content worth serving.	11	Without purgatory, sync would fetch all L2/L3 events (patches, issues, etc.) for announcements that may never receive git data. This wastes bandwidth and creates orphaned events.
12		12
13	## Solution Overview	13	## Solution Overview
14		14
@@ -57,15 +57,37 @@ This ensures we only serve announcements for repos that actually have content.
57		57
58	Why: Prevents premature expiry during slow sync operations or multi-step pushes.	58	Why: Prevents premature expiry during slow sync operations or multi-step pushes.
59		59
60	### 5. State Events Consider Purgatory Announcements	60	### 5. Authorization Must Check Purgatory Announcements
61		61
62	Decision: When validating state events, check purgatory announcements for authorization.	62	Decision: When validating state events or git operations, check purgatory announcements in addition to the database.
63		63
64	Why: State events may arrive before git data promotes the announcement. They still need authorization from the announcement's maintainer set.	64	Why: State events and git pushes may arrive before git data promotes the announcement. They still need authorization from the announcement's maintainer set.
65		65
66	### 6. We need to request State Events in sysc for announcement in purgatory but not other l2 or l3 events because they will be rejected.	66	Where: `fetch_repository_data()` and related authorization functions must query both DB and purgatory.
67		67
68	### 7. When creating the authorised maintainers for a repositoriy we need to also get relivant announcement events from purgatory as well as db.	68	### 6. Sync Only State Events for Purgatory Announcements
		69
		70	Decision: Purgatory announcements trigger sync for state events only, not other L2/L3 events (patches, issues, PRs, etc.).
		71
		72	Why: Other L2/L3 events would be rejected anyway (no promoted announcement in DB). Syncing them wastes bandwidth and creates work for announcements that may never promote.
		73
		74	How: Sync uses a `SyncLevel` concept - `Full` for promoted repos, `StateOnly` for purgatory. On promotion, upgrade to `Full`.
		75
		76	### 7. Soft Expiry Preserves Event Without Bare Repo
		77
		78	Decision: When a purgatory announcement expires, delete the bare repo but retain the announcement event for an extended period (e.g., 24h).
		79
		80	Why: This handles the case where a state event arrives after initial expiry. Without soft expiry, we'd either:
		81	- Add to `failed_events` and reject the state event (losing potential revival)
		82	- Re-fetch the announcement repeatedly (wasting sync bandwidth)
		83
		84	Behavior during soft expiry:
		85	- Bare repo is deleted (saves disk space)
		86	- Announcement event retained in purgatory with `soft_expired` flag
		87	- Sync continues requesting state events (same as active purgatory)
		88	- If state event arrives: recreate bare repo, clear `soft_expired`, extend expiry
		89	- If announcement republished directly to us: treat as fresh arrival
		90	- After extended expiry: fully remove from purgatory
69		91
70	## Data Structure	92	## Data Structure
71		93
@@ -78,13 +100,14 @@ pub struct AnnouncementPurgatoryEntry {
78	pub identifier: String,	100	pub identifier: String,
79	pub owner: PublicKey,	101	pub owner: PublicKey,
80	pub repo_path: PathBuf,	102	pub repo_path: PathBuf,
		103	pub relays: HashSet<String>, // For sync registration
81	pub created_at: Instant,	104	pub created_at: Instant,
82	pub expires_at: Instant,	105	pub expires_at: Instant,
		106	pub soft_expired: bool, // Bare repo deleted, event retained
83	}	107	}
84	```	108	```
85		109
86	Indexed by `(pubkey, identifier)` because identifier is not unique across different owners.	110	Indexed by `(pubkey, identifier)` because identifier is not unique across different owners. Lookups are primarily from nostr events which have pubkey and identifier readily available.
87	question: would it be more efficent to index by repo_path? this contains the pubkey and identifier data?
88		111
89	## Flows	112	## Flows
90		113
@@ -138,27 +161,51 @@ Is there an active announcement?
138		161
139	## Edge Cases	162	## Edge Cases
140		163
141	\| Scenario \| Behavior \|	164	\| Scenario \| Behavior \|
142	\| ------------------------------------------------------ \| -------------------------------------------------------------------------------------------------------------------------------------- \|	165	\| ------------------------------------------------------ \| ------------------------------------------------------------------------------------------------------- \|
143	\| Git data before announcement \| Push fails (no repo exists) \|	166	\| Git data before announcement \| Push fails (no repo exists) \|
144	\| Announcement expires, no git data \| Delete bare repo, discard announcement \|	167	\| Announcement expires, no git data \| Delete bare repo, set `soft_expired` flag, retain event for extended period \|
145	\| State expires, announcement in purgatory \| Announcement keeps its own expiry \|	168	\| Soft-expired announcement fully expires \| Remove from purgatory entirely \|
146	\| Multiple owners, same identifier \| Each tracked separately by `(pubkey, identifier)` \|	169	\| State event arrives for soft-expired announcement \| Recreate bare repo, clear `soft_expired`, extend expiry \|
147	\| Newer announcement replaces older (same pubkey) \| Replace purgatory entry, extend expiry, and state event expiry \|	170	\| State expires, announcement in purgatory \| Announcement keeps its own expiry \|
148	\| Newer announcement changes services (unacceptable) \| Clear older announcement from purgatory for that `(pubkey, identifier)`, delete bare repo, remove state event for puragatory if exists \|	171	\| Multiple owners, same identifier \| Each tracked separately by `(pubkey, identifier)` \|
149	\| Deletion event for purgatory announcement \| Remove from purgatory, delete bare repo \|	172	\| Newer announcement replaces older (same pubkey) \| Replace purgatory entry, extend expiry, and state event expiry \|
		173	\| Newer announcement changes services (unacceptable) \| Clear older announcement from purgatory, delete bare repo, remove state events from purgatory if exists \|
		174	\| Deletion event for purgatory announcement \| Remove from purgatory, delete bare repo \|
		175
		176	## Purgatory Lifecycle
150		177
151	## Purgatory Exit Conditions	178	An announcement progresses through purgatory states:
152		179
153	An announcement leaves purgatory via:	180	```
		181	┌─────────────────────────────────────┐
		182	│ │
		183	v │
		184	Announcement ──> ACTIVE ──────────────────────────────────┤
		185	arrives (bare repo exists) │
		186	│ │
		187	├── Git data ──> PROMOTED (exit) │
		188	│ │
		189	├── Deletion ──> REMOVED (exit) │
		190	│ │
		191	v │
		192	SOFT_EXPIRED ──────────────────────────────┘
		193	(bare repo deleted, ^
		194	event retained) │
		195	│ │
		196	├── State event arrives (revival)
		197	│
		198	└── Extended expiry ──> REMOVED (exit)
		199	```
154		200
155	\| Exit \| Trigger \| Action \|	201	\| Exit \| Trigger \| Action \|
156	\| ------------------ \| ---------------------------------------------- \| ---------------------------------- \|	202	\| -------------- \| ---------------------------------------------- \| -------------------------------------------- \|
157	\| Promotion \| Git data arrives \| Move to database, serve to clients \|	203	\| Promotion \| Git data arrives \| Move to database, upgrade sync to Full \|
158	\| Expiry \| Timeout \| Delete bare repo, discard \|	204	\| Soft expiry\| Initial timeout \| Delete bare repo, retain event, continue sync\|
159	\| Deletion \| Kind 5 event \| Delete bare repo, discard \|	205	\| Full expiry\| Extended timeout (soft-expired) \| Remove from purgatory entirely \|
160	\| Replacement \| Newer announcement (same pubkey, identifier) \| Replace entry \|	206	\| Deletion \| Kind 5 event \| Delete bare repo, remove from purgatory \|
161	\| Service change \| Newer announcement no longer lists our service \| Discard old entry \|	207	\| Replacement\| Newer announcement (same pubkey, identifier) \| Replace entry \|
		208	\| Service change \| Newer announcement removes our service \| Remove from purgatory \|
162		209
163	## Integration Points	210	## Integration Points
164		211
@@ -169,24 +216,33 @@ An announcement leaves purgatory via:
169	\| `src/nostr/policy/announcement.rs` \| Route new announcements to purgatory \|	216	\| `src/nostr/policy/announcement.rs` \| Route new announcements to purgatory \|
170	\| `src/git/receive.rs` \| Promote on git data arrival \|	217	\| `src/git/receive.rs` \| Promote on git data arrival \|
171	\| `src/git/auth.rs` \| Extend purgatory expiry when extending state event expiry \|	218	\| `src/git/auth.rs` \| Extend purgatory expiry when extending state event expiry \|
		219	\| `src/git/authorization.rs` \| Check purgatory announcements for maintainer authorization\|
172	\| `src/nostr/policy/state.rs` \| Check purgatory for authorization \|	220	\| `src/nostr/policy/state.rs` \| Check purgatory for authorization \|
		221	\| `src/sync/mod.rs` \| Add `SyncLevel` to `RepoSyncNeeds` \|
		222	\| `src/sync/filters.rs` \| Respect sync level when building filters \|
		223	\| `src/sync/self_subscriber.rs` \| Register purgatory announcements with `StateOnly` level \|
		224
		225	See [announcements-purgatory-implementation.md](./announcements-purgatory-implementation.md) for detailed implementation notes.
173		226
174	## Testing	227	## Testing
175		228
176	- Announcement to purgatory, git data promotes it	229	- Announcement to purgatory, git data promotes it
177	- Announcement expires without git data (repo deleted)	230	- Announcement soft-expires without git data (repo deleted, event retained)
		231	- State event revives soft-expired announcement (repo recreated)
		232	- Soft-expired announcement fully expires after extended period
178	- State event extends purgatory expiry	233	- State event extends purgatory expiry
179	- Git auth extends purgatory expiry	234	- Git auth extends purgatory expiry
180	- Newer announcement replaces older in purgatory	235	- Newer announcement replaces older in purgatory
181	- Service change clears purgatory entry	236	- Service change clears purgatory entry
182	- `(pubkey, identifier)` indexing with multiple owners	237	- `(pubkey, identifier)` indexing with multiple owners
		238	- Sync requests only state events for purgatory announcements
		239	- Sync upgrades to full on promotion
183		240
184	## Risks	241	## Risks
185		242
186	\| Risk \| Mitigation \|	243	\| Risk \| Mitigation \|
187	\| ------------------------------------ \| ------------------------------------ \|	244	\| ------------------------------------ \| ------------------------------------------------------- \|
188	\| Disk exhaustion from purgatory repos \| Short expiry, monitor purgatory size \|	245	\| Disk exhaustion from purgatory repos \| Short expiry, soft expiry deletes repo early \|
189	\| Race between promotion and expiry \| Atomic operations \|	246	\| Race between promotion and expiry \| Atomic operations \|
190	\| Sync re-fetching expired events \| Track expired event IDs \|	247	\| Sync re-fetching expired events \| Soft expiry retains event; no need for `failed_events` \|
191		248	\| Filter explosion from many purgatory \| Existing consolidation handles this (threshold at 70) \|
192	question: do expired annoucements go on failed_events list? what if a new state event comes in for it? surely then we want it again? but if not we dont want to keep donwloading it and havea a repo made for it. Should we have a longer period were we keep the event just in case, but delete the bare repo and only remake it when the state event arrives?