upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/docs/explanation
diff options
context:
space:
mode:
Diffstat (limited to 'docs/explanation')
-rw-r--r--docs/explanation/architecture.md64
-rw-r--r--docs/explanation/grasp-02-proactive-sync.md35
-rw-r--r--docs/explanation/inline-authorization.md93
-rw-r--r--docs/explanation/monitoring.md96
4 files changed, 287 insertions, 1 deletions
diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md
index 6da2295..d2a9bf7 100644
--- a/docs/explanation/architecture.md
+++ b/docs/explanation/architecture.md
@@ -239,6 +239,22 @@ pub struct RepositoryAnnouncement { ... }
239pub struct RepositoryState { ... } 239pub struct RepositoryState { ... }
240``` 240```
241 241
242#### [`policy/state.rs`](src/nostr/policy/state.rs) - State Event Authorization
243
244State events undergo authorization checks at multiple points:
245
246```rust
247/// State event authorization checks:
248/// 1. Announcement must exist for the repository identifier
249/// 2. Author must be in maintainer set of accepted announcement
250/// 3. Validated on arrival, announcement acceptance, and git data arrival
251```
252
253**Defense-in-depth authorization:**
254- **On arrival** (StatePolicy): Initial authorization check
255- **On announcement acceptance**: Purgatory re-evaluation of waiting state events
256- **On git data arrival**: Final authorization before database save
257
242### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/)) 258### 5. Purgatory System ([`src/purgatory/`](../../src/purgatory/))
243 259
244The purgatory system solves the "which arrives first?" problem where either nostr events or git pushes can arrive in any order. It provides an in-memory holding area for events and git data awaiting their counterparts. 260The purgatory system solves the "which arrives first?" problem where either nostr events or git pushes can arrive in any order. It provides an in-memory holding area for events and git data awaiting their counterparts.
@@ -457,6 +473,7 @@ The ngit-grasp relay implements **Proactive Sync of Nostr Eevents**, which synch
457- **Health tracking** with exponential backoff for failing relays 473- **Health tracking** with exponential backoff for failing relays
458- **Daily sync** with random 23-25h timer to detect state drift 474- **Daily sync** with random 23-25h timer to detect state drift
459- **Filter consolidation** when count exceeds 70 to prevent subscription explosion 475- **Filter consolidation** when count exceeds 70 to prevent subscription explosion
476- **Rejected events index** - prevents wasteful re-fetching during negentropy sync
460 477
461**Architecture:** 478**Architecture:**
462 479
@@ -479,6 +496,14 @@ The ngit-grasp relay implements **Proactive Sync of Nostr Eevents**, which synch
479│ │ Health Tracker │ Exponential backoff, dead detection │ 496│ │ Health Tracker │ Exponential backoff, dead detection │
480│ │ (DashMap) │ │ 497│ │ (DashMap) │ │
481│ └──────────────────┘ │ 498│ └──────────────────┘ │
499│ ┌──────────────────────────────────────────────────────┐ │
500│ │ Rejected Events Index (Two-Tier) │ │
501│ │ ┌────────────────┐ ┌──────────────────────┐ │ │
502│ │ │ Hot Cache │───▶│ Cold Index │ │ │
503│ │ │ (2 min) │ │ (7 days) │ │ │
504│ │ │ Full events │ │ Metadata only │ │ │
505│ │ └────────────────┘ └──────────────────────┘ │ │
506│ └──────────────────────────────────────────────────────┘ │
482└─────────────────────────────────────────────────────────────┘ 507└─────────────────────────────────────────────────────────────┘
483``` 508```
484 509
@@ -486,6 +511,45 @@ The ngit-grasp relay implements **Proactive Sync of Nostr Eevents**, which synch
486 511
487For full design details, see [grasp-02-proactive-sync-v4.md](grasp-02-proactive-sync-v4.md). 512For full design details, see [grasp-02-proactive-sync-v4.md](grasp-02-proactive-sync-v4.md).
488 513
514### Rejected Events Index
515
516The rejected events index solves two critical problems during sync:
517
5181. **Negentropy sync efficiency**: Prevents repeatedly downloading events that will be rejected again
5192. **Race condition resolution**: Enables immediate re-processing when event dependencies are satisfied
520
521**Two-Tier Architecture:**
522
523| Tier | Duration | Storage | Purpose |
524|------|----------|---------|---------|
525| Hot Cache | 2 minutes | Full events | Immediate re-processing when dependencies arrive |
526| Cold Index | 7 days | Metadata only | Prevent re-fetch during negentropy sync |
527
528**Event Flow:**
529
530```
531Event Rejected (e.g., maintainer before owner announcement)
532
533 ├──▶ Store full event in Hot Cache (2 min expiry)
534 └──▶ Store metadata in Cold Index (7 day expiry)
535
536Dependency Arrives (e.g., owner announcement accepted)
537
538 ├──▶ Invalidate from Cold Index
539 ├──▶ Retrieve from Hot Cache (if still available)
540 └──▶ Re-process immediately (<1 second vs 24 hours)
541
542Negentropy Sync
543
544 └──▶ Exclude Cold Index IDs from "missing events" calculation
545```
546
547**Tracked Events:**
548- Repository announcements (kind 30617) rejected for not listing this service or maintainer validation failure
549- State events (kind 30618) rejected for missing announcements or unauthorized authors
550
551**Source Code:** [`src/sync/rejected_index.rs`](../../src/sync/rejected_index.rs)
552
489## Future Extensions 553## Future Extensions
490 554
491### GRASP-02: Proactive Sync 555### GRASP-02: Proactive Sync
diff --git a/docs/explanation/grasp-02-proactive-sync.md b/docs/explanation/grasp-02-proactive-sync.md
index e983316..ed8fdbf 100644
--- a/docs/explanation/grasp-02-proactive-sync.md
+++ b/docs/explanation/grasp-02-proactive-sync.md
@@ -729,6 +729,41 @@ If negentropy fails (relay doesn't support NIP-77, network error, etc.):
7292. The sync falls back to traditional REQ+EOSE 7292. The sync falls back to traditional REQ+EOSE
7303. No error is raised - fallback is automatic 7303. No error is raised - fallback is automatic
731 731
732### Integration with Rejected Events Index
733
734The rejected events index prevents wasteful re-fetching during negentropy sync by excluding rejected event IDs from the reconciliation process:
735
736**During Negentropy Reconciliation:**
737
7381. **Build "already have" set**: Combine event IDs from:
739 - Events in database
740 - Events in purgatory
741 - **Events in rejected index (hot cache + cold index)**
742
7432. **Send to negentropy**: This combined set represents "events we already have or don't want"
744
7453. **Receive differences**: Relay only sends events we don't have and haven't rejected
746
7474. **Process received events**: New events go through normal validation:
748 - If accepted → saved to database
749 - If rejected → added to rejected index
750 - If waiting for dependencies → added to purgatory
751
752**Why This Matters:**
753
754Without the rejected events index, negentropy would repeatedly download events that don't list this service or are from unauthorized maintainers, wasting bandwidth on every sync cycle.
755
756**Re-Processing on Dependency Arrival:**
757
758When a dependency is satisfied (e.g., owner announcement accepted):
7591. Related entries are **invalidated** (removed) from cold index
7602. If event still in hot cache → immediate re-processing
7613. If event expired from hot cache → will be re-fetched on next sync (now that dependency exists)
762
763This prevents permanently excluding events that could become valid after dependencies arrive.
764
765See [work/rejected-events-index-summary.md](../../work/rejected-events-index-summary.md) for complete implementation details.
766
732--- 767---
733 768
734## REQ+EOSE Pagination 769## REQ+EOSE Pagination
diff --git a/docs/explanation/inline-authorization.md b/docs/explanation/inline-authorization.md
index a71a217..7081f63 100644
--- a/docs/explanation/inline-authorization.md
+++ b/docs/explanation/inline-authorization.md
@@ -352,6 +352,99 @@ pub async fn authorize_push(
352 - If no event found, create placeholder (git-data-first scenario) 352 - If no event found, create placeholder (git-data-first scenario)
353 - Collect PR events from purgatory for post-push processing 353 - Collect PR events from purgatory for post-push processing
354 354
355## State Event Authorization
356
357State events (kind 30618) undergo authorization checks at three points (defense-in-depth):
358
359### 1. On Arrival (StatePolicy)
360
361When a state event arrives via WebSocket or sync:
362
363```rust
364// src/nostr/policy/state.rs
365impl StatePolicy {
366 async fn admit_event(&self, event: &Event) -> Result<Decision, Error> {
367 // Check 1: Does announcement exist for this repository?
368 let announcements = query_announcements(pubkey, identifier);
369 if announcements.is_empty() {
370 return Reject("No announcement exists for repository");
371 }
372
373 // Check 2: Is author in maintainer set?
374 let maintainers = build_maintainer_set(announcements);
375 if !maintainers.contains(&event.author) {
376 return Reject("Author not in maintainer set");
377 }
378
379 // If git data doesn't exist yet, goes to purgatory
380 // Otherwise, accepted to database
381 }
382}
383```
384
385### 2. On Announcement Acceptance (Purgatory Re-evaluation)
386
387When a repository announcement is accepted, waiting state events are re-evaluated:
388
389```rust
390// After announcement is saved to database
391for state_event in purgatory.get_state_events(identifier) {
392 // Re-check authorization now that announcement exists
393 if author_in_maintainer_set(state_event.author, identifier) {
394 // If git data now exists, save to database
395 // Otherwise, keep in purgatory
396 } else {
397 // Remove from purgatory - not authorized
398 }
399}
400```
401
402### 3. On Git Data Arrival (Purgatory Sync)
403
404When git data is pushed, purgatory state events are validated before saving:
405
406```rust
407// src/git/handlers.rs - after successful git push
408for state_event in purgatory.get_matching_state_events(identifier) {
409 // Final authorization check before database save
410 if author_in_maintainer_set(state_event.author, identifier) {
411 database.save(state_event);
412 purgatory.remove(state_event);
413 } else {
414 purgatory.remove(state_event); // Not authorized
415 }
416}
417```
418
419### Why Three Checkpoints?
420
421**Defense-in-depth** ensures authorization is always validated:
422
4231. **On arrival**: Prevents unauthorized events from entering the system
4242. **On announcement acceptance**: Handles race condition where state arrives before announcement
4253. **On git data arrival**: Final check before committing to database
426
427This prevents scenarios where:
428- Unauthorized state events are saved after maintainer changes
429- Race conditions bypass authorization
430- Purgatory holds events that will never be authorized
431
432### Rejection Tracking
433
434State events rejected during authorization are tracked in the rejected events index:
435
436- **Reason: MaintainerNotYetValid** - Author not in maintainer set (may become valid later)
437- **Reason: Other** - Other validation failures
438
439When a repository announcement is accepted, rejected state events for that repository are:
4401. **Invalidated** from cold index (removed from negentropy exclusion)
4412. **Retrieved** from hot cache (if still available within 2 minutes)
4423. **Re-processed** immediately with new maintainer set
443
444This enables rapid recovery from race conditions where state events arrive before maintainer announcements.
445
446See [work/rejected-events-index-summary.md](../../work/rejected-events-index-summary.md) for complete details on rejection tracking and re-processing.
447
355--- 448---
356 449
357## Comparison with Reference Implementation 450## Comparison with Reference Implementation
diff --git a/docs/explanation/monitoring.md b/docs/explanation/monitoring.md
index 7520813..bd652be 100644
--- a/docs/explanation/monitoring.md
+++ b/docs/explanation/monitoring.md
@@ -204,4 +204,98 @@ sum(ngit_sync_relay_status == 5) # RateLimited
2044. **Restart behavior**: Conservative initial backoff (5s + jitter) avoids thundering herd on restart 2044. **Restart behavior**: Conservative initial backoff (5s + jitter) avoids thundering herd on restart
2055. **Historical data**: Prometheus retains health history; in-memory state only needs current status 2055. **Historical data**: Prometheus retains health history; in-memory state only needs current status
206 206
207See [GRASP-02 Proactive Sync](grasp-02-proactive-sync.md) for full architecture details. \ No newline at end of file 207See [GRASP-02 Proactive Sync](grasp-02-proactive-sync.md) for full architecture details.
208
209## Rejected Events Index Metrics
210
211The rejected events index tracks rejected repository announcements and state events to prevent wasteful re-fetching during negentropy sync and enable race condition resolution.
212
213### Rejected Events Metrics
214
215All metrics are parameterized by `event_type` label with values "announcement" or "state":
216
217| Metric | Type | Labels | Description |
218|--------|------|--------|-------------|
219| `ngit_rejected_hot_cache_current` | Gauge | event_type | Current number of entries in hot cache |
220| `ngit_rejected_cold_index_current` | Gauge | event_type | Current number of entries in cold index |
221| `ngit_rejected_hot_cache_hits` | Counter | event_type | Events successfully retrieved from hot cache for re-processing |
222| `ngit_rejected_hot_cache_misses` | Counter | event_type | Events expired from hot cache before dependency arrived |
223| `ngit_rejected_hot_cache_expired` | Counter | event_type | Entries cleaned up from hot cache (2 min expiry) |
224| `ngit_rejected_cold_index_expired` | Counter | event_type | Entries cleaned up from cold index (7 day expiry) |
225| `ngit_rejected_invalidated` | Counter | event_type | Entries invalidated when dependency was satisfied |
226
227### Example Grafana Queries
228
229```promql
230# Hot cache efficiency - how often we successfully re-process from cache
231rate(ngit_rejected_hot_cache_hits_total[5m])
232/ (rate(ngit_rejected_hot_cache_hits_total[5m]) + rate(ngit_rejected_hot_cache_misses_total[5m]))
233
234# Current rejected events by type
235ngit_rejected_hot_cache_current{event_type="announcement"}
236ngit_rejected_hot_cache_current{event_type="state"}
237ngit_rejected_cold_index_current{event_type="announcement"}
238ngit_rejected_cold_index_current{event_type="state"}
239
240# Race condition resolution rate - invalidations indicate successful dependency arrival
241rate(ngit_rejected_invalidated_total[5m])
242
243# Cache hit ratio over time (higher is better, means dependencies arriving quickly)
244sum(rate(ngit_rejected_hot_cache_hits_total[5m]))
245/ sum(rate(ngit_rejected_hot_cache_hits_total[5m]) + rate(ngit_rejected_hot_cache_misses_total[5m]))
246```
247
248### Example Alerts
249
250```yaml
251# Alert if hot cache hit rate is too low (suggests timing issues)
252- alert: RejectedEventsCacheMissRate
253 expr: |
254 sum(rate(ngit_rejected_hot_cache_misses_total[5m]))
255 / sum(rate(ngit_rejected_hot_cache_hits_total[5m]) + rate(ngit_rejected_hot_cache_misses_total[5m]))
256 > 0.8
257 for: 15m
258 labels:
259 severity: warning
260 annotations:
261 summary: "High rejected events cache miss rate ({{ $value | humanizePercentage }})"
262 description: "Most rejected events are expiring before dependencies arrive"
263
264# Alert if cold index growing too large
265- alert: RejectedEventsColdIndexSize
266 expr: ngit_rejected_cold_index_current > 10000
267 for: 1h
268 labels:
269 severity: info
270 annotations:
271 summary: "Rejected events cold index has {{ $value }} entries"
272 description: "Consider investigating why many events are being rejected"
273```
274
275### Two-Tier Architecture
276
277**Hot Cache (2 minutes):**
278- Stores full event objects
279- Enables immediate re-processing when dependencies arrive
280- Cleaned up every 60 seconds
281- Memory: ~200 KB typical, ~20 MB worst case
282
283**Cold Index (7 days):**
284- Stores metadata only (event ID, pubkey, identifier, reason)
285- Prevents re-downloading during negentropy sync
286- Cleaned up daily
287- Memory: ~1 MB typical
288
289### Use Cases
290
291**Race Condition Resolution:**
292When a maintainer announcement arrives before the owner announcement:
2931. Maintainer event rejected → hot cache + cold index
2942. Owner announcement accepted → invalidate from cold index
2953. If still in hot cache → immediate re-processing (<1 second)
2964. If expired from hot cache → will be re-fetched on next sync
297
298**Negentropy Sync Efficiency:**
299During sync, cold index IDs are excluded from "missing events" calculation, preventing wasteful re-download of events that will be rejected again.
300
301See [work/rejected-events-index-summary.md](../../work/rejected-events-index-summary.md) for complete implementation details. \ No newline at end of file