npub1tkq8unhsd5jqx6ueex5lcpsgknrpquxuk44ftpjlpm3ulaake7xs76txrw/ngit-grasp-mirror/src/sync/rejected_index.rs, branch master

feat(sync): add rejected events cache persistence and integrate with shutdown/startup

2026-01-14T10:19:18+00:00

Implement save/restore functionality for rejected events cache and integrate persistence with relay shutdown/startup lifecycle. Both purgatory and rejected cache now survive relay restarts. Key features: - Serialize rejected events cache to JSON (rejected-events-cache.json) - Save both hot cache (2min, full events) and cold index (7day, metadata) - Restore with downtime adjustment (preserves remaining TTL) - Graceful degradation (missing/corrupted files don't crash) - File cleanup after successful restore - Automatic restoration in SyncManager::new() Integration: - Shutdown hook saves both purgatory and rejected cache - Startup hook restores both and re-queues repositories - Non-fatal errors (logs warnings, continues on failure) Files: - src/sync/rejected_index.rs: save_to_disk/restore_from_disk methods - src/sync/mod.rs: SyncManager integration and auto-restore - src/main.rs: Shutdown/startup hooks for both caches - tests/purgatory_persistence.rs: 17 integration tests Tests: 13 unit tests + 17 integration tests covering full lifecycle

refactor(sync): add EventType enum and unify rejected index methods

2026-01-09T20:51:49+00:00

Add EventType enum (Announcement, State) to distinguish event types within RejectedEventsIndex. This consolidates the two-tier index design into a single unified interface. Changes: - Add EventType enum with Announcement and State variants - Add event_type field to HotCacheEntry and ColdIndexEntry - Create unified invalidate_and_get() with optional event_type filter - Update cleanup_expired_for_type() to handle both types - Remove deprecated wrapper methods (invalidate_and_get_events, invalidate_and_get_state_events, cleanup_expired, cleanup_states_expired) Consolidates phases 2, 3, and 7 of rejected events index refactoring.

chore: cargo fmt

2026-01-09T19:58:41+00:00

feat(sync): fix race condition with announcement-before-state event ordering

2026-01-09T19:18:46+00:00

**Problem:** Integration test `test_concurrent_state_and_pr_sync` was timing out because of a race condition: when syncing from remote relays, state events can arrive BEFORE their announcements (no ordering guarantee). The system was rejecting these state events with "no announcement exists" but NOT tracking them for re-processing when the announcement later arrived. **Solution:** Implemented announcement → state event re-processing (GRASP-02 PR4.1) to handle the race condition, mirroring the existing maintainer announcement re-processing logic (GRASP-02 PR3). **What Changed:** 1. **Announcement → State Event Re-processing (GRASP-02 PR4.1)**: When a repository announcement is accepted, the system now invalidates and re-processes state events that were rejected with "no announcement exists". This ensures state events arriving before their announcements are eventually processed correctly. 2. **State Event → State Event Re-processing (GRASP-02 PR4.2)**: When a state event is accepted (git data arrives), the system invalidates and re-processes other rejected state events for the same repository from the hot cache. (Renamed from PR4 for clarity - this was already implemented in previous commit) 3. **Proper Rejection Tracking**: Extended rejection reason detection to include "no announcement exists" and "not authorized" messages, ensuring these state events are properly tracked in the rejected events index for re-processing. 4. **Proper State Event Metrics**: State events now use `add_state()` instead of `add_announcement()` when rejected, ensuring correct metrics tracking. 5. **Removed Redundant Field**: Removed `event_id` field from `ColdIndexEntry` since it's already stored as the HashMap key. This eliminates dead code while preserving the cold index's core purpose: preventing re-fetch of rejected events during negentropy sync via `get_all_event_ids()`. 6. **Fixed Doc Test**: Changed doc test from `no_run` to `ignore` since it uses undefined variables for illustration purposes. 7. **Fixed Clippy Warnings**: - Added `#[allow(dead_code)]` for `reason` fields (reserved for future metrics) - Fixed unused variable warning - Collapsed nested if statement **Why:** The two-tier rejected events index was handling two scenarios: - GRASP-02 PR3: Maintainer announcement arrives → re-process announcements - GRASP-02 PR4.2: State event with git data arrives → re-process state events But it was missing: - GRASP-02 PR4.1: Repository announcement arrives → re-process state events This created a race condition where state events arriving before their announcements would be rejected and never re-processed. **Implementation Details:** The fix follows the same pattern as maintainer re-processing: 1. When announcement accepted, parse it to get pubkey + identifier 2. Call `invalidate_and_get_state_events()` to get rejected state events 3. Re-process each state event from hot cache using `process_event_static()` 4. Log results (Saved, Purgatory, Duplicate, or still rejected) **Test Results:** ✅ All tests pass (578 total): - 248 unit tests pass - 330 integration tests pass (including the previously failing test) - All clippy warnings fixed - Doc tests pass ✅ Target test now passes consistently: - `test_concurrent_state_and_pr_sync` completes in ~2.7s (was timing out at 30s) **Impact:** - Fixes race condition in sync ordering (state before announcement) - No breaking changes - only adds re-processing capability - Follows existing patterns - mirrors GRASP-02 PR3 maintainer re-processing - Minimal code changes - ~86 lines added to handle new re-processing path **Files Changed:** ``` src/sync/mod.rs | 86 +++++++++++++++++++++++++++++++++++++++++++++ src/sync/rejected_index.rs | 6 ++-- 2 files changed, 87 insertions(+), 5 deletions(-) ``` Co-authored-by: Assistant

feat: implement state event authorization per GRASP-01 spec

2026-01-09T17:04:06+00:00

Add comprehensive authorization checks to ensure state events are only accepted from maintainers of accepted repository announcements. This implements the core GRASP-01 requirement that pushes must match the latest state announcement "respecting the maintainer set." Changes: 1. StatePolicy authorization (src/nostr/policy/state.rs): - Check authorization BEFORE git data validation (fail-fast) - Reject if no announcement exists for repository - Reject if author not in maintainer set - Use existing helpers: fetch_repository_data() and pubkey_authorised_for_repo_owners() - Structured logging for all rejections 2. Purgatory invalidation (src/nostr/builder.rs): - New method: check_purgatory_state_events_for_identifier() - Called when announcements accepted (Accept and AcceptMaintainer) - Re-evaluates state events in purgatory for the identifier - Processes newly-authorized events (releases from purgatory) - Keeps unauthorized events for natural expiry (30 min) - Enables retroactive authorization when announcements arrive late 3. Purgatory sync authorization (src/git/sync.rs): - Check authorization BEFORE processing git data - Remove unauthorized events from purgatory (permanent rejection) - Prevents processing even if git data arrives first - Structured logging for monitoring 4. Rejected events tracking (src/sync/rejected_index.rs): - Add support for tracking rejected state events - New methods: add_state(), contains_state() - Separate metrics for state rejections - Enables sync to avoid re-fetching rejected states 5. Sync metrics (src/sync/metrics.rs, src/sync/mod.rs): - Add state-specific metrics (hot cache, cold index) - Track rejected states separately from announcements - Support monitoring of authorization rejections 6. Comprehensive tests (tests/state_authorization.rs): - test_reject_state_without_announcement - test_reject_state_from_unauthorized_author - test_accept_state_from_announcement_author - test_accept_state_from_maintainer Security Impact: - Before: State events could be published by anyone - After: Only maintainers can publish state events - Defense-in-depth: Authorization checked at 3 points: 1. On arrival (StatePolicy) 2. On announcement acceptance (purgatory re-evaluation) 3. On git data arrival (purgatory sync) All tests pass: - 248 unit tests - 51 NIP-34 announcement tests - 4 new state authorization tests - 9 rejected index tests Closes: State authorization requirement from GRASP-01 spec

feat(sync): add cleanup loops and metrics for rejected events index

2026-01-09T16:27:38+00:00

Add automatic cleanup and Prometheus metrics for the two-tier rejected events index that caches rejected announcements for re-processing. Cleanup loops: - Hot cache: Every 60 seconds (events expire after 2 minutes) - Cold index: Every 24 hours (metadata expires after 7 days) - Background task with graceful shutdown support New Prometheus metrics (7): - Gauges: hot_cache_current, cold_index_current - Counters: hits, misses, hot_expired, cold_expired, invalidated This completes the maintainer announcement re-processing feature, reducing wait time from 24 hours to <1 second when a maintainer's announcement arrives before the repository owner's announcement. Memory is bounded through automatic cleanup, and comprehensive metrics enable monitoring of hit rates, memory usage, and cleanup effectiveness. Changes: - src/sync/metrics.rs: Added 7 metrics with recording methods - src/sync/rejected_index.rs: Added optional metrics support - src/sync/mod.rs: Added cleanup background task Tests: 248 library tests passing, 3 integration tests passing

feat: Switch SyncManager to use two-tier RejectedEventsIndex

2026-01-09T15:49:17+00:00

Replaces the simple HashSet with the sophisticated two-tier RejectedEventsIndex from PR1, enabling future immediate re-processing when maintainer dependencies resolve. ## Changes ### Config (src/config.rs) - Add `rejected_hot_cache_duration_secs` (default: 120 = 2 minutes) - Add `rejected_cold_index_expiry_secs` (default: 604800 = 7 days) - Both configurable via CLI flags or environment variables ### SyncManager (src/sync/mod.rs) **Type Change:** - Before: `Arc>>` (simple event ID set) - After: `Arc` (two-tier storage) **Initialization:** - Pass config durations to RejectedEventsIndex::new() - Creates hot cache (2 min) + cold index (7 days) **Event Processing (process_event_static):** - Extract identifier from 'd' tag - Determine rejection reason from error message - Call `add_announcement()` with full event + metadata - Stores in both hot cache and cold index **Negentropy Sync (derive_relay_targets):** - Call `get_all_event_ids()` to get rejected IDs - Returns union of hot cache + cold index event IDs - Excludes from negentropy reconciliation **Event Loop (relay_connection):** - Use `contains()` method instead of direct HashSet access - Simpler API, same skip-rejected behavior ### RejectedEventsIndex (src/sync/rejected_index.rs) **New Method:** - `get_all_event_ids()`: Returns HashSet from both tiers - Used for negentropy exclusion (replaces direct HashSet access) ### Tests Updated **test_rejected_events_index_tracks_announcements:** - Create RejectedEventsIndex with config durations - Add 'd' tag to test announcement - Use `add_announcement()` with full event - Verify both hot cache and cold index populated - Check lengths with `hot_cache_len()` and `cold_index_len()` **test_rejected_events_excluded_from_negentropy:** - Create RejectedEventsIndex instead of HashSet - Build full event with 'd' tag - Add to index with `add_announcement()` - Get IDs with `get_all_event_ids()` - Verify excluded from reconciliation ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ SyncManager │ │ │ │ rejected_events_index: Arc │ │ ├─ Hot Cache (2 min): Full events for re-processing │ │ └─ Cold Index (7 days): Metadata for dedup │ └─────────────────────────────────────────────────────────────┘ │ │ On rejection ▼ ┌─────────────────────────────────────────────────────────────┐ │ add_announcement(event, pubkey, identifier, reason) │ │ ├─ Store full event in hot cache │ │ └─ Store metadata in cold index │ └─────────────────────────────────────────────────────────────┘ │ │ On negentropy sync ▼ ┌─────────────────────────────────────────────────────────────┐ │ get_all_event_ids() → HashSet │ │ ├─ Union of hot cache IDs │ │ └─ Union of cold index IDs │ └─────────────────────────────────────────────────────────────┘ ``` ## Benefits ### Immediate - **Better tracking**: Store rejection reason + metadata - **Configurable**: Tune cache/index durations per deployment - **Observable**: Separate hot/cold metrics (future PR4) ### Future (PR3) - **Immediate re-processing**: Get events from hot cache when valid - **No 24h delay**: Maintainer announcements accepted in <1 second - **Automatic recovery**: Hot cache for immediate, cold index for later ## Backward Compatibility **No breaking changes:** - Same rejection behavior (skip events in index) - Same negentropy exclusion (union with purgatory IDs) - Default config values match previous implicit behavior **Migration:** - Existing deployments continue working with defaults - Optional: Tune durations via new config flags ## Testing All tests passing: - ✅ 9 rejected_index tests (hot cache, cold index, two-tier) - ✅ 139 sync module tests (including updated integration tests) - ✅ 247 total library tests ## Next Steps **PR3: Add invalidation + immediate re-processing** - Invalidate cold index when owner announcement accepted - Get events from hot cache for re-processing - Recursive call to process_event_static - Integration tests for <1s maintainer acceptance **PR4: Add cleanup + metrics** - Hot cache cleanup task (every 60s) - Cold index cleanup task (daily) - Prometheus metrics for both tiers - Monitor hot cache hits vs misses ## Configuration Examples ```bash # Default (2 min hot cache, 7 day cold index) ngit-grasp # Longer hot cache for slow relays ngit-grasp --rejected-hot-cache-duration-secs 300 # Shorter cold index for memory-constrained systems ngit-grasp --rejected-cold-index-expiry-secs 86400 # Environment variables export NGIT_REJECTED_HOT_CACHE_DURATION_SECS=180 export NGIT_REJECTED_COLD_INDEX_EXPIRY_SECS=259200 ngit-grasp ``` Part of: Maintainer chain discovery fix See: work/SOLUTION-SUMMARY-V2.md for full design Previous: PR1 (rejected_index.rs implementation) Next: PR3 (invalidation + re-processing)

feat: Add two-tier rejected events index

2026-01-09T15:38:58+00:00

Implements a sophisticated two-tier storage system for rejected repository announcements to enable immediate re-processing when dependencies resolve. ## Architecture **Tier 1: Hot Cache (2 minutes)** - Stores full event objects for immediate re-processing - Enables <1 second re-processing vs 24 hour wait - Auto-expires to prevent memory growth - Memory: ~200 KB typical, ~20 MB worst case **Tier 2: Cold Index (7 days)** - Stores metadata only (event_id, pubkey, identifier) - Prevents repeated downloads of rejected events - Enables invalidation when circumstances change - Memory: ~1 MB typical ## Problem Solved Without this system, maintainer announcements face a timing gap: 00:00 - Maintainer announcement rejected → Event discarded 00:02 - Owner announcement accepted (lists maintainer) → Want to re-process 00:02 - ❌ Maintainer announcement GONE → Must wait 24h for next sync With two-tier system: 00:00 - Maintainer announcement rejected → Stored in both tiers 00:02 - Owner announcement accepted → Invalidate + get from hot cache 00:02 - ✅ Re-process immediately → Accepted in <1 second ## Implementation New module: src/sync/rejected_index.rs - RejectedEventsIndex: Public API combining both tiers - HotCache: Internal struct for full event storage - ColdIndex: Internal struct for metadata storage - RejectionReason: Enum for tracking why events were rejected Key methods: - add_announcement(): Add to both tiers - contains(): Check if event is rejected - invalidate_and_get_events(): Remove from cold index, get from hot cache - cleanup_expired(): Remove expired entries from both tiers ## Testing 9 comprehensive unit tests covering: - Hot cache storage and retrieval - Hot cache expiration - Cold index metadata tracking - Cold index invalidation - Two-tier integration - Cleanup of expired entries - Hot cache misses after expiry - Multiple maintainer repositories All tests passing. ## Next Steps PR2: Switch SyncManager to use new RejectedEventsIndex PR3: Add invalidation + immediate re-processing logic PR4: Add cleanup task + Prometheus metrics Part of: Maintainer chain discovery fix See: work/SOLUTION-SUMMARY-V2.md for full design