diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-09 15:49:17 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-09 15:49:17 +0000 |
| commit | 02e957ec97c9a9e6e37eca9c9d4aa6aef4bcd363 (patch) | |
| tree | a4d6966452d66eb3fa0592311c1d85c570473a1f /src/sync/rejected_index.rs | |
| parent | e4cfecbfc909c9ca4983101cf6a5855959a5d49f (diff) | |
feat: Switch SyncManager to use two-tier RejectedEventsIndex
Replaces the simple HashSet<EventId> with the sophisticated two-tier
RejectedEventsIndex from PR1, enabling future immediate re-processing
when maintainer dependencies resolve.
## Changes
### Config (src/config.rs)
- Add `rejected_hot_cache_duration_secs` (default: 120 = 2 minutes)
- Add `rejected_cold_index_expiry_secs` (default: 604800 = 7 days)
- Both configurable via CLI flags or environment variables
### SyncManager (src/sync/mod.rs)
**Type Change:**
- Before: `Arc<RwLock<HashSet<EventId>>>` (simple event ID set)
- After: `Arc<RejectedEventsIndex>` (two-tier storage)
**Initialization:**
- Pass config durations to RejectedEventsIndex::new()
- Creates hot cache (2 min) + cold index (7 days)
**Event Processing (process_event_static):**
- Extract identifier from 'd' tag
- Determine rejection reason from error message
- Call `add_announcement()` with full event + metadata
- Stores in both hot cache and cold index
**Negentropy Sync (derive_relay_targets):**
- Call `get_all_event_ids()` to get rejected IDs
- Returns union of hot cache + cold index event IDs
- Excludes from negentropy reconciliation
**Event Loop (relay_connection):**
- Use `contains()` method instead of direct HashSet access
- Simpler API, same skip-rejected behavior
### RejectedEventsIndex (src/sync/rejected_index.rs)
**New Method:**
- `get_all_event_ids()`: Returns HashSet<EventId> from both tiers
- Used for negentropy exclusion (replaces direct HashSet access)
### Tests Updated
**test_rejected_events_index_tracks_announcements:**
- Create RejectedEventsIndex with config durations
- Add 'd' tag to test announcement
- Use `add_announcement()` with full event
- Verify both hot cache and cold index populated
- Check lengths with `hot_cache_len()` and `cold_index_len()`
**test_rejected_events_excluded_from_negentropy:**
- Create RejectedEventsIndex instead of HashSet
- Build full event with 'd' tag
- Add to index with `add_announcement()`
- Get IDs with `get_all_event_ids()`
- Verify excluded from reconciliation
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ SyncManager │
│ │
│ rejected_events_index: Arc<RejectedEventsIndex> │
│ ├─ Hot Cache (2 min): Full events for re-processing │
│ └─ Cold Index (7 days): Metadata for dedup │
└─────────────────────────────────────────────────────────────┘
│
│ On rejection
▼
┌─────────────────────────────────────────────────────────────┐
│ add_announcement(event, pubkey, identifier, reason) │
│ ├─ Store full event in hot cache │
│ └─ Store metadata in cold index │
└─────────────────────────────────────────────────────────────┘
│
│ On negentropy sync
▼
┌─────────────────────────────────────────────────────────────┐
│ get_all_event_ids() → HashSet<EventId> │
│ ├─ Union of hot cache IDs │
│ └─ Union of cold index IDs │
└─────────────────────────────────────────────────────────────┘
```
## Benefits
### Immediate
- **Better tracking**: Store rejection reason + metadata
- **Configurable**: Tune cache/index durations per deployment
- **Observable**: Separate hot/cold metrics (future PR4)
### Future (PR3)
- **Immediate re-processing**: Get events from hot cache when valid
- **No 24h delay**: Maintainer announcements accepted in <1 second
- **Automatic recovery**: Hot cache for immediate, cold index for later
## Backward Compatibility
**No breaking changes:**
- Same rejection behavior (skip events in index)
- Same negentropy exclusion (union with purgatory IDs)
- Default config values match previous implicit behavior
**Migration:**
- Existing deployments continue working with defaults
- Optional: Tune durations via new config flags
## Testing
All tests passing:
- ✅ 9 rejected_index tests (hot cache, cold index, two-tier)
- ✅ 139 sync module tests (including updated integration tests)
- ✅ 247 total library tests
## Next Steps
**PR3: Add invalidation + immediate re-processing**
- Invalidate cold index when owner announcement accepted
- Get events from hot cache for re-processing
- Recursive call to process_event_static
- Integration tests for <1s maintainer acceptance
**PR4: Add cleanup + metrics**
- Hot cache cleanup task (every 60s)
- Cold index cleanup task (daily)
- Prometheus metrics for both tiers
- Monitor hot cache hits vs misses
## Configuration Examples
```bash
# Default (2 min hot cache, 7 day cold index)
ngit-grasp
# Longer hot cache for slow relays
ngit-grasp --rejected-hot-cache-duration-secs 300
# Shorter cold index for memory-constrained systems
ngit-grasp --rejected-cold-index-expiry-secs 86400
# Environment variables
export NGIT_REJECTED_HOT_CACHE_DURATION_SECS=180
export NGIT_REJECTED_COLD_INDEX_EXPIRY_SECS=259200
ngit-grasp
```
Part of: Maintainer chain discovery fix
See: work/SOLUTION-SUMMARY-V2.md for full design
Previous: PR1 (rejected_index.rs implementation)
Next: PR3 (invalidation + re-processing)
Diffstat (limited to 'src/sync/rejected_index.rs')
| -rw-r--r-- | src/sync/rejected_index.rs | 20 |
1 files changed, 19 insertions, 1 deletions
diff --git a/src/sync/rejected_index.rs b/src/sync/rejected_index.rs index f89783a..80d6b5b 100644 --- a/src/sync/rejected_index.rs +++ b/src/sync/rejected_index.rs | |||
| @@ -85,7 +85,7 @@ | |||
| 85 | //! ``` | 85 | //! ``` |
| 86 | 86 | ||
| 87 | use nostr_sdk::{Event, EventId, PublicKey}; | 87 | use nostr_sdk::{Event, EventId, PublicKey}; |
| 88 | use std::collections::HashMap; | 88 | use std::collections::{HashMap, HashSet}; |
| 89 | use std::sync::{Arc, RwLock}; | 89 | use std::sync::{Arc, RwLock}; |
| 90 | use std::time::{Duration, Instant}; | 90 | use std::time::{Duration, Instant}; |
| 91 | 91 | ||
| @@ -396,6 +396,24 @@ impl RejectedEventsIndex { | |||
| 396 | pub fn cold_index_len(&self) -> usize { | 396 | pub fn cold_index_len(&self) -> usize { |
| 397 | self.cold_index.len() | 397 | self.cold_index.len() |
| 398 | } | 398 | } |
| 399 | |||
| 400 | /// Get all rejected event IDs (from both hot cache and cold index) | ||
| 401 | /// | ||
| 402 | /// Used for excluding rejected events from negentropy sync. | ||
| 403 | /// Note: This creates a snapshot - events may be added/removed concurrently. | ||
| 404 | pub fn get_all_event_ids(&self) -> HashSet<EventId> { | ||
| 405 | let mut ids = HashSet::new(); | ||
| 406 | |||
| 407 | // Add from hot cache | ||
| 408 | let hot_entries = self.hot_cache.entries.read().unwrap(); | ||
| 409 | ids.extend(hot_entries.keys().cloned()); | ||
| 410 | |||
| 411 | // Add from cold index | ||
| 412 | let cold_entries = self.cold_index.entries.read().unwrap(); | ||
| 413 | ids.extend(cold_entries.keys().cloned()); | ||
| 414 | |||
| 415 | ids | ||
| 416 | } | ||
| 399 | } | 417 | } |
| 400 | 418 | ||
| 401 | #[cfg(test)] | 419 | #[cfg(test)] |