npub1tkq8unhsd5jqx6ueex5lcpsgknrpquxuk44ftpjlpm3ulaake7xs76txrw/ngit-grasp-mirror/src/sync/algorithms.rs, branch v1.0.0Unnamed repository; edit this file 'description' to name the repository.
https://upleb.uk/npub1tkq8unhsd5jqx6ueex5lcpsgknrpquxuk44ftpjlpm3ulaake7xs76txrw/ngit-grasp-mirror/atom?h=v1.0.02026-02-13T19:59:36+00:00feat: add SyncLevel to sync system for purgatory announcement state-only sync2026-02-13T19:59:36+00:00DanConwayDevDanConwayDev@protonmail.com2026-02-13T19:59:36+00:00urn:sha1:e922e14e3ec4b898c111b2100cd63dddbe2fcdb1
Purgatory announcements need state events (kind 30618) synced from
external relays, but not full L2/L3 events (patches, issues, PRs)
which would be rejected anyway. This implements the SyncLevel concept
from the design doc (decision #6):
- Add SyncLevel enum (Full vs StateOnly) to RepoSyncNeeds
- When announcement enters purgatory during sync, register in
RepoSyncIndex with SyncLevel::StateOnly
- Add build_sync_level_aware_filters() that partitions repos by level:
StateOnly repos only get state event filters (kind 30618)
- Update derive_relay_targets to track state_only_repos separately
- Update compute_actions to handle both repo sets
- SelfSubscriber always uses SyncLevel::Full (promoted repos)
Fix sync tests after Syncing status introduction2026-01-09T15:32:02+00:00DanConwayDevDanConwayDev@protonmail.com2026-01-09T15:32:02+00:00urn:sha1:29be4cb7d0fbd29325c995a76ba1b1f47beecca5
- Fix relay_connected() helper to check v >= 2 (Syncing/Connected states)
- Fix unit test to use status value 3 (Connected) instead of 1 (Connecting)
- Fix clippy warning: use .to_vec() instead of .iter().cloned().collect()
All 61 sync integration tests now passing.
All 238 unit tests passing.
Clippy clean.
feat(sync): add ConnectedDegraded status for failed historic sync2026-01-09T14:12:24+00:00DanConwayDevDanConwayDev@protonmail.com2026-01-09T14:12:24+00:00urn:sha1:93a1684f068603b354ba3c05957a25459c73de05
- Add ConnectionStatus::ConnectedDegraded (status=4 in metrics)
- Track batch failures via PendingBatch.failed field
- Track relay-level failures via RelayState.historic_sync_had_failures
- Transition to ConnectedDegraded when any batch fails during historic sync
- Add is_live_sync_active() helper for cleaner match patterns
- Update state machine diagram with ConnectedDegraded transitions
- Update metrics docs with status=4 and example queries
Fixes issue where relays with failed negentropy retries would
incorrectly transition to Connected status despite missing data.
Now operators can distinguish 'fully synced' vs 'degraded (partial data)'.
feat(sync): prevent infinite retry loop in negentropy validation2026-01-09T09:24:17+00:00DanConwayDevDanConwayDev@protonmail.com2026-01-09T09:24:17+00:00urn:sha1:eb10e85f199266affd3bca0a3d4cd934f74f3e7f
Add retry protection to negentropy event validation:
- Track retry_count in PendingBatch (incremented on each retry attempt)
- Detect when retry makes zero progress (relay returns no requested events)
- Abort retry and complete batch with partial results when stuck
- Log error with full details when retry protection triggers
This prevents infinite loops when:
- Relay has bugs and returns wrong events for ID queries
- Relay is malicious and returns unrelated events
- Relay has eventual consistency issues
- Network corruption causes incorrect responses
The protection triggers when received_count == 0 on a retry (relay
returned nothing we asked for), indicating the relay will never
provide the missing events.
Future work: Track failed batches in Prometheus metrics
(sync_failed_batches_total) for monitoring and alerting.
feat(sync): validate negentropy event receipt and retry missing events2026-01-09T09:21:55+00:00DanConwayDevDanConwayDev@protonmail.com2026-01-09T09:21:55+00:00urn:sha1:e1806540b5d905646b786e21a6060e4498e9aff1
Add validation that all events requested by ID during negentropy sync
are actually received from the relay. When events are missing:
- Log detailed information (requested/received/missing counts and IDs)
- Create retry subscriptions for missing events (chunked by 300)
- Update batch to track only missing events in next round
- Only complete batch after all events received or retry fails
This handles relays that have limits on ID-based queries (e.g., max 150
events per query) by automatically retrying in smaller chunks.
Also excludes purgatory and rejected announcement events from negentropy
requests to avoid re-requesting events we know we can't/won't store.
Note: Current implementation lacks retry limit - infinite loop protection
needed (tracked as future work).
feat(sync): implement pagination for historic_sync REQ+EOSE flow2025-12-19T16:40:13+00:00DanConwayDevDanConwayDev@protonmail.com2025-12-19T16:37:28+00:00urn:sha1:39242bfec6f6592c478c651f2e89e88e3e66ff2a
Add automatic pagination support for non-Negentropy historic sync to handle
large result sets efficiently. When a subscription receives >= 75 events,
the system automatically fetches the next page using the 'until' parameter.
Changes:
- Add PaginationState struct to track event counts and min timestamps
- Add pagination_state HashMap to PendingBatch for per-subscription tracking
- Add PAGINATION_THRESHOLD constant (75 events)
- Pass pending_sync_index to event processor for state updates
- Track events and timestamps as they arrive
- Check threshold on EOSE and launch follow-up subscriptions
- Initialize pagination state when creating historic sync subscriptions
- Update test fixtures in algorithms.rs
The pagination continues recursively until a page returns fewer than 75 events,
ensuring complete historic data retrieval without overwhelming relay limits.
sync: new connection logic2025-12-18T12:06:57+00:00DanConwayDevDanConwayDev@protonmail.com2025-12-18T12:06:57+00:00urn:sha1:e98a3850b6dcd3bbd5d251896ef56199cd49dc33proactive sync prep - some helper functions written but not enabled2025-12-16T15:26:55+00:00DanConwayDevDanConwayDev@protonmail.com2025-12-16T15:26:55+00:00urn:sha1:7821b107190cc116a30a4c339f935bc16a1d5197fix: unify sync state tracking for negentropy and REQ+EOSE paths2025-12-12T10:20:42+00:00DanConwayDevDanConwayDev@protonmail.com2025-12-12T10:09:47+00:00urn:sha1:de07e31fad60f9c68a08807cde17ff81d8371a65
When negentropy (NIP-77) sync was enabled, the RelaySyncIndex was never
updated to reflect historical sync completion. This caused the three-way
diff algorithm in compute_actions() to malfunction, leading to:
- Repeated sync attempts for the same items
- Incorrect filter counting for consolidation
- Potential premature relay disconnection
This fix unifies both sync paths (REQ+EOSE and Negentropy) through a
consistent PendingBatch flow:
1. Added SyncMethod enum to distinguish between sync types
2. Updated PendingBatch struct to include sync_method field
3. Extracted confirm_batch() method for unified batch confirmation
4. Modified negentropy_sync_and_process() to:
- Create a PendingBatch before sync
- Add batch to pending_sync_index
- On success: Remove batch and call confirm_batch()
- On failure: Remove batch without confirming
The confirm_batch() method moves repos and root_events from the batch
to the RelayState.repos and RelayState.root_events, ensuring the
three-way diff works correctly regardless of sync method.
Closes: negentropy-sync-state-tracking.md
fix: resolve all fmt and clippy warnings2025-12-11T16:53:03+00:00DanConwayDevDanConwayDev@protonmail.com2025-12-11T16:53:03+00:00urn:sha1:2a9160836bb87fdea3ae891563b0169c68d1c2ab
Main lib (src/):
- Add #[allow(dead_code)] for build_info field (stored to prevent Prometheus unregistration)
- Add #[allow(dead_code)] for first_seen field (reserved for future rate limiting)
- Replace .or_insert_with(RelaySyncNeeds::default) with .or_default()
- Replace manual div_ceil implementations with .div_ceil(100)
Test code (tests/):
- Replace .expect(&format!(...)) with .unwrap_or_else(|_| panic!(...))
- Remove needless borrows in fetch_metrics() calls
- Add #[allow(dead_code)] and #[allow(unused_imports)] to test helpers module
grasp-audit:
- Apply cargo fmt to fix formatting