diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-19 16:37:28 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-19 16:40:13 +0000 |
| commit | 39242bfec6f6592c478c651f2e89e88e3e66ff2a (patch) | |
| tree | ef358d70e7d33bc9edd086e64d9b4f64ee8bcd3a /src/sync/relay_connection.rs | |
| parent | 1adbd93e5bb8e14403ba64a76d5dc93209227514 (diff) | |
feat(sync): implement pagination for historic_sync REQ+EOSE flow
Add automatic pagination support for non-Negentropy historic sync to handle
large result sets efficiently. When a subscription receives >= 75 events,
the system automatically fetches the next page using the 'until' parameter.
Changes:
- Add PaginationState struct to track event counts and min timestamps
- Add pagination_state HashMap to PendingBatch for per-subscription tracking
- Add PAGINATION_THRESHOLD constant (75 events)
- Pass pending_sync_index to event processor for state updates
- Track events and timestamps as they arrive
- Check threshold on EOSE and launch follow-up subscriptions
- Initialize pagination state when creating historic sync subscriptions
- Update test fixtures in algorithms.rs
The pagination continues recursively until a page returns fewer than 75 events,
ensuring complete historic data retrieval without overwhelming relay limits.
Diffstat (limited to 'src/sync/relay_connection.rs')
| -rw-r--r-- | src/sync/relay_connection.rs | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/src/sync/relay_connection.rs b/src/sync/relay_connection.rs index 37094be..5a61777 100644 --- a/src/sync/relay_connection.rs +++ b/src/sync/relay_connection.rs | |||
| @@ -23,8 +23,8 @@ use crate::nostr::builder::SharedDatabase; | |||
| 23 | /// Events from a relay connection | 23 | /// Events from a relay connection |
| 24 | #[derive(Debug)] | 24 | #[derive(Debug)] |
| 25 | pub enum RelayEvent { | 25 | pub enum RelayEvent { |
| 26 | /// A new event was received | 26 | /// A new event was received (event, subscription_id) |
| 27 | Event(Event), | 27 | Event(Event, SubscriptionId), |
| 28 | /// End of stored events for a subscription | 28 | /// End of stored events for a subscription |
| 29 | EndOfStoredEvents(SubscriptionId), | 29 | EndOfStoredEvents(SubscriptionId), |
| 30 | /// Connection was closed | 30 | /// Connection was closed |
| @@ -216,7 +216,7 @@ impl RelayConnection { | |||
| 216 | sub_id = %subscription_id, | 216 | sub_id = %subscription_id, |
| 217 | "Received event" | 217 | "Received event" |
| 218 | ); | 218 | ); |
| 219 | if event_sender.send(RelayEvent::Event(*event)).await.is_err() { | 219 | if event_sender.send(RelayEvent::Event(*event, subscription_id.clone())).await.is_err() { |
| 220 | tracing::debug!(relay = %url, "Event sender closed, stopping event loop"); | 220 | tracing::debug!(relay = %url, "Event sender closed, stopping event loop"); |
| 221 | break; | 221 | break; |
| 222 | } | 222 | } |