diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-19 16:37:28 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2025-12-19 16:40:13 +0000 |
| commit | 39242bfec6f6592c478c651f2e89e88e3e66ff2a (patch) | |
| tree | ef358d70e7d33bc9edd086e64d9b4f64ee8bcd3a /src/sync/algorithms.rs | |
| parent | 1adbd93e5bb8e14403ba64a76d5dc93209227514 (diff) | |
feat(sync): implement pagination for historic_sync REQ+EOSE flow
Add automatic pagination support for non-Negentropy historic sync to handle
large result sets efficiently. When a subscription receives >= 75 events,
the system automatically fetches the next page using the 'until' parameter.
Changes:
- Add PaginationState struct to track event counts and min timestamps
- Add pagination_state HashMap to PendingBatch for per-subscription tracking
- Add PAGINATION_THRESHOLD constant (75 events)
- Pass pending_sync_index to event processor for state updates
- Track events and timestamps as they arrive
- Check threshold on EOSE and launch follow-up subscriptions
- Initialize pagination state when creating historic sync subscriptions
- Update test fixtures in algorithms.rs
The pagination continues recursively until a page returns fewer than 75 events,
ensuring complete historic data retrieval without overwhelming relay limits.
Diffstat (limited to 'src/sync/algorithms.rs')
| -rw-r--r-- | src/sync/algorithms.rs | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/src/sync/algorithms.rs b/src/sync/algorithms.rs index a6e0787..f4b1f5c 100644 --- a/src/sync/algorithms.rs +++ b/src/sync/algorithms.rs | |||
| @@ -401,6 +401,7 @@ mod tests { | |||
| 401 | }, | 401 | }, |
| 402 | outstanding_subs: HashSet::new(), | 402 | outstanding_subs: HashSet::new(), |
| 403 | sync_method: SyncMethod::ReqEose, | 403 | sync_method: SyncMethod::ReqEose, |
| 404 | pagination_state: HashMap::new(), | ||
| 404 | }], | 405 | }], |
| 405 | ); | 406 | ); |
| 406 | 407 | ||
| @@ -512,6 +513,7 @@ mod tests { | |||
| 512 | }, | 513 | }, |
| 513 | outstanding_subs: HashSet::new(), | 514 | outstanding_subs: HashSet::new(), |
| 514 | sync_method: SyncMethod::ReqEose, | 515 | sync_method: SyncMethod::ReqEose, |
| 516 | pagination_state: HashMap::new(), | ||
| 515 | }], | 517 | }], |
| 516 | ); | 518 | ); |
| 517 | 519 | ||