upleb.uk

Public git repos — served from a NIP-34 GRASP relay at git.upleb.uk

summaryrefslogtreecommitdiff
path: root/src/sync/algorithms.rs
diff options
context:
space:
mode:
authorDanConwayDev <DanConwayDev@protonmail.com>2025-12-19 16:37:28 +0000
committerDanConwayDev <DanConwayDev@protonmail.com>2025-12-19 16:40:13 +0000
commit39242bfec6f6592c478c651f2e89e88e3e66ff2a (patch)
treeef358d70e7d33bc9edd086e64d9b4f64ee8bcd3a /src/sync/algorithms.rs
parent1adbd93e5bb8e14403ba64a76d5dc93209227514 (diff)
feat(sync): implement pagination for historic_sync REQ+EOSE flow
Add automatic pagination support for non-Negentropy historic sync to handle large result sets efficiently. When a subscription receives >= 75 events, the system automatically fetches the next page using the 'until' parameter. Changes: - Add PaginationState struct to track event counts and min timestamps - Add pagination_state HashMap to PendingBatch for per-subscription tracking - Add PAGINATION_THRESHOLD constant (75 events) - Pass pending_sync_index to event processor for state updates - Track events and timestamps as they arrive - Check threshold on EOSE and launch follow-up subscriptions - Initialize pagination state when creating historic sync subscriptions - Update test fixtures in algorithms.rs The pagination continues recursively until a page returns fewer than 75 events, ensuring complete historic data retrieval without overwhelming relay limits.
Diffstat (limited to 'src/sync/algorithms.rs')
-rw-r--r--src/sync/algorithms.rs2
1 files changed, 2 insertions, 0 deletions
diff --git a/src/sync/algorithms.rs b/src/sync/algorithms.rs
index a6e0787..f4b1f5c 100644
--- a/src/sync/algorithms.rs
+++ b/src/sync/algorithms.rs
@@ -401,6 +401,7 @@ mod tests {
401 }, 401 },
402 outstanding_subs: HashSet::new(), 402 outstanding_subs: HashSet::new(),
403 sync_method: SyncMethod::ReqEose, 403 sync_method: SyncMethod::ReqEose,
404 pagination_state: HashMap::new(),
404 }], 405 }],
405 ); 406 );
406 407
@@ -512,6 +513,7 @@ mod tests {
512 }, 513 },
513 outstanding_subs: HashSet::new(), 514 outstanding_subs: HashSet::new(),
514 sync_method: SyncMethod::ReqEose, 515 sync_method: SyncMethod::ReqEose,
516 pagination_state: HashMap::new(),
515 }], 517 }],
516 ); 518 );
517 519