diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-07 11:57:55 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-07 11:57:55 +0000 |
| commit | 8babcee8fdfa5b0f460aa1e6d8057feb7d2fda49 (patch) | |
| tree | 0e85dce4da81e57aa26ead9810589fe0f44f89ca /src/purgatory/sync/mod.rs | |
| parent | 1c5dac680b5a446e26b161208a17030f5fbd8a88 (diff) | |
Add sync_identifier orchestration and ThrottleManager queue processing
Implement the main sync orchestration function and trigger-based queue
processing for throttled domains:
sync_identifier function:
- Orchestrates syncing git data for a single identifier
- Tries all non-throttled URLs in sequence
- Checks completion after each fetch (no pending events or all OIDs fetched)
- Enqueues with throttled domains when non-throttled URLs are exhausted
- Returns true if complete, false if events remain (for backoff)
ThrottleManager enhancements:
- Add set_context() to provide SyncContext for queue processing
- Add try_process_next() to spawn tasks when capacity frees
- Add process_queued_identifier() to handle queued work
- Update complete_request() to trigger processing on completion
- Update enqueue_identifier() to trigger processing when capacity available
- Add internal methods for non-Arc testing compatibility
Generic function updates:
- Add ?Sized bound to sync_identifier_next_url, sync_identifier_from_url,
sync_identifier, and get_throttled_domains_with_untried_urls for
dynamic dispatch support (Arc<dyn SyncContext>)
Tests:
- sync_identifier_tries_multiple_urls_until_complete: verifies sequential
URL fetching until all OIDs are available
- sync_identifier_enqueues_throttled_domains_when_incomplete: verifies
throttled domains get the identifier enqueued for later processing
- has_queued_work_reflects_queue_state: verifies queue state tracking
Diffstat (limited to 'src/purgatory/sync/mod.rs')
| -rw-r--r-- | src/purgatory/sync/mod.rs | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/src/purgatory/sync/mod.rs b/src/purgatory/sync/mod.rs index d26c1f0..8ac9216 100644 --- a/src/purgatory/sync/mod.rs +++ b/src/purgatory/sync/mod.rs | |||
| @@ -13,8 +13,8 @@ mod throttle; | |||
| 13 | 13 | ||
| 14 | pub use context::{ProcessResult, SyncContext}; | 14 | pub use context::{ProcessResult, SyncContext}; |
| 15 | pub use functions::{ | 15 | pub use functions::{ |
| 16 | get_throttled_domains_with_untried_urls, sync_identifier_from_url, sync_identifier_next_url, | 16 | get_throttled_domains_with_untried_urls, sync_identifier, sync_identifier_from_url, |
| 17 | ThrottledDomainInfo, | 17 | sync_identifier_next_url, ThrottledDomainInfo, |
| 18 | }; | 18 | }; |
| 19 | pub use queue::SyncQueueEntry; | 19 | pub use queue::SyncQueueEntry; |
| 20 | pub use throttle::{DomainThrottle, ThrottleManager}; | 20 | pub use throttle::{DomainThrottle, ThrottleManager}; |