diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-07 11:44:27 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-07 11:44:27 +0000 |
| commit | 1c5dac680b5a446e26b161208a17030f5fbd8a88 (patch) | |
| tree | 76a7113edcdd7436872665473727c1f535d7ab83 /src/purgatory/sync/mod.rs | |
| parent | 5bd6b9b93cd52da2075bc00a08cf7feca4b85d5c (diff) | |
Add core sync functions for identifier-based purgatory synchronization
Implement sync_identifier_next_url and sync_identifier_from_url functions
that provide the core URL selection and fetch logic for purgatory sync.
sync_identifier_next_url:
- Pure URL selection logic with no side effects
- Filters out our own domain and already-tried URLs
- Respects domain throttling when domain parameter is None
- Can target a specific domain when domain parameter is Some
sync_identifier_from_url:
- Fetches OIDs from a specific URL via the SyncContext
- Tracks request start/completion with ThrottleManager for rate limiting
- Calls process_newly_available_git_data on successful fetch
Also adds get_throttled_domains_with_untried_urls helper for the main
sync loop to know which DomainThrottle queues to enqueue identifiers to.
These functions are designed to be called by both:
- Main sync loop (tries non-throttled URLs immediately)
- DomainThrottle queue processing (when capacity frees up)
Includes 10 unit tests covering:
- Throttled domain skipping
- Tried URL skipping
- Our domain filtering
- Specific domain targeting
- Fetch success/failure handling
- Throttle request tracking
Diffstat (limited to 'src/purgatory/sync/mod.rs')
| -rw-r--r-- | src/purgatory/sync/mod.rs | 5 |
1 files changed, 5 insertions, 0 deletions
diff --git a/src/purgatory/sync/mod.rs b/src/purgatory/sync/mod.rs index 1ac0cb1..d26c1f0 100644 --- a/src/purgatory/sync/mod.rs +++ b/src/purgatory/sync/mod.rs | |||
| @@ -7,10 +7,15 @@ | |||
| 7 | //! - Debouncing for burst event arrivals | 7 | //! - Debouncing for burst event arrivals |
| 8 | 8 | ||
| 9 | mod context; | 9 | mod context; |
| 10 | mod functions; | ||
| 10 | mod queue; | 11 | mod queue; |
| 11 | mod throttle; | 12 | mod throttle; |
| 12 | 13 | ||
| 13 | pub use context::{ProcessResult, SyncContext}; | 14 | pub use context::{ProcessResult, SyncContext}; |
| 15 | pub use functions::{ | ||
| 16 | get_throttled_domains_with_untried_urls, sync_identifier_from_url, sync_identifier_next_url, | ||
| 17 | ThrottledDomainInfo, | ||
| 18 | }; | ||
| 14 | pub use queue::SyncQueueEntry; | 19 | pub use queue::SyncQueueEntry; |
| 15 | pub use throttle::{DomainThrottle, ThrottleManager}; | 20 | pub use throttle::{DomainThrottle, ThrottleManager}; |
| 16 | 21 | ||