diff options
| author | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-07 11:26:48 +0000 |
|---|---|---|
| committer | DanConwayDev <DanConwayDev@protonmail.com> | 2026-01-07 11:26:48 +0000 |
| commit | 08eba888fe510896f1863be6f2b5a72a91616975 (patch) | |
| tree | c5c947246dd34fbffbc1cb15b48102bb472d3d4f /Cargo.toml | |
| parent | 2665811f54f62f147b7d773c76bd26d032b8f9cb (diff) | |
Add DomainThrottle for per-domain rate limiting
Implement per-domain throttling for purgatory sync operations:
- Concurrent request limit (max in-flight requests per domain)
- Rate limit (max requests per minute via sliding window)
- Fair round-robin queue processing across identifiers
- In-progress tracking to prevent duplicate fetches
- Tried URL tracking per identifier
Add indexmap dependency for ordered iteration in round-robin queue.
Includes 6 unit tests covering:
- Concurrent limit enforcement
- Rate limit enforcement (sliding window)
- Round-robin fair processing
- In-progress identifier skipping
- Round-robin index adjustment on removal
- Tried URL merging on re-enqueue
Diffstat (limited to 'Cargo.toml')
| -rw-r--r-- | Cargo.toml | 3 |
1 files changed, 3 insertions, 0 deletions
| @@ -33,6 +33,9 @@ prometheus = "0.13" | |||
| 33 | dashmap = "5" | 33 | dashmap = "5" |
| 34 | lazy_static = "1.4" | 34 | lazy_static = "1.4" |
| 35 | 35 | ||
| 36 | # Data structures | ||
| 37 | indexmap = "2" | ||
| 38 | |||
| 36 | # Random (for startup jitter) | 39 | # Random (for startup jitter) |
| 37 | rand = "0.8" | 40 | rand = "0.8" |
| 38 | 41 | ||