| Age | Commit message (Collapse) | Author |
|---|
|
fetch_repository_data_{excluding,with}_purgatory
The old name was ambiguous - it wasn't clear whether purgatory was
included or not. The two variants are now explicitly named:
- fetch_repository_data_excluding_purgatory: DB only
- fetch_repository_data_with_purgatory: DB + purgatory overlay
SyncContext trait method also renamed to fetch_repository_data_with_purgatory
to match the free function it delegates to.
|
|
Previously, sync_identifier_from_url passed all needed OIDs to
process_newly_available_git_data, not just the OIDs that were
successfully fetched. This caused incorrect logging (new_oids_count
would show all needed OIDs, not just fetched ones).
While this didn't break functionality (the actual processing uses
can_apply_state which checks the repository on disk), it made
debugging confusing.
Changes:
- Rename oids_fetched to fetched_oids and change type from usize to Vec<String>
- Return Vec<String> from match arms instead of counts
- Pass fetched_oids (not needed_oids) to process_newly_available_git_data
- Return fetched_oids.len() at the end
This ensures logging accurately reflects which OIDs were actually
fetched from the remote.
|
|
When fetch_oids returns Ok(vec![]) (all requested OIDs missing from
remote), the log message now says 'Fetch returned no OIDs (not available
on remote)' instead of the misleading 'Fetch succeeded' with oids_fetched=0.
|
|
Implement domain-level naughty list tracking for git remotes, reusing the
existing NaughtyListTracker from relay sync. This prevents repeated attempts
to fetch from git domains with persistent infrastructure issues (SSL/TLS
certificate errors, DNS failures).
Changes:
- Updated NaughtyListTracker to track both relay URLs and git domains
- Added git_naughty_list field to RealSyncContext for error classification
- Modified fetch_oids() to classify git fetch errors and record naughty domains
- Updated sync_identifier_next_url() to filter out naughty domains during URL selection
- Added git_naughty_list parameter to ThrottleManager for domain queue processing
- Threaded naughty list through start_sync_loop and all sync functions
- Updated all tests to pass naughty list parameter
The naughty list uses 12-hour expiration (configurable) to allow domains to
recover from infrastructure issues. First occurrence logs WARN, repeats log DEBUG.
|
|
- Update nostr-relay-builder, nostr-sdk, nostr-lmdb to latest revision
- Update grasp-audit nostr-sdk dependency
- Fix clippy warnings:
- Replace .clone() with std::slice::from_ref() in src/git/sync.rs
- Change &PathBuf to &Path in tests/common/git_server.rs
- Replace vec![] with array literal in src/purgatory/sync/functions.rs
- Update PR_TEST_COMMIT_HASH in grasp-audit due to event generation changes
All 249 tests passing, no breaking changes required.
|
|
|
|
- Prefix unused variable auth_result with underscore
- Prefix unused field git_data_path with underscore in Purgatory struct
- Add #[allow(clippy::too_many_arguments)] to handle_receive_pack
- Replace len() >= 1 with !is_empty()
- Replace .last() with .next_back() on DoubleEndedIterator
- Fix doc list item overindentation
- Replace map_or(true, ...) with is_none_or(...)
- Replace map_or(false, ...) with is_some_and(...)
|
|
Add support for extracting clone URLs from PR/PR-Update events (kind 1618/1619)
during purgatory sync, per NIP-34 specification. This enables fetching PR commits
from URLs specified in the PR event itself, not just from repository announcement
clone URLs.
Changes:
- Add collect_pr_clone_urls() to SyncContext trait
- Implement in RealSyncContext: extract clone tags from PR events in purgatory
- Implement in MockSyncContext: configurable PR clone URLs for testing
- Update sync_identifier_next_url to merge PR clone URLs with announcement URLs
- Update get_throttled_domains_with_untried_urls with same merge logic
- Add unit tests for PR clone URL extraction and filtering
|
|
Implement the unified function that handles all post-git-data-available
processing, regardless of how data arrived (git push or purgatory sync).
This function:
- Discovers satisfiable events from purgatory (state and PR events)
- Syncs OIDs to authorized owner repos
- Aligns refs and sets HEAD
- Saves events to database
- Notifies WebSocket subscribers
- Removes from purgatory
New additions:
- ProcessResult struct for tracking processing outcomes
- process_newly_available_git_data async function in src/git/sync.rs
- Helper functions: extract_identifier_from_repo_path, extract_identifier_from_pr_event
- Purgatory::find_prs_for_identifier method for PR event discovery
- Unit tests for all helper functions
Also fixes:
- Simplified extract_domain to avoid url crate dependency
- Removed unused imports in sync/loop.rs
|
|
Implement the main sync orchestration function and trigger-based queue
processing for throttled domains:
sync_identifier function:
- Orchestrates syncing git data for a single identifier
- Tries all non-throttled URLs in sequence
- Checks completion after each fetch (no pending events or all OIDs fetched)
- Enqueues with throttled domains when non-throttled URLs are exhausted
- Returns true if complete, false if events remain (for backoff)
ThrottleManager enhancements:
- Add set_context() to provide SyncContext for queue processing
- Add try_process_next() to spawn tasks when capacity frees
- Add process_queued_identifier() to handle queued work
- Update complete_request() to trigger processing on completion
- Update enqueue_identifier() to trigger processing when capacity available
- Add internal methods for non-Arc testing compatibility
Generic function updates:
- Add ?Sized bound to sync_identifier_next_url, sync_identifier_from_url,
sync_identifier, and get_throttled_domains_with_untried_urls for
dynamic dispatch support (Arc<dyn SyncContext>)
Tests:
- sync_identifier_tries_multiple_urls_until_complete: verifies sequential
URL fetching until all OIDs are available
- sync_identifier_enqueues_throttled_domains_when_incomplete: verifies
throttled domains get the identifier enqueued for later processing
- has_queued_work_reflects_queue_state: verifies queue state tracking
|
|
Implement sync_identifier_next_url and sync_identifier_from_url functions
that provide the core URL selection and fetch logic for purgatory sync.
sync_identifier_next_url:
- Pure URL selection logic with no side effects
- Filters out our own domain and already-tried URLs
- Respects domain throttling when domain parameter is None
- Can target a specific domain when domain parameter is Some
sync_identifier_from_url:
- Fetches OIDs from a specific URL via the SyncContext
- Tracks request start/completion with ThrottleManager for rate limiting
- Calls process_newly_available_git_data on successful fetch
Also adds get_throttled_domains_with_untried_urls helper for the main
sync loop to know which DomainThrottle queues to enqueue identifiers to.
These functions are designed to be called by both:
- Main sync loop (tries non-throttled URLs immediately)
- DomainThrottle queue processing (when capacity frees up)
Includes 10 unit tests covering:
- Throttled domain skipping
- Tried URL skipping
- Our domain filtering
- Specific domain targeting
- Fetch success/failure handling
- Throttle request tracking
|