| Age | Commit message (Collapse) | Author |
|---|
|
When a replacement 30617 announcement arrived for an entry already in
purgatory (e.g. the same event fetched from a second relay during sync,
or a user re-submitting a slightly updated announcement), the policy
returned Accept instead of AcceptPurgatory. This caused the event to be
saved to the database immediately, bypassing the purgatory gate, without
the corresponding git data or state events ever arriving.
Fix: return AcceptPurgatory when replacing a purgatory entry so the
updated event stays in purgatory until git data arrives. The purgatory
entry is still updated with the newer event via replace_purgatory_announcement
before the return.
|
|
NIP-01 places no restriction on d tag characters and NIP-34 only
recommends kebab-case without mandating it. Rejecting identifiers with
whitespace or other URL-unsafe characters was therefore overly strict.
The correct approach (per NIP-34 PR #2312 and GRASP-01) is to store
identifiers verbatim on disk and percent-encode them when constructing
URLs. The previous commit already handled the incoming direction
(percent-decoding URL paths before filesystem lookup); this commit
handles the outgoing direction and removes the validation restriction.
Changes:
- validate_identifier: drop whitespace rejection; only reject chars
that are unsafe as filesystem directory names (/, \, null, . / ..)
- git/mod.rs: add percent_encode() alongside percent_decode()
- landing.rs: percent-encode identifier in nostr:// clone URL and
gitworkshop link (also fixes a pre-existing bug where the clone URL
displayed literal '{npub}' / '{identifier}' instead of the values)
|
|
Two bugs allowed a repository announcement with a space-containing
identifier ('kuboslopp by Shakespeare') to enter purgatory and create
a bare repo on disk, but then fail to serve git data over HTTP.
Bug 1 (serving): parse_git_url and parse_repo_url did not percent-decode
the URL path before resolving the filesystem path. A client requesting
/npub.../kuboslopp%20by%20Shakespeare.git/info/refs had the identifier
extracted as 'kuboslopp%20by%20Shakespeare' (literal %20), which did not
match the on-disk directory 'kuboslopp by Shakespeare.git'.
Fix: add percent_decode() in src/git/mod.rs and apply it to the repo
component in both parse_git_url and parse_repo_url.
Bug 2 (validation): validate_announcement did not check that the
identifier is safe as a filesystem path component and URL segment.
Identifiers containing whitespace, path separators, null bytes, or
reserved names (. / ..) should be rejected at acceptance time.
Fix: add validate_identifier() in src/nostr/events.rs and call it from
validate_announcement before any other policy checks.
|
|
When NGIT_MAX_CONNECTIONS is unset the relay imposes no connection cap,
deferring to OS fd limits and infrastructure controls. The option remains
available for operators who want an explicit ceiling.
|
|
Fix pre-existing clippy lints:
- &PathBuf -> &Path in audit_cleanup.rs
- too_many_arguments on process_newly_available_git_data,
process_purgatory_announcements, and HttpService::new
- clone_on_copy for PublicKey (Copy type) in purgatory cleanup loop
|
|
State events (kind 30618) can include refs/tags/<name>^{} entries which
are git's notation for the dereferenced commit behind an annotated tag.
These are not real git refs and are never sent as part of a push.
extract_refs_from_state and RepositoryState::from_event were treating
them as real refs, causing can_satisfy_state to reject valid annotated
tag pushes: the would-be state after the push lacked the spurious ^{}
entry, so the exact-equality check always failed.
|
|
|
|
fetch_repository_data_{excluding,with}_purgatory
The old name was ambiguous - it wasn't clear whether purgatory was
included or not. The two variants are now explicitly named:
- fetch_repository_data_excluding_purgatory: DB only
- fetch_repository_data_with_purgatory: DB + purgatory overlay
SyncContext trait method also renamed to fetch_repository_data_with_purgatory
to match the free function it delegates to.
|
|
|
|
Kind 5 deletion events referencing a PR or PR-update event by e-tag now
remove the matching purgatory entry, provided the deletion author matches
the PR event author. Placeholders (git data arrived before the event) are
not removed since they have no author to verify against.
PR purgatory is keyed by event ID hex so this is an O(1) lookup, checked
before the O(n) announcement and state event scans.
|
|
The previous tests deleted purgatory announcements (kind 30617) and checked
for bare-repo absence via git ls-remote, which would corrupt shared-mode
test state by destroying repos other tests depend on.
New approach tests deletion of purgatory state events (kind 30618) instead:
- e-tag test: promotes a repo, creates a unique commit locally, submits a
state event pointing to it (enters purgatory), deletes the state event by
event ID, then verifies git push of that commit is rejected.
- a-tag coordinate test: promotes a repo, generates a fresh maintainer
keypair, sends a replacement announcement adding that maintainer, submits
a state event signed by the new maintainer (enters purgatory), deletes by
coordinate 30618:<new_maintainer_pubkey>:<identifier>, then verifies git
push is rejected.
Also extends DeletionPolicy to handle kind 30618 state events in purgatory
for both e-tag (event ID) and a-tag (coordinate) deletion paths.
|
|
Kind 5 deletion events signed by the announcement author now evict the
corresponding purgatory entry and delete the bare repository from disk.
Both NIP-09 reference styles are supported:
- e tag (event ID): matches the purgatory entry whose event ID equals the tag value
- a tag (coordinate 30617:<pubkey>:<identifier>): matches by coordinate, only
removes entries with created_at <= deletion event created_at per NIP-09 spec
Author-only enforcement: coordinate pubkey and e-tag owner must match the
deletion event pubkey; third-party deletion attempts are silently ignored.
Includes 6 unit tests and 2 integration tests (event ID and coordinate paths).
|
|
Per design doc decision #4: state event arrival resets the 30-minute
protocol timer for purgatory announcements. This prevents premature
expiry during slow sync operations where the repo is actively receiving
metadata but git data hasn't arrived yet.
Extends expiry for all owners whose announcement authorized the state
event, and triggers revival if the announcement was soft-expired.
|
|
When a state event arrives and the required commits already exist in
another maintainer's repo on the same relay, process_state_with_git_data
copies the OIDs across and aligns refs — but never called
process_purgatory_announcements for the target repos. Any announcement
waiting in purgatory for that repo stayed there indefinitely.
Fix: after process_state_with_git_data, call process_newly_available_git_data
for each target repo (those that received copied OIDs) so purgatory
announcements are promoted immediately.
|
|
Instead of threading repo_sync_index through PolicyContext/builder.rs/main.rs
to handle user-submitted purgatory announcements, add a simple background
timer (run_purgatory_announcement_sync, every 5s) that scans the purgatory
for announcement entries and registers them in repo_sync_index as StateOnly.
This is simpler and covers both flows:
- Sync-path announcements: inline registration still happens during event
processing (sync/mod.rs:1839+), timer provides a safety net
- User-submitted announcements: SelfSubscriber never sees them (rejected
from DB), timer is the primary registration path
The timer calls sync_purgatory_announcements_to_index() which:
1. Snapshots purgatory via new announcements_for_sync() public method
2. Or_inserts StateOnly entries (never downgrades Full entries)
3. Detects newly added relay URLs and calls handle_new_sync_filters to
connect and subscribe - fixing the failing test that expected relay
discovery from a user-submitted purgatory announcement
Removes: repo_sync_index field from PolicyContext, set/get_repo_sync_index
methods, set_repo_sync_index on Nip34WritePolicy, wiring in main.rs, and
the inline AcceptPurgatory registration block in builder.rs.
|
|
negentropy fallback
Three targeted fixes for purgatory announcement sync:
1. SelfSubscriber sync_level upgrade: After or_insert_with in process_batch,
always set entry.sync_level = SyncLevel::Full so that when a promoted
announcement is broadcast via notify_event and SelfSubscriber receives it,
an existing StateOnly entry gets upgraded to Full and PR event subscriptions
are triggered immediately (not delayed up to 24h).
2. Negentropy fallback filter split: In handle_eose, when falling back from
negentropy to REQ+EOSE, split batch_repos by SyncLevel and call
build_sync_level_aware_filters instead of build_layer2_and_layer3_filters.
Prevents StateOnly (purgatory) repos from getting Layer 2 #a/#A/#q filters
prematurely, which caused nostr-sdk client deduplication to permanently
drop PR events after orphan rejection.
3. Recompute sync filters after announcement batch EOSE: Add
recompute_new_sync_filters_for_relay calls at all three batch-completion
paths in handle_eose for generic filter (announcement) batches. This
triggers state-only subscriptions for any purgatory repos registered during
that batch, fixing the 24h delay before state event sync starts.
4. User-submitted purgatory announcements: Add repo_sync_index field to
PolicyContext with setter/getter, wire in main.rs after SyncManager
creation, and register in AcceptPurgatory handler so user-submitted
announcements get StateOnly sync started immediately.
5. Update archive tests: test_archive_without_state_events_does_not_sync_git
updated to reflect that StateOnly subscription now proactively fetches
state events from source relays. test_archive_read_only_creates_bare_repo
un-ignored as it now works end-to-end.
|
|
after announcement promotion"
This reverts commit d76003b629a4a03dba23a8a1c41da6e4ac4c30cf.
|
|
announcement promotion
When git data arrives for a purgatory announcement and promotes it to the
database, the relay now:
1. Upgrades the announcement's sync level in RepoSyncIndex from StateOnly
to Full (git/sync.rs: process_purgatory_announcements)
2. Sends AddFilters actions to SyncManager for all connected relays, using
Full sync filters (Layer 2 #a/#A/#q) to subscribe to PR events
(purgatory/sync/context.rs: RealSyncContext.process_newly_available_git_data)
3. For user-submitted purgatory announcements, registers the repo in
RepoSyncIndex with StateOnly level and sends AddFilters to SyncManager
so it discovers and connects to relays listed in the announcement tags
(nostr/builder.rs: handle_announcement AcceptPurgatory path)
The RealSyncContext now accepts optional repo_sync_index and sync_action_tx
parameters. main.rs wires these up from SyncManager. PolicyContext gains
repo_sync_index and sync_action_tx fields for the write policy path.
|
|
purgatory
remove_purgatory_announcement() was unconditionally wiping all state
events for an identifier when one owner's announcement was evicted.
State events are keyed by identifier alone, so this incorrectly
discarded state events belonging to a different owner's repository
sharing the same identifier string. Now only removes state events if
no other owner's announcement remains in purgatory for that identifier.
|
|
An older rejected announcement (e.g. a relay replay of a superseded
event) was incorrectly evicting a newer purgatory entry for the same
pubkey+identifier. Now only evict when the incoming event's created_at
is strictly greater than the stored entry's created_at.
|
|
Previously, has_active_announcement() only queried the database, so when
a newer announcement arrived for the same (pubkey, identifier) while the
original was still in purgatory, it was incorrectly routed as a brand-new
announcement (AcceptPurgatory) rather than replacing the existing entry.
This change splits the logic into two cases:
- If the existing entry is in the database: return Accept (replacement) as before
- If the existing entry is only in purgatory: replace the purgatory entry via
add_announcement() (which overwrites by key) and extend expiries for both the
announcement and any waiting state events, then return Accept
- If the owner sends a Reject-classified announcement (service removed) but has
a purgatory entry: clear the purgatory entry, delete the bare repo, and remove
any waiting state events before rejecting
Also add an explicit comment to find_accepted_repository() in related.rs
clarifying that it intentionally only checks the database. Related events
should only be accepted after the repository announcement has been promoted
(validated via git data) - this is correct behaviour, not a missing check.
|
|
is_maintainer_in_any_announcement only queried the database, missing
announcements still in purgatory. A maintainer's announcement (which
lists the recursive maintainer) may arrive and enter purgatory before
the recursive maintainer's announcement does, causing the maintainer
exception check to return false and reject the recursive maintainer's
announcement.
|
|
Add comments explaining that PR event processing (both incoming and
purgatory) should only use database announcements, not purgatory ones.
This is intentional because:
- Incoming PR events should only be accepted for validated announcements
- Purgatory PR events should only be released when announcement is promoted
- This prevents accepting PR events for announcements that fail validation
Differs from state event processing which uses fetch_repository_data_with_purgatory
because state events check authorization without releasing from purgatory.
|
|
Route new announcements to purgatory instead of accepting immediately.
Announcements are promoted to the database when git data arrives,
ensuring we only serve announcements for repos with actual content.
Implemented:
- AnnouncementPurgatoryEntry type and DashMap store
- Route new announcements to purgatory (replacement announcements skip)
- Promote announcements on git data arrival (process_purgatory_announcements)
- Authorization checks purgatory announcements (fetch_repository_data_with_purgatory)
- State policy uses purgatory announcements for maintainer validation
- Cleanup task handles announcement expiry
- Updated count()/cleanup() to 3-tuples
Known broken:
- test_archive_read_only_creates_bare_repo fails: sync module does not
treat purgatory announcements as confirmed repos, so per-repo sync
(state events, PRs) is never triggered for purgatory announcements
- Announcement persistence (save/restore) not implemented
- SyncLevel (StateOnly vs Full) not implemented
- Soft expiry two-phase not implemented
- Expiry extension on state event / git auth not wired up
|
|
- Derive Default for config structs instead of manual impl
- Fix doc comment formatting in ArchiveConfig::matches
- Collapse nested if statement in validate_announcement
- Allow too_many_arguments for SyncManager::new
|
|
|
|
This merge includes critical bug fixes and comprehensive migration tooling
developed during the relay.ngit.dev migration effort.
Bug Fixes:
- Fix git protocol error handling to return HTTP 200 with ERR pkt-line
- Fix naughty list false positives and DNS failure identification
- Fix database query filters in load_existing_events (remove .since())
- Fix OID fetch tracking to distinguish 0 OIDs from successful fetches
- Fix purgatory event source tracking for filtered expiry logging
- Implement OID retry logic for 'not our ref' errors
Migration Tools & Documentation:
- Complete 5-phase migration analysis pipeline with orchestration script
- Phase 1: Event fetching from source relay
- Phase 2: Git sync verification
- Phase 3: Categorization and relay comparison
- Phase 4: Log extraction (parse failures, purgatory expiry)
- Phase 5: Action classification for migration decisions
- Comprehensive migration guide with lessons learned
- Troubleshooting guide for permission and corruption issues
Configuration:
- Add NGIT_LOG_LEVEL configuration option
- Update git throttle limits to 60/minute
- Improve logging throughout for better observability
|
|
Add EventSource enum (Direct/Sync) to purgatory entries to distinguish
between user-submitted events and sync-fetched events. This enables:
- WARN-level logging for direct submissions that expire (user should know)
- DEBUG-level logging for sync-fetched expirations (expected behavior)
- Source upgrade from Sync→Direct if user submits after sync
- Expiry timer reset on source upgrade (fresh 30-min window for user)
The source is included in [PURGATORY_EXPIRED] logs as source=direct or
source=sync for easy filtering.
|
|
- Add [PARSE_FAIL] logging when event parsing fails
- Add [PURGATORY_EXPIRED] logging when repos expire from purgatory
- Logs include: kind, event_id, repo, npub, reason
- Supports Phase 4 migration scripts (30-extract-*.sh)
- All 382 tests pass
|
|
Enables relay operators to backup/archive specific GRASP servers by domain.
Includes configuration, validation, documentation, and integration tests.
|
|
NIP-34 specifies single clone/relays tags with multiple values, not multiple
tags with single values. Update test helper to match spec.
|
|
Combined Accept and AcceptArchive match arms in builder.rs to ensure
bare repositories are created for both cases. Previously AcceptArchive
had duplicate code that didn't call ensure_bare_repository().
Also includes:
- Config fix: effective_git_data_path() respects explicit paths with memory backend
- TestRelay: Added git_data_path() and archive config support for testing
- Integration tests for archive_read_only behavior
|
|
- Make RateLimit explicit in relay builder (500 subs, 60 events/min)
- Add NGIT_MAX_CONNECTIONS config option (default: 500)
- Update all 4 config locations (src, nix, docs, .env.example)
- Fix documentation error: filter limit 5000→500
- Document Phase 2 deferral decision (per-IP enforcement)
Addresses primary DoS vector (connection exhaustion) with minimal code.
Per-IP rate limiting deferred until abuse detected in production.
Related: issue ff38 (git endpoint throttling - separate concern)
|
|
Adds NGIT_EVENT_BLACKLIST option for blocking all events from specific npubs,
taking precedence over all other validation to enable comprehensive moderation
without affecting curation policy.
Key features:
- Simple npub-only format: <npub>,<npub>,...
- Checked FIRST before any other validation (including repository blacklist)
- Blocks ALL event types (announcements, state events, PRs, comments, etc.)
- Events never reach relay storage or purgatory
- Specific rejection reason for operator debugging
Implementation:
- Add EventBlacklistConfig struct with check() method
- Add NGIT_EVENT_BLACKLIST config option and event_blacklist_config() method
- Add config field to PolicyContext for policy access
- Add check_event_blacklist() to Nip34WritePolicy
- Check event blacklist first in admit_event() method (before any other validation)
- 4 new unit tests covering all blacklist behavior
Configuration synced across all four sources:
- src/config.rs: Core implementation with EventBlacklistConfig
- .env.example: Comprehensive documentation with examples
- docs/reference/configuration.md: Complete reference documentation
- nix/module.nix: NixOS module option with environment mapping
README updates:
- Add comprehensive "Curation & Moderation" section
- Document repository whitelists (GRASP-01 and GRASP-05 modes)
- Document repository and event blacklists with precedence order
- Add configuration table for all curation/moderation settings
- Provide real-world examples for different relay configurations
Testing:
- 4 new tests for event blacklist functionality
- All 336 library tests passing
- All 64 integration tests passing
- All 38 filter support tests passing
Verification:
- Repository blacklist confirmed to apply to sync (uses same admit_event flow)
- Sync events validated through process_event_static -> write_policy.admit_event
Use cases:
- Block spam/abusive users completely
- Prevent malicious actors from submitting any events
- Temporary blocks for investigation
- Moderation without affecting whitelist curation policy
|
|
Adds NGIT_REPOSITORY_BLACKLIST option for blocking repositories, taking precedence
over all whitelists (archive and repository) to enable moderation without affecting
curation policy.
Key features:
- Three blacklist formats: <npub>, <npub>/<identifier>, <identifier>
- Blacklist checked first before any other validation
- Overrides archive whitelist and repository whitelist
- Specific rejection reasons based on match type (npub/identifier/both)
- Not flagged in NIP-11 curation (operational, not policy)
Implementation:
- Add BlacklistConfig struct with check() method returning detailed reasons
- Add NGIT_REPOSITORY_BLACKLIST config option and blacklist_config() method
- Update validate_announcement() to check blacklist first with specific reasons
- 12 new unit tests covering all blacklist behavior and precedence
Configuration synced across all four sources:
- src/config.rs: Core implementation with BlacklistConfig
- .env.example: Comprehensive documentation with examples
- docs/reference/configuration.md: Complete reference documentation
- nix/module.nix: NixOS module option with environment mapping
Testing:
- 12 new tests for blacklist functionality (config + validation)
- All 332 library tests passing
- All 38 integration tests passing
Use cases:
- Block spam/malware repos by identifier
- Block abusive users by npub
- Block specific problematic repos by npub/identifier
- Temporary blocks for investigation
|
|
config methods
Refactors configuration validation to fail fast on fatal errors at startup
while gracefully handling recoverable issues (e.g., malformed whitelist entries).
Changes:
- Add Config::validate() for eager validation called immediately after load
- Remove Result<> from archive_config() and repository_config() methods
- WhitelistEntry::parse_whitelist() skips invalid entries with warnings
- Validate relay_owner_nsec format in Config::validate()
- Update all call sites to remove Result handling from config getters
Benefits:
- Fatal config errors (incompatible settings) fail at startup, not runtime
- Recoverable errors (bad whitelist entries) logged as warnings and skipped
- No Result handling scattered throughout runtime code after validation
- Config methods safe to call without error handling after validate()
Testing:
- Add 7 new tests for validation edge cases and error handling
- Total config tests: 40 (up from 33)
- All 320 library tests passing
Breaking change: Config users must call config.validate() after Config::load()
to ensure configuration is valid. This is enforced in main.rs.
|
|
Adds NGIT_REPOSITORY_WHITELIST option for curated relay operation that
accepts only whitelisted repositories while maintaining GRASP-01 compliance
(announcements must list the service). This differs from archive whitelist
which enables GRASP-05 mode and doesn't require service listing.
Key features:
- Supports three whitelist formats: npub, npub/identifier, identifier
- Enforces mutual exclusivity with archive read-only mode
- Updates NIP-11 curation field when whitelist is enabled
- Maintains GRASP-01 compliance (doesn't add GRASP-05 support)
Configuration synced across all four sources: src/config.rs, docs/reference/configuration.md,
nix/module.nix, and .env.example as required by AGENTS.md.
|
|
Implements NGIT_ARCHIVE_READ_ONLY configuration option that defaults to true
when archive mode is enabled, allowing relays to operate as read-only syncs
of archived repositories.
Key changes:
- Add NGIT_ARCHIVE_READ_ONLY config option (defaults to true if archive enabled)
- NIP-11 advertises GRASP-05 support and includes curation field when read-only
- Validation logic rejects non-whitelisted repos in read-only mode
- Comprehensive tests for read-only behavior and defaults
- Full documentation in config reference, .env.example, and NixOS module
Read-only mode enables passive mirroring without being listed in announcements,
useful for backup/archive operations while preventing accidental write acceptance.
|
|
Implements GRASP-05 specification for accepting repository announcements
that don't list this relay, enabling archive, mirror, and backup use cases.
Core Features:
- Three whitelist formats: <npub>, <npub>/<identifier>, <identifier>
- Archive-all mode for complete ecosystem mirrors
- Fail-fast npub validation at startup
- Read-only enforcement (archived repos reject pushes)
- Full GRASP-02 sync (git data + Nostr events)
- Dynamic archive status (no flags/metadata)
Implementation:
- Add ArchiveWhitelistEntry enum with Pubkey/Repository/Identifier variants
- Add ArchiveConfig with validation and matching logic
- Update AnnouncementResult to include AcceptArchive variant
- Refactor validate_announcement() to return AnnouncementResult with archive check
- Update AnnouncementPolicy with catch-all pattern for cleaner code
- Wire archive config through builder and policy layers
Configuration:
- NGIT_ARCHIVE_ALL: Accept all announcements (⚠️ storage risk)
- NGIT_ARCHIVE_WHITELIST: Comma-separated whitelist entries
- Updated docs, .env.example, and nix/module.nix
Testing:
- 28 unit tests for config parsing and whitelist matching
- 7 integration tests for archive mode validation
- All 296 tests passing
Validation Priority:
1. Lists our service → Accept (GRASP-01, read/write)
2. Is maintainer → AcceptMaintainer (multi-maintainer, read/write)
3. Matches archive config → AcceptArchive (GRASP-05, read-only)
4. None of above → Reject
Security Considerations:
- Archive-all mode has storage/bandwidth DoS risk
- Identifier-only format matches any pubkey (use npub/identifier for high-value)
- Invalid npubs cause startup failure (fail-fast)
Documentation:
- Concise explanation focused on rationale
- Reference docs updated with all config options
- README updated to reflect completed feature
- Removed from roadmap, added to compliance section
See docs/explanation/grasp-05-archive.md for details.
|
|
Announcements were being rejected when clone URLs or relay URLs had
trailing slashes that didn't match. Added URL normalization to strip
trailing slashes before comparison, allowing announcements to be
accepted regardless of trailing slash presence.
- Add normalize_url_for_comparison() helper
- Update has_clone_url() and has_relay() to normalize before matching
- Add comprehensive tests for trailing slash scenarios
Fixes issue in work/active-issues/clone-relays-mismatch-validation.md
|
|
State events from remote relays for repos we don't host are expected
rejections during proactive sync. Changed to only WARN for user-submitted
events (potential misconfiguration/attack) while using DEBUG for synced
events (normal operation).
This reduces log noise from ~1967 warnings to <10 warnings in a 30-second
production sync test, making real issues visible again.
|
|
|
|
Add comprehensive authorization checks to ensure state events are only
accepted from maintainers of accepted repository announcements. This
implements the core GRASP-01 requirement that pushes must match the
latest state announcement "respecting the maintainer set."
Changes:
1. StatePolicy authorization (src/nostr/policy/state.rs):
- Check authorization BEFORE git data validation (fail-fast)
- Reject if no announcement exists for repository
- Reject if author not in maintainer set
- Use existing helpers: fetch_repository_data() and
pubkey_authorised_for_repo_owners()
- Structured logging for all rejections
2. Purgatory invalidation (src/nostr/builder.rs):
- New method: check_purgatory_state_events_for_identifier()
- Called when announcements accepted (Accept and AcceptMaintainer)
- Re-evaluates state events in purgatory for the identifier
- Processes newly-authorized events (releases from purgatory)
- Keeps unauthorized events for natural expiry (30 min)
- Enables retroactive authorization when announcements arrive late
3. Purgatory sync authorization (src/git/sync.rs):
- Check authorization BEFORE processing git data
- Remove unauthorized events from purgatory (permanent rejection)
- Prevents processing even if git data arrives first
- Structured logging for monitoring
4. Rejected events tracking (src/sync/rejected_index.rs):
- Add support for tracking rejected state events
- New methods: add_state(), contains_state()
- Separate metrics for state rejections
- Enables sync to avoid re-fetching rejected states
5. Sync metrics (src/sync/metrics.rs, src/sync/mod.rs):
- Add state-specific metrics (hot cache, cold index)
- Track rejected states separately from announcements
- Support monitoring of authorization rejections
6. Comprehensive tests (tests/state_authorization.rs):
- test_reject_state_without_announcement
- test_reject_state_from_unauthorized_author
- test_accept_state_from_announcement_author
- test_accept_state_from_maintainer
Security Impact:
- Before: State events could be published by anyone
- After: Only maintainers can publish state events
- Defense-in-depth: Authorization checked at 3 points:
1. On arrival (StatePolicy)
2. On announcement acceptance (purgatory re-evaluation)
3. On git data arrival (purgatory sync)
All tests pass:
- 248 unit tests
- 51 NIP-34 announcement tests
- 4 new state authorization tests
- 9 rejected index tests
Closes: State authorization requirement from GRASP-01 spec
|
|
|
|
- Replace KIND_REPOSITORY_ANNOUNCEMENT with Kind::GitRepoAnnouncement
- Replace KIND_REPOSITORY_STATE with Kind::RepoState
- Replace KIND_PR with Kind::GitPullRequest
- Replace KIND_PR_UPDATE with Kind::GitPullRequestUpdate
- Replace KIND_USER_GRASP_LIST with Kind::GitUserGraspList
- Replace KIND_PATCH with Kind::GitPatch
- Replace KIND_ISSUE with Kind::GitIssue
- Replace KIND_COMMENT with Kind::Comment
- Replace all Kind::Custom(30617|30618|1617|1618|1619|1621|1111|10317) patterns
- Remove all hardcoded KIND_* constants from events.rs
- Update all match statements to use Kind enum directly
- Update all filter builders to use Kind variants
- Update all test helpers and assertions
Benefits:
- Type safety: compiler prevents wrong kind numbers
- Readability: Kind::GitRepoAnnouncement is self-documenting
- Maintainability: single source of truth (rust-nostr)
- IDE support: full autocompletion and refactoring
- Standards: aligns with rust-nostr best practices
Files modified: 21
Constants removed: 9
Patterns replaced: 100+
Tests passing: 222/222
|
|
|
|
Adds expired event tracking to prevent proactive sync from repeatedly
fetching and re-adding events that expired from purgatory without
finding git data.
Key features:
- Track expired events for 7 days to prevent re-sync loops
- Distinguish synced vs user-submitted events (via socket address)
- Allow users to retry expired events (git data might now be available)
- Reject synced expired events (prevents infinite loop)
- Daily cleanup of expired event records older than 7 days
Implementation:
- Added expired_events: DashMap<EventId, Instant> to Purgatory
- Updated event_ids() to include both purgatory + expired events
- Added is_expired(), mark_expired(), cleanup_expired_events()
- Updated cleanup() to mark expired events automatically
- Added is_synced detection in WritePolicy (localhost:0 = synced)
- Policy layer checks is_synced && is_expired() before rejecting
Behavior:
- Negentropy: Filters expired events before fetching (optimal)
- REQ+EOSE: Rejects synced expired events at policy layer
- User submissions: Always allowed to retry (skip expired check)
Testing:
- Added 5 new tests for expired event tracking
- All 222 tests passing
Fixes the infinite re-sync loop where events without git data would
expire, get synced again, expire again, repeat forever.
|
|
Eliminates code duplication by extracting core event processing into
reusable functions. All state and PR event processing now uses the same
unified logic from src/git/process.rs.
Changes:
- Add src/git/process.rs with unified processing functions
- process_state_with_git_data() for state events
- process_pr_with_git_data() for PR events
- Pure functions with comprehensive result types
- Refactor policy handlers to use unified processing
- src/nostr/policy/state.rs: Remove ~70 lines of duplicated logic
- src/nostr/policy/pr_event.rs: Remove ~40 lines of duplicated logic
- Refactor purgatory processing to use unified functions
- src/git/sync.rs: Remove ~125 lines of duplicated logic
- Make extract_owner_from_repo_path() public for reuse
Benefits:
- DRY: Single source of truth for event processing
- Testable: Pure functions with clear contracts
- Maintainable: Changes happen in one place
- Consistent: All code paths use same logic
All 217 unit tests + 40 integration tests pass (257/257).
|
|
Phase 13 of purgatory-sync-redesign:
- Add sync loop startup in main.rs (RealSyncContext + ThrottleManager + start_sync_loop)
- Update add_state() and add_pr() to automatically enqueue for background sync
- Remove start_state_sync() call from state.rs (now handled by sync loop)
- Remove orphaned legacy functions: sync_state_git_data, fetch_missing_oids_from_server,
get_most_complete_local_repo, identify_missing_oids, get_date_of_most_recent_commit_on_default_branch
- Clean up unused imports in purgatory/mod.rs
|
|
- Remove duplicate AlignmentResult struct from nostr/policy/state.rs
- Remove duplicate align_repository_with_state method from StatePolicy
- Import and use the canonical implementation from git::sync
- Re-export AlignmentResult from git::sync in policy/mod.rs
The git::sync version is preferred as it:
- Handles symbolic refs (ref:) properly by skipping them
- Uses git::oid_exists which is more general than git::commit_exists
- Has a cleaner iteration pattern (delete first, then update/create)
|
|
|
|
|
|
don't save new events destined for purgatory events directly to db
or serve on websockets
don't download events already in purgatory via negentropy sync
|
|
|
|
|
|
|
|
|
|
so we can more easily support grasp purgatory feature
|
|
|
|
Main lib (src/):
- Add #[allow(dead_code)] for build_info field (stored to prevent Prometheus unregistration)
- Add #[allow(dead_code)] for first_seen field (reserved for future rate limiting)
- Replace .or_insert_with(RelaySyncNeeds::default) with .or_default()
- Replace manual div_ceil implementations with .div_ceil(100)
Test code (tests/):
- Replace .expect(&format!(...)) with .unwrap_or_else(|_| panic!(...))
- Remove needless borrows in fetch_metrics() calls
- Add #[allow(dead_code)] and #[allow(unused_imports)] to test helpers module
grasp-audit:
- Apply cargo fmt to fix formatting
|
|
|
|
- Add relay discovery from stored announcements
- Implement FilterService with three-layer strategy
- Support multiple simultaneous relay connections
- Filter batching for large tag sets
|
|
- Add src/sync/ module with SyncManager
- Add NGIT_SYNC_RELAY_URL config option
- Subscribe to kind 30617 on configured relay
- Validate synced events through Nip34WritePolicy
- Integration test with two TestRelay instances
|
|
Split the ~900 line Nip34WritePolicy into focused sub-policies for improved
testability and maintainability:
- AnnouncementPolicy - Repository announcement validation
- StatePolicy - State event validation + ref alignment
- PrEventPolicy - PR/PR Update validation
- RelatedEventPolicy - Forward/backward reference checking
The main Nip34WritePolicy now delegates to these sub-policies via a shared
PolicyContext that provides domain, database, and git_data_path.
Also updates:
- README.md: Accurate project structure reflecting actual implementation
- docs/learnings: Marks this technical debt item as complete
|
|
|
|
|
|
- Add nostr-lmdb dependency (v0.44) for persistent storage
- Create SharedDatabase type alias for database abstraction
- Update all database-related functions to use trait object
- Support runtime selection via NGIT_DATABASE_BACKEND env var
Database backends:
- memory: In-memory (default, fastest, no persistence)
- lmdb: LMDB backend (persistent, general purpose)
All 34 tests pass with the new implementation.
|
|
if we have the OIDs
|
|
|
|
incorrect ref on event receive
|
|
|
|
|
|
|
|
|
|
we dont need it
|
|
but do we really nedd to create a blank commit?
I dont think ngit-relay does that.
Do we need to se the default branch or is this automatic?
|
|
|
|
Add environment variable configuration for database backend selection:
- Added DatabaseBackend enum (memory, nostrdb, lmdb) in src/config.rs
- Updated relay builder to use configured backend in src/nostr/builder.rs
- Added NGIT_DATABASE_BACKEND to .env.example with documentation
- Updated docs/reference/configuration.md with backend comparison table
NostrDB and LMDB backends prepared for future implementation when
nostr-relay-builder adds support. Currently defaults to in-memory
database with warning logs when persistent backends are selected.
|
|
- Fixed bug where regular replaceable events (10000-19999) were using
wrong address format (3 colons instead of 2)
- Regular replaceable now use kind:pubkey format (1 colon)
- Parameterized replaceable (30000-39999) correctly use kind:pubkey:d-identifier (2 colons)
- Refactored to eliminate code duplication between both replaceable event types
- Updated documentation to reflect correct addressing for both types
|
|
- Remove uppercase 'Q' tag (not in Nostr spec)
- Add support for addressable references in 'q' tags
- Optimize queries based on event type:
- Addressable events (kind >= 30000): only check a, A, q with coordinates
- Regular events: only check e, E, q with event IDs
- Handle addressable events without 'd' tag (empty identifier)
- Reduce query count from up to 6 to maximum 3 per event type
|
|
- Replace individual queries with batched operations
- Group addressable references by kind to reduce queries
- Query all event IDs in single batch operation
- Reduces N+M queries to ~K+1 queries (75% reduction typical case)
- All 37 tests passing, functionality preserved
|
|
- Add Nip34WritePolicy with Arc<MemoryDatabase> for stateful event validation
- Implement full GRASP-01 event acceptance policy:
* Accept events referencing accepted repositories (via a, A, q tags)
* Accept events referencing accepted events (transitive, via e, E, q tags)
* Support forward references (events referenced by accepted events)
* Reject orphan events with no valid references
- Extract and validate all reference tag types (a, A, q, e, E)
- Query database for repository and event existence checks
- Implement fail-secure error handling for database query failures
Test improvements:
- Fix send_and_verify_rejected to handle relay rejection errors properly
- Fix RepoWithIssue fixture usage in forward reference tests
- Add database synchronization polling for race condition mitigation
- Achieve 94% test pass rate (16/17 integration tests passing)
|
|
|
|
|
|
|
|
- WebSocket-based relay using tokio-tungstenite
- Full NIP-01 protocol support (EVENT, REQ, CLOSE)
- Event validation (signature and ID)
- In-memory event storage
- Filter support (IDs, authors, kinds, since/until)
- Configuration via environment variables
- Nix flake for reproducible builds
- Test automation script
All 6 NIP-01 smoke tests passing (100%)
|