diff options
| author | Your Name <you@example.com> | 2026-05-21 15:44:46 +0530 |
|---|---|---|
| committer | Your Name <you@example.com> | 2026-05-21 15:44:46 +0530 |
| commit | 243e4808246555f86d464d1c476681a902e644ff (patch) | |
| tree | 4334ebf0f8c2e2ee7f5ebccb93dedd68a2e95b18 | |
| parent | 46d9c97e11ea0b03a7dc03fdef6bd360cbe53d44 (diff) | |
feat(wallet): lazy keyset loading with exponential backoff + jitter
- Remove eager load_keysets() from init_wallet(), load on first use
- Add ensure_keysets() with binary exponential backoff + random jitter
- Guard all wallet operations (receive, send, melt, swap_all) with ensure_keysets()
- Skip checkstate on TLS failure, let wallet swap verify proofs instead
- Pin HTTP server to core 0 (fixes TLS cert verification in checkstate)
- Rewrite nucula http.c to use manual open/write/read (same as working health probe)
- Disable hardware MPI (CONFIG_MBEDTLS_HARDWARE_MPI=n)
- Add http.h stub for unit tests
Known issue: progressive TLS failure after 2-3 connections (PK verify 0x4290),
wallet receive swap fails. Needs deeper mbedTLS/PSRAM investigation.
| -rw-r--r-- | PLAN_lazy_keysets.md | 67 | ||||
| -rw-r--r-- | components/nucula_lib/nucula_wallet.cpp | 83 | ||||
| -rw-r--r-- | main/cashu.c | 1 | ||||
| -rw-r--r-- | main/tollgate_api.c | 23 | ||||
| -rw-r--r-- | sdkconfig.defaults | 2 | ||||
| -rw-r--r-- | tests/unit/stubs/http.h | 41 |
6 files changed, 194 insertions, 23 deletions
diff --git a/PLAN_lazy_keysets.md b/PLAN_lazy_keysets.md new file mode 100644 index 0000000..a92dbf3 --- /dev/null +++ b/PLAN_lazy_keysets.md | |||
| @@ -0,0 +1,67 @@ | |||
| 1 | # Plan: Lazy Keyset Loading + Jittered Exponential Backoff | ||
| 2 | |||
| 3 | ## Problem | ||
| 4 | `nucula_wallet_init()` calls `load_keysets()` which does HTTPS to the mint. At boot (~3s after STA gets IP), other services are initializing, internal RAM is fragmented, and TLS allocation fails. Wallet ends up with zero keysets → `receive()` fails because `active_keyset()` returns null. | ||
| 5 | |||
| 6 | ## Solution | ||
| 7 | Lazy keyset loading (load on first use) + exponential backoff with jitter on failure. Blocking approach — no background task. | ||
| 8 | |||
| 9 | ## Checklist | ||
| 10 | |||
| 11 | ### Implementation — DONE | ||
| 12 | - [x] Add per-slot state tracking (`s_keysets_loaded`, `s_keyset_attempts`, delay constants) | ||
| 13 | - [x] Add `ensure_keysets(int slot)` helper with exponential backoff + jitter | ||
| 14 | - [x] Remove eager `load_keysets()` from `init_wallet()` | ||
| 15 | - [x] Add `ensure_keysets()` guard to `nucula_wallet_receive()` | ||
| 16 | - [x] Add `ensure_keysets()` guard to `nucula_wallet_send()` | ||
| 17 | - [x] Add `ensure_keysets()` guard to `nucula_wallet_melt()` | ||
| 18 | - [x] Add `ensure_keysets()` guard to `nucula_wallet_swap_all()` | ||
| 19 | - [x] Add required includes (`freertos/FreeRTOS.h`, `freertos/task.h`, `esp_random.h`) | ||
| 20 | - [x] Fix lazy keyset loading success log (save attempts before reset) | ||
| 21 | |||
| 22 | ### TLS Fixes — DONE | ||
| 23 | - [x] Skip checkstate on TLS failure, let wallet swap verify (tollgate_api.c) | ||
| 24 | - [x] Pin HTTP server to core 0 (fixes checkstate and keyset TLS) | ||
| 25 | - [x] Rewrite nucula http.c to use manual open/write/read path (same as working health probe) | ||
| 26 | - [x] Disable hardware MPI (CONFIG_MBEDTLS_HARDWARE_MPI=n, did not fix but kept for safety) | ||
| 27 | - [x] Add 2s delay after ensure_keysets before receive to let TLS resources free | ||
| 28 | |||
| 29 | ### Verification — DONE | ||
| 30 | - [x] `make test-unit` passes (407/407) | ||
| 31 | - [x] Firmware builds and flashes to Board A | ||
| 32 | - [x] Payment accepted (session created with correct allotment) | ||
| 33 | - [x] Keyset loading succeeds on first use (lazy loading confirmed via serial) | ||
| 34 | - [x] Checkstate succeeds (Certificate validated via serial) | ||
| 35 | |||
| 36 | ### Remaining Issue — TLS Progressive Failure | ||
| 37 | - [ ] After 2-3 TLS connections, subsequent connections fail with `PK verify failed 0x4290` | ||
| 38 | - [ ] This affects the wallet's swap POST (3rd TLS connection in payment flow) | ||
| 39 | - [ ] Root cause: internal RAM fragmentation after multiple TLS handshakes | ||
| 40 | - [ ] Workaround tried: core pinning, manual HTTP client, hardware MPI disable, delays | ||
| 41 | - [ ] **Next step**: Investigate mbedTLS internal allocations, consider forcing SSL context to PSRAM, or use `mbedtls_ssl_session_reset()` to reuse SSL context | ||
| 42 | |||
| 43 | ## Design Details | ||
| 44 | |||
| 45 | ### Parameters | ||
| 46 | - `BASE_DELAY_MS = 1000` (1s initial delay) | ||
| 47 | - `MAX_DELAY_MS = 30000` (30s cap) | ||
| 48 | - `JITTER_MS = 500` (random 0-500ms added) | ||
| 49 | - No max attempts — retries indefinitely until success | ||
| 50 | |||
| 51 | ### ensure_keysets(slot) logic | ||
| 52 | 1. If `s_keysets_loaded[slot]` → return true (no-op) | ||
| 53 | 2. If `s_wallets[slot]->keysets()` non-empty → set loaded flag, return true (NVS cache hit) | ||
| 54 | 3. Compute `delay = min(BASE_DELAY_MS << attempts, MAX_DELAY_MS) + (esp_random() % JITTER_MS)` | ||
| 55 | 4. `vTaskDelay(delay / portTICK_PERIOD_MS)` | ||
| 56 | 5. Call `s_wallets[slot]->load_keysets()` | ||
| 57 | - Success → set loaded flag, log success, return true | ||
| 58 | - Failure → increment attempts counter, return false | ||
| 59 | |||
| 60 | ### Key Changes | ||
| 61 | | File | Change | | ||
| 62 | |------|--------| | ||
| 63 | | `components/nucula_lib/nucula_wallet.cpp` | Lazy keysets + ensure_keysets guards | | ||
| 64 | | `main/tollgate_api.c` | Skip checkstate on failure, core_id=0 | | ||
| 65 | | `nucula_src/main/http.c` | Manual open/write/read (same as health probe) | | ||
| 66 | | `sdkconfig.defaults` | MBEDTLS_HARDWARE_MPI=n | | ||
| 67 | | `tests/unit/stubs/http.h` | Stub for unit tests | | ||
diff --git a/components/nucula_lib/nucula_wallet.cpp b/components/nucula_lib/nucula_wallet.cpp index 7c9141d..278c9bc 100644 --- a/components/nucula_lib/nucula_wallet.cpp +++ b/components/nucula_lib/nucula_wallet.cpp | |||
| @@ -4,8 +4,11 @@ | |||
| 4 | #include "crypto.h" | 4 | #include "crypto.h" |
| 5 | #include "hex.h" | 5 | #include "hex.h" |
| 6 | #include "esp_log.h" | 6 | #include "esp_log.h" |
| 7 | #include "esp_random.h" | ||
| 7 | #include "secp256k1.h" | 8 | #include "secp256k1.h" |
| 8 | #include "cJSON.h" | 9 | #include "cJSON.h" |
| 10 | #include "freertos/FreeRTOS.h" | ||
| 11 | #include "freertos/task.h" | ||
| 9 | #include <cstring> | 12 | #include <cstring> |
| 10 | #include <string> | 13 | #include <string> |
| 11 | #include <vector> | 14 | #include <vector> |
| @@ -18,6 +21,55 @@ static secp256k1_context *s_ctx = nullptr; | |||
| 18 | static cashu::Wallet *s_wallets[MAX_WALLETS] = {}; | 21 | static cashu::Wallet *s_wallets[MAX_WALLETS] = {}; |
| 19 | static int s_wallet_count = 0; | 22 | static int s_wallet_count = 0; |
| 20 | static char s_wallet_urls[MAX_WALLETS][256] = {}; | 23 | static char s_wallet_urls[MAX_WALLETS][256] = {}; |
| 24 | static bool s_keysets_loaded[MAX_WALLETS] = {}; | ||
| 25 | static int s_keyset_attempts[MAX_WALLETS] = {}; | ||
| 26 | |||
| 27 | static const int KEYSET_BASE_DELAY_MS = 1000; | ||
| 28 | static const int KEYSET_MAX_DELAY_MS = 30000; | ||
| 29 | static const int KEYSET_JITTER_MS = 500; | ||
| 30 | |||
| 31 | static bool ensure_keysets(int slot) | ||
| 32 | { | ||
| 33 | if (slot < 0 || slot >= MAX_WALLETS || !s_wallets[slot]) | ||
| 34 | return false; | ||
| 35 | |||
| 36 | if (s_keysets_loaded[slot]) | ||
| 37 | return true; | ||
| 38 | |||
| 39 | if (!s_wallets[slot]->keysets().empty()) { | ||
| 40 | s_keysets_loaded[slot] = true; | ||
| 41 | return true; | ||
| 42 | } | ||
| 43 | |||
| 44 | int delay_ms = KEYSET_BASE_DELAY_MS << s_keyset_attempts[slot]; | ||
| 45 | if (delay_ms > KEYSET_MAX_DELAY_MS) | ||
| 46 | delay_ms = KEYSET_MAX_DELAY_MS; | ||
| 47 | delay_ms += (int)(esp_random() % KEYSET_JITTER_MS); | ||
| 48 | |||
| 49 | ESP_LOGI(TAG, "Keyset load attempt %d for slot %d, waiting %d ms", | ||
| 50 | s_keyset_attempts[slot] + 1, slot, delay_ms); | ||
| 51 | |||
| 52 | vTaskDelay(pdMS_TO_TICKS(delay_ms)); | ||
| 53 | |||
| 54 | if (s_wallets[slot]->load_keysets()) { | ||
| 55 | int attempts = s_keyset_attempts[slot] + 1; | ||
| 56 | s_keysets_loaded[slot] = true; | ||
| 57 | s_keyset_attempts[slot] = 0; | ||
| 58 | ESP_LOGI(TAG, "Keyset load succeeded for slot %d after %d attempt(s)", | ||
| 59 | slot, attempts); | ||
| 60 | return true; | ||
| 61 | } | ||
| 62 | |||
| 63 | s_keyset_attempts[slot]++; | ||
| 64 | return false; | ||
| 65 | } | ||
| 66 | |||
| 67 | static int wallet_slot(const cashu::Wallet *w) | ||
| 68 | { | ||
| 69 | for (int i = 0; i < s_wallet_count; i++) | ||
| 70 | if (s_wallets[i] == w) return i; | ||
| 71 | return -1; | ||
| 72 | } | ||
| 21 | 73 | ||
| 22 | static cashu::Wallet *find_wallet_for_token(const cashu::Token &tok) | 74 | static cashu::Wallet *find_wallet_for_token(const cashu::Token &tok) |
| 23 | { | 75 | { |
| @@ -61,14 +113,9 @@ static esp_err_t init_wallet(int slot, const char *mint_url) | |||
| 61 | 113 | ||
| 62 | s_wallets[slot]->load_from_nvs(); | 114 | s_wallets[slot]->load_from_nvs(); |
| 63 | 115 | ||
| 64 | if (!s_wallets[slot]->load_keysets()) { | 116 | ESP_LOGI(TAG, "Wallet[%d] initialized: url=%s balance=%d proofs=%d (keysets lazy)", |
| 65 | ESP_LOGW(TAG, "Keyset load failed for slot %d (may be offline)", slot); | ||
| 66 | } | ||
| 67 | |||
| 68 | ESP_LOGI(TAG, "Wallet[%d] initialized: url=%s balance=%d proofs=%d keysets=%d", | ||
| 69 | slot, mint_url, s_wallets[slot]->balance(), | 117 | slot, mint_url, s_wallets[slot]->balance(), |
| 70 | (int)s_wallets[slot]->proofs().size(), | 118 | (int)s_wallets[slot]->proofs().size()); |
| 71 | (int)s_wallets[slot]->keysets().size()); | ||
| 72 | return ESP_OK; | 119 | return ESP_OK; |
| 73 | } | 120 | } |
| 74 | 121 | ||
| @@ -136,6 +183,13 @@ esp_err_t nucula_wallet_receive(const char *token_str) | |||
| 136 | return ESP_FAIL; | 183 | return ESP_FAIL; |
| 137 | } | 184 | } |
| 138 | 185 | ||
| 186 | if (!ensure_keysets(wallet_slot(w))) { | ||
| 187 | ESP_LOGE(TAG, "Keysets not available for receive"); | ||
| 188 | return ESP_FAIL; | ||
| 189 | } | ||
| 190 | |||
| 191 | vTaskDelay(pdMS_TO_TICKS(2000)); | ||
| 192 | |||
| 139 | std::vector<cashu::Proof> proofs_out; | 193 | std::vector<cashu::Proof> proofs_out; |
| 140 | if (!w->receive(tok, proofs_out)) { | 194 | if (!w->receive(tok, proofs_out)) { |
| 141 | ESP_LOGE(TAG, "Receive failed"); | 195 | ESP_LOGE(TAG, "Receive failed"); |
| @@ -157,6 +211,11 @@ esp_err_t nucula_wallet_send(uint64_t amount_sat, char *token_out, size_t token_ | |||
| 157 | cashu::Wallet *w = find_wallet_for_send(amount); | 211 | cashu::Wallet *w = find_wallet_for_send(amount); |
| 158 | if (!w) return ESP_FAIL; | 212 | if (!w) return ESP_FAIL; |
| 159 | 213 | ||
| 214 | if (!ensure_keysets(wallet_slot(w))) { | ||
| 215 | ESP_LOGE(TAG, "Keysets not available for send"); | ||
| 216 | return ESP_FAIL; | ||
| 217 | } | ||
| 218 | |||
| 160 | std::vector<cashu::Proof> selected, remaining; | 219 | std::vector<cashu::Proof> selected, remaining; |
| 161 | if (!w->select_proofs(amount, selected, remaining)) { | 220 | if (!w->select_proofs(amount, selected, remaining)) { |
| 162 | ESP_LOGE(TAG, "Insufficient balance for %d sat", amount); | 221 | ESP_LOGE(TAG, "Insufficient balance for %d sat", amount); |
| @@ -246,6 +305,11 @@ esp_err_t nucula_wallet_swap_all(void) | |||
| 246 | auto &proofs = mutable_proofs(s_wallets[i]); | 305 | auto &proofs = mutable_proofs(s_wallets[i]); |
| 247 | if (proofs.empty()) continue; | 306 | if (proofs.empty()) continue; |
| 248 | 307 | ||
| 308 | if (!ensure_keysets(i)) { | ||
| 309 | ESP_LOGE(TAG, "Keysets not available for swap_all wallet[%d]", i); | ||
| 310 | continue; | ||
| 311 | } | ||
| 312 | |||
| 249 | int old_balance = s_wallets[i]->balance(); | 313 | int old_balance = s_wallets[i]->balance(); |
| 250 | 314 | ||
| 251 | std::vector<cashu::Proof> inputs = proofs; | 315 | std::vector<cashu::Proof> inputs = proofs; |
| @@ -301,6 +365,11 @@ esp_err_t nucula_wallet_melt(const char *bolt11_invoice, uint64_t max_fee_sats) | |||
| 301 | } | 365 | } |
| 302 | if (!w) return ESP_FAIL; | 366 | if (!w) return ESP_FAIL; |
| 303 | 367 | ||
| 368 | if (!ensure_keysets(wallet_slot(w))) { | ||
| 369 | ESP_LOGE(TAG, "Keysets not available for melt"); | ||
| 370 | return ESP_FAIL; | ||
| 371 | } | ||
| 372 | |||
| 304 | cashu::MeltQuote quote; | 373 | cashu::MeltQuote quote; |
| 305 | if (!w->request_melt_quote(std::string(bolt11_invoice), quote)) { | 374 | if (!w->request_melt_quote(std::string(bolt11_invoice), quote)) { |
| 306 | ESP_LOGE(TAG, "Melt quote request failed"); | 375 | ESP_LOGE(TAG, "Melt quote request failed"); |
diff --git a/main/cashu.c b/main/cashu.c index da12ff9..4bcda4d 100644 --- a/main/cashu.c +++ b/main/cashu.c | |||
| @@ -199,6 +199,7 @@ esp_err_t cashu_check_proof_states(const char *mint_url, const cashu_token_t *to | |||
| 199 | if (!client) { free(post_body); free(resp_buf); return ESP_FAIL; } | 199 | if (!client) { free(post_body); free(resp_buf); return ESP_FAIL; } |
| 200 | 200 | ||
| 201 | esp_http_client_set_header(client, "Content-Type", "application/json"); | 201 | esp_http_client_set_header(client, "Content-Type", "application/json"); |
| 202 | |||
| 202 | esp_err_t err = esp_http_client_open(client, strlen(post_body)); | 203 | esp_err_t err = esp_http_client_open(client, strlen(post_body)); |
| 203 | if (err != ESP_OK) { | 204 | if (err != ESP_OK) { |
| 204 | ESP_LOGE(TAG, "checkstate open failed: %s", esp_err_to_name(err)); | 205 | ESP_LOGE(TAG, "checkstate open failed: %s", esp_err_to_name(err)); |
diff --git a/main/tollgate_api.c b/main/tollgate_api.c index 80fe6ed..7f2102e 100644 --- a/main/tollgate_api.c +++ b/main/tollgate_api.c | |||
| @@ -280,16 +280,8 @@ static esp_err_t api_post_payment(httpd_req_t *req) | |||
| 280 | err = cashu_check_proof_states(mint_url, token, states, &state_count); | 280 | err = cashu_check_proof_states(mint_url, token, states, &state_count); |
| 281 | ESP_LOGI(TAG, "Stack HWM after checkstate: %u", uxTaskGetStackHighWaterMark(NULL)); | 281 | ESP_LOGI(TAG, "Stack HWM after checkstate: %u", uxTaskGetStackHighWaterMark(NULL)); |
| 282 | if (err != ESP_OK) { | 282 | if (err != ESP_OK) { |
| 283 | free(states); | 283 | ESP_LOGW(TAG, "Checkstate failed, proceeding without spend check (wallet swap will verify)"); |
| 284 | free(token); | 284 | state_count = 0; |
| 285 | cJSON *notice = create_notice("error", "payment-error-verification", "Failed to verify token with mint"); | ||
| 286 | char *json = cJSON_PrintUnformatted(notice); | ||
| 287 | httpd_resp_set_status(req, "502 Bad Gateway"); | ||
| 288 | httpd_resp_set_type(req, "application/json"); | ||
| 289 | httpd_resp_send(req, json, strlen(json)); | ||
| 290 | cJSON_free(json); | ||
| 291 | cJSON_Delete(notice); | ||
| 292 | return ESP_OK; | ||
| 293 | } | 285 | } |
| 294 | 286 | ||
| 295 | for (int i = 0; i < state_count; i++) { | 287 | for (int i = 0; i < state_count; i++) { |
| @@ -795,11 +787,12 @@ esp_err_t tollgate_api_start(void) | |||
| 795 | { | 787 | { |
| 796 | if (s_api_server) return ESP_OK; | 788 | if (s_api_server) return ESP_OK; |
| 797 | 789 | ||
| 798 | httpd_config_t config = HTTPD_DEFAULT_CONFIG(); | 790 | httpd_config_t config = HTTPD_DEFAULT_CONFIG(); |
| 799 | config.server_port = 2121; | 791 | config.server_port = 2121; |
| 800 | config.ctrl_port = 32769; | 792 | config.ctrl_port = 32769; |
| 801 | config.max_uri_handlers = 16; | 793 | config.max_uri_handlers = 16; |
| 802 | config.stack_size = 16384; | 794 | config.stack_size = 16384; |
| 795 | config.core_id = 0; | ||
| 803 | 796 | ||
| 804 | esp_err_t ret = httpd_start(&s_api_server, &config); | 797 | esp_err_t ret = httpd_start(&s_api_server, &config); |
| 805 | if (ret != ESP_OK) { | 798 | if (ret != ESP_OK) { |
diff --git a/sdkconfig.defaults b/sdkconfig.defaults index bed04fe..f67ce67 100644 --- a/sdkconfig.defaults +++ b/sdkconfig.defaults | |||
| @@ -40,6 +40,7 @@ CONFIG_MBEDTLS_CERTIFICATE_BUNDLE=y | |||
| 40 | CONFIG_MBEDTLS_DYNAMIC_BUFFER=y | 40 | CONFIG_MBEDTLS_DYNAMIC_BUFFER=y |
| 41 | CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=4096 | 41 | CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=4096 |
| 42 | CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096 | 42 | CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096 |
| 43 | CONFIG_MBEDTLS_HARDWARE_MPI=n | ||
| 43 | 44 | ||
| 44 | # PSRAM (ESP32-S3 has 8MB) | 45 | # PSRAM (ESP32-S3 has 8MB) |
| 45 | CONFIG_SPIRAM=y | 46 | CONFIG_SPIRAM=y |
| @@ -48,4 +49,3 @@ CONFIG_SPIRAM_SPEED_80M=y | |||
| 48 | CONFIG_SPIRAM_USE_MALLOC=y | 49 | CONFIG_SPIRAM_USE_MALLOC=y |
| 49 | CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=4096 | 50 | CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=4096 |
| 50 | CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=32768 | 51 | CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=32768 |
| 51 | CONFIG_MBEDTLS_DYNAMIC_BUFFER=y | ||
diff --git a/tests/unit/stubs/http.h b/tests/unit/stubs/http.h new file mode 100644 index 0000000..753ebd8 --- /dev/null +++ b/tests/unit/stubs/http.h | |||
| @@ -0,0 +1,41 @@ | |||
| 1 | #ifndef STUBS_HTTP_H | ||
| 2 | #define STUBS_HTTP_H | ||
| 3 | |||
| 4 | #include "esp_err.h" | ||
| 5 | #include <stddef.h> | ||
| 6 | |||
| 7 | typedef struct { | ||
| 8 | int status; | ||
| 9 | char *body; | ||
| 10 | size_t body_len; | ||
| 11 | } http_response_t; | ||
| 12 | |||
| 13 | static inline esp_err_t http_get(const char *url, http_response_t *resp) | ||
| 14 | { | ||
| 15 | (void)url; (void)resp; | ||
| 16 | return ESP_FAIL; | ||
| 17 | } | ||
| 18 | |||
| 19 | static inline esp_err_t http_post_json(const char *url, const char *json_body, | ||
| 20 | http_response_t *resp) | ||
| 21 | { | ||
| 22 | (void)url; (void)json_body; (void)resp; | ||
| 23 | return ESP_FAIL; | ||
| 24 | } | ||
| 25 | |||
| 26 | static inline esp_err_t http_post_json_timeout(const char *url, const char *json_body, | ||
| 27 | http_response_t *resp, int timeout_ms) | ||
| 28 | { | ||
| 29 | (void)url; (void)json_body; (void)resp; (void)timeout_ms; | ||
| 30 | return ESP_FAIL; | ||
| 31 | } | ||
| 32 | |||
| 33 | static inline void http_response_free(http_response_t *resp) | ||
| 34 | { | ||
| 35 | if (resp && resp->body) { | ||
| 36 | free(resp->body); | ||
| 37 | resp->body = NULL; | ||
| 38 | } | ||
| 39 | } | ||
| 40 | |||
| 41 | #endif | ||