diff --git a/CLAUDE.md b/CLAUDE.md index 495ce60..6b5b4e9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -231,11 +231,9 @@ backup because this was deferred — do not repeat. - **MAVLinkRouter** is the confirmed router. Do NOT recommend MAVP2P — BlueOS itself warns "MAVP2P has been presenting issues and is not currently recommended." - MAVP2P was tried and reverted. Do not suggest it again. -- The BlueOS ardupilot-manager FD exhaustion bug (Docker default `nofile=1024`) is - fixed permanently via `/etc/docker/daemon.json` with `nofile` hard/soft = 65536. - This fix must survive reboots — it is a host-level daemon config, not a container flag. - Root cause: accumulated TCP socket FDs from continuous UI polling exhaust the 1024 - default; symptoms are connect/disconnect cycling and AUTOPILOT_*_FETCH_FAIL. +- **ardupilot-manager FD exhaustion — CORRECTED ROOT CAUSE (9 Jul 2026).** The earlier claim that this was "fixed permanently" by `/etc/docker/daemon.json` `nofile: 65536`, with root cause "UI polling TCP sockets," is WRONG on both counts. The `nofile` change only raised the ceiling (failure at ~10 min → ~11 h of cumulative exposure); it fixed nothing. The leaked FDs are `/dev/i2c-1` handles, NOT TCP sockets. Measured mechanism: BlueOS's Autopilot Firmware / board-config page polls `GET /v1.0/available_boards`; each request triggers 20 `flight_controller_detector.linux.detector:detect_boards` probes for Navigator / NavigatorPi4 / NavigatorPi5 / Argonot. Each Linux-board probe opens `/dev/i2c-1` and never closes it on failure. We run a Pixhawk 2.4.8 on USB serial (PlatformType.Serial), so every probe fails and every probe leaks. Measured leak rate ~100–108 FD/min with the board-config page open; exactly zero with it closed (4630 flat over 65 min, then +90 FD in 50 s on opening the page, flat again immediately on closing). Confirmed cause-and-effect, reversible. Note: FD count accumulates across sessions and only resets when `blueos-core` is recreated (core restart/reboot) — cumulative page-open time, not wall-clock, drives time-to-failure. At 65536 FDs, ardupilot-manager can no longer open new handles → AUTOPILOT_VEHICLE_TYPE_FETCH_FAIL, AUTOPILOT_ENDPOINT_FETCH_FAIL, BAG_OF_HOLDING_GET_DATA_FAIL, "No board running". Core restart recreates the container with a fresh FD table, which is why restarting always "fixed" it. This is an upstream BlueOS 1.4.3 bug; do NOT patch BlueOS (updates would revert it). +- **Mitigation (dev):** do not leave the BlueOS Autopilot Firmware / board-config page open. Open, act, close. Other BlueOS pages and Cockpit are unaffected. +- **Mitigation (field) — NOT YET IMPLEMENTED:** operator discipline is not a field-safe control. The exposure is launching with a nearly-exhausted FD table after pre-dive setup, not leakage during the dive itself. Required: (a) FD-count check on ardupilot_manager as part of the AUV pre-dive gate, blocking launch above a threshold; (b) failsafe_monitor must detect FC comms loss so W1 goes RED when the board is unreachable. See open items. ### Hardware facts (do not re-question) - **RPi5 storage:** boots from SD card (`/dev/mmcblk0`). NVMe holds DATA ONLY (`/data` — @@ -381,6 +379,20 @@ entries; correct forward with a new dated entry if something needs revising. (viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn delay — no failsafe assessment in that window; flagged for future design consideration. +- 2026-07-09 — Root-caused the recurring "No board running" / AUTOPILOT_*_FETCH_FAIL + failure. NOT the previously-documented TCP-socket/UI-polling cause. Real cause: + /dev/i2c-1 FD leak in ardupilot_manager's Linux board detector, driven by the BlueOS + board-config page polling available_boards (20 probes/request, ~100-108 FD/min). + Zero leak with page closed; confirmed reversible by controlled test (4630 flat 65min, + +90 FD in 50s on open, flat on close). nofile=65536 only delayed failure to ~11h + cumulative page-open time. Upstream BlueOS 1.4.3 bug — not patching. Corrected the + stale root cause in this file. +- 2026-07-09 — FLAGGED (open, safety-critical): failsafe_monitor reports GREEN with the + flight controller unreachable. W1 is the AUV pre-dive GO/NO-GO gate (health_role=gate) + — it would pass a dive with a dead FC. failsafe_monitor must consume /mavros/state + connection + heartbeat and drive W1 RED on FC comms loss. W1 status line should carry + cause text (e.g. "Vehicle Disconnected", "Heartbeat Lost", "No Board Running"). Also + flagged: no FD-count check on ardupilot_manager in the pre-dive gate. ---