diff --git a/docs/design_intent_register.md b/docs/design_intent_register.md index 7622286..18d9850 100644 --- a/docs/design_intent_register.md +++ b/docs/design_intent_register.md @@ -1,7 +1,7 @@ # Design Intent Register — Argonaut 3 -**Version:** 1.2 -**Date:** 11 July 2026 +**Version:** 1.3 +**Date:** 13 July 2026 **File:** `docs/design_intent_register.md` (repo-tracked, no version number in filename) **Purpose of this register:** Capture operator/owner *intent* — what each behaviour MEANS and MUST/MUST NOT do — separately from mechanism (topics, code). Mechanism is recorded elsewhere. This register is the source of truth for *why*, against which any implementation is validated. Every entry dated and attributed. @@ -12,17 +12,22 @@ | Version | Date | Change | |---|---|---| +| 1.3 | 13 Jul 2026 | **New Parked Design Item:** mode-aware safe-zone-reached arrival event (relates to the DIR-7 "in-mission recovery is mode-dependent" addendum) — the permanent `/rov/nav/safe_zone_reached` interface, mode-aware/sensor-derived arrival judgement (GPS at surface + EKF/dead-reckoning underwater), GPS recorded as first-class across mission types (not a hull/jacket-specific surface concern), gate-only scope, and the temporary W1-button bench-scaffolding publisher pending navigation. Implemented same day in `failsafe_monitor` (`bf815cc`) — see `docs/handover.md` §15f for the fix narrative; this entry records the design intent, not the commit detail. DIR-7 addendum's Implementation note updated: the mode-dependent recovery mechanism is no longer mode-blind (`2a3e577`, 13 Jul 2026). | | 1.2 | 11 Jul 2026 | **DIR-7's FC-connectivity MUST — RESOLVED**, four commits 11 Jul 2026 (full detail in `docs/handover.md` Changelog); open-flag callout updated accordingly. **DIR-7 gains two addenda:** in-mission recovery is mode-dependent (AUV commits to recovery on condition-clear, does not auto-resume; ROV holds and hands the continue-or-recover decision to the operator); and in-mission AMBER runtime-response (AMBER means action-required, not just informational, once a mission is underway — response branches by mode, AUV's autonomous curtailment logic left unspecified pending further design). **New "Parked Design Items" section** — auto-recovery (failsafe_monitor should attempt to re-establish the FC link itself before escalating) and a dedicated `string cause` field in `FailsafeStatus.msg` (replacing the `cockpit_bridge` message-substring shortcut) — both tracked as open implementation gaps against existing DIR-7/12 intent, not new decisions. Filename de-versioned as part of the 11 Jul doc restructure — this file moved into the repo `docs/` folder. | | 1.1 | 9 Jul 2026 | **DIR-1 corrected** — the *tether/HILT link*, not the power source, determines operating mode. A tethered vehicle may run on battery or topside supply. **DIR-7 expanded** — AUV `gate` vs ROV `advisory` health semantics; and a MUST that health assessment include flight-controller connectivity (a monitor with no FC input must never report GREEN). **DIR-9 superseded by DIR-9a** — recording is a single ARM-bounded black-box tier, not continuous-from-boot. **DIR-12 added** — vehicle health is distinct from autonomy health. | | 1.0 | Jun 2026 | Initial register. | -> ⚠ **Known implementation drift (updated 11 Jul 2026):** `recording_manager.py` still implements +> ⚠ **Known implementation drift (updated 13 Jul 2026):** `recording_manager.py` still implements > the superseded DIR-9 (continuous-from-boot) model, NOT the governing DIR-9a — **still open.** -> `failsafe_monitor`'s DIR-7 FC-connectivity MUST is **resolved** (11 Jul 2026, see DIR-7 below); -> DIR-12's broader vehicle-health framing remains open beyond the FC-link piece (FD-headroom -> check on `ardupilot_manager` not built). The two new DIR-7 addenda (mode-dependent in-mission -> recovery, AMBER runtime-response) and both Parked Design Items are **intent recorded, not -> implemented** — see their sections. +> `failsafe_monitor`'s DIR-7 FC-connectivity MUST is **resolved** (11 Jul 2026, see DIR-7 below). +> The DIR-7 addendum "in-mission recovery is mode-dependent" is **essentially implemented** (13 +> Jul 2026 — `failsafe_monitor` now reads `health_role` and branches recovery accordingly, +> including the gate-mode arrival exit via the new `/rov/nav/safe_zone_reached` interface); +> remaining gap is the event's real publisher (navigation not yet built) — see the new "Parked +> Design Items" entry below. The AMBER runtime-response addendum, auto-recovery, and the +> dedicated `string cause` field remain **intent recorded, not implemented**. DIR-12's broader +> vehicle-health framing remains open beyond the FC-link piece (FD-headroom check on +> `ardupilot_manager` not built). --- @@ -231,11 +236,18 @@ what happens next is mode-dependent — recovery is not one behaviour: - ROV: surface the continue-or-recover choice explicitly; do not auto-decide on the operator's behalf. -**Implementation note:** the FSM-level fix landed 11 Jul 2026 (commit `1b0050d`) makes -`RETURN_TO_SAFE` and `HOLD_AND_RECOVER` both able to return to `NORMAL` on condition-clear — -this is the **mechanism**. It is currently **mode-blind** (`failsafe_monitor` does not yet -read the mode profile), so today it behaves the same way in both AUV and ROV. Making that -mechanism mode-aware, per this addendum, is open — see Parked Design Items below. +**Implementation note:** the FSM-level fix landed 11 Jul 2026 (commit `1b0050d`) made +`RETURN_TO_SAFE` and `HOLD_AND_RECOVER` both able to return to `NORMAL` on condition-clear — the +**mechanism**. It was initially **mode-blind** (`failsafe_monitor` did not yet read the mode +profile), so at that point it behaved the same way in both AUV and ROV. + +> ✅ **Made mode-aware 13 Jul 2026** (commit `2a3e577`): `failsafe_monitor` now subscribes to +> `/rov/mode/profile` and branches on `health_role` per this addendum — gate commits +> `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` instead of returning to NORMAL; advisory holds for an +> explicit operator resume (placeholder topic). This surfaced a further gap — gate-mode +> `RETURN_TO_SAFE` had no exit — closed by the new mode-aware safe-zone-reached arrival event, +> commit `bf815cc`. See the new Parked Design Item below for the arrival-event design detail, +> and `docs/handover.md` §15f for the full fix narrative. --- @@ -436,6 +448,46 @@ is about the monitor actively trying to make the condition go away. retiring the substring-parsing shortcut. Same intent as already specified in DIR-7; this is purely a mechanism cleanup, not a new decision. +### Mode-aware safe-zone-reached arrival event (relates to the DIR-7 addendum, "In-mission recovery is mode-dependent") + +Gate-mode (AUV) committed recovery, once entered, **runs to completion** (per the addendum +above) and must be signalled complete by an **arrival event**, not by the triggering condition +merely clearing — condition-clear and arrival are different facts, and conflating them would let +a transient blip that clears immediately end a recovery the vehicle has not actually completed. + +**Permanent interface:** the topic `/rov/nav/safe_zone_reached` (`std_msgs/Bool`), which +`failsafe_monitor` subscribes to and which the **navigation subsystem** will publish to. This +topic and the subscription to it are permanent — `failsafe_monitor` must only ever consume this +event, never compute position itself; that responsibility belongs to navigation. + +**The arrival judgement is mode-aware and sensor-derived, not a single fixed method:** +- **GPS provides fixes at the surface**, including the pattern where an AUV takes a start fix, + dives to dead-reckon a survey on sensor data, and periodically resurfaces for GPS fixes to + bound drift before diving again. +- **Underwater, in both modes**, the position estimate is EKF- and dead-reckoning-based — there + is no GPS underwater. + +**GPS is a first-class part of the navigation system across mission types** — e.g. pipeline +survey, which routinely involves the resurface-for-a-fix pattern above — not a marginal, +surface-only concern specific to hull and jacket inspection. This corrects any earlier framing +that treated it that way. + +**Gate-only by design:** the arrival event applies to gate (AUV) mode only. Advisory (ROV) mode +keeps its existing operator-resume mechanism (see the addendum above) and ignores the arrival +event entirely — there is no reason for an operator-present mode to wait on an autonomous +arrival signal it does not need. + +**Until navigation can raise this event, its publisher is temporary throwaway scaffolding:** a +W1-contextual operator button that appears only while W1 shows the recovering/`RETURN_TO_SAFE` +state, backed by a `rov_api` endpoint. This scaffolding is a **dumb publisher to the topic +only** — it never reaches into `failsafe_monitor` internals — so it can be deleted cleanly in +one cut when navigation takes over publishing to the same permanent topic. + +> **Status:** Permanent interface (topic + `failsafe_monitor` subscription + FSM behaviour) +> implemented 13 Jul 2026 (`bf815cc`) — see `docs/handover.md` §15f. Temporary bench-scaffolding +> publisher (W1 button + `rov_api` endpoint) and the real navigation-raised event are both still +> to come. + --- ## Proposed build staging (REVISED to align with research) diff --git a/docs/handover.md b/docs/handover.md index ffa16ea..d631320 100644 --- a/docs/handover.md +++ b/docs/handover.md @@ -1,7 +1,7 @@ # ROV Autonomous Inspection System — Project Handover **Project Name:** Argonaut 3 -**Version:** 2.24 -**Date:** 11 July 2026 +**Version:** 2.25 +**Date:** 13 July 2026 **File:** `docs/handover.md` (repo-tracked, no version number in filename — see Version History for versioning) **This document is the single source of truth for current project state** — what's built, @@ -16,9 +16,13 @@ it — that change only raised the ceiling (failure at ~10 min → ~11 h of cumu real cause is a `/dev/i2c-1` file-descriptor leak in `ardupilot_manager`'s Linux board detector, driven by the BlueOS Autopilot Firmware / board-config page. See §15a. **This bug is OPEN.** -✅ **Resolved this session** — the vehicle-health/FC-comms-loss fault (was OPEN, -SAFETY-CRITICAL as of v2.22) is fixed: `failsafe_monitor` now drives W1 RED on FC comms loss, -verified live end-to-end. See §15d and Changelog, 11 Jul entries. +✅ **Resolved this session (13 Jul 2026)** — the operating-mode profile system was found to +have been dark at runtime since it was built: `mode_profile_loader` was never wired into the +stack launch file, so `/rov/mode/profile` had zero publishers. Fixed and verified live +(`464e17e`). With the mode system now actually publishing, `failsafe_monitor`'s mode-aware +recovery (`2a3e577`) surfaced a second gap — gate-mode committed recovery entered +`RETURN_TO_SAFE` and could never leave it. Fixed with a new permanent arrival interface, +verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 Jul entries. > Full narrative for each prior version is in Version History below — not repeated here. @@ -38,7 +42,7 @@ verified live end-to-end. See §15d and Changelog, 11 Jul entries. `ardupilot_manager`'s Linux board detector. **Bug is OPEN.** Dev mitigation: do not leave the BlueOS Autopilot Firmware / board-config page open. - `cockpit_bridge` node — ROS2 → WebSocket :9001 → Cockpit data lake. 19 variables live (8 telemetry + 11 mode-profile flags). -- **Operating-mode profile system** — `mode_profile_loader` reads `mode_profiles.yaml` (ROV/AUV defaults + custom hybrid via Save As), publishes the resolved flag set latched on `/rov/mode/profile`, reloadable via `/rov/mode/reload` (std_srvs/Trigger). `cockpit_bridge` surfaces the flags to the data lake as `external/rov-*`. Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified end-to-end: ROV/AUV resolve + live reload. Commit `4e48dc2`. +- **Operating-mode profile system** — `mode_profile_loader` reads `mode_profiles.yaml` (ROV/AUV defaults + custom hybrid via Save As), publishes the resolved flag set latched on `/rov/mode/profile`, reloadable via `/rov/mode/reload` (std_srvs/Trigger). `cockpit_bridge` surfaces the flags to the data lake as `external/rov-*`. Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified end-to-end: ROV/AUV resolve + live reload. Commit `4e48dc2`. **Found never wired into the stack launch file (13 Jul 2026)** — built and committed 7 Jul but never added to `rov_full.launch.py`, so `/rov/mode/profile` had zero publishers at runtime the whole time. Fixed with respawn, commit `464e17e`, verified live (publisher count 0→1). See §15f. - Recording Manager — `recording_manager.py` node active in `argonaut.service`. Publishes `/rov/recording/active` (Bool). Confirmed `data: true` in full stack. Bags to `/data/bags/`. Commit `5d75a9d`. @@ -50,6 +54,19 @@ verified live end-to-end. See §15d and Changelog, 11 Jul entries. end-to-end: kill mavros → RED + "Heartbeat Lost", restart → recover to GREEN. Four commits `86b2728`/`1b0050d`/`dbe9515`/`3409ae5` — full detail in Changelog below. §15d (was OPEN, SAFETY-CRITICAL) is resolved. +- **Mode-aware failsafe recovery (13 Jul 2026)** — `failsafe_monitor` now subscribes to + `/rov/mode/profile` and branches condition-clear recovery on `health_role` (`2a3e577`): gate + (AUV) commits `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` on condition-clear instead of returning to + NORMAL; advisory (ROV) holds and waits for an explicit operator resume (placeholder topic). + This surfaced a new gap once the mode system was actually live — gate-mode `RETURN_TO_SAFE` + had no exit, a terminal trap. Closed by a new PERMANENT interface, + `/rov/nav/safe_zone_reached` (std_msgs/Bool), that transitions the FSM to NORMAL on arrival + (`bf815cc`). Verified live end-to-end on the bench: published the event by hand, watched the + FSM release `RETURN_TO_SAFE`→NORMAL and the `motion_controller` recovery broadcast stop. The + topic/subscription are permanent; today's publisher is a manual bench command only — the + real navigation-raised event and a temporary W1-contextual operator button are both still to + come. See §15f, DIR-7 addendum, and the new Parked Design Item in + `docs/design_intent_register.md`. - W6 Altitude Widget — deployed. NO SONAR state confirmed (expected — no Ping2 connected). Commit `0640353`. - Backup system — push model, both Pis → server. RPi4: 00:00 daily. RPi5: 00:30 daily. @@ -95,15 +112,34 @@ verified live end-to-end. See §15d and Changelog, 11 Jul entries. W0–W7 into a single unified widget. Flagged thought, not yet designed. - W2, W4, W5: untested. Require system testing with live mission / backend pages. - Optional: RPi5 remote was on hostname form after DNS fix — consistent with laptop. Done. +- **`flag_manual_abort` in `failsafe_monitor` is a one-way latch, never reset to `False`** + (found 13 Jul 2026, pre-existing, not caused by that session's work). Once any manual abort + fires, the priority-6 handler returns early on every subsequent evaluation cycle for the + rest of the process lifetime — permanently shadowing all lower-priority handling, including + the new mode-aware recovery and arrival-event logic. Needs a reset condition (candidates: on + reaching NORMAL, on an explicit operator reset, or process-restart only — undecided, needs + its own session). **Field-deployment concern:** one manual abort currently disables all + future automatic recovery for the rest of that dive. See §15f. +- **`failsafe_monitor` produces no log output in `journalctl`** (found 13 Jul 2026). State had + to be read directly off `/rov/failsafe` every time rather than watched in the service log. + Should log state transitions and cause on each one. Minor, but cost real diagnostic time. + See §15f. **PARKED (design decisions deferred, priority order — moved here from CLAUDE.md 11 Jul 2026):** 1. ~~**Doc restructure**~~ — DONE this session (11 Jul 2026): this document, `CLAUDE.md`, and `docs/design_intent_register.md` restructured/updated; all docs moved into repo `docs/`. 2. **Mode-dependent failsafe recovery** — AUV (`health_role: gate`) commits to recovery on condition-clear, does not auto-resume; ROV (`advisory`) holds and hands the - continue-or-recover decision to the operator. Requires `failsafe_monitor` to read the mode - profile (currently mode-blind). **Intent now recorded** as a DIR-7 addendum (11 Jul 2026, - `docs/design_intent_register.md`) — implementation still open. + continue-or-recover decision to the operator. **Essentially complete (13 Jul 2026):** + `failsafe_monitor` now reads the mode profile and branches on `health_role` (`2a3e577`), and + gate-mode `RETURN_TO_SAFE` now has a real exit via the new permanent + `/rov/nav/safe_zone_reached` interface (`bf815cc`), verified live end-to-end on the bench. + **Remaining:** the event's publisher — navigation cannot yet raise it, so the immediate next + step is disposable bench scaffolding, a temporary W1-contextual operator button (shown only + while W1 reads the recovering/`RETURN_TO_SAFE` state, backed by a `rov_api` endpoint), to be + deleted outright once navigation publishes the real GPS/EKF-derived event. Advisory mode's + operator resume is also still a placeholder topic pending the proper `rov_api` endpoint. See + DIR-7 addendum + new Parked Design Item, `docs/design_intent_register.md`. 3. **Auto-recovery** — `failsafe_monitor` should attempt to re-establish the FC link itself, escalating/holding RED only if it cannot. Real-world dives have no operator to restore comms. **Intent now recorded** in DIR's Parked Design Items (11 Jul 2026) — implementation still open. @@ -119,16 +155,21 @@ verified live end-to-end. See §15d and Changelog, 11 Jul entries. a dedicated design session. **NEXT (priority order, carried from §12):** -1. **W3 widget code**: update label to "RETURN TO SAFE", set `FASTAPI_HOST` = +1. **Safe-zone-reached arrival button (bench scaffolding)** — temporary W1-contextual operator + button, shown only while W1 reads the recovering/`RETURN_TO_SAFE` state, backed by a new + `rov_api` endpoint that publishes `True` to `/rov/nav/safe_zone_reached`. Disposable — + deleted outright once navigation can raise the real GPS/EKF-derived arrival event on the + same (permanent) topic. See §15f, DIR-7 addendum, and the new Parked Design Item. +2. **W3 widget code**: update label to "RETURN TO SAFE", set `FASTAPI_HOST` = `http://192.168.1.101:8081`, then test W3 end-to-end (button → /abort → FSM, already verified server-side). -2. **Stage 2b design decisions** (deferred pending Grant): return budget "shortest path to +3. **Stage 2b design decisions** (deferred pending Grant): return budget "shortest path to safety" concept; W7 backup notice (Option A confirmed: server read-only JSON, RPi5 polls outbound). -3. **W2 testing**: test with live mission via `POST /mission/start` (backend now available) +4. **W2 testing**: test with live mission via `POST /mission/start` (backend now available) + loaded plan. -4. **Ping2 sonar purchase** (~$280 USD): required for W6 full end-to-end test. -5. **Pre-field hardening pass**: UFW + RPi5 SSH password (bundled, triggered before remote +5. **Ping2 sonar purchase** (~$280 USD): required for W6 full end-to-end test. +6. **Pre-field hardening pass**: UFW + RPi5 SSH password (bundled, triggered before remote ops/field). **DATA LAKE VARIABLES (19 total, published by cockpit_bridge):** @@ -159,6 +200,9 @@ verified live end-to-end. See §15d and Changelog, 11 Jul entries. | Hash | Description | |---|---| +| `bf815cc` | failsafe_monitor: /rov/nav/safe_zone_reached permanent interface closes gate-mode RETURN_TO_SAFE terminal trap | +| `464e17e` | rov_full.launch.py: wire mode_profile_loader in (was built 4e48dc2, never launched) + respawn | +| `2a3e577` | failsafe_monitor: mode-aware recovery (health_role gate/advisory) per DIR-7 addendum | | `3409ae5` | W1 widget: state word on #w1-message (no flicker), cause text red 11px on #w1-footer | | `dbe9515` | cockpit_bridge: publish external/rov-failsafe-cause (substring stopgap per DIR-7) | | `d854641` | Fix cockpit_bridge publishing failsafe_state (FSM) instead of assessment_state to rov-failsafe — real cause of that day's "no data" symptom | @@ -185,6 +229,7 @@ verified live end-to-end. See §15d and Changelog, 11 Jul entries. | Version | Date | Changes | |---|---|---| +| 2.25 | 13 Jul 2026 | **Two-part fix session, both surfaced by putting the 11 Jul mode-aware work under load.** (1) `mode_profile_loader` was found never to have been wired into `rov_full.launch.py` despite being committed 7 Jul (`4e48dc2`) — `/rov/mode/profile` had zero publishers at runtime since then, so the entire operating-mode profile system had been dark and `failsafe_monitor`'s `health_role` had been silently running on its gate default the whole time. Fixed by adding the node to the launch file with respawn (`464e17e`), verified live (publisher count 0→1). (2) With the mode system now actually publishing, `failsafe_monitor`'s mode-aware recovery (`2a3e577`, same session — subscribes to the resolved profile, branches condition-clear recovery on `health_role`) surfaced a second gap: gate-mode committed recovery entered `RETURN_TO_SAFE` and could never leave it, because that branch deliberately ignores further condition-clear so recovery runs to completion, and nothing signalled completion. Fixed by a new PERMANENT interface, `/rov/nav/safe_zone_reached` (std_msgs/Bool), that transitions the FSM `RETURN_TO_SAFE`→`NORMAL` on arrival when gate-mode (`bf815cc`), verified live end-to-end on the bench (published the event by hand, watched the FSM release and the `motion_controller` recovery broadcast stop). Today's publisher is temporary bench scaffolding only — real navigation-raised arrival and a temporary W1-contextual operator button are both still to come; see the new Parked Design Item in `docs/design_intent_register.md` v1.3. §0 Confirmed Working, PARKED, and NEXT updated; new §15f narrative section. **Two pre-existing `failsafe_monitor` defects found this session (not caused by it):** `flag_manual_abort` is a one-way latch never reset, permanently shadowing all lower-priority handling (including this session's own new logic) after any single manual abort — field-deployment concern, needs its own session; and `failsafe_monitor` produced no log output in `journalctl` this session, costing diagnostic time. Both added to §0 Open Items and §15f. Companion DIR bump: `docs/design_intent_register.md` v1.3 adds the mode-aware safe-zone-reached Parked Design Item — see that changelog for detail. | | 2.24 | 11 Jul 2026 | **§15d resolved.** Vehicle health / FC-comms-loss fault (discovered 9 Jul, flagged OPEN/SAFETY-CRITICAL in v2.22) is fixed as of four commits today: `86b2728` (unconditional RED on comms loss + `fcu_ever_connected` latch + cause text), `1b0050d` (FSM recovery fix — `RETURN_TO_SAFE` was a terminal trap), `dbe9515` (new `external/rov-failsafe-cause` variable), `3409ae5` (W1 widget shows state + cause text). Verified live end-to-end. Related fault found+fixed same session: `cockpit_bridge` was publishing `failsafe_state` (FSM, 3+) not `assessment_state` (0/1/2) to `rov-failsafe` — the actual cause of that day's "widgets show no data," fixed in `d854641`, compounded by a restart-storm wedging port 9001 (→ CLAUDE.md's one-restart-then-verify-9001 rule). §0 Confirmed Working and Recent Commits updated; PARKED list updated to reflect intent now recorded in the DIR (mode-dependent recovery, auto-recovery, dedicated cause field, AMBER runtime-response — all still open on implementation, closed on design intent). Companion DIR bump: `docs/design_intent_register.md` v1.2 adds the two DIR-7 addenda and a Parked Design Items section — see that changelog for detail, not repeated here. | | 2.23 | 11 Jul 2026 | **Doc restructure.** This document is now the single source of truth for current state (Quick Status Board gains PARKED and NEXT subsections, folding in the list previously buried in CLAUDE.md's SESSION LOG and §12's Next Priorities). New **Changelog** section (below) replaces CLAUDE.md's SESSION LOG — same entries, same append-only format, moved here. The embedded Design Intent Register (formerly here, ~200 lines) is **removed** — full DIR now lives only in `docs/design_intent_register.md`; this document keeps a short pointer + summary of which DIRs govern current work. All docs moved into the repo under `docs/` (git-tracked, no version number in filenames — see CLAUDE.md Documentation Discipline). Fixed: duplicate `## 15d` heading (second instance renamed `15e`); §15's "Recording control" locked-decision row corrected from "DIR-9 — always-on" to DIR-9a (ARM-bounded); duplicate 8-variable data-lake table in §9 replaced with a pointer to §0's canonical 19-variable table; stale "Widget JSON not in Gitea" row in §16 marked resolved (W6 committed per `785ec56`, already reflected in §12). No new facts asserted beyond what was already recorded elsewhere in this document, CLAUDE.md, or the DIR — this pass is consolidation, not new findings. | | 2.22 | 9 Jul 2026 | **⚠ ROOT-CAUSE CORRECTION (supersedes v2.10 and v2.19 claims).** The BlueOS "FD exhaustion" bug was NEVER fixed. `nofile=65536` was a ceiling raise, not a fix — it moved failure from ~10 min to ~11 h of cumulative exposure. Real cause, measured: `ardupilot_manager`'s Linux board detector opens `/dev/i2c-1` on each failed probe and never closes it. The BlueOS **Autopilot Firmware / board-config page** polls `GET /v1.0/available_boards`; each request fires 20 `detect_boards` probes (Navigator/NavigatorPi4/NavigatorPi5/Argonot). We run a Pixhawk on USB serial, so every probe fails and leaks. Measured ~100–108 FD/min with the page open; **exactly zero with it closed** (4630 flat over 65 min; +90 FD in 50 s on opening; flat immediately on closing). FD composition: 2740/2745 were `/dev/i2c-1`, 3 sockets. Probe:request ratio 20:1 (8920:445, from logs). Count accumulates across sessions; resets only when `blueos-core` is recreated. Upstream BlueOS 1.4.3 defect — **do NOT patch BlueOS** (updates revert). Dev mitigation: don't leave the board-config page open. Field mitigation NOT built. §15a rewritten. **New §15d — OPEN, SAFETY-CRITICAL:** `failsafe_monitor` reported GREEN while the FC was unreachable; W1 is the AUV pre-dive GO/NO-GO gate (`health_role: gate`) and would pass a dive with a dead flight controller. All widget indicators reflect the RPi5 stack, not the vehicle. **Also this session:** `rov_api` `/mode` endpoints (GET/POST, validate-before-write + `/rov/mode/reload` trigger) built, deployed, tested incl. rejection path (`9109439`). **W8 mode indicator** built and validated live — render + data (`87000e6`). **Chronic "widgets show no data" root-caused:** `cockpit_bridge` was in the launch file but had no respawn; a boot-time race left it permanently dead, taking the whole data-lake feed with it — fixed with `respawn=True` (`c099c00`), verified across a cold stack restart. Respawn extended to `failsafe_monitor`, `motion_controller`, `ekf_filter_node`, `depth_node`, `mission_executor` (`a5e1c99`); deliberately NOT applied to `mavros_node` (would mask FCU-link instability), `recording_manager` (pending DIR-9a rework), perception (not operational), `foxglove_bridge` (viz-only). **FLAGGED OPEN:** `recording_manager.py` still implements superseded DIR-9 (continuous-from-boot) rather than governing DIR-9a (ARM-bounded); and `argonaut-stack.sh` passes `record:=${RECORD}` to a launch file that declares no such arg, so `RECORD=false` is silently ignored. Claude Code (VS Code) adopted for repo edits; SESSION LOG added to `CLAUDE.md` to bridge Claude Code ↔ claude.ai. | @@ -287,6 +332,41 @@ context needed. Full trigger-rule text lives in `CLAUDE.md`. moved this log here from CLAUDE.md, removed the embedded DIR copy in favour of a pointer to `docs/design_intent_register.md`, moved all project docs into the repo `docs/` folder. See Version History v2.23 above for detail. +- 2026-07-13 — Found mode_profile_loader (rov_mission) was built and committed 7 Jul in + 4e48dc2 but never added to rov_full.launch.py, so /rov/mode/profile had zero publishers at + runtime since then — the whole operating-mode profile system had been dark, and + failsafe_monitor's health_role had been silently running on its gate default the whole + time. Fixed by adding the Node to the launch file (with the other core nodes, before + Perception) with respawn=True/respawn_delay=5.0, matching the other core nodes (c099c00, + a5e1c99). Verified live: publisher count 0 -> 1 (464e17e). +- 2026-07-13 — failsafe_monitor made mode-aware (2a3e577, same session): subscribes to the + resolved /rov/mode/profile and branches condition-clear recovery on health_role per the + DIR-7 addendum — gate (AUV) commits HOLD_AND_RECOVER -> RETURN_TO_SAFE instead of returning + to NORMAL; advisory (ROV) holds for an explicit operator resume (placeholder topic). This + surfaced a new gap once the mode system was actually live (previous entry): gate-mode + RETURN_TO_SAFE had no exit, a terminal trap, because that branch deliberately ignores + further condition-clear so committed recovery runs to completion, and nothing signalled + completion. Fixed (bf815cc) with a new subscription, /rov/nav/safe_zone_reached + (std_msgs/Bool) — THIS TOPIC AND SUBSCRIPTION ARE THE PERMANENT nav-arrival interface; + today's publisher is temporary bench scaffolding only. On gate + RETURN_TO_SAFE + True, + transitions to NORMAL with cause text "Safe zone reached — recovery complete" (DIR-7 cause + MUST, via the existing message-prefix mechanism, no FailsafeStatus.msg change). Verified + live end-to-end on the bench: published the event by hand, watched the FSM release + RETURN_TO_SAFE -> NORMAL and the motion_controller recovery broadcast stop. Real + navigation-raised arrival event and a temporary W1-contextual operator button (rov_api- + backed bench scaffolding, next step) are both still to come — see new Parked Design Item, + docs/design_intent_register.md. +- 2026-07-13 — FLAGGED (open, pre-existing, found not caused by this session's work): (1) + flag_manual_abort in failsafe_monitor is a one-way latch, never reset to False anywhere — + once any manual abort fires, the priority-6 handler returns early every subsequent cycle + for the rest of the process lifetime, permanently shadowing all lower-priority handling + including this session's new mode-aware recovery and arrival-event logic. Needs a reset + condition (candidates: on reaching NORMAL, on explicit operator reset, or process-restart + only — undecided, needs its own session). Field-deployment concern: one manual abort + disables all future automatic recovery for the rest of that dive. (2) failsafe_monitor + produced no log output at all in journalctl this session — state had to be read directly + off /rov/failsafe every time. Should log state transitions and cause. Minor but cost real + diagnostic time this session. --- @@ -1002,7 +1082,9 @@ restart-storm that wedged port 9001 while diagnosing it (see CLAUDE.md's one-restart-then-verify-9001 rule, added as a direct result). ### Still open (not resolved by this fix — see DIR-7 addenda and Parked Design Items) -- Mode-dependent in-mission recovery (AUV commit-to-recovery vs ROV operator handoff) +- ~~Mode-dependent in-mission recovery (AUV commit-to-recovery vs ROV operator handoff)~~ — + **ESSENTIALLY COMPLETE 13 Jul 2026, see §15f.** Remaining: the arrival-event publisher + (temporary operator button, then real navigation). - Auto-recovery (failsafe_monitor re-establishing the FC link itself) - In-mission AMBER runtime-response, branched by mode - Dedicated `string cause` field on `FailsafeStatus.msg` (current fix is a substring stopgap) @@ -1011,6 +1093,72 @@ one-restart-then-verify-9001 rule, added as a direct result). --- +## 15f. Mode system launch-wiring gap + gate-mode recovery arrival event — 13 Jul 2026 + +### Fault 1 — mode_profile_loader never wired into the stack launch file + +`mode_profile_loader` (`rov_mission`) was built and committed 7 Jul 2026 (`4e48dc2`) — node, +`ModeProfile.msg`, `mode_profiles.yaml`, `cockpit_bridge` flag surfacing, all verified working +in isolation at the time. It was never added to `rov_full.launch.py`. Confirmed on RPi5, 12 Jul +2026: `ros2 node list` showed no mode node; `ros2 topic info /rov/mode/profile` reported +`Publisher count: 0`, 2 subscribers waiting. The entire operating-mode profile system had been +dark at runtime — zero publishers — for the six days since it was built. `failsafe_monitor`'s +`health_role` had been silently running on its gate default the whole time, not the actually +configured mode. + +**Fix (`464e17e`):** added the `Node` action for `mode_profile_loader` to `rov_full.launch.py`, +placed with the other core nodes (before Perception) rather than down in the mission-package +include further below, so consumers of its latched output (`failsafe_monitor`, `cockpit_bridge`) +wait as little as possible on a publisher. `respawn=True`/`respawn_delay=5.0`, matching the other +core nodes (`c099c00`, `a5e1c99`) — if this node dies the whole mode system goes dark again. +Verified live: publisher count 0 → 1. + +### Fault 2 — gate-mode RETURN_TO_SAFE was a terminal trap + +With the mode system now actually publishing a real gate profile, a second gap surfaced. +`failsafe_monitor`'s mode-aware recovery (`2a3e577`, same session) correctly commits gate-mode +(AUV) `HOLD_AND_RECOVER` → `RETURN_TO_SAFE` on condition-clear per the DIR-7 addendum — but then +never left `RETURN_TO_SAFE`. That branch deliberately ignores further condition-clear (recovery, +once committed, must run to completion — see the addendum) but nothing had ever been wired to +signal that the recovery HAD completed. + +**Fix (`bf815cc`):** new subscription, `/rov/nav/safe_zone_reached` (std_msgs/Bool), same plain +QoS and latch-on-True callback pattern as the existing `flag_manual_abort` subscription. When +`health_role == gate` and `fsm_state == RETURN_TO_SAFE` and a `True` is received, the FSM +transitions to `NORMAL` and the cause text reads "Safe zone reached — recovery complete" (DIR-7's +cause-text MUST, via the existing message-prefix mechanism — no `FailsafeStatus.msg` change, no +`cockpit_bridge`/W1 change needed). Any other combination (not in `RETURN_TO_SAFE`, or advisory +mode) is ignored — advisory mode keeps its separate, untouched operator-resume mechanism. + +**This topic and this subscription are the PERMANENT nav-arrival interface** — the navigation +subsystem will publish the real event here (GPS at surface, EKF dead-reckoning underwater; see +the new Parked Design Item in `docs/design_intent_register.md` for the full sensor-derivation +detail). Only today's PUBLISHER is temporary: verified live end-to-end on the bench by publishing +the event by hand and watching the FSM release `RETURN_TO_SAFE` → `NORMAL` and the +`motion_controller` recovery broadcast stop. The permanent replacement publisher — a temporary +W1-contextual operator button (bench scaffolding, `rov_api`-backed) ahead of navigation raising +the event for real — is next (see §0 NEXT). + +### Pre-existing defects found this session (not caused by this session's work) + +**`flag_manual_abort` is a one-way latch, never reset.** Set `True` in `_abort_callback` on any +`/rov/mission/abort` message; nothing anywhere in `failsafe_monitor` ever sets it back to +`False`. Once any manual abort fires, Priority 6 (`if self.flag_manual_abort: ...; return`) +returns early on every subsequent evaluation cycle for the rest of the process lifetime — +permanently shadowing every lower-priority handler, including the new mode-aware recovery and +arrival-event logic added this session (neither is reachable once this latch is set). Discovered +during this session's work on `2a3e577`; the bug itself predates it. Needs a reset condition — +candidates: on reaching `NORMAL`, on an explicit operator reset, or only on process restart — +undecided, needs its own session. **Field-deployment concern:** a single manual abort currently +disables all future automatic recovery for the rest of that dive. + +**`failsafe_monitor` produces no log output at all.** Silent in `journalctl` this session — no +info/warn lines observed for state transitions or startup. Its state had to be read directly off +`/rov/failsafe` (`ros2 topic echo`) every time rather than watched in the service log. Minor, but +cost real diagnostic time this session. Should log state transitions and cause on each one. + +--- + ### MAVLinkRouter vs MAVP2P BlueOS itself warns "MAVP2P has been presenting issues and is not currently recommended." MAVLinkRouter is the confirmed stable router. Do not recommend or use MAVP2P.