# ROV Autonomous Inspection System — Project Handover
**Project Name:** Argonaut 3
**Version:** 2.26
**Date:** 13 July 2026
**File:** `docs/handover.md` (repo-tracked, no version number in filename — see Version History for versioning)

**This document is the single source of truth for current project state** — what's built,
verified, broken, parked, and next. Rules for how Claude works are in `CLAUDE.md` (repo root).
Design intent ("why") is in `docs/design_intent_register.md`. Domain specs (UI, failsafe, sim/ML,
recording lifecycle) are in `docs/`, referenced from §0 and inline, not duplicated here.

**Status:**
⚠ **CORRECTION (still open) — the BlueOS "FD exhaustion" root cause recorded in v2.10–v2.21 was
WRONG.** It is NOT TCP-socket/uvicorn exhaustion from UI polling, and `nofile=65536` did NOT fix
it — that change only raised the ceiling (failure at ~10 min → ~11 h of cumulative exposure). The
real cause is a `/dev/i2c-1` file-descriptor leak in `ardupilot_manager`'s Linux board detector,
driven by the BlueOS Autopilot Firmware / board-config page. See §15a. **This bug is OPEN.**

✅ **Resolved this session (13 Jul 2026)** — the operating-mode profile system was found to
have been dark at runtime since it was built: `mode_profile_loader` was never wired into the
stack launch file, so `/rov/mode/profile` had zero publishers. Fixed and verified live
(`464e17e`). With the mode system now actually publishing, `failsafe_monitor`'s mode-aware
recovery (`2a3e577`) surfaced a second gap — gate-mode committed recovery entered
`RETURN_TO_SAFE` and could never leave it. Fixed with a new permanent arrival interface,
verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 Jul entries.

⚠ **CORRECTED later the same session (13 Jul 2026)** — the condition-clear-commits-to-recovery
entry trigger described immediately above (`2a3e577`) is **KNOWN-WRONG**, not merely
mode-aware-and-working. It produced a contradictory live state: W1 showing GREEN ("Systems
nominal") while the vehicle was simultaneously in `RETURN_TO_SAFE` ("Recovering to safe zone").
Corrected intent recorded in `docs/design_intent_register.md` v1.4 (`255096f`): committed
recovery must be triggered by fault PERSISTENCE or RECURRENCE, never by a condition clearing —
GREEN is unconditionally a working state. The safe-zone-reached arrival EXIT mechanism
(`bf815cc`) is unaffected and still correct; only the ENTRY trigger needs rework. This is the
**top-priority next-session task** — see §0 NEXT and §15g.

> Full narrative for each prior version is in Version History below — not repeated here.

---

## 0. QUICK STATUS BOARD (read this first)

**CONFIRMED WORKING (verified across sessions to date):**
- Full autonomy stack — 19 nodes, EKF ~11 Hz on `/odometry/filtered`, `/tf` valid.
  Runs as systemd `argonaut.service` (manual-start in dev — verify BlueOS heartbeat first).
- Pixhawk 2.4.8 — confirmed active on `/dev/ttyACM0`. `connected: true`, `system_status: 3`
  (STANDBY), `mode: MANUAL`. SITL is NOT used in production. BlueOS `preferred_board: Pixhawk1`.
- MAVLink routing — `mavlink-routerd` running on `ttyACM0:115200`. **MAVLinkRouter is the
  confirmed router. MAVP2P has known issues (BlueOS own warning) — do not use, do not suggest.**
- ~~BlueOS FD exhaustion — permanently fixed~~ **✗ RETRACTED (9 Jul 2026). NOT FIXED. See §15a.**
  The `nofile: 65536` change raised the ceiling only. Real cause is a `/dev/i2c-1` FD leak in
  `ardupilot_manager`'s Linux board detector. **Bug is OPEN.** Dev mitigation: do not leave the
  BlueOS Autopilot Firmware / board-config page open.
- `cockpit_bridge` node — ROS2 → WebSocket :9001 → Cockpit data lake. 19 variables live (8 telemetry + 11 mode-profile flags).
- **Operating-mode profile system** — `mode_profile_loader` reads `mode_profiles.yaml` (ROV/AUV defaults + custom hybrid via Save As), publishes the resolved flag set latched on `/rov/mode/profile`, reloadable via `/rov/mode/reload` (std_srvs/Trigger). `cockpit_bridge` surfaces the flags to the data lake as `external/rov-*`. Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified end-to-end: ROV/AUV resolve + live reload. Commit `4e48dc2`. **Found never wired into the stack launch file (13 Jul 2026)** — built and committed 7 Jul but never added to `rov_full.launch.py`, so `/rov/mode/profile` had zero publishers at runtime the whole time. Fixed with respawn, commit `464e17e`, verified live (publisher count 0→1). See §15f.
- Recording Manager — `recording_manager.py` node active in `argonaut.service`. Publishes
  `/rov/recording/active` (Bool). Confirmed `data: true` in full stack. Bags to `/data/bags/`.
  Commit `5d75a9d`.
- `/data/bags/` — created on NVMe. Recording live at ~2.5 MB/s (nav/MAVLink topics, no camera).
  512GB NVMe will last weeks at this rate.
- W0 Data Lake Inspector — confirmed working. 706/706 vars live.
- **W1 System Health Indicator — failsafe_monitor now drives RED on FC comms loss (11 Jul
  2026).** Previously visual-confirmed only (GREEN, full test pending); now verified live
  end-to-end: kill mavros → RED + "Heartbeat Lost", restart → recover to GREEN. Four commits
  `86b2728`/`1b0050d`/`dbe9515`/`3409ae5` — full detail in Changelog below. §15d (was OPEN,
  SAFETY-CRITICAL) is resolved.
- **Mode-aware failsafe recovery — PARTIALLY correct, entry trigger KNOWN-WRONG (13 Jul 2026,
  corrected later the same day).** `failsafe_monitor` correctly subscribes to
  `/rov/mode/profile` and reads `health_role` (`2a3e577`) — this mechanism is confirmed working
  and correct, unaffected by the correction below. The safe-zone-reached arrival EXIT
  mechanism — a new PERMANENT interface, `/rov/nav/safe_zone_reached` (std_msgs/Bool), that
  transitions gate-mode `RETURN_TO_SAFE` to NORMAL on arrival (`bf815cc`) — is also confirmed
  working and correct, verified live end-to-end on the bench (published the event by hand,
  watched the FSM release `RETURN_TO_SAFE`→NORMAL and the `motion_controller` recovery
  broadcast stop). **However, the ENTRY trigger built the same day (`2a3e577`: gate commits
  `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` ON CONDITION-CLEAR) is KNOWN-WRONG** — corrected in
  `docs/design_intent_register.md` v1.4 (`255096f`) after it produced a contradictory live
  state (W1 GREEN "Systems nominal" while the vehicle sat in `RETURN_TO_SAFE` "Recovering to
  safe zone"). Corrected intent: committed recovery must be triggered by fault PERSISTENCE or
  RECURRENCE, never by condition-clear — GREEN is unconditionally a working state. Rework to
  the count-based model is the **top-priority next-session task** — see §0 NEXT, §15g, and the
  DIR-7 addendum in `docs/design_intent_register.md`.
- W6 Altitude Widget — deployed. NO SONAR state confirmed (expected — no Ping2 connected).
  Commit `0640353`.
- Backup system — push model, both Pis → server. RPi4: 00:00 daily. RPi5: 00:30 daily.
  Server monitor alerts on FAILED (always) and OVERDUE ≥2 days.
- **FastAPI backend (`rov_api`)** — deployed and verified end-to-end on RPi5:8081. Runs as
  `argonaut-api.service` (manual-start in dev). Endpoints confirmed: `GET /health` (live
  mission/recording/failsafe), `POST /abort` (W3 → `/rov/mission/abort` → failsafe_monitor
  FSM NORMAL→HOLD_AND_RECOVER), `POST /mission/start` (MissionCommand START), `POST /mission/stop`
  (MissionCommand ABORT). Commit `66a18ef`.
- **Deploy files version-controlled** — repo `deploy/` folder holds both services' units +
  wrappers + README. Commit `97decac`.
- **Hostname resolution (DNS)** — `symbytech`/`blueos`/`rov-brain` resolve on laptop (Windows
  hosts) and RPi5 (`/etc/hosts`). Git remotes on hostname form.
- Gitea remote — hostname form: `ssh://git@symbytech:222/administrator/rov-autonomy.git`
- All three (Gitea/laptop/RPi5) level at `4e48dc2`.

**UNTESTED (visual/UI confirmed, real data not flowing yet):**
- W2 Mission Status — UI confirmed (IDLE state + progress bar). Untested: needs a mission
  running. Test: `POST /mission/start` via FastAPI (now available) with a loaded plan, observe widget.
- W4 Mission Setup Button — UI confirmed (cyan button, opens setup URL). Untested: needs a
  setup page served by FastAPI (not yet built — Stage 2b/Phase 6).
- W5 Return Budget — UI confirmed (shows pending). Untested: bridge does not yet publish
  `external/rov-return-budget`. Backend deferred to Stage 2b (return budget approach under
  discussion — "shortest path to safety" concept to be defined).

**BLOCKED / NOT BUILT:**
- W3 RETURN TO SAFE button — backend now LIVE (`/abort` verified end-to-end). Remaining:
  update widget code label from "ABORT MISSION" to "RETURN TO SAFE" and set `FASTAPI_HOST`
  to `http://192.168.1.101:8081`.
- W4 setup page — the page FastAPI would serve is not built (Stage 2b/Phase 6).
- W5 return budget — `/return_budget` endpoint deferred to Stage 2b.
- W7 Backup Due Notice — not built. Depends on FastAPI `/backup/*` (Stage 2b, Option A design:
  server exposes read-only JSON status, RPi5 polls outbound).
- `external/rov-return-budget` — not published by cockpit_bridge. W5 budget figure pending.

**OPEN ITEMS / HOUSEKEEPING:**
- W3 widget code: update button label to "RETURN TO SAFE"; set `FASTAPI_HOST` constant.
- W6 full test: hardware-gated on Ping2 sonar purchase (Phase 3, ~$280 USD).
- RPi5 SSH password: weak dev password. Harden before field deployment (bundled with UFW).
- UFW: inactive on RPi5. Deferred to pre-field/pre-remote hardening pass (no online exposure
  currently — Tailscale disabled, no proxy to Pis). Bundle with SSH password change.
- Widget consolidation: Grant to review as development progresses — may consolidate
  W0–W7 into a single unified widget. Flagged thought, not yet designed.
- W2, W4, W5: untested. Require system testing with live mission / backend pages.
- Optional: RPi5 remote was on hostname form after DNS fix — consistent with laptop. Done.
- **`flag_manual_abort` in `failsafe_monitor` is a one-way latch, never reset to `False`**
  (found 13 Jul 2026, pre-existing, not caused by that session's work). Once any manual abort
  fires, the priority-6 handler returns early on every subsequent evaluation cycle for the
  rest of the process lifetime — permanently shadowing all lower-priority handling, including
  the new mode-aware recovery and arrival-event logic. Needs a reset condition (candidates: on
  reaching NORMAL, on an explicit operator reset, or process-restart only — undecided, needs
  its own session). **Field-deployment concern:** one manual abort currently disables all
  future automatic recovery for the rest of that dive. See §15f.
- **`failsafe_monitor` produces no log output in `journalctl`** (found 13 Jul 2026). State had
  to be read directly off `/rov/failsafe` every time rather than watched in the service log.
  Should log state transitions and cause on each one. Minor, but cost real diagnostic time.
  See §15f.
- **W1 safe-zone-reached button — PAUSED, not abandoned (13 Jul 2026).** Supporting
  infrastructure is built and committed: `rov_api`'s `POST /nav/safe_zone_reached` endpoint and
  the `external/rov-failsafe-state` data lake variable it depends on (`2def2b1`); a pre-edit
  backup of the W1 widget was made per CREATE-THEN-BACKUP
  (`widgets/w1_system_health_indicator.json.2026-07-13.bak`). The widget edit itself was not
  started — the button would surface FSM state reached via a trigger now known to be wrong (see
  the mode-aware recovery correction above), so it waits on that rework. See §15g.
- **Stray nested colcon workspace caused recurring `rov_mission` build failures — root-caused
  and fixed (13 Jul 2026).** `build/`/`install`/`log` directories had been created inside
  `src/rov-autonomy` on RPi5 (from `colcon build` being run from inside the repo checkout
  instead of the workspace root), colliding with the real workspace at `/data/ros2_ws` and
  causing builds to resolve `mode_profiles.yaml` through a stale nested `install/` tree. This
  had recurred 2-3 times. Fixed by removing the stray directories (gitignored/untracked — local
  filesystem cleanup only, no git change). **Durable guard: `colcon build` must only ever be
  run from `/data/ros2_ws`, never from inside `src/rov-autonomy`.** See §15g.

**PARKED (design decisions deferred, priority order — moved here from CLAUDE.md 11 Jul 2026):**
1. ~~**Doc restructure**~~ — DONE this session (11 Jul 2026): this document, `CLAUDE.md`,
   and `docs/design_intent_register.md` restructured/updated; all docs moved into repo `docs/`.
2. **Mode-dependent failsafe recovery — CORRECTED intent (13 Jul 2026), rework pending.**
   `failsafe_monitor` correctly reads the mode profile and branches on `health_role`
   (`2a3e577`) — that mechanism stands. But `2a3e577`'s ENTRY trigger (gate commits
   `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` **on condition-clear**) is **KNOWN-WRONG**: it produced a
   contradictory live state (W1 GREEN while the FSM sat in `RETURN_TO_SAFE`). **Corrected intent**
   (`docs/design_intent_register.md` v1.4, `255096f`): GREEN is unconditionally a working state;
   committed recovery is triggered by fault **persistence** (config-driven raw failure count,
   transient tier) or **recurrence/hard-persistence** (RED that doesn't clear, immediate commit)
   — never by condition-clear. All thresholds are config-wizard-driven via the resolved mode
   profile, like `health_role`. The safe-zone-reached arrival EXIT mechanism
   (`/rov/nav/safe_zone_reached`, `bf815cc`) is unaffected and still correct — it still
   completes a committed recovery; only entry into one was wrong. **This rework is the
   top-priority NEXT item** (see §0 NEXT). The W1-contextual operator button (supporting
   `rov_api` endpoint + data lake variable already built, `2def2b1`) is PAUSED pending this
   rework — see Open Items above and §15g. Advisory mode's operator resume is also still a
   placeholder topic pending the proper `rov_api` endpoint.
3. **Auto-recovery** — `failsafe_monitor` should attempt to re-establish the FC link itself,
   escalating/holding RED only if it cannot. Real-world dives have no operator to restore comms.
   **Intent now recorded** in DIR's Parked Design Items (11 Jul 2026) — implementation still open.
4. **Dedicated `string cause` field** in `FailsafeStatus.msg`, to replace the message-substring
   shortcut currently used in `cockpit_bridge`. **Intent now recorded** in DIR's Parked Design
   Items (11 Jul 2026) — implementation still open.
5. **Operator control surface** (start/pause/stop via `rov_api`) and one-command stack
   startup + healthcheck.
6. **UFW / SSH pre-field hardening** — see Open Items above.
7. **In-mission AMBER runtime-response** — AMBER means action-required once a mission is
   underway, not just informational; response branches by mode. **Intent now recorded** as a
   DIR-7 addendum (11 Jul 2026) — AUV's autonomous curtailment logic is unspecified, left for
   a dedicated design session.

**NEXT (priority order, carried from §12):**
1. **FSM entry-trigger rework — count-based fault-persistence model (TOP PRIORITY, 13 Jul
   2026).** `failsafe_monitor`'s condition-clear-commits-to-recovery entry logic (`2a3e577`) is
   KNOWN-WRONG (see `docs/design_intent_register.md` v1.4, DIR-7 addendum, `255096f`). Rework
   to: GREEN unconditionally a working state; committed recovery triggered by fault persistence
   (transient tier — config-driven raw failure count, no time window) or hard-persistence (RED
   that doesn't clear — immediate commit); all thresholds config-wizard-driven via the resolved
   mode profile, like `health_role`. The safe-zone-reached arrival EXIT mechanism (`bf815cc`)
   needs no rework — only the entry trigger.
2. **Safe-zone-reached arrival button (bench scaffolding) — PAUSED, waiting on item 1.**
   `rov_api`'s `POST /nav/safe_zone_reached` endpoint and the `external/rov-failsafe-state` data
   lake variable it depends on are built and committed (`2def2b1`); a pre-edit W1 widget backup
   was made per CREATE-THEN-BACKUP. The widget edit itself is not started — the button would
   surface FSM state reached via the known-wrong trigger above, so it is meaningless until item
   1 lands. Disposable once navigation can raise the real GPS/EKF-derived arrival event on the
   same permanent topic. See §15g, DIR-7 addendum, and the Parked Design Items.
3. **W3 widget code**: update label to "RETURN TO SAFE", set `FASTAPI_HOST` =
   `http://192.168.1.101:8081`, then test W3 end-to-end (button → /abort → FSM, already
   verified server-side).
4. **Stage 2b design decisions** (deferred pending Grant): return budget "shortest path to
   safety" concept; W7 backup notice (Option A confirmed: server read-only JSON, RPi5 polls
   outbound).
5. **W2 testing**: test with live mission via `POST /mission/start` (backend now available)
   + loaded plan.
6. **Ping2 sonar purchase** (~$280 USD): required for W6 full end-to-end test.
7. **Pre-field hardening pass**: UFW + RPi5 SSH password (bundled, triggered before remote
   ops/field).

**DATA LAKE VARIABLES (19 total, published by cockpit_bridge):**

| Variable | Source topic | Notes |
|---|---|---|
| `external/rov-failsafe` | `/rov/failsafe` | 0=GREEN, 1=AMBER, 2=RED; -1=no data |
| `external/rov-depth` | `/rov/depth` | Depth in metres (3 dp) |
| `external/rov-voltage` | `/mavros/battery` | Battery volts (2 dp) |
| `external/rov-heading` | `/mavros/mavros/compass_hdg` | Compass degrees (1 dp) |
| `external/rov-ms` | `/rov/mission/status` | Mission state: 0=IDLE…4=ABORTED; -1=no data |
| `external/rov-mp` | `/rov/mission/status` | Mission progress 0-100% |
| `external/rov-recording` | `/rov/recording/active` | 1=recording, 0=not recording |
| `external/rov-altitude` | `/ping2/altitude` | Ping2 altitude metres; -1.0=NO SONAR |
| `external/rov-mode` | `/rov/mode/profile` | Active profile name (string): ROV / AUV / hybrid |
| `external/rov-mode-base` | `/rov/mode/profile` | Base mode: 0=ROV, 1=AUV |
| `external/rov-hilt` | `/rov/mode/profile` | HILT link: 1=present, 0=absent |
| `external/rov-health-role` | `/rov/mode/profile` | W1 role: 0=advisory, 1=gate |
| `external/rov-rec-control` | `/rov/mode/profile` | Recording: 0=manual, 1=auto (ARM-bounded) |
| `external/rov-nogo-gate` | `/rov/mode/profile` | Recording no-go: 0=relaxed, 1=enforced |
| `external/rov-rec-reminder` | `/rov/mode/profile` | ARM reminder (manual): 1=on, 0=off |
| `external/rov-return-mode` | `/rov/mode/profile` | Return: 0=operator-selectable, 1=autonomous |
| `external/rov-live-telem` | `/rov/mode/profile` | Live telemetry: 0=on, 1=predive-only |
| `external/rov-nav-display` | `/rov/mode/profile` | Nav display: 1=shown to pilot, 0=autonomy only |
| `external/rov-camera-role` | `/rov/mode/profile` | Camera: 0=pilot-primary, 1=record-only |

**RECENT COMMITS (latest first):**

| Hash | Description |
|---|---|
| `255096f` | docs(dir): correct inverted DIR-7 in-mission-recovery intent — count-based fault-persistence model, GREEN unconditional working state |
| `2def2b1` | rov_api: POST /nav/safe_zone_reached endpoint; cockpit_bridge: external/rov-failsafe-state variable |
| `bf815cc` | failsafe_monitor: /rov/nav/safe_zone_reached permanent interface closes gate-mode RETURN_TO_SAFE terminal trap |
| `464e17e` | rov_full.launch.py: wire mode_profile_loader in (was built 4e48dc2, never launched) + respawn |
| `2a3e577` | failsafe_monitor: mode-aware recovery (health_role gate/advisory) per DIR-7 addendum |
| `3409ae5` | W1 widget: state word on #w1-message (no flicker), cause text red 11px on #w1-footer |
| `dbe9515` | cockpit_bridge: publish external/rov-failsafe-cause (substring stopgap per DIR-7) |
| `d854641` | Fix cockpit_bridge publishing failsafe_state (FSM) instead of assessment_state to rov-failsafe — real cause of that day's "no data" symptom |
| `1b0050d` | failsafe_monitor FSM fix: RETURN_TO_SAFE was a terminal trap, both states now recover to NORMAL on condition-clear |
| `86b2728` | failsafe_monitor: comms loss forces unconditional RED + fcu_ever_connected latch + cause text on FailsafeStatus.message |
| `4e48dc2` | feat(mode): operating-mode profile system — ModeProfile.msg, mode_profile_loader, mode_profiles.yaml, cockpit_bridge flags |
| `785ec56` | Widgets housekeeping: add W6 (`w6_altitude.json`), remove probe widget |
| `97decac` | Add deploy/ folder — systemd units and launch wrappers + README |
| `66a18ef` | Phase 3 Stage 2: rov_api FastAPI backend (core) — /health, /abort, /mission/start, /mission/stop |
| `0640353` | W6: Add rov-altitude to cockpit_bridge for Ping2 downward sonar |
| `5d75a9d` | Phase 3 Stage 1: Recording Manager (DIR-9) |
| `2cb7b74` | W0 layout fix (3-column, horizontal scroll) |
| `66f9bd7` | W0–W5 all confirmed working, committed |
| `c94412d` | Full stack zero-crash launch (MAVROS fcu_url, plugin_allowlist, QoS, battery guard) |
| `12b5161` | cockpit_bridge node — ROS2→WebSocket :9001 |

> Table has historically lagged actual commit history (e.g. `9109439`, `87000e6`, `c099c00`,
> `a5e1c99` from the 8–9 Jul session were never added) — not reconstructed retroactively here
> without measuring actual git log; flagged so it doesn't look more complete than it is.

---

## Version History

| Version | Date | Changes |
|---|---|---|
| 2.26 | 13 Jul 2026 | **Same-day correction, extending v2.25's record.** The mode-aware recovery built earlier today (`2a3e577`) was found to be internally contradictory: its gate-mode entry trigger commits to `RETURN_TO_SAFE` ON CONDITION-CLEAR, which produced a live state that should not be possible — W1 showing GREEN ("Systems nominal") while the vehicle sat in `RETURN_TO_SAFE` ("Recovering to safe zone"). Corrected in `docs/design_intent_register.md` v1.4 (`255096f`): GREEN is unconditionally a working state; committed recovery is triggered by fault PERSISTENCE (transient tier — config-driven raw failure count, deliberately no time window) or immediate hard-persistence (RED that doesn't clear), never by condition-clear; all thresholds config-wizard-driven via the resolved mode profile like `health_role`. Survives unaffected: `health_role` mode-awareness (`2a3e577`'s subscription mechanism) and the safe-zone-reached arrival EXIT mechanism (`bf815cc`) — only ENTRY into committed recovery was wrong. `2a3e577`'s entry logic is now flagged KNOWN-WRONG pending rework, which is the new top-priority NEXT item, ahead of everything previously listed. Also recorded: (1) the W1 safe-zone-reached button is PAUSED, not abandoned — its `rov_api` endpoint (`POST /nav/safe_zone_reached`) and the `external/rov-failsafe-state` data lake variable are built and committed (`2def2b1`), a pre-edit W1 widget backup was made per CREATE-THEN-BACKUP, but the widget edit itself was not started because the FSM state it would surface is reached via a known-wrong trigger; (2) a recurring `rov_mission` build failure (recurred 2-3 times) was root-caused to stray nested `build`/`install`/`log` directories inside `src/rov-autonomy` on RPi5 (from `colcon build` run from the wrong directory), colliding with the real workspace at `/data/ros2_ws` — fixed by removing the stray directories (gitignored, local cleanup only, no git change); durable guard recorded: `colcon build` only ever from `/data/ros2_ws`. §0 Confirmed Working, Open Items, PARKED, NEXT, Recent Commits all updated; new §15g narrative section. Companion DIR bump: `docs/design_intent_register.md` v1.4 — see that changelog for the full corrected-intent detail, not repeated here. |
| 2.25 | 13 Jul 2026 | **Two-part fix session, both surfaced by putting the 11 Jul mode-aware work under load.** (1) `mode_profile_loader` was found never to have been wired into `rov_full.launch.py` despite being committed 7 Jul (`4e48dc2`) — `/rov/mode/profile` had zero publishers at runtime since then, so the entire operating-mode profile system had been dark and `failsafe_monitor`'s `health_role` had been silently running on its gate default the whole time. Fixed by adding the node to the launch file with respawn (`464e17e`), verified live (publisher count 0→1). (2) With the mode system now actually publishing, `failsafe_monitor`'s mode-aware recovery (`2a3e577`, same session — subscribes to the resolved profile, branches condition-clear recovery on `health_role`) surfaced a second gap: gate-mode committed recovery entered `RETURN_TO_SAFE` and could never leave it, because that branch deliberately ignores further condition-clear so recovery runs to completion, and nothing signalled completion. Fixed by a new PERMANENT interface, `/rov/nav/safe_zone_reached` (std_msgs/Bool), that transitions the FSM `RETURN_TO_SAFE`→`NORMAL` on arrival when gate-mode (`bf815cc`), verified live end-to-end on the bench (published the event by hand, watched the FSM release and the `motion_controller` recovery broadcast stop). Today's publisher is temporary bench scaffolding only — real navigation-raised arrival and a temporary W1-contextual operator button are both still to come; see the new Parked Design Item in `docs/design_intent_register.md` v1.3. §0 Confirmed Working, PARKED, and NEXT updated; new §15f narrative section. **Two pre-existing `failsafe_monitor` defects found this session (not caused by it):** `flag_manual_abort` is a one-way latch never reset, permanently shadowing all lower-priority handling (including this session's own new logic) after any single manual abort — field-deployment concern, needs its own session; and `failsafe_monitor` produced no log output in `journalctl` this session, costing diagnostic time. Both added to §0 Open Items and §15f. Companion DIR bump: `docs/design_intent_register.md` v1.3 adds the mode-aware safe-zone-reached Parked Design Item — see that changelog for detail. |
| 2.24 | 11 Jul 2026 | **§15d resolved.** Vehicle health / FC-comms-loss fault (discovered 9 Jul, flagged OPEN/SAFETY-CRITICAL in v2.22) is fixed as of four commits today: `86b2728` (unconditional RED on comms loss + `fcu_ever_connected` latch + cause text), `1b0050d` (FSM recovery fix — `RETURN_TO_SAFE` was a terminal trap), `dbe9515` (new `external/rov-failsafe-cause` variable), `3409ae5` (W1 widget shows state + cause text). Verified live end-to-end. Related fault found+fixed same session: `cockpit_bridge` was publishing `failsafe_state` (FSM, 3+) not `assessment_state` (0/1/2) to `rov-failsafe` — the actual cause of that day's "widgets show no data," fixed in `d854641`, compounded by a restart-storm wedging port 9001 (→ CLAUDE.md's one-restart-then-verify-9001 rule). §0 Confirmed Working and Recent Commits updated; PARKED list updated to reflect intent now recorded in the DIR (mode-dependent recovery, auto-recovery, dedicated cause field, AMBER runtime-response — all still open on implementation, closed on design intent). Companion DIR bump: `docs/design_intent_register.md` v1.2 adds the two DIR-7 addenda and a Parked Design Items section — see that changelog for detail, not repeated here. |
| 2.23 | 11 Jul 2026 | **Doc restructure.** This document is now the single source of truth for current state (Quick Status Board gains PARKED and NEXT subsections, folding in the list previously buried in CLAUDE.md's SESSION LOG and §12's Next Priorities). New **Changelog** section (below) replaces CLAUDE.md's SESSION LOG — same entries, same append-only format, moved here. The embedded Design Intent Register (formerly here, ~200 lines) is **removed** — full DIR now lives only in `docs/design_intent_register.md`; this document keeps a short pointer + summary of which DIRs govern current work. All docs moved into the repo under `docs/` (git-tracked, no version number in filenames — see CLAUDE.md Documentation Discipline). Fixed: duplicate `## 15d` heading (second instance renamed `15e`); §15's "Recording control" locked-decision row corrected from "DIR-9 — always-on" to DIR-9a (ARM-bounded); duplicate 8-variable data-lake table in §9 replaced with a pointer to §0's canonical 19-variable table; stale "Widget JSON not in Gitea" row in §16 marked resolved (W6 committed per `785ec56`, already reflected in §12). No new facts asserted beyond what was already recorded elsewhere in this document, CLAUDE.md, or the DIR — this pass is consolidation, not new findings. |
| 2.22 | 9 Jul 2026 | **⚠ ROOT-CAUSE CORRECTION (supersedes v2.10 and v2.19 claims).** The BlueOS "FD exhaustion" bug was NEVER fixed. `nofile=65536` was a ceiling raise, not a fix — it moved failure from ~10 min to ~11 h of cumulative exposure. Real cause, measured: `ardupilot_manager`'s Linux board detector opens `/dev/i2c-1` on each failed probe and never closes it. The BlueOS **Autopilot Firmware / board-config page** polls `GET /v1.0/available_boards`; each request fires 20 `detect_boards` probes (Navigator/NavigatorPi4/NavigatorPi5/Argonot). We run a Pixhawk on USB serial, so every probe fails and leaks. Measured ~100–108 FD/min with the page open; **exactly zero with it closed** (4630 flat over 65 min; +90 FD in 50 s on opening; flat immediately on closing). FD composition: 2740/2745 were `/dev/i2c-1`, 3 sockets. Probe:request ratio 20:1 (8920:445, from logs). Count accumulates across sessions; resets only when `blueos-core` is recreated. Upstream BlueOS 1.4.3 defect — **do NOT patch BlueOS** (updates revert). Dev mitigation: don't leave the board-config page open. Field mitigation NOT built. §15a rewritten. **New §15d — OPEN, SAFETY-CRITICAL:** `failsafe_monitor` reported GREEN while the FC was unreachable; W1 is the AUV pre-dive GO/NO-GO gate (`health_role: gate`) and would pass a dive with a dead flight controller. All widget indicators reflect the RPi5 stack, not the vehicle. **Also this session:** `rov_api` `/mode` endpoints (GET/POST, validate-before-write + `/rov/mode/reload` trigger) built, deployed, tested incl. rejection path (`9109439`). **W8 mode indicator** built and validated live — render + data (`87000e6`). **Chronic "widgets show no data" root-caused:** `cockpit_bridge` was in the launch file but had no respawn; a boot-time race left it permanently dead, taking the whole data-lake feed with it — fixed with `respawn=True` (`c099c00`), verified across a cold stack restart. Respawn extended to `failsafe_monitor`, `motion_controller`, `ekf_filter_node`, `depth_node`, `mission_executor` (`a5e1c99`); deliberately NOT applied to `mavros_node` (would mask FCU-link instability), `recording_manager` (pending DIR-9a rework), perception (not operational), `foxglove_bridge` (viz-only). **FLAGGED OPEN:** `recording_manager.py` still implements superseded DIR-9 (continuous-from-boot) rather than governing DIR-9a (ARM-bounded); and `argonaut-stack.sh` passes `record:=${RECORD}` to a launch file that declares no such arg, so `RECORD=false` is silently ignored. Claude Code (VS Code) adopted for repo edits; SESSION LOG added to `CLAUDE.md` to bridge Claude Code ↔ claude.ai. |
| 2.21 | 7 Jul 2026 | **Operating-mode profile system** built, verified end-to-end, committed (`4e48dc2`) and pushed. New `ModeProfile.msg` (rov_interfaces) — atomic resolved flag set. New `mode_profile_loader` node + `config/mode_profiles.yaml` (rov_mission): ROV/AUV default profiles plus custom hybrid via Save As; resolved flags published latched on `/rov/mode/profile`; reloadable via `/rov/mode/reload` (std_srvs/Trigger); fail-safe validation rejects bad flag values (caught a YAML 1.1 `on`→bool trap; fixed by quoting + a normaliser). `cockpit_bridge` now surfaces 11 mode flags to the data lake — **19 `external/rov-*` variables total** (was 8). Widgets read flags, not modes; hybrid = any saved profile with overrides. Verified: ROV and AUV both resolve + live reload. **DIR reworked for consistency:** DIR-9 collapsed to a stub; **DIR-9a** governing — single **ARM-bounded black-box recording** (AUV: ARM starts / DISARM stops; ROV: operator start/stop + ARM reminder; purpose black-box/debug/ML/fault-finding only; certified deliverable capture out of scope pending Class-approved devices). DIR-1 corrected (tether/HILT, not power source, determines mode; tethered may use battery or topside supply). DIR-7 gains AUV-mode health semantics (GREEN/AMBER/RED = within/approaching/terminate, parameter-driven). DIR-4/5/10 recording contradictions removed (always-on continuous "core" dropped; base set now recorded within the armed run). **Backup:** overdue alert resolved (07-05 was a one-night hostname-resolution transient, not a config regression); segment `_9` size-mismatch re-synced (captured mid-rotation); RPi5 backup script hardened (`--exclude='*.mcap'` so in-progress bags are not captured mid-write; active-dive prune guard). |
| 2.20 | July 2026 | Phase 3 Stage 2 core complete: `rov_api` FastAPI backend deployed on RPi5:8081 (commit 66a18ef) — HTTP-to-ROS2 bridge, uvicorn main thread + embedded rclpy node in daemon thread. Endpoints verified end-to-end: GET /health (mission/recording/failsafe), POST /abort (W3 RETURN TO SAFE → /rov/mission/abort → failsafe_monitor FSM NORMAL→HOLD_AND_RECOVER), POST /mission/start (MissionCommand START), POST /mission/stop (MissionCommand ABORT). Runs as argonaut-api.service (manual-start, same policy as argonaut.service). fastapi/uvicorn installed system-wide (--ignore-installed typing_extensions). Deploy files version-controlled in repo deploy/ folder (commit 97decac): both services' units + wrappers + README documenting locations, pip deps, install steps. Hostname resolution (DNS) fixed on laptop (Windows hosts) and RPi5 (/etc/hosts) — symbytech/blueos/rov-brain resolve; git remotes returned to hostname form. Widget housekeeping (commit 785ec56): W6 committed as w6_altitude.json (underscore convention), probe widget removed from Gitea, 7 local Copy/working cruft files deleted, W0–W5 line-ending-only churn discarded. All three (Gitea/laptop/RPi5) level at 785ec56. Stage 2b deferred (return budget, /backup/*, /mission/upload) — return budget approach pending "shortest path to safety" discussion; W7 backup design chosen (Option A: server read-only JSON, RPi5 polls outbound). UFW deferred to pre-field hardening pass (no online exposure). |
| 2.19 | July 2026 | Phase 3 Stage 1 complete: `recording_manager` deployed — continuous rosbag2 MCAP recorder, zstd compression, 500MB bag splitting, 10GB retention policy, 1Hz health monitoring, hard no-go gate on `/rov/recording/active`. Pixhawk 2.4.8 confirmed active FC (SITL removed from BlueOS); BlueOS FD exhaustion claimed fixed (`/etc/docker/daemon.json`, nofile=65536) — **✗ this claim was WRONG, retracted in v2.22, see §15a**. MAVLinkRouter confirmed; MAVP2P not recommended (BlueOS own warning). W6 Altitude Widget built and deployed; cockpit_bridge updated to 8 variables (`rov-recording`, `rov-altitude` added). W3 button label decided: "RETURN TO SAFE" (per DIR-5 — assessed vehicle return-to-safe, not emergency hard-stop). Tailscale stopped/disabled on RPi5. Gitea remote updated to IP-based URL. ardupilot_manager_monitor.sh deleted from RPi5. Probe widget deleted (dev tool, not part of W0–W7). |
| 2.18 | July 2026 | Simulation & ML tooling scoped into new companion doc `Argonaut3_Simulation_ML_Tooling_Plan_v1_0.md`. Two tracks defined: Track 1 (Gazebo Harmonic + ArduSub SITL, CPU-led, runs on existing laptop) for nav/mission/failsafe bench; Track 2 (HoloOcean/UNav-Sim/Isaac, RTX GPU) for perception synthetic-data + model training; convergence = structure-relative visual localization (the real bottleneck). Dev-PC tiers costed (min $1.2–1.6k / rec $2.0–2.8k / high $3.5–6k+); cloud alternative costed (free: Kaggle/Colab/Lightning; paid RTX 4090 ~$0.33/hr Vast/RunPod). Training-asset sources listed (Fab/Megascans free; SubPipe/COU datasets; YOLO/SAM2 base models) — NO public jacket dataset exists, CAD+synthetic is the path. RECORD CORRECTED: past SITL failures were under-resourced hosts (server 2-vCPU VM; RPi4 + half-connected Pixhawk) — not intrinsic to SITL. R210 server cannot host GPU sim (1U). PCIe Gen 3 on RPi5 NVMe: option logged, DEFERRED to Phase 3 camera-to-MCAP. |
| 2.17 | June 2026 | W0–W5 all committed to Gitea (W0 layout fix 2cb7b74). Stale server clone `~/rov-autonomy` deleted. MAVROS recovery: confirmed data point — RPi4 BlueOS core restart ALONE recovered the system this session (RPi5 untouched). DESIGN INTENT REGISTER added as Section 1 — dual-capability AUV/ROV platform, HILT as override channel, mission lifecycle, abort/return-to-safe model (mission-bounded), Emergency Surface shelved. Re-scopes W3/W4 backend: near-term need is tethered HILT path. FastAPI backend NOT started — intent locked first. |
| 2.16 | June 2026 | W1–W5 all confirmed working in Cockpit v1.18.2 with live data. W1 failsafe variable name corrected (`rov-failsafe` full word, not truncated `rov-failsa`). Outstanding widget work is backend-side. Flagged items consolidated. |
| 2.15 | June 2026 | Cockpit v1.18.2 DIY widget data lake getter CONFIRMED via probe widget. `getDataLakeVariableData(id)` returns live values. W1–W5 widget blocker resolved. Full `window.cockpit` surface documented. |
| 2.14 | June 2026 | COCKPIT DATA BRIDGE BUILT & WORKING (the P2 keystone). New `cockpit_bridge` ROS2 node serves a WebSocket on :9001; all 6 vars land in data lake as `external/rov-*` (commit 12b5161). RPi4 SD replaced + backup system (pi4-backup.sh) in place & verified. Logging flood (NaN TF) mitigated. W0 working (scroll fixed). |
| 2.13 | June 2026 | RPi5 storage RE-ARCHITECTED to OS-on-SD + data-on-NVMe. OS migrated SD via rsync; NVMe wiped to single ext4 data volume at /data; workspace moved to /data/ros2_ws. Boot fixed by switching root mount from LABEL to PARTUUID. Full stack now runs as systemd service argonaut.service. |
| 2.12 | June 2026 | Full stack now launches end-to-end with ZERO crashes (commit c94412d). Fixed: MAVROS fcu_url, plugin_allowlist, timestamped MCAP bag dir, BEST_EFFORT QoS, battery<0=UNKNOWN guard. |
| 2.11 | June 2026 | Full autonomy stack deployed and launched end-to-end on RPi5. THREE repo bugs found and fixed. foxglove_bridge :8765 + MCAP recording to /data/bags confirmed working. |
| 2.10 | June 2026 | MAVROS link RPi5->RPi4 verified. BlueOS "too many open files" re-diagnosed as uvicorn FD exhaustion. **✗ WRONG — retracted in v2.22, see §15a.** |
| 2.9 | June 2026 | RPi4 fully reinstalled. BlueOS 1.4.3 on fresh SD. Static IP 192.168.1.100. ArduSub SITL 4.5.7 STABLE running. Both Pis operational. |
| 2.8 | June 2026 | RPi5 base stack complete. Ubuntu 24.04.4 installed. Hostname rov-brain, static IP 192.168.1.101, Tailscale. ROS2 Jazzy, Docker, MAVROS installed. |
| 2.7 | May 2026 | Cockpit version corrected to v1.18.0. Generic WebSocket confirmed as correct dev injection path. Data lake key format confirmed with external/ prefix. |
| 2.6 | May 2026 | All 6 widgets built and confirmed working in Cockpit native desktop app. Widget files committed to Gitea. |
| 2.5 | May 2026 | Widget approach confirmed (DIY JSON), Data Lake Inspector built. |
| 2.4 | May 2026 | Architecture overhaul — three-prong system, Foxglove, server separation. |
| 2.3 | May 2026 | Clean dev environment — BlueOS VM + internal SITL + Cockpit working. |
| 2.2 | May 2026 | Dev environment attempt — standalone Docker SITL stack (superseded). |
| 2.1 | May 2026 | Server dev environment plan, shopping list update. |
| 2.0 | May 2026 | Workspace, build pipeline, failsafe design, UI design. |
| 1.0 | April 2026 | Initial handover document. |

---

## Changelog

**Moved here from `CLAUDE.md`'s SESSION LOG (11 Jul 2026 restructure).** This is the granular,
session-by-session record that bridges Claude Code (VS Code) sessions and the claude.ai chat —
finer-grained than the Version History table above, which narrates per-doc-version. Append-only;
correct forward with a new dated entry, never edit past ones.

**Claude Code appends an entry here at the end of any session where one of these happened** — not
for routine edits, refactors, or in-progress work: a design/architecture decision was made or
changed; a hardware fact was confirmed or changed; a security-relevant change was made; a
backup/recovery-relevant change was made; a bug was root-caused and fixed (not just patched); a
previously-deferred item was completed or its trigger condition changed; anything Grant would
reasonably expect this chat to already know next time he opens it. Skip typo fixes, comment-only
changes, WIP work, or anything already fully captured by the git commit message with no extra
context needed. Full trigger-rule text lives in `CLAUDE.md`.

- 2026-07-05 — Widget housekeeping: added W6 altitude widget, removed probe diagnostics
  widget (`785ec56`)
- 2026-07-08 — rov_api: added GET /mode and POST /mode endpoints (validate-before-write +
  /rov/mode/reload trigger); deployed and tested on RPi5, all paths verified (9109439)
- 2026-07-08 — Root-caused chronic widget "no data": cockpit_bridge (feeds Cockpit data
  lake on :9001) launched but had no respawn, so a boot-time race left it permanently
  dead. Added respawn=True to its launch Node. W8 mode indicator built + validated live
  (render+data). W1/W8 confirmed GREEN/AUV once bridge up.
- 2026-07-08 — FLAGGED (open): recording_manager.py still implements superseded DIR-9
  (continuous-from-boot recording), NOT the governing DIR-9a (ARM-bounded black-box:
  AUV ARM-start/DISARM-stop, ROV operator manual + arm reminder). Design decision
  exists only in handover + CLAUDE.md; code never updated. foxglove_mcap.launch.py
  comments also still say DIR-9. Also: argonaut-stack.sh passes record:=${RECORD} but
  rov_full.launch.py declares no such arg, so RECORD=false is silently ignored and
  bags record regardless. Both deferred to a dedicated DIR-9a recorder-rework session
  — do NOT wire RECORD into the current DIR-9 model.
- 2026-07-08 — Added respawn=True/respawn_delay=5.0 to failsafe_monitor,
  motion_controller, ekf_filter_node, depth_node, mission_executor. Deliberately NOT
  applied to: mavros_node (respawn could mask/worsen the chronic FCU-link instability
  — belongs in that investigation), recording_manager (pending DIR-9a rework),
  camera_node/feature_detector (perception not operational yet), foxglove_bridge
  (viz-only). NOTE: failsafe_monitor respawn has a coverage gap during the 5s respawn
  delay — no failsafe assessment in that window; flagged for future design
  consideration.
- 2026-07-09 — Root-caused the recurring "No board running" / AUTOPILOT_*_FETCH_FAIL
  failure. NOT the previously-documented TCP-socket/UI-polling cause. Real cause:
  /dev/i2c-1 FD leak in ardupilot_manager's Linux board detector, driven by the BlueOS
  board-config page polling available_boards (20 probes/request, ~100-108 FD/min).
  Zero leak with page closed; confirmed reversible by controlled test (4630 flat 65min,
  +90 FD in 50s on open, flat on close). nofile=65536 only delayed failure to ~11h
  cumulative page-open time. Upstream BlueOS 1.4.3 bug — not patching. Corrected the
  stale root cause in this file.
- 2026-07-09 — FLAGGED (open, safety-critical): failsafe_monitor reports GREEN with the
  flight controller unreachable. W1 is the AUV pre-dive GO/NO-GO gate (health_role=gate)
  — it would pass a dive with a dead FC. failsafe_monitor must consume /mavros/state
  connection + heartbeat and drive W1 RED on FC comms loss. W1 status line should carry
  cause text (e.g. "Vehicle Disconnected", "Heartbeat Lost", "No Board Running"). Also
  flagged: no FD-count check on ardupilot_manager in the pre-dive gate.
- 2026-07-11 — RESOLVED the 2026-07-09 flagged safety-critical fault: failsafe_monitor now
  drives W1 RED on FC comms loss. Four commits: (1) 86b2728 — comms loss forces
  unconditional RED (was scored as one marginal parameter, only reached AMBER) plus
  fcu_ever_connected latch so the startup window before any heartbeat is not reported GREEN;
  cause text "Heartbeat Lost"/"Vehicle Disconnected" added to FailsafeStatus.message per
  DIR-7. (2) 1b0050d — FSM recovery fix: RETURN_TO_SAFE was a terminal trap (only
  HOLD_AND_RECOVER could return to NORMAL); now both recover when conditions clear. (3)
  dbe9515 — cockpit_bridge publishes new data lake variable rov-failsafe-cause (substring of
  message before first "|"; empty when nominal). (4) 3409ae5 — W1 shows state word on
  #w1-message (state-driven, no flicker) and cause text on #w1-footer, footer restyled red
  11px, blank when nominal. All four verified live end-to-end: kill mavros -> RED +
  "Heartbeat Lost", restart -> recover to GREEN. Also corrected: cockpit_bridge was
  publishing failsafe_state (FSM, can be 3+) not assessment_state (0/1/2) to rov-failsafe —
  fixed in dbe9515's sibling commit d854641; this was the real cause of the day's "widgets
  show no data" (value 3 out of W1's range), compounded by a restart-storm wedging 9001.
- 2026-07-11 — Doc restructure (this pass): consolidated current-state into this document,
  moved this log here from CLAUDE.md, removed the embedded DIR copy in favour of a pointer
  to `docs/design_intent_register.md`, moved all project docs into the repo `docs/` folder.
  See Version History v2.23 above for detail.
- 2026-07-13 — Found mode_profile_loader (rov_mission) was built and committed 7 Jul in
  4e48dc2 but never added to rov_full.launch.py, so /rov/mode/profile had zero publishers at
  runtime since then — the whole operating-mode profile system had been dark, and
  failsafe_monitor's health_role had been silently running on its gate default the whole
  time. Fixed by adding the Node to the launch file (with the other core nodes, before
  Perception) with respawn=True/respawn_delay=5.0, matching the other core nodes (c099c00,
  a5e1c99). Verified live: publisher count 0 -> 1 (464e17e).
- 2026-07-13 — failsafe_monitor made mode-aware (2a3e577, same session): subscribes to the
  resolved /rov/mode/profile and branches condition-clear recovery on health_role per the
  DIR-7 addendum — gate (AUV) commits HOLD_AND_RECOVER -> RETURN_TO_SAFE instead of returning
  to NORMAL; advisory (ROV) holds for an explicit operator resume (placeholder topic). This
  surfaced a new gap once the mode system was actually live (previous entry): gate-mode
  RETURN_TO_SAFE had no exit, a terminal trap, because that branch deliberately ignores
  further condition-clear so committed recovery runs to completion, and nothing signalled
  completion. Fixed (bf815cc) with a new subscription, /rov/nav/safe_zone_reached
  (std_msgs/Bool) — THIS TOPIC AND SUBSCRIPTION ARE THE PERMANENT nav-arrival interface;
  today's publisher is temporary bench scaffolding only. On gate + RETURN_TO_SAFE + True,
  transitions to NORMAL with cause text "Safe zone reached — recovery complete" (DIR-7 cause
  MUST, via the existing message-prefix mechanism, no FailsafeStatus.msg change). Verified
  live end-to-end on the bench: published the event by hand, watched the FSM release
  RETURN_TO_SAFE -> NORMAL and the motion_controller recovery broadcast stop. Real
  navigation-raised arrival event and a temporary W1-contextual operator button (rov_api-
  backed bench scaffolding, next step) are both still to come — see new Parked Design Item,
  docs/design_intent_register.md.
- 2026-07-13 — FLAGGED (open, pre-existing, found not caused by this session's work): (1)
  flag_manual_abort in failsafe_monitor is a one-way latch, never reset to False anywhere —
  once any manual abort fires, the priority-6 handler returns early every subsequent cycle
  for the rest of the process lifetime, permanently shadowing all lower-priority handling
  including this session's new mode-aware recovery and arrival-event logic. Needs a reset
  condition (candidates: on reaching NORMAL, on explicit operator reset, or process-restart
  only — undecided, needs its own session). Field-deployment concern: one manual abort
  disables all future automatic recovery for the rest of that dive. (2) failsafe_monitor
  produced no log output at all in journalctl this session — state had to be read directly
  off /rov/failsafe every time. Should log state transitions and cause. Minor but cost real
  diagnostic time this session.
- 2026-07-13 — CORRECTED (later same session): the DIR-7 in-mission-recovery intent
  implemented earlier today (2a3e577) was inverted. It had gate-mode AUV commit to recovery
  ON CONDITION-CLEAR, which produced a contradictory live state: W1 GREEN ("Systems nominal")
  while the vehicle was simultaneously in RETURN_TO_SAFE ("Recovering to safe zone"). Corrected
  intent recorded in docs/design_intent_register.md v1.4 (255096f): GREEN is unconditionally a
  working state, a cleared fault returns to GREEN and the mission continues/resumes, and
  committed recovery is triggered by fault PERSISTENCE or RECURRENCE, never by clearing. Three
  fault tiers recorded: uncontrollable (power loss etc., out of FSM scope, adjacent to shelved
  DIR-6); transient/self-resetting (system attempts restore, returns to GREEN and resumes on
  success, commits to RETURN_TO_SAFE only once a raw config-driven failure count is exceeded,
  deliberately no time window); hard/persistent (RED that doesn't clear, commits immediately).
  All thresholds are config-wizard-driven via the resolved mode profile, like health_role, not
  hardcoded. Survives unaffected: health_role mode-awareness (2a3e577's subscription
  mechanism) and the safe-zone-reached arrival EXIT mechanism (bf815cc) — only ENTRY into
  committed recovery was wrong, the exit mechanism is unchanged and correct. 2a3e577's
  condition-clear entry logic is now KNOWN-WRONG pending rework — new top-priority NEXT item,
  ahead of everything previously listed. New Parked Design Item: adaptive restart-timing
  (monitor own restore count/duration per fault, weigh future attempts against observed
  history instead of a fixed time) — depends on the count-based retry machinery above existing
  first, design-captured not build-specified.
- 2026-07-13 — W1 safe-zone-reached button work PAUSED, not abandoned. Supporting
  infrastructure built and committed: rov_api POST /nav/safe_zone_reached endpoint (mirrors
  the existing /abort pattern exactly) and the new external/rov-failsafe-state data lake
  variable it depends on (surfaces FailsafeStatus.failsafe_state, distinct from the existing
  external/rov-failsafe assessment integer) — both in 2def2b1. Pre-edit backup of the W1
  widget made per CREATE-THEN-BACKUP: widgets/w1_system_health_indicator.json.2026-07-13.bak.
  The widget edit itself was deliberately not started — the button would surface FSM state
  (rov-failsafe-state == 3 / RETURN_TO_SAFE) reached via the trigger just found to be
  known-wrong above, so it would be meaningless until the FSM rework lands.
- 2026-07-13 — Root-caused a recurring rov_mission build failure (recurred 2-3 times across
  sessions): stray nested build/, install/, and log/ directories existed inside
  src/rov-autonomy on RPi5, created when colcon build was at some point run from inside the
  repo source directory instead of the actual workspace root, /data/ros2_ws. These collided
  with the real workspace and caused rov_mission builds to fail resolving mode_profiles.yaml
  through a stale nested install/ tree. Fixed by removing the stray directories from RPi5 —
  gitignored and untracked, so this was a local filesystem cleanup only, no git change.
  Durable guard: colcon build must only ever be run from /data/ros2_ws, never from inside
  src/rov-autonomy. Recorded so this is not reintroduced.

---

## Design Intent Register — pointer

The full Design Intent Register (DIR-1 through DIR-12) lives in **`docs/design_intent_register.md`**,
git-tracked in the repo, not duplicated here. That file is the source of truth for *why* —
read it before any design/architecture decision.

**DIRs governing current work, for quick orientation:**
- **DIR-1/DIR-2** — tether/HILT link (not power source) determines AUV vs ROV mode; HILT is an
  override channel, not a confirmation gate.
- **DIR-7/DIR-12** — health has two distinct subjects (vehicle vs autonomy stack); AUV
  `health_role: gate` vs ROV `advisory`; health assessment must include FC connectivity and
  must carry cause text. (Implemented 11 Jul 2026 — see Changelog above.)
- **DIR-9a (governing, supersedes DIR-9)** — recording is a single ARM-bounded black-box tier,
  keyed to the `recording_control` flag, not the mode name. `recording_manager.py` still
  implements the superseded DIR-9 model — **known drift, open**.
- **DIR-5/5a/5b** — return-to-safe is mission-bounded, vehicle-layer, and mode-dependent
  (AUV automatic, ROV alert+override); operator override continues the mission, it does not
  pause it.

---

## 1. Project Overview

### Goal
A fully autonomous underwater inspection system targeting hull and jacket surveys
with no human operator during field operations. Built on a shoestring budget using
commodity hardware, designed for field deployment from day one.

### Target Structures
- **Jackets:** Fixed offshore structures — predictable geometry
- **Hulls:** In-port vessels — moored, requires online re-localisation

### Key Design Values
- Cost-conscious — use what is available, upgrade when a dead end is reached
- Field-operational from day one — same codebase, different configs per environment
- Full autonomy — no pilot required during operation
- Cockpit as access hub — Argonaut 3 widgets only, not a container for everything
- QGroundControl retained as dev/debug only

---

## 2. Platform

**Vehicle:** BlueROV2 Heavy (Blue Robotics)
**Operating mode:** Vehicle-side compute only
**Tether:** Single Ethernet cable — all topside/bottomside communication over one cable

---

## 3. Hardware Architecture

### On the ROV

| Board | Role | OS |
|---|---|---|
| Raspberry Pi 4 (8GB) | BlueOS gateway — MAVLink routing, Pixhawk FC host | BlueOS 1.4.3 (armv7) |
| Raspberry Pi 5 (8GB) | Autonomy brain — ROS2 stack, foxglove_bridge, MCAP recorder | Ubuntu 24.04 LTS + ROS2 Jazzy |

> **RPi4:** BlueOS 1.4.3 running. Static IP 192.168.1.100. Tailscale 100.100.119.125
> (node renamed `blueos-1`). **Pixhawk 2.4.8 active on `/dev/ttyACM0`** — SITL is NOT
> in production use. BlueOS `preferred_board: Pixhawk1`. Docker `nofile` ulimit = 65536
> (ceiling raise only — **NOT a fix**; see §15a, ardupilot-manager `/dev/i2c-1` FD leak is an
> OPEN bug). Pirate Mode enabled.
>
> **RPi5:** Ubuntu 24.04.4, **boots from SD card** (`/dev/mmcblk0`). NVMe (`/data`) =
> data only (ros2_ws, bags, logs — NOT the OS). Static IP 192.168.1.101. Tailscale
> installed but **STOPPED and DISABLED** (re-enable: `sudo systemctl enable --now tailscaled`).
> ROS2 Jazzy, Docker, MAVROS installed. `argonaut.service` running (manual-start in dev).
> **Weak SSH password — harden before field deployment.**

### Sensors

| Sensor | Status |
|---|---|
| Xsens MTI-30-2A5G4 IMU | Have unit — CA-USB-MTI cable on order |
| IP cameras (2+) | Have units — not yet integrated |
| Ping2 Sonar (downward) | Phase 3 purchase (~$280 USD) — required for W6 full test |
| Ping2 Sonar (forward) | Phase 4 purchase |
| Ping360 Scanning Sonar | Phase 4-5 purchase |
| DVL (Doppler Velocity Log) | Phase 7 — critical for accurate X/Y positioning |
| Tritech Gemini 720im | Phase 6+ |

> **DVL is the most important missing sensor.** Without it, X/Y position is
> dead-reckoned from IMU only and drifts over time. Z axis (Bar30) is accurate.
> All position displays must reflect this uncertainty honestly.

### Power

| Device | PSU | Status |
|---|---|---|
| RPi4 | 5.1V / 3A RPi4 official PSU | In use |
| RPi5 | 5V / 5A 27W USB-C official RPi5 PSU | In use |

### Storage

| Device | Storage | Notes |
|---|---|---|
| RPi4 | Fresh microSD (Patriot 64GB or equivalent) | BlueOS OS drive |
| RPi5 (OS) | SD card (`/dev/mmcblk0`) | Ubuntu 24.04, hardened (log2ram, tmpfs, journald capped) |
| RPi5 (Data) | RPi M.2 HAT+ + RPi SSD 512GB (2230 NVMe) at `/data` | ros2_ws, bags, logs |

---

## 4. Software Stack

### RPi4 (BlueOS Gateway)
- BlueOS 1.4.3 (armv7) — running, static IP 192.168.1.100
- Tailscale 100.100.119.125 (`blueos-1`)
- **Pixhawk 2.4.8** — active flight controller on `/dev/ttyACM0:115200`
- **MAVLinkRouter** — routes Pixhawk to MAVROS on RPi5 and to Edge PC for QGC debug.
  **MAVP2P has known issues (BlueOS own warning) — do not use.**
- `/etc/docker/daemon.json` — `nofile: 65536` (**ceiling raise only, NOT a fix** — see §15a)
- Pirate Mode — enabled

### RPi5 (Autonomy Brain)
- Ubuntu Server 24.04.4 LTS (Noble, arm64) — boots from SD, NVMe = `/data`
- ROS2 Jazzy Jalisco — auto-sourced in `.bashrc`
- MAVROS + GeographicLib datasets
- Docker 29.5.3 — runs without sudo
- Argonaut 3 autonomy stack (7 packages) — deployed, running as `argonaut.service`
- `foxglove_bridge` — confirmed running on `:8765`
- `recording_manager` — confirmed active, bags to `/data/bags/`
- `cockpit_bridge` — WebSocket `:9001`, 8 variables
- Tailscale — installed, service stopped and disabled
- Xsens MTi ROS2 driver — pending CA-USB-MTI cable

### Edge PC / Field Laptop BB8 (Topside)
- **Cockpit native desktop app v1.18.2** — primary GCS, Argonaut 3 widgets
- Foxglove Studio — live monitoring (Prong 2) and post-dive replay (Prong 3)
- Mission Planner (Phase 6+) — standalone 3D web app, locally installed
- Local model + mission cache — glTF twins and YAMLs pre-synced, offline capable

> **Cockpit native app is required.** The BlueOS browser extension version is
> "Lite" and does not support DIY widgets.

---

## 5. System Architecture — Three Zones

```
ONSHORE                    OFFSHORE TOPSIDE              OFFSHORE BOTTOMSIDE
------------------         ----------------------        -----------------------------
Dev Server                 Edge PC / Field Laptop        BlueROV2 Heavy
  Harbor registry            Cockpit native v1.18.2        RPi4 (BlueOS)
  Gitea source control       Foxglove Studio                 MAVLink Router
  ARM64 build pipeline*      Mission Planner*                Pixhawk 2.4.8 (ArduSub)
                             Local cache                 RPi5 (ROS2 Jazzy)
Ops Server (separate)                                      Autonomy stack (7 packages)
  Digital twin storage     <-- Ethernet Tether -->         foxglove_bridge :8765
  Mission archive                                           recording_manager
  Post-dive data                                            cockpit_bridge :9001
  Model conversion                                        Sensors
  Tailscale endpoint                                        Xsens, Bar30, cameras, sonar
```

*ARM64 builds: laptop only (server overheats under compilation load)
*Mission Planner: Phase 6+ standalone app

### Ethernet Tether — Single Cable

| Flow | Direction |
|---|---|
| BlueOS web UI :80 | Tether to Edge PC browser |
| MAVLink UDP :14550 | Tether to QGC on Edge PC (debug) |
| foxglove WebSocket :8765 | Tether to Foxglove Studio on Edge PC |
| cockpit_bridge WebSocket :9001 | Tether to Cockpit on Edge PC |
| Mission YAML upload | Edge PC to RPi5 via SSH/SCP |
| MCAP sync post-dive | RPi5 to Edge PC via SSH/SCP |
| Container deploy | Edge PC to RPi5 via SSH |

---

## 6. Three-Prong Mission System

### Prong 1 — Mission Creation + Editing
- glTF model synced to Edge PC local cache pre-dive
- Mission Planner on Edge PC: Three.js 3D scene with structure twin
- Hull mode: parametric lawnmower path generator
- Jacket mode: 3D waypoint editor
- Output: Mission YAML (rich format, structure-relative 3D coordinates)
- Status: Phase 6+ — standalone web app, not in Cockpit

### Prong 2 — Mission Execution + Monitoring
- `rov_mission` on RPi5 executes YAML via state machine
- MAVROS translates ROS2 setpoints to MAVLink to RPi4 to ArduSub to thrusters
- All ROS2 topics streamed via `foxglove_bridge` WebSocket to Foxglove Studio
- Cockpit widgets on Edge PC: Health / Mission Status / Return To Safe
- Action path: Cockpit widget fetch() -> FastAPI on RPi5 -> `/rov/mission/abort`

### Prong 3 — Data Collection + Replay
- MCAP recorder logs ALL ROS2 topics to NVMe SSD during dive
- Post-dive: MCAP transferred to Edge PC via SCP over tether
- Foxglove Studio on Edge PC: immediate replay, no server needed
- When server reachable: MCAP archived to Ops Server via Tailscale

---

## 7. Digital Twin

| Format | Origin | Status |
|---|---|---|
| EBM / EBMX | CADMATIC — hull model | Have file |
| NWD | Autodesk Navisworks — platform models | Have files (x2 platforms) |
| IFC | Open BIM standard — intermediate | Conversion target |
| glTF / GLB | Khronos Group — web render target | Final format for all tools |

Conversion pipeline: EBM / NWD -> CAD Exchanger SDK -> IFC -> glTF / GLB
Currently manual. Automated pipeline is a later infrastructure task.

---

## 8. ROS2 Workspace

**Location on RPi5:** `/data/ros2_ws/src/rov-autonomy/`
**Gitea repo:** `administrator/rov-autonomy`
**Gitea SSH remote:** `ssh://git@symbytech:222/administrator/rov-autonomy.git`
(hostname form — resolves via hosts entries on laptop and RPi5)

### Package Structure

| Package | Purpose | Key nodes |
|---|---|---|
| `rov_interfaces` | Custom messages and services | — (msgs/srvs only) |
| `rov_navigation` | State estimation, EKF, depth | `state_estimator`, `depth_node` |
| `rov_perception` | Camera nodes, feature detection | `camera_node`, `feature_detector` |
| `rov_control` | Motion controller, failsafe monitor | `motion_controller`, `failsafe_monitor` |
| `rov_mission` | Mission executor, cockpit bridge, recorder | `mission_executor`, `cockpit_bridge`, `recording_manager` |
| `rov_bringup` | Top-level launch files | — (launch files only) |
| `rov_api` | FastAPI backend — HTTP→ROS2 bridge on :8081 | `api_node` |
| `rov_simulation` | Simulation nodes (dev only) | Do NOT deploy in production launch files |

### Key Topics

| Topic | Type | Publisher |
|---|---|---|
| `/imu/data` | sensor_msgs/Imu | Xsens driver |
| `/mavros/imu/data` | sensor_msgs/Imu | MAVROS |
| `/odometry/filtered` | nav_msgs/Odometry | EKF node |
| `/rov/failsafe` | rov_interfaces/FailsafeStatus | failsafe_monitor |
| `/rov/mission/status` | rov_interfaces/MissionStatus | mission_executor |
| `/rov/recording/active` | std_msgs/Bool | recording_manager |
| `/rov/recording/bag_dir` | std_msgs/String | recording_manager |
| `/ping2/altitude` | sensor_msgs/Range | ping2 driver (hardware-gated) |

### Key Commits

| Hash | Description |
|---|---|
| `0640353` | W6: Add rov-altitude to cockpit_bridge for Ping2 downward sonar |
| `5d75a9d` | Phase 3 Stage 1: Recording Manager (DIR-9) |
| `2cb7b74` | W0 layout fix (3-column, horizontal scroll) |
| `66f9bd7` | W0–W5 all confirmed working, committed |
| `12b5161` | cockpit_bridge node — ROS2→WebSocket :9001 data-lake injection |
| `c94412d` | Full stack zero-crash launch |
| `45b3301` | Fix full-stack launch: fcu_url, plugin_allowlist, QoS, battery guard |
| `7590faf` | Missing setup.cfg in all 5 Python packages |
| `f1c6710` | Missing config/ dir in rov_bringup |

### Build/Deploy Procedure

```bash
# On RPi5 — build (symlink-install works correctly with setup.cfg files in place)
cd /data/ros2_ws
colcon build --packages-select <pkg> --symlink-install
source /data/ros2_ws/install/setup.bash
```

---

## 9. UI Architecture — Argonaut 3 / Cockpit

### Cockpit Setup

**App:** Cockpit native desktop v1.18.2 — installed on field laptop BB8 (Windows)
**Vehicle connection:** Enter RPi4 IP (192.168.1.100) on first launch
**Generic WebSocket:** `ws://192.168.1.101:9001` (cockpit_bridge on RPi5)

> Cockpit native desktop app is mandatory for DIY widget work.
> The BlueOS browser extension version is "Lite" and does NOT support DIY widgets.
> Generic WebSocket data injection requires v1.18.0 minimum.

### Widget Approach — DIY Widgets (Confirmed)

Widgets are Cockpit DIY widgets: plain HTML/CSS/JS stored as JSON files and
imported via the gear icon on a placed DIY widget.

**Import format:** JSON file with three string fields:
```json
{
  "html": "...",
  "css":  "...",
  "js":   "...",
  "inheritCockpitStyles": false
}
```

**CSS rules for DIY widgets (hard-won — do not break these):**
- No `body {}` or `:root {}` blocks — bleed into Cockpit's own styles and crash the renderer
- No `@keyframes` animations — cause renderer instability
- No unicode characters in JS strings — use plain ASCII equivalents
- No `position: fixed` overlays — use `display: block/none` toggling instead
- No `async/await` — use `.then()/.catch()` for fetch calls
- Use element ID/class selectors only — no global tag selectors (e.g. `div {}`)

### Data Injection Path — Confirmed Production Path

```
cockpit_bridge (ROS2 node on RPi5, rov_mission pkg)
  WebSocket server :9001
    <-- Cockpit connects as client
      --> Sends 'variable-name=value' messages at 2Hz
        --> Data lake populated as external/{variable-name}
          --> DIY widget reads via getDataLakeVariableData('external/{variable-name}')
```

**Confirmed data lake read method:** `window.cockpit.getDataLakeVariableData(variableId)`
**Does NOT exist:** `getDataLakeValue()`, `listenToDatalakeVariable()` (not in DIY widget scope)
**Metadata only:** `getAllDataLakeVariablesInfo()` — enumeration only, not value reads

### Data Lake Variables

Full current list (19 total: 8 telemetry + 11 mode-profile flags) is in §0 QUICK STATUS BOARD —
not duplicated here to avoid the two tables drifting apart, which is exactly what happened
between v2.21 (8 vars, this table) and v2.22 (19 vars, §0's table) before this restructure.

### Widget File Locations

**Gitea repo:** `rov-autonomy/widgets/`
**Laptop local:** `C:\Users\grant\Desktop\Autonomous ROV System\rov-autonomy\widgets\`

### Widget Build Status

| # | File | Widget | Test Status | Variable |
|---|---|---|---|---|
| 0 | `w0_data_lake_inspector.json` | Data Lake Inspector | **Confirmed working** (706 vars live) | None — reads all |
| 1 | `w1_system_health_indicator.json` | System Health Indicator | **Visual confirmed** — full test pending | `external/rov-failsafe` |
| 2 | `w2_mission_status.json` | Mission Status | **Visual confirmed** — untested (needs mission running) | `external/rov-ms`, `external/rov-mp` |
| 3 | `w3_abort_button.json` | Return To Safe Button | **UI confirmed** — label rename pending ("ABORT MISSION" → "RETURN TO SAFE"); backend not built | FastAPI `/abort` POST |
| 4 | `w4_mission_setup_button.json` | Mission Setup Button | **Visual confirmed** — untested (needs FastAPI backend) | None — opens URL |
| 5 | `w5_battery_return_budget.json` | Return Budget | **Visual confirmed** — untested (`rov-return-budget` not published) | `external/rov-return-budget` |
| 6 | `w6_altitude.json` | Altitude | **NO SONAR confirmed** — hardware-gated for full test | `external/rov-altitude` |
| 7 | `w7_backup_due_notice.json` | Backup Due Notice | **Not built** — depends on FastAPI backend | FastAPI `/backup/status` |

> **Testing note:** W0 is the only widget with confirmed live data end-to-end. W1–W5
> display correctly but have not been tested with actual operational data. Full testing
> requires a live mission, FastAPI backend, and/or Ping2 hardware depending on the widget.
> W6 full test requires Ping2 sonar hardware. W7 not started.

### How to Install a Widget

1. Open Cockpit native desktop app
2. Connect to vehicle (192.168.1.100)
3. Enable Pirate Mode (Settings → General → Enable Pirate Mode) — resets each session
4. Enter edit mode (pencil icon, top right)
5. Click Add Widget at the bottom — scroll right to find the `</>` DIY widget
6. Drag it onto the main view area and resize
7. Click the gear icon on the placed widget
8. Click Import — select the `.json` file from the widgets folder
9. Widget loads immediately

---

## 10. Failsafe Design Summary

Full design in `docs/failsafe_design.md`.

### Three States

| State | Meaning |
|---|---|
| GREEN | All systems nominal |
| AMBER | One or more parameters degraded — mission continues with reduced confidence |
| RED | Critical failure — safe action triggered automatically |

### Trigger Conditions

| Trigger | Threshold | State |
|---|---|---|
| Comms loss | >2s (configurable) | RED → SURFACE |
| Battery low | <15% above return budget | AMBER |
| Battery critical | <5% above return budget | RED → SURFACE |
| Depth exceeded | Configurable per site | RED → SURFACE |
| IMU fault | Data gap >500ms | AMBER |
| DVL fault (Phase 7) | Data gap >1s | AMBER |

---

## 11. Infrastructure — SymbyTech Server

| Item | Value |
|---|---|
| Hardware | Dell PowerEdge R210 |
| OS | Ubuntu Server 24.04 LTS |
| Static IP | 192.168.1.175 (on eno2 — NOT eno1) |
| RAID Mount | /mnt/raid-data/ |
| Tailscale IP | 100.104.236.104 |
| CPU cores | 4 (host) |

> **WARNING:** eno2 is the primary NIC, not eno1. Never change this in netplan.

### Gitea

| Item | Value |
|---|---|
| Public URL | https://git.symbytech.com |
| SSH remote | `ssh://git@192.168.1.175:222/administrator/REPO.git` |
| ROV repo | `administrator/rov-autonomy` |
| Data location | /mnt/raid-data/forgejo/ |

> Use the server IP directly (`192.168.1.175:222`) — the hostname `symbytech` may not
> resolve on all networks. Do NOT use `git.symbytech.com` for SSH port 222.

### Harbor Registry

| Item | Value |
|---|---|
| Public URL | https://registry.symbytech.com |
| Local URL | http://192.168.1.175:8090 |
| Storage | /mnt/raid-data/harbor/ |

### Backup System (built 30 Jun 2026)

Push model — each Pi self-backs-up via root cron, pushing to server over SSH.

| Device | Script | Schedule | Log |
|---|---|---|---|
| RPi4 | `/usr/local/bin/rov-brain-pi4-backup.sh` | 00:00 daily | `/var/log/rov-brain-pi4-backup.log` |
| RPi5 | `/usr/local/bin/rov-brain-pi5-backup.sh` | 00:30 daily (staggered) | `/var/log/rov-brain-pi5-backup.log` |

Server monitor (`/usr/local/bin/symbytech-monitor.sh` → `check_rov_backup`) reads
`*/backup-status.txt` under `/mnt/raid-data/backup/`. Alerts: FAILED (always),
OVERDUE ≥2 days (throttled to once/7 days per device, regardless of online/offline).

RPi5 backup images `/dev/mmcblk0` (SD card OS) and rsyncs `/data` (NVMe data).
**RPi5 boots from SD — `/dev/mmcblk0` is the OS drive. NVMe is data only.**

### Development VM — DECOMMISSIONED

The BlueOS KVM VM (`blueos-vm`, NAT IP 192.168.122.89, Tailscale 100.84.141.120) was
used during early development before real RPi4/RPi5 hardware was operational. It is
no longer active (`virsh list --all` returns empty). The NPM proxy rule
`arov.symbytech.com → 192.168.122.89:80` is a leftover dead rule with no active target.
All `[VM]` commands in this document are historical only — do not use.

---

## 12. Phase Task Status

### Phase 1 — Foundation

| Task | Status |
|---|---|
| Flash BlueOS onto RPi4 | DONE — BlueOS 1.4.3, static IP 192.168.1.100 |
| Install Ubuntu 24.04 on RPi5 | DONE — Ubuntu 24.04.4, hostname rov-brain, IP 192.168.1.101 |
| Install ROS2 Jazzy on RPi5 | DONE — Jazzy Jalisco, auto-sourced in .bashrc |
| Confirm network comms between Pis | DONE — MAVROS `/mavros/state connected:true` against Pixhawk |
| Install Docker on RPi5 | DONE — Docker 29.5.3, no sudo required |
| Install MAVROS | DONE — running in argonaut.service |
| Install Tailscale on RPi5 | DONE — installed; service stopped and disabled |
| Install Tailscale on RPi4 | DONE — Tailscale IP 100.100.119.125 |
| Harbor project and user setup | DONE |
| Connect Xsens IMU | PENDING — CA-USB-MTI cable on order |
| foxglove_bridge install on RPi5 | DONE — ros-jazzy-foxglove-bridge, :8765 confirmed |
| Create /data/bags on NVMe | DONE — recording live ~2.5 MB/s |
| Deploy ROS2 workspace from Gitea | DONE — all 7 packages built, argonaut.service running |
| Automated backup system | DONE — both Pis → server, monitoring active |
| BlueOS FD exhaustion fix | ✗ **NOT DONE — root cause was wrong.** `nofile=65536` is a ceiling raise. Real cause: `/dev/i2c-1` FD leak in ardupilot_manager board detector. **OPEN** — see §15a |
| Pixhawk 2.4.8 as active FC | DONE — confirmed ttyACM0, MAVROS connected:true |

### Phase 2 — Widgets and Data Path

| Task | Status |
|---|---|
| Confirm widget approach | DONE — Cockpit DIY widgets, JSON format |
| Confirm Cockpit version required | DONE — native desktop v1.18.2 |
| Establish CSS rules for DIY widgets | DONE |
| W0 Data Lake Inspector | DONE — confirmed working (706 vars live) |
| W1 System Health Indicator | DONE — visual confirmed |
| W2 Mission Status | DONE — visual confirmed; full test pending |
| W3 Return To Safe Button | DONE (UI); label rename pending; backend not built |
| W4 Mission Setup Button | DONE — visual confirmed; full test pending |
| W5 Return Budget | DONE — visual confirmed; full test pending |
| W6 Altitude Widget | DONE — NO SONAR state confirmed (hardware-gated) |
| W7 Backup Due Notice | NOT STARTED — depends on FastAPI backend |
| All W0–W6 committed to Gitea | W0–W5: committed. W6 widget JSON: pending re-import from laptop |
| cockpit_bridge node | DONE — 8 variables, commits 5d75a9d + 0640353 |
| Probe widget | DELETED — dev tool only, not part of W0–W7 |

### Phase 3 — Recording and Mission Control (Active)

| Task | Status |
|---|---|
| Recording Manager (DIR-9) | DONE — Stage 1 complete, commit 5d75a9d |
| FastAPI backend core (RPi5:8081) | DONE — Stage 2 core, commit 66a18ef, verified end-to-end |
| Deploy files version-controlled | DONE — deploy/ folder, commit 97decac |
| Hostname/DNS resolution fix | DONE — laptop + RPi5 hosts entries, git remotes on hostname |
| Widget housekeeping (W6 add, probe remove) | DONE — commit 785ec56 |
| W3 backend `/abort` POST end-to-end | DONE — verified: FSM NORMAL→HOLD_AND_RECOVER |
| W3 label update to "RETURN TO SAFE" | PENDING — widget code + FASTAPI_HOST constant |
| W5 return budget publishing | DEFERRED — Stage 2b (return budget approach TBD) |
| W7 Backup Due Notice + /backup/* | DEFERRED — Stage 2b (Option A design) |
| /mission/upload endpoint | DEFERRED — Stage 2b |
| Ping2 sonar purchase (W6 full test) | PENDING — Phase 3, ~$280 USD |

### Next Priorities
See §0 QUICK STATUS BOARD — NEXT (moved there 11 Jul 2026 restructure to avoid two lists
drifting apart).

---

## 13. Session Lessons Learned — Data Injection Investigation (Task 11)

This section documents what was learned during the Task 11 data injection investigation
so it is not repeated. These are permanent historical records.

### What was attempted and why it failed

| Approach | Why it failed |
|---|---|
| UDP to VM port 14550 from server host | KVM NAT silently drops packets |
| UDP to VM port 14001 from server | Same KVM NAT issue |
| HTTP POST to mavlink2rest /mavlink | Stored in REST, not broadcast via WebSocket — not in data lake |
| UDP with sysid=255, compid=190 | mavlink2rest only processes sysid 1 messages for data lake |
| UDP with sysid=1, compid=1 from Python | Caused ArduSub to detect heartbeat conflict — do not impersonate compid=1 |

### What works

| Approach | Result |
|---|---|
| Generic WebSocket (Cockpit v1.18.0+) | Confirmed working. WebSocket server on RPi5, Cockpit connects as client. Variables appear in data lake as `external/{name}` immediately. **This is the correct and supported path.** |
| Native ArduSub NAMED_VALUE messages | Confirmed working via W0 — flow: Pixhawk → ttyACM0 → mavlink-routerd → mavlink2rest UDP → mavlink2rest REST + WebSocket → Cockpit. |

### Key architectural insight
The ONLY path that gets external data into the Cockpit data lake is:
1. Messages from the Pixhawk via ttyACM0 → mavlink-routerd → mavlink2rest — for production ArduSub/MAVROS data
2. Generic WebSocket connections (Cockpit v1.18.0+) — for custom external data via `cockpit_bridge`

---

## 14. Roadmap Summary

| Phase | Title | Status |
|---|---|---|
| 1 | Foundation | COMPLETE |
| 2 | Widgets and Data Path | COMPLETE (W6 deployed; W7 not started; backend pending) |
| 3 | Recording and Mission Control | IN PROGRESS — Stage 1 complete |
| 4 | Surface Following and Pattern Flight | Not started |
| 5 | Perception and Feature Detection | Not started |
| 6 | Full Autonomy + Mission Planner | Not started |
| 7 | Field Trials and Hardware Evolution | DVL integration |
| 8 | Commercial Product | Not started |

---

## 15. Key Architecture Decisions (Locked)

| Decision | Choice | Rationale |
|---|---|---|
| Autonomy compute | Vehicle-side RPi5 | Lowest latency to sensors and thrusters |
| RPi4 role | Thin gateway only | Separation of concerns |
| OS on RPi5 | Ubuntu 24.04 LTS | Only clean ROS2 Jazzy support |
| RPi5 storage | OS-on-SD, data-on-NVMe | Physical separation; SD hardened for low writes |
| ROS2 version | Jazzy Jalisco | Current LTS |
| Container registry | Harbor (self-hosted) | No rate limits, private |
| Cockpit version | Native desktop v1.18.2 | Browser extension is Lite; v1.18.2 required for DIY widgets |
| Widget approach | Cockpit DIY widgets (JSON format) | Confirmed working, version-controlled in Gitea |
| MAVLink router | MAVLinkRouter | MAVP2P has known issues per BlueOS own warning |
| Flight controller | Pixhawk 2.4.8 (real hardware) | SITL retired from production |
| Data injection | Generic WebSocket via cockpit_bridge | Confirmed working; MAVLink injection paths do not populate data lake |
| WebSocket variable naming | Use hyphens not underscores | Cockpit normalises underscores to hyphens in data lake IDs |
| Recording format | MCAP (Foxglove native) | Seekable replay, no conversion needed |
| Recording control | recording_manager ROS2 node | DIR-9a (governing) — single ARM-bounded black-box tier, keyed to `recording_control` flag, not mission state. `recording_manager.py` implementation still lags on superseded DIR-9 (continuous-from-boot) — known drift, open. |
| foxglove_bridge role | Visualisation ONLY | Does NOT record MCAP (DIR-9a) |
| ARM64 builds | Laptop only | Server overheats under compilation load |
| Acoustic positioning | Excluded | Unreliable near steel |
| DVL | Deferred to Phase 7 | Budget constraint — highest priority future sensor |

---

## 15a. ardupilot-manager FD Exhaustion — **OPEN BUG** (root cause corrected 9 Jul 2026)

> ⚠ **This section replaces the previous "RESOLVED — Permanent Fix Applied" entry, which was
> wrong on both diagnosis and fix.** The stale root cause survived from v2.10 to v2.21 and
> actively prevented this being solved, because it made every recurrence look inexplicable.

### What the old entry claimed (WRONG — do not reinstate)
- ✗ "Root cause: accumulated TCP socket FDs from continuous UI polling exhausting `nofile=1024`."
- ✗ "Permanently fixed via `/etc/docker/daemon.json` `nofile: 65536`."
- ✗ (v2.10) "re-diagnosed as uvicorn FD exhaustion."

None of these are correct. The leaked FDs are not sockets. `nofile: 65536` fixed nothing — it
raised the ceiling, converting a failure at ~10 minutes into a failure at ~11 hours, which
disguised a hard leak as an intermittent annoyance.

### Actual root cause (measured 9 Jul 2026)

`ardupilot_manager` leaks **`/dev/i2c-1` file descriptors**, one per failed Linux-board probe.

Mechanism, end to end:
1. The BlueOS **Autopilot Firmware / board-config page** polls `GET /v1.0/available_boards`.
2. Each request triggers **20** `flight_controller_detector.linux.detector:detect_boards` probes
   — for `Navigator`, `NavigatorPi4`, `NavigatorPi5`, and `Argonot`.
3. Each Linux-board probe opens `/dev/i2c-1` and **never closes it when detection fails**.
4. We run a **Pixhawk 2.4.8 on USB serial** (`PlatformType.Serial`), not a Navigator. Every
   probe fails. Every probe leaks.
5. At 65536 FDs, `ardupilot_manager` can no longer open new handles. Everything requiring a
   fresh handle fails: `AUTOPILOT_VEHICLE_TYPE_FETCH_FAIL`, `AUTOPILOT_ENDPOINT_FETCH_FAIL`,
   `BAG_OF_HOLDING_GET_DATA_FAIL`, **"No board running"**.
6. Restarting the core **recreates the `blueos-core` container with a fresh FD table** — which
   is why a core restart has always appeared to "fix" it.

### Evidence (controlled measurement, RPi4, 9 Jul 2026)

| Time | Board-config page | FD count on PID 122 (`ardupilot_manager/main.py`) |
|---|---|---|
| 07:51 → 08:56 (65 min) | CLOSED | 4630 → 4630 — **absolutely flat** |
| 09:03:35 → 09:04:24 (50 s) | **OPEN** | 4630 → 4720 — **+90 FDs (~108/min)** |
| 09:04:24 → 09:04:47 | CLOSED | 4720 → 4720 — **flat immediately** |

FD composition at 2740 total: **2740 × `/dev/i2c-1`**, 3 sockets, 2 pipes. Not a socket leak.
Probe-to-request ratio confirmed from logs: 8920 `detect_boards` ÷ 445 `available_boards` = **20:1**.

Cause and effect: isolated, reproducible, reversible.

### Critical property — the count does NOT reset at launch
FD count accumulates **across sessions** and only resets when `blueos-core` is recreated (core
restart / reboot). **Cumulative page-open time**, not wall-clock time, drives time-to-failure
(~11 h at ~100 FD/min). Leave that page open two hours a day and the wall is hit inside a week
without a single long session.

### Why this matters for field operations
The exposure is **not** leakage during a dive (topside is disconnected; nothing polls). The
exposure is **launching with a nearly-exhausted FD table** after pre-dive setup — precisely when
the board-config page is most likely to have been open for hours. Mid-dive, with no operator,
`ardupilot_manager` then fails on the next handle it needs.

### Mitigation

**Dev (effective immediately, zero code):**
Do not leave the BlueOS **Autopilot Firmware / board-config** page open. Open it, do the job,
close it. Other BlueOS pages and Cockpit are unaffected and do not leak.

**Field — NOT YET IMPLEMENTED (both are open items):**
- **(a)** FD-count check on `ardupilot_manager` as part of the AUV pre-dive gate, blocking launch
  above a threshold. Operator discipline is not a field-safe control.
- **(b)** `failsafe_monitor` must detect FC comms loss so W1 goes RED when the board is
  unreachable — see §15d.

**Do NOT patch BlueOS or ArduPilot.** Any upstream update reverts local changes. This is an
upstream BlueOS 1.4.3 defect; the correct route is an upstream report. All countermeasures must
live on our side of the boundary.

**The `nofile: 65536` setting in `/etc/docker/daemon.json` remains in place** and is harmless —
but it is a ceiling raise, **not a fix**, and must never again be described as one.

---

## 15d. Vehicle health in the failsafe assessment — **RESOLVED 11 Jul 2026**

> Previously titled "Vehicle health is NOT in the failsafe assessment — OPEN, SAFETY-CRITICAL."
> Discovered 9 Jul 2026 (below), fixed 11 Jul 2026 via four commits. Corrected root cause of a
> second, related fault (`cockpit_bridge` publishing the wrong variable) found and fixed in the
> same session — see the last commit below.

### Original fault (9 Jul 2026)

Discovered while the flight controller was unreachable ("No board running"):

**W1 showed GREEN / "Systems nominal".**

Every widget indicator reflected the **RPi5 autonomy stack**, not the vehicle:
- **W1** read `external/rov-failsafe` from `failsafe_monitor` — which did not gate on FC
  connectivity.
- **W8** reads `external/rov-mode`, sourced from a YAML file. It would show `AUV` with the
  Pixhawk in a drawer.
- **Foxglove "Connected: true"** means connected to `foxglove_bridge` on the RPi5 — not to
  the vehicle.

**Why this was critical:** in AUV mode, `health_role: gate` makes W1 the pre-dive GO/NO-GO
gate. A gate that cannot see the flight controller is not a gate — it would pass a dive with
a dead FC.

### Fix (11 Jul 2026) — four commits, verified live end-to-end

1. **`86b2728`** — comms loss now forces **unconditional RED** (previously scored as one
   marginal parameter among several, only reaching AMBER). Added an `fcu_ever_connected`
   latch so the startup window before any heartbeat has ever been received is not reported
   GREEN by default. Cause text (`"Heartbeat Lost"`, `"Vehicle Disconnected"`) added to
   `FailsafeStatus.message`, per DIR-7's "state must carry its cause" MUST.
2. **`1b0050d`** — FSM recovery fix: `RETURN_TO_SAFE` was a terminal trap (only
   `HOLD_AND_RECOVER` could transition back to `NORMAL`); both states now recover on
   condition-clear. (Mode-blind — see DIR-7's 11 Jul in-mission-recovery addendum for the
   still-open mode-dependent design intent this mechanism needs to grow into.)
3. **`dbe9515`** — `cockpit_bridge` publishes a new data lake variable
   `external/rov-failsafe-cause` (substring of `message` before the first `|`; empty when
   nominal). Stopgap per DIR-7's cause MUST — the DIR's Parked Design Items records the
   proper fix (a dedicated `string cause` field on `FailsafeStatus.msg`).
4. **`3409ae5`** — W1 widget shows the state word on `#w1-message` (state-driven, no
   flicker) and cause text on `#w1-footer` (red, 11px, blank when nominal).

**Verified live:** kill mavros → RED + "Heartbeat Lost"; restart mavros → recovers to GREEN.

### Related fault found and fixed in the same session

`cockpit_bridge` was publishing `failsafe_state` (the FSM state, which can be 3 or higher)
to `external/rov-failsafe`, not `assessment_state` (0/1/2) as W1 expects — fixed in commit
`d854641` (sibling of `dbe9515`). This was the **actual cause** of that day's "widgets show
no data" symptom (a value of 3 falling outside W1's expected 0–2 range), compounded by a
restart-storm that wedged port 9001 while diagnosing it (see CLAUDE.md's
one-restart-then-verify-9001 rule, added as a direct result).

### Still open (not resolved by this fix — see DIR-7 addenda and Parked Design Items)
- ~~Mode-dependent in-mission recovery (AUV commit-to-recovery vs ROV operator handoff)~~ —
  **ESSENTIALLY COMPLETE 13 Jul 2026, see §15f.** Remaining: the arrival-event publisher
  (temporary operator button, then real navigation).
- Auto-recovery (failsafe_monitor re-establishing the FC link itself)
- In-mission AMBER runtime-response, branched by mode
- Dedicated `string cause` field on `FailsafeStatus.msg` (current fix is a substring stopgap)
- FD-headroom check on `ardupilot_manager` as part of the AUV pre-dive gate (§15a mitigation
  (a), still not built)

---

## 15f. Mode system launch-wiring gap + gate-mode recovery arrival event — 13 Jul 2026

### Fault 1 — mode_profile_loader never wired into the stack launch file

`mode_profile_loader` (`rov_mission`) was built and committed 7 Jul 2026 (`4e48dc2`) — node,
`ModeProfile.msg`, `mode_profiles.yaml`, `cockpit_bridge` flag surfacing, all verified working
in isolation at the time. It was never added to `rov_full.launch.py`. Confirmed on RPi5, 12 Jul
2026: `ros2 node list` showed no mode node; `ros2 topic info /rov/mode/profile` reported
`Publisher count: 0`, 2 subscribers waiting. The entire operating-mode profile system had been
dark at runtime — zero publishers — for the six days since it was built. `failsafe_monitor`'s
`health_role` had been silently running on its gate default the whole time, not the actually
configured mode.

**Fix (`464e17e`):** added the `Node` action for `mode_profile_loader` to `rov_full.launch.py`,
placed with the other core nodes (before Perception) rather than down in the mission-package
include further below, so consumers of its latched output (`failsafe_monitor`, `cockpit_bridge`)
wait as little as possible on a publisher. `respawn=True`/`respawn_delay=5.0`, matching the other
core nodes (`c099c00`, `a5e1c99`) — if this node dies the whole mode system goes dark again.
Verified live: publisher count 0 → 1.

### Fault 2 — gate-mode RETURN_TO_SAFE was a terminal trap

With the mode system now actually publishing a real gate profile, a second gap surfaced.
`failsafe_monitor`'s mode-aware recovery (`2a3e577`, same session) correctly commits gate-mode
(AUV) `HOLD_AND_RECOVER` → `RETURN_TO_SAFE` on condition-clear per the DIR-7 addendum — but then
never left `RETURN_TO_SAFE`. That branch deliberately ignores further condition-clear (recovery,
once committed, must run to completion — see the addendum) but nothing had ever been wired to
signal that the recovery HAD completed.

**Fix (`bf815cc`):** new subscription, `/rov/nav/safe_zone_reached` (std_msgs/Bool), same plain
QoS and latch-on-True callback pattern as the existing `flag_manual_abort` subscription. When
`health_role == gate` and `fsm_state == RETURN_TO_SAFE` and a `True` is received, the FSM
transitions to `NORMAL` and the cause text reads "Safe zone reached — recovery complete" (DIR-7's
cause-text MUST, via the existing message-prefix mechanism — no `FailsafeStatus.msg` change, no
`cockpit_bridge`/W1 change needed). Any other combination (not in `RETURN_TO_SAFE`, or advisory
mode) is ignored — advisory mode keeps its separate, untouched operator-resume mechanism.

**This topic and this subscription are the PERMANENT nav-arrival interface** — the navigation
subsystem will publish the real event here (GPS at surface, EKF dead-reckoning underwater; see
the new Parked Design Item in `docs/design_intent_register.md` for the full sensor-derivation
detail). Only today's PUBLISHER is temporary: verified live end-to-end on the bench by publishing
the event by hand and watching the FSM release `RETURN_TO_SAFE` → `NORMAL` and the
`motion_controller` recovery broadcast stop. The permanent replacement publisher — a temporary
W1-contextual operator button (bench scaffolding, `rov_api`-backed) ahead of navigation raising
the event for real — is next (see §0 NEXT).

### Pre-existing defects found this session (not caused by this session's work)

**`flag_manual_abort` is a one-way latch, never reset.** Set `True` in `_abort_callback` on any
`/rov/mission/abort` message; nothing anywhere in `failsafe_monitor` ever sets it back to
`False`. Once any manual abort fires, Priority 6 (`if self.flag_manual_abort: ...; return`)
returns early on every subsequent evaluation cycle for the rest of the process lifetime —
permanently shadowing every lower-priority handler, including the new mode-aware recovery and
arrival-event logic added this session (neither is reachable once this latch is set). Discovered
during this session's work on `2a3e577`; the bug itself predates it. Needs a reset condition —
candidates: on reaching `NORMAL`, on an explicit operator reset, or only on process restart —
undecided, needs its own session. **Field-deployment concern:** a single manual abort currently
disables all future automatic recovery for the rest of that dive.

**`failsafe_monitor` produces no log output at all.** Silent in `journalctl` this session — no
info/warn lines observed for state transitions or startup. Its state had to be read directly off
`/rov/failsafe` (`ros2 topic echo`) every time rather than watched in the service log. Minor, but
cost real diagnostic time this session. Should log state transitions and cause on each one.

---

### MAVLinkRouter vs MAVP2P
BlueOS itself warns "MAVP2P has been presenting issues and is not currently recommended."
MAVLinkRouter is the confirmed stable router. Do not recommend or use MAVP2P.

**Symptoms (for reference — should not recur):**
- Frontend connected/disconnected cycling
- "No board running" / "No endpoints available" in BlueOS UI
- `AUTOPILOT_*_FETCH_FAIL` (timeout 30000ms) in notifications
- `OSError: [Errno 24] Too many open files` in ardupilot-manager logs

---

## 15g. DIR-7 recovery-intent correction, W1 button paused, and a build-workspace footgun — 13 Jul 2026 (later same session)

### DIR-7 in-mission-recovery intent corrected — condition-clear entry trigger was inverted

Later the same day as §15f, the DIR-7 "In-mission recovery is mode-dependent" addendum was
found to be internally contradictory as implemented. `2a3e577`'s gate-mode entry trigger —
commit to `RETURN_TO_SAFE` ON CONDITION-CLEAR — produced a live state that should not be
possible: **W1 showing GREEN ("Systems nominal") while the vehicle was simultaneously in
`RETURN_TO_SAFE` ("Recovering to safe zone")** — a working-state light next to an
active-recovery action.

**Corrected intent** (full detail: `docs/design_intent_register.md` v1.4, DIR-7 addendum,
commit `255096f` — pointer only here, not duplicated):
- **GREEN is unconditionally a working state.** A fault that clears returns the vehicle to
  GREEN and the mission continues or resumes. A cleared condition must **never** trigger or
  commit a recovery.
- Committed recovery is triggered by a fault **persisting or recurring**, never by it clearing.
- **Three fault tiers:** (1) **uncontrollable** — power loss or anything that disables the
  safety logic itself, out of scope for the FSM entirely, adjacent to the shelved DIR-6
  Emergency Surface concern; (2) **transient/self-resetting** — e.g. a comms blip or sensor
  re-sync; the system attempts restore, returns to GREEN and resumes on success, and only
  commits to `RETURN_TO_SAFE` once a **raw, config-driven failure count** (no time window, no
  fixed duration) is exceeded; (3) **hard/persistent** — a RED that does not clear, commits to
  `RETURN_TO_SAFE` immediately.
- All thresholds (the acceptable failure count, and which specific faults are transient vs
  hard) are **configuration-driven**, set in the setup wizard and read from the resolved mode
  profile the same way `health_role` is — never hardcoded.

**What this means for the code right now:**
- **KNOWN-WRONG, pending rework (top-priority next-session task):** `2a3e577`'s
  condition-clear-commits-to-recovery entry logic (the `health_role == HEALTH_GATE` /
  `HOLD_AND_RECOVER` branch in `failsafe_monitor._apply_failsafe_priority`'s "all clear"
  block). This is what produces the contradictory GREEN-while-recovering state above, and must
  be reworked into the count-based tier model.
- **Survives, confirmed correct, no rework needed:** `failsafe_monitor` subscribing to
  `/rov/mode/profile` and reading `health_role` (`2a3e577`'s mode-awareness mechanism) — this
  is the right way to know which mode's rules apply, and is unaffected by the correction. Also
  survives: the safe-zone-reached arrival EXIT mechanism and its permanent
  `/rov/nav/safe_zone_reached` topic (`bf815cc`) — it still correctly completes a **committed**
  `RETURN_TO_SAFE` recovery on arrival. Only the trigger for **entering** committed recovery
  was wrong; how a genuinely-committed recovery **exits** is untouched and correct.

**§0 Confirmed Working, PARKED, and NEXT updated accordingly** — mode-aware recovery is no
longer described as "essentially complete"; the FSM entry-trigger rework is now the top NEXT
priority, ahead of the W1 button (which is paused, not the immediate next step — see below).

### W1 safe-zone-reached button — PAUSED, not abandoned

The supporting infrastructure for the temporary W1-contextual operator button is built and
committed: the `rov_api` endpoint `POST /nav/safe_zone_reached` (mirrors the existing `/abort`
pattern exactly) and the new `external/rov-failsafe-state` data lake variable it depends on
(surfaces `FailsafeStatus.failsafe_state`, distinct from the existing `external/rov-failsafe`
assessment integer), both in commit `2def2b1`. A pre-edit backup of the W1 widget was made per
CREATE-THEN-BACKUP: `widgets/w1_system_health_indicator.json.2026-07-13.bak`.

**The W1 widget edit itself was deliberately not started.** The button would surface FSM state
(specifically, `rov-failsafe-state == 3` / `RETURN_TO_SAFE`) that the DIR-7 correction above
just established is reached via a known-wrong trigger — the button would be meaningless,
possibly misleading, until the FSM correctly represents recovery state per the corrected
model. This work is paused, waiting on the FSM rework above, not abandoned or deprioritised
for any other reason.

### Stray nested colcon workspace — recurring rov_mission build failures root-caused

A recurring build failure (recurred 2-3 times across sessions) was root-caused this session:
stray nested `build/`, `install/`, and `log/` directories existed inside `src/rov-autonomy` on
RPi5 — created at some point when `colcon build` was run from inside the repo source directory
instead of the actual workspace root, `/data/ros2_ws`. These nested directories collided with
the real workspace, and caused `rov_mission` builds to fail resolving `mode_profiles.yaml`
through a stale nested `install/` tree rather than the real one.

**Fix:** the stray nested `build/`, `install/`, and `log/` directories were removed from RPi5.
They are gitignored and untracked, so this was a **local filesystem cleanup only — no git
change**, and does not appear in any commit.

**Durable guard:** `colcon build` must **only ever** be run from `/data/ros2_ws` (the actual
workspace root) — **never** from inside `src/rov-autonomy` (the repo source checkout within
it). Recorded here so this is not reintroduced.

---

## 15b. Session 6 Lessons Learned — RPi5 Base Stack Install

### SSH on Ubuntu 24.04 Server (cloud-init image)
Ubuntu 24.04 preinstalled server image disables password authentication by default. To enable persistently:
```bash
echo "PasswordAuthentication yes" | sudo tee /etc/ssh/sshd_config.d/99-password-auth.conf
sudo systemctl restart ssh
```
**Do NOT add KbdInteractiveAuthentication** — removed in OpenSSH 9.x, causes sshd to fail on Ubuntu 24.04.

### NVMe clone via dd — PARTUUID handling
`dd` copies the full disk bit-for-bit including MBR disk identifier, resulting in identical PARTUUIDs on both SD and NVMe. Fix with fdisk. However Ubuntu 24.04 preinstalled image uses `LABEL=` in both `cmdline.txt` and `fstab`, so the duplicate PARTUUID issue does not cause boot failure on this image.

### rpi-clone does not support NVMe partition naming
`rpi-clone` fails on NVMe devices (expects `sda1` style, not `nvme0n1p1`). Use `dd` for full disk clone.

### initramfs rebuild required after NVMe clone on Ubuntu
After cloning to NVMe, rebuild initramfs before booting:
```bash
sudo chroot /mnt/nvme-root update-initramfs -u -k all
```

### Ethernet cable quality matters
Two cables failed silently (no LEDs, interface DOWN) before a third worked. Always test with a known-good cable before debugging the OS.

---

## 15c. Session 7 Lessons Learned — Workspace Deploy & First Full Launch

### The committed Gitea workspace was never build-tested — three latent bugs

1. **Missing `config/` dir in rov_bringup** (commit f1c6710)
2. **Missing `setup.cfg` in all 5 Python packages** (commit 7590faf) — without them, setuptools installs executables to `bin/` instead of `lib/<pkg>/`, so `ros2 launch` / `ros2 run` cannot find them.
3. **ekf.yaml mixed int/float** (commit 45b3301) — ROS2 YAML parser requires uniform sequence types.

**Process lesson:** future handovers must distinguish "written" from "built and run".

### colcon: `--symlink-install` works correctly with setup.cfg files in place
With `setup.cfg` present in each Python package, `--symlink-install` correctly places
executables in `lib/<pkg>/`. This is now the standard build command.

### MAVROS full-launch fixes (commit c94412d) — reusable knowledge
- **fcu_url format:** `udp://@<remote_ip>:14550` is correct (bind any local, connect remote).
- **MAVROS plugin collisions on Jazzy:** fix with `plugin_allowlist`: `['sys_status','sys_time','command','imu','global_position','setpoint_velocity']`.
- **QoS on MAVROS topics:** use `qos_profile_sensor_data` (BEST_EFFORT) on MAVROS-sourced subs.
- **SITL/real hardware battery:** treat `battery_percent < 0` as UNKNOWN, not critically low.
- **Rebuild discipline:** `colcon build --packages-select <pkg>` required after editing installed launch/config files.

---

## 15e. Session — RPi5 Storage Re-Architecture

**RPi5 boot facts:**
- BOOT_ORDER is a hex value in EEPROM (nibbles, read right-to-left: 1=SD, 6=NVMe, 4=USB, f=restart-loop).
- FIX for wrong-root boot: change `/boot/firmware/cmdline.txt` and `/etc/fstab` from `LABEL=` to `PARTUUID=` (PARTUUIDs differ between drives; labels collided).
- End-state BOOT_ORDER: `0xf41` (SD-first). NVMe is data-only — nothing to compete.

**SD hardening:**
- `log2ram` — `/var/log` is a RAM disk (128M).
- `/tmp` → tmpfs (256M) via fstab.
- journald capped: `SystemMaxUse=20M` in `/etc/systemd/journald.conf`.

**systemd service:**
- `/etc/argonaut/rov.env` — EnvironmentFile: `ENV=dev`, `BLUEOS_IP=192.168.1.100`.
- `/usr/local/bin/argonaut-stack.sh` — wrapper: sources ROS2 setup, execs launch.
- `/etc/systemd/system/argonaut.service` — `User=ubuntu`, `EnvironmentFile`, `Restart=on-failure`, `RestartSec=5`.
- **DEV POLICY: installed but NOT enabled-on-boot.** Start manually: `sudo systemctl start argonaut`. Field: `sudo systemctl enable argonaut`.

---

## 16. Known Gaps and Risks

| Gap | Notes | Priority |
|---|---|---|
| DVL absent | X/Y position dead-reckoned from IMU — drifts | High — Phase 7 |
| Ping2 sonar not purchased | W6 full test blocked; downward altitude not flowing | Phase 3 (~$280 USD) |
| FastAPI backend not built | W3 abort, W4 setup, W5 budget, W7 backup all blocked | High — Phase 3 Stage 2 |
| W3 button label not updated | Widget still shows "ABORT MISSION"; should be "RETURN TO SAFE" | Next session |
| ~~Widget JSON not in Gitea~~ | RESOLVED — W6 committed as `w6_altitude.json`, probe widget removed (`785ec56`), see §12 | Closed |
| RPi5 weak SSH password | Password auth enabled, Tailscale service disabled but installed | Pre-field |
| RPi5 thermal in sealed enclosure | Conduction cooling path unresolved | Phase 3 |
| ML inference compute | RPi5 has no GPU. ONNX/TFLite on-device or Jetson Orin Nano. | Phase 5 |
| Sim/ML dev compute | Track 2 (perception) needs RTX GPU dev PC or cloud. Track 1 (Gazebo) runs on laptop. | Decision pending |
| Jacket training data | No public dataset. Path = synthetic-from-CAD + SubPipe/COU + field captures. | Phase 5+ |
| PCIe Gen 3 (RPi5 NVMe) | `dtparam=pciex1_gen=3` → ~2× throughput but UNCERTIFIED. Benefit nil until camera→MCAP. | Phase 3 revisit |
| Ops Server separation | Currently same machine as Dev Server | Near-term |
| Model conversion pipeline | Manual for now | Phase 6 |
| Widget consolidation | Grant to review — may consolidate W0–W7 into single unified widget | Future |

---

## 17. Shopping List

| Item | Supplier | Approx Cost | Priority |
|---|---|---|---|
| CA-USB-MTI (Xsens cable) | Newark Electronics | — | High — long lead time |
| Ping2 Sonar (downward) | Blue Robotics | ~$280 USD | Phase 3 — unblocks W6 full test |
| Ops Server (cloud VPS or local machine) | TBD | — | Medium — before field ops |

---

## 18. Reference Commands

> Terminal label convention and shell prompt mapping:
> `[RPi4]` → `pi@blueos:~$` (192.168.1.100)
> `[RPi5]` → `ubuntu@rov-brain:~$` (192.168.1.101)
> `[SERVER]` → `grant@symbytech:~$` (192.168.1.175)
> `[LAPTOP — Git Bash]` → `grant@BB8 MINGW64 ...`
> Label appears as plain text ABOVE the code block, never inside it.

### RPi4 — SSH Access

[LAPTOP — Git Bash]
```bash
ssh pi@192.168.1.100
```

### RPi4 — BlueOS UI

[BROWSER]
```
http://192.168.1.100
```

### RPi4 — BlueOS Container

[RPi4]
```bash
sudo docker ps --format "table {{.Names}}\t{{.Status}}"
```

[RPi4]
```bash
sudo docker restart blueos-core
```

[RPi4]
```bash
sudo docker logs blueos-core --tail 30 2>&1
```

### RPi5 — SSH Access

[LAPTOP — Git Bash]
```bash
ssh ubuntu@192.168.1.101
```

### RPi5 — argonaut.service

[RPi5]
```bash
sudo systemctl start argonaut.service
```

[RPi5]
```bash
sudo systemctl status argonaut.service
```

[RPi5]
```bash
journalctl -u argonaut.service -f
```

### RPi5 — argonaut-api.service (FastAPI backend :8081)

[RPi5]
```bash
sudo systemctl start argonaut-api.service
```

[RPi5]
```bash
systemctl is-active argonaut-api.service && curl -s http://localhost:8081/health
```

[RPi5]
```bash
curl -s -X POST http://localhost:8081/abort
```

[RPi5]
```bash
curl -s -X POST http://localhost:8081/mission/start -H "Content-Type: application/json" -d '{"mission_id":"test01","parameters":[]}'
```

### RPi5 — ROS2 Stack Health

[RPi5]
```bash
source /data/ros2_ws/install/setup.bash && ros2 node list | wc -l
```

[RPi5]
```bash
ros2 topic echo /mavros/state --once
```

[RPi5]
```bash
ros2 topic echo /rov/recording/active --once
```

[RPi5]
```bash
ros2 topic echo /rov/failsafe --once
```

### RPi5 — Bags

[RPi5]
```bash
ls -lh /data/bags/
```

[RPi5]
```bash
du -sh /data/bags/*
```

### RPi5 — Build

[RPi5]
```bash
cd /data/ros2_ws && colcon build --packages-select rov_mission --symlink-install 2>&1 | tail -10
```

### Gitea

[LAPTOP — Git Bash]
```bash
cd "C:\Users\grant\Desktop\Autonomous ROV System\rov-autonomy"
```

[LAPTOP — Git Bash]
```bash
git pull origin master
```

[RPi5]
```bash
cd /data/ros2_ws/src/rov-autonomy && git pull origin master && git push origin master
```

### Server — General

[SERVER]
```bash
df -h && docker ps
```

[SERVER]
```bash
cd ~/harbor && sudo docker compose ps
```

### Server — Harbor

[SERVER]
```bash
docker login registry.symbytech.com
```

### Browser — Dev URLs

[BROWSER]
```
http://192.168.1.100           BlueOS (RPi4)
http://192.168.1.101:8765      Foxglove bridge WebSocket (connect via Foxglove Studio)
https://registry.symbytech.com Harbor
https://git.symbytech.com      Gitea
http://192.168.1.175:81        NPM admin
http://192.168.1.175:9000      Portainer
```

### Cockpit — Widget Import

[COCKPIT]
```
1. Connect to vehicle (192.168.1.100)
2. Settings -> General -> Enable Pirate Mode (required for DIY widgets, resets each session)
3. Enter edit mode (pencil icon, top right)
4. Add Widget at bottom — scroll right to find </> DIY widget
5. Drag onto main view area and resize
6. Click gear icon on widget -> Import -> select .json from rov-autonomy/widgets/
```

[COCKPIT]
```
Generic WebSocket (one-time setup, persists):
Settings -> Generic WebSocket Connections
Add: ws://192.168.1.101:9001
```

---

## 19. Document Index

All project docs live in the repo under `docs/` (git-tracked, no version number in the
filename — the `Version:` field inside each file is authoritative; project knowledge holds a
mirror copy for chat context, re-uploaded after any commit that changes these). Do not cite a
version number from memory — check the file itself.

| Document | Purpose |
|---|---|
| `docs/handover.md` | This document — master reference, single source of truth for current state |
| `CLAUDE.md` (repo root) | Rules for Claude — behaviour, conventions, safety, research requirements |
| `docs/design_intent_register.md` | Design Intent Register — authoritative design decisions, DIR-1…DIR-12 |
| `docs/failsafe_design.md` | Failsafe state machine and sensor roadmap |
| `docs/ui_design.md` | Cockpit widget specs, data paths, and build status |
| `docs/simulation_ml_plan.md` | Simulation + ML tooling: dev-PC tiers, cloud options, training assets |
| `docs/mission_recording_lifecycle.md` | Lifecycle research backing DIR-9/9a/10 |

Server/infra docs (SymbyTech server, Harbor registry) and task/shopping lists stay
project-knowledge-only — not part of the `rov-autonomy` repo `docs/` folder.

---

*Update this document at the end of each working session.*
*Increment version number and add rows to Version History and Changelog as appropriate.*
*Commit to the repo (`docs/handover.md`); re-upload to Claude project knowledge for chat context.*