docs: record corrected in-mission recovery intent, top-priority FSM rework, paused W1 button, build fix

v2.26, same-day continuation of v2.25:

- Records the DIR-7 in-mission-recovery intent correction just made in
  docs/design_intent_register.md v1.4 (255096f): the condition-clear-commits-
  to-recovery entry trigger built earlier today (2a3e577) is KNOWN-WRONG —
  it produced a contradictory live state (W1 GREEN while the vehicle sat in
  RETURN_TO_SAFE). Corrected model (pointer to the DIR, not duplicated):
  GREEN is unconditionally a working state; committed recovery is triggered
  by fault persistence/recurrence, never by condition-clear.

- Marks the FSM entry-trigger rework as the new top-priority NEXT item,
  ahead of everything previously listed. Confirmed Working, Open Items, and
  PARKED sections rewritten so nothing claims mode-aware recovery works
  correctly end to end — the mode-awareness mechanism (2a3e577) and the
  safe-zone-reached arrival exit (bf815cc) are both confirmed correct and
  unaffected; only entry into committed recovery was wrong.

- Records the W1 safe-zone-reached button as PAUSED, not abandoned: the
  rov_api endpoint and external/rov-failsafe-state variable are built and
  committed (2def2b1), the pre-edit W1 widget backup exists, but the widget
  edit itself was not started since the FSM state it would surface is
  currently known-wrong.

- Records a recurring rov_mission build failure, root-caused and fixed:
  stray nested build/install/log directories inside src/rov-autonomy
  (from colcon being run from the wrong directory) collided with the real
  workspace at /data/ros2_ws. Local filesystem cleanup only, no git change.
  Durable guard recorded: colcon build only ever from /data/ros2_ws.

New §15g narrative section; Recent Commits, Version History, and Changelog
all updated. Documentation only — no code, widget, or DIR file touched in
this commit. Not uploaded to Claude project knowledge; that remains a
manual step.
This commit is contained in:
Grant 2026-07-13 18:13:14 +02:00
parent 255096f20b
commit 406bad8fbe

View File

@ -1,6 +1,6 @@
# ROV Autonomous Inspection System — Project Handover # ROV Autonomous Inspection System — Project Handover
**Project Name:** Argonaut 3 **Project Name:** Argonaut 3
**Version:** 2.25 **Version:** 2.26
**Date:** 13 July 2026 **Date:** 13 July 2026
**File:** `docs/handover.md` (repo-tracked, no version number in filename — see Version History for versioning) **File:** `docs/handover.md` (repo-tracked, no version number in filename — see Version History for versioning)
@ -24,6 +24,16 @@ recovery (`2a3e577`) surfaced a second gap — gate-mode committed recovery ente
`RETURN_TO_SAFE` and could never leave it. Fixed with a new permanent arrival interface, `RETURN_TO_SAFE` and could never leave it. Fixed with a new permanent arrival interface,
verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 Jul entries. verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 Jul entries.
**CORRECTED later the same session (13 Jul 2026)** — the condition-clear-commits-to-recovery
entry trigger described immediately above (`2a3e577`) is **KNOWN-WRONG**, not merely
mode-aware-and-working. It produced a contradictory live state: W1 showing GREEN ("Systems
nominal") while the vehicle was simultaneously in `RETURN_TO_SAFE` ("Recovering to safe zone").
Corrected intent recorded in `docs/design_intent_register.md` v1.4 (`255096f`): committed
recovery must be triggered by fault PERSISTENCE or RECURRENCE, never by a condition clearing —
GREEN is unconditionally a working state. The safe-zone-reached arrival EXIT mechanism
(`bf815cc`) is unaffected and still correct; only the ENTRY trigger needs rework. This is the
**top-priority next-session task** — see §0 NEXT and §15g.
> Full narrative for each prior version is in Version History below — not repeated here. > Full narrative for each prior version is in Version History below — not repeated here.
--- ---
@ -54,19 +64,22 @@ verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 J
end-to-end: kill mavros → RED + "Heartbeat Lost", restart → recover to GREEN. Four commits end-to-end: kill mavros → RED + "Heartbeat Lost", restart → recover to GREEN. Four commits
`86b2728`/`1b0050d`/`dbe9515`/`3409ae5` — full detail in Changelog below. §15d (was OPEN, `86b2728`/`1b0050d`/`dbe9515`/`3409ae5` — full detail in Changelog below. §15d (was OPEN,
SAFETY-CRITICAL) is resolved. SAFETY-CRITICAL) is resolved.
- **Mode-aware failsafe recovery (13 Jul 2026)**`failsafe_monitor` now subscribes to - **Mode-aware failsafe recovery — PARTIALLY correct, entry trigger KNOWN-WRONG (13 Jul 2026,
`/rov/mode/profile` and branches condition-clear recovery on `health_role` (`2a3e577`): gate corrected later the same day).** `failsafe_monitor` correctly subscribes to
(AUV) commits `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` on condition-clear instead of returning to `/rov/mode/profile` and reads `health_role` (`2a3e577`) — this mechanism is confirmed working
NORMAL; advisory (ROV) holds and waits for an explicit operator resume (placeholder topic). and correct, unaffected by the correction below. The safe-zone-reached arrival EXIT
This surfaced a new gap once the mode system was actually live — gate-mode `RETURN_TO_SAFE` mechanism — a new PERMANENT interface, `/rov/nav/safe_zone_reached` (std_msgs/Bool), that
had no exit, a terminal trap. Closed by a new PERMANENT interface, transitions gate-mode `RETURN_TO_SAFE` to NORMAL on arrival (`bf815cc`) — is also confirmed
`/rov/nav/safe_zone_reached` (std_msgs/Bool), that transitions the FSM to NORMAL on arrival working and correct, verified live end-to-end on the bench (published the event by hand,
(`bf815cc`). Verified live end-to-end on the bench: published the event by hand, watched the watched the FSM release `RETURN_TO_SAFE`→NORMAL and the `motion_controller` recovery
FSM release `RETURN_TO_SAFE`→NORMAL and the `motion_controller` recovery broadcast stop. The broadcast stop). **However, the ENTRY trigger built the same day (`2a3e577`: gate commits
topic/subscription are permanent; today's publisher is a manual bench command only — the `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` ON CONDITION-CLEAR) is KNOWN-WRONG** — corrected in
real navigation-raised event and a temporary W1-contextual operator button are both still to `docs/design_intent_register.md` v1.4 (`255096f`) after it produced a contradictory live
come. See §15f, DIR-7 addendum, and the new Parked Design Item in state (W1 GREEN "Systems nominal" while the vehicle sat in `RETURN_TO_SAFE` "Recovering to
`docs/design_intent_register.md`. safe zone"). Corrected intent: committed recovery must be triggered by fault PERSISTENCE or
RECURRENCE, never by condition-clear — GREEN is unconditionally a working state. Rework to
the count-based model is the **top-priority next-session task** — see §0 NEXT, §15g, and the
DIR-7 addendum in `docs/design_intent_register.md`.
- W6 Altitude Widget — deployed. NO SONAR state confirmed (expected — no Ping2 connected). - W6 Altitude Widget — deployed. NO SONAR state confirmed (expected — no Ping2 connected).
Commit `0640353`. Commit `0640353`.
- Backup system — push model, both Pis → server. RPi4: 00:00 daily. RPi5: 00:30 daily. - Backup system — push model, both Pis → server. RPi4: 00:00 daily. RPi5: 00:30 daily.
@ -124,22 +137,41 @@ verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 J
to be read directly off `/rov/failsafe` every time rather than watched in the service log. to be read directly off `/rov/failsafe` every time rather than watched in the service log.
Should log state transitions and cause on each one. Minor, but cost real diagnostic time. Should log state transitions and cause on each one. Minor, but cost real diagnostic time.
See §15f. See §15f.
- **W1 safe-zone-reached button — PAUSED, not abandoned (13 Jul 2026).** Supporting
infrastructure is built and committed: `rov_api`'s `POST /nav/safe_zone_reached` endpoint and
the `external/rov-failsafe-state` data lake variable it depends on (`2def2b1`); a pre-edit
backup of the W1 widget was made per CREATE-THEN-BACKUP
(`widgets/w1_system_health_indicator.json.2026-07-13.bak`). The widget edit itself was not
started — the button would surface FSM state reached via a trigger now known to be wrong (see
the mode-aware recovery correction above), so it waits on that rework. See §15g.
- **Stray nested colcon workspace caused recurring `rov_mission` build failures — root-caused
and fixed (13 Jul 2026).** `build/`/`install`/`log` directories had been created inside
`src/rov-autonomy` on RPi5 (from `colcon build` being run from inside the repo checkout
instead of the workspace root), colliding with the real workspace at `/data/ros2_ws` and
causing builds to resolve `mode_profiles.yaml` through a stale nested `install/` tree. This
had recurred 2-3 times. Fixed by removing the stray directories (gitignored/untracked — local
filesystem cleanup only, no git change). **Durable guard: `colcon build` must only ever be
run from `/data/ros2_ws`, never from inside `src/rov-autonomy`.** See §15g.
**PARKED (design decisions deferred, priority order — moved here from CLAUDE.md 11 Jul 2026):** **PARKED (design decisions deferred, priority order — moved here from CLAUDE.md 11 Jul 2026):**
1. ~~**Doc restructure**~~ — DONE this session (11 Jul 2026): this document, `CLAUDE.md`, 1. ~~**Doc restructure**~~ — DONE this session (11 Jul 2026): this document, `CLAUDE.md`,
and `docs/design_intent_register.md` restructured/updated; all docs moved into repo `docs/`. and `docs/design_intent_register.md` restructured/updated; all docs moved into repo `docs/`.
2. **Mode-dependent failsafe recovery** — AUV (`health_role: gate`) commits to recovery on 2. **Mode-dependent failsafe recovery — CORRECTED intent (13 Jul 2026), rework pending.**
condition-clear, does not auto-resume; ROV (`advisory`) holds and hands the `failsafe_monitor` correctly reads the mode profile and branches on `health_role`
continue-or-recover decision to the operator. **Essentially complete (13 Jul 2026):** (`2a3e577`) — that mechanism stands. But `2a3e577`'s ENTRY trigger (gate commits
`failsafe_monitor` now reads the mode profile and branches on `health_role` (`2a3e577`), and `HOLD_AND_RECOVER`→`RETURN_TO_SAFE` **on condition-clear**) is **KNOWN-WRONG**: it produced a
gate-mode `RETURN_TO_SAFE` now has a real exit via the new permanent contradictory live state (W1 GREEN while the FSM sat in `RETURN_TO_SAFE`). **Corrected intent**
`/rov/nav/safe_zone_reached` interface (`bf815cc`), verified live end-to-end on the bench. (`docs/design_intent_register.md` v1.4, `255096f`): GREEN is unconditionally a working state;
**Remaining:** the event's publisher — navigation cannot yet raise it, so the immediate next committed recovery is triggered by fault **persistence** (config-driven raw failure count,
step is disposable bench scaffolding, a temporary W1-contextual operator button (shown only transient tier) or **recurrence/hard-persistence** (RED that doesn't clear, immediate commit)
while W1 reads the recovering/`RETURN_TO_SAFE` state, backed by a `rov_api` endpoint), to be — never by condition-clear. All thresholds are config-wizard-driven via the resolved mode
deleted outright once navigation publishes the real GPS/EKF-derived event. Advisory mode's profile, like `health_role`. The safe-zone-reached arrival EXIT mechanism
operator resume is also still a placeholder topic pending the proper `rov_api` endpoint. See (`/rov/nav/safe_zone_reached`, `bf815cc`) is unaffected and still correct — it still
DIR-7 addendum + new Parked Design Item, `docs/design_intent_register.md`. completes a committed recovery; only entry into one was wrong. **This rework is the
top-priority NEXT item** (see §0 NEXT). The W1-contextual operator button (supporting
`rov_api` endpoint + data lake variable already built, `2def2b1`) is PAUSED pending this
rework — see Open Items above and §15g. Advisory mode's operator resume is also still a
placeholder topic pending the proper `rov_api` endpoint.
3. **Auto-recovery**`failsafe_monitor` should attempt to re-establish the FC link itself, 3. **Auto-recovery**`failsafe_monitor` should attempt to re-establish the FC link itself,
escalating/holding RED only if it cannot. Real-world dives have no operator to restore comms. escalating/holding RED only if it cannot. Real-world dives have no operator to restore comms.
**Intent now recorded** in DIR's Parked Design Items (11 Jul 2026) — implementation still open. **Intent now recorded** in DIR's Parked Design Items (11 Jul 2026) — implementation still open.
@ -155,21 +187,31 @@ verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 J
a dedicated design session. a dedicated design session.
**NEXT (priority order, carried from §12):** **NEXT (priority order, carried from §12):**
1. **Safe-zone-reached arrival button (bench scaffolding)** — temporary W1-contextual operator 1. **FSM entry-trigger rework — count-based fault-persistence model (TOP PRIORITY, 13 Jul
button, shown only while W1 reads the recovering/`RETURN_TO_SAFE` state, backed by a new 2026).** `failsafe_monitor`'s condition-clear-commits-to-recovery entry logic (`2a3e577`) is
`rov_api` endpoint that publishes `True` to `/rov/nav/safe_zone_reached`. Disposable — KNOWN-WRONG (see `docs/design_intent_register.md` v1.4, DIR-7 addendum, `255096f`). Rework
deleted outright once navigation can raise the real GPS/EKF-derived arrival event on the to: GREEN unconditionally a working state; committed recovery triggered by fault persistence
same (permanent) topic. See §15f, DIR-7 addendum, and the new Parked Design Item. (transient tier — config-driven raw failure count, no time window) or hard-persistence (RED
2. **W3 widget code**: update label to "RETURN TO SAFE", set `FASTAPI_HOST` = that doesn't clear — immediate commit); all thresholds config-wizard-driven via the resolved
mode profile, like `health_role`. The safe-zone-reached arrival EXIT mechanism (`bf815cc`)
needs no rework — only the entry trigger.
2. **Safe-zone-reached arrival button (bench scaffolding) — PAUSED, waiting on item 1.**
`rov_api`'s `POST /nav/safe_zone_reached` endpoint and the `external/rov-failsafe-state` data
lake variable it depends on are built and committed (`2def2b1`); a pre-edit W1 widget backup
was made per CREATE-THEN-BACKUP. The widget edit itself is not started — the button would
surface FSM state reached via the known-wrong trigger above, so it is meaningless until item
1 lands. Disposable once navigation can raise the real GPS/EKF-derived arrival event on the
same permanent topic. See §15g, DIR-7 addendum, and the Parked Design Items.
3. **W3 widget code**: update label to "RETURN TO SAFE", set `FASTAPI_HOST` =
`http://192.168.1.101:8081`, then test W3 end-to-end (button → /abort → FSM, already `http://192.168.1.101:8081`, then test W3 end-to-end (button → /abort → FSM, already
verified server-side). verified server-side).
3. **Stage 2b design decisions** (deferred pending Grant): return budget "shortest path to 4. **Stage 2b design decisions** (deferred pending Grant): return budget "shortest path to
safety" concept; W7 backup notice (Option A confirmed: server read-only JSON, RPi5 polls safety" concept; W7 backup notice (Option A confirmed: server read-only JSON, RPi5 polls
outbound). outbound).
4. **W2 testing**: test with live mission via `POST /mission/start` (backend now available) 5. **W2 testing**: test with live mission via `POST /mission/start` (backend now available)
+ loaded plan. + loaded plan.
5. **Ping2 sonar purchase** (~$280 USD): required for W6 full end-to-end test. 6. **Ping2 sonar purchase** (~$280 USD): required for W6 full end-to-end test.
6. **Pre-field hardening pass**: UFW + RPi5 SSH password (bundled, triggered before remote 7. **Pre-field hardening pass**: UFW + RPi5 SSH password (bundled, triggered before remote
ops/field). ops/field).
**DATA LAKE VARIABLES (19 total, published by cockpit_bridge):** **DATA LAKE VARIABLES (19 total, published by cockpit_bridge):**
@ -200,6 +242,8 @@ verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 J
| Hash | Description | | Hash | Description |
|---|---| |---|---|
| `255096f` | docs(dir): correct inverted DIR-7 in-mission-recovery intent — count-based fault-persistence model, GREEN unconditional working state |
| `2def2b1` | rov_api: POST /nav/safe_zone_reached endpoint; cockpit_bridge: external/rov-failsafe-state variable |
| `bf815cc` | failsafe_monitor: /rov/nav/safe_zone_reached permanent interface closes gate-mode RETURN_TO_SAFE terminal trap | | `bf815cc` | failsafe_monitor: /rov/nav/safe_zone_reached permanent interface closes gate-mode RETURN_TO_SAFE terminal trap |
| `464e17e` | rov_full.launch.py: wire mode_profile_loader in (was built 4e48dc2, never launched) + respawn | | `464e17e` | rov_full.launch.py: wire mode_profile_loader in (was built 4e48dc2, never launched) + respawn |
| `2a3e577` | failsafe_monitor: mode-aware recovery (health_role gate/advisory) per DIR-7 addendum | | `2a3e577` | failsafe_monitor: mode-aware recovery (health_role gate/advisory) per DIR-7 addendum |
@ -229,6 +273,7 @@ verified live end-to-end on the bench (`bf815cc`). See §15f and Changelog, 13 J
| Version | Date | Changes | | Version | Date | Changes |
|---|---|---| |---|---|---|
| 2.26 | 13 Jul 2026 | **Same-day correction, extending v2.25's record.** The mode-aware recovery built earlier today (`2a3e577`) was found to be internally contradictory: its gate-mode entry trigger commits to `RETURN_TO_SAFE` ON CONDITION-CLEAR, which produced a live state that should not be possible — W1 showing GREEN ("Systems nominal") while the vehicle sat in `RETURN_TO_SAFE` ("Recovering to safe zone"). Corrected in `docs/design_intent_register.md` v1.4 (`255096f`): GREEN is unconditionally a working state; committed recovery is triggered by fault PERSISTENCE (transient tier — config-driven raw failure count, deliberately no time window) or immediate hard-persistence (RED that doesn't clear), never by condition-clear; all thresholds config-wizard-driven via the resolved mode profile like `health_role`. Survives unaffected: `health_role` mode-awareness (`2a3e577`'s subscription mechanism) and the safe-zone-reached arrival EXIT mechanism (`bf815cc`) — only ENTRY into committed recovery was wrong. `2a3e577`'s entry logic is now flagged KNOWN-WRONG pending rework, which is the new top-priority NEXT item, ahead of everything previously listed. Also recorded: (1) the W1 safe-zone-reached button is PAUSED, not abandoned — its `rov_api` endpoint (`POST /nav/safe_zone_reached`) and the `external/rov-failsafe-state` data lake variable are built and committed (`2def2b1`), a pre-edit W1 widget backup was made per CREATE-THEN-BACKUP, but the widget edit itself was not started because the FSM state it would surface is reached via a known-wrong trigger; (2) a recurring `rov_mission` build failure (recurred 2-3 times) was root-caused to stray nested `build`/`install`/`log` directories inside `src/rov-autonomy` on RPi5 (from `colcon build` run from the wrong directory), colliding with the real workspace at `/data/ros2_ws` — fixed by removing the stray directories (gitignored, local cleanup only, no git change); durable guard recorded: `colcon build` only ever from `/data/ros2_ws`. §0 Confirmed Working, Open Items, PARKED, NEXT, Recent Commits all updated; new §15g narrative section. Companion DIR bump: `docs/design_intent_register.md` v1.4 — see that changelog for the full corrected-intent detail, not repeated here. |
| 2.25 | 13 Jul 2026 | **Two-part fix session, both surfaced by putting the 11 Jul mode-aware work under load.** (1) `mode_profile_loader` was found never to have been wired into `rov_full.launch.py` despite being committed 7 Jul (`4e48dc2`) — `/rov/mode/profile` had zero publishers at runtime since then, so the entire operating-mode profile system had been dark and `failsafe_monitor`'s `health_role` had been silently running on its gate default the whole time. Fixed by adding the node to the launch file with respawn (`464e17e`), verified live (publisher count 0→1). (2) With the mode system now actually publishing, `failsafe_monitor`'s mode-aware recovery (`2a3e577`, same session — subscribes to the resolved profile, branches condition-clear recovery on `health_role`) surfaced a second gap: gate-mode committed recovery entered `RETURN_TO_SAFE` and could never leave it, because that branch deliberately ignores further condition-clear so recovery runs to completion, and nothing signalled completion. Fixed by a new PERMANENT interface, `/rov/nav/safe_zone_reached` (std_msgs/Bool), that transitions the FSM `RETURN_TO_SAFE`→`NORMAL` on arrival when gate-mode (`bf815cc`), verified live end-to-end on the bench (published the event by hand, watched the FSM release and the `motion_controller` recovery broadcast stop). Today's publisher is temporary bench scaffolding only — real navigation-raised arrival and a temporary W1-contextual operator button are both still to come; see the new Parked Design Item in `docs/design_intent_register.md` v1.3. §0 Confirmed Working, PARKED, and NEXT updated; new §15f narrative section. **Two pre-existing `failsafe_monitor` defects found this session (not caused by it):** `flag_manual_abort` is a one-way latch never reset, permanently shadowing all lower-priority handling (including this session's own new logic) after any single manual abort — field-deployment concern, needs its own session; and `failsafe_monitor` produced no log output in `journalctl` this session, costing diagnostic time. Both added to §0 Open Items and §15f. Companion DIR bump: `docs/design_intent_register.md` v1.3 adds the mode-aware safe-zone-reached Parked Design Item — see that changelog for detail. | | 2.25 | 13 Jul 2026 | **Two-part fix session, both surfaced by putting the 11 Jul mode-aware work under load.** (1) `mode_profile_loader` was found never to have been wired into `rov_full.launch.py` despite being committed 7 Jul (`4e48dc2`) — `/rov/mode/profile` had zero publishers at runtime since then, so the entire operating-mode profile system had been dark and `failsafe_monitor`'s `health_role` had been silently running on its gate default the whole time. Fixed by adding the node to the launch file with respawn (`464e17e`), verified live (publisher count 0→1). (2) With the mode system now actually publishing, `failsafe_monitor`'s mode-aware recovery (`2a3e577`, same session — subscribes to the resolved profile, branches condition-clear recovery on `health_role`) surfaced a second gap: gate-mode committed recovery entered `RETURN_TO_SAFE` and could never leave it, because that branch deliberately ignores further condition-clear so recovery runs to completion, and nothing signalled completion. Fixed by a new PERMANENT interface, `/rov/nav/safe_zone_reached` (std_msgs/Bool), that transitions the FSM `RETURN_TO_SAFE`→`NORMAL` on arrival when gate-mode (`bf815cc`), verified live end-to-end on the bench (published the event by hand, watched the FSM release and the `motion_controller` recovery broadcast stop). Today's publisher is temporary bench scaffolding only — real navigation-raised arrival and a temporary W1-contextual operator button are both still to come; see the new Parked Design Item in `docs/design_intent_register.md` v1.3. §0 Confirmed Working, PARKED, and NEXT updated; new §15f narrative section. **Two pre-existing `failsafe_monitor` defects found this session (not caused by it):** `flag_manual_abort` is a one-way latch never reset, permanently shadowing all lower-priority handling (including this session's own new logic) after any single manual abort — field-deployment concern, needs its own session; and `failsafe_monitor` produced no log output in `journalctl` this session, costing diagnostic time. Both added to §0 Open Items and §15f. Companion DIR bump: `docs/design_intent_register.md` v1.3 adds the mode-aware safe-zone-reached Parked Design Item — see that changelog for detail. |
| 2.24 | 11 Jul 2026 | **§15d resolved.** Vehicle health / FC-comms-loss fault (discovered 9 Jul, flagged OPEN/SAFETY-CRITICAL in v2.22) is fixed as of four commits today: `86b2728` (unconditional RED on comms loss + `fcu_ever_connected` latch + cause text), `1b0050d` (FSM recovery fix — `RETURN_TO_SAFE` was a terminal trap), `dbe9515` (new `external/rov-failsafe-cause` variable), `3409ae5` (W1 widget shows state + cause text). Verified live end-to-end. Related fault found+fixed same session: `cockpit_bridge` was publishing `failsafe_state` (FSM, 3+) not `assessment_state` (0/1/2) to `rov-failsafe` — the actual cause of that day's "widgets show no data," fixed in `d854641`, compounded by a restart-storm wedging port 9001 (→ CLAUDE.md's one-restart-then-verify-9001 rule). §0 Confirmed Working and Recent Commits updated; PARKED list updated to reflect intent now recorded in the DIR (mode-dependent recovery, auto-recovery, dedicated cause field, AMBER runtime-response — all still open on implementation, closed on design intent). Companion DIR bump: `docs/design_intent_register.md` v1.2 adds the two DIR-7 addenda and a Parked Design Items section — see that changelog for detail, not repeated here. | | 2.24 | 11 Jul 2026 | **§15d resolved.** Vehicle health / FC-comms-loss fault (discovered 9 Jul, flagged OPEN/SAFETY-CRITICAL in v2.22) is fixed as of four commits today: `86b2728` (unconditional RED on comms loss + `fcu_ever_connected` latch + cause text), `1b0050d` (FSM recovery fix — `RETURN_TO_SAFE` was a terminal trap), `dbe9515` (new `external/rov-failsafe-cause` variable), `3409ae5` (W1 widget shows state + cause text). Verified live end-to-end. Related fault found+fixed same session: `cockpit_bridge` was publishing `failsafe_state` (FSM, 3+) not `assessment_state` (0/1/2) to `rov-failsafe` — the actual cause of that day's "widgets show no data," fixed in `d854641`, compounded by a restart-storm wedging port 9001 (→ CLAUDE.md's one-restart-then-verify-9001 rule). §0 Confirmed Working and Recent Commits updated; PARKED list updated to reflect intent now recorded in the DIR (mode-dependent recovery, auto-recovery, dedicated cause field, AMBER runtime-response — all still open on implementation, closed on design intent). Companion DIR bump: `docs/design_intent_register.md` v1.2 adds the two DIR-7 addenda and a Parked Design Items section — see that changelog for detail, not repeated here. |
| 2.23 | 11 Jul 2026 | **Doc restructure.** This document is now the single source of truth for current state (Quick Status Board gains PARKED and NEXT subsections, folding in the list previously buried in CLAUDE.md's SESSION LOG and §12's Next Priorities). New **Changelog** section (below) replaces CLAUDE.md's SESSION LOG — same entries, same append-only format, moved here. The embedded Design Intent Register (formerly here, ~200 lines) is **removed** — full DIR now lives only in `docs/design_intent_register.md`; this document keeps a short pointer + summary of which DIRs govern current work. All docs moved into the repo under `docs/` (git-tracked, no version number in filenames — see CLAUDE.md Documentation Discipline). Fixed: duplicate `## 15d` heading (second instance renamed `15e`); §15's "Recording control" locked-decision row corrected from "DIR-9 — always-on" to DIR-9a (ARM-bounded); duplicate 8-variable data-lake table in §9 replaced with a pointer to §0's canonical 19-variable table; stale "Widget JSON not in Gitea" row in §16 marked resolved (W6 committed per `785ec56`, already reflected in §12). No new facts asserted beyond what was already recorded elsewhere in this document, CLAUDE.md, or the DIR — this pass is consolidation, not new findings. | | 2.23 | 11 Jul 2026 | **Doc restructure.** This document is now the single source of truth for current state (Quick Status Board gains PARKED and NEXT subsections, folding in the list previously buried in CLAUDE.md's SESSION LOG and §12's Next Priorities). New **Changelog** section (below) replaces CLAUDE.md's SESSION LOG — same entries, same append-only format, moved here. The embedded Design Intent Register (formerly here, ~200 lines) is **removed** — full DIR now lives only in `docs/design_intent_register.md`; this document keeps a short pointer + summary of which DIRs govern current work. All docs moved into the repo under `docs/` (git-tracked, no version number in filenames — see CLAUDE.md Documentation Discipline). Fixed: duplicate `## 15d` heading (second instance renamed `15e`); §15's "Recording control" locked-decision row corrected from "DIR-9 — always-on" to DIR-9a (ARM-bounded); duplicate 8-variable data-lake table in §9 replaced with a pointer to §0's canonical 19-variable table; stale "Widget JSON not in Gitea" row in §16 marked resolved (W6 committed per `785ec56`, already reflected in §12). No new facts asserted beyond what was already recorded elsewhere in this document, CLAUDE.md, or the DIR — this pass is consolidation, not new findings. |
@ -367,6 +412,44 @@ context needed. Full trigger-rule text lives in `CLAUDE.md`.
produced no log output at all in journalctl this session — state had to be read directly produced no log output at all in journalctl this session — state had to be read directly
off /rov/failsafe every time. Should log state transitions and cause. Minor but cost real off /rov/failsafe every time. Should log state transitions and cause. Minor but cost real
diagnostic time this session. diagnostic time this session.
- 2026-07-13 — CORRECTED (later same session): the DIR-7 in-mission-recovery intent
implemented earlier today (2a3e577) was inverted. It had gate-mode AUV commit to recovery
ON CONDITION-CLEAR, which produced a contradictory live state: W1 GREEN ("Systems nominal")
while the vehicle was simultaneously in RETURN_TO_SAFE ("Recovering to safe zone"). Corrected
intent recorded in docs/design_intent_register.md v1.4 (255096f): GREEN is unconditionally a
working state, a cleared fault returns to GREEN and the mission continues/resumes, and
committed recovery is triggered by fault PERSISTENCE or RECURRENCE, never by clearing. Three
fault tiers recorded: uncontrollable (power loss etc., out of FSM scope, adjacent to shelved
DIR-6); transient/self-resetting (system attempts restore, returns to GREEN and resumes on
success, commits to RETURN_TO_SAFE only once a raw config-driven failure count is exceeded,
deliberately no time window); hard/persistent (RED that doesn't clear, commits immediately).
All thresholds are config-wizard-driven via the resolved mode profile, like health_role, not
hardcoded. Survives unaffected: health_role mode-awareness (2a3e577's subscription
mechanism) and the safe-zone-reached arrival EXIT mechanism (bf815cc) — only ENTRY into
committed recovery was wrong, the exit mechanism is unchanged and correct. 2a3e577's
condition-clear entry logic is now KNOWN-WRONG pending rework — new top-priority NEXT item,
ahead of everything previously listed. New Parked Design Item: adaptive restart-timing
(monitor own restore count/duration per fault, weigh future attempts against observed
history instead of a fixed time) — depends on the count-based retry machinery above existing
first, design-captured not build-specified.
- 2026-07-13 — W1 safe-zone-reached button work PAUSED, not abandoned. Supporting
infrastructure built and committed: rov_api POST /nav/safe_zone_reached endpoint (mirrors
the existing /abort pattern exactly) and the new external/rov-failsafe-state data lake
variable it depends on (surfaces FailsafeStatus.failsafe_state, distinct from the existing
external/rov-failsafe assessment integer) — both in 2def2b1. Pre-edit backup of the W1
widget made per CREATE-THEN-BACKUP: widgets/w1_system_health_indicator.json.2026-07-13.bak.
The widget edit itself was deliberately not started — the button would surface FSM state
(rov-failsafe-state == 3 / RETURN_TO_SAFE) reached via the trigger just found to be
known-wrong above, so it would be meaningless until the FSM rework lands.
- 2026-07-13 — Root-caused a recurring rov_mission build failure (recurred 2-3 times across
sessions): stray nested build/, install/, and log/ directories existed inside
src/rov-autonomy on RPi5, created when colcon build was at some point run from inside the
repo source directory instead of the actual workspace root, /data/ros2_ws. These collided
with the real workspace and caused rov_mission builds to fail resolving mode_profiles.yaml
through a stale nested install/ tree. Fixed by removing the stray directories from RPi5 —
gitignored and untracked, so this was a local filesystem cleanup only, no git change.
Durable guard: colcon build must only ever be run from /data/ros2_ws, never from inside
src/rov-autonomy. Recorded so this is not reintroduced.
--- ---
@ -1171,6 +1254,87 @@ MAVLinkRouter is the confirmed stable router. Do not recommend or use MAVP2P.
--- ---
## 15g. DIR-7 recovery-intent correction, W1 button paused, and a build-workspace footgun — 13 Jul 2026 (later same session)
### DIR-7 in-mission-recovery intent corrected — condition-clear entry trigger was inverted
Later the same day as §15f, the DIR-7 "In-mission recovery is mode-dependent" addendum was
found to be internally contradictory as implemented. `2a3e577`'s gate-mode entry trigger —
commit to `RETURN_TO_SAFE` ON CONDITION-CLEAR — produced a live state that should not be
possible: **W1 showing GREEN ("Systems nominal") while the vehicle was simultaneously in
`RETURN_TO_SAFE` ("Recovering to safe zone")** — a working-state light next to an
active-recovery action.
**Corrected intent** (full detail: `docs/design_intent_register.md` v1.4, DIR-7 addendum,
commit `255096f` — pointer only here, not duplicated):
- **GREEN is unconditionally a working state.** A fault that clears returns the vehicle to
GREEN and the mission continues or resumes. A cleared condition must **never** trigger or
commit a recovery.
- Committed recovery is triggered by a fault **persisting or recurring**, never by it clearing.
- **Three fault tiers:** (1) **uncontrollable** — power loss or anything that disables the
safety logic itself, out of scope for the FSM entirely, adjacent to the shelved DIR-6
Emergency Surface concern; (2) **transient/self-resetting** — e.g. a comms blip or sensor
re-sync; the system attempts restore, returns to GREEN and resumes on success, and only
commits to `RETURN_TO_SAFE` once a **raw, config-driven failure count** (no time window, no
fixed duration) is exceeded; (3) **hard/persistent** — a RED that does not clear, commits to
`RETURN_TO_SAFE` immediately.
- All thresholds (the acceptable failure count, and which specific faults are transient vs
hard) are **configuration-driven**, set in the setup wizard and read from the resolved mode
profile the same way `health_role` is — never hardcoded.
**What this means for the code right now:**
- **KNOWN-WRONG, pending rework (top-priority next-session task):** `2a3e577`'s
condition-clear-commits-to-recovery entry logic (the `health_role == HEALTH_GATE` /
`HOLD_AND_RECOVER` branch in `failsafe_monitor._apply_failsafe_priority`'s "all clear"
block). This is what produces the contradictory GREEN-while-recovering state above, and must
be reworked into the count-based tier model.
- **Survives, confirmed correct, no rework needed:** `failsafe_monitor` subscribing to
`/rov/mode/profile` and reading `health_role` (`2a3e577`'s mode-awareness mechanism) — this
is the right way to know which mode's rules apply, and is unaffected by the correction. Also
survives: the safe-zone-reached arrival EXIT mechanism and its permanent
`/rov/nav/safe_zone_reached` topic (`bf815cc`) — it still correctly completes a **committed**
`RETURN_TO_SAFE` recovery on arrival. Only the trigger for **entering** committed recovery
was wrong; how a genuinely-committed recovery **exits** is untouched and correct.
**§0 Confirmed Working, PARKED, and NEXT updated accordingly** — mode-aware recovery is no
longer described as "essentially complete"; the FSM entry-trigger rework is now the top NEXT
priority, ahead of the W1 button (which is paused, not the immediate next step — see below).
### W1 safe-zone-reached button — PAUSED, not abandoned
The supporting infrastructure for the temporary W1-contextual operator button is built and
committed: the `rov_api` endpoint `POST /nav/safe_zone_reached` (mirrors the existing `/abort`
pattern exactly) and the new `external/rov-failsafe-state` data lake variable it depends on
(surfaces `FailsafeStatus.failsafe_state`, distinct from the existing `external/rov-failsafe`
assessment integer), both in commit `2def2b1`. A pre-edit backup of the W1 widget was made per
CREATE-THEN-BACKUP: `widgets/w1_system_health_indicator.json.2026-07-13.bak`.
**The W1 widget edit itself was deliberately not started.** The button would surface FSM state
(specifically, `rov-failsafe-state == 3` / `RETURN_TO_SAFE`) that the DIR-7 correction above
just established is reached via a known-wrong trigger — the button would be meaningless,
possibly misleading, until the FSM correctly represents recovery state per the corrected
model. This work is paused, waiting on the FSM rework above, not abandoned or deprioritised
for any other reason.
### Stray nested colcon workspace — recurring rov_mission build failures root-caused
A recurring build failure (recurred 2-3 times across sessions) was root-caused this session:
stray nested `build/`, `install/`, and `log/` directories existed inside `src/rov-autonomy` on
RPi5 — created at some point when `colcon build` was run from inside the repo source directory
instead of the actual workspace root, `/data/ros2_ws`. These nested directories collided with
the real workspace, and caused `rov_mission` builds to fail resolving `mode_profiles.yaml`
through a stale nested `install/` tree rather than the real one.
**Fix:** the stray nested `build/`, `install/`, and `log/` directories were removed from RPi5.
They are gitignored and untracked, so this was a **local filesystem cleanup only — no git
change**, and does not appear in any commit.
**Durable guard:** `colcon build` must **only ever** be run from `/data/ros2_ws` (the actual
workspace root) — **never** from inside `src/rov-autonomy` (the repo source checkout within
it). Recorded here so this is not reintroduced.
---
## 15b. Session 6 Lessons Learned — RPi5 Base Stack Install ## 15b. Session 6 Lessons Learned — RPi5 Base Stack Install
### SSH on Ubuntu 24.04 Server (cloud-init image) ### SSH on Ubuntu 24.04 Server (cloud-init image)