graph_watchdog design ===================== Role ---- Detects silent faults in the ROS 2 graph. A ``GatewayPlugin`` hosting a fleet of detectors; each detector observes the ROS 2 graph and raises faults via a ``ReportFault`` service client on the gateway node. Faults reach the SOVD ``/faults`` API via FaultManager - the plugin injects no entities. Its only HTTP surface is a read-only reliability status route (see `Reliability core`_ below). Structure --------- - **Plugin shell** (``GraphWatchdogPlugin``): loads via the gateway plugin ABI (v7). In ``set_context`` it casts the context with ``as_ros_plugin_context``, creates one ``rclcpp::Client`` on the gateway node, and starts a dedicated tick thread. The tick is deliberately NOT a gateway wall timer: detectors do blocking parameter/service reads, and the gateway's small executor is only safe because blocking never runs on an executor thread. No child node. Every ROS entity the plugin owns - the fault client here and the ``LifecycleWatcher``'s ``~/transition_event`` subscriptions below - is created in a callback group of its own with ``automatically_add_to_executor_with_node = false``, and added only to a private ``SingleThreadedExecutor`` that the tick thread pumps between ticks. Neither executor is ever spun on a thread of its own. The reason is the same in both cases and is a correctness requirement, not tuning: ``rclcpp::AnyExecutable`` holds a strong reference to the subscription or client it dispatches and is a local of the executor thread, so an entity left in the node's default group can have its last reference released - and its destructor run, mutating the node's rcl entity registry - on a gateway executor thread, concurrently with this plugin creating an entity on the same node. No lock the plugin holds can reach a gateway executor thread, so the fix is to keep the entities out of every gateway executor; that puts creation, destruction and callback execution on the tick thread alone. For the fault client the drain is load-bearing for a second reason: ``DetectorContext::raise_fault`` is fire-and-forget (it guards on ``service_is_ready()`` and discards the future), and rclcpp keeps per-request state until an executor processes the response, so an unpumped client would leak one pending entry per raised fault for the life of the process. - **Detector pattern**: ``Detector`` + ``DetectorContext``; each detector is one self-registering file under ``src/detectors/`` (``REGISTER_DETECTOR``), globbed by CMake so parallel authors never edit a shared file. A detector's source needs no CMake or registry edit; adding a per-detector unit test is the one shared touch - it adds an ``ament_add_gtest`` entry to the test block. - **Fault-code contract**: the frozen ``GRAPH_*`` namespace. - **mode seam**: ``raise`` / ``advisory`` / ``off`` per detector. - **Reliability core**: see the dedicated section below - central warmup and lifecycle gating enforced inside ``raise_fault``, detector-consulted clock validity, and the ``x-medkit-watchdog`` status route. Reliability core ----------------- - **ReliabilityGate** composes two independent trackers and is the single entry point both the tick loop and the HTTP route go through: - ``WarmupTracker`` (pure, no ROS): an entity arms once continuously present for ``warmup_cycles`` ticks. A disappearance longer than a short forget grace followed by a reappearance (a real mid-run restart) re-warms it from scratch; a transient one-tick discovery gap is absorbed and does not re-warm, so recurring DDS churn cannot re-arm (and thus permanently suppress) a still-running entity. Unknown ``source_id``\ s (e.g. a topic, not a tracked app) fall back to a global bringup grace window keyed off the first tick the graph was seen non-empty, which re-arms whenever the graph empties out so a full-stack restart gets the grace again. - ``LifecycleWatcher`` (event-driven): discovers managed ``rclcpp_lifecycle`` nodes from the introspection snapshot, seeds their state via a ``GetState`` service call, then keeps it fresh by subscribing to each node's ``~/transition_event`` topic (reliable + volatile, matching the ``rcl_lifecycle`` publisher). A node still cached non-active shortly after discovery is briefly re-seeded via ``GetState`` so an ``active`` transition lost during the subscription's DDS endpoint-matching window self-heals rather than suppressing the node for the process lifetime; the blocking seeds are bounded per tick so a batch bringup cannot stall the tick loop. An entry's identity is its BINDING - the pair (fqn, ``GetState`` path) - not its ``App::id``, which can survive a graph sweep while pointing at a different node. When either half moves, the entry is dropped (the old binding recorded as departed under its own fqn) and re-seeded from scratch, so a moved binding can never keep enforcing the old node's label. The subscription callback holds only a ``weak_ptr`` to the watcher's shared state, so a callback in flight during teardown bails instead of touching freed memory. Non-managed nodes are never gated - ``node_ok`` returns true for them unconditionally, and so is a managed node whose ``GetState`` has never answered: its label stays empty, and ``node_ok`` reads an unread label as permission on purpose, since gating on it would silence every detector for a node whose lifecycle service is broken. Only a KNOWN non-active label suppresses. ``presence_ownership()`` asks the STRICTER question the presence class needs, without changing that permissive answer, and answers it with a GROUND rather than a boolean: EARNED for a state read as ``active`` or for a node with no lifecycle at all, PROVISIONAL for one whose state was asked for as often as it ever will be and never came (``measurement_pending()``, which reads the per-node GetState re-seed budget charged only for reads that actually ran), and neither while the asking is still going on. Only the first is permanent - see the ``node_death`` section. Those subscriptions follow the private-callback-group rule described in the plugin shell bullet above: created on the shared gateway node, but in the watcher's own group and pumped only by the watcher's own executor, so they are created, run and destroyed on the tick thread alone. This is also why the tick thread drains in short slices rather than once per tick - a transition would otherwise be delayed by up to a whole tick interval, and a bringup burst could overrun the subscription's queue and lose the intermediate transitions the departure classification reads. - **Central enforcement.** ``DetectorContext::raise_fault`` runs every raise through ``reliability_allows(gate, source_id)`` before the fault client sends anything; a detector raising about a still-warming-up entity or a lifecycle-inactive node is silently suppressed, transparent to the detector. ``clear_fault`` is never gated - a fault can always be cleared. A null gate (not yet wired) always allows, so unit tests that construct a bare ``DetectorContext`` are unaffected. - **Clock validity is detector-consulted, not central.** ``WatchdogClock::time_is_valid()`` flags a paused or absent ``/clock`` under ``use_sim_time`` by comparing wall-clock advancement against sim-time advancement on each ``mark_tick()``. Unlike warmup/lifecycle, this is not enforced inside ``raise_fault`` - a time-based detector must check it itself and skip its own age/grace-period math for the tick when it is false. - **tf_static QoS helper.** ``tf_static_qos()`` returns a depth-1 ``transient_local`` ``rclcpp::QoS``, available to detectors alongside ``time_is_valid()``. ``/tf_static`` publishers latch with transient-local durability, so a late subscriber must match it; the actual ``/tf_static`` subscription belongs to a later detector. - **Status endpoint.** ``GraphWatchdogPlugin::get_routes()`` registers ``GET x-medkit-watchdog`` (mounted at ``/api/v1/x-medkit-watchdog`` by the gateway), returning ``ReliabilityGate::status_json()``: schema version, ``warmup_cycles``, a ``global_state`` (``armed``/``warming_up``), and one entry per known entity (``id``, ``first_seen_tick``, ``armed``, ``state``, ``lifecycle``). Returns 503 with ``ERR_SERVICE_UNAVAILABLE`` if the gate has not been constructed yet or has already been torn down by ``shutdown()``. Beside the gate's own state the payload carries a ``detectors`` object, one block per detector whose ``Detector::status_json()`` returns something, omitted entirely when none does. It exists for a condition that belongs to a SINGLE detector and would otherwise live only in the gateway log, which no HTTP client and no e2e assertion can read - ``lifecycle_expectation`` reports ``tracking_saturated`` (it has refused to track a required node) beside the live ``tracked_nodes`` count and the ``tracked_node_cap`` in force. The handler runs on an HTTP thread while ``tick()`` runs on the tick thread and holds no lock a detector takes, so a ``status_json()`` implementation must build its payload from atomics and must not block. - **Build note.** ``LifecycleWatcher`` reuses the gateway's own lifecycle-state helpers (``lifecycle_status_helpers.cpp``, ``ros2_lifecycle_state_reader.cpp``) compiled in via ``GATEWAY_SRC_DIR`` - the same non-header-only reuse pattern other gateway plugins use - rather than reimplementing lifecycle-state parsing. Detectors --------- ``qos_mismatch``, ``orphan``, ``param_drift``, ``lifecycle_expectation`` and ``node_death`` are the detectors this package ships so far. Two silent-fault classes remain undelivered, ``GRAPH_TF_STALE`` and ``GRAPH_LATENCY_BUDGET``; they land in follow-up changes, each against its own issue. ``qos_mismatch`` raises ``GRAPH_QOS_MISMATCH``. It watches every topic's publisher/subscriber QoS pairs rather than parameter values: each tick it enumerates every topic via ``get_topic_names_and_types()`` and, for each one, every currently-connected publisher and subscriber (``get_publishers_info_by_topic`` / ``get_subscriptions_info_by_topic``, which report the RESOLVED live profile - never a hand-built one), checking each pub x sub pair for incompatibility. **Two levels behind one fault code.** Per-pair incompatibility is not reported uniformly. A subscriber incompatible with EVERY publisher on its topic receives nothing and raises at ``SEVERITY_ERROR``. A subscriber incompatible with some publisher but not all raises at ``SEVERITY_WARN``: DDS refuses that one pair, so that producer's data is silently discarded while the topic keeps looking alive - a real silent fault, since an RxO-incompatible pair is a match DDS has already computed, not a heuristic. One fault code carries one severity, so the emitted fault reflects the worst finding in the sweep. Before this split the partial case was indistinguishable from healthy, which meant e.g. a second ``/tf`` broadcaster or a hand-rolled ``BEST_EFFORT`` publisher on ``/diagnostics`` went unreported forever. **RxO rule.** ``qos_policy.hpp``'s ``qos_incompatibility(pub, sub)`` implements RxO (Request <= Offered) compatibility for the QoS policies that silently starve a subscriber with no error surfaced anywhere in the graph: reliability, durability, liveliness kind, deadline, and liveliness lease duration. Publisher = offered, subscriber = requested; a pair is incompatible only in the one strict direction per policy - a ``BEST_EFFORT`` publisher against a ``RELIABLE`` subscriber, a ``VOLATILE`` publisher against a ``TRANSIENT_LOCAL`` subscriber, an ``AUTOMATIC``-liveliness publisher against a ``MANUAL_BY_TOPIC`` subscriber, or an offered deadline / lease duration greater than the requested one. The reverse direction (e.g. a ``RELIABLE`` publisher feeding a ``BEST_EFFORT`` subscriber) is compatible by design - the subscriber asked for no more than it is offered - even though the QoS profiles differ, which is exactly the discriminator the integration and e2e tests each pin with a positive control (see "Test tiers" below). For deadline and lease, an unspecified/infinite offer is unbounded (fails any finite request), while an unspecified subscriber value does not constrain the policy (always compatible); history and depth are not RxO dimensions and are not checked. The checks match only concrete incompatible enum pairs, so ``SYSTEM_DEFAULT``/``UNKNOWN`` (never reported by a live endpoint, which always carries the resolved profile) never raise. **Aggregation via the shared helper.** ``qos_mismatch`` was the first detector to use the ``AggregatedFault`` helper (``aggregated_fault.hpp``); ``orphan`` and ``param_drift`` go through the same one, so no detector reimplements the level-triggered raise/clear pattern. The rationale: the fault_manager identifies a fault by ``fault_code`` alone, so one ``GRAPH_QOS_MISMATCH`` per mismatched topic would collide into a single record under the shared code. The detector keeps one ``AggregatedFault`` instance for the whole graph and, each tick, hands it every currently-mismatched topic's description (keyed by topic name so a repeat mismatch on the same topic overwrites rather than duplicates); an empty map on a clean tick clears (``EVENT_PASSED``), level-triggered semantics (see "Closing the loop" in the README for the ``healing_enabled`` requirement to actually reach HEALED). ``graph_source_id()``, also in ``aggregated_fault.hpp``, returns the constant ``kGraphWatchdogEntityId`` (``"graph_watchdog"``) and nothing else, so every detector's aggregated faults land under the same ``source_id``. It deliberately does NOT fall back to the host Component id: a runtime host Component built by ``HostInfoProvider`` never sets ``external``, ``collect_component_app_fqns`` only puts a Component's bare id in the fault scope set when it does, and a fault raised under it is therefore listed by no entity endpoint at all. The ``ctx`` argument is unused and kept only because every call site already has it in hand. **Coverage is exhaustive, not budgeted.** Unlike a budgeted round-robin sweep (bounded by a parameter-service round-trip budget), ``qos_mismatch`` reads no external service - ``get_topic_names_and_types()`` and the two ``get_*_info_by_topic()`` calls are local graph-cache queries, so every topic is checked every tick with no coverage-latency trade-off to configure or budget knob to tune. ``/rosout`` and ``/parameter_events`` are skipped - ROS 2 system topics with their own well-known QoS conventions, not useful signal for this detector. The sweep polls ``ctx.cancelled`` between topics so a shutdown mid-sweep bails promptly, the same shutdown-responsiveness contract every detector honors. **Test tiers.** Three tiers each prove a different layer, deliberately not overlapping: 1. **Unit** (``test_qos_policy.cpp``): pure ``qos_incompatibility()`` logic against hand-built ``rmw_qos_profile_t`` values - the RxO trio (reliability, durability, liveliness), the reverse-direction compatible case, identical-profile compatibility, and the never-raise guarantee for ``SYSTEM_DEFAULT``/``UNKNOWN`` enums a live endpoint never actually reports. 2. **Integration** (``test_qos_mismatch_integration.cpp``): a real ``rclcpp::Node`` publisher/subscriber pair over real DDS, read through the actual ``get_publishers_info_by_topic``/``get_subscriptions_info_by_topic`` API - proving the detector reads the RESOLVED live profile correctly, not just that the pure comparison function is correct - against a fake ``ReportFault`` service. A second, concurrent RxO-compatible-but-different-QoS topic pair proves the detector does not over-fire on "QoS differs somewhere in the graph". 3. **E2e** (``test/e2e/test_qos_e2e.test.py``): the Acceptance gate - a real gateway process with the plugin ``.so`` loaded, a real fault_manager, and the operator-visible ``GET /api/v1/faults`` surface, proving the whole raise/clear story reaches a SOVD fault through the real tick timer, not just the detector's own ``tick()`` called directly. ``orphan`` raises ``GRAPH_ORPHAN``. It looks for the mistake no ROS 2 tool reports: a topic that exists but has no counterpart, because one side was spelled slightly differently. Each tick it counts publishers and subscribers per topic, keeps the one-sided ones, and pairs a publisher-only topic with a subscriber-only topic when their names are within ``max_edit_distance`` Levenshtein edits of each other. Only a pair is reported, never a lone one-sided topic: a publisher with no subscriber is normal on almost every robot, so it carries no signal on its own. **Three guards keep this from crying wolf.** - Names must sit in the same parent namespace. ``/a/scan`` and ``/b/scan`` differ by one character but belong to two different robots, and pairing them would be nonsense. The leaf and the namespace carry separate edit budgets and the namespace one defaults to zero, so this is an exact match unless an operator raises ``namespace_edit_distance``. Raising it is what makes a misspelled namespace (``/robott/scan`` against ``/robot/scan``) visible at all, at the price of reporting every robot of a fleet whose namespaces are themselves near-misses, which is why it is opt-in rather than tuned. - A pair that matches once every run of digits collapses to one placeholder is an enumeration, not a typo. ``/lidar_1`` next to ``/lidar_2``, each one-sided, is the ordinary state of a multi-sensor machine, and reporting it would fire on every such machine. Collapsing runs rather than comparing character positions also covers ``/lidar_9`` next to ``/lidar_10``, where the index changes length. The price is a real typo in a digit going unreported, which is a deliberate trade. - ``grace`` consecutive sweeps must agree before anything is raised. During bringup one side of a healthy pair is routinely missing for a moment, which looks exactly like a typo until the other side appears. An operator who still gets a false alarm puts the topic in ``allowlist``. What this detector cannot see at all is a namespace added or dropped by accident: ``/scan`` against ``/robot/scan`` is far past any sensible edit distance. **Test tiers.** ``test_orphan_policy.cpp`` pins the pure matching rules, including each guard and the cases it deliberately lets through. ``test_orphan_integration.cpp`` drives the detector over a real ``rclcpp`` graph and a fake ``ReportFault`` service, covering the grace counter, the clear on repair, and the allowlist. ``test/e2e/test_orphan_e2e.test.py`` is the acceptance gate: a real gateway with the plugin loaded, a real fault_manager, and the fault read back from ``GET /api/v1/faults``. It carries three pairs at once - one reported, one allowlisted, one visible only under a raised ``namespace_edit_distance`` - so it also proves a string array and an integer survive the parameter path into ``configure()``, which no C++ tier can show. ``param_drift`` raises ``GRAPH_PARAM_DRIFT``. It reads node parameters over the parameter service and reports a value that no longer matches its reference. The reference is either self-captured, the value the parameter had when the node armed, or pinned in config through ``expect``. A node absent for longer than the window the reliability gate absorbs loses its captured baseline, so a restart - the documented way to apply a new configuration - re-captures instead of reporting the change as drift. A shorter gap keeps the baseline on purpose: DDS discovery drops a node from a single poll routinely, and re-capturing on that would take the node's current, possibly already drifted, value as the new reference and heal a real fault that could then never fire again. Only the pinned form can catch a value that was already wrong at startup; self-capture by construction treats whatever it first sees as correct. **A budgeted sweep, unlike every other detector here.** ``qos_mismatch`` and ``orphan`` read the local graph cache and are therefore exhaustive every tick. This one makes real service calls to other nodes, so it is bounded by ``max_reads_per_tick`` and sweeps the graph round-robin. The budget is spent in service ROUND TRIPS at the watched node rather than in transport calls, because one call is several requests there: a self-capture read is a list plus a batched get, and an ``expect`` pin the node declares is a list, a get and a descriptor read. A pin the node does NOT declare stops at the list, and is charged two rather than the one it costs, because a second and rarer NOT_FOUND path does cost two and the error code cannot tell them apart - a load bound may overstate, never understate. That buys a bound on the load the sweep puts on watched nodes and pays with coverage latency: a drift is invisible until its node's turn comes, and so is a repair. The reads run on a thread the plugin owns rather than the gateway executor, so a node that stops answering stalls this sweep only. **Three horizons for a stale entry.** A node missing from one sweep is not forgotten immediately. Its captured BASELINE is dropped on the third consecutive sweep without it - the two the warmup tracker's forget grace absorbs, plus one - so that a one-tick discovery gap does not silently reset the baseline and hide a real drift behind a fresh capture. Its FINDING survives ``prune_grace`` consecutive absences and is dropped on the next one, so that the same gap does not clear the aggregated fault either. Only the finding horizon is configurable: ``prune_grace`` is a per-detector config key, while the baseline horizon is the compile-time ``kBaselineForgetGrace``. A ``prune_grace`` below 2 makes the finding the shorter of the two and the baseline then goes on the same sweep, because the absence counter it rides on is dropped with the finding. The third horizon is not about absence at all. A node that stays PRESENT but leaves the reliability gate's ``active`` state is never re-evaluated (it is not armed) and never pruned (it is still there), so its finding would otherwise be re-raised for ever - an operator who fixes the parameter and leaves the node deactivated would watch ``GRAPH_PARAM_DRIFT`` stay CONFIRMED naming a value that is no longer true. ``kFrozenHoldTicks`` (60 consecutive gate-denied ticks, one minute at the default tick period) bounds that: past it the finding is released and the app also stops holding back a clear, and both re-derive from a fresh read if it ever arms again. Because it is a compile-time constant and counted in ticks rather than in rotations, neither ``max_reads_per_tick`` nor ``tick_interval_ms`` shortens it. **Severity is WARN, deliberately.** The detector knows a value moved; it does not know whether that breaks the robot. Grading the consequence needs knowledge of the machine that the graph does not carry. **Nothing is cleared that was not measured.** A clear asserts that nothing is drifting anywhere, which is only true if every app was actually looked at, so the aggregate emits nothing at all while any app still has no usable read. That covers three states which are all "nothing is known about this app": the round-robin has not reached it, its reads keep failing, or the reliability gate has not armed it - and the last of those is the ordinary state of every app for the first ``warmup_cycles`` ticks of a run, so without it a restart clears before issuing a single round trip. A cached read counts as evidence only until a read of that app FAILS, after which the app is unmeasured again. Each hold is bounded from the other side: an app is given up on after both sixty failed attempts and sixty ticks unread, and a gate-denied one after the frozen hold. The bound on attempts alone would not do, because an attempt is not a duration - the reader paces itself off ``max_reads_per_tick``, so at the accepted ceiling sixty-one attempts are spent in milliseconds and a node still being discovered would be written off. Finally, a clear withheld for a minute of ticks is logged once with the count of apps behind it, because from outside the process a correctly withheld clear and a detector that has nothing to report look exactly the same. **Test tiers.** ``test_param_drift_policy.cpp`` pins the pure comparison, rendering and config rules. ``test_param_drift_integration.cpp`` drives real nodes with real parameters over the real parameter service against a fake ``ReportFault`` service, including a node that never answers, which must not hang the tick, and a node that answers past the per-read bound, which must not be read at all. ``test/e2e/test_param_drift_e2e.test.py`` is the acceptance gate: one gateway launch carrying the whole raise-and-clear story for the detector's default self-capturing mode, with a real fault_manager and both ends read back from ``GET /api/v1/faults``. ``test/e2e/test_param_roundtrip_e2e.test.py`` measures what one parameter read really costs the node that serves it, against a node that answers the parameter services by hand and counts every request - so a round trip added to or removed from the TRANSPORT is caught, instead of quietly invalidating the round-trip figures the budget is built on. Nothing else in the package can notice that: every other test proves the arithmetic GIVEN those figures, and the stand-in transport they use counts calls. It does not, however, read the detector's own charge constants at all - it runs with ``param_drift`` switched off - so what holds those to the measurement is the timing suite in ``test_param_drift_integration.cpp``, which times the reader against a configured budget and therefore moves when a charge does. Cost and charge are pinned separately and on purpose. ``test/e2e/test_config_plumbing_e2e.test.py`` drives three further gateway launches covering the nested ``expect`` sub-object reader, ``mode: "off"``, and the bare ``mode: off`` that the ROS parser types as a YAML 1.1 boolean; those three prove configuration delivery and suppression only, and assert no clear. ``lifecycle_expectation`` raises three independent faults, ``GRAPH_NODE_INACTIVE``, ``GRAPH_NODE_UNREADABLE`` and ``GRAPH_NODE_NOT_MANAGED``. It watches an operator-declared set of ``require_active`` node names rather than topics, parameters, or presence. Each tick it matches every configured entry against the live apps by ``App::id`` OR the stable ``effective_fqn()`` OR that fqn's bare leaf name - ``App::id`` alone is recomputed each sweep and gets namespaced on a same-bare-name collision, so an id-only match would silently stop covering a multi-robot graph; a bare name therefore matches that node in every namespace (use a full FQN to pin one). For each matched present node it reads the node's raw lifecycle label via ``ReliabilityGate::lifecycle_state_of()`` and hands the matches - a ``std::vector`` of (entry, node fqn, state) - to ``lifecycle_expectation_tracker.hpp``'s ``LifecycleExpectationTracker``, the detector's pure, ROS-free core, independently unit-tested before the detector ever touches a live gate. **The model: one observed state per node per tick, two clocks - not three fault codes each running their own bookkeeping.** All three faults are outputs of a SINGLE per-node state machine inside the tracker. Every tick, a matched node is classified into exactly one of ``ACTIVE``, ``INACTIVE`` (non-empty label, not ``active``), ``UNREADABLE`` (a managed node whose label has never been read - ``optional("")``), or ``NOT_MANAGED`` (``nullopt`` - no tracked lifecycle at all); a node not matched at all this tick is a fifth case, ``ABSENT``, handled by a wholly separate path. Two clocks track history: - The **violation streak** advances on ``INACTIVE``, resets ONLY on ``ACTIVE``. Past ``grace`` the node is ``GRAPH_NODE_INACTIVE``'s content. - The **unmeasured clock** advances on ``UNREADABLE`` OR ``NOT_MANAGED``, resets ONLY on a real measurement (``ACTIVE`` or ``INACTIVE``). It is deliberately BLIND to which of the two causes it is seeing on any given tick - a node alternating between "matched but never read" and "not a tracked lifecycle node at all" keeps this ONE clock climbing instead of each cause resetting a separate counter of its own. That blindness closes an entire class of alternation, not one more special case for it: three review rounds each found a pair of "I cannot measure this" causes whose separate counters could erase each other's progress, leaving a node invisible to every code that existed. Past ``unmeasured_hold_ticks`` (a fixed 60 ticks, not configurable), the clock MATURES and takes ownership of the node. **Ownership is exclusive.** A node is content of at most one of the three faults at a time. The moment the unmeasured clock matures, the violation streak is RELEASED - actually reset to 0, not merely excluded from ``GRAPH_NODE_INACTIVE``'s content this tick - so that fault's clear is free to flow immediately; a node returning to ``INACTIVE`` afterwards re-earns ``grace`` from zero. Which of the two unmeasured codes a matured node reports under is STICKY: it keeps reporting under whichever code the clock matured under even if the LIVE cause later flips, and only a real measurement (resetting the whole clock) changes it. The rejected alternative, current-cause-wins, is more literal but makes a node that flaps between the two causes flap the fault surface too - a raise/clear/raise churn across two codes for a node whose actual situation never changed. **Absence continues the last CORROBORATED observation, it never erases.** A node not matched at all this tick is ABSENT, a fact separate from any observed state (there is no label to classify). For up to ``absence_grace`` (a fixed 3 ticks) consecutive absent ticks a node's whole state is held unchanged - the blink tolerance. Past ``absence_grace`` absence resets nothing, but what it ADVANCES depends on whether the node's SETTLED observation is already CONTENT, and - for a below-grace violation streak only - on whether the presence detector is CURRENTLY tracking the node: a settled-``INACTIVE`` node ALREADY reported under ``GRAPH_NODE_INACTIVE`` keeps its streak exactly as it was, so the fault stays raised, regardless of ownership. One not yet past ``grace`` splits on that fact, read every tick from the key set ``node_death`` publishes and never remembered: a node in that set is HELD - neither advanced (no fault born from evidence gathered while nobody could observe the node, since ``node_death`` is tracking this exact departure) nor erased (it resumes rather than restarts on return) - while a node outside it keeps climbing on absence exactly as it would on a present tick, because nothing else in the plugin will ever report its departure either. A settled-``UNREADABLE``/``NOT_MANAGED`` node keeps climbing its unmeasured clock under that same cause regardless of maturity or ownership - that clock's own absence behaviour is unrelated to this distinction. A settled-``ACTIVE`` node advances nothing at all - anything it had started but not corroborated is released, so the entry becomes idle and is reclaimed silently. So a departure never heals a fault it has already earned, and never starts one out of evidence gathered while the node could not be observed AND could be reported some other way; what it changes, for an already-raised fault, is the detail phrase, which then says the node has since left the graph. **What "settled" means, and why an unmeasured reading needs corroborating.** A real measurement (``ACTIVE`` or a non-active label) settles at once - a label is a fact about the node, and no discovery artifact invents one. An UNMEASURED reading settles at once too on a node nothing has ever measured, since it is then the only thing known about it (this is what keeps a crash-looping node accumulating toward its report rather than being released on every absence). Only when an unmeasured reading would OVERRIDE a real measurement must it first hold for ``kDefaultObservationSettleTicks`` (6) consecutive matched ticks. The transient it guards against is real and cheap to hit: ``LifecycleWatcher::update()`` drops a tracked id whose ``get_state`` path is absent from the current sweep, and ``discover_apps()`` can yield an app with no services when a sweep races service enumeration - so a present, healthy, managed node reads ``NOT_MANAGED`` for a tick. If that tick is the last one before a clean shutdown, continuing "the last observation" literally would mature a healthy departure into a permanent ``GRAPH_NODE_NOT_MANAGED``. Six is the same bar the typo warning already sets against the same transient, and a tenth of the 60-tick unmeasured hold, so R30 is untouched: a node that is genuinely unmeasurable when it leaves corroborated that long before the hold that reports it. **Where "a departure never heals a fault" ends.** It holds within one gateway lifetime. Across a RESTART it does not: the restarted tracker has no measurements, the departed node is not in the graph, and its ``require_active`` entry matches nothing - which this detector cannot tell apart from a misspelt entry, because the only component that knows the difference is the fault store and it is not read at startup. The never-matched hold is deliberately bounded (a typo must not block healing forever), so once it lapses the level-triggered clear flows and the record heals with nothing having been measured. Re-seeding the tracker from the fault store at startup would change that and is not implemented; the boundary is pinned by the ``restart_departed`` e2e scenario rather than left to be discovered. Why the erasure went away rather than moving, for the two UNMEASURED causes: every erasure horizon is an evasion for a node that touches it periodically. A node in a restart loop - start, crash, respawn delay, start - touches absence by construction, and discarding its evidence would let it alternate ``(UNREADABLE, ABSENT x N)`` or ``(NOT_MANAGED, ABSENT x N)`` forever without ever accumulating enough of anything to be reported. The VIOLATION streak needed the identical rule for the identical reason once - a node alternating ``(INACTIVE, ABSENT x N)`` would otherwise never accumulate ``grace`` + 1 either. Where the presence class CAN also report the same departure, it no longer does: ``GRAPH_NODE_DISAPPEARED`` (this package's own ``node_death`` detector) independently reports it, whether or not the node was ever measured not-active first. But ``node_death`` only ever tracks an App once the reliability gate says it OWNS that App's departure, on one of one of the two grounds ``ReliabilityGate::presence_ownership()`` accepts - so a ``require_active`` node that never reaches ``active`` is never tracked by ``node_death``, and neither is a managed node whose ``GetState`` has not answered while the watcher is still asking. The gate would permit that second one to RAISE - ``LifecycleWatcher::node_ok()`` treats an unread label as permission, deliberately, or a broken lifecycle service would silence ``qos_mismatch``, ``orphan`` and ``param_drift`` too - but permission is not knowledge. Once the asking stops the node IS tracked, provisionally, and a label arriving afterwards over ``~/transition_event`` and reading non-active takes it straight back out again. A node that is merely RE-WARMING is not that: warmup says nothing about who a node belongs to, so the gate distinguishes ``kUnclaimed`` (nothing known yet) from ``kDisowned`` (measured as another detector's) and only the second withdraws a key already held. That withdrawal happens inside ``node_death``'s own tick, and only on a tick that still sees the app present, so it leaves a window of about one entity-cache refresh in which a node that dies right after its label arrives is reported by the presence code anyway (measured: 210 ms between the label reaching the status route and the release). A withdrawal that HAS happened is final, which needed its own guard: a dying managed node loses its lifecycle services from the snapshot before it loses its App entry, and a node with no managed record answers ``kEarned`` - so dying erased the very measurement that disowned the node and re-admitted the key on the tick before it departed. A released key is therefore readmitted on a label reading ``active``, never on the mere absence of a label - but the refusal runs for a bounded window rather than the node's whole life: after ``miss_grace + 1`` consecutive record-less PRESENT ticks (this detector's own "absent this long means dead" span, already floored to ``kMinNodeDeathWindowMs`` of wall clock) the key is taken back, because a node still present past that window is not mid-death, it has simply stopped being managed - and holding the refusal open would leave its eventual death reported by nobody at all. It is a mis-attribution of a TRUE report, not a false positive, and it cannot be closed at report time: the departed record keeps only the last label, which reads ``inactive`` both for a node earned and then deactivated - a death this detector must report - and for one only ever held provisionally. Separating them after the fact needs per-key ownership history surviving the departure, which is the unbounded state the tracked-key prune exists to prevent. So the split is on whether the presence detector is tracking the node RIGHT NOW - read from the key set that detector publishes each sweep, not from the reliability gate and not guessed from the observed label: while it holds the key, a below-``grace`` streak that goes absent is simply HELD rather than matured on the strength of the absence alone - two codes standing at once for the same node is not a problem to fix, it is ``GRAPH_NODE_INACTIVE`` and ``GRAPH_NODE_DISAPPEARED`` saying different, both-true things. Reading membership rather than latching it is what makes the hold end when the claim behind it does: a key handed back on a measured disown, one reclaimed by ``prune()`` after a durable suppression, and one collapsed under the cap all leave the set, and the streak resumes climbing for each. A node outside the set gets no backstop - ``GRAPH_NODE_DISAPPEARED`` will not report it - so absence keeps advancing its streak exactly as a present tick would. Reporting a HEALTHY node that left remains ``GRAPH_NODE_DISAPPEARED``'s job as well, for either kind of node - a healthy departure never starts a violation regardless of ownership. **Content follows the clocks, not the snapshot.** Whether a node was in this tick's matches decides nothing about what it reports: a node past ``grace`` stays in ``GRAPH_NODE_INACTIVE``'s content while it blinks, and a matured node stays in its own code's content. A fault's content is what has been MEASURED about the node, and a missing snapshot entry measures nothing either way. **Presence is not enough here - and absence is someone else's fault.** A present-but-non-active managed node (``inactive``, ``unconfigured``, ``finalized``) is exactly the case ``GRAPH_NODE_INACTIVE`` exists to catch: on a Nav2 stack, a ``controller_server`` stuck ``inactive`` means the robot silently will not act, with no crash and no signal on ``/diagnostics`` or ``/rosout``. A node that vanishes having only ever been measured HEALTHY is the presence fault class's business, and this detector raises nothing for it; a node that vanishes while already reported under any of the three faults KEEPS that fault, because leaving the graph answers nothing the operator asked - the integration test's ``AbsenceAfterRaiseKeepsTheFaultAndSaysTheNodeIsGone`` and ``HealthyNodeThatVanishesRaisesNothingAtAll`` pin the two halves of that split. **Safe default: off.** ``require_active`` is empty by default, so the detector emits nothing at all - not even clears - until an operator opts specific nodes in (zero false positives, the same config-scoped posture ``param_drift``'s ``expect`` uses). **Keyed by node, not by config entry.** The bare-name form of ``require_active`` is deliberately fleet-wide ("all ``controller_server``\ s must be active"), so one entry legitimately covers N nodes. The tracker therefore keys every fact by the node's stable fqn: two namesakes are two report entries (one cannot silently replace the other, and a healthy one cannot mask a broken one), and the fault description names each offending node's FQN with the entry that demanded it as context - the description is the only carrier, since every ``GRAPH_*`` fault shares one ``source_id``. Two entries naming the SAME node advance its clocks once per tick, not once per matching entry. **Why grace, not the reliability gate (the design decision this detector turns on).** Every detector's raise is subject to the central ``reliability_allows(gate, source_id)`` gate inside ``ctx.raise_fault()``, which ANDs the entity's warmup state with ``LifecycleWatcher::node_ok()`` - true only when a managed node's lifecycle is ``active`` (or it is not tracked at all). Gating THIS detector's raise the same way on the required node's OWN lifecycle would be self-defeating: ``node_ok()`` is exactly FALSE for the inactive node this detector exists to report, so the central gate would suppress the signal forever. Instead the tracker counts consecutive not-active ticks itself, per node, entirely independent of ``reliability_allows``. This does not bypass bringup-quiesce entirely: the aggregated fault's own outer ``ctx.raise_fault`` call is still subject to the central gate for the aggregate's own ``source_id`` (``graph_watchdog``) - only the per-node lifecycle check bypasses it, because that check IS what this detector reports on. **Bounded by evidence, not by age.** The map is NOT simply bounded by the operator-declared ``require_active`` set: keys are node fqns, and one bare-name entry matches a node of that name in every namespace, so identity churn (nodes reappearing under ever-new fqns) would grow it without a bound. Since absence no longer erases anything, an entry carrying a live clock is not reclaimed by age either - those are the same defect seen twice, and moving a horizon rather than removing it leaves the evasion in place at a different N. So ``prune_ticks`` (the operator's ``prune_grace``, used as written - there is no ``grace + 1`` clamp any more, because there is nothing left for one to protect) reclaims IDLE entries only: both clocks at zero and no matured ownership, i.e. nothing to lose, and still atomically - ONE map entry per node, gone in the same tick, never partially. A non-idle entry is never pruned by age. An entry whose UNMEASURED clock is still climbing cannot grow without bound in time either: past the absence grace it advances every tick, so it matures within at most ``60 + absence_grace + 1`` ticks and is reported - the longest ``GRAPH_NODE_UNREADABLE`` or ``GRAPH_NODE_NOT_MANAGED``'s clear can be withheld by one departed node. A below-``grace`` VIOLATION streak on a node the presence detector is not tracking shares that same bound, for the same reason it advances at all while absent: it matures within at most ``grace + absence_grace + 1`` ticks, because nothing else will ever report that node's departure either. Only once a node HAS been owned does its below-grace streak lose the bound: absence then holds it rather than advancing it, so it neither matures nor becomes idle for as long as the node is away. That is not a new way to withhold ``GRAPH_NODE_INACTIVE``'s clear - the clear was already gated on every required node's status being settled, so one node this indecisive already blocked it; what changes is only that the fault never NAMES an OWNED node's departure, since that node's evidence belongs to ``GRAPH_NODE_DISAPPEARED`` instead. ``grace`` is still capped at 300 instead of being accepted up to ``INT_MAX - 1``, because it independently bounds how long a PRESENT node may go unreported and how long a returning node takes to re-mature: at the old maximum that PRESENT-side bound was roughly 24 days at the shipped cadence, with no warning - silence indistinguishable from a working detector finding nothing. What bounds the map is ``tracked_node_cap`` (default 512, ``kDefaultTrackedNodeCap``, accepted range 1..16384): at the cap idle entries are reclaimed first, then entries for DEPARTED nodes are collapsed into per-code COUNTS - lexicographically last first, keeping at most ``kMaxNamedDepartedEntries`` (3) named, since three maximally-long details are all one 480-character description holds - and only if every tracked node is PRESENT and carrying evidence is the NEWCOMER refused, never a live violation evicted. Collapsing follows from the fault being keyed by CODE rather than by node: five hundred entries for dead identities keep the same one fault raised that a single entry would, so holding them buys nothing while the slots they occupy can cost total blindness. Under identity churn a cap full of the dead would refuse a genuinely broken PRESENT node - which then never reaches ``affected`` or ``pending``, is not covered by the never-matched hold either (its entry HAS matched), and so ``GRAPH_NODE_INACTIVE`` would emit a level-triggered CLEAR every tick while that node sat there not-active. The count is content, so collapsing an entry heals nothing; it is ordered ahead of the individually named entries but behind anything crossing on this tick, so a node that just broke is never displaced by it. It only grows within one tracker lifetime: a collapsed entry's fqn is no longer known, so a node returning under it is tracked and measured afresh. A refused node is a required node going unchecked, so refusing one is never silent: it withholds ``GRAPH_NODE_INACTIVE``'s clear for as long as it lasts, is logged once per saturation EPISODE (the latch re-arms when the episode ends, so a later, real saturation is not silenced by an earlier one), and is reported on ``GET /x-medkit-watchdog`` under ``detectors.lifecycle_expectation``. That withhold is deliberately UNBOUNDED, unlike the never-matched hold beside it: the never-matched hold is bounded because a typo must not block healing forever, while saturation - once departed entries can no longer crowd out present ones - means genuinely more required PRESENT nodes than the cap allows, a capacity condition the operator resolves rather than a transient that resolves itself. **The clear is withheld until the required set has actually been measured - and this is entirely about** ``GRAPH_NODE_INACTIVE``. The other two faults have no withheld-clear guard of their own; see "Three independent faults, not one shared record" below for why they don't need one. A clear asserts that every required node is free of a CONFIRMED violation. That is the restart-heal hazard: a gateway restart brings every detector counter back to zero while the ``GRAPH_NODE_INACTIVE`` raised before it is still in the fault_manager's store (a separate process), so a clear emitted on the strength of an empty affected map heals a fault that is still real. TWO things produce that empty map without the assertion being true, both the ordinary state of bookkeeping that just started over (a restarted gateway, a reconfigure - ``configure()`` rebuilds the tracker - or a node that respawned stuck): - **Not matched yet.** Before the entity snapshot catches up with the graph a ``require_active`` entry matches nothing at all, so the clear is about a node the detector has never once looked at. The plugin ticks as soon as it is loaded, so every restart passes through this window. Bounded the same way node-keyed state is (60 ticks), so a misspelt entry cannot block healing for the process lifetime. - **A node's status is UNSETTLED** (the tracker's own ``pending`` set). A violation streak that has not yet passed ``grace``, an unmeasured clock still climbing under EITHER cause, or a streak HELD while the node is inside an unmeasured spell. Absence never puts a node here on its own - content follows the clocks, so a node already past ``grace`` stays in the fault's content through a blink rather than dropping into a withheld limbo. A node whose unmeasured clock has MATURED is deliberately NOT in this set - ownership passed to its own fault code and the violation streak was released, so it stops counting toward this withhold the exact tick it stops being uncertain, rather than continuing to poison ``GRAPH_NODE_INACTIVE``'s withhold decision the way a shared record used to. Either reason withholds the emission entirely, neither raise nor clear. A raise is never withheld: a violation read from the nodes that DID answer is real regardless of the unmeasured ones. The never-matched leg and both unmeasured causes are bounded: they release after 60 consecutive ticks (a minute at the shipped cadence, mirroring ``param_drift``'s frozen hold), whether the node is present or gone. The pending leg releases the same way - as soon as the node reads ``active``, or a PRESENT tick pushes its streak past grace and the fault is raised again - but while the node stays absent that release has no timeout of its own: absence holds a below-grace streak rather than advancing it, so the hold lasts for exactly as long as the node does not return. Every hold releases by SETTLING the node's status, never by giving up on it. Because a correctly withheld clear and a detector with nothing to report look identical from outside, a hold that lives past 10 consecutive ticks is explained in the log once per episode, naming every reason in force and, for the node-keyed ones, the count behind each and one node by name - the not-managed and unreadable reasons are still named separately even though both now release the same way (into their own fault code), since an operator reading the log wants to know WHICH of the two is happening. **Three independent faults, not one shared record.** ``GRAPH_NODE_INACTIVE``, ``GRAPH_NODE_UNREADABLE`` and ``GRAPH_NODE_NOT_MANAGED`` are each raised through the shared ``AggregatedFault`` helper every ``GRAPH_*`` detector uses (one graph-level record per code, since the fault_manager identifies a fault by ``fault_code`` alone), but as three SEPARATE, fixed-severity class members - ``GRAPH_NODE_INACTIVE`` always ``SEVERITY_ERROR``, the other two always ``SEVERITY_WARN`` - the same shape ``orphan_detector`` and ``param_drift_detector`` use, rather than one record whose severity is chosen from that tick's mixed content the way ``qos_mismatch_detector``'s ``any_starved ? kStarvedSeverity : kPartialSeverity`` does for its own single code. Raises are fully independent: each fault's content comes from its own measurement, and one raising, healing, or changing severity never forces, blocks, or reflects onto another. ``GRAPH_NODE_INACTIVE``'s own clear is not simply "nothing CONFIRMED non-active this tick" though - it is withheld exactly as described above; that is the ORIGINAL withheld-clear guarantee this detector always gave, scoped to ``GRAPH_NODE_INACTIVE`` alone. The other two have no such guard: each one's own clear needs nothing beyond its own content going empty, because once the unmeasured clock has matured "still cannot be measured" is a settled fact, not a pending one. A node is content of at most one of the three at a time, and healing one never forces, blocks, or changes the severity of another. Content under either unmeasured code survives for as long as a node stays that way, with no further bound past the initial hold. **A re-bind is a fresh binding.** ``LifecycleWatcher`` keys its tracked map by ``App::id``, and an id can survive a graph sweep while pointing at a DIFFERENT node (id assignment shifts under bare-name collisions) - an entry kept across such a move would keep enforcing the old node's label, and its old ``~/transition_event`` subscription, against the new binding. An entry's binding identity is its fqn plus its ``GetState`` service path, both captured at first sighting, and ``update()`` re-checks that identity every tick. A moved binding is two events at once: the OLD binding departed (recorded under ITS OWN fqn in ``recently_departed_``, same record and retention as a vanish), and the id is new again (erased and re-seeded through the ordinary new-node path: fresh ``GetState``, fresh subscription, fresh self-heal budget). For this detector the consequence is that a re-bind is never enforced with the departed node's label: the new binding starts unknown (benign) until its own label is read. No straggler from the old subscription has to be filtered out of the new entry. Erasing the entry destroys that subscription, on the tick thread: the private executor that runs these callbacks is pumped by the same thread that runs ``update()`` (see the plugin-shell bullet above), so nothing else holds a reference to it and no message it had queued is delivered afterwards. The same property removes the other ordering question this seam used to carry - a re-seed's blocking ``GetState`` cannot be overtaken by a ``~/transition_event``, because while ``update()`` blocks nothing is pumping the events at all. **New violations are named first, not alphabetically - in EACH fault's own description independently.** The description used to list affected nodes in fqn order (``AggregatedFault::emit``'s default), which is fine when every entry is equally interesting but not once the cap is full: a fleet sharing ``require_active: ["controller_server"]`` across a dozen robots fills the 480-char cap from the alphabetically-earliest ones, and a THIRTEENTH robot going inactive afterward would be silently invisible forever - one shared ``fault_code``, one record, no way to tell the operator which of thirteen actually broke. The tracker reports which fqns entered EACH fault's content on THIS tick - ``newly_affected``, ``newly_unreadable``, ``newly_not_managed`` - and each list orders only its OWN fault's ``AggregatedFault::emit_ordered`` call, since the three faults never share a description: a fresh entry is named FIRST in whichever fault it belongs to; every other affected node in that same fault still appears, in the same fqn order as before, once the fresh ones are placed. When every node crosses on the same tick (``grace: 0`` during a bringup burst, for instance), there is nothing to distinguish them by and the order degrades to fqn order. Two budgets protect each description, applied before it ever reaches the 480-char cap: the lifecycle label - which arrives verbatim off a remote ``~/transition_event`` and is therefore untrusted and unbounded, and only ever appears in ``GRAPH_NODE_INACTIVE``'s own detail, never the two unmeasured faults' - is trimmed to 32 characters before it is interpolated (more than double the longest label a conforming implementation produces, ``errorprocessing`` at 15 characters), and the whole per-node detail is then capped at 150 characters as a backstop against a pathological fqn or a long ``require_active`` "required by" list, sized so at least three worst-case details still fit inside the 480-char cap (``3 * 150 + 2 * 2 = 454 <= 480``). **Test tiers.** Three tiers each prove a different layer, deliberately not overlapping - plus the shared-watcher seam the re-bind behaviour lives in: 1. **Unit** (``test_lifecycle_expectation_tracker.cpp``): pure ``LifecycleExpectationTracker`` logic over hand-built matches. Covers the violation streak (stuck-inactive past grace raising keyed by the node with the entry as context, active never raising, reaching active within grace resetting the streak, both namesakes reported separately, two entries not halving the grace, a violating read winning a duplicate match's tie-break); the unmeasured clock shared by UNREADABLE and NOT_MANAGED (neither ever confirms a violation while climbing; each matures into its OWN fault at the exact hold boundary; the clock resets ONLY on a real measurement, in both directions; maturity RELEASES the violation streak entirely, so a node returning to inactive re-earns grace from zero; the sticky-cause decision, pinned directly against the rejected current-cause-wins alternative); the alternation this redesign closes - a node alternating between unreadable and not-managed, indefinitely and on every single tick, still matures the shared clock; the ``INACTIVE``/``NOT_MANAGED`` alternation across absence gaps longer than the absence grace, now counting only the MEASURED not-active legs; the violation streak surviving non-maturing unmeasured ticks and RESUMING rather than restarting, counted exactly and read through ``pending_violation``; absence CONTINUING an already-matured fault exactly as it was on its last present tick - a blink holds either clock unchanged; past the blink tolerance an unmeasured clock still climbing keeps climbing and matures on absence alone, while a below-grace violation streak is instead HELD and RESUMES rather than restarts once the node returns, proven both from one long absence and from many short ones interleaved with matched reads; a matured violation fault survives a departure, a node measured ACTIVE that vanishes raises nothing and is reclaimed, and the two ``(UNREADABLE/NOT_MANAGED, ABSENT x N)`` restart-loop shapes are swept at the absence grace and past it (their ``INACTIVE`` sibling pinned to the opposite claim at the same two N: it never crosses grace from absence alone, for a node the presence detector is tracking - a node outside its key set gets the opposite result instead, maturing from absence alone within ``grace + absence_grace + 1`` ticks exactly like the unmeasured clock does, and resuming rather than restarting on return the same way an owned node's held streak does); the ``pending`` set and its per-reason breakdown; new-first ordering for all three fault-shaped maps; the remote-supplied label's own trim budget ahead of the whole-detail backstop, reapplied to a matured unreadable node's detail (which carries no label, only a fqn and a "required by" list); the age horizon reclaiming IDLE entries only, atomically, and never one that carries evidence even when it undercuts the absence grace; the SETTLING rule that decides what absence may continue - an uncorroborated unmeasured run before a healthy departure raising nothing, swept across every run length below the bound and with the entry released rather than left holding the clear hostage, the same run one tick longer still being reported, and a single MEASURED not-active read before a departure still confirming; and the tracked-node cap - idle entries reclaimed first, departed entries collapsed into a count so a present broken node is always admitted and reported, at most three left named, the count surviving into the description as content, a node returning after its entry was collapsed measured afresh, the newcomer refused only when every entry is PRESENT and carrying evidence, and saturation reported as a LEVEL on every refused tick with its edge re-arming when an episode ends - swept at a shrunk cap and again at the real shipped 512. The re-bind seam is shared infrastructure and is pinned separately in ``test_lifecycle_watcher.cpp``. 2. **Integration** (``test_lifecycle_expectation_integration.cpp``): the detector driven against a fake ``ReportFault`` service, with a REAL ``ReliabilityGate`` arming the global bringup grace and feeding the detector its labels through ``lifecycle_state_of()`` - injected via ``set_lifecycle_state_for_test()`` strictly AFTER the last ``gate.update()`` call for most cases, or, for the alternation and not-managed cases (which need a genuine ``nullopt`` the injection seam cannot produce - it can only ever SET a tracked value, never remove tracking), by toggling whether the matched app carries lifecycle services and driving the gate's real discovery path instead. The fixture cases cover the ``GRAPH_NODE_INACTIVE`` raise/clear round trip naming the stuck node; the active-from-arming positive control; the fault SURVIVING the required node's departure while a healthy node's departure raises nothing at all; bare-name and full-FQN matching against a namespaced app; the zero-config default measured as zero fault_manager requests of any kind; the unmanaged-entry and no-match warnings; the withheld-clear guard's releases and its once-per-episode log line; the blink-plus-unread-re-seed sequence; a filler batch sized (from the real detail-building code) to exceed the 480-char cap aggregating into one fault with the fresh crossing named first; a PRESENT node crossing grace fresh while a batch of already-departed, already-matured ones sits absent-and-content, named ahead of them rather than truncated away by them; a required node appearing mid-run; a re-bind under the same ``App::id``; and the reconfigure/config-validation edge cases. The unmeasured clock's own split is pinned directly, for BOTH codes symmetrically: a managed node whose ``GetState`` genuinely never answers (through ``set_managed_app`` and a real, failing seed - proven by asserting ``lifecycle_state_of()`` actually returns ``optional("")`` first, not assumed) is reported under ``GRAPH_NODE_UNREADABLE``; a node with no tracked lifecycle at all is reported under ``GRAPH_NODE_NOT_MANAGED`` (no longer released into silence, the deliberate behaviour change this redesign makes); either hold releases into its report on the exact tick past its bound; a node already reported under one of the two clears once genuinely read, returning to ordinary ``GRAPH_NODE_INACTIVE`` tracking; a confirmed node healing while an unreadable OR not-managed sibling stays present clears ``GRAPH_NODE_INACTIVE`` promptly without touching either unmeasured fault; content under either code survives with no window and no expiry; 25 nodes of either cause aggregate into one capped fault at ``SEVERITY_WARN``; a node whose hold expires opens its fault's own description over an already-reported filler batch with new-first ordering; an already-reported node of either cause KEEPING its own record once it vanishes, with its description switching to say the node has left the graph; a node returning from that absence staying reported with no clear/re-raise churn; each of the two UNMEASURED restart-loop shapes raising its own code from absence alone (their ``INACTIVE`` sibling instead pinned to counting only real reads, never crossing from any of the interleaved absence gaps), and the ``inactive``/``not-managed`` alternation across absence gaps now counting only the MEASURED not-active legs; a below-grace streak surviving the tightest ``prune_grace`` without being confirmed while the node stays absent, and resuming (not restarting) once it returns; a withheld ``GRAPH_NODE_INACTIVE`` clear releasing when the absent node's clock MATURES into its sibling's content rather than when the node is given up on; the two wire strings pinned as hand-typed literals; and the independence claim in both directions. The ``configure()``-level cases pin the config contract: the unknown-key warning for ``require_activ`` (the worst-case typo - it also leaves the detector unconfigured, so the warning must precede the zero-config early return), a fully-valid config producing zero warnings (including ``grace: 0`` and ``prune_grace: 0``, the documented low endpoints), negative, non-integer, past-the-int-range and exact-``kMaxGrace``-boundary ``grace`` (both sides), ``prune_grace`` out of 0..3600 rejected on the WIDE integer (never truncated into a hair-trigger prune) plus both range endpoints accepted, non-array ``require_active`` and empty-string and non-string entries each warning, the unclamped prune horizon reaching idle bookkeeping at exactly the configured ``prune_grace`` (0, 1 and 4 - the smallest positive value included, since neither documented endpoint sweeps it) while a node carrying evidence survives it - including the ``grace: 0, prune_grace: 0`` corner (an UNMEASURED clock, the only kind that keeps climbing while absent) and a wide ``grace`` beside the tightest ``prune_grace`` for a below-grace VIOLATION streak, whose instrument is that it is neither confirmed nor pruned while the node stays absent, and resumes rather than restarts once it returns - ``tracked_node_cap`` validated at both range endpoints and one value past each with the key proven IN FORCE at both ends, boundedness under identity churn at the real 512-node cap by collapsing the departed rather than refusing the live node, and saturation reported once per EPISODE with a second episode reported again after the first ends. 3. **E2e** (``test/e2e/test_lifecycle_expectation_e2e.test.py``): the acceptance gate - one source file, TWELVE CTest targets (the config-plumbing pattern: the plugin reads its config once at ``set_context()``, so different configs need different gateway launches), each bringing up a real gateway with the plugin ``.so`` and a real fault_manager, asserting on the operator-visible ``GET /api/v1/faults`` surface. Six scenarios drive the ``managed_lifecycle`` demo node (a real ``rclcpp_lifecycle::LifecycleNode``, which always answers ``GetState``); two, ``unreadable`` and ``departure_keeps``, drive a fixture built to never answer it (see below); the last two, ``not_managed`` and ``restart_loop``, drive ``calibration`` - a PLAIN demo service node with no lifecycle interface at all, so no purpose-built fixture was needed for ``GRAPH_NODE_NOT_MANAGED``, unlike ``GRAPH_NODE_UNREADABLE``. The main scenario proves raise-survives-CONFIGURE-survives-RESTART-heals-on-ACTIVATE through real ``lifecycle_msgs/srv/ChangeState`` transitions, including the entity-scoped ``/apps/graph_watchdog/faults`` surface. The restart leg is the withheld-clear guard at the only tier that can reach it: the gateway is SIGTERMed, its port is waited down, the relaunched process is gated on being armed again, and the fault about the still-inactive node must come back with ``last_passed`` unset. The default-config scenario launches with no ``lifecycle_expectation`` config at all against the same inactive node and holds a sustained silence window, and the negative control does the same with the self-activating variant of the same executable; the discriminating variable is the node's actual lifecycle state. Both silence scenarios gate on three facts before asserting absence: the plugin is armed; ``GET /faults`` answers 200 in THIS launch; and the target's label was actually READ, pinned through the plugin's own ``GET /x-medkit-watchdog`` route. The fourth scenario, ``healing_threshold``, is the withheld-clear guard's pending leg at the only tier that runs the REAL fault_manager debounce state machine at all. The required node, already reported stuck, is SIGTERM'd and respawned twice under the same name (a real snapshot blink, not a lifecycle transition), each one comfortably inside the tracker's fixed absence grace, against a ``healing_threshold`` of 1. This same run is also this package's only e2e coverage for a node oscillating readable/unreadable producing no ``GRAPH_NODE_INACTIVE`` event churn, read raw off ``GET /faults/stream``: no ``fault_cleared`` frame for ``GRAPH_NODE_INACTIVE`` at any point, and every ``fault_confirmed``/ ``fault_updated`` frame carrying ``severity_label: "ERROR"`` - its only possible value now that severity is fixed per code. The fifth scenario, ``unreadable``, is where the 60-consecutive-tick failure that matures ``GRAPH_NODE_UNREADABLE`` is actually reached. It launches ``unreadable_lifecycle_node.cpp``: a plain ``rclcpp::Node`` that advertises ``get_state``/``change_state`` with the service TYPES ``find_lifecycle_get_state_path`` actually checks for, whose ``get_state`` handler stores every request and never answers until its own ``start_answering`` parameter is set true. Against that fixture the scenario proves the node is present and matched with its lifecycle label read as ``""``; ``GRAPH_NODE_INACTIVE`` stays silent for it throughout; ``GRAPH_NODE_UNREADABLE`` raises once the hold expires, names the node, and carries ``SEVERITY_WARN`` - a node is content of at most one of the three, never more than one; and once the fixture starts answering ``"active"`` the watcher reads it through a real ``GetState`` round trip and ``GRAPH_NODE_UNREADABLE`` clears. The sixth scenario, ``departure_keeps``, proves what a DEPARTURE does to an already-reported unmeasured fault: nothing. The same fixture is SIGTERM'd permanently once ``GRAPH_NODE_UNREADABLE`` has raised, confirmed gone from ``GET /apps``, and past every horizon that could have discarded its evidence the fault is still active, has never once been reported PASSED (``last_passed``, which catches even a transient clear), and its description now says the node has left the graph. The seventh, ``not_managed``, is the sibling proof for the OTHER cause: ``calibration`` is present and matched from launch, ``GRAPH_NODE_NOT_MANAGED`` raises once its hold expires naming the node at ``SEVERITY_WARN``, and neither ``GRAPH_NODE_INACTIVE`` nor ``GRAPH_NODE_UNREADABLE`` ever raises for it - the one live proof that the NOT-MANAGED cause specifically never bleeds into the UNREADABLE code it shares a clock with. Its departure leg mirrors ``departure_keeps``'s own shape. The eighth scenario, ``restart_loop``, is the acceptance gate for the whole evidence-retention model. The same plain ``calibration`` node, but respawning, is SIGTERM'd over and over on a cadence that never lets it accumulate the 60 consecutive PRESENT ticks the unmeasured hold would otherwise need. Every cycle's absence is proven rather than assumed - ``GET /apps`` is polled until the node is gone and again until it is back, and the measured gap must EXCEED the 3-tick absence grace, on top of launch's own enforced ``respawn_delay`` floor - so every cycle genuinely crosses the horizon past which evidence used to be discarded. ``GRAPH_NODE_NOT_MANAGED`` must raise anyway and survive the restarts that follow. A required node in a crash loop is the case this detector most exists to catch, and the one a design that discarded evidence on absence made permanently silent. The last four scenarios are about the bounds themselves. ``cap_pressure`` runs ``tracked_node_cap: 1`` against TWO required nodes, so one is refused on every tick - reachable only because the cap is a config key; against a compile-time 512 it would need 513 real lifecycle nodes. It proves the refusal is visible on ``GET /x-medkit-watchdog``, that it WITHHOLDS ``GRAPH_NODE_INACTIVE``'s clear (measured through ``last_passed``, which catches even a transient clear, at the moment the tracked node's unmeasured clock matures and that fault's content goes empty), and that the entry for a node that then LEAVES is collapsed so the present, still-broken node is admitted and named. ``unsettled_departure`` runs two healthy managed nodes built from this package's own ``droppable_lifecycle_node.cpp`` - which looks managed, answers a chosen label, and stops advertising its lifecycle services on command - and drops the services of each: one is killed immediately (a single missed sweep before a clean shutdown, about which nothing may ever be reported) and the other holds the dropped state past the settling budget before being killed (genuinely not managed when it left, so it must still be reported). The first leg's window is MEASURED against the settling budget rather than assumed, so it cannot silently turn into the second. ``wide_grace`` configures the value that used to be the accepted ``grace`` maximum and proves it is refused and the documented default applied - under it the detector could neither raise nor heal for days. ``restart_departed`` records where "a departure never heals a fault" ends: the required node is killed while its fault is outstanding, the gateway is restarted, and the record then heals - because the restarted detector cannot tell an entry for a departed node from a misspelt one. ``node_death`` raises ``GRAPH_NODE_DISAPPEARED``. It watches every ARMED App in the graph for one that goes offline - zero-config, unlike ``lifecycle_expectation``'s operator-declared ``require_active`` list, because every armed App is a candidate. Liveness is ``App::is_online``, never mere membership in the entity snapshot: in runtime-only discovery the two happen to coincide (a dead node's App leaves the snapshot entirely), but a manifest keeps a bound App present with only ``is_online`` cleared once its ROS binding disappears, and hybrid discovery inherits that shape - counting snapshot membership alone would make a manifest-declared node immortal. A managed lifecycle node that merely deactivates keeps ``is_online: true`` (its process is still running, only its ROS 2 lifecycle state changed), so a deactivation is never mistaken for a death either - that is ``lifecycle_expectation``'s concern, not this one's. Tracking is keyed on the STABLE fqn (``App::effective_fqn()``), never ``App::id``: an id is recomputed every sweep and only gains a namespace prefix once a same-bare-name collision currently exists anywhere in the graph, so a live node's id can change out from under a key built from it. **What is never tracked.** A peer-aggregated app (``app.source`` starting ``peer:``) carries no ROS binding of its own, so ``effective_fqn()`` is empty for every one of them; tracking them would collapse an entire peer fleet onto one ``""`` key. A ``_ros2cli_`` node - rcl's own hidden-node naming convention for every ``ros2`` CLI invocation, matched structurally rather than guessed at - is excluded too, or every CLI invocation would accumulate one permanently "dead" entry for the life of the gateway. An App that never comes online is skipped by the ``is_online`` filter above and so is never admitted to tracking in the first place - the identical protection a node still inside ``warmup_cycles`` gets. A MANAGED App whose lifecycle state has not been read YET is excluded too, and for a different reason: the gate's raise permission for it is deliberately PERMISSIVE (an unread label must not silence ``qos_mismatch``, ``orphan`` or ``param_drift``), but this detector reports an ABSENCE and cannot attribute the departure of a node whose state nothing has read - so tracking asks ``presence_ownership()`` instead. That exclusion is BOUNDED, and the readmission that follows it is PROVISIONAL. The watcher charges a GetState re-seed budget per node and charges it only for a read that actually ran, so an empty label with attempts left is "we have not finished asking" and an empty label with none left is "we asked and failed". Past that point the node is admitted here after all - nothing will ASK again, and ``lifecycle_expectation`` returns before it looks at anything unless an operator named the node in ``require_active``, which defaults to empty, so refusing forever would mean a whole class of deaths reported by nobody. But asking is not the only channel: the ``~/transition_event`` subscription is created independently of the seed budget and is never torn down when it runs out, so a label can still arrive unasked. One that reads non-active says the node was ``lifecycle_expectation``'s all along, and this detector releases the key while the node is still alive rather than reporting a death it was never entitled to. Ownership EARNED from a measurement is not released that way: a node that ran and then deactivated is still a death when it stops running. Stickiness is earned by knowledge, never by ignorance. What that does NOT close: an App counts as managed only once the snapshot shows it advertising a ``GetState``-typed service, and until then it is indistinguishable from a plain node, so a managed node whose services have not yet been discovered can still be taken for one. That window is bounded by the arming warmup, which already needs several consecutive ticks of presence, and node and services come from the same introspection sweep. Nor does anything here close the case of a node that appears, lives fewer than ``warmup_cycles`` ticks and vanishes: the gate has always required completed warmup before it arms anything, so such a node is owned by nobody and reported by nobody. That is the bringup-quiesce trade-off working as designed, not a gap this boundary introduces. Once tracked, though, a node's continued life-or-death judgement rests on PRESENCE alone, not on staying armed: a tracked node that later goes lifecycle-inactive is not thereby mistaken for dead. Composable/component nodes hosted in one process die together - if the container process dies, every node it hosted leaves the snapshot on the same tick, and all of them are named in the one aggregated fault rather than raising one fault each. **The wall-clock floor on** ``miss_grace``. The entity snapshot a tick reads is not rebuilt every tick: runtime discovery rebuilds it off a graph event debounced to about one refresh per second, so several consecutive ticks between two refreshes see the IDENTICAL snapshot. A ``miss_grace`` counted purely in ticks does not count independent samples of the graph - shorten ``tick_interval_ms`` enough and one stale cache generation gets re-counted as several misses. The floor raises ``miss_grace``, for the configured ``tick_interval_ms``, to the smallest value whose ``(miss_grace + 1) * tick_interval_ms`` window is still at least 3000 ms - one graph-cache refresh cycle plus margin. At the shipped 1000 ms tick the floor is exactly the shipped default (``miss_grace: 2``), so the default configuration is unaffected; only a faster-than-default tick ever raises the effective grace, with a warning naming the window it actually spans. **Suppression: opt-in, and only by name.** Nothing is suppressed unless the operator names the mechanism in ``suppress`` - a configured ``allowlist`` that ``suppress`` does not name has NO effect, and a startup warning says so. Two mechanisms exist, activated independently of each other: - ``"allowlist"`` builds an ``AllowlistSuppressor`` over the configured ``allowlist`` set - the same three-way match (fqn, bare leaf, ``App::id``) ``lifecycle_expectation``'s ``require_active`` uses, so an operator who has one working can expect the other to accept the same shapes. Exact match only, never a prefix or a shared suffix. - ``"lifecycle"`` builds a (stateless) ``LifecycleShutdownSuppressor``: it suppresses a departure the reliability gate classifies as a clean managed-lifecycle shutdown - see "Clean-shutdown suppression" below for what counts as clean. Every field is re-validated from scratch on every ``configure()`` call, so a malformed ``allowlist`` entry, an unrecognized ``suppress`` name, or a wrongly-typed entry each produce their own named warning rather than being silently dropped. A candidate is suppressed the moment ANY active mechanism votes yes, order-independent - which one runs first never changes the result. Suppression is independent of ``mode``: a suppressed key never becomes content in the first place, while ``mode`` separately decides whether non-suppressed content is actually sent. **Durability, and why only a durable veto may reclaim bookkeeping.** Whether a suppressor is durable answers whether its "yes" is a standing fact about the key rather than a condition that can later stop holding. Both mechanisms here are durable: an operator-declared allowlist entry does not start matching and then stop on its own, and a departure's observed shutdown shape does not change after the fact. Durability is what makes reclaiming a suppressed key's tracker bookkeeping (``prune_grace`` consecutive suppressed ticks) sound - a NON-durable veto could lift later even after its key's bookkeeping was discarded, leaving a live, unsuppressed death with nothing left to report it: a false heal that silently outlives the very condition that produced it. A durable veto never lifts for a given key once it has fired, so reclaiming under it loses nothing. ``Suppressor::durable()`` defaults to false, the safe assumption for a suppressor nobody has reasoned about yet; both suppressors here override it explicitly to true. Reclaiming is re-checked against the durable suppressors specifically, never merely "no longer in this tick's report", because a key can be dropped from a tick's report by ANY suppressor in the chain while a durable one also happens to cover it - whether a key may be reclaimed depends on WHICH suppressor vetoed it, not on whether it survived the filter. The generic chain is reusable (``Suppressor`` plus the free function ``apply_suppressors()``, independently unit-tested), but this detector's own filtering is hand-inlined rather than a call to that helper, because it has to interleave the id-form check above - answerable only while the node is still present - with the ordinary fqn/leaf dispatch the helper's plain string-keyed signature does not have room for. **Clean-shutdown suppression - what counts as clean.** ``LifecycleShutdownSuppressor`` reads the lifecycle watcher's cached label for the departed node's LAST observed transition. ``shuttingdown`` alone suppresses - it is only reachable via a deliberate SHUTDOWN transition, so the label alone is enough. ``finalized`` suppresses only WITH an observed transition and never through the error branch; without an observed transition, or reached through the error branch, it does not, because ``finalized`` is also where a node's ``on_error`` override lands after ``ON_ERROR_FAILURE``/``ON_ERROR_ERROR`` out of ``errorprocessing`` - the standard way a driver reports a hardware fault it cannot recover from, which is precisely the death an operator needs reported, not silenced. ``unconfigured`` is deliberately excluded even though it looks quiet: it is also the resting state of a node whose ``configure()`` failed or that never activated at all, and suppressing on it would hide exactly that startup failure. Any other label, or no departure on record at all, abstains. The suppressor itself is stateless by construction - a detector's ``configure()`` runs before any per-tick context exists, so it stores nothing at construction and reads the gate fresh on every call. **The verdict is latched; the evidence is not.** ``node_death`` remembers that a durable suppressor vetoed a given departed key and stops re-asking while that key stays tracked and absent. The label the suppressor reads is retained for a bounded number of ticks, while the veto must hold to ``prune()``'s reclaim - which needs it UNBROKEN for ``prune_ticks`` consecutive ticks. Re-asking every tick made those windows race, and the retention clock is anchored on the wrong event to win it: it starts when the node's lifecycle SERVICES leave the graph, while the reclaim tick counts from when its App does, one or more sweeps later. Once the label expired mid-veto the suppressor abstained, the streak reset, and with the label gone for good the key could never be suppressed again - a cleanly shut-down node named for the life of the process. Latching is sound because that is what ``durable()`` means: the verdict never lifts for a given key, so remembering it cannot diverge from asking again. Only positive verdicts from durable suppressors are latched, and an entry is dropped the moment its key is present again, so one departure's verdict never carries into the next. The retention window is still not this detector's own to size: the plugin predicts the same ``prune_ticks_`` (``max(prune_grace, miss_grace + 1)``) and sizes the watcher's departed-node retention from it. With the verdict latched, that window only has to outlive the FIRST tick suppression is evaluated on; it stays sized to the reclaim tick as slack against the services-before-App lead (``GraphWatchdogPlugin::compute_departed_retention_ticks()``). **Bounded by evidence, not by age - and what the cap actually bounds.** This detector is zero-config over every armed App in the graph - a strictly larger scope than ``lifecycle_expectation``'s named ``require_active`` set - so identity churn would grow the tracker's map without a bound if nothing capped it. An unsuppressed, still-dead entry is never reclaimed by age (``prune_grace`` only ever reclaims a DURABLY suppressed key), so ``tracked_node_cap`` (default 512, accepted range 1..16384) is what bounds memory - specifically the DEPARTED subset of the map. A PRESENT/armed key never counts against the cap and is never evicted to make room for anything, at any map size (``NodeLivenessTrackerCap.PresentEntriesExceedingTheCapAreAllStillIndividuallyReportableOnDeath``): it is bounded by the live graph rather than by churn, so a graph far larger than the cap is not capacity pressure at all. Under pressure the cap collapses departed entries into one synthetic count, keeping at most three individually named (``NodeLivenessTrackerCap.MaturedDepartedEntriesAreCollapsedIntoACountUnderCapPressure``) - but only entries that have actually crossed ``miss_grace``. An entry still mid-grace is never a collapse candidate, because collapsing erases the identity and would report a death the node has not earned, permanently (``NodeLivenessTrackerCap.ImmatureDepartedEntriesAreNeverCollapsedOrReportedUnderCapPressure``). Collapsing is one-way, and the cost is worth stating plainly: the identity is DISCARDED, the collapsed count only grows within a tracker's life, and nothing decrements it when one of those nodes returns. So the description stops naming them, and - since the aggregated fault clears only on an EMPTY dead set - the synthetic collapsed entry keeps that set non-empty, which means ``GRAPH_NODE_DISAPPEARED`` cannot clear again for the life of the process once anything has been collapsed. It ends by operator acknowledge or by a restart, never by the graph recovering. Neither half is repairable here: decrementing needs the identity the collapse threw away, and keeping identities is the unbounded state the cap exists to prevent, while decaying the count by time would heal a fault by waiting - which this package refuses everywhere else. What bounds the damage is what collapsing may touch: only a MATURED entry, whose death has already been reported by name, so nothing becomes UNREPORTED by being collapsed. The attribution and the self-healing are lost, not the warning. So saturation IS reachable here: when immature departures alone exceed the cap, ``tracking_saturated`` goes true and the departed set stays oversized for a few ticks - bounded by how fast departures arrive times ``miss_grace``, not by process uptime, which is the growth the cap exists to prevent. What differs from ``LifecycleExpectationTracker`` is not whether the flag can fire but what it costs: there a refused newcomer withholds ``GRAPH_NODE_INACTIVE``'s clear, whereas here nothing is refused and no report is lost, the map is merely temporarily larger than its target. **The boundary with** ``lifecycle_expectation``. ``GRAPH_NODE_INACTIVE`` and ``GRAPH_NODE_DISAPPEARED`` can both be raised for the same node at the same time, and that is CORRECT, not a defect this detector removes: ``GRAPH_NODE_INACTIVE`` says a required node is present but not active, ``GRAPH_NODE_DISAPPEARED`` says a node is gone, and an operator facing each has a different repair - reactivate it, or find out why it left. A node CONFIRMED ``GRAPH_NODE_INACTIVE`` that then departs keeps that fault (its description switches to say the node has since left the graph) AND now also raises ``GRAPH_NODE_DISAPPEARED``, since this detector tracks every App it owns independently of whatever ``lifecycle_expectation`` thinks of it. What changed is narrower than that, and it only applies to a node this detector could ever have tracked. Before this detector existed, ``lifecycle_expectation``'s own violation streak let a SUSTAINED absence mature an unconfirmed (below-``grace``) streak into a confirmed ``GRAPH_NODE_INACTIVE`` - the only way a node stuck in a restart loop, never observed long enough to cross ``grace`` while present, would ever be reported at all. Where this detector can independently report the same departure - which needs the presence detector to have OWNED the node at least once, since it only ever tracks an App the reliability gate has admitted for ownership, and ownership needs a MANAGED node's lifecycle state known to read ``active`` or asked for as often as it ever will be, plus the node being online - that absence-alone maturity is gone: absence CONTINUES a violation that has already matured past ``grace`` (the fault stays raised, still naming the node), but no longer CREATES one that has not. A node that was briefly non-active and then died is reported as gone (``GRAPH_NODE_DISAPPEARED``) rather than also acquiring an inactive fault born from ticks gathered while nobody could observe it. A ``require_active`` node that never reaches ``active`` is never admitted, is never tracked here, and its departure can never raise ``GRAPH_NODE_DISAPPEARED``; the same holds for an app that is not online. For those, ``lifecycle_expectation`` keeps the older behaviour: absence still matures a below-``grace`` streak, because it is structurally the only detector that will ever get to report them. See ``lifecycle_expectation``'s own "Absence continues the last CORROBORATED observation, it never erases" section above for the mechanism this rests on. **Repeated failures: what** ``occurrence_count`` **and the captured evidence answer, and don't.** An operator asking "how many times did this node die in the last hour" reads it off the fault record itself, not off this detector: ``GRAPH_NODE_DISAPPEARED``'s own ``occurrence_count`` starts at 1 on the first raise and increments by exactly one each time a FAILED report reactivates a record that was CLEARED - never merely re-raised while still CONFIRMED, and never merely HEALED. Healing (the fault manager's debounce counter crossing ``healing_threshold`` on clean sweeps, see "Closing the loop" in the README) and clearing are different things: ``DELETE /api/v1/apps/graph_watchdog/faults/GRAPH_NODE_DISAPPEARED`` - the fault manager's own ``~/clear_fault`` underneath - is what acknowledges an occurrence and closes its cycle. A still-CONFIRMED, or still-HEALED-but-unacknowledged, fault that fires FAILED again is the SAME occurrence continuing, not a new one, so restarting the dead node fast enough to heal it before anyone acknowledges it will not move the count. The honest limits, read off this branch's fault-manager storage rather than assumed: the per-fault rosbag store enforces ``fault_code`` as UNIQUE, so a fault can hold at most one recording at a time - a later confirmation's capture replaces the earlier one on disk rather than accumulating a history. The freeze frame is the same shape: one row per ``fault_code``, overwritten on every capture, so a fifth occurrence's captured values overwrite the first's. What survives every occurrence by default is the count itself; a hash-chained record of every raise/clear/heal transition also exists, but only once the fault manager's own ``audit_log.enabled`` is turned on, which it is not by default. Recordings and the freeze frame do not accumulate a per-occurrence history either way. **A second death while the first is still outstanding gets no evidence of its own.** This detector folds every currently-affected node into ONE ``GRAPH_NODE_DISAPPEARED`` record via ``AggregatedFault::emit()`` (see "The boundary with" ``lifecycle_expectation`` above): a second node dying while the fault is still CONFIRMED is added to the description on the next tick, but the ``ReportFault`` call that carries it lands on ``fault_storage.cpp``'s already-CONFIRMED, still-FAILED branch - the same re-report-not-a-new-occurrence path "Repeated failures" above describes for one node dying twice, reached here by two different nodes sharing one code instead. ``occurrence_count`` does not move, the fault manager publishes ``EVENT_UPDATED`` rather than ``EVENT_CONFIRMED``, and ``just_confirmed`` in ``fault_manager_node.cpp`` stays false - which is what gates ``capture_pool_``, so neither a freeze frame nor a recording is captured for the second node. Acknowledging between the two deaths avoids this: the DELETE route's ``~/clear_fault`` closes the first occurrence, so the second node's next FAILED report reactivates a CLEARED record instead of updating a CONFIRMED one - a genuine new occurrence, with its own ``occurrence_count`` and its own capture. Two fixes were measured and rejected here, so this is not open for reconsideration without new evidence: - **Clear-then-raise from inside the detector does nothing.** ``AggregatedFault`` already has both halves available - ``emit()`` sends a PASSED-shaped ``clear_fault()`` when ``affected`` is empty, a FAILED-shaped ``raise_fault()`` otherwise - so sending one right after the other on the same tick looks like a free fix. It is not: ``compute_debounce_status()``'s hysteresis latch holds a CONFIRMED (or HEALED) status until the debounce counter itself crosses the OPPOSITE threshold, and a single PASSED event only moves that counter by one step. Unless the fault happened to sit exactly one step short of ``healing_threshold`` already, the clear changes nothing the status can see, and the raise that follows lands right back on the same already-CONFIRMED branch as before. - **Calling the real** ``~/clear_fault`` **service instead is worse than doing nothing.** ``SqliteFaultStorage::clear_fault()`` runs ``DELETE FROM snapshots WHERE fault_code = ?`` - it deletes the per-topic readings captured for the fault it clears, keeping only the freeze-frame row. And ``ClearFault.srv``'s own ``skip_correlation_auto_clear`` defaults to false, so clearing a root cause also clears every symptom the correlation engine attributes to it unless the caller explicitly opts out (the gateway's own REST DELETE routes do; a detector reaching for this service directly would have to remember to as well). Having the detector call this automatically the moment a second node dies would destroy the FIRST node's evidence - snapshots and correlated symptoms alike - to chase evidence for the second, before anyone has necessarily seen the first. A correct fix does not belong in this detector at all: it needs an operation in the fault manager that re-confirms a CONFIRMED record - bumping ``occurrence_count``, publishing ``EVENT_CONFIRMED`` again, and re-arming ``just_confirmed`` for a fresh capture - without routing through CLEARED and without deleting anything the first occurrence already earned. **Test tiers.** Three tiers each prove a different layer, deliberately not overlapping, plus the suppression chain the detector shares no code with any sibling for: 1. **Unit** (``test_node_liveness_tracker.cpp``): the pure presence/absence state machine - a key present but never armed is never tracked; once armed, presence alone (not continued arming) keeps a tracked key's miss counter at zero; a key is reported dead only once misses exceed ``miss_grace``; freshness ordering keeps a brand-new death out of a capped description's blind spot; ``prune()`` reclaims a key only once it has been suppressed for MORE than ``prune_ticks`` CONSECUTIVE calls, resets the streak the moment a veto lifts even once, and never reclaims an unsuppressed death no matter how long it stays dead; the cap leaves every PRESENT key individually tracked and reportable however far the map is past it, collapses only departed entries that have crossed ``miss_grace``, keeps the collapsed count monotone within one tracker lifetime, and - the property most at odds with a naive read of the sibling detector - reports ``tracking_saturated`` for a departed set that immature entries alone keep oversized, which unlike the sibling's saturation refuses nothing and loses no report. ``test_suppressor.cpp`` pins the ``Suppressor`` interface and the free ``apply_suppressors()`` helper (order-independent, null-safe, returns the dropped count); ``test_allowlist_suppressor.cpp`` and ``test_lifecycle_shutdown_suppressor.cpp`` pin each suppressor's own matching rules independently of any detector. 2. **Integration** (``test_node_death_integration.cpp``): the full config contract (``miss_grace``, ``prune_grace`` and ``tick_interval_ms`` range checks and their floor interaction; malformed ``allowlist``/``suppress`` entries; ``tracked_node_cap`` range checks, including a value far past ``INT_MAX`` rejected rather than wrapped; an unknown top-level key), that a peer-aggregated app and an ``is_online: false`` app are never tracked, that the tracked count stays bounded and shrinks after a reclaim under sustained identity churn, that a clean managed-lifecycle departure is suppressed exactly AT the ``miss_grace`` boundary and stays reclaimed - not re-raised - past the retention window the plugin sizes for it, that the allowlist's id-form suppresses only the colliding node and not its namesake, and the ungated-clear guard's four properties: a stored fault stays unhealed while an unrelated node arms alone, a clear flows once this process instance has genuinely raised, the guard tracks DELIVERY rather than intent, and a reconfigure while the node is absent does not withhold a clear the process had already earned. 3. **E2e**, four files, each launching its own real gateway + fault_manager + demo-node stack: ``test/e2e/test_node_death_e2e.test.py`` (ten scenarios covering the raise/clear round trip, no-heal-standalone, the lifecycle-deactivate/manifest-never-online/ros2cli non-cases, a bare-name collision naming only the node that exited, a fast tick alone raising nothing, a three-cycle restart loop reaching ``occurrence_count: 3``, and a fault surviving a gateway restart); ``test/e2e/test_node_death_suppression_e2e.test.py`` (five scenarios covering the allowlist, its inertness when unnamed in ``suppress``, the clean-shutdown/still-active contrast, opt-in suppression, and pruning without a false heal); and ``test/e2e/test_node_death_boundary_e2e.test.py`` (eight scenarios proving the seam with ``lifecycle_expectation`` directly: a node stuck inactive but never gone raises ``GRAPH_NODE_INACTIVE`` alone, an ARMED node killed below ``grace`` raises ``GRAPH_NODE_DISAPPEARED`` alone, a node killed AFTER ``GRAPH_NODE_INACTIVE`` has confirmed raises BOTH at once, a healthy node that is simply killed raises ``GRAPH_NODE_DISAPPEARED`` alone, a restart-looping required node is caught every cycle regardless of lifecycle maturity, a NEVER-armed node killed below ``grace`` raises ``GRAPH_NODE_INACTIVE`` alone instead - node_death cannot track a node the gate never admitted for ownership, so absence has to mature it here - a large ``miss_grace`` delays but does not swallow the report, and a restarted gateway's warmup window never produces a spurious PASSED); and ``test/e2e/test_presence_ownership_e2e.test.py`` (seven scenarios against the fixture whose ``GetState`` never answers, so the unread state is permanent rather than the race the boundary file's B6 row has to live with: a managed node the watcher asked and could not read, killed, raises BOTH ``GRAPH_NODE_DISAPPEARED`` and ``GRAPH_NODE_UNREADABLE``; the same death with no ``require_active`` entry anywhere - the shipped default, where nothing else is watching at all - still raises ``GRAPH_NODE_DISAPPEARED``, and that pair also rules out an implementation deciding ownership from ``require_active`` membership; the SAME node told to start answering, measured ``active``, then killed, IS reported too, under the identical configuration as the first row so the only variable is the lifecycle state; a node measured ``active`` that then LOSES its lifecycle services and is killed is still reported; and with an unmeasured managed node in the graph ``GRAPH_PARAM_DRIFT`` still names it - the one leg of the three that passes through the app-keyed gate at all - while the topic-keyed ``GRAPH_QOS_MISMATCH`` and ``GRAPH_ORPHAN`` still report too; and the one row that asserts an ABSENCE, where the same never-answering node is owned provisionally, then announces ``inactive`` over ~/transition_event, then dies: ``GRAPH_NODE_INACTIVE`` names it and ``GRAPH_NODE_DISAPPEARED`` never appears, which is also what rejects a detector wired to the permissive ``reliability_allows()``; and a crash loop shorter than the warmup, where the node is killed, reported, brought back for less than one re-warm and killed again - the second death must still be reported, and no PASSED may be emitted while it is dead). Status --------------- The plugin loads, ticks the graph, and shuts down cleanly. The reliability core is real and already ticking. Five silent-fault detector classes raise through it today, ``qos_mismatch``, ``orphan``, ``param_drift``, ``lifecycle_expectation`` and ``node_death``. Two classes remain undelivered, ``GRAPH_TF_STALE`` and ``GRAPH_LATENCY_BUDGET``; they land in follow-up changes, each against its own issue.