• v0.4.3 7db213a368

    agorise released this 2026-09-12 01:09:06 +00:00 | 4 commits to main since this release

    hidden-rpc v0.4.3

    The report paid for itself immediately: three findings

    1. The node could never have recovered, and now it repairs itself in a minute.
    kc's state held block 1 while a 23 GB block_log sat on the disk next to it. blurtd
    was asking peers for blocks, receiving them, and rejecting every single one —
    unlinkable block ... does not link to known chain — because it has no chain to link
    them to. A node in that state will never catch up, however long you leave it.

    That is provable from two numbers we already collect: the head block, and the size of
    the block_log. The supervisor now checks for it directly and rebuilds the state from
    the block_log within a minute, instead of waiting out the six-hour
    "it has not served" timeout. A node genuinely syncing from scratch (low head block,
    no block_log) is untouched — there is nothing to replay.

    2. The six-hour timer was being reset by every installer run. LAST_SERVED_AT
    was written to the persist file but never read back — the three restore lines landed
    outside load_persist when I edited it, and nothing checked. Since every installer
    run restarts the supervisor, the not-serving clock went back to zero each time. On a
    node being actively worked on, the outcome watchdog could effectively never fire.
    Fixed, with a test that round-trips every persisted field.

    3. The report's own service table said failed=active for everything, which is
    what systemctl is-failed prints when a unit is fine. It now says "not failed" or
    "FAILED", which is what the column was for.

    The installer was destroying rebuilds. Interlocked now.

    v0.4.1's installer printed "Replay flag removed" on a node that was hours into
    rebuilding its chain state, and threw that work away. Its health test was "does the
    RPC answer" — and a rebuilding node answers, with head block 1.

    This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
    was the error: the decision was being made in two places, by two different tests.

    The guard is now a hard interlock inside the repair library itself, so no caller
    can get it wrong:

    • blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
      not serving a real head block. It reads the container's command and the head block
      — data, not logs, which is what defeated the earlier attempts;
    • blurtd_deflag and blurtd_recreate both refuse while it is true;
    • the installer no longer makes this decision at all. It reports "this node is
      rebuilding its chain state — leaving it completely alone" and moves on. The
      supervisor removes the flag, once, when the node genuinely serves.

    Tested from both callers, including the exact case that bit us twice: a rebuilding
    node whose log shows no replay lines.

    A random-failure bug in seventeen places

    Chasing an intermittent test failure turned up something worse than the test: every
    script here runs under set -o pipefail, and seventeen of them used
    something | grep -q pattern. grep -q exits the instant it matches, the process
    writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
    sometimes, depending on whether the writer had finished. Measured on an idle
    machine: about one run in three.

    Two of those were in the installer's decision about whether the node container
    exists. A random false there sends the installer down the wrong branch — recreating
    a container it should have left alone, or leaving one alone it should have rebuilt.
    Some of the confusing behaviour on this fleet may well have been this.

    All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
    passes 25 runs out of 25.

    sudo hidden-rpc-report — every fact in one file

    Diagnosing a node one question per round is a bad way to work; each round costs a
    reinstall and a wait. This writes down everything that could explain a node's
    behaviour, in one pass, and changes nothing:

    • the head block sampled three times ten seconds apart (does it move?)
    • the container's exact command, state, exit code and restart count
    • how much block_log is really on disk
    • blurtd's last 60 log lines and any replay progress lines
    • every unit's active/enabled/failed state, including start-limit latches
    • the supervisor's status file, persisted state and last 60 journal lines
    • torrc stanza, onion hostname, i2pd tunnel and keys, key backups
    • loopback RPC, both proxy ports, clock, disk, memory

    Also available as hidden-rpc-setup --report.

    The hidden-rpc icon, and a plain answer about restarting

    The tray icon and the menu entries now use the project's own icon rather than
    borrowed system ones. The tray shows the mark with a small status badge in the
    corner — green for serving, amber while rebuilding, red when it is not serving, grey
    when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
    entries now live under Internet.

    Setup also finishes with a plain-language check of exactly what happens when you
    restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
    the tray icon. If any of them would not come back on their own, it says so and tells
    you the fix (run the installer again).

    Stopping frees the machine, starting hands it back

    Stop the node stops blurtd, which is effectively all of the memory and CPU
    hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
    addresses stay published. If you want every last megabyte there is now a second menu
    item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

    Start the node brings everything back, including anything a --all stop took down.
    The node loads its state, catches up on the blocks it missed, and becomes reachable
    from outside again by itself; the self-healer takes over from there. Your .onion
    and .b32.i2p addresses never change.

    An icon by your clock, and menu entries — no terminal needed

    You should not have to open a terminal to answer "is my node running?". On a machine
    with a desktop, setup now adds a tray icon next to the clock:

    • green arrows — serving the chain, all good
    • spinning arrows — rebuilding or starting up; hours, nothing to do
    • warning triangle — not serving; the node is repairing itself
    • crossed-out plug — stopped, or the supervisor is not running

    Hover it for a one-line summary (which transport is reachable, what block it is on,
    how long the current phase has been running). Click it to stop the node, start
    it again
    , or open the live view.

    Your applications menu also gets hidden-rpc: Node status, Start the node and
    Stop the node. Stopping asks for your password through the normal desktop
    dialog — no sudo, no terminal.

    Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
    write its state out (cutting that short is what causes hours of rebuilding later),
    and it turns maintenance mode on so the self-healer will not quietly start the node
    again while you are using the machine for something else. Starting turns it back off.

    The icon needs no privileges: it reads the same status file everything else does.

    Which processes are hidden-rpc?

    None of them run as you, which is why System Monitor shows nothing by that name —
    switch it to All Processes and you will find:

    process who runs it what it is
    blurtd root the chain node itself (inside Docker)
    transport-supervisor.sh root the self-healer
    tor debian-tor your .onion
    i2pd i2pd your .b32.i2p

    "Catching up — 63,560,408 blocks to go" was the last dishonest line

    A node answering from an empty state (head block 1) is not behind, and it is not
    catching up: it is not serving, and it will not get there on its own. The installer
    said otherwise because it compared the head block against the live chain without
    asking whether that head meant anything. It now says plainly that the node is not
    serving and points at the repair.

    Everything the tools print about this node's health is now derived from the same
    test the supervisor uses.

    The installer is now installer/setup.sh

    Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
    old name is gone rather than aliased; update any script or note that referenced it.

    Every command this project prints can now be typed exactly as printed

    setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
    gives command not found. The full-path form the installer printed instead was long
    enough to wrap, and inconsistent with the two short commands beside it.

    Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
    of these work from any directory:

    hidden-rpc-status                 # everything: node, reachability, self-repairs
    hidden-rpc-status --watch         # live progress, one calm line
    hidden-rpc-setup --doctor         # check and repair everything right now
    hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
    sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
    

    The wrapper runs whichever installer last set this node up, recorded in
    /etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
    command that is neither an installed command nor an explicit bash <path>.

    --doctor, --reindex, --headless and --desktop did nothing at all

    They parsed into an action that had no case in the dispatch, so the installer exited
    silently, printing nothing. My patch adding those cases never matched the file and I
    did not check that it had applied. They work now.

    The reason this kept happening is worth more than the fix: three separate defects in
    this series have been installer paths that no test ever executed. So the lint suite
    now walks the installer's own flag table and asserts that every flag reaches a case
    in the dispatch, and that every case calls a function that exists — verified to fail
    when a case is removed. Documented flags and accepted flags are checked against each
    other too.

    --doctor also now reports how much block_log is actually on disk as its first line,
    since that single number decides which repair a stuck node needs.

    One rule above all the others: is this node serving?

    Every release since v0.3.0 fixed a real fault and then met a new one nobody had
    thought of. That is the wrong shape for software that is meant to be set up once and
    forgotten, because it means each unanticipated failure is an outage that waits for a
    human. v0.3.6 changes the shape.

    Serving is now defined by one fact, and nothing else: the head block is advancing,
    and it is a real head rather than a freshly-initialised one.
    Blurt produces a block
    every three seconds, so a node that is genuinely serving cannot fail that test, and
    nothing that is not serving can pass it — whatever its logs, its container state, its
    flags or its daemon status happen to say. (A node answering RPC with head block 1
    was being reported as "Serving", because the code asked the narrower question "is a
    replay running?" and got the wrong answer.)

    And above every diagnosis there is now an outcome watchdog. If a node has not
    served the chain for six hours, something is wrong that the specific checks did not
    catch, and it escalates anyway:

    1. rebuild the state by replaying the local block_log — if the block_log is intact;
    2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
      not there, so fetch the whole chain again from the signed on-chain pointer;
    3. if a rebuild already happened and the node still is not serving, the data itself
      is suspect — fetch it again;
    4. and the reboot rung still applies underneath all of it.

    Visible progress (a running replay, a moving log) buys more time, but only up to
    twice the window, so nothing can stall forever by printing. Maintenance mode
    suppresses all of it. Each step is rate-limited and recorded across restarts.

    This is the property that makes "set it and forget it" honest: a node can recover
    from a failure mode that did not exist when this code was written
    , because the top
    rule asks about the outcome rather than the cause.

    New commands and reporting

    hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
    serving says so, with its head block and what the supervisor will do next:

    ○ NOT serving — its head block is not advancing  (for 3h)
      head block 1. The supervisor escalates on its own if this lasts:
      rebuild the state from the block_log, then refetch the chain if that fails.
    

    --doctor now diagnoses from the two facts that decide which repair is right: it
    samples the head block twice fifteen seconds apart, and reports how much block_log is
    actually on disk. There is also a new last-resort command:

    ./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
    

    Stop guessing from log text: watch the head block instead

    v0.3.4 decided whether a node was rebuilding its state or serving by grepping
    blurtd's log for replay progress lines. On a real node that grep came back empty at
    the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
    — was reported as "Serving. head block 1".

    That was not just a cosmetic error. From "serving", the supervisor's next step is to
    remove the pinned --replay-blockchain flag, which would have stopped the rebuild
    and left a state at block 1 against a 63-million-block log. blurtd would then refuse
    to open with "does not match block log", crash-loop, be detected, and start another
    rebuild. Hours, on a loop, indefinitely.

    The decision no longer depends on log text at all. It uses the one fact that cannot
    be missed: is the head block moving? A serving node advances every three seconds.
    A node rebuilding its state answers with head block 1 and sits there. Two samples
    settle it, and the flag can only ever be removed from a node that has been seen
    advancing.

    Everything downstream got the same treatment: the rebuild is never interrupted, the
    wedged-node timer cannot fire during one, and both hidden-rpc-status and the
    installer now say "rebuilding its state by replaying the block_log" instead of
    computing a meaningless "63,559,304 blocks to go".

    Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
    first node check before publishing a status file, so the first thing you read is
    what the node is actually doing.

    I broke I2P in v0.3.1, and this fixes it

    The low-power i2pd tuning added in v0.3.1 was appended to the end of
    /etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
    header belongs to that section. So notransit = true at the end of the file became
    cpuext.notransit, and i2pd refused to start at all:

    unrecognised option 'cpuext.notransit'
    

    Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
    since that install
    — the .onion kept working, so it looked like a flaky tunnel.

    Fixed three ways:

    • the block is now inserted before the first section header, which is the only
      place global options work. Re-running the installer rewrites a misplaced block in
      the right place;
    • applying it is verified: the installer restarts i2pd and checks it actually
      came up. If it did not, the config is put back exactly as it was and you are told
      I2P is running on defaults instead;
    • the supervisor's deep repair now suspects our own changes first: if i2pd is
      down and our managed block is in its config, the block comes out and i2pd is
      restarted without it.

    An optimisation that stops a daemon starting is not an optimisation, and the code
    that adds one has to be able to take it back.

    A replaying node is no longer mistaken for a serving one

    During a block_log replay blurtd answers RPC with head block 1 and stays there for
    hours. v0.3.3 read that as "serving", which started the head-stall timer against a
    node doing exactly what it was told, and tried to strip the replay flag off a replay
    in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
    warning every minute for hours). The supervisor now checks whether a replay is
    running before calling a node serving, and hidden-rpc-status shows real replay
    progress instead of head block 1.

    A node that cannot open its own database now fixes itself

    On a real node, blurtd was exiting immediately on every start with:

    Chain state does not match block log. Please reindex blockchain.
    

    This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
    block_log disagree, so it refuses to open, exits 1, the container restart policy
    starts it again, and it exits again — forever. Restarting it, rebuilding the
    container, rebooting the machine: none of them can ever help, because the state
    itself has to be rebuilt.

    Two things came out of it.

    A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
    was busy if its log kept moving — and a crash loop churns its log constantly while
    the container reports running. The supervisor now checks the container runtime's
    restart counter, which is the only thing that tells those two apart, before it
    looks at log activity.

    blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
    and what the fix is, in plain English, in its log. When the supervisor sees that
    message it rebuilds the state by replaying the block_log once — hours, unattended —
    records that it did, and removes the replay flag again as soon as the node serves.
    It will not start a second replay on top of a running one, will not repeat it
    within 12 hours, and will not start one at all while maintenance mode is on.

    The installer handles it too: it reads the old container's log before removing it,
    so a re-run on an affected node starts the rebuild automatically and says what it is
    doing. There is also an explicit command:

    ./installer/setup.sh --reindex
    

    hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
    not match the block_log, so it is replaying once."

    Since v0.3.1: two judgement bugs found on a real node

    v0.3.1 installs and runs. Watching a node come up on real hardware showed two
    places where the supervisor was judging by a stopwatch instead of by evidence.

    1. A slow load could be mistaken for a hang. The node was considered "still
    loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
    chain on a slow disk, reading the state into memory can take longer — and the one
    mistake that makes things genuinely worse is restarting a node halfway through
    loading, because then it never finishes. The supervisor now asks the node instead:
    if blurtd's log keeps moving, it is working, and it is left alone however long it
    takes. It only acts when the node is running, silent, and not answering — both
    conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
    1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
    working — leave it alone"
    .

    2. Slow probes looked like waking from suspend. Resume detection timed the
    whole loop. But a tick in which both hidden-service self-dials time out
    legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
    tripped the "woke up" threshold — so an unreachable node forced a full re-check and
    reset its repair ladders on every single cycle. The check now measures the sleep
    only, which is immune to slow probes and still catches a real suspend. Verified: a
    tick spending 8 seconds in probes with a 2-second threshold now produces zero false
    wake-ups.

    Also in this release: the installer no longer holds your terminal for up to 30
    minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
    prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
    already watching the node by then, so there is nothing to sit and guard.


    If you tried v0.3.0: this is the fix

    v0.3.0 could not install. It stopped at "Starting your node" on every machine,
    with no useful explanation. My mistake, and an avoidable one.

    The installer loaded its shared repair library like this:

    DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
    

    A variable assignment written in front of the . builtin is a temporary
    environment assignment: bash throws it away when the command finishes. While it
    was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
    as already set and skipped its default — so DOCKER_CMD ended up unset, and
    every container command after that ran as timeout 120 run -d …, which fails
    instantly. The library's error went to /dev/null, so all you saw was "the node
    didn't start on this try".

    Fixed in three ways, so it cannot happen again in this shape or any other:

    • one load_heal_lib function does the loading with plain assignments, and
      verifies afterwards that the helpers exist and the command wrappers are
      non-empty — stopping with a clear message if not, before touching anything;
    • container failures now report what the container runtime actually said,
      instead of a shrug;
    • a third test suite, supervisor/tests/installer-lint.sh, runs the real
      start_node out of the real installer against a fake docker and asserts the
      exact command line it produces. That is the test that was missing: v0.3.0's
      supervisor was tested thoroughly and its installer was never executed at all.

    If v0.3.0 removed the container on your node before failing, nothing was lost —
    it was stopped gracefully first and the chain data lives in the Docker volume.
    Re-running the installer rebuilds the container and the node loads its state in
    the usual few minutes. No replay.

    Everything below is the v0.3.0 release, unchanged.


    A node that went down and stayed down was the whole problem. This release is
    about making that impossible — and about a node that stops eating the machine it
    lives on.

    Nothing about your addresses, your chain data or your setup changes. Re-run the
    installer on each node and it picks all of this up.

    Why nodes were going dark and staying dark

    Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:

    • DEGRADED was a dead end. Once a transport used up its restart budget, the
      supervisor kept probing it but never repaired it again. The only way back
      was a recovery that happened on its own. Any node that reached this state was
      off the network until a human walked over to it.
    • "The node is down" meant "do nothing", forever. When the chain daemon
      stopped answering, the supervisor correctly refused to restart Tor and I2P —
      and then stopped there. Nothing in the system ever restarted blurtd, its
      container, or Docker.
    • Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
      start command is literally /bin/true, so restarting it can report success
      while the real daemon stays dead. And after five quick failures systemd latches
      a service into a permanent failed state that refuses every further restart until
      someone runs reset-failed — which nothing ever did, including for the
      supervisor itself. Self-healing could be switched off by a single bad minute.
    • The replay flag was pinned forever. A node set up from the block_log kept
      --replay-blockchain in its container command permanently, so every reboot
      re-ran a multi-hour replay. Which looks exactly like "it went down and never
      came back".

    What's new

    • The supervisor now owns the whole node, not just the two transports: the
      chain daemon, the container, the container runtime, the clock, the disk and the
      power settings. A stopped node is started; a missing container is rebuilt from a
      saved spec; a node whose head block has frozen is treated as wedged and
      repaired; a dead Docker is restarted.
    • DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
      again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
      power. Restart thrash is still capped.
    • Every repair is verified. After a restart the supervisor checks that the
      unit really is active with a live process, that the proxy port answers, and that
      the address still exists — and escalates if not. It resolves the real Tor unit
      at runtime instead of trusting a name guessed at install time, and runs
      reset-failed before every restart.
    • Deeper repairs for the failures restarting can never fix: a torrc that lost
      its hidden-service stanza (put back, verified with tor --verify-config, rolled
      back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
      (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
      hidden-service keys (restored from a local backup, so your .onion and
      .b32.i2p survive a purge or a bad restore).
    • Reboots are boring now. The replay flag is removed automatically as soon as
      a node proves it is serving; a new shutdown unit gives the blockchain the full
      time it needs to save itself before Docker goes away, so there are no more dirty
      shutdowns and no surprise replays; and everything is enabled to come back on its
      own. Move a node to another room whenever you like.
    • Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
      the systemd level, where no desktop power setting can override them. If a
      machine does sleep anyway, waking up triggers an immediate full re-check.
    • A controlled reboot as the genuine last resort — only when every transport
      is down, only after two hours of failed repairs, never during a replay, never
      just after booting, at most once a day, and always after stopping the blockchain
      cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.

    A much lighter node

    Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.

    • No more permanent replay. This was the single largest CPU cost in the fleet,
      and it is gone.
    • The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
      loop no longer spawns anything at all: it reads /proc with shell builtins
      instead of shelling out to ip, awk, cksum, date and sed several times
      every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
    • Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
      5 minutes once a transport has been steady for half an hour, and snap back
      instantly on any network change, wake-up or failure.
    • I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
      on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
      node carries, for no benefit to its own reachability. Now notransit = true,
      with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
      server tunnel is unaffected.
    • blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
      connections with a smaller sync prefetch. Neither needs a replay.
    • Optional --headless stops the desktop on a node nobody looks at, freeing
      RAM and the GPU. Reversible with --desktop.

    New commands

    hidden-rpc-status                  # everything: node, reachability, self-repairs, host
    hidden-rpc-status --watch          # live progress, one calm line
    sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
    sudo hidden-rpc-nodectl start      # and back on again
    ./installer/setup.sh --doctor    # check and repair everything, right now
    

    hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
    the node alone while you work; start turns it back off. A node can never be left
    quietly un-supervised. hidden-rpc-transport-status still works and shows the
    transports-only view.

    Upgrading

    Re-run the installer on each node (safe anytime, never harms a node):

    bash installer/setup.sh
    

    On a node that is currently pinned to the replay flag, this removes it with one
    short restart — after which reboots take minutes instead of hours. Your
    /etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
    this version adds are appended.

    To fix a node without a full installer run: setup.sh --doctor.

    Minimum requirements

    • A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
    • About 90 GB free for a first-time trustless setup.
    • Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
      successfully around 50% of the time versus 100% for Ethernet.

    Testing

    Both suites run anywhere, with no Tor, I2P or Docker needed:

    bash supervisor/tests/fsm-sim.sh          # state machine + policy
    bash supervisor/tests/heal-sim.sh         # the repairs themselves
    bash supervisor/tests/installer-lint.sh   # the installer, actually executed
    

    heal-sim.sh drives the repair code against fake systemctl and docker
    commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
    start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
    are in docs/SELF-HEALING.md.

    Downloads