hidden-rpc

agorise/hidden-rpc

Watch

Fork 0

26 releases 26 tags

RSS feed

v1.1.0 785c19b8ed

hidden-rpc v1.1.0 Stable
agorise released this 2026-09-16 18:44:23 +00:00 | 0 commits to main since this release
hidden-rpc v1.1.0

The bug that took Tor off this fleet

Every node this installer has touched since v1.0.0 has had no Tor at all. Not a
degraded Tor — none. The .onion was unreachable across every reboot, and there was
no failed unit, no error state and nothing in hidden-rpc-status pointing at the
cause.

The installer wrote a systemd drop-in adding Restart=always to tor.service. On
Debian and Linux Mint, tor.service is not the Tor daemon: it is a Type=oneshot
wrapper whose ExecStart is literally /bin/true, and whose only job is to pull in
tor@default.service, which is the daemon. Restart= is illegal on a oneshot, and
systemd's response is not a warning and not a failed unit — it refuses to load the
unit entirely. So nothing pulled in tor@default; and because tor@default names
tor.service in its own PartOf=, even starting it by hand was refused. Both units
sat quietly at inactive, main PID 0, is-failed reporting inactive.

The supervisor did everything it was supposed to: it noticed, restarted, deep-repaired,
escalated, and reported honestly that the restart had produced no daemon. It could not
have succeeded. You cannot restart a unit that systemd has thrown away.

Three things changed, so that this class of fault heals itself:
- The installer reads each unit's Type= off disk before writing, and never puts
  Restart= in a oneshot. A oneshot still gets the [Unit] half (StartLimitIntervalSec=0),
  which is legal everywhere.
- The supervisor removes a drop-in of ours that stops a unit loading — at startup
  and on every deep repair, so a node running an old version digs itself out without
  waiting for an upgrade. It disables our files one at a time until the unit loads,
  puts them all back if the cause turns out not to be ours, and never touches a
  drop-in somebody else wrote.
- A failed restart now logs what systemctl said about it. The single line that
  explained everything was being discarded into /dev/null.
Regression tests were added for each: Restart= never reaching a oneshot, the rescue
removing ours and only ours, and the rescue actually being wired into the supervisor
and the installer.

On an already-broken node, upgrading fixes it. To fix one by hand right now:
```
sudo rm -f /etc/systemd/system/tor.service.d/hidden-rpc-restart.conf
sudo systemctl daemon-reload
sudo systemctl reset-failed tor tor@default
sudo systemctl start tor
```
hidden-rpc no longer restarts your computer

The last-resort rung used to reboot the machine. It no longer does —
ALLOW_REBOOT now defaults to 0, and upgrading an existing node turns it off.

A node is not necessarily the only thing on its machine. One node in this fleet also
runs a video transcoder that does not come back on its own, and software that reboots
the computer it lives on cannot know what it is interrupting.

And it did not work. The one time the rung ever fired, the cause was the tor.service
bug above — a unit systemd had refused to load, which no reboot can fix. The node
rebooted itself once a day for three days, took the transcoder down with it each time,
and was no closer to serving.

What happens instead, at exactly the same point in the ladder:
- an [ERROR] ATTENTION: line in the event log;
- the tray icon turns red with a !, and its menu says what is stuck, how long it
  has been stuck, and what you could do about it;
- hidden-rpc-status prints a NEEDS A LOOK block;
- one of the menu options is Restart this computer — a button you press, behind a
  confirmation, which stops blurtd cleanly first so the machine does not come back into
  a multi-hour replay (hidden-rpc-nodectl reboot does the same from a terminal).
The ladder keeps retrying underneath all of this, and the red badge clears itself the
moment the node is reachable again.

The tray colours now mean something specific, which they did not before: green is
serving, amber is busy or repairing itself (hours, nothing to do), red ! is
the only state that ever wants something from you, and grey is stopped. "Not
serving" while the ladder is still working is amber, not red — a red dot that means
"wait a few minutes" is a red dot people learn to ignore.

To put the automatic reboot back on a machine that really is nothing but a node, set
ALLOW_REBOOT="1" in /etc/hidden-rpc/supervisor.conf. The upgrade leaves a note in
that file recording that it changed the value, and checks for that note first, so it
will never override your choice twice.

Logging that survives a reboot and deletes itself after two weeks

A node that misbehaves at 3am is only diagnosable if its logs are still there when
someone looks days later. Two things were missing.

The journal could be volatile. systemd's shipped default is Storage=auto —
persistent only if /var/log/journal exists. Ubuntu and Mint create it on a fresh
install, but not on an upgrade, and not if /var was ever rebuilt. On such a machine
logs live in /run and are wiped on every reboot: the reboot destroys the evidence
for why it rebooted. Now set to Storage=persistent explicitly.

Nothing pruned by time. MaxRetentionSec is unset by default, so the journal is
pruned only on size — up to 4 GB and months of history on a big disk. Now two weeks
or 200 MB, whichever comes first, with 2 GB always left free on the filesystem.

The node now records its own decisions

/var/lib/hidden-rpc/events.log — the short story, one line per decision, kept
14 days and hard-capped at 2 MB. It survives independently of systemd, so it is
still there if journald has rotated:
```
2026-09-16T17:46:33Z [INFO] boot marker: machine has been up 8 minutes, 532s since boot
2026-09-16T17:46:33Z [INFO] supervisor v1.1.0 started: watching node tor i2p
2026-09-16T17:46:33Z [INFO] tor: HEALTHY -> SUSPECT
2026-09-16T17:46:34Z [INFO] tor: nudged; grace 45s
2026-09-16T17:46:40Z [WARN] tor: plain restarts have not worked — checking configuration, keys and host health
2026-09-16T17:46:42Z [ERROR] tor: 3 restart cycles exhausted -> DEGRADED; will deep-repair and retry in 900s
2026-09-16T17:46:49Z [INFO] tor: auto-recovered
```
Only decisions are recorded — a probe is not an event, a repair is. The boot marker
includes uptime, so a machine reboot is distinguishable from a supervisor restart
even when hidden-rpc was not the cause.

Nothing here can fill a disk. Four independent limits: the event log pruned to 14 days
hourly and hard-capped by size; the journal capped at 200 MB with 2 GB kept free; the
deep-repair log rotated weekly at 20 MB; and the existing disk_repair rung that
vacuums automatically below 5 GB free. The 14-day boundary and the size cap are both
verified by tests using real timestamps.

Two new commands
```
hidden-rpc-logs               # what this node decided, newest last
hidden-rpc-logs --boots       # every restart, and whether it was clean
hidden-rpc-logs --where       # where the logs are and how long they are kept
hidden-rpc-logs --all         # everything, from the journal

sudo hidden-rpc-forensics     # why did it reboot / why is a transport stuck?
```
hidden-rpc-forensics is standalone and version-independent — copy it to a node
that has not been upgraded and run it there:
```
scp hidden-rpc-forensics star:/tmp/ && ssh star 'sudo bash /tmp/hidden-rpc-forensics'
```
It reads only: changes nothing, starts nothing, stops nothing, safe on a live node
including one part-way through a replay. It answers, in order: did hidden-rpc reboot
this machine itself (from its own persisted record); was the shutdown clean or
unexpected; what do the logs from before the reboot say (panic, OOM, thermal, disk,
or a polite request, and who made it); and why a transport is stuck — unit state,
start-limit latch, torrc stanza, keys, clock, and a live self-dial through Tor.

Full detail in docs/LOGGING.md.

hidden-rpc v1.0.2

The tray icon now comes back after a cold boot

It was autostarted one way only: the XDG entry in /etc/xdg/autostart. On a cold
boot the session can launch that before the panel's system-tray applet is listening
— so the icon has nothing to attach to, gives up, and never appears, with no error
anywhere to find. Logging out and back in "fixed" it, which is exactly the kind of
thing nobody should have to know.

Four changes, because any one of them can still lose a race:
- two independent start paths — the XDG entry (now with an 8-second delay so it
  stops racing the panel) and a systemd user service enabled for every user with
  systemctl --global enable, which needs nothing in anyone's home directory;
- a lock in the indicator, so two start paths can never produce two icons —
  whichever wins keeps the tray, the other exits quietly (verified headlessly);
- Mint's own XApp tray API is preferred where present, with the Ayatana
  AppIndicator standard and the legacy GTK tray behind it. XApp copes with the panel
  applet appearing later, which is the cold-boot case exactly;
- it retries every 10 seconds for ten minutes if no tray is available yet,
  instead of exiting silently.
If it is ever missing again:
```
hidden-rpc-setup --tray      # bring it back right now
sudo hidden-rpc-report       # and say why it was missing
```
The report now prints the whole picture — whether the binary, the autostart entry
and the user unit are installed, which tray backends exist on the machine, the
desktop session, and whether the icons are in the theme.

Threat-modelled, red-teamed, and honest about latency

docs/AUDIT.md now carries the two structured exercises that were missing: a
STRIDE model applied per trust boundary (the world → the hidden service, the
chain → the installer, the desktop user → root, and the node → its own operator),
and a red-team attack-tree pass working backwards from four attacker goals —
get root, deanonymise the operator, silence the fleet, make a node lie. Each leaf is
marked closed or residual. Every path this software controls is closed; what remains
is inherent to running a public blockchain node, or lives in blurtd upstream.

On "it must be lightning fast"

Straight answer: sub-second cannot be promised over Tor or I2P, and no setting on
the node changes that. A request to a v3 onion traverses six volunteer relays
meeting at a rendezvous somewhere in the world. Hundreds of milliseconds is normal,
occasional seconds happen, and the first request to a cold service pays extra.

What is controllable is the part that is ours: the time blurtd spends answering,
measured on loopback, where single-digit to low-tens of milliseconds — and, more
importantly, steady — is the target. The right answer for a client that needs speed
is to race several nodes and take the first reply, which is exactly what Morphit
does; a seven-node race turns a long-tailed distribution into a short one, and that
buys far more than any per-box tuning.

So instead of a claim, this release ships a measurement:
```
hidden-rpc-bench            # p50/p90/p99 on loopback, Tor and I2P
hidden-rpc-bench --local    # loopback only, builds no circuits
hidden-rpc-bench --json     # for trending
```
Three calls of increasing weight, no privileges needed. No latency numbers are
quoted in the audit on purpose — figures from a build container would be
meaningless, and the only ones that matter come off your actual laptops.

Two latency changes made as a result
- webserver-thread-pool-size 16 → 32 (Blurt's own default). Cutting it to 16
  was a footprint decision that quietly created a latency floor: a client racing
  several nodes opens several connections at once, and waiting for a free thread is
  paid on top of six network hops. Where footprint and answer time conflict on the
  read path, answer time wins now. Still 8× below the 256 this project shipped.
- vm.swappiness = 10, the single biggest latency cliff on this hardware: a
  ~48 GB memory-mapped state on a 8–16 GB machine, served from swap, turns 20 ms
  into seconds. Reversible by deleting one file. hidden-rpc-report now shows swap
  in use and the current swappiness.
And the footprint verdict, honestly

Not as small as possible. The largest remaining win — dropping blurtd's follow and
tags plugins — is deliberately unpulled: it needs a multi-hour replay, and whether
any Morphit code path calls tags_api/follow_api cannot be answered from this
repository. Two smaller levers (reduced Tor padding, shorter I2P tunnels) are
documented and left alone because they trade away this project's first priority.
docs/AUDIT.md §6 has the full accounting.

hidden-rpc v1.0.0

Stable release. A machine you set up once and stop thinking about: it
serves the Blurt chain over Tor and I2P, repairs itself when anything breaks, and
tells you how it is doing from an icon next to your clock.

What 1.0 means here

Every failure this fleet actually hit is now either prevented, detected and repaired
automatically, or — where a fault cannot be named in advance — caught by a rule that
does not need to know what broke:
- the node is supervised as a whole — chain daemon, container, container runtime,
  Tor, I2P, clock, disk, power settings — each with its own repair ladder, and every
  repair verified afterwards rather than assumed;
- "serving" has exactly one definition: the head block is advancing, and it is a
  real head. Nothing that is not serving can pass that test, whatever its logs,
  flags or daemon status say;
- above every diagnosis sits an outcome watchdog. If a node has not served for
  six hours, it escalates anyway: rebuild the state from the block_log, refetch the
  chain if the block_log is gone or a rebuild did not help, reboot as a last resort.
  A node can therefore recover from a failure mode that did not exist when this code
  was written;
- DEGRADED is a slow retry, never a resting place. It keeps deep-repairing on a
  doubling backoff for as long as the machine has power;
- reboots are boring. Clean shutdown for the blockchain, no pinned replay flag,
  everything enabled at boot, the tray icon back at login. Move the machine to
  another room whenever you like.
Audited before release

docs/AUDIT.md is the full record: every file, shellcheck and pyflakes and bandit
clean, adversarial fuzzing of the one input a stranger controls, and resource
behaviour measured over time rather than reasoned about.

It found seven real defects, two of them local privilege escalations — root
executing a script from a user-writable home directory, and a predictable
root-written temp file open to a symlink attack. Both fixed and regression-tested,
along with a slow memory creep in the supervisor (now bounded by a daily same-pid
re-exec), zombie processes from the tray menu, and a URL field that accepted more
characters than it should.
```
shellcheck   0 errors      pyflakes  0 issues      bandit  0 high, 0 medium
239 assertions across 4 test suites, all passing from a clean extraction
```
If you are upgrading
```
bash installer/setup.sh
```
Safe at any time, including while a node is rebuilding its state — the installer now
refuses to touch a rebuild in progress. Your addresses, chain data and settings are
untouched.

Requirements
- A 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Everyday commands
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
sudo hidden-rpc-report            # every relevant fact in one file, changes nothing
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
Or use the icon by the clock: green serving, amber rebuilding, red not serving, grey
stopped. Click it for progress, or to stop and start the node.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Earlier releases

hidden-rpc v0.4.8

Upgrades now replace the tray icon that is actually running

If your tray menu still says "Watch progress…", you are looking at the old
indicator: v0.4.6 renamed those entries to "Show progress…" and "Open in a
terminal…". The installer wrote the new file to disk and then skipped starting it,
because one was already running — so the old process stayed in the tray, with the
old menu and the old bug, until the next login. The fix shipped and went on doing
nothing, which is the worst way for a fix to behave.

Setup now stops the running indicator and starts the new one, reading the session
environment (DISPLAY, X authority, session bus) from the process it is replacing
rather than guessing it, and confirms the new one came up.

The indicator also answers --version now, so "which one is actually in the tray?"
is a question with an answer:
```
/usr/local/bin/hidden-rpc-indicator --version
```
To swap it immediately without waiting for anything:
```
pkill -f hidden-rpc-indicator
setsid /usr/local/bin/hidden-rpc-indicator >/dev/null 2>&1 &
```
A stray directory was being packaged and shipped

The v0.4.6 tarball contained a directory tree named
supervisor/ExecStart=/usr/sbin/i2pd --conf=.... An edit had briefly left
i2p_tunnel_file echoing a systemd ExecStart line along with the path it was
supposed to return; the caller then created that as a directory. The cause was
already fixed, but the debris had been committed and packaged, because nothing was
checking what went into the tarball.

The lint suite now refuses to pass if the repository contains build artefacts,
editor leftovers, paths with = or spaces in them (the signature of a mangled shell
redirect), or unexpected top-level directories. Verified by planting the exact
directory again and watching it fail.

Nothing was broken by it on a node — it was never installed anywhere, only shipped.

"Watch progress…" did nothing. Now there is a window.

It ran x-terminal-emulator -e "hidden-rpc-status --watch". Newer gnome-terminal —
what Mint uses — ignores -e, so the click did nothing, silently, with no error
anywhere to notice.

The menu now has two entries:
- Show progress… opens a small window of our own, with a real progress bar
  during a rebuild (49.9% — block 31,400,000 of 62,905,338), the headline state,
  and both transports. It depends on nothing but the tray icon itself, so it always
  works.
- Open in a terminal… still gives you the live text view, now invoking each
  terminal the way that terminal actually accepts (gnome-terminal, mate-terminal,
  xfce4-terminal, tilix, konsole, xterm) and keeping the window open afterwards, so
  a failure is visible instead of a flash. If no terminal is found it falls back to
  the window rather than doing nothing.
Asking how a node is doing no longer asks for your password

hidden-rpc-status ran sudo docker logs to read the replay percentage, which
dropped a password prompt into the middle of its own output — you saw exactly that
while watching kc rebuild.

Reading blurtd's log needs root; the supervisor already is root, so it now
publishes the progress numbers in its status file (replay_percent,
replay_block, replay_total). The status tool and the tray icon read them from
there. Neither needs any privileges to look at a node now — only stopping and
starting does.

Found why kc lost its name

The node's name was stored only in $HOME/.hidden-rpc/name. Run the installer with
sudo — or anything that reaches the setup step as root — and $HOME is /root, so it
found no name file, concluded the node was unnamed, and wrote that empty answer into
the machine-wide supervisor.conf. The name was silently erased.

Two fixes: the name is now looked up in the invoking user's real home (via
SUDO_USER) and, failing that, read back from supervisor.conf, which is the
authoritative machine-wide copy; and an empty name is never written over a real one.
When a name is found, both copies are kept in step.

To restore kc's name:
```
hidden-rpc-setup --name kc
```
Cosmetic: no contradictory warnings during a rebuild

A node deliberately replaying its block_log was showing
⚠ the node is down — being repaired next to transports reported as healthy —
two statements that cannot both be true, on a node where nothing was wrong. During a
rebuild the RPC being unavailable is the expected state, not a fault, and the warning
is now suppressed for it (and never shown beside a healthy transport).

The report paid for itself immediately: three findings

1. The node could never have recovered, and now it repairs itself in a minute.
kc's state held block 1 while a 23 GB block_log sat on the disk next to it. blurtd
was asking peers for blocks, receiving them, and rejecting every single one —
unlinkable block ... does not link to known chain — because it has no chain to link
them to. A node in that state will never catch up, however long you leave it.

That is provable from two numbers we already collect: the head block, and the size of
the block_log. The supervisor now checks for it directly and rebuilds the state from
the block_log within a minute, instead of waiting out the six-hour
"it has not served" timeout. A node genuinely syncing from scratch (low head block,
no block_log) is untouched — there is nothing to replay.

2. The six-hour timer was being reset by every installer run. LAST_SERVED_AT
was written to the persist file but never read back — the three restore lines landed
outside load_persist when I edited it, and nothing checked. Since every installer
run restarts the supervisor, the not-serving clock went back to zero each time. On a
node being actively worked on, the outcome watchdog could effectively never fire.
Fixed, with a test that round-trips every persisted field.

3. The report's own service table said failed=active for everything, which is
what systemctl is-failed prints when a unit is fine. It now says "not failed" or
"FAILED", which is what the column was for.

The installer was destroying rebuilds. Interlocked now.

v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.

This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.

The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:
- blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
  not serving a real head block. It reads the container's command and the head block
  — data, not logs, which is what defeated the earlier attempts;
- blurtd_deflag and blurtd_recreate both refuse while it is true;
- the installer no longer makes this decision at all. It reports "this node is
  rebuilding its chain state — leaving it completely alone" and moves on. The
  supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.

A random-failure bug in seventeen places

Chasing an intermittent test failure turned up something worse than the test: every
script here runs under set -o pipefail, and seventeen of them used
something | grep -q pattern. grep -q exits the instant it matches, the process
writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.

Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.

All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.

sudo hidden-rpc-report — every fact in one file

Diagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:
- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as hidden-rpc-setup --report.

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v1.1.0.tar.gz
  0 downloads · 2026-09-16 18:44:30 +00:00 · 224 KiB
- hidden-rpc-v1.1.0.tar.gz.sha256
  0 downloads · 2026-09-16 18:44:32 +00:00 · 91 B

v1.0.2 8f12c88591

hidden-rpc v1.0.2 Stable
agorise released this 2026-09-12 22:24:51 +00:00 | 1 commits to main since this release
hidden-rpc v1.0.2

The tray icon now comes back after a cold boot

It was autostarted one way only: the XDG entry in /etc/xdg/autostart. On a cold
boot the session can launch that before the panel's system-tray applet is listening
— so the icon has nothing to attach to, gives up, and never appears, with no error
anywhere to find. Logging out and back in "fixed" it, which is exactly the kind of
thing nobody should have to know.

Four changes, because any one of them can still lose a race:
- two independent start paths — the XDG entry (now with an 8-second delay so it
  stops racing the panel) and a systemd user service enabled for every user with
  systemctl --global enable, which needs nothing in anyone's home directory;
- a lock in the indicator, so two start paths can never produce two icons —
  whichever wins keeps the tray, the other exits quietly (verified headlessly);
- Mint's own XApp tray API is preferred where present, with the Ayatana
  AppIndicator standard and the legacy GTK tray behind it. XApp copes with the panel
  applet appearing later, which is the cold-boot case exactly;
- it retries every 10 seconds for ten minutes if no tray is available yet,
  instead of exiting silently.
If it is ever missing again:
```
hidden-rpc-setup --tray      # bring it back right now
sudo hidden-rpc-report       # and say why it was missing
```
The report now prints the whole picture — whether the binary, the autostart entry
and the user unit are installed, which tray backends exist on the machine, the
desktop session, and whether the icons are in the theme.

Threat-modelled, red-teamed, and honest about latency

docs/AUDIT.md now carries the two structured exercises that were missing: a
STRIDE model applied per trust boundary (the world → the hidden service, the
chain → the installer, the desktop user → root, and the node → its own operator),
and a red-team attack-tree pass working backwards from four attacker goals —
get root, deanonymise the operator, silence the fleet, make a node lie. Each leaf is
marked closed or residual. Every path this software controls is closed; what remains
is inherent to running a public blockchain node, or lives in blurtd upstream.

On "it must be lightning fast"

Straight answer: sub-second cannot be promised over Tor or I2P, and no setting on
the node changes that. A request to a v3 onion traverses six volunteer relays
meeting at a rendezvous somewhere in the world. Hundreds of milliseconds is normal,
occasional seconds happen, and the first request to a cold service pays extra.

What is controllable is the part that is ours: the time blurtd spends answering,
measured on loopback, where single-digit to low-tens of milliseconds — and, more
importantly, steady — is the target. The right answer for a client that needs speed
is to race several nodes and take the first reply, which is exactly what Morphit
does; a seven-node race turns a long-tailed distribution into a short one, and that
buys far more than any per-box tuning.

So instead of a claim, this release ships a measurement:
```
hidden-rpc-bench            # p50/p90/p99 on loopback, Tor and I2P
hidden-rpc-bench --local    # loopback only, builds no circuits
hidden-rpc-bench --json     # for trending
```
Three calls of increasing weight, no privileges needed. No latency numbers are
quoted in the audit on purpose — figures from a build container would be
meaningless, and the only ones that matter come off your actual laptops.

Two latency changes made as a result
- webserver-thread-pool-size 16 → 32 (Blurt's own default). Cutting it to 16
  was a footprint decision that quietly created a latency floor: a client racing
  several nodes opens several connections at once, and waiting for a free thread is
  paid on top of six network hops. Where footprint and answer time conflict on the
  read path, answer time wins now. Still 8× below the 256 this project shipped.
- vm.swappiness = 10, the single biggest latency cliff on this hardware: a
  ~48 GB memory-mapped state on a 8–16 GB machine, served from swap, turns 20 ms
  into seconds. Reversible by deleting one file. hidden-rpc-report now shows swap
  in use and the current swappiness.
And the footprint verdict, honestly

Not as small as possible. The largest remaining win — dropping blurtd's follow and
tags plugins — is deliberately unpulled: it needs a multi-hour replay, and whether
any Morphit code path calls tags_api/follow_api cannot be answered from this
repository. Two smaller levers (reduced Tor padding, shorter I2P tunnels) are
documented and left alone because they trade away this project's first priority.
docs/AUDIT.md §6 has the full accounting.

hidden-rpc v1.0.0

Stable release. A machine you set up once and stop thinking about: it
serves the Blurt chain over Tor and I2P, repairs itself when anything breaks, and
tells you how it is doing from an icon next to your clock.

What 1.0 means here

Every failure this fleet actually hit is now either prevented, detected and repaired
automatically, or — where a fault cannot be named in advance — caught by a rule that
does not need to know what broke:
- the node is supervised as a whole — chain daemon, container, container runtime,
  Tor, I2P, clock, disk, power settings — each with its own repair ladder, and every
  repair verified afterwards rather than assumed;
- "serving" has exactly one definition: the head block is advancing, and it is a
  real head. Nothing that is not serving can pass that test, whatever its logs,
  flags or daemon status say;
- above every diagnosis sits an outcome watchdog. If a node has not served for
  six hours, it escalates anyway: rebuild the state from the block_log, refetch the
  chain if the block_log is gone or a rebuild did not help, reboot as a last resort.
  A node can therefore recover from a failure mode that did not exist when this code
  was written;
- DEGRADED is a slow retry, never a resting place. It keeps deep-repairing on a
  doubling backoff for as long as the machine has power;
- reboots are boring. Clean shutdown for the blockchain, no pinned replay flag,
  everything enabled at boot, the tray icon back at login. Move the machine to
  another room whenever you like.
Audited before release

docs/AUDIT.md is the full record: every file, shellcheck and pyflakes and bandit
clean, adversarial fuzzing of the one input a stranger controls, and resource
behaviour measured over time rather than reasoned about.

It found seven real defects, two of them local privilege escalations — root
executing a script from a user-writable home directory, and a predictable
root-written temp file open to a symlink attack. Both fixed and regression-tested,
along with a slow memory creep in the supervisor (now bounded by a daily same-pid
re-exec), zombie processes from the tray menu, and a URL field that accepted more
characters than it should.
```
shellcheck   0 errors      pyflakes  0 issues      bandit  0 high, 0 medium
239 assertions across 4 test suites, all passing from a clean extraction
```
If you are upgrading
```
bash installer/setup.sh
```
Safe at any time, including while a node is rebuilding its state — the installer now
refuses to touch a rebuild in progress. Your addresses, chain data and settings are
untouched.

Requirements
- A 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Everyday commands
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
sudo hidden-rpc-report            # every relevant fact in one file, changes nothing
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
Or use the icon by the clock: green serving, amber rebuilding, red not serving, grey
stopped. Click it for progress, or to stop and start the node.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Earlier releases

hidden-rpc v0.4.8

Upgrades now replace the tray icon that is actually running

If your tray menu still says "Watch progress…", you are looking at the old
indicator: v0.4.6 renamed those entries to "Show progress…" and "Open in a
terminal…". The installer wrote the new file to disk and then skipped starting it,
because one was already running — so the old process stayed in the tray, with the
old menu and the old bug, until the next login. The fix shipped and went on doing
nothing, which is the worst way for a fix to behave.

Setup now stops the running indicator and starts the new one, reading the session
environment (DISPLAY, X authority, session bus) from the process it is replacing
rather than guessing it, and confirms the new one came up.

The indicator also answers --version now, so "which one is actually in the tray?"
is a question with an answer:
```
/usr/local/bin/hidden-rpc-indicator --version
```
To swap it immediately without waiting for anything:
```
pkill -f hidden-rpc-indicator
setsid /usr/local/bin/hidden-rpc-indicator >/dev/null 2>&1 &
```
A stray directory was being packaged and shipped

The v0.4.6 tarball contained a directory tree named
supervisor/ExecStart=/usr/sbin/i2pd --conf=.... An edit had briefly left
i2p_tunnel_file echoing a systemd ExecStart line along with the path it was
supposed to return; the caller then created that as a directory. The cause was
already fixed, but the debris had been committed and packaged, because nothing was
checking what went into the tarball.

The lint suite now refuses to pass if the repository contains build artefacts,
editor leftovers, paths with = or spaces in them (the signature of a mangled shell
redirect), or unexpected top-level directories. Verified by planting the exact
directory again and watching it fail.

Nothing was broken by it on a node — it was never installed anywhere, only shipped.

"Watch progress…" did nothing. Now there is a window.

It ran x-terminal-emulator -e "hidden-rpc-status --watch". Newer gnome-terminal —
what Mint uses — ignores -e, so the click did nothing, silently, with no error
anywhere to notice.

The menu now has two entries:
- Show progress… opens a small window of our own, with a real progress bar
  during a rebuild (49.9% — block 31,400,000 of 62,905,338), the headline state,
  and both transports. It depends on nothing but the tray icon itself, so it always
  works.
- Open in a terminal… still gives you the live text view, now invoking each
  terminal the way that terminal actually accepts (gnome-terminal, mate-terminal,
  xfce4-terminal, tilix, konsole, xterm) and keeping the window open afterwards, so
  a failure is visible instead of a flash. If no terminal is found it falls back to
  the window rather than doing nothing.
Asking how a node is doing no longer asks for your password

hidden-rpc-status ran sudo docker logs to read the replay percentage, which
dropped a password prompt into the middle of its own output — you saw exactly that
while watching kc rebuild.

Reading blurtd's log needs root; the supervisor already is root, so it now
publishes the progress numbers in its status file (replay_percent,
replay_block, replay_total). The status tool and the tray icon read them from
there. Neither needs any privileges to look at a node now — only stopping and
starting does.

Found why kc lost its name

The node's name was stored only in $HOME/.hidden-rpc/name. Run the installer with
sudo — or anything that reaches the setup step as root — and $HOME is /root, so it
found no name file, concluded the node was unnamed, and wrote that empty answer into
the machine-wide supervisor.conf. The name was silently erased.

Two fixes: the name is now looked up in the invoking user's real home (via
SUDO_USER) and, failing that, read back from supervisor.conf, which is the
authoritative machine-wide copy; and an empty name is never written over a real one.
When a name is found, both copies are kept in step.

To restore kc's name:
```
hidden-rpc-setup --name kc
```
Cosmetic: no contradictory warnings during a rebuild

A node deliberately replaying its block_log was showing
⚠ the node is down — being repaired next to transports reported as healthy —
two statements that cannot both be true, on a node where nothing was wrong. During a
rebuild the RPC being unavailable is the expected state, not a fault, and the warning
is now suppressed for it (and never shown beside a healthy transport).

The report paid for itself immediately: three findings

1. The node could never have recovered, and now it repairs itself in a minute.
kc's state held block 1 while a 23 GB block_log sat on the disk next to it. blurtd
was asking peers for blocks, receiving them, and rejecting every single one —
unlinkable block ... does not link to known chain — because it has no chain to link
them to. A node in that state will never catch up, however long you leave it.

That is provable from two numbers we already collect: the head block, and the size of
the block_log. The supervisor now checks for it directly and rebuilds the state from
the block_log within a minute, instead of waiting out the six-hour
"it has not served" timeout. A node genuinely syncing from scratch (low head block,
no block_log) is untouched — there is nothing to replay.

2. The six-hour timer was being reset by every installer run. LAST_SERVED_AT
was written to the persist file but never read back — the three restore lines landed
outside load_persist when I edited it, and nothing checked. Since every installer
run restarts the supervisor, the not-serving clock went back to zero each time. On a
node being actively worked on, the outcome watchdog could effectively never fire.
Fixed, with a test that round-trips every persisted field.

3. The report's own service table said failed=active for everything, which is
what systemctl is-failed prints when a unit is fine. It now says "not failed" or
"FAILED", which is what the column was for.

The installer was destroying rebuilds. Interlocked now.

v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.

This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.

The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:
- blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
  not serving a real head block. It reads the container's command and the head block
  — data, not logs, which is what defeated the earlier attempts;
- blurtd_deflag and blurtd_recreate both refuse while it is true;
- the installer no longer makes this decision at all. It reports "this node is
  rebuilding its chain state — leaving it completely alone" and moves on. The
  supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.

A random-failure bug in seventeen places

Chasing an intermittent test failure turned up something worse than the test: every
script here runs under set -o pipefail, and seventeen of them used
something | grep -q pattern. grep -q exits the instant it matches, the process
writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.

Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.

All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.

sudo hidden-rpc-report — every fact in one file

Diagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:
- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as hidden-rpc-setup --report.

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v1.0.2.tar.gz
  0 downloads · 2026-09-12 22:24:58 +00:00 · 190 KiB
- hidden-rpc-v1.0.2.tar.gz.sha256
  0 downloads · 2026-09-12 22:24:59 +00:00 · 91 B

v1.0.1 67ee3bf311

hidden-rpc v1.0.1 Stable
agorise released this 2026-09-12 18:44:36 +00:00 | 2 commits to main since this release
hidden-rpc v1.0.1

Threat-modelled, red-teamed, and honest about latency

docs/AUDIT.md now carries the two structured exercises that were missing: a
STRIDE model applied per trust boundary (the world → the hidden service, the
chain → the installer, the desktop user → root, and the node → its own operator),
and a red-team attack-tree pass working backwards from four attacker goals —
get root, deanonymise the operator, silence the fleet, make a node lie. Each leaf is
marked closed or residual. Every path this software controls is closed; what remains
is inherent to running a public blockchain node, or lives in blurtd upstream.

On "it must be lightning fast"

Straight answer: sub-second cannot be promised over Tor or I2P, and no setting on
the node changes that. A request to a v3 onion traverses six volunteer relays
meeting at a rendezvous somewhere in the world. Hundreds of milliseconds is normal,
occasional seconds happen, and the first request to a cold service pays extra.

What is controllable is the part that is ours: the time blurtd spends answering,
measured on loopback, where single-digit to low-tens of milliseconds — and, more
importantly, steady — is the target. The right answer for a client that needs speed
is to race several nodes and take the first reply, which is exactly what Morphit
does; a seven-node race turns a long-tailed distribution into a short one, and that
buys far more than any per-box tuning.

So instead of a claim, this release ships a measurement:
```
hidden-rpc-bench            # p50/p90/p99 on loopback, Tor and I2P
hidden-rpc-bench --local    # loopback only, builds no circuits
hidden-rpc-bench --json     # for trending
```
Three calls of increasing weight, no privileges needed. No latency numbers are
quoted in the audit on purpose — figures from a build container would be
meaningless, and the only ones that matter come off your actual laptops.

Two latency changes made as a result
- webserver-thread-pool-size 16 → 32 (Blurt's own default). Cutting it to 16
  was a footprint decision that quietly created a latency floor: a client racing
  several nodes opens several connections at once, and waiting for a free thread is
  paid on top of six network hops. Where footprint and answer time conflict on the
  read path, answer time wins now. Still 8× below the 256 this project shipped.
- vm.swappiness = 10, the single biggest latency cliff on this hardware: a
  ~48 GB memory-mapped state on a 8–16 GB machine, served from swap, turns 20 ms
  into seconds. Reversible by deleting one file. hidden-rpc-report now shows swap
  in use and the current swappiness.
And the footprint verdict, honestly

Not as small as possible. The largest remaining win — dropping blurtd's follow and
tags plugins — is deliberately unpulled: it needs a multi-hour replay, and whether
any Morphit code path calls tags_api/follow_api cannot be answered from this
repository. Two smaller levers (reduced Tor padding, shorter I2P tunnels) are
documented and left alone because they trade away this project's first priority.
docs/AUDIT.md §6 has the full accounting.

hidden-rpc v1.0.0

Stable release. A machine you set up once and stop thinking about: it
serves the Blurt chain over Tor and I2P, repairs itself when anything breaks, and
tells you how it is doing from an icon next to your clock.

What 1.0 means here

Every failure this fleet actually hit is now either prevented, detected and repaired
automatically, or — where a fault cannot be named in advance — caught by a rule that
does not need to know what broke:
- the node is supervised as a whole — chain daemon, container, container runtime,
  Tor, I2P, clock, disk, power settings — each with its own repair ladder, and every
  repair verified afterwards rather than assumed;
- "serving" has exactly one definition: the head block is advancing, and it is a
  real head. Nothing that is not serving can pass that test, whatever its logs,
  flags or daemon status say;
- above every diagnosis sits an outcome watchdog. If a node has not served for
  six hours, it escalates anyway: rebuild the state from the block_log, refetch the
  chain if the block_log is gone or a rebuild did not help, reboot as a last resort.
  A node can therefore recover from a failure mode that did not exist when this code
  was written;
- DEGRADED is a slow retry, never a resting place. It keeps deep-repairing on a
  doubling backoff for as long as the machine has power;
- reboots are boring. Clean shutdown for the blockchain, no pinned replay flag,
  everything enabled at boot, the tray icon back at login. Move the machine to
  another room whenever you like.
Audited before release

docs/AUDIT.md is the full record: every file, shellcheck and pyflakes and bandit
clean, adversarial fuzzing of the one input a stranger controls, and resource
behaviour measured over time rather than reasoned about.

It found seven real defects, two of them local privilege escalations — root
executing a script from a user-writable home directory, and a predictable
root-written temp file open to a symlink attack. Both fixed and regression-tested,
along with a slow memory creep in the supervisor (now bounded by a daily same-pid
re-exec), zombie processes from the tray menu, and a URL field that accepted more
characters than it should.
```
shellcheck   0 errors      pyflakes  0 issues      bandit  0 high, 0 medium
239 assertions across 4 test suites, all passing from a clean extraction
```
If you are upgrading
```
bash installer/setup.sh
```
Safe at any time, including while a node is rebuilding its state — the installer now
refuses to touch a rebuild in progress. Your addresses, chain data and settings are
untouched.

Requirements
- A 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Everyday commands
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
sudo hidden-rpc-report            # every relevant fact in one file, changes nothing
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
Or use the icon by the clock: green serving, amber rebuilding, red not serving, grey
stopped. Click it for progress, or to stop and start the node.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Earlier releases

hidden-rpc v0.4.8

Upgrades now replace the tray icon that is actually running

If your tray menu still says "Watch progress…", you are looking at the old
indicator: v0.4.6 renamed those entries to "Show progress…" and "Open in a
terminal…". The installer wrote the new file to disk and then skipped starting it,
because one was already running — so the old process stayed in the tray, with the
old menu and the old bug, until the next login. The fix shipped and went on doing
nothing, which is the worst way for a fix to behave.

Setup now stops the running indicator and starts the new one, reading the session
environment (DISPLAY, X authority, session bus) from the process it is replacing
rather than guessing it, and confirms the new one came up.

The indicator also answers --version now, so "which one is actually in the tray?"
is a question with an answer:
```
/usr/local/bin/hidden-rpc-indicator --version
```
To swap it immediately without waiting for anything:
```
pkill -f hidden-rpc-indicator
setsid /usr/local/bin/hidden-rpc-indicator >/dev/null 2>&1 &
```
A stray directory was being packaged and shipped

The v0.4.6 tarball contained a directory tree named
supervisor/ExecStart=/usr/sbin/i2pd --conf=.... An edit had briefly left
i2p_tunnel_file echoing a systemd ExecStart line along with the path it was
supposed to return; the caller then created that as a directory. The cause was
already fixed, but the debris had been committed and packaged, because nothing was
checking what went into the tarball.

The lint suite now refuses to pass if the repository contains build artefacts,
editor leftovers, paths with = or spaces in them (the signature of a mangled shell
redirect), or unexpected top-level directories. Verified by planting the exact
directory again and watching it fail.

Nothing was broken by it on a node — it was never installed anywhere, only shipped.

"Watch progress…" did nothing. Now there is a window.

It ran x-terminal-emulator -e "hidden-rpc-status --watch". Newer gnome-terminal —
what Mint uses — ignores -e, so the click did nothing, silently, with no error
anywhere to notice.

The menu now has two entries:
- Show progress… opens a small window of our own, with a real progress bar
  during a rebuild (49.9% — block 31,400,000 of 62,905,338), the headline state,
  and both transports. It depends on nothing but the tray icon itself, so it always
  works.
- Open in a terminal… still gives you the live text view, now invoking each
  terminal the way that terminal actually accepts (gnome-terminal, mate-terminal,
  xfce4-terminal, tilix, konsole, xterm) and keeping the window open afterwards, so
  a failure is visible instead of a flash. If no terminal is found it falls back to
  the window rather than doing nothing.
Asking how a node is doing no longer asks for your password

hidden-rpc-status ran sudo docker logs to read the replay percentage, which
dropped a password prompt into the middle of its own output — you saw exactly that
while watching kc rebuild.

Reading blurtd's log needs root; the supervisor already is root, so it now
publishes the progress numbers in its status file (replay_percent,
replay_block, replay_total). The status tool and the tray icon read them from
there. Neither needs any privileges to look at a node now — only stopping and
starting does.

Found why kc lost its name

The node's name was stored only in $HOME/.hidden-rpc/name. Run the installer with
sudo — or anything that reaches the setup step as root — and $HOME is /root, so it
found no name file, concluded the node was unnamed, and wrote that empty answer into
the machine-wide supervisor.conf. The name was silently erased.

Two fixes: the name is now looked up in the invoking user's real home (via
SUDO_USER) and, failing that, read back from supervisor.conf, which is the
authoritative machine-wide copy; and an empty name is never written over a real one.
When a name is found, both copies are kept in step.

To restore kc's name:
```
hidden-rpc-setup --name kc
```
Cosmetic: no contradictory warnings during a rebuild

A node deliberately replaying its block_log was showing
⚠ the node is down — being repaired next to transports reported as healthy —
two statements that cannot both be true, on a node where nothing was wrong. During a
rebuild the RPC being unavailable is the expected state, not a fault, and the warning
is now suppressed for it (and never shown beside a healthy transport).

The report paid for itself immediately: three findings

1. The node could never have recovered, and now it repairs itself in a minute.
kc's state held block 1 while a 23 GB block_log sat on the disk next to it. blurtd
was asking peers for blocks, receiving them, and rejecting every single one —
unlinkable block ... does not link to known chain — because it has no chain to link
them to. A node in that state will never catch up, however long you leave it.

That is provable from two numbers we already collect: the head block, and the size of
the block_log. The supervisor now checks for it directly and rebuilds the state from
the block_log within a minute, instead of waiting out the six-hour
"it has not served" timeout. A node genuinely syncing from scratch (low head block,
no block_log) is untouched — there is nothing to replay.

2. The six-hour timer was being reset by every installer run. LAST_SERVED_AT
was written to the persist file but never read back — the three restore lines landed
outside load_persist when I edited it, and nothing checked. Since every installer
run restarts the supervisor, the not-serving clock went back to zero each time. On a
node being actively worked on, the outcome watchdog could effectively never fire.
Fixed, with a test that round-trips every persisted field.

3. The report's own service table said failed=active for everything, which is
what systemctl is-failed prints when a unit is fine. It now says "not failed" or
"FAILED", which is what the column was for.

The installer was destroying rebuilds. Interlocked now.

v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.

This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.

The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:
- blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
  not serving a real head block. It reads the container's command and the head block
  — data, not logs, which is what defeated the earlier attempts;
- blurtd_deflag and blurtd_recreate both refuse while it is true;
- the installer no longer makes this decision at all. It reports "this node is
  rebuilding its chain state — leaving it completely alone" and moves on. The
  supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.

A random-failure bug in seventeen places

Chasing an intermittent test failure turned up something worse than the test: every
script here runs under set -o pipefail, and seventeen of them used
something | grep -q pattern. grep -q exits the instant it matches, the process
writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.

Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.

All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.

sudo hidden-rpc-report — every fact in one file

Diagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:
- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as hidden-rpc-setup --report.

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v1.0.1.tar.gz
  0 downloads · 2026-09-12 18:44:42 +00:00 · 185 KiB
- hidden-rpc-v1.0.1.tar.gz.sha256
  0 downloads · 2026-09-12 18:44:44 +00:00 · 91 B

v0.4.7 214465c8b9

hidden-rpc v0.4.7 Stable
agorise released this 2026-09-12 17:47:41 +00:00 | 3 commits to main since this release
hidden-rpc v0.4.7

A stray directory was being packaged and shipped

The v0.4.6 tarball contained a directory tree named
supervisor/ExecStart=/usr/sbin/i2pd --conf=.... An edit had briefly left
i2p_tunnel_file echoing a systemd ExecStart line along with the path it was
supposed to return; the caller then created that as a directory. The cause was
already fixed, but the debris had been committed and packaged, because nothing was
checking what went into the tarball.

The lint suite now refuses to pass if the repository contains build artefacts,
editor leftovers, paths with = or spaces in them (the signature of a mangled shell
redirect), or unexpected top-level directories. Verified by planting the exact
directory again and watching it fail.

Nothing was broken by it on a node — it was never installed anywhere, only shipped.

"Watch progress…" did nothing. Now there is a window.

It ran x-terminal-emulator -e "hidden-rpc-status --watch". Newer gnome-terminal —
what Mint uses — ignores -e, so the click did nothing, silently, with no error
anywhere to notice.

The menu now has two entries:
- Show progress… opens a small window of our own, with a real progress bar
  during a rebuild (49.9% — block 31,400,000 of 62,905,338), the headline state,
  and both transports. It depends on nothing but the tray icon itself, so it always
  works.
- Open in a terminal… still gives you the live text view, now invoking each
  terminal the way that terminal actually accepts (gnome-terminal, mate-terminal,
  xfce4-terminal, tilix, konsole, xterm) and keeping the window open afterwards, so
  a failure is visible instead of a flash. If no terminal is found it falls back to
  the window rather than doing nothing.
Asking how a node is doing no longer asks for your password

hidden-rpc-status ran sudo docker logs to read the replay percentage, which
dropped a password prompt into the middle of its own output — you saw exactly that
while watching kc rebuild.

Reading blurtd's log needs root; the supervisor already is root, so it now
publishes the progress numbers in its status file (replay_percent,
replay_block, replay_total). The status tool and the tray icon read them from
there. Neither needs any privileges to look at a node now — only stopping and
starting does.

Found why kc lost its name

The node's name was stored only in $HOME/.hidden-rpc/name. Run the installer with
sudo — or anything that reaches the setup step as root — and $HOME is /root, so it
found no name file, concluded the node was unnamed, and wrote that empty answer into
the machine-wide supervisor.conf. The name was silently erased.

Two fixes: the name is now looked up in the invoking user's real home (via
SUDO_USER) and, failing that, read back from supervisor.conf, which is the
authoritative machine-wide copy; and an empty name is never written over a real one.
When a name is found, both copies are kept in step.

To restore kc's name:
```
hidden-rpc-setup --name kc
```
Cosmetic: no contradictory warnings during a rebuild

A node deliberately replaying its block_log was showing
⚠ the node is down — being repaired next to transports reported as healthy —
two statements that cannot both be true, on a node where nothing was wrong. During a
rebuild the RPC being unavailable is the expected state, not a fault, and the warning
is now suppressed for it (and never shown beside a healthy transport).

The report paid for itself immediately: three findings

1. The node could never have recovered, and now it repairs itself in a minute.
kc's state held block 1 while a 23 GB block_log sat on the disk next to it. blurtd
was asking peers for blocks, receiving them, and rejecting every single one —
unlinkable block ... does not link to known chain — because it has no chain to link
them to. A node in that state will never catch up, however long you leave it.

That is provable from two numbers we already collect: the head block, and the size of
the block_log. The supervisor now checks for it directly and rebuilds the state from
the block_log within a minute, instead of waiting out the six-hour
"it has not served" timeout. A node genuinely syncing from scratch (low head block,
no block_log) is untouched — there is nothing to replay.

2. The six-hour timer was being reset by every installer run. LAST_SERVED_AT
was written to the persist file but never read back — the three restore lines landed
outside load_persist when I edited it, and nothing checked. Since every installer
run restarts the supervisor, the not-serving clock went back to zero each time. On a
node being actively worked on, the outcome watchdog could effectively never fire.
Fixed, with a test that round-trips every persisted field.

3. The report's own service table said failed=active for everything, which is
what systemctl is-failed prints when a unit is fine. It now says "not failed" or
"FAILED", which is what the column was for.

The installer was destroying rebuilds. Interlocked now.

v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.

This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.

The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:
- blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
  not serving a real head block. It reads the container's command and the head block
  — data, not logs, which is what defeated the earlier attempts;
- blurtd_deflag and blurtd_recreate both refuse while it is true;
- the installer no longer makes this decision at all. It reports "this node is
  rebuilding its chain state — leaving it completely alone" and moves on. The
  supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.

A random-failure bug in seventeen places

Chasing an intermittent test failure turned up something worse than the test: every
script here runs under set -o pipefail, and seventeen of them used
something | grep -q pattern. grep -q exits the instant it matches, the process
writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.

Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.

All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.

sudo hidden-rpc-report — every fact in one file

Diagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:
- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as hidden-rpc-setup --report.

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.4.7.tar.gz
  0 downloads · 2026-09-12 17:47:47 +00:00 · 162 KiB
- hidden-rpc-v0.4.7.tar.gz.sha256
  0 downloads · 2026-09-12 17:47:49 +00:00 · 91 B

v0.4.3 7db213a368

hidden-rpc v0.4.3 Stable
agorise released this 2026-09-12 01:09:06 +00:00 | 4 commits to main since this release
hidden-rpc v0.4.3

The report paid for itself immediately: three findings

1. The node could never have recovered, and now it repairs itself in a minute.
kc's state held block 1 while a 23 GB block_log sat on the disk next to it. blurtd
was asking peers for blocks, receiving them, and rejecting every single one —
unlinkable block ... does not link to known chain — because it has no chain to link
them to. A node in that state will never catch up, however long you leave it.

That is provable from two numbers we already collect: the head block, and the size of
the block_log. The supervisor now checks for it directly and rebuilds the state from
the block_log within a minute, instead of waiting out the six-hour
"it has not served" timeout. A node genuinely syncing from scratch (low head block,
no block_log) is untouched — there is nothing to replay.

2. The six-hour timer was being reset by every installer run. LAST_SERVED_AT
was written to the persist file but never read back — the three restore lines landed
outside load_persist when I edited it, and nothing checked. Since every installer
run restarts the supervisor, the not-serving clock went back to zero each time. On a
node being actively worked on, the outcome watchdog could effectively never fire.
Fixed, with a test that round-trips every persisted field.

3. The report's own service table said failed=active for everything, which is
what systemctl is-failed prints when a unit is fine. It now says "not failed" or
"FAILED", which is what the column was for.

The installer was destroying rebuilds. Interlocked now.

v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.

This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.

The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:
- blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
  not serving a real head block. It reads the container's command and the head block
  — data, not logs, which is what defeated the earlier attempts;
- blurtd_deflag and blurtd_recreate both refuse while it is true;
- the installer no longer makes this decision at all. It reports "this node is
  rebuilding its chain state — leaving it completely alone" and moves on. The
  supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.

A random-failure bug in seventeen places

Chasing an intermittent test failure turned up something worse than the test: every
script here runs under set -o pipefail, and seventeen of them used
something | grep -q pattern. grep -q exits the instant it matches, the process
writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.

Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.

All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.

sudo hidden-rpc-report — every fact in one file

Diagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:
- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as hidden-rpc-setup --report.

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.4.3.tar.gz
  0 downloads · 2026-09-12 01:09:12 +00:00 · 158 KiB
- hidden-rpc-v0.4.3.tar.gz.sha256
  0 downloads · 2026-09-12 01:09:13 +00:00 · 91 B

v0.4.2 b98b45724b

hidden-rpc v0.4.2 Stable
agorise released this 2026-09-11 23:24:54 +00:00 | 5 commits to main since this release
hidden-rpc v0.4.2

The installer was destroying rebuilds. Interlocked now.

v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.

This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.

The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:
- blurtd_rebuild_in_progress is true when the replay flag is pinned AND the node is
  not serving a real head block. It reads the container's command and the head block
  — data, not logs, which is what defeated the earlier attempts;
- blurtd_deflag and blurtd_recreate both refuse while it is true;
- the installer no longer makes this decision at all. It reports "this node is
  rebuilding its chain state — leaving it completely alone" and moves on. The
  supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.

A random-failure bug in seventeen places

Chasing an intermittent test failure turned up something worse than the test: every
script here runs under set -o pipefail, and seventeen of them used
something | grep -q pattern. grep -q exits the instant it matches, the process
writing to the pipe gets SIGPIPE, and pipefail turns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.

Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.

All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.

sudo hidden-rpc-report — every fact in one file

Diagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:
- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as hidden-rpc-setup --report.

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.4.2.tar.gz
  0 downloads · 2026-09-11 23:25:00 +00:00 · 155 KiB
- hidden-rpc-v0.4.2.tar.gz.sha256
  0 downloads · 2026-09-11 23:25:01 +00:00 · 91 B

v0.4.1 cdd4841ad1

hidden-rpc v0.4.1 Stable
agorise released this 2026-09-11 22:48:25 +00:00 | 6 commits to main since this release
hidden-rpc v0.4.1

The hidden-rpc icon, and a plain answer about restarting

The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.

Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).

Stopping frees the machine, starting hands it back

Stop the node stops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).

Start the node brings everything back, including anything a --all stop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your .onion
and .b32.i2p addresses never change.

An icon by your clock, and menu entries — no terminal needed

You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:
- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.

Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.

Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.

The icon needs no privileges: it reads the same status file everything else does.

Which processes are hidden-rpc?

None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:

process who runs it what it is

blurtd root the chain node itself (inside Docker)

transport-supervisor.sh root the self-healer

tor debian-tor your .onion

i2pd i2pd your .b32.i2p

"Catching up — 63,560,408 blocks to go" was the last dishonest line

A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.

Everything the tools print about this node's health is now derived from the same
test the supervisor uses.

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.4.1.tar.gz
  0 downloads · 2026-09-11 22:48:30 +00:00 · 148 KiB
- hidden-rpc-v0.4.1.tar.gz.sha256
  0 downloads · 2026-09-11 22:48:32 +00:00 · 91 B

v0.3.8 cd515b56bc

hidden-rpc v0.3.8 Stable
agorise released this 2026-09-11 22:16:45 +00:00 | 7 commits to main since this release
hidden-rpc v0.3.8

The installer is now installer/setup.sh

Shorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.

Every command this project prints can now be typed exactly as printed

setup.sh --doctor was printed as advice and is not on anyone's PATH, so typing it
gives command not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.

Setup now installs a small hidden-rpc-setup wrapper into /usr/local/bin, so all
of these work from any directory:
```
hidden-rpc-status                 # everything: node, reachability, self-repairs
hidden-rpc-status --watch         # live progress, one calm line
hidden-rpc-setup --doctor         # check and repair everything right now
hidden-rpc-setup --rebootstrap    # last resort: fetch the whole chain again
sudo hidden-rpc-nodectl stop      # stop it safely before unplugging or moving it
```
The wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicit bash <path>.

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.3.8.tar.gz
  0 downloads · 2026-09-11 22:16:50 +00:00 · 136 KiB
- hidden-rpc-v0.3.8.tar.gz.sha256
  0 downloads · 2026-09-11 22:16:52 +00:00 · 91 B

v0.3.7 ef76101523

hidden-rpc v0.3.7 Stable
agorise released this 2026-09-11 22:02:13 +00:00 | 8 commits to main since this release
hidden-rpc v0.3.7

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/hidden-rpc-setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/hidden-rpc-setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/hidden-rpc-setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/hidden-rpc-setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: hidden-rpc-setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.3.7.tar.gz
  0 downloads · 2026-09-11 22:02:19 +00:00 · 134 KiB
- hidden-rpc-v0.3.7.tar.gz.sha256
  0 downloads · 2026-09-11 22:02:21 +00:00 · 91 B

v0.3.6 ecd8958d01

hidden-rpc v0.3.6 Stable
agorise released this 2026-09-11 21:52:40 +00:00 | 9 commits to main since this release
hidden-rpc v0.3.6

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/hidden-rpc-setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/hidden-rpc-setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/hidden-rpc-setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/hidden-rpc-setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: hidden-rpc-setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.3.6.tar.gz
  0 downloads · 2026-09-11 21:52:46 +00:00 · 132 KiB
- hidden-rpc-v0.3.6.tar.gz.sha256
  0 downloads · 2026-09-11 21:52:48 +00:00 · 91 B

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

process	who runs it	what it is
`blurtd`	root	the chain node itself (inside Docker)
`transport-supervisor.sh`	root	the self-healer
`tor`	debian-tor	your .onion
`i2pd`	i2pd	your .b32.i2p

1 2 3

hidden-rpc v1.1.0 Stable

hidden-rpc v1.1.0

The bug that took Tor off this fleet

hidden-rpc no longer restarts your computer

Logging that survives a reboot and deletes itself after two weeks

The node now records its own decisions

Two new commands

hidden-rpc v1.0.2

The tray icon now comes back after a cold boot

Threat-modelled, red-teamed, and honest about latency

On "it must be lightning fast"

Two latency changes made as a result

And the footprint verdict, honestly

hidden-rpc v1.0.0

What 1.0 means here

Audited before release

If you are upgrading

Requirements

Everyday commands

Earlier releases

hidden-rpc v0.4.8

Upgrades now replace the tray icon that is actually running

A stray directory was being packaged and shipped

"Watch progress…" did nothing. Now there is a window.

Asking how a node is doing no longer asks for your password

Found why kc lost its name

Cosmetic: no contradictory warnings during a rebuild

The report paid for itself immediately: three findings

The installer was destroying rebuilds. Interlocked now.

A random-failure bug in seventeen places

sudo hidden-rpc-report — every fact in one file

The hidden-rpc icon, and a plain answer about restarting

Stopping frees the machine, starting hands it back

An icon by your clock, and menu entries — no terminal needed

Which processes are hidden-rpc?

"Catching up — 63,560,408 blocks to go" was the last dishonest line

The installer is now installer/setup.sh

Every command this project prints can now be typed exactly as printed

--doctor, --reindex, --headless and --desktop did nothing at all

One rule above all the others: is this node serving?

New commands and reporting

Stop guessing from log text: watch the head block instead

I broke I2P in v0.3.1, and this fixes it

A replaying node is no longer mistaken for a serving one

A node that cannot open its own database now fixes itself

Since v0.3.1: two judgement bugs found on a real node

If you tried v0.3.0: this is the fix

Why nodes were going dark and staying dark

What's new

A much lighter node

New commands

Upgrading

Minimum requirements

Testing

hidden-rpc v1.0.2 Stable

hidden-rpc v1.0.2

The tray icon now comes back after a cold boot

Threat-modelled, red-teamed, and honest about latency

On "it must be lightning fast"

Two latency changes made as a result

And the footprint verdict, honestly

hidden-rpc v1.0.0

What 1.0 means here

Audited before release

If you are upgrading

Requirements

Everyday commands

Earlier releases

hidden-rpc v0.4.8

Upgrades now replace the tray icon that is actually running

A stray directory was being packaged and shipped

"Watch progress…" did nothing. Now there is a window.

Asking how a node is doing no longer asks for your password

Found why kc lost its name

Cosmetic: no contradictory warnings during a rebuild

The report paid for itself immediately: three findings

The installer was destroying rebuilds. Interlocked now.

A random-failure bug in seventeen places

sudo hidden-rpc-report — every fact in one file

The hidden-rpc icon, and a plain answer about restarting

`sudo hidden-rpc-report` — every fact in one file

The installer is now `installer/setup.sh`

`--doctor`, `--reindex`, `--headless` and `--desktop` did nothing at all

`sudo hidden-rpc-report` — every fact in one file

The installer is now `installer/setup.sh`

`--doctor`, `--reindex`, `--headless` and `--desktop` did nothing at all

`sudo hidden-rpc-report` — every fact in one file

The installer is now `installer/setup.sh`

`--doctor`, `--reindex`, `--headless` and `--desktop` did nothing at all