hidden-rpc v0.3.7

agorise/hidden-rpc

Watch

Fork 0

26 releases 26 tags

RSS feed

v0.3.7 ef76101523

hidden-rpc v0.3.7 Stable
agorise released this 2026-09-11 22:02:13 +00:00 | 8 commits to main since this release
hidden-rpc v0.3.7

--doctor, --reindex, --headless and --desktop did nothing at all

They parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.

The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.

--doctor also now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.

One rule above all the others: is this node serving?

Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.

Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC with head block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)

And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:
1. rebuild the state by replaying the local block_log — if the block_log is intact;
2. if the block_log is missing or truncated, replaying cannot conjure blocks that are
  not there, so fetch the whole chain again from the signed on-chain pointer;
3. if a rebuild already happened and the node still is not serving, the data itself
  is suspect — fetch it again;
4. and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.

This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.

New commands and reporting

hidden-rpc-status no longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:
```
○ NOT serving — its head block is not advancing  (for 3h)
  head block 1. The supervisor escalates on its own if this lasts:
  rebuild the state from the block_log, then refetch the chain if that fails.
```
--doctor now diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:
```
./installer/hidden-rpc-setup.sh --rebootstrap   # throw the local chain away, fetch it again
```
Stop guessing from log text: watch the head block instead

v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".

That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned --replay-blockchain flag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.

The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.

Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and both hidden-rpc-status and the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".

Also: hidden-rpc-status no longer shows ? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.

I broke I2P in v0.3.1, and this fixes it

The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a [section]
header belongs to that section. So notransit = true at the end of the file became
cpuext.notransit, and i2pd refused to start at all:
```
unrecognised option 'cpuext.notransit'
```
Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the .onion kept working, so it looked like a flaky tunnel.

Fixed three ways:
- the block is now inserted before the first section header, which is the only
  place global options work. Re-running the installer rewrites a misplaced block in
  the right place;
- applying it is verified: the installer restarts i2pd and checks it actually
  came up. If it did not, the config is put back exactly as it was and you are told
  I2P is running on defaults instead;
- the supervisor's deep repair now suspects our own changes first: if i2pd is
  down and our managed block is in its config, the block comes out and i2pd is
  restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.

A replaying node is no longer mistaken for a serving one

During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, and hidden-rpc-status shows real replay
progress instead of head block 1.

A node that cannot open its own database now fixes itself

On a real node, blurtd was exiting immediately on every start with:
```
Chain state does not match block log. Please reindex blockchain.
```
This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.

Two things came out of it.

A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reports running. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.

blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.

The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:
```
./installer/hidden-rpc-setup.sh --reindex
```
hidden-rpc-status reports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."

Since v0.3.1: two judgement bugs found on a real node

v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.

1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, and GRACE_NODE, now
1 hour). hidden-rpc-status says which it is: "its log is still moving, so it is
working — leave it alone".

2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takes PROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P = 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.

Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to use hidden-rpc-status --watch — the self-healer is
already watching the node by then, so there is nothing to sit and guard.

If you tried v0.3.0: this is the fix

v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.

The installer loaded its shared repair library like this:
```
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"
```
A variable assignment written in front of the . builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own : "${DOCKER_CMD:=docker}" saw the name
as already set and skipped its default — so DOCKER_CMD ended up unset, and
every container command after that ran as timeout 120 run -d …, which fails
instantly. The library's error went to /dev/null, so all you saw was "the node
didn't start on this try".

Fixed in three ways, so it cannot happen again in this shape or any other:
- one load_heal_lib function does the loading with plain assignments, and
  verifies afterwards that the helpers exist and the command wrappers are
  non-empty — stopping with a clear message if not, before touching anything;
- container failures now report what the container runtime actually said,
  instead of a shrug;
- a third test suite, supervisor/tests/installer-lint.sh, runs the real
  start_node out of the real installer against a fake docker and asserts the
  exact command line it produces. That is the test that was missing: v0.3.0's
  supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.

Everything below is the v0.3.0 release, unchanged.

A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.

Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.

Why nodes were going dark and staying dark

Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
  supervisor kept probing it but never repaired it again. The only way back
  was a recovery that happened on its own. Any node that reached this state was
  off the network until a human walked over to it.
- "The node is down" meant "do nothing", forever. When the chain daemon
  stopped answering, the supervisor correctly refused to restart Tor and I2P —
  and then stopped there. Nothing in the system ever restarted blurtd, its
  container, or Docker.
- Repairs were never checked. On Linux Mint, tor.service is a wrapper whose
  start command is literally /bin/true, so restarting it can report success
  while the real daemon stays dead. And after five quick failures systemd latches
  a service into a permanent failed state that refuses every further restart until
  someone runs reset-failed — which nothing ever did, including for the
  supervisor itself. Self-healing could be switched off by a single bad minute.
- The replay flag was pinned forever. A node set up from the block_log kept
  --replay-blockchain in its container command permanently, so every reboot
  re-ran a multi-hour replay. Which looks exactly like "it went down and never
  came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
  chain daemon, the container, the container runtime, the clock, the disk and the
  power settings. A stopped node is started; a missing container is rebuilt from a
  saved spec; a node whose head block has frozen is treated as wedged and
  repaired; a dead Docker is restarted.
- DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
  again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
  power. Restart thrash is still capped.
- Every repair is verified. After a restart the supervisor checks that the
  unit really is active with a live process, that the proxy port answers, and that
  the address still exists — and escalates if not. It resolves the real Tor unit
  at runtime instead of trusting a name guessed at install time, and runs
  reset-failed before every restart.
- Deeper repairs for the failures restarting can never fix: a torrc that lost
  its hidden-service stanza (put back, verified with tor --verify-config, rolled
  back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
  (now written to tunnels.conf.d, where upgrades cannot touch it), and missing
  hidden-service keys (restored from a local backup, so your .onion and
  .b32.i2p survive a purge or a bad restore).
- Reboots are boring now. The replay flag is removed automatically as soon as
  a node proves it is serving; a new shutdown unit gives the blockchain the full
  time it needs to save itself before Docker goes away, so there are no more dirty
  shutdowns and no surprise replays; and everything is enabled to come back on its
  own. Move a node to another room whenever you like.
- Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
  the systemd level, where no desktop power setting can override them. If a
  machine does sleep anyway, waking up triggers an immediate full re-check.
- A controlled reboot as the genuine last resort — only when every transport
  is down, only after two hours of failed repairs, never during a replay, never
  just after booting, at most once a day, and always after stopping the blockchain
  cleanly. Set ALLOW_REBOOT="0" in /etc/hidden-rpc/supervisor.conf to forbid it.
A much lighter node

Full detail, with measurements and the trade-offs, in docs/FOOTPRINT.md.
- No more permanent replay. This was the single largest CPU cost in the fleet,
  and it is gone.
- The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
  loop no longer spawns anything at all: it reads /proc with shell builtins
  instead of shelling out to ip, awk, cksum, date and sed several times
  every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check.
- Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
  5 minutes once a transport has been steady for half an hour, and snap back
  instantly on any network change, wake-up or failure.
- I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
  on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
  node carries, for no benefit to its own reachability. Now notransit = true,
  with the distro's always-on IRC tunnel switched off too. Your own .b32.i2p
  server tunnel is unaffected.
- blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
  connections with a smaller sync prefetch. Neither needs a replay.
- Optional --headless stops the desktop on a node nobody looks at, freeing
  RAM and the GPU. Reversible with --desktop.
New commands
```
hidden-rpc-status                  # everything: node, reachability, self-repairs, host
hidden-rpc-status --watch          # live progress, one calm line
sudo hidden-rpc-nodectl stop       # stop a node safely before unplugging or moving it
sudo hidden-rpc-nodectl start      # and back on again
./installer/hidden-rpc-setup.sh --doctor    # check and repair everything, right now
```
hidden-rpc-nodectl stop also turns maintenance mode on, so the supervisor leaves
the node alone while you work; start turns it back off. A node can never be left
quietly un-supervised. hidden-rpc-transport-status still works and shows the
transports-only view.

Upgrading

Re-run the installer on each node (safe anytime, never harms a node):
```
bash installer/hidden-rpc-setup.sh
```
On a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.conf is kept as you have it; only the new settings
this version adds are appended.

To fix a node without a full installer run: hidden-rpc-setup.sh --doctor.

Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
  successfully around 50% of the time versus 100% for Ethernet.
Testing

Both suites run anywhere, with no Tor, I2P or Docker needed:
```
bash supervisor/tests/fsm-sim.sh          # state machine + policy
bash supervisor/tests/heal-sim.sh         # the repairs themselves
bash supervisor/tests/installer-lint.sh   # the installer, actually executed
```
heal-sim.sh drives the repair code against fake systemctl and docker
commands that reproduce the real traps: the /bin/true Tor wrapper, systemd's
start limiter, a pinned replay flag, and unless-stopped. The hardware scenarios
are in docs/SELF-HEALING.md.
- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
- Source code (ZIP)
  0 downloads
- Source code (TAR.GZ)
  0 downloads
- hidden-rpc-v0.3.7.tar.gz
  0 downloads · 2026-09-11 22:02:19 +00:00 · 134 KiB
- hidden-rpc-v0.3.7.tar.gz.sha256
  0 downloads · 2026-09-11 22:02:21 +00:00 · 91 B

hidden-rpc v0.3.7 Stable

--doctor, --reindex, --headless and --desktop did nothing at all

One rule above all the others: is this node serving?

New commands and reporting

Stop guessing from log text: watch the head block instead

I broke I2P in v0.3.1, and this fixes it

A replaying node is no longer mistaken for a serving one

A node that cannot open its own database now fixes itself

Since v0.3.1: two judgement bugs found on a real node

If you tried v0.3.0: this is the fix

Why nodes were going dark and staying dark

What's new

A much lighter node

New commands

Upgrading

Minimum requirements

Testing

`--doctor`, `--reindex`, `--headless` and `--desktop` did nothing at all