-
hidden-rpc v0.4.2 Stable
released this
2026-09-11 23:24:54 +00:00 | 5 commits to main since this releasehidden-rpc v0.4.2
The installer was destroying rebuilds. Interlocked now.
v0.4.1's installer printed "Replay flag removed" on a node that was hours into
rebuilding its chain state, and threw that work away. Its health test was "does the
RPC answer" — and a rebuilding node answers, with head block 1.This is the same mistake the supervisor made in v0.3.4. Fixing it there and not here
was the error: the decision was being made in two places, by two different tests.The guard is now a hard interlock inside the repair library itself, so no caller
can get it wrong:blurtd_rebuild_in_progressis true when the replay flag is pinned AND the node is
not serving a real head block. It reads the container's command and the head block
— data, not logs, which is what defeated the earlier attempts;blurtd_deflagandblurtd_recreateboth refuse while it is true;- the installer no longer makes this decision at all. It reports "this node is
rebuilding its chain state — leaving it completely alone" and moves on. The
supervisor removes the flag, once, when the node genuinely serves.
Tested from both callers, including the exact case that bit us twice: a rebuilding
node whose log shows no replay lines.A random-failure bug in seventeen places
Chasing an intermittent test failure turned up something worse than the test: every
script here runs underset -o pipefail, and seventeen of them used
something | grep -q pattern.grep -qexits the instant it matches, the process
writing to the pipe gets SIGPIPE, andpipefailturns that into a failed pipeline —
sometimes, depending on whether the writer had finished. Measured on an idle
machine: about one run in three.Two of those were in the installer's decision about whether the node container
exists. A random false there sends the installer down the wrong branch — recreating
a container it should have left alone, or leaving one alone it should have rebuilt.
Some of the confusing behaviour on this fleet may well have been this.All seventeen are rewritten to consume the whole pipe. The previously flaky suite now
passes 25 runs out of 25.sudo hidden-rpc-report— every fact in one fileDiagnosing a node one question per round is a bad way to work; each round costs a
reinstall and a wait. This writes down everything that could explain a node's
behaviour, in one pass, and changes nothing:- the head block sampled three times ten seconds apart (does it move?)
- the container's exact command, state, exit code and restart count
- how much block_log is really on disk
- blurtd's last 60 log lines and any replay progress lines
- every unit's active/enabled/failed state, including start-limit latches
- the supervisor's status file, persisted state and last 60 journal lines
- torrc stanza, onion hostname, i2pd tunnel and keys, key backups
- loopback RPC, both proxy ports, clock, disk, memory
Also available as
hidden-rpc-setup --report.The hidden-rpc icon, and a plain answer about restarting
The tray icon and the menu entries now use the project's own icon rather than
borrowed system ones. The tray shows the mark with a small status badge in the
corner — green for serving, amber while rebuilding, red when it is not serving, grey
when stopped — so it is recognisably hidden-rpc and readable at a glance. The menu
entries now live under Internet.Setup also finishes with a plain-language check of exactly what happens when you
restart the computer, item by item: the node, Docker, the self-healer, Tor, I2P and
the tray icon. If any of them would not come back on their own, it says so and tells
you the fix (run the installer again).Stopping frees the machine, starting hands it back
Stop the nodestops blurtd, which is effectively all of the memory and CPU
hidden-rpc uses. Tor and I2P keep running — a few tens of megabytes — so your
addresses stay published. If you want every last megabyte there is now a second menu
item, Stop everything, including Tor and I2P (hidden-rpc-nodectl stop --all).Start the nodebrings everything back, including anything a--allstop took down.
The node loads its state, catches up on the blocks it missed, and becomes reachable
from outside again by itself; the self-healer takes over from there. Your.onion
and.b32.i2paddresses never change.An icon by your clock, and menu entries — no terminal needed
You should not have to open a terminal to answer "is my node running?". On a machine
with a desktop, setup now adds a tray icon next to the clock:- green arrows — serving the chain, all good
- spinning arrows — rebuilding or starting up; hours, nothing to do
- warning triangle — not serving; the node is repairing itself
- crossed-out plug — stopped, or the supervisor is not running
Hover it for a one-line summary (which transport is reachable, what block it is on,
how long the current phase has been running). Click it to stop the node, start
it again, or open the live view.Your applications menu also gets hidden-rpc: Node status, Start the node and
Stop the node. Stopping asks for your password through the normal desktop
dialog — no sudo, no terminal.Stopping from the icon is the safe stop: it gives blurtd the full time it needs to
write its state out (cutting that short is what causes hours of rebuilding later),
and it turns maintenance mode on so the self-healer will not quietly start the node
again while you are using the machine for something else. Starting turns it back off.The icon needs no privileges: it reads the same status file everything else does.
Which processes are hidden-rpc?
None of them run as you, which is why System Monitor shows nothing by that name —
switch it to All Processes and you will find:process who runs it what it is blurtdroot the chain node itself (inside Docker) transport-supervisor.shroot the self-healer tordebian-tor your .onion i2pdi2pd your .b32.i2p "Catching up — 63,560,408 blocks to go" was the last dishonest line
A node answering from an empty state (head block 1) is not behind, and it is not
catching up: it is not serving, and it will not get there on its own. The installer
said otherwise because it compared the head block against the live chain without
asking whether that head meant anything. It now says plainly that the node is not
serving and points at the repair.Everything the tools print about this node's health is now derived from the same
test the supervisor uses.The installer is now
installer/setup.shShorter, and it no longer stutters ("hidden-rpc/installer/hidden-rpc-setup.sh"). The
old name is gone rather than aliased; update any script or note that referenced it.Every command this project prints can now be typed exactly as printed
setup.sh --doctorwas printed as advice and is not on anyone's PATH, so typing it
givescommand not found. The full-path form the installer printed instead was long
enough to wrap, and inconsistent with the two short commands beside it.Setup now installs a small
hidden-rpc-setupwrapper into/usr/local/bin, so all
of these work from any directory:hidden-rpc-status # everything: node, reachability, self-repairs hidden-rpc-status --watch # live progress, one calm line hidden-rpc-setup --doctor # check and repair everything right now hidden-rpc-setup --rebootstrap # last resort: fetch the whole chain again sudo hidden-rpc-nodectl stop # stop it safely before unplugging or moving itThe wrapper runs whichever installer last set this node up, recorded in
/etc/hidden-rpc/blurtd.run. A lint test now fails the build if any tool prints a
command that is neither an installed command nor an explicitbash <path>.--doctor,--reindex,--headlessand--desktopdid nothing at allThey parsed into an action that had no case in the dispatch, so the installer exited
silently, printing nothing. My patch adding those cases never matched the file and I
did not check that it had applied. They work now.The reason this kept happening is worth more than the fix: three separate defects in
this series have been installer paths that no test ever executed. So the lint suite
now walks the installer's own flag table and asserts that every flag reaches a case
in the dispatch, and that every case calls a function that exists — verified to fail
when a case is removed. Documented flags and accepted flags are checked against each
other too.--doctoralso now reports how much block_log is actually on disk as its first line,
since that single number decides which repair a stuck node needs.One rule above all the others: is this node serving?
Every release since v0.3.0 fixed a real fault and then met a new one nobody had
thought of. That is the wrong shape for software that is meant to be set up once and
forgotten, because it means each unanticipated failure is an outage that waits for a
human. v0.3.6 changes the shape.Serving is now defined by one fact, and nothing else: the head block is advancing,
and it is a real head rather than a freshly-initialised one. Blurt produces a block
every three seconds, so a node that is genuinely serving cannot fail that test, and
nothing that is not serving can pass it — whatever its logs, its container state, its
flags or its daemon status happen to say. (A node answering RPC withhead block 1
was being reported as "Serving", because the code asked the narrower question "is a
replay running?" and got the wrong answer.)And above every diagnosis there is now an outcome watchdog. If a node has not
served the chain for six hours, something is wrong that the specific checks did not
catch, and it escalates anyway:- rebuild the state by replaying the local block_log — if the block_log is intact;
- if the block_log is missing or truncated, replaying cannot conjure blocks that are
not there, so fetch the whole chain again from the signed on-chain pointer; - if a rebuild already happened and the node still is not serving, the data itself
is suspect — fetch it again; - and the reboot rung still applies underneath all of it.
Visible progress (a running replay, a moving log) buys more time, but only up to
twice the window, so nothing can stall forever by printing. Maintenance mode
suppresses all of it. Each step is rate-limited and recorded across restarts.This is the property that makes "set it and forget it" honest: a node can recover
from a failure mode that did not exist when this code was written, because the top
rule asks about the outcome rather than the cause.New commands and reporting
hidden-rpc-statusno longer says "Serving" unless the node is. A node that is not
serving says so, with its head block and what the supervisor will do next:○ NOT serving — its head block is not advancing (for 3h) head block 1. The supervisor escalates on its own if this lasts: rebuild the state from the block_log, then refetch the chain if that fails.--doctornow diagnoses from the two facts that decide which repair is right: it
samples the head block twice fifteen seconds apart, and reports how much block_log is
actually on disk. There is also a new last-resort command:./installer/setup.sh --rebootstrap # throw the local chain away, fetch it againStop guessing from log text: watch the head block instead
v0.3.4 decided whether a node was rebuilding its state or serving by grepping
blurtd's log for replay progress lines. On a real node that grep came back empty at
the wrong moment, so a node sitting at head block 1 — hours into a state rebuild
— was reported as "Serving. head block 1".That was not just a cosmetic error. From "serving", the supervisor's next step is to
remove the pinned--replay-blockchainflag, which would have stopped the rebuild
and left a state at block 1 against a 63-million-block log. blurtd would then refuse
to open with "does not match block log", crash-loop, be detected, and start another
rebuild. Hours, on a loop, indefinitely.The decision no longer depends on log text at all. It uses the one fact that cannot
be missed: is the head block moving? A serving node advances every three seconds.
A node rebuilding its state answers with head block 1 and sits there. Two samples
settle it, and the flag can only ever be removed from a node that has been seen
advancing.Everything downstream got the same treatment: the rebuild is never interrupted, the
wedged-node timer cannot fire during one, and bothhidden-rpc-statusand the
installer now say "rebuilding its state by replaying the block_log" instead of
computing a meaningless "63,559,304 blocks to go".Also:
hidden-rpc-statusno longer shows? starting. The supervisor now runs its
first node check before publishing a status file, so the first thing you read is
what the node is actually doing.I broke I2P in v0.3.1, and this fixes it
The low-power i2pd tuning added in v0.3.1 was appended to the end of
/etc/i2pd/i2pd.conf. That file is an INI file: every key after a[section]
header belongs to that section. Sonotransit = trueat the end of the file became
cpuext.notransit, and i2pd refused to start at all:unrecognised option 'cpuext.notransit'Verified against i2pd 2.49. On any node that took v0.3.1-v0.3.3, I2P has been down
since that install — the.onionkept working, so it looked like a flaky tunnel.Fixed three ways:
- the block is now inserted before the first section header, which is the only
place global options work. Re-running the installer rewrites a misplaced block in
the right place; - applying it is verified: the installer restarts i2pd and checks it actually
came up. If it did not, the config is put back exactly as it was and you are told
I2P is running on defaults instead; - the supervisor's deep repair now suspects our own changes first: if i2pd is
down and our managed block is in its config, the block comes out and i2pd is
restarted without it.
An optimisation that stops a daemon starting is not an optimisation, and the code
that adds one has to be able to take it back.A replaying node is no longer mistaken for a serving one
During a block_log replay blurtd answers RPC with head block 1 and stays there for
hours. v0.3.3 read that as "serving", which started the head-stall timer against a
node doing exactly what it was told, and tried to strip the replay flag off a replay
in progress (harmless — the rebuild refuses mid-replay — but it logged an alarming
warning every minute for hours). The supervisor now checks whether a replay is
running before calling a node serving, andhidden-rpc-statusshows real replay
progress instead ofhead block 1.A node that cannot open its own database now fixes itself
On a real node, blurtd was exiting immediately on every start with:
Chain state does not match block log. Please reindex blockchain.This is what a hard kill or a power cut leaves behind: blurtd's stored state and the
block_log disagree, so it refuses to open, exits 1, the container restart policy
starts it again, and it exits again — forever. Restarting it, rebuilding the
container, rebooting the machine: none of them can ever help, because the state
itself has to be rebuilt.Two things came out of it.
A crash loop was being mistaken for a node hard at work. v0.3.2 decided a node
was busy if its log kept moving — and a crash loop churns its log constantly while
the container reportsrunning. The supervisor now checks the container runtime's
restart counter, which is the only thing that tells those two apart, before it
looks at log activity.blurtd's own diagnosis is now read and acted on. It says exactly what is wrong
and what the fix is, in plain English, in its log. When the supervisor sees that
message it rebuilds the state by replaying the block_log once — hours, unattended —
records that it did, and removes the replay flag again as soon as the node serves.
It will not start a second replay on top of a running one, will not repeat it
within 12 hours, and will not start one at all while maintenance mode is on.The installer handles it too: it reads the old container's log before removing it,
so a re-run on an affected node starts the rebuild automatically and says what it is
doing. There is also an explicit command:./installer/setup.sh --reindexhidden-rpc-statusreports it as "Rebuilding the chain state — its stored state did
not match the block_log, so it is replaying once."Since v0.3.1: two judgement bugs found on a real node
v0.3.1 installs and runs. Watching a node come up on real hardware showed two
places where the supervisor was judging by a stopwatch instead of by evidence.1. A slow load could be mistaken for a hang. The node was considered "still
loading" for a fixed 30 minutes and "stuck" after that. On a machine with a large
chain on a slow disk, reading the state into memory can take longer — and the one
mistake that makes things genuinely worse is restarting a node halfway through
loading, because then it never finishes. The supervisor now asks the node instead:
if blurtd's log keeps moving, it is working, and it is left alone however long it
takes. It only acts when the node is running, silent, and not answering — both
conditions, past both windows (NODE_STALL, default 30 min, andGRACE_NODE, now
1 hour).hidden-rpc-statussays which it is: "its log is still moving, so it is
working — leave it alone".2. Slow probes looked like waking from suspend. Resume detection timed the
whole loop. But a tick in which both hidden-service self-dials time out
legitimately takesPROBE_TIMEOUT_TOR + PROBE_TIMEOUT_I2P= 105 seconds, which
tripped the "woke up" threshold — so an unreachable node forced a full re-check and
reset its repair ladders on every single cycle. The check now measures the sleep
only, which is immune to slow probes and still catches a real suspend. Verified: a
tick spending 8 seconds in probes with a 2-second threshold now produces zero false
wake-ups.Also in this release: the installer no longer holds your terminal for up to 30
minutes waiting for a node to finish loading. It waits 5 minutes, then hands the
prompt back and tells you to usehidden-rpc-status --watch— the self-healer is
already watching the node by then, so there is nothing to sit and guard.
If you tried v0.3.0: this is the fix
v0.3.0 could not install. It stopped at "Starting your node" on every machine,
with no useful explanation. My mistake, and an avoidable one.The installer loaded its shared repair library like this:
DOCKER_CMD="$SUDO docker" . "$SUP_SRC/heal.sh"A variable assignment written in front of the
.builtin is a temporary
environment assignment: bash throws it away when the command finishes. While it
was in effect, though, the library's own: "${DOCKER_CMD:=docker}"saw the name
as already set and skipped its default — soDOCKER_CMDended up unset, and
every container command after that ran astimeout 120 run -d …, which fails
instantly. The library's error went to/dev/null, so all you saw was "the node
didn't start on this try".Fixed in three ways, so it cannot happen again in this shape or any other:
- one
load_heal_libfunction does the loading with plain assignments, and
verifies afterwards that the helpers exist and the command wrappers are
non-empty — stopping with a clear message if not, before touching anything; - container failures now report what the container runtime actually said,
instead of a shrug; - a third test suite,
supervisor/tests/installer-lint.sh, runs the real
start_nodeout of the real installer against a fake docker and asserts the
exact command line it produces. That is the test that was missing: v0.3.0's
supervisor was tested thoroughly and its installer was never executed at all.
If v0.3.0 removed the container on your node before failing, nothing was lost —
it was stopped gracefully first and the chain data lives in the Docker volume.
Re-running the installer rebuilds the container and the node loads its state in
the usual few minutes. No replay.Everything below is the v0.3.0 release, unchanged.
A node that went down and stayed down was the whole problem. This release is
about making that impossible — and about a node that stops eating the machine it
lives on.Nothing about your addresses, your chain data or your setup changes. Re-run the
installer on each node and it picks all of this up.Why nodes were going dark and staying dark
Four separate mechanisms, each found in the v0.2.1 code rather than guessed at:
- DEGRADED was a dead end. Once a transport used up its restart budget, the
supervisor kept probing it but never repaired it again. The only way back
was a recovery that happened on its own. Any node that reached this state was
off the network until a human walked over to it. - "The node is down" meant "do nothing", forever. When the chain daemon
stopped answering, the supervisor correctly refused to restart Tor and I2P —
and then stopped there. Nothing in the system ever restarted blurtd, its
container, or Docker. - Repairs were never checked. On Linux Mint,
tor.serviceis a wrapper whose
start command is literally/bin/true, so restarting it can report success
while the real daemon stays dead. And after five quick failures systemd latches
a service into a permanent failed state that refuses every further restart until
someone runsreset-failed— which nothing ever did, including for the
supervisor itself. Self-healing could be switched off by a single bad minute. - The replay flag was pinned forever. A node set up from the block_log kept
--replay-blockchainin its container command permanently, so every reboot
re-ran a multi-hour replay. Which looks exactly like "it went down and never
came back".
What's new
- The supervisor now owns the whole node, not just the two transports: the
chain daemon, the container, the container runtime, the clock, the disk and the
power settings. A stopped node is started; a missing container is rebuilt from a
saved spec; a node whose head block has frozen is treated as wedged and
repaired; a dead Docker is restarted. - DEGRADED is now a slow retry, never a grave. It deep-repairs and tries
again on a doubling backoff (15m → 30m → 60m) for as long as the machine has
power. Restart thrash is still capped. - Every repair is verified. After a restart the supervisor checks that the
unit really is active with a live process, that the proxy port answers, and that
the address still exists — and escalates if not. It resolves the real Tor unit
at runtime instead of trusting a name guessed at install time, and runs
reset-failedbefore every restart. - Deeper repairs for the failures restarting can never fix: a torrc that lost
its hidden-service stanza (put back, verified withtor --verify-config, rolled
back if it would break Tor), an i2pd tunnel file reverted by a package upgrade
(now written totunnels.conf.d, where upgrades cannot touch it), and missing
hidden-service keys (restored from a local backup, so your.onionand
.b32.i2psurvive a purge or a bad restore). - Reboots are boring now. The replay flag is removed automatically as soon as
a node proves it is serving; a new shutdown unit gives the blockchain the full
time it needs to save itself before Docker goes away, so there are no more dirty
shutdowns and no surprise replays; and everything is enabled to come back on its
own. Move a node to another room whenever you like. - Laptops stay awake. Suspend, hibernate and the lid switch are disabled at
the systemd level, where no desktop power setting can override them. If a
machine does sleep anyway, waking up triggers an immediate full re-check. - A controlled reboot as the genuine last resort — only when every transport
is down, only after two hours of failed repairs, never during a replay, never
just after booting, at most once a day, and always after stopping the blockchain
cleanly. SetALLOW_REBOOT="0"in/etc/hidden-rpc/supervisor.confto forbid it.
A much lighter node
Full detail, with measurements and the trade-offs, in
docs/FOOTPRINT.md.- No more permanent replay. This was the single largest CPU cost in the fleet,
and it is gone. - The supervisor went from ~104,000 processes a day to roughly 7,000. Its idle
loop no longer spawns anything at all: it reads/procwith shell builtins
instead of shelling out toip,awk,cksum,dateandsedseveral times
every five seconds. Measured, 200 runs: 5.23 ms → 0.54 ms per network check. - Tor circuits cut by ~80%. Health self-dials relax from every 60s to every
5 minutes once a transport has been steady for half an hour, and snap back
instantly on any network change, wake-up or failure. - I2P stops relaying strangers' traffic. Stock i2pd accepts transit tunnels;
on a ten-year-old laptop that is the biggest ongoing CPU and bandwidth cost the
node carries, for no benefit to its own reachability. Nownotransit = true,
with the distro's always-on IRC tunnel switched off too. Your own.b32.i2p
server tunnel is unaffected. - blurtd's web thread pool 256 → 16 (Blurt's own default is 32) and fewer p2p
connections with a smaller sync prefetch. Neither needs a replay. - Optional
--headlessstops the desktop on a node nobody looks at, freeing
RAM and the GPU. Reversible with--desktop.
New commands
hidden-rpc-status # everything: node, reachability, self-repairs, host hidden-rpc-status --watch # live progress, one calm line sudo hidden-rpc-nodectl stop # stop a node safely before unplugging or moving it sudo hidden-rpc-nodectl start # and back on again ./installer/setup.sh --doctor # check and repair everything, right nowhidden-rpc-nodectl stopalso turns maintenance mode on, so the supervisor leaves
the node alone while you work;startturns it back off. A node can never be left
quietly un-supervised.hidden-rpc-transport-statusstill works and shows the
transports-only view.Upgrading
Re-run the installer on each node (safe anytime, never harms a node):
bash installer/setup.shOn a node that is currently pinned to the replay flag, this removes it with one
short restart — after which reboots take minutes instead of hours. Your
/etc/hidden-rpc/supervisor.confis kept as you have it; only the new settings
this version adds are appended.To fix a node without a full installer run:
setup.sh --doctor.Minimum requirements
- A standard 64-bit PC (amd64), Linux Mint or Ubuntu, Docker-capable.
- About 90 GB free for a first-time trustless setup.
- Wired Ethernet is strongly recommended. In this fleet, Wi-Fi connects
successfully around 50% of the time versus 100% for Ethernet.
Testing
Both suites run anywhere, with no Tor, I2P or Docker needed:
bash supervisor/tests/fsm-sim.sh # state machine + policy bash supervisor/tests/heal-sim.sh # the repairs themselves bash supervisor/tests/installer-lint.sh # the installer, actually executedheal-sim.shdrives the repair code against fakesystemctlanddocker
commands that reproduce the real traps: the/bin/trueTor wrapper, systemd's
start limiter, a pinned replay flag, andunless-stopped. The hardware scenarios
are indocs/SELF-HEALING.md.- Repository: https://git.agorise.net/agorise/hidden-rpc
- License: AGPL-3.0-or-later
Downloads
-
Source code (ZIP)
0 downloads
-
Source code (TAR.GZ)
0 downloads