Fix a problem

Search this page for the exact message you see. Each entry has the cause, the fix, and where the problem came up:

• Seen on a mainnet run means it happened during a full mainnet run of the 3 steps on 18 Oct 2026. The run log has the details: run-log.md
• Found in review means a tester found it while checking the guide against a live node or the code.
• Reproduced in testing means it was caused on purpose to record the exact message.

Step 1: rent a server

Hetzner refuses the volume

You see: volumes size limit exceeded (resource_limit_exceeded)

Why: New Hetzner accounts can rent at most 1024 GB of volumes in total. Asking for 2000 GB gives this error.

Fix: Ask for 1000 GB, as the Full run does. Your limits are at console.hetzner.cloud/limits.

VOLUME_GB=1000

Where: Seen on a mainnet run, when step 1 asked for 2000 GB. Nothing was created, so it cost nothing.

The project already has a server with that name

You see: this project already has: server snapchain-read-node, with Nothing was created or changed.

Why: A server named $NAME, the volume $NAME-data, or a firewall $NAME-fw that is applied to a server already exists in this Hetzner project. It may be someone else’s node, so provision.sh stops before it changes anything.

Fix: Use an empty Hetzner project. For a second node in the same project, pick another name and keep it for every later step and for teardown:

export NAME=snapchain-read-node-2

If it is your own earlier run, REUSE=1 ./provision.sh continues with it.

Where: Found in review by a first-time tester in a shared project; the check was added to provision.sh afterwards. Covered by unit tests, not yet hit on a live run.

The server type isn’t available to rent

You see: server type cax31: not available in fsn1 right now. when you set a type, or none of [cx43 cax31 cpx42 ccx23] is usable in fsn1. with SERVER_TYPE="". Both list the types you can rent there, cheapest first.

Why: Hetzner can’t rent you that type in that location right now. Which types it can rent changes over time.

Fix: Run step 1 again with a type from the list, or with another location. The guide has been tested on cax31 only; times on other types vary.

SERVER_TYPE=cpx42      # or another type from the error
LOCATION=nbg1          # fsn1, nbg1, hel1, ash, hil or sin

A typo in LOCATION gives unknown LOCATION=…, followed by the list of valid names.

Where: Seen on a mainnet run, for cx43 and later for cax31.

SSH stops working

You see: SSH to the server hangs, then times out. It worked before.

Why: The firewall lets SSH in only from the address in SSH_SOURCE_CIDRS, and your public address changed. Home connections and some cloud machines do this.

Fix: Point the SSH rule of the firewall $NAME-fw at your new address. This changes only that rule; the server keeps running.

MY_IP=$(curl -4 -s https://ifconfig.me) && echo "$MY_IP"
hcloud firewall describe "$NAME-fw" -o json \
  | jq --arg ip "$MY_IP/32" '[.rules[] | if .protocol == "tcp" and .port == "22" then .source_ips = [$ip] else . end]' \
  | hcloud firewall replace-rules --rules-file - "$NAME-fw"

It prints Firewall Rules for Firewall <id> updated. Then connect again. If a watchdog runs, its progress check also uses SSH: until the rule is fixed it logs a warning and deletes nothing.

Where: Seen on a mainnet run: the machine running the steps kept getting new public IPs. Opening SSH to every address, with key-only login, is the other way out.

$IP is empty

You see: ssh: Could not resolve hostname : Name or service not known on Linux, with nothing after hostname.

Why: $IP lives only in the terminal where step 1 ran.

Fix: Set the name from step 1 again, then get the IP:

export NAME=snapchain-read-node
IP=$(hcloud server ip "$NAME")

Where: Found in review, while a tester checked the steps. Reproduced in testing for the exact message.

The watchdog deletes the node during the sync

This only applies if you left out WATCHDOG=0 in step 1.

You see: The server is gone. .provision-state/watchdog.log gives the reason, for example wall-clock cap 86400s reached.

Why: The watchdog deletes everything 24 hours after step 1, even if the node is healthy. Our cax31 sync took 14 h 06 m. An earlier version would also have deleted it 2 hours into the sync when it had no SSH key file, because it could not see any progress.

Fix: Stop the watchdog to keep the node. Nothing is deleted.

touch .provision-state/watchdog.disarm

The current watchdog turns its progress check off when it has no SSH key file.

Where: Found in review, by reading the code.

The watchdog deletes the node to stay under MAX_USD

This only applies if you left out WATCHDOG=0 in step 1.

You see: The server is gone. .provision-state/watchdog.log has a line TEARDOWN triggered: budget: projected <amount> >= MAX_USD <limit> after <seconds>s at <rate>/h.

Why: Every minute the watchdog multiplies the live hourly price of the server, its IPv4 address and the volume by the hours that have started, each counted in full, plus the next check and 15 minutes for the teardown. When that total reaches MAX_USD, in your Hetzner account’s currency, it runs teardown.sh. That deletes the server, the volume and the firewall. The node’s database was on the volume, so it is gone too, and a new node loads the snapshot from the start.

The watchdog runs on your computer, not on the server. While your computer sleeps, it checks nothing, and Hetzner keeps billing. It measures time by the clock, so after you wake the computer it counts the hours you slept, and it can delete everything at its next check, within 60 seconds. The 24-hour limit works the same way.

Fix: Run the 3 steps again. To avoid it next time:

• Set MAX_USD and EXPECTED_HOURS for the whole time you want the node. provision.sh refuses to start if the price for EXPECTED_HOURS would pass MAX_USD, so raise both together.
• The watchdog also deletes everything 24 hours after step 1. For longer, set WATCHDOG_MAX_WALL_SECS in step 1 (172800 is 48 hours) and raise MAX_USD to match.
• To keep the node with no limit, use WATCHDOG=0 in step 1. To stop a running watchdog without deleting anything, create its disarm file. It reads the file at its next check, deletes it and exits:

touch .provision-state/watchdog.disarm

Where: Found in review, by reading watchdog.sh.

The watchdog never started

You see: watchdog did not start (see …/watchdog.out). Nothing was created. The file says HCLOUD_TOKEN not set. NOT RUNNING: no spending cap is active.

Why: A new terminal does not have your Hetzner token.

Fix: Set the token in this terminal, then run step 1 again.

printf 'Hetzner token: '; stty -echo; read -r HCLOUD_TOKEN; stty echo; echo; export HCLOUD_TOKEN

Where: Found in review. The tests now check this message.

The watchdog would delete everything at its first check

This only applies if you left out WATCHDOG=0 in step 1.

You see: WATCHDOG_T0 must be a Unix time in seconds (date +%s), got '<value>', or the watchdog would delete everything at its first check (WATCHDOG_T0=<epoch> is too far back for MAX_WALL_SECS=86400 / MAX_USD=8). Nothing was created. Unset WATCHDOG_T0 or raise the caps.

Why: WATCHDOG_T0 tells the watchdog when the run started, in seconds since 1970, as date +%s prints it. It is only for restarting a watchdog partway through a run. If it is not a whole number, provision.sh stops. If it is so far back that the 24-hour cap (MAX_WALL_SECS, 86400 seconds) or the MAX_USD budget is already reached, the watchdog would run teardown.sh at its first check while step 1 is still creating the server, so provision.sh stops first.

Fix: Unset it, then run step 1 again:

unset WATCHDOG_T0

To restart a watchdog for a run that is still going, set WATCHDOG_T0 to the date +%s value from when step 1 ran. If that run is close to a limit, raise WATCHDOG_MAX_WALL_SECS or MAX_USD.

Where: Found in testing with a fake hcloud; the check was added to provision.sh afterwards. Covered by unit tests, not yet hit on a live run.

MAX_USD is below the watchdog’s fallback rate

This only applies if you left out WATCHDOG=0 in step 1.

You see: MAX_USD=<amount> must be above the watchdog's FALLBACK_RATE (0.35/h, used until the server exists), or it deletes everything at its first check. Nothing was created.

Why: Until the server exists, the watchdog cannot read its live price, so it uses FALLBACK_RATE: 0.35 an hour by default, in your Hetzner account’s currency. It counts the first started hour in full, so with MAX_USD at or below that rate it would delete everything at its first check. provision.sh stops before it creates anything.

Fix: Raise MAX_USD together with EXPECTED_HOURS, then run step 1 again. The defaults, MAX_USD=8 EXPECTED_HOURS=24, pass for cax31 with the 1000 GB volume:

IP=$(SSH_KEY=<key-name> SERVER_TYPE="" VOLUME_GB=1000 MAX_USD=8 EXPECTED_HOURS=24 SSH_SOURCE_CIDRS="$MY_IP/32" ./provision.sh)

Where: Found in testing: the real watchdog.sh with MAX_USD=0.3 and no server yet deleted everything after 0 seconds. The check was added to provision.sh afterwards. Covered by unit tests, not yet hit on a live run.

Step 2: start the node

The disk fills while the snapshot loads

You see: The data disk fills up hours into step 2. If /root/snapchain-DISK-FULL exists, disk-guard.sh stopped the node at 90% full.

Why: The official restore keeps every downloaded chunk, an unpacked copy of each shard and the database until the end. The snapshot our run downloaded measured 485 to 505 GB, and that does not fit on a 1000 GB volume.

Fix: On disks under 1500 GB, node.sh installs two helpers. snapshot-space-saver.sh deletes chunks the node has already read. disk-guard.sh stops the node before the disk is full. Check that they run:

ssh root@$IP systemctl status snapchain-space-saver snapchain-disk-guard.timer

Where: Seen on a mainnet run, during the snapshot restore. The helpers were installed by hand before the disk filled, and the disk peaked at 699 GB (75%). The automatic install in node.sh has unit tests but has not run during a real sync yet.

The node crashes on a 404 while it loads the snapshot

You see: In the node log, Failed to download snapshot chunk .rocks.snapshot/shard-0/chunk_0001.bin: HTTP status client error (404 Not Found), then a panic. From node.sh: Snapchain could not download the shard 0 snapshot. The shard number can differ.

Why: Snapchain reads the snapshot list of every shard once, at the start. It then loads shard 1, shard 2 and shard 0, one after another. Shard 0’s snapshot is replaced once a day and the old files are deleted. If that happens before the node reaches shard 0, every download fails with 404. The node tries each chunk 5 times, 10 seconds apart, then crashes. Docker restarts it, but the database folder is no longer empty, so it skips the snapshot and runs without shard 0.

Fix: node.sh does this for you while it waits for sync. It restores only the failed shard from the current snapshot and leaves the other shards alone. To run it yourself:

ssh root@$IP 'bash -s repair-shard 0' < node.sh

It downloads the chunks 4 at a time, starts over with the new snapshot list if it gets a 404, and gives up after 3 tries. It needs 200 MiB free for each chunk in the list. It stops the node only to swap in the restored shard. SNAPSHOT_REPAIR=0 turns the automatic repair off.

Where: Reported by a tester on a mainnet sync. The repair is tested offline against a fake bucket and a fake Docker. It has not run on a real node yet.

Upstream bug: snapchain #1044. A tester fixed shard 0 by hand on mainnet with the same steps this command runs. Read his report.

The API does not answer for hours

You see: HTTP API not answering yet (snapshot download in progress?).

Why: The node serves no API while it loads the snapshot. On our cax31 run that took 8 h 34 m.

Fix: Wait. The used space on the server should keep growing:

ssh root@$IP df -h /root/snapchain

Where: Seen on a mainnet run, for the whole snapshot restore.

node.sh stops waiting before the node is in sync

You see: not in sync after 43200s. Re-run './node.sh wait' later; it is safe to re-run.

Why: node.sh waits for a set time. It used to be 12 hours, shorter than our cax31 sync (14 h 06 m). It is 24 hours now, so the message says 86400s.

Fix: Wait again. The node kept syncing the whole time.

ssh root@$IP 'bash -s wait' < node.sh

Where: Seen on a mainnet run, with the old 12-hour wait.

Your computer sleeps or SSH drops

You see: The terminal running step 2 stops, or SSH disconnects.

Why: The node runs on the server, not on your computer.

Fix: Reconnect and wait again:

export NAME=snapchain-read-node   # the name from step 1
IP=$(hcloud server ip "$NAME")
ssh root@$IP 'bash -s wait' < node.sh

Where: Seen on a mainnet run. The computer running the steps paused at least 13 times during the sync, and the node kept syncing each time.

Warnings about peers in the node log

You see: InsufficientPeers ("read-node-peers") warnings, 540 in 30 minutes on our run. HandshakeTimedOut to two bootstrap peers every 30 seconds.

Why: None of the connected peers listen on the read-node-peers topic. The two bootstrap peers do not answer.

Fix: Nothing, while blockDelay stays in single digits. If blockDelay keeps growing, the validators may have stopped making blocks. The Snapchain docs say read nodes log this warning nonstop during such a halt (node-types).

ssh root@$IP 'curl -s 127.0.0.1:3381/v1/info' | jq '.shardInfos[] | {shardId, blockDelay}'

Where: Seen on a mainnet run, for the whole sync.

Step 3: open the app

Node.js is too old

You see: [deploy-client] ERROR: Node.js 22.12+ (22.x), 24.x or 26+ is required (found: v20.19.2).

Why: The app’s build tools need Node.js 22.12 or later.

Fix: Install Node.js 22.12+ or 24: nvm install 24 && nvm use 24 (nvm install docs), or download it from nodejs.org. Node.js 20.19 can run npm run dev, but not the tests that deploy-client.sh runs. Then check:

node --version

Where: Reproduced in testing with Node.js 20.19.2. Found in review first, when the site build failed on Node.js 20.

SSH_OPTS_ARR[@]: unbound variable on a Mac

You see: ./deploy-client.sh: line 28: SSH_OPTS_ARR[@]: unbound variable, right after [deploy-client] uploading to root@<ip>:/opt/snapchain-client.

Why: A Mac runs the script with bash 3.2. If you don’t set SSH_OPTS, the script has an empty list of SSH options. Bash 3.2 treats an empty list as a missing variable and stops.

Fix: Get the fixed script with git pull, then run step 3 again. Running it twice is safe. On an older copy, set any SSH option to get past it:

SSH_OPTS="-o BatchMode=yes" ./deploy-client.sh "$IP"

Where: Reported by a tester on a Mac. Reproduced in testing under bash 3.2.57, built from the GNU source, offline. The fix is tested there too, along with the other scripts that run on your computer.

Docker is not on the server yet

You see: bash: line 1: docker: command not found after [deploy-client] uploading to root@<ip>:/opt/snapchain-client, or another error from docker compose.

Why: deploy-client.sh starts Caddy on the server with docker compose. A new server has no Docker until step 2 installs it, so step 3 fails if it runs before step 2 gets that far.

Fix: Wait for step 2 to install Docker. Its output shows [node.sh] Installing Docker Engine from download.docker.com ..., and Docker is ready once [node.sh] Downloading official bootstrap: follows. To check from another terminal:

ssh root@$IP docker compose version

When that prints a version, run step 3 again. Running it twice is safe.

Where: Found in review.

Port 5317 is already in use

You see: Error: Port 5317 is already in use when you run npm run dev for the local copy of the app.

Why: The local copy always uses port 5317, which keeps its saved app key apart from other apps on your computer. If the port is busy, it stops instead of moving.

Fix: Find the program that uses the port, stop it, then start the app again:

lsof -i :5317

Where: Reproduced in testing. Found in review first: the app used to run on port 5173, the default for Vite projects.

Port 3381 does not answer from outside

You see: curl http://<server-ip>:3381/v1/info hangs, then times out.

Why: provision.sh keeps the node’s own ports 3381 and 3383 closed to the internet. An earlier version left them open, so anyone could read the node and send it messages.

Fix: Ask through the app’s address, or open an SSH tunnel:

curl -s https://<name>/v1/info
# or
ssh -N -L 3381:127.0.0.1:3381 root@$IP     # then, in another terminal:
curl -s 127.0.0.1:3381/v1/info

Where: Found in review. The first review reached port 3381 from outside. The second confirmed it now times out.

Adding an app key

No FID for your recovery phrase

You see: In the app: This recovery phrase doesn't own a Farcaster account that your node knows. The underlying error is no FID is registered to custody address 0x….

Why: The app turns your recovery phrase into one Ethereum address, the first account at the path m/44'/60'/0'/0/0, and asks your node which FID that address owns. Your node knows of none. The usual causes:

• A word is wrong. A different phrase gives a different address.
• The address that owns your FID is not that first account, for example a second account your wallet made from the same phrase, or an address the FID was moved to later. The app does not try other accounts.
• Your node has not synced. It finds FIDs in its own copy of the chain, so it cannot see an FID registered after the last block it has.

Fix: Check the words, then check that every shard shows a blockDelay in single digits, and try again:

ssh root@$IP 'curl -s 127.0.0.1:3381/v1/info' | jq '.shardInfos[] | {shardId, blockDelay}'

If your FID belongs to another account, the app has no way to use it today. Never paste the phrase into an AI assistant or a chat to debug this.

Where: Found in review, by reading client/src/signer.ts.

Reading from the node

reverse=1 returns 400

You see: provided string was not true or false

Why: The node accepts only true or false.

Fix: Use reverse=true.

Where: Found in review, on a live node.

A repeated parameter returns 400

You see: Multiple values for one key: "requesterFids"

Why: The node wants list parameters with brackets.

Fix: Use requesterFids[]=1&requesterFids[]=2.

Where: Found in review, on a live node.

mempoolSize shows 4294967295

You see: "mempoolSize": 4294967295 for shards 1 and 2 in /v1/info.

Why: A read node has no mempool. The number is a placeholder.

Fix: Nothing.

Where: Seen on a live node.

Ask an assistant

Paste your exact command and its output, after removing tokens, IPs and hostnames:

Copy debugging prompt

Show prompt

Context: I am following the RunFarcaster guide (3 commands: a Hetzner server running a Snapchain read node, and a web app that reads and posts through it).
Repo: https://github.com/sudharshanx/runfarcaster (MIT). I cloned it to my computer (git clone https://github.com/sudharshanx/runfarcaster, then cd runfarcaster) and run every command from that folder. Scripts: provision.sh (step 1), node.sh (step 2), deploy-client.sh (step 3), teardown.sh, and watchdog.sh (optional).
Server name: export NAME=snapchain-read-node (or another name I pick), in every terminal; all steps and teardown.sh use it. In any new terminal, get the server IP again with: IP=$(hcloud server ip "$NAME")

I am on step <1 rent a server | 2 start the node | 3 open the app | teardown>.
The exact command I ran:
<paste command>

What it printed (tokens, IPs and hostnames removed):
<paste output>

What the guide said I should see: <expected output>

Tell me the most likely cause, based on the repo scripts and the guide's Fix a problem page, and give me one next command to run. Stay with what the output shows.

Rules for you:
- Never ask for my recovery phrase (also called a seed phrase) or for any private key, and refuse one if I paste it. Tell me to stop, and to treat that account as exposed.
- Never print, log or echo HCLOUD_TOKEN or any other token. Read it from the environment only.
- Ask me before you run teardown.sh, delete anything, or change MAX_USD.
- Show me real output only. If a command fails, show the exact error and stop.