Hello everyone. I happen to have “fast catchup” not working on my mainnet node. The device is a Raspberry PI 4 Model B 4GB ARM-Cortex-A72 4X 1.50 GHz, 4 GB RAM with 32-bit Raspberry OS on 16 GB microSD (the node is located in the home folder of the device). When it gets to 1000 Catchpoint downloaded blocks: pi@raspberrypi:~/node $ ./goal node status -d data/ -w 1000 Last committed block: 361 Sync Time: 6564.2s Catchpoint: 11020000#KT5CVUYN44C2N7RQR4GRFWSRM7QEWGL3U45CTC53J3XCHBBQT3GA Catchpoint total accounts: 7542015 Catchpoint accounts processed: 7542015 Catchpoint accounts verified: 7542015 Catchpoint total blocks: 1000 Catchpoint downloaded blocks: 15 Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8= give up all work and started again with a normal non-fast sync: Time since last block: 0.1s Last committed block: 681 Time since last block: 0.5s Last committed block: 706 Time since last block: 0.3s Last committed block: 706 Time since last block: 1.3s Sync Time: 35.5s Last consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Next consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Round for next consensus protocol: 707 I have this setting Catchup on my config.json: “CatchpointFileHistoryLength”: 365, “CatchpointInterval”: 10000, “CatchpointTracking”: 0, “CatchupBlockDownloadRetryAttempts”: 1000, “CatchupFailurePeerRefreshRate”: 10, “CatchupGossipBlockFetchTimeoutSec”: 4, “CatchupHTTPBlockFetchTimeoutSec”: 4, “CatchupLedgerDownloadRetryAttempts”: 50, “CatchupParallelBlocks”: 16, I don’t understand what the problem is, whether it is due to my network connection, number of processors or something else.

When I was discussing this with him on Discord, it appeared it hit a hardcoded ‘give up’ time duration. His machine has a slow connection and its a slower device, but even if it takes hours for the fast catchup that’s nothing compared to the time it would likely take his device to catch up from scratch (and pruning the whole way). Assuming there is still ‘activity’ it should never reach a stop-dead total elapsed time wall, throw away all of its work and start over not using fast catchup.

@aojjazz, I think that your intuition is correct here. It does make sense that as long as the node is capable of moving forward, it should keep doing so. The number of failures is troubling regardless.

I tried “fast catchup” on testnet always with the same device and without changing the .json and it works perfectly and also very fast: pi@raspberrypi:~/node $ ./goal node status -d testnetdata/ -w 1000 Last committed block: 429 Sync Time: 694.8s Catchpoint: 11320000#5HSFH424JSO7FSQCHDOF4SL7WACEMEZ6JGJ7VKDFVC4SYXA4PQIA Catchpoint total accounts: 266145 Catchpoint accounts processed: 266145 Catchpoint accounts verified: 266145 Catchpoint total blocks: 1000 Catchpoint downloaded blocks: 1000 Genesis ID: testnet-v1.0 Genesis hash: SGO1GKSzyE7IEPItTxCByw9x8FmnrCDexi9/cOUJOiI= So the problem is on the mainnet … I try to change the CatchupBlockDownloadRetryAttempts on the mainnet to 20000 and let’s see what happens.

Here is the proof with CatchupBlockDownloadRetryAttempts set to 20000 on the mainnet: pi@raspberrypi:~/node $ ./goal node status -d data/ -w 1000 Last committed block: 4058 Sync Time: 7118.6s Catchpoint: 11070000#Z5GVD2WLQ2MIG37UJPGPMA2JG7EWAFEDJKHBXAAOWP63ABV553TA Catchpoint total accounts: 7593802 Catchpoint accounts processed: 7593802 Catchpoint accounts verified: 0 Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8= it got stuck like that and meanwhile the sync time continued: pi@raspberrypi:~/node $ ./goal node status -d data/ -w 1000 Last committed block: 4058 Sync Time: 7702.6s Catchpoint: 11070000#Z5GVD2WLQ2MIG37UJPGPMA2JG7EWAFEDJKHBXAAOWP63ABV553TA Catchpoint total accounts: 7593802 Catchpoint accounts processed: 7593802 Catchpoint accounts verified: 0 Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

UPDATE: is running the “Catchpoint accounts verified”: pi@raspberrypi:~/node $ ./goal node status -d data/ -w 1000 Last committed block: 4058 Sync Time: 8994.8s Catchpoint: 11070000#Z5GVD2WLQ2MIG37UJPGPMA2JG7EWAFEDJKHBXAAOWP63ABV553TA Catchpoint total accounts: 7593802 Catchpoint accounts processed: 7593802 Catchpoint accounts verified: 7323648 Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8= it remained so. It does not go further, it is blocked, and in the meantime the sync time advances: pi@raspberrypi:~/node $ ./goal node status -d data/ -w 1000 Last committed block: 4058 Sync Time: 9580.2s Catchpoint: 11070000#Z5GVD2WLQ2MIG37UJPGPMA2JG7EWAFEDJKHBXAAOWP63ABV553TA Catchpoint total accounts: 7593802 Catchpoint accounts processed: 7593802 Catchpoint accounts verified: 7323648 Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

NEW UPDATE: “Catchpoint accounts verified” concluded: pi@raspberrypi:~/node $ ./goal node status -d data/ -w 1000 Last committed block: 4058 Sync Time: 9802.4s Catchpoint: 11070000#Z5GVD2WLQ2MIG37UJPGPMA2JG7EWAFEDJKHBXAAOWP63ABV553TA Catchpoint total accounts: 7593802 Catchpoint accounts processed: 7593802 Catchpoint accounts verified: 7593802 Catchpoint total blocks: 1000 Catchpoint downloaded blocks: 3 Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8= performed the “Catchpoint downloaded blocks”. Arriving at “Catchpoint downloaded blocks” = 500 approximately, everything fails: Last committed block: 6176 Time since last block: 0.0s Sync Time: 170.8s Last consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Next consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Round for next consensus protocol: 6177 Next consensus protocol supported: true Last Catchpoint: Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=

Nothing to do unfortunately …

I’ll try that tomorrow and see what the outcomes when it runs on my side. The one active thing that can be done here is to identify the relays that have failed to provide the needed blocks. I can’t tell if the issues that you’ve been seeing are a result of a networking or something bigger than that ( my guess is that it’s unrelated to your network ); I’ll keep you posted tomorrow with my findings. One note is that I won’t be able to test it on RaspbPI ( since I don’t have one ). If it’s a RaspPI specific issue, it can take longer to confirm and resolve.

I’m not sure if the telemetry is sufficiently detailed to provide constructive information around that. Your node.log file contains details regarding the catchup failuire. If you could grep the catchpoint related entries from there, it would be great.

Fast catchup does not work on Raspberry Pi 4B

General

salvatorecorvaglia December 27, 2020, 8:52am 14

Is it possible that the developers of Algorand can see what happened if I turned telemetry of my node?

Topic		Replies	Views
Fast Catchup not working for our node on MainNet Uncategorized	53	5583	October 7, 2020
Can't complete catchup operation on mainnet -- timeout after 45 minutes (~4m accounts) General	13	1499	November 10, 2020
Node continues normally despite fast catchup General	5	568	January 15, 2021
Catchpoint gets stuck, fails, and restarts on pi4 4GB Uncategorized	2	502	September 26, 2022
Mainnet Fast Catchup Stalling General fast-catchup	6	535	February 7, 2022

Fast catchup does not work on Raspberry Pi 4B

Related topics