Hi,
I am thinking to run an indexer node myself which requires an archival node with all ledger data, but I am not sure if I have enough disk space to store the whole ledger.
If anyone is running an archival node with the full ledger data, could you tell me how big the current mainnet & testnet full data is? 300G, 500G, etc?
Thank you in advance!
On 2021-05-02:
EDIT: For up-to-date numbers, please use [topic not in archive]
Thank you very much for the information!
Another question for the indexer data, I have read some info about the indexer DB here:
The Indexer DB takes up a fraction of disk space compared to the actual blockchain data with archival mode on. For example, 100 GB of blockchain data takes about 1 GB of data in the Indexer DB.
So I thought the indexer data will take much less space than the ledger data, but from your info, seems they are pretty close. Any reason that caused the indexer’s data size so big? Thank you
The indexer indexes transactions rather than blocks. Therefore, empty blocks can still consume quite some space on the blockchain, whereas they wouldn’t consume any space on the indexer.
On the flip side, the indexer have multiple indices meant for the quick retrieval and analysis of transactions. These could exceed the size of the transaction itself, hence, on a high-transactional-volume blockchain, the indexer would become larger than the blockchain.
got it, that makes sense, thank you!
Hello,
What is the current size of full chain (testnet and mainnet) of Algorand?
I will set up a node to use all data and wonder about the required disk space to process full chain data.
By the way, the link provided in this forum topic doesn’t work: Howbigisalgorand.com
Thanks!
Today 2022-04-04:
How long does it take to get to 1.1TB? My archival node has been running for almost a week and yet the disk usage is about 40GB.
It seems the disk usage increases about 1~2GB per day. My node has 1Gbps network bandwidth.
I think it should take around one week with a 1Gbps network. 1-2GB per day seems way too slow.
Do you have enough RAM? Is your disk a not-too-slow SSD?
HDD and SD cards are now way too slow.
My node is running on AWS. It has NVMe SSD, 32GB RAM, and 8 vCPUs.
The system load is super low. Is there something I should check?
That’s very strange.
Can you run and indicate the output of the following commands:
goal node status
goal version -v
du -hs $ALGORAND_DATA/*
cat $ALGORAND_DATA/config.json
Hi Fabrice,
I used ‘catchup’ for fast sync. It looks like it’s not supported for an archival or relay node. Perhaps, this limitation can be documented in the install guide.
Anyhow, since then, I’ve restarted my node and can see upticks in my disk usage.
My node has been synced to block # 8,368,820 and consumed about 158GB of disk space in the past 14 hours. The system load, disk i/o, cpu, etc seems to be still very light.
Here’re outputs:
$ goal node status
Last committed block: 8368820
Time since last block: 0.1s
Sync Time: 52611.9s
Last consensus protocol: GitHub - algorandfoundation/specs at e5f565421d720c6f75cdd186f7098495caf9101f
Next consensus protocol: GitHub - algorandfoundation/specs at e5f565421d720c6f75cdd186f7098495caf9101f
Round for next consensus protocol: 8368821
Next consensus protocol supported: true
Last Catchpoint:
Genesis ID: mainnet-v1.0
Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=
$ goal version -v
Version: [v1 v2]
GenesisID: mainnet-v1.0
Build: 3.5.1.stable [rel/stable] (commit #aa2fb0ee)
$ sudo du -hs /var/lib/algorand/
158G /var/lib/algorand/
$ cat /var/lib/algorand/config.json
{
“Version”: 21,
“AccountUpdatesStatsInterval”: 5000000000,
“AccountsRebuildSynchronousMode”: 1,
“AgreementIncomingBundlesQueueLength”: 7,
“AgreementIncomingProposalsQueueLength”: 25,
“AgreementIncomingVotesQueueLength”: 10000,
“AnnounceParticipationKey”: true,
“Archival”: true,
“BaseLoggerDebugLevel”: 4,
“BlockServiceCustomFallbackEndpoints”: “”,
“BroadcastConnectionsLimit”: -1,
“CadaverSizeTarget”: 1073741824,
“CatchpointFileHistoryLength”: 365,
“CatchpointInterval”: 10000,
“CatchpointTracking”: 0,
“CatchupBlockDownloadRetryAttempts”: 1000,
“CatchupBlockValidateMode”: 0,
“CatchupFailurePeerRefreshRate”: 10,
“CatchupGossipBlockFetchTimeoutSec”: 4,
“CatchupHTTPBlockFetchTimeoutSec”: 4,
“CatchupLedgerDownloadRetryAttempts”: 50,
“CatchupParallelBlocks”: 16,
“ConnectionsRateLimitingCount”: 60,
“ConnectionsRateLimitingWindowSeconds”: 1,
“DNSBootstrapID”: “.algorand.network”,
“DNSSecurityFlags”: 0,
“DeadlockDetection”: 0,
“DeadlockDetectionThreshold”: 30,
“DisableLocalhostConnectionRateLimit”: true,
“DisableNetworking”: false,
“DisableOutgoingConnectionThrottling”: false,
“EnableAccountUpdatesStats”: false,
“EnableAgreementReporting”: false,
“EnableAgreementTimeMetrics”: false,
“EnableAssembleStats”: false,
“EnableBlockService”: false,
“EnableBlockServiceFallbackToArchiver”: true,
“EnableCatchupFromArchiveServers”: false,
“EnableDeveloperAPI”: false,
“EnableGossipBlockService”: true,
“EnableIncomingMessageFilter”: false,
“EnableLedgerService”: false,
“EnableMetricReporting”: false,
“EnableOutgoingNetworkMessageFiltering”: true,
“EnablePingHandler”: true,
“EnableProcessBlockStats”: false,
“EnableProfiler”: false,
“EnableRequestLogger”: false,
“EnableTopAccountsReporting”: false,
“EnableVerbosedTransactionSyncLogging”: false,
“EndpointAddress”: “127.0.0.1:0”,
“FallbackDNSResolverAddress”: “”,
“ForceFetchTransactions”: false,
“ForceRelayMessages”: false,
“GossipFanout”: 4,
“IncomingConnectionsLimit”: 800,
“IncomingMessageFilterBucketCount”: 5,
“IncomingMessageFilterBucketSize”: 512,
“IsIndexerActive”: false,
“LedgerSynchronousMode”: 2,
“LogArchiveMaxAge”: “”,
“LogArchiveName”: “node.archive.log”,
“LogSizeLimit”: 1073741824,
“MaxAPIResourcesPerAccount”: 100000,
“MaxCatchpointDownloadDuration”: 7200000000000,
“MaxConnectionsPerIP”: 30,
“MinCatchpointFileDownloadBytesPerSecond”: 20480,
“NetAddress”: “:4160”,
“NetworkMessageTraceServer”: “”,
“NetworkProtocolVersion”: “”,
“NodeExporterListenAddress”: “:9100”,
“NodeExporterPath”: “./node_exporter”,
“OptimizeAccountsDatabaseOnStartup”: false,
“OutgoingMessageFilterBucketCount”: 3,
“OutgoingMessageFilterBucketSize”: 128,
“ParticipationKeysRefreshInterval”: 60000000000,
“PeerConnectionsUpdateInterval”: 3600,
“PeerPingPeriodSeconds”: 0,
“PriorityPeers”: {},
“ProposalAssemblyTime”: 250000000,
“PublicAddress”: “”,
“ReconnectTime”: 60000000000,
“ReservedFDs”: 256,
“RestConnectionsHardLimit”: 2048,
“RestConnectionsSoftLimit”: 1024,
“RestReadTimeoutSeconds”: 15,
“RestWriteTimeoutSeconds”: 120,
“RunHosted”: false,
“SuggestedFeeBlockHistory”: 3,
“SuggestedFeeSlidingWindowSize”: 50,
“TLSCertFile”: “”,
“TLSKeyFile”: “”,
“TelemetryToLog”: true,
“TransactionSyncDataExchangeRate”: 0,
“TransactionSyncSignificantMessageThreshold”: 0,
“TxPoolExponentialIncreaseFactor”: 2,
“TxPoolSize”: 15000,
“TxSyncIntervalSeconds”: 60,
“TxSyncServeResponseSize”: 1000000,
“TxSyncTimeoutSeconds”: 30,
“UseXForwardedForAddressField”: “”,
“VerifiedTranscationsCacheSize”: 30000
}
$ iostat
Linux 5.13.0-1019-aws 04/05/22 x86_64 (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
6.75 0.00 0.54 0.26 0.44 92.01
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.00 0.02 0.00 0.00 9316 0 0
loop1 0.00 0.00 0.00 0.00 504 0 0
loop10 0.00 0.00 0.00 0.00 14 0 0
loop2 0.00 0.00 0.00 0.00 1259 0 0
loop3 0.00 0.03 0.00 0.00 15494 0 0
loop4 0.00 0.00 0.00 0.00 2383 0 0
loop5 0.00 0.04 0.00 0.00 25016 0 0
loop6 0.00 0.00 0.00 0.00 519 0 0
loop7 0.00 0.00 0.00 0.00 2434 0 0
loop8 0.00 0.02 0.00 0.00 10415 0 0
loop9 0.00 0.00 0.00 0.00 1249 0 0
nvme0n1 297.09 31.00 3833.87 0.00 18780402 2322982024 0
This is the issue indeed.
Just made a PR.
Small comment about config.json, in general I prefer to just put the modified fields in the config.json so that if default values changes, it automatically get them:
{
“Archival”: true
}
Also, don’t hesitate to use ``` for any output to improve formatting.
Hi fabrice,
I started the mainnet data of Algorand to be loaded on a VM disk. In 2 days it became 165GB. But after 13 days more it can only reach 318 GB. How can I make this process faster? The indexer also works in parallel.
Is there a variable that should be set to a different value in config.json file to make this process faster?
The output of the commands you’ve sent here are like that:
$ goal node status
Last committed block: 13401207
Time since last block: 2.9s
Sync Time: 1322034.2s
Last consensus protocol: GitHub - algorandfoundation/specs at 3a83c4c743f8b17adfd73944b4319c25722a6782
Next consensus protocol: GitHub - algorandfoundation/specs at ac2255d586c4474d4ebcf3809acccb59b7ef34ff
Round for next consensus protocol: 13409876
Next consensus protocol supported: true
Last Catchpoint:
Genesis ID: mainnet-v1.0
Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=
$ goal version -v
Version: [v1 v2]
GenesisID: mainnet-v1.0
Build: 3.5.1.stable [rel/stable] (commit #aa2fb0ee)
ubuntu@algorand-node:~$ du -hs $ALGORAND_DATA/*
577M /mnt/algorand/agreement.cdv
1.1G /mnt/algorand/agreement.cdv.archive
4.0K /mnt/algorand/algod.admin.token
0 /mnt/algorand/algod.lock
4.0K /mnt/algorand/algod.net
4.0K /mnt/algorand/algod.pid
4.0K /mnt/algorand/algod.token
4.0K /mnt/algorand/config.json
28K /mnt/algorand/genesis.json
4.0K /mnt/algorand/goal.cache
4.0K /mnt/algorand/logging.config
du: cannot read directory ‘/mnt/algorand/mainnet-v1.0’: Permission denied
4.0K /mnt/algorand/mainnet-v1.0
1.1G /mnt/algorand/node.archive.log
806M /mnt/algorand/node.log
$ sudo du -hs /mnt/algorand
313G /mnt/algorand
$ cat $ALGORAND_DATA/config.json
{
“Version”: 19,
“AccountUpdatesStatsInterval”: 5000000000,
“AccountsRebuildSynchronousMode”: 1,
“AnnounceParticipationKey”: true,
“Archival”: true,
“BaseLoggerDebugLevel”: 4,
“BlockServiceCustomFallbackEndpoints”: “”,
“BroadcastConnectionsLimit”: -1,
“CadaverSizeTarget”: 1073741824,
“CatchpointFileHistoryLength”: 365,
“CatchpointInterval”: 10000,
“CatchpointTracking”: 0,
“CatchupBlockDownloadRetryAttempts”: 1000,
“CatchupBlockValidateMode”: 0,
“CatchupFailurePeerRefreshRate”: 10,
“CatchupGossipBlockFetchTimeoutSec”: 4,
“CatchupHTTPBlockFetchTimeoutSec”: 4,
“CatchupLedgerDownloadRetryAttempts”: 50,
“CatchupParallelBlocks”: 4,
“ConnectionsRateLimitingCount”: 60,
“ConnectionsRateLimitingWindowSeconds”: 1,
“DNSBootstrapID”: “.algorand.network”,
“DNSSecurityFlags”: 1,
“DeadlockDetection”: 0,
“DisableLocalhostConnectionRateLimit”: true,
“DisableNetworking”: false,
“DisableOutgoingConnectionThrottling”: false,
“EnableAccountUpdatesStats”: false,
“EnableAgreementReporting”: false,
“EnableAgreementTimeMetrics”: false,
“EnableAssembleStats”: false,
“EnableBlockService”: false,
“EnableBlockServiceFallbackToArchiver”: true,
“EnableCatchupFromArchiveServers”: false,
“EnableDeveloperAPI”: false,
“EnableGossipBlockService”: true,
“EnableIncomingMessageFilter”: false,
“EnableLedgerService”: false,
“EnableMetricReporting”: false,
“EnableOutgoingNetworkMessageFiltering”: true,
“EnablePingHandler”: true,
“EnableProcessBlockStats”: false,
“EnableProfiler”: false,
“EnableRequestLogger”: false,
“EnableTopAccountsReporting”: false,
“EnableVerbosedTransactionSyncLogging”: false,
“EndpointAddress”: “172.23.149.211:4001”,
“FallbackDNSResolverAddress”: “”,
“ForceFetchTransactions”: false,
“ForceRelayMessages”: false,
“GossipFanout”: 4,
“IncomingConnectionsLimit”: 800,
“IncomingMessageFilterBucketCount”: 5,
“IncomingMessageFilterBucketSize”: 512,
“IsIndexerActive”: true,
“LedgerSynchronousMode”: 2,
“LogArchiveMaxAge”: “”,
“LogArchiveName”: “node.archive.log”,
“LogSizeLimit”: 1073741824,
“MaxCatchpointDownloadDuration”: 7200000000000,
“MaxConnectionsPerIP”: 30,
“MinCatchpointFileDownloadBytesPerSecond”: 20480,
“NetAddress”: “”,
“NetworkMessageTraceServer”: “”,
“NetworkProtocolVersion”: “”,
“NodeExporterListenAddress”: “:9100”,
“NodeExporterPath”: “./node_exporter”,
“OptimizeAccountsDatabaseOnStartup”: false,
“OutgoingMessageFilterBucketCount”: 3,
“OutgoingMessageFilterBucketSize”: 128,
“ParticipationKeysRefreshInterval”: 60000000000,
“PeerConnectionsUpdateInterval”: 3600,
“PeerPingPeriodSeconds”: 0,
“PriorityPeers”: {},
“ProposalAssemblyTime”: 250000000,
“PublicAddress”: “”,
“ReconnectTime”: 60000000000,
“ReservedFDs”: 256,
“RestReadTimeoutSeconds”: 15,
“RestWriteTimeoutSeconds”: 120,
“RunHosted”: false,
“SuggestedFeeBlockHistory”: 3,
“SuggestedFeeSlidingWindowSize”: 50,
“TLSCertFile”: “”,
“TLSKeyFile”: “”,
“TelemetryToLog”: true,
“TransactionSyncDataExchangeRate”: 0,
“TransactionSyncSignificantMessageThreshold”: 0,
“TxPoolExponentialIncreaseFactor”: 2,
“TxPoolSize”: 15000,
“TxSyncIntervalSeconds”: 60,
“TxSyncServeResponseSize”: 1000000,
“TxSyncTimeoutSeconds”: 30,
“UseXForwardedForAddressField”: “”,
“VerifiedTranscationsCacheSize”: 30000
}
$ iostat
Linux 5.4.0-105-generic (algorand-node-and-kafka) 04/20/22 x86_64 (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8.41 0.00 2.71 8.86 0.13 79.89
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
loop0 0.01 0.01 0.00 0.00 32466 0 0
loop1 0.00 0.00 0.00 0.00 3604 0 0
loop2 0.00 0.00 0.00 0.00 3129 0 0
loop3 0.00 0.00 0.00 0.00 3645 0 0
loop4 0.01 0.01 0.00 0.00 13905 0 0
loop5 0.03 0.03 0.00 0.00 67308 0 0
loop6 0.00 0.00 0.00 0.00 3650 0 0
loop7 0.00 0.00 0.00 0.00 13 0 0
vda 12.87 13.42 327.12 0.00 32053649 781264694 0
vdc 1039.83 45.77 9275.26 0.00 109312677 22152265092 0
@crytocurv Did you set CatchupParallelBlocks value to 16 to increase the load speed of the mainnet?
What helped you to increase the speed?
This is way too slow.
You can see the current size of the ledger there:
Total syncing time should be less than 3 weeks and definitely 318GB in 2 weeks is much less than expected.
I would start by using the simplified config.json:
{
“Archival”: true
}
Thanks for the information. “Archival” parameter is already set to true in my config.json file. Do you mean deleting all other parameters in config.json file and edit the file to only include the parameter below:
{
“Archival”: true
}
Do I need to restart the node after editing the config.json, or will the node automatically begin working with the newest version of config.json after editing it?
Yes, just keeping "Archival": true and nothing else.
It is possible some of the parameters you have are bad.
In particular:
“IsIndexerActive”: true,
may create issues nowadays as this indexer is completely deprecated.
You need to restart the node afterwards.
No, I didn’t. I don’t think the config change would help much because the bottleneck is mostly disk I/O.
My nodes are now running with local NVMe SSD storage. Network attached storage, e.g., EC2 EBS, wouldn’t cut it because of high latency and jitter.
Check this web site below; it has very good info for node runners :