Indexer error: overflowed subtracting rewards for block

d13coBit D13Apr 23, 2022

Our indexer has gotten stuck with this error:

{“error”:“AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415”,“level”:“error”,“msg”:“adding block 20537415 to database failed”,“time”:“2022-04-23T20:09:44+02:00”}

We may have inadvertently caused this by running a background reindex (REINDEX CONCURRENTLY) while the indexer was running.

Any tips on how to get it unstuck?

fabriceFabrice BenhamoudaApr 25, 2022

First check that you have the latest version of the indexer.
If you have the latest version, the simplest solution is to clear the database and resync from the beginning but it takes a lot of time.

d13coBit D13Apr 25, 2022

We are on 2.10.0, which seems to be the latest version according to github releases.

The simplest solution you mention is the last resort, as it took us 77 days to sync from block 6M to current (no 2TB SSD servers available in our budget). We will likely abandon the endeavor of running our own indexer if we are unable to fix this instance.

Any ideas about the nature of the error? We couldn’t find it in the source code.

We found the indexer logs around the time the error was first encountered but nothing seems amiss:

Apr 21 10:06:51 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537413","time":"2022-04-21T10:06:51+02:00"}
Apr 21 10:06:52 k44 algorand-indexer[85573]: {"level":"info","msg":"round r=20537413 (41 txn) imported in 818.390509ms","time":"2022-04-21T10:06:52+02:00"}
Apr 21 10:06:56 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537414","time":"2022-04-21T10:06:56+02:00"}
Apr 21 10:06:56 k44 algorand-indexer[85573]: {"level":"info","msg":"round r=20537414 (30 txn) imported in 714.184434ms","time":"2022-04-21T10:06:56+02:00"}
Apr 21 10:07:00 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:00+02:00"}
Apr 21 10:07:01 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:01+02:00"}
Apr 21 10:07:02 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:02+02:00"}
Apr 21 10:07:02 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:02+02:00"}
Apr 21 10:07:03 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:03+02:00"}
Apr 21 10:07:04 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:03+02:00"}
Apr 21 10:07:04 k44 algorand-indexer[85573]: {"level":"info","msg":"adding block 20537415","time":"2022-04-21T10:07:04+02:00"}
Apr 21 10:07:04 k44 algorand-indexer[85573]: {"error":"AddBlock() err: TxWithRetry() err: attemptTx() err: AddBlock() eval err: EvalForIndexer() err: overflowed subtracting rewards for block 20537415","level":"error","msg":"adding block 20537415 to database failed","time":"2022-04-21T10:07:04+02:00"}

d13coBit D13Apr 25, 2022

The algod that feeds the indexer seems normal for those rounds as well.

Previous round header - processed OK:

{
  "block": {
    "earn": 218250,
    "fees": "Y76M3MSY6DKBRHBL7C3NNDXGS5IIMQVQVUAB6MP4XEMMGVF2QWNPL226CA",
    "frac": 332612560,
    "gen": "mainnet-v1.0",
    "gh": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
    "prev": "blk-XXDSGYHLHZZDXVVIKADQBE4SN2QRXWVFIXTBCYPENPILQ4RQ3LPA",
    "proto": "https://github.com/algorandfoundation/specs/tree/d5ac876d7ede07367dbaa26e149aa42589aac1f7",
    "rate": 600000,
    "rnd": 20537414,
    "rwcalr": 21000000,
    "rwd": "737777777777777777777777777777777777777777777777777UFEJ2CI",
    "seed": "FT3SeQnJSkZ0qmWcFCZl6P7x7HVYRtLIphqxPm4gJxk=",
    "tc": 710532082,
    "ts": 1650528411,
    "txn": "wNvbGfCyVSfjCMyIadbN1Wm0QxI28FFA1rwZCxMn1IA=",
    "txns": [

Round we are stuck on:

{
  "block": {
    "earn": 218250,
    "fees": "Y76M3MSY6DKBRHBL7C3NNDXGS5IIMQVQVUAB6MP4XEMMGVF2QWNPL226CA",
    "frac": 333212560,
    "gen": "mainnet-v1.0",
    "gh": "wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8=",
    "prev": "blk-6FDIWYYAY7PH3FJQFRDHJNEFFYHUEJONEGIJDVNSXQUA7VVR3S3A",
    "proto": "https://github.com/algorandfoundation/specs/tree/d5ac876d7ede07367dbaa26e149aa42589aac1f7",
    "rate": 600000,
    "rnd": 20537415,
    "rwcalr": 21000000,
    "rwd": "737777777777777777777777777777777777777777777777777UFEJ2CI",
    "seed": "WyCWA2l9/bD9IGR5NB4dNkUwPKp9jc/EPKoHuHot8bA=",
    "tc": 710532112,
    "ts": 1650528416,
    "txn": "KItphoDC0K4CpYaJounQFKIDb3f6EcDWCwbUXTYzUic=",
    "txns": [
      {

d13coBit D13Apr 26, 2022

Found the source of the error in the source code - it is in go-algorand repo:

github.com

algorand/go-algorand/blob/9b1375631151563dec1b1f8390957ca55f643078/ledger/internal/eval.go#L793

      
        
            
            
	err = eval.state.Put(poolAddr, poolNew)
            	if err != nil {
            		return nil, err
            	}
            
            
	// ensure that we have at least MinBalance after withdrawing rewards
            	ot.SubA(poolNew.MicroAlgos, basics.MicroAlgos{Raw: proto.MinBalance})
            	if ot.Overflowed {
            		// TODO this should never happen; should we panic here?
            		return nil, fmt.Errorf("overflowed subtracting rewards for block %v", hdr.Round)
            	}
            
            
	return eval, nil
            }
            
            
// hotfix for testnet stall 08/26/2019; move some algos from testnet bank to rewards pool to give it enough time until protocol upgrade occur.
            // hotfix for testnet stall 11/07/2019; do the same thing
            func (eval *BlockEvaluator) workaroundOverspentRewards(rewardPoolBalance ledgercore.AccountData, headerRound basics.Round) (poolOld ledgercore.AccountData, err error) {
            	// verify that we patch the correct round.
            	if headerRound != 1499995 && headerRound != 2926564 {

johnleeJohn LeeApr 26, 2022

Hmm, unfortunately I think this is broken, because the database seems to be in an inconsistent state. For data integrity I’d unfortunately recommend re-indexing, because the block evaluator is getting a different answer on the indexer side.

CryptocurvApr 27, 2022

↩ in reply to d13co

If your project is small, there’s a free indexer API service available at AlgoNode | Free Algorand API provider. I think they have a good reputation.

Also, the website provides a blockchain snapshot for a quick, full sync-up for non-production environment - i.e., use it on your own risk.

joeJoe RhodesMay 10, 2022

This particular error can happen if you upgraded Indexer (I think it was 2.6 → 2.8) without doing a re-sync of the DB. There was a pretty significant DB change and I don’t think Indexer could update the DB. I vaguely recall it throwing this error.

Also, from 2.9 to 2.10 there was a DB update that takes about 1 hour on good hardware to update.

It’s unfortunate, but like others said, you may be in for a re-index from block 0

d13coBit D13May 14, 2022

Thanks all. This was very disappointing and we will likely not run an indexer going forward. We’ll make do with the free/community options.

We may attempt to fix it by deleting the last few blocks entered but not holding our breaths about doing this correctly.

There was no indexer upgrade involved BTW.