How to search the blockchain?

Hi, are there some best practicies how to search the whole blockchain for some data?

I am getting timeout for searching for specific amount with limit 1, so I assume there is a bug in the indexer. I have filed the bug report here, so hopefully someone will check it out…

To search the blockchain by the amount should be the basic feature that the blockchain apis provides… Am I wrong with my opinion?

What should be considered as best practice? Should i search with minround and maxround? What is recommended round amount in one search? If these ints are properly indexed, why the amount int is not?

Can you link the issue here?
What exact query did you try?

The indexer uses an SQL database behind the scenes.
Every query you make is converted to an SQL query.
Indexes have been chosen to quickly answer usual queries.
But some queries may require you to manually add indexes.

The reason is: adding indexes is a tradeoff.
When you have more indexes, you can answer more queries faster, but on the other hand, inserting new transactions take more time and the database grows significantly.

Sure no problem… if the markup link is not enough for you, here you go

https://algoexplorerapi.io/idx2/v2/transactions?currency-greater-than=300000000&currency-less-than=300000000&limit=1

I dont want to have indexer running. (I expect from the blockchain network to provide some basic free tools to operate on it)

However I have not seen the response in your answer… What is the best practice to search the network? If the answer was that every developer should run the full archvial node and full indexer node and he should modify it for his need I hope i read it wrong…

Sorry I missed the link.
I’ve answered directly in the issue.

But in short, the issue is that greater-than and less-than are “strict”.
So you need:

https://algoexplorerapi.io/idx2/v2/transactions?currency-greater-than=299999999&currency-less-than=300000001&limit=1

which is immediate.

But I agree that there should be a clear error message in that case saying that the query is not correct.

It would be cool to have there this validation check… thanks for correcting me…

What is wrong with this timeouted request?

https://algoexplorerapi.io/idx2/v2/transactions?currency-greater-than=299999997&currency-less-than=299999999&limit=1

I still think there is some index missing because it should return zero transaction list right away… no?

I haven’t looked at the details but it may indeed be a lack of index.

Do you have an example of dApp or use where you would need such a query?

The issue with indexes is that they are a complex trade-off. Adding more of them is very costly in terms of storage and time to add new transactions.
If you have a concrete use case that absolutely needs it, I would recommend commenting on your Github issue.

Sure, have you read Governance - proposal for technical realisation for voting - #23 by scholtz ?

There is no real means of searching the blockchain, and only thing how it can be solved is the custom asset or have the amount indexed properly… I do not consider the search by note prefix as effecient enough because it never can be indexed properly by the current databases, but to index by the amount it is very possible, but it needs to be indexed…

Index by the tx amount can be usefull for a lot of usecases… this is just one of them…

We’ve actually been working on facilitating such a use case: finding transactions related to a specific dApp, by providing a special format for the note field:

It is not yet implemented in the indexer but once the proposal is finalized, the goal is to allow the indexer to easily filter such transactions.

Would this be a solution for your use case?

No, because i do not want to have dapp… I want to search the blockchain for the note field not related to any application…

I agree that the search over the applications should also be improved, for example by searching by the hash of approval program. This is not related to this thread thou, so if you feel start the new thread for this… i have created github issue for this in the indexer, so perhaps you can reply there…

Sorry, the term dApp above is confusing.
It is to be understood in a very large sense.
It includes anything making transactions to the blockchain.
It does not restrict to smart contracts at all.

ok, I think I understand… you want everyone to use the specific format for note fields like avote/v1:j{json}

i can comply with this, this is good idea if the indexers would allow to search by the app name …

also it would be good for the wallets to display the note field and know the data type of related data {text|messagePack|json|binary}

have you considered also adding there other compression protocols such as FAST for example?

Yes, that’s the goal that indexer and blockchain explorers would support it.

It can be a good idea to allow for compressed data. A very naive question I have (I really did not look at it) is whether there exist compression algorithm that would work well with such a small output restriction (1kB). For example, compression “Hello” using zip on my computer yields a file of size 168B.
So I’m wondering whether compression would help that much if you’re limited to 1kB output.

the fast protocol is meant to do templating of predefined messages… so you know that the first 8 fields are boolean, so you use only one byte for the raw data… it is being used for example in the exchange systems to provide market data feeds and more… there is not much compression over text, but i can imagine that with additional meta data stored elsewhere other effecient compression methods might exists

the 1 byte plus few bytes to identify the template id versus json 86 bytes {“a1”:true,“a2”:false,“a3”:true,“a4”:false,“a5”:true,“a6”:false,“a7”:true,“a8”:false}
is quite significant…

also there is some json binary encoder called cbor… it has been used in eu green certificates… see https://ec.europa.eu/health/sites/default/files/ehealth/docs/trust-framework_interoperability_certificates_en.pdf

json > cbor > cose > zlib > aztec qr