I’ve been reading the puya-ts compiler and found a bug that let two contract members share the same storage key without any error, which silently overwrote state (I sent a fix: https://github.com/algorandfoundation/puya-ts/pull/408). It made me wonder what other mistakes pass the current unit tests.
I’m considering a small open-source tool that runs property-based and fuzz tests against Algorand contracts written in Algorand TypeScript or Puya Python. It would generate random inputs and call sequences, and check invariants you define, such as “total supply never changes” or “only the owner can call this method”.
Before I build anything, I’d like to hear from people who write contracts:
-
Have you had a bug that your unit tests missed? What was it?
-
Would you use something like this, and in which language: TypeScript or Python?
-
Is there something already doing this that I’ve missed?
-
What would it need to do on day one to be worth trying?
If nobody needs it, I’d rather know now. Thanks for any replies.
Update: I tried the idea on the Foundation’s auction example from algorand-typescript-testing, using fast-check to generate random sequences of bids and claims by three accounts, and checking after every step that the contract holds exactly what it owes.
It found a problem within 8 random runs and shrank it to the smallest case: one account bids 1, then bids 2. The contract holds 3 but only owes 2, so 1 is locked forever. The cause is that bid() sets the claimable amount to the new bid instead of adding to the earlier deposit. With a one line change to add instead, the same test passes 300 random runs.
This is an example contract, not a production one, so I’m not calling it a vulnerability. But it is the kind of bug that single-case unit tests don’t catch and random sequences do.
I’m now turning this into a small open-source library, so that writing a test like this takes a few lines. Same questions as above: would you use it, in TypeScript or Python, and what would it need to do first?
Update: the tool now exists and is on npm as algo-property-test (MIT, TypeScript). You describe the calls your contract accepts and the rules that must always hold, it runs random call sequences from several accounts, and when a rule breaks it shrinks the failure to the shortest sequence and prints a seed to reproduce it. It supports payments and asset transfers, and works with the Foundation’s algorand-typescript-testing library.
Source and README: GitHub - ss1738/algo-property-test: Property-based testing for Algorand TypeScript smart contracts: random call sequences, invariants, shrinking · GitHub
I ran it on the Foundation’s own examples and opened two issues from what it found: the voting example counts one voter once per question (the existing spec expects the wrong number), and in the marketplace example a fully accepted bid leaves a stale bidder that blocks lower bids. They are issues 147 and 148 on algorand-typescript-testing.
If you write contracts in TypeScript, I would like to know whether it runs on yours and what is missing. A failing example or a rule it could not express would help most.