The function can be seen here: https://github.com/algorand/go-algorand/blob/3ff5722f4ca5645b670d9b34a1071b4c17fa60fe/data/pools/transactionPool.go#L304.
Congestion fees are based on how many blocks the transactions in the mempool could fill.
More information on fees can be found on the developer portal: Structure - Algorand Developer Portal