Profiling the proposals' transaction processing loop

[DrZoltanFazekas]

The ongoing stress tests revealed that the more transactions (simple ZIL transfers) a block contains, the larger the gap between the block's timestamp and the next block's timestamp. This is due to the fact that every validator incl. the next block proposer spends a large amount of time looping through the transactions in the block they receive, checking and executing them. It's roughly ~40ms per transaction. During this loop there is no interrupt or exit condition i.e. the validators don't time out after 5s (default value of the exponential backoff), it just delays the creation of the next block. We must find the expensive operations performed on every transactions and try to optimize/eliminate them.

[DrZoltanFazekas]

Related to the benchmarking James has been doing...