Support "quiet softmax" (Attention Is Off By One)

[wbrickner]

Feature description

• Softmax activation that adds 1 to the denominator: exp(x_i) / [ 1 + ∑ exp(x_j) ],
• TranformerEncoder / MultiHeadAttention attention implementation that can conditionally use this activation instead of the default softmax.

Feature motivation

• Attention is Off By One

TLDR:

• Attention heads are forced (by the normal softmax implementation) to deposit information even when no useful information exists in a head for the present sequence. This is bad. It also results in large variance of weights that make quantization and compression difficult.
• This is fixed by adding 1 to the denominator.

Suggest a Solution

• I propose a quiet_softmax activation function
• I propose a use_quiet_softmax configuration field for MultiHeadAttention (on MultiHeadAttentionConfig), and to provide this field as well for all the layers that use MultiHeadAttention internally (like TransformerEncoder).
• There is a case to be made that this should be the default softmax implementation

[wbrickner]

PR submitted which resolves this: #692

[wbrickner]

Merged, closed.