Number of dimensions of each input embedding
Optional initSequenceLength: numberInitial sequence length that the positional embedding should be computed for, or DEFAULT_SEQUENCE_LENGTH if not specified
Private dimPrivate encodingPrivate encodingReadonly lengthReadonly nameReturns the name of the function. Function names are read-only and can not be changed.
Private sequenceStatic Readonly DEFAULT_The default initSequenceLength if none is supplied in the constructor.
Static Readonly ENCODING_The base of the exponent in the positional encoding.
Determines whether the given value inherits from this function if this function was used as a constructor function.
A constructor function can control which objects are recognized as its instances by 'instanceof' by overriding this method.
Calls the function, substituting the specified object for the this value of the function, and the specified array for the arguments of the function.
The object to be used as the this object.
Optional argArray: anyA set of arguments to be passed to the function.
For a given function, creates a bound function that has the same body as the original function. The this object of the bound function is associated with the specified object, and has the specified initial parameters.
An object to which the this keyword can refer inside the new function.
Rest ...argArray: any[]A list of arguments to be passed to the new function.
Calculate positional encodings at a given range of sequence positions.
a Tensor of calculated positional embeddings with shape [end - start, dim]
Start of the range to calculate
End of the range to calculate
Calls a method of an object, substituting another object for the current object.
The object to be used as the current object.
Rest ...argArray: any[]A list of arguments to be passed to the method.
Generated using TypeDoc
A module to generate the positional encoding for a Transformer of a given input dimension,
$$ \mathrm{PE}_{i, 2z} = \sin \left( \frac{i}{10000^{2z/d}} \right) $$
$$ \mathrm{PE}_{i, 2z + 1} = \cos \left( \frac{i}{10000^{2z/d}} \right) $$
where $i$ is the sequence position, $2z$ and $2z+1$ are the dimensions of the input embedding, and $d$ is the dimensionality of the input embedding.
The multiplicative factors $\frac{1}{10000^{2z/d}}$ are precomputed during object creation as they are constant for all $i$.
The full PE is initially precomputed for all $i$ up to 256 (configurable). If the module is called with a sequence length larger than the initial value, the additional values are also calculated and stored.