mindformers.modules¶

MindFormers Transformers API.

mindformers.modules.layers¶

`mindformers.modules.layers.Dropout`	A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for parallel training.
`mindformers.modules.layers.FixedSparseAttention`	Fixed Sparse Attention Layer.
`mindformers.modules.layers.LayerNorm`	A self-defined layer norm operation using reduce sum and reduce mean
`mindformers.modules.layers.Linear`	The dense connected layer.

mindformers.modules.transformer¶

`mindformers.modules.transformer.AttentionMask`	Get the Lower triangular matrix from the input mask.
`mindformers.modules.transformer.EmbeddingOpParallelConfig`	The parallel config of `VocabEmbedding` for the setting data parallel or model parallel for the embedding table.
`mindformers.modules.transformer.FeedForward`	The multilayer perceptron with two linear layers with dropout applied at final output.
`mindformers.modules.transformer.MoEConfig`	The configuration of MoE (Mixture of Expert).
`mindformers.modules.transformer.MultiHeadAttention`	This is an implementation of multihead attention in the paper Attention is all you need.Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following.
`mindformers.modules.transformer.OpParallelConfig`	OpParallelConfig for the setting data parallel and model parallel.
`mindformers.modules.transformer.Transformer`	Transformer module including encoder and decoder.
`mindformers.modules.transformer.TransformerDecoder`	Transformer Decoder module with multi-layer stacked of TransformerDecoderLayer, including multihead self attention, cross attention and feedforward layer.
`mindformers.modules.transformer.TransformerDecoderLayer`	Transformer Decoder Layer.
`mindformers.modules.transformer.TransformerEncoder`	Transformer Encoder module with multi-layer stacked of TransformerEncoderLayer, including multihead self attention and feedforward layer.
`mindformers.modules.transformer.TransformerEncoderLayer`	Transformer Encoder Layer.
`mindformers.modules.transformer.TransformerOpParallelConfig`	TransformerOpParallelConfig for setting parallel configuration, such as the data parallel and model parallel.
`mindformers.modules.transformer.TransformerRecomputeConfig`	TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers.
`mindformers.modules.transformer.VocabEmbedding`	The embedding lookup table from the 0-th dim of the parameter table.