mindformers.modules.layers.LayerNorm

class mindformers.modules.layers.LayerNorm(normalized_shape, eps=1e-05, param_init_type=mindspore.float32, is_self_defined=False)

A self-defined layer norm operation using reduce sum and reduce mean

Args:

normalized_shape (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type.

Inputs:

• x (Tensor) - Tensor of shape \((batch, seq\_length, hidden\_size)\).

Outputs:

Tensor of shape \((batch, seq_length, hidden_size)\).

shard(strategy)

Set the shard for the layer norm. the strategy size should be equal to the inputs.

Note:

It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored.

Args:

strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs.

Examples:

>>> import mindspore
>>> net = mindformers.modules.transformer.LayerNorm(normalized_shape=(1024, 10))
>>> net.shard(((10, 2, 1),))