mindformers.modules.layers.LayerNorm

class mindformers.modules.layers.LayerNorm(normalized_shape, eps=1e-05, param_init_type=mindspore.float32, is_self_defined=False)

A self-defined layer norm operation using reduce sum and reduce mean

参数

• normalized_shape (tuple) – The shape of the input tensor

• eps (float) – The epsilon value of the denominator. Default 1e-5.

• param_init_type – The param init type.

Inputs:

• x (Tensor) - Tensor of shape \((batch, seq\_length, hidden\_size)\).

Outputs:

Tensor of shape \((batch, seq_length, hidden_size)\).

shard(strategy)

Set the shard for the layer norm. the strategy size should be equal to the inputs.

注解

It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored.

参数

strategy (tuple) – The strategy for the dropout. Should be the same shape as the inputs.

实际案例

>>> import mindspore
>>> net = mindformers.modules.transformer.LayerNorm(normalized_shape=(1024, 10))
>>> net.shard(((10, 2, 1),))