mindformers.models.t5.T5Config

class mindformers.models.t5.T5Config(vocab_size: int = 32128, hidden_size: int = 512, d_kv: int = 64, d_ff: int = 2048, num_layers: int = 6, num_decoder_layers: int = None, num_heads: int = 8, relative_attention_num_buckets: int = 32, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, embedding_dropout_prob: float = 0.1, layer_norm_epsilon: float = 1e-06, initializer_factor: float = 1.0, is_encoder_decoder: bool = True, use_cache: bool = True, pad_token_id: int = 0, start_token_id: int = 0, eos_token_id: int = 1, batch_size: int = 1, seq_length: int = 1024, max_position_embeddings: int = 1024, initializer_range: float = 0.02, max_decode_length: int = 128, length_penalty_weight: float = 1.0, compute_dtype: str = 'float32', has_relative_bias: bool = True, scale_output: bool = True, parallel_config: mindformers.modules.transformer.transformer.TransformerOpParallelConfig = <mindformers.modules.transformer.transformer.TransformerOpParallelConfig object>, checkpoint_name_or_path: str = None, top_p: float = 0.95, top_k: int = 1, repetition_penalty: float = 1.0, max_length: int = 20, do_sample: bool = False, param_init_type: str = 'float32', layernorm_compute_type: str = 'float32', softmax_compute_type: str = 'float32', hidden_act: str = 'relu', post_layernorm_residual: bool = False, offset: int = 0, use_past: bool = False, moe_config: mindformers.modules.transformer.moe.MoEConfig = <mindformers.modules.transformer.moe.MoEConfig object>, **kwargs)

T5 config class which defines the model size