mindformers.models.gpt2.GPT2Config

class mindformers.models.gpt2.GPT2Config(batch_size: int = None, eos_token_id: int = 50256, pad_token_id: int = 50256, bos_token_id: int = 50256, unk_token_id: int = 50256, seq_length: int = 1024, vocab_size: int = 50257, hidden_size: int = 768, num_layers: int = 12, num_heads: int = 12, expand_ratio: int = 4, embedding_dropout_prob: float = 0.1, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, param_init_type: str = 'float32', layernorm_compute_type: str = 'float32', softmax_compute_type: str = 'float32', compute_dtype: str = 'float16', hidden_act: str = 'gelu', use_past: bool = False, post_layernorm_residual: bool = False, offset: int = 0, parallel_config: mindformers.modules.transformer.transformer.TransformerOpParallelConfig = <mindformers.modules.transformer.transformer.TransformerOpParallelConfig object>, checkpoint_name_or_path: str = '', moe_config: mindformers.modules.transformer.moe.MoEConfig = <mindformers.modules.transformer.moe.MoEConfig object>, repetition_penalty: float = 1.0, max_decode_length: int = 1024, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, **kwargs)

Gpt config class which defines the model size