mindformers.models.llama.LlamaConfig

class mindformers.models.llama.LlamaConfig(batch_size: int = 1, seq_length: int = 2048, hidden_size: int = 4096, num_layers: int = 32, num_heads: int = 32, vocab_size: int = 32000, multiple_of: int = 256, rms_norm_eps: float = 1e-05, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 32000, ignore_token_id: int = -100, compute_dtype: str = 'float16', layernorm_compute_type: str = 'float32', softmax_compute_type: str = 'float32', param_init_type: str = 'float16', parallel_config: mindformers.modules.transformer.transformer.TransformerOpParallelConfig = <mindformers.modules.transformer.transformer.TransformerOpParallelConfig object>, use_past: bool = False, offset: int = 0, checkpoint_name_or_path: str = '', repetition_penalty: float = 1.0, max_decode_length: int = 1024, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, **kwargs)

LLaMA config class which defines the model size.