mindformers.trainer.MaskedLanguageModelingTrainer

class mindformers.trainer.MaskedLanguageModelingTrainer(model_name: Optional[str] = None)

MaskedLanguageModeling Task For Trainer. Args:

model_name (str): The model name of Task-Trainer. Default: None

Examples:

>>> from mindformers import MaskedLanguageModelingTrainer
>>> def generator():
>>>     data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32)
>>>     input_mask = np.ones_like(data)
>>>     token_type_id = np.zeros_like(data)
>>>     next_sentence_lables = np.array([1]).astype(np.int32)
>>>     masked_lm_positions = np.array([1, 2]).astype(np.int32)
>>>     masked_lm_ids = np.array([1, 2]).astype(np.int32)
>>>     masked_lm_weights = np.ones_like(masked_lm_ids)
>>>     train_data = (data, input_mask, token_type_id, next_sentence_lables,
...                   masked_lm_positions, masked_lm_ids, masked_lm_weights)
>>>     for _ in range(512):
...         yield train_data
>>> dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids",
...                                                     "next_sentence_labels", "masked_lm_positions",
...                                                     "masked_lm_ids", "masked_lm_weights"])
>>> dataset = dataset.batch(batch_size=16)
>>> mlm_trainer = MaskedLanguageModelingTrainer(model_name="bert_tiny_uncased")
>>> mlm_trainer.train(dataset=dataset)
>>> res = mlm_trainer.predict(input_data = "hello world [MASK]")

Raises:

NotImplementedError: If train method or evaluate method or predict method not implemented.

predict(config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[str, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs)

Executes the predict of the trainer.

Args:

config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]):

The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None.

input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[str, BaseModel]]): The network for trainer. It support model name supported

or BaseModel class. Supported model name can refer to model support list. For . Default: None.

tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text.

Default: None.

Returns:

A list of prediction.

train(config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs)

Train task for MaskedLanguageModeling Trainer. This function is used to train or fine-tune the network.

The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback.

Args:

config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]):

The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None.

network (Optional[Union[Cell, BaseModel]]): The network for trainer.

It supports model name or BaseModel or MindSpore Cell class. Default: None.

dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset.

It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None.

optimizer (Optional[Optimizer]): The training network’s optimizer. It support Optimizer class of MindSpore.

Default: None.

wrapper (Optional[TrainOneStepCell]): Wraps the network with the optimizer.

It support TrainOneStepCell class of MindSpore. Default: None.

callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function.

It support CallBack or CallBack List of MindSpore. Default: None.

Raises:

NotImplementedError: If wrapper not implemented.