mindformers.models.bert.BertForQuestionAnswering

class mindformers.models.bert.BertForQuestionAnswering(config={'assessment_method': '', 'attention_probs_dropout_prob': 0.1, 'batch_size': 16, 'checkpoint_name_or_path': '', 'compute_dtype': mindspore.float16, 'dropout_prob': 0.1, 'dtype': mindspore.float32, 'hidden_act': 'gelu', 'hidden_dropout_prob': 0.1, 'hidden_size': 768, 'initializer_range': 0.02, 'intermediate_size': 3072, 'is_training': True, 'layernorm_dtype': mindspore.float32, 'max_position_embeddings': 128, 'model_type': 'bert', 'moe_config': <mindformers.modules.transformer.moe.MoEConfig object>, 'num_attention_heads': 12, 'num_hidden_layers': 12, 'num_labels': 1, 'parallel_config': <mindformers.modules.transformer.transformer.TransformerOpParallelConfig object>, 'post_layernorm_residual': True, 'seq_length': 128, 'softmax_dtype': mindspore.float32, 'type_vocab_size': 2, 'use_one_hot_embeddings': False, 'use_past': False, 'use_relative_positions': False, 'vocab_size': 30522})

Bert with dense layer for question answering task.

Args:

config (BertConfig): The config of BertModel.

Returns:

Tensor, loss, logits.

Examples:

>>> from mindformers import BertForQuestionAnswering, BertTokenizer
>>> model = BertForQuestionAnswering.from_pretrained('qa_bert_base_uncased')
>>> tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased')
>>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "],
...                  return_tensors='ms', max_length=384, padding='max_length')
>>> input_ids = data['input_ids']
>>> attention_mask = data['attention_mask']
>>> token_type_ids = data['token_type_ids']
>>> output = model(input_ids, attention_mask, token_type_ids)
>>> print(output)
(Tensor(shape=[1, 384], dtype=Float32, value=
[[ 5.30713797e-02,  1.08105361e-01,  1.55855455e-02 ... -3.73562761e-02,
  -1.89271923e-02, -2.99206823e-02]]), Tensor(shape=[1, 384], dtype=Float32, value=
[[ 2.39873882e-02,  2.35675126e-01,  1.52210891e-01 ... -2.11214088e-02,
   4.84664962e-02, -4.52579707e-02]]))