mindformers.models.bert.BertTokenizer

class mindformers.models.bert.BertTokenizer(vocab_file, do_lower_case=True, do_basic_tokenize=True, unk_token='[UNK]', sep_token='[SEP]', pad_token='[PAD]', cls_token='[CLS]', mask_token='[MASK]', is_tokenize_char=False, **kwargs)

Bert Tokenizer.

save_vocabulary(save_directory, filename_prefix)

write the word to the files

property vocab_size

Return the vocab size