mindformers.models.clip.CLIPTextConfig¶

class mindformers.models.clip.CLIPTextConfig(vocab_size: Optional[int] = 49408, hidden_size: Optional[int] = 512, intermediate_size: Optional[int] = 2048, num_hidden_layers: Optional[int] = 12, num_attention_heads: Optional[int] = 8, max_position_embeddings: Optional[int] = 77, hidden_act: Optional[str] = 'quick_gelu', attention_dropout: Optional[float] = 0.0, drop_out: Optional[float] = 0.0, initializer_range: Optional[float] = 0.02, initializer_factor: Optional[float] = 1.0, **kwargs)[源代码]¶

Config For CLIP Text Module

参数

vocab_size (Optional[int]) – Vocabulary size of the CLIP text model.
hidden_size (Optional[int]) – The dims of text features.
intermediate_size (Optional[int]) – Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.
num_hidden_layers (Optional[int]) – The number of transformer layers in text encoder.
num_attention_heads (Optional[int]) – Number of attention heads for each attention layer in the Transformer encoder.
max_position_embeddings (Optional[int]) – The maximum sequence length that this model might ever be used with.
hidden_act (Optional[str]) – The non-linear activation function (function or string) in the encoder and pooler. Only “quick_gelu” supported currently.
attention_dropout (Optional[float]) – The dropout ratio for the attention probabilities.
dropout (Optional[float]) – The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
initializer_range (Optional[float]) – The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
initializer_factor (Optional[float]) – A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).

实际案例

>>> from mindformers import CLIPTextConfig
>>> CLIPTextConfig(hidden_size=256, vocab_size=40000)
    {'vocab_size': 40000, 'hidden_size': 256, 'intermediate_size': 2048,
     'num_hidden_layers': 12, 'num_attention_heads': 8, 'max_position_embeddings': 77,
      'hidden_act': 'quick_gelu', 'attention_dropout': 0.0, 'initializer_range': 0.02,
       'initializer_factor': 1.0}