mindformers.models.clip.CLIPTextConfig

class mindformers.models.clip.CLIPTextConfig(vocab_size: Optional[int] = 49408, hidden_size: Optional[int] = 512, intermediate_size: Optional[int] = 2048, num_hidden_layers: Optional[int] = 12, num_attention_heads: Optional[int] = 8, max_position_embeddings: Optional[int] = 77, hidden_act: Optional[str] = 'quick_gelu', attention_dropout: Optional[float] = 0.0, drop_out: Optional[float] = 0.0, initializer_range: Optional[float] = 0.02, initializer_factor: Optional[float] = 1.0, **kwargs)

Config For CLIP Text Module

Args:

vocab_size (Optional[int]): Vocabulary size of the CLIP text model. hidden_size (Optional[int]): The dims of text features. intermediate_size (Optional[int]): Dimensionality of the “intermediate”

(i.e., feed-forward) layer in the Transformer encoder.

num_hidden_layers (Optional[int]): The number of transformer layers in text encoder. num_attention_heads (Optional[int]): Number of attention heads for each

attention layer in the Transformer encoder.

max_position_embeddings (Optional[int]): The maximum sequence length that

this model might ever be used with.

hidden_act (Optional[str]): The non-linear activation function

(function or string) in the encoder and pooler. Only “quick_gelu” supported currently.

attention_dropout (Optional[float]): The dropout ratio for the attention probabilities. dropout (Optional[float]): The dropout probabilitiy for all fully connected

layers in the embeddings, encoder, and pooler.

initializer_range (Optional[float]): The standard deviation of the

truncated_normal_initializer for initializing all weight matrices.

initializer_factor (Optional[float]): A factor for initializing all weight matrices

(should be kept to 1, used internally for initialization testing).

Examples:

>>> from mindformers import CLIPTextConfig
>>> CLIPTextConfig(hidden_size=256, vocab_size=40000)
    {'vocab_size': 40000, 'hidden_size': 256, 'intermediate_size': 2048,
     'num_hidden_layers': 12, 'num_attention_heads': 8, 'max_position_embeddings': 77,
      'hidden_act': 'quick_gelu', 'attention_dropout': 0.0, 'initializer_range': 0.02,
       'initializer_factor': 1.0}