Friday, 14 January 2022

sentence transformer using huggingface/transformers pre-trained model vs SentenceTransformer

This page has two scripts

When should one use 1st method shown below vs 2nd? As nli-distilroberta-base-v2 trained specially for finding sentence embedding wont that always be better than the first method?

training_stsbenchmark.py1 -

from sentence_transformers import SentenceTransformer,  LoggingHandler, losses, models, util
#You can specify any huggingface/transformers pre-trained model here, for example, bert-base-uncased, roberta-base, xlm-roberta-base
model_name = sys.argv[1] if len(sys.argv) > 1 else 'distilbert-base-uncased'

# Use Huggingface/transformers model (like BERT, RoBERTa, XLNet, XLM-R) for mapping tokens to embeddings
word_embedding_model = models.Transformer(model_name)

# Apply mean pooling to get one fixed sized sentence vector
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(),
                               pooling_mode_mean_tokens=True,
                               pooling_mode_cls_token=False,
                               pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

training_stsbenchmark_continue_training.py -

from sentence_transformers import SentenceTransformer, LoggingHandler, losses, util, InputExample
model_name = 'nli-distilroberta-base-v2'
model = SentenceTransformer(model_name)


from sentence transformer using huggingface/transformers pre-trained model vs SentenceTransformer

No comments:

Post a Comment