Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [huggingface]

91 questions
7
votes
1 answer

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part already handled. (I already have a tensor of…
Pablo Messina
  • 377
  • 2
  • 10
4
votes
1 answer

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long document, but there's no pretrained in spanish so, is…
Simone
  • 242
  • 2
  • 9
4
votes
1 answer

How to measure the accuracy of an NLP paraphrasing model?

I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with t5-base, the other with t5-small) for this task?…
carrot_142
  • 63
  • 3
3
votes
2 answers

How to use is_split_into_words with Huggingface NER pipeline

I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train. My incoming text has already been split into words. When tokenizing during training/fine-tuning I can use…
Alan Buxton
  • 131
  • 4
2
votes
1 answer

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead I am getting loss: nan - accuracy: 0.0000e+00. My code is largely per the boiler…
JasonExcel
  • 21
  • 2
2
votes
1 answer

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary word/token, that is being split into em,bed,ding,s. I would…
cerofrais
  • 121
  • 4
2
votes
1 answer

LMM Fine Tuning - Supervised Fine Tuning Trainer (SFTTrainer) vs transformers Trainer

When should one opt for the Supervised Fine Tuning Trainer (SFTTrainer) instead of the regular Transformers Trainer when it comes to instruction fine-tuning for Language Models (LLMs)? From what I gather, the regular Transformers Trainer typically…
Marvin Martin
  • 21
  • 1
2
votes
2 answers

Dynamic batching and padding batches for NLP in deep learning libraries

This is the usual way we train modern deep learning models for NLP, e.g. with Huggingface libraries where we have a fix length for the input no. of tokens/subwoords unit. https://huggingface.co/docs/transformers/pad_truncation In the follow example,…
alvas
  • 2,340
  • 6
  • 25
  • 38
2
votes
1 answer

What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?

I am building a project for my bachelor thesis and am wondering how to prepare my raw data. The goal is to program some kind of semantic search for job postings. My data set consists of stored web pages in HTML format, each containing the detail…
nesquick
  • 23
  • 3
2
votes
0 answers

How to train a Task Specific Knowledge Distillation model using Hugging face model

I was referring to this code: https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker/blob/master/knowledge-distillation.ipynb From @philschmid I could follow most of the code, but had few doubts. Please help me to…
MAC
  • 277
  • 2
  • 9
2
votes
2 answers

How to improve language model ex: BERT on unseen text in training?

I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a specific similar sentence from the training data and…
IS92
  • 123
  • 3
1
vote
0 answers

How to do batch inference on Hugging face pretrained models?

I want to do batch inference on MarianMT model. Here's the code: from transformers import MarianTokenizer tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-de') src_texts = [ "I am a small frog.", "Tom asked his teacher for…
sai_varshittha
  • 11
  • 1
1
vote
0 answers

Train and validation sets splits using load_data

I'm using the package "datasets". The code I have: squad_v2 = True model_checkpoint = "roberta-base" batch_size = 10 from datasets import load_dataset, load_metric datasets = load_dataset("squad_v2") datasets and output: It's working fine, but as…
Adolf Miszka
  • 149
  • 6
1
vote
0 answers

Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling

I intend to use 2 tasks of modelling including (a) Causal language modelling & (b) Mask language modelling for training my new added tokens My pseudo-code is below ##add new tokenizer model_name = "vinai/phobert-large" model =…
Lắc Lê
  • 11
  • 1
1
vote
1 answer

Masked Language Modeling on Domain-specific Data

My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even training a language model(like BERT) from scratch,…
mitra mirshafiee
  • 153
  • 3
1
2 3 4 5 6 7