3

Thanks to GPT2 pretrained model now it is possible to generate meaningful sequence of words with or without prefix. However a sentence should end with a proper endings (.,!,?). I am just wondering how to generate a sentence (with proper ending) of length N?

One possible approach is post-processing, that is process many sequences and choose the ones the serve the purpose! However, it could be a really daunting task to use in any pipeline.

Is there any suggestion, perhaps a secondary algorithm, to tune the hyper-parameter such that it produces sentence of desired length with higher probabilities.

user185597
  • 31
  • 2

2 Answers2

1

Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.

Noah Weber
  • 5,609
  • 1
  • 11
  • 26
  • Interesting note. My knowledge of GPT2 is limited but I have played around with internal structure to some extent. It did not seem to me that it is possible to fix the output length (let alone meaningfulness) with changing parameters though. If you happen to know any ref I truly appreciate that. – user185597 Mar 04 '20 at 15:12
0

According to the tests I have performed GPT2 model is strong enough to end the sentences with meaningful punctuation at the end. Basically its upto the size and quality of data you have used if you fine tuning it.

Regarding length, if you are so sure you want you output to be N token long, then that can be be by editing the script's length argument in interact_model function.

But fixing your output to certain length will not guarantee to generate a sentence with proper ending as it totally depends on the N-1th term of your output.

Pooja Sonkar
  • 11
  • 4