Hey Snehal, yes I'm sure splitting into proper sentences/paragraphs would improve the results of…

Some issues with splitting text by token: At the breakage point, we might loose proper sentence structure. Depend on the model, it might not be an issue. But let's say we want to ensure splitting of longer text into "n" chunks where each chunk contain valid sentences, we can use a sentence segmentation logic to split the original text into a chunks of size that's appropriate for downstream model to process.
Some issues with splitting text by token: At the breakage point, we might loose proper sentence…
24
1
Snehal
James Briggs
·Follow
Mar 13, 2021
--
Hey Snehal, yes I'm sure splitting into proper sentences/paragraphs would improve the results of this method for most (if not all) models, thanks for sharing!
--
--
Written by James Briggs11.4K Followers
·36 Following
Freelance ML engineer learning and writing about everything. I post a lot on YT https://www.youtube.com/c/jamesbriggs
No responses yet
Help
Status
About
Careers
Press
Blog
Privacy
Terms
Text to speech
Teams