The success of transformer models is in large part thanks to the ability to take a model that has been pre-trained on gigantic datasets by the likes of Google and OpenAI — and apply them to our own use-cases.

Sometimes, this is all we need — we take the model and roll with it as is.

But at other times, we find that we really need to fine-tune the model. We need to train it a little bit more on our specific use case.

Each transformer model is different, and fine-tuning for different use-cases is different too — so we’ll…

Python 3.10 development has stabilized and we can finally test out all of the new features that will be included in the final release.

We’ll cover some of the most interesting additions to Python — structural pattern matching, parenthesized context managers, more typing, and the new and improved error messages.

Check out the video version of the article here:

Structural Pattern Matching

Structural pattern matching is an incredible feature to be added to Python — truly awesome.

Imagine an if-else statement that looks like this:

You take that and you modify the syntax so it looks more…

Next sentence prediction (NSP) is one-half of the training process behind the BERT model (the other being masked-language modeling — MLM).

Although NSP (and MLM) are used to pre-train BERT models, we can use these exact methods to fine-tune our models to better understand the specific style of language in our own use-cases.

So, in this article, we’ll cover exactly how we take an unstructured body of text, and use it to fine-tune a BERT model using NSP.

Training

So, how can we fine-tune a model using NSP?

First, we need data. Because we’re essentially just switching between consecutive sentences…

Next sentence prediction (NSP) is one-half of the training process behind the BERT model (the other being masked-language modeling — MLM).

Where MLM teaches BERT to understand relationships between words — NSP teaches BERT to understand longer-term dependencies across sentences.

Without NSP, BERT performs worse on every single metric [1] — so it’s important.

Now, when we use a pre-trained BERT model, training with NSP and MLM has already been done, so why do we need to know about it?

Well, we can actually fine-tune these pre-trained BERT models so that they better understand the language used in…

BERT, everyone’s favorite transformer costs Google ~$7K to train [1] (and who knows how much in R&D costs). From there, we write a couple of lines of code to use the same model — all for free.

BERT has enjoyed unparalleled success in NLP thanks to two unique training approaches, masked-language modeling (MLM), and next sentence prediction (NSP).

In many cases, we might be able to take the pre-trained BERT model out-of-the-box and apply it successfully to our own language tasks.

But often, we might need to fine-tune the model.

Further training with MLM allows us to…

Named entity recognition (NER) consists of extracting ‘entities’ from text — what we mean by that is given the sentence:

“Apple reached an all-time high stock price of 143 dollars this January.”

We might want to extract the key pieces of information — or ‘entities’ — and categorize each of those entities. Like so:

Apple — Organization

143 dollars — Monetary Value

this January — Date

For us humans, this is easy. But how can we teach a machine to distinguish between a granny smith apple and the Apple we trade on NASDAQ?

(No, we can’t rely on the ‘A’…

All we ever seem to talk about nowadays are BERT this, BERT that. I want to write about something else, but BERT is just too good — so this article will be about BERT and sequence similarity!

A big part of NLP relies on similarity in highly-dimensional spaces. Typically an NLP solution will take some text, process it to create a big vector/array representing said text — then perform several transformations.

It’s highly-dimensional magic.

Sentence similarity is one of the clearest examples of how powerful highly-dimensional magic can be.

The logic is this:

• Take a sentence, convert it into a…

Type annotations have a long and convoluted history with Python, going all the way back to the first release of Python 3 with the initial implementation of function annotations.