If you didn’t catch part 1, read that first, because this is a continuation.
In part 1, I talked about how ChatGPT is an interface to the underlying GPT engine, and that developers can access the engine directly and write applications that modify prompts before they go to GPT. This changes its behaviour, and this is called Prompt Engineering.
In this article I’ll take you further into how you can shape GPT to fit your business, and introduce a couple of new concepts: Fine-tuning and Semantic Search. Both of these techniques help you make GPT applications that are distinctly yours.
To make an application for your business that is different to ChatGPT, there’s a couple of key things you probably want to achieve:
Before we get to how to do that, let me explain a little but more about GPT. There are a couple of substantial things that OpenAI did to make ChatGPT what it is today. First, they trained their model on a lot of data - in the order of a trillion tokens (a token is a snippet of text). But the second thing was fine-tuning. Essentially what they did was get a small army of humans to rate the answers GPT gave to questions, and fed that feedback back into the model. By all reports, it was this fine-tuning process that took the GPT model from being an incoherent mess, to the outperforming, philosophy-inducing triumph that it is today.
You’ll notice when you interact with ChatGPT that it has a quite particular conversational style. For example, if you correct it, or get annoyed with it, it will apologise profusely and try to provide an improved response. And it will generally repeat your question back to you before it answers, and often again after it has answered. Given the amount of work that went into making GPT work this way, you’ll be surprised at how easy it is to change this conversational style.
Fine-tuning is no more complicated than providing a set of sample questions and answers for GPT to learn from. There’s an art to doing it well, but fundamentally, all you need is about 200-250 really great examples of how you want the model to behave. That’s it - just 250 examples, and you can radically change the style of the conversation.
For example, it might be important that if the model doesn’t know the answer to the question, that it says it doesn’t, rather than just making something up. Also, it might be important for the model to politely decline to answer when asked about something which is off-topic for it. So, if you can provide high-quality examples of these kinds of behaviours, for the breadth of your use case, that’s how you start with fine-tuning.
What actually happens is that you take a copy of the GPT engine of your own, and then fine-tune that. When you build your application, you use your fine-tuned model behind the scenes, instead of the original GPT model. This ability to make a copy of GPT and use it via APIs is a capability that OpenAI provides.
When I spoke about Prompt Engineering in part 1, I said that it could get you most of the way to this same outcome, and that’s true. There will be many use cases and scenarios where Prompt Engineering is all you need, but others where fine-tuning will be more cost-effective. Essentially, your usage of APIs is priced on tokens (snippets of text), and so Prompt Engineering will cost more per-request, whereas fine-tuning will cost more up-front, but less per-request.
But that’s part of a much larger discussion of running costs - for another article!
Okay, so the other half of the make it yours was extending GPT’s knowledge to include your own content, and to keep it on-topic. This is where Semantic Search comes in. In the simplest terms, what we’re going to do is put our content in a database, use GPT to understand what each bit of content is about, and then when the user asks a question, we’ll find the content in the database that is most related to the question, feed that content to GPT, and then have GPT turn that into an answer.
Phew - let me break that down a bit.
Okay, so first step was to put our content into a database. It doesn’t matter what content, or what database. Your content might already be in a database, or maybe its in documents on a file share - in any case, we’ll take the content and create tokens (remember those? Snippets of text!) and put them in some kind of database.
The next step is to take each token, and ask GPT to create an embedding for it. That’s a fancy word, but it’s just a numerical representation of the meaning (or semantics) of the token. So two tokens that have a similar meaning - even though they might use completely different words - will have embeddings that are numerically “close”. So we store all these embeddings with our tokens.
Then, when we build our application, what we’re going to do when someone asks a question, is ask GPT to create an embedding of the question. We can then - just using some simple maths - find the content in our database that’s most semantically similar to the question - hence the term, Semantic Search. These are the documents/paragraphs/knowledge-base articles etc etc that are likely to be about the question, and hence most likely to contain the answer to the question.
Okay, so the user asked a question, and we now have (say) the top 5 paragraphs from our database that are most likely to contain the answer to the question. What now? Well, this is where GPT really shines. We can use our Prompt Engineering techniques to wrap the question and the information together, and have GPT turn them into an answer.
Something like this:
Answer the following question concisely, based on the following information:
QUESTION: [The user’s question]
INFORMATION: [The relevant content from our database]
This is how we “extend” GPT’s knowledge to include our own content. And to keep it on topic. Combined with other settings (like “temperature”) and fine-tuning, this can be a very powerful way to create a GPT-based application that is bent into the shape you want.
Hopefully I’ve given you just enough knowledge to visualise how you can take ChatGPT’s technology and shape it to your own context. I hope that’s useful. There’s a lot more to it, and I’ll be writing more articles about how to do this, and what the costs & benefits are, and so on.
But for now, I hope you’re excited about the possibilities. If you’ve got an idea, we’d love to hear from you!
Reach out about a problem you have, an opportunity you've
spotted, or an idea that inspires you.
We'd love to hear about it!