ChatGPT broke our product this weekend.
When you sign up for our trial we ask you to give us a URL, which we send to ChatGPT so that it can visit the site and extract a set of search terms for the user.
Doing this gives new users a richer experience, because it creates a nice tailored search for them and we can give them relevant new contracting opportunities.
Except this weekend it didn't. In our prompt we ask ChatGPT to come up with "two or three word" search terms.
That has worked for 6 months, without fail.
This weekend ChatGPT started returning only three word search terms, which returned much fewer search results and often zero results.
We hadn't changed anything.
Same model.
Same prompt.
Different result.
This is the only place in our tool where our users engage with an LLM, so it's not like we're just running a wrapper, but it didn't cross my mind that the LLM could change like this.
Obviously we've added a quick fix, but looking forwards we have to take a much more defensive approach to working with these tools. Here's what we're going to do going forwards:
Be super precise - if you want "two or three word" search terms. Send one request for two word terms and another for three word terms.
Monitor everything - if you're precise you can check the results and know it is / isn't working.
Have a back up LLM - be ready to move to a new LLM or mirror your requests into a second LLM so that you can choose the best results.