InstructGPT > Google Search
I’ve talked about it on my podcast, and tweeted about it endlessly:
… but I think it’s quite clear. For various reasons, the quality of Google Search results have significantly declined. To name just a few issues I have with Google results … there are far more ads, it often misidentifies the intention of my search query, I can tell its AI is confusing the context of my keywords, and it has become so overloaded with SEO and content marketing posts, it has become a lot more difficult to know who to trust. I often walk away with unclear, indefinite answers and find myself frustrated regularly from the search result experience.
Not only has the quality of Google Search results declined, but as a GPT-3 developer, I’ve also noticed how much of our technological capabilities have grown in comparison to Google’s core search technology over the last few years. Through Large Language Models (LLM) like GPT-3, we’ve seen great advancements in many of the classic NLP areas like summarization, Q&A, and content generation.
GPT-3 has become so impressive and capable, I have found myself using it far more for answers to basic questions instead of Google.
Some of the queries I enter into the GPT-3 playground are just for the purpose of having information synthesized and answered immediately in a simple way:
Others are more driven by curiosity:
Regardless, what I just find so fascinating is that I’ve been using GPT-3 more and more often as a search engine alternative. Especially when I have a broad question, need information synthesized, or am looking for a simple explanation - things I feel I can no longer get answered through Google search.
Comparing GPT-3 Completions to Google Search Results
Here is a comparison of some of my most recent personal Google Search queries and their respective responses, I’ve included the GPT-3 responses too:
“what makes artists choose some art over others”
“what does it mean to be histrionic”
“cats love boxes / why do cats love boxes”
In fairness, here’s some of my searches I have omitted which I only looked up on Google:
weather (local area)
looking up email/social profile of a contact
“fire emoji”
“how to join 2 clips in [adobe] premiere [pro]”
looking up restaurant names (actually trying to find their hours of operation)
“tesa® Powerstrips Large”
“share apps between apple accounts”
“how to clean water heater drain”
“intelligence is a gift to be used for the good of mankind”
“kanye west and pharrell at funeral”
“time present and time past eliot”
Why use GPT-3 over Google Search?
Between the two results services, while the responses may be far more simplified for broad questions, I would still prefer the GPT-3 responses and say that they are a breath of fresh air.
Comparing the cognitive load of a google search results page to a simple plain-text answer from GPT-3’s Davinci engine is such a huge difference. I definitely prefer having the information synthesized and explained to me in simple terms. I love that I don’t have to click and review many different links (which are themselves distracting) before getting to a crisp, mental summary understanding like this.
In the past, while Google did incorporate snippets at the top of a results page to answer specific user queries directly … they were actually excerpts from web pages. They were not the kinds of summaries that GPT-3 is synthesizing and rewording on your behalf … saving you so much time and mental work. Since Google search doesn't generate its own content, its responses are limited to ranking the things others have said. I think this approach seriously limits the actual answers people may get back, which could have been answered in a more direct, nuanced, and personalized way, better matching the user’s original search query intent.
Also, I think I’m just a curious person. I have so many questions and areas I’m interested in. I feel like GPT-3 helps me better explore these curiosities as an aid but also as a navigation agent by compressing the vast amount of information that is truly out there in a crisp and human-friendly way.
PageRank Results Listings Page vs. LLM Plain Text Completions
With Google Search, I feel like I’m just looking at an old paradigm. This feel likes an older technology where the onus was on the user to browse various links over many keyword searches, synthesize the information, compare multiple perspectives, summarize all of this in their own heads, and try to make some kind of assessment of their own.
The future feels a lot more like many steps in this process have already been completed for you through language models, liberating you to apply that compressed knowledge understanding how you see fit. You could continue with your day or do more with this information, such as ask follow up questions, brainstorm ideas, or explore the topic in new or deeper ways.
My understanding is that Google, of course, has large language models of their own, but applying them safely globally across Google web search would be a huge technical endeavour. At the same time, I believe this new paradigm of plain-text responses, the kind GPT-3 offers, could rock the boat too much with their core advertising business model which depends on the user spending at least some amount of time on the search results page, across various search queries.
GPT-3 is very far from perfect
I wouldn’t be surprised if OpenAI itself would not advocate for using GPT-3 as a web search engine alternative. GPT-3 has serious safety risks, makes up information, says offensive things, and so much more. Even now, I have to carefully review GPT-3’s answers to my questions very thoroughly, go with my gut about what seems true, and sometimes cross reference GPT-3’s completion answer with Google search results. Believe me, I am aware that this defeats the purpose of using GPT-3 as a Google search alternative, but it is what it is.
Google Search is still a part of my everyday life
It goes without saying, Google Search is still an essential part of my daily information retrieval toolkit, however, nowadays, I would say it serves specific buckets of my daily search volume where I feel it’s a stronger choice.
Looking up exact information
Since GPT-3 has a history of making up facts, sounding truthful but being incorrect, saying offensive things, and also giving inconsistent non-deterministic answers, it makes sense to use Google for verifying stuff. This includes dates, quotations, and other kinds of exact information. Also, since GPT-3 can’t use a web browser (fully yet), I’ll use google - obviously - for exact searches with advanced flags like “site:stackoverflow.com”. I often still follow up some GPT-3 completions I’m not sure about with Google search, just to verify that GPT-3 is not making stuff up.
Looking up current information
GPT-3 was trained in 2019, it is not up to date on the latest news and other events going on in the world. To give you a small example, GPT-3 still thinks Trump is President and barely knows anything about COVID-19. So, I find myself looking stuff up on Google for answers to the latest events (like celebrity gossip) which are in today’s context.
Looking up highly local information
GPT-3 has a lot of embedded information found in its massive neural network, but probably no where near the level of granularity found in Google’s massive search database. It’s not just about having lots of data about local areas and businesses, but understanding how that impacts peoples’ search habits, search query intents, and desired results. Context is key.
Personalized Search Experience
Unlike Google, GPT-3 does not personalize your answers based on your location, demographic information, preferences, interests, or past search history.
Still the best keyword search
Sometimes google is really good, at a typing efficiency level, if you want to enter the least amount of text possible. To be honest, if you entered just two keywords like “local restaurant”, GPT-3 may not know what you are referring to. As a result, it could generate a poem about local restaurants, tell you what that phrase could mean, or list example made-up restaurant names - definitely not what you would be looking for. However, this type of keyword search is super common and an area where Google search excels at. I’d say for super ambiguous and curt user search queries, for the most part, Google does a good job of aligning your search intent with the results you’re looking for.
Searching certain categories of information
Some things like Travel, product searches, image search, currency conversion, and maps are definitely search queries I don’t see GPT-3 being able to answer anytime soon.
General Handiness
Google is just everywhere - so I find myself using it just because it’s a part of Chrome (which is my default browser), Android, and even a part of my daily habits. Sometimes, I’m too lazy to type a website into my address bar so I end up going through Google!
The Significance of InstructGPT
In the past, you needed to work really hard to get GPT-3 to answer questions in a straight forward way like in my earlier example:
… you would need to write a clever GPT-3 prompt, adjust your configuration settings, or even fine tune GPT-3 to get this level of comprehensive and reliable completions.
However, about two weeks back, OpenAI announced InstructGPT, which attempts to align GPT-3’s completions with safer and commercially valuable GPT-3 developer goals. This includes improvements in targeted categories such as summarization, classification, and more. OpenAI has effectively fine tuned GPT-3 to give better answers to questions at a higher quality which most users would expect. InstructGPT demonstrates meaningful strides towards more truthful and accurate completions. Due to its success and ease of use, the InstructGPT series is now the default GPT-3 engine used on the OpenAI platform.
So, what I’m saying is, the tweets/screenshots you saw above were from a single attempt, with the default engine, with no completion examples (ie. zero shot), and basically no changes to the default GPT-3 configurations. The only things I changed were the response length (so I can get a longer answer) and the temperature value to 0.3. This means that, if you’re interested to use it as a search engine alternative, GPT-3 has now become a lot more reliable and a practical alternative as well to do so.
InstructGPT will only continue to improve. At the same time, OpenAI has even released additional research called WebGPT which literally searches the web, browses the links on your behalf, summarizes information, and even provides citation references when answering your query.
So, this is an even greater example of offering a unique approach to information retrieval that search users want which even Google does not currently offer.
Closing thoughts
So, altogether, I’m saying, we’re definitely heading towards a new direction for web based search. I guess I’m just surprised that I’m already using today’s existing GPT technology as a complementary Google search alternative so often. All of this is just a taste of a much greater, new paradigm headed our way. I’m not sure how Google will respond.
Update February 28, 2022
I talked about this article on my podcast in some more detail here: