Web search on vs. off
What we actually can compare, and what we can't
This page was going to compare OpenAI with and without web search. While analyzing the data we found that, in this cut, only 2 of the 88 no-search calls returned capturable text: not enough of a sample for a fair comparison. We're reporting it anyway, because it's a real finding about capture reliability, not about the assistant's behavior.
Summary
With web search on, all 57 calls in this cut returned capturable text: 219 mentions, 106 distinct restaurants, and cited URLs in 46 of the 57 responses (81%). Without search, 86 of 88 calls have no capturable text due to the capture gap documented in the time-slot analysis; the 2 remaining usable calls aren't a sufficient base for any comparison.
What happens with web search on
With the search tool enabled, the assistant cites real sources in most of its responses: TripAdvisor, TheFork, local guides like Denia.com or VisitDenia.com, and several restaurants' own websites.
81% of with-search responses include at least one cited URL, versus 0% of this cut's without-search responses (though again, with only 2 usable without-search calls, that specific comparison should be read with a lot of caution).
Real citation examples with search on
These are real query excerpts and the URLs the assistant cited when answering them.
"I'm traveling with young kids, which fresh seafood spots in Dénia are good?"
"Looking for a place with a terrace and sea views for pizza or Italian food"
Why the without-search comparison isn't reliable in this cut
Of the 88 calls to OpenAI without web search, 86 have null response text: the run completed and used tokens, but the text couldn't be recovered. It's a capture-pipeline gap, not a signal that the assistant recommended nothing; we cover it in more detail in the time-slot analysis.
The 2 calls that did return text mentioned 7 restaurants in total (La Catrina Cantina Mexicana, Carrer Loreto, Tasca Eulàlia, Els Tomassets, Ca Pepa Teresa, Tasca Ramonet, and Plaça del Convent), with no cited URLs, as expected without the search tool enabled. That's an anecdotal sample, not a basis for generalizing.
What we can't claim from this
- We can't reliably compare with-search and without-search responses in this specific cut: the without-search sample is too small.
- "Cited URL" doesn't equal "verified information": we haven't checked whether each cited page actually confirms the claim next to it.
- We don't use "hallucination" as a generic accusation: with a sample this small without search, we couldn't document concrete examples of incorrect claims, so we don't use the term here.
- A future cut with more without-search calls (and without the capture gap) would allow a real comparison.
Keep exploring the dataset
This is one of six dimensions we analyze. The rest of the variables, and the full dataset to explore on your own, are on the data page.