[study]
We checked 192 products Shopify describes to AI agents
On one day in September we read what Shopify’s catalog tells AI agents about 192 top-ranked products, and checked every claim against the brand’s own words.
The Hearsay team Sep 25, 2026 2 min read
What we did
Shopify summarizes each listing before sharing it with AI agents like ChatGPT, Gemini, Copilot and Meta’s Muse. The summary comes with its own “top features,” “tech specs” and selling points. Merchants can’t see it, so we went and read it.
We ran 20 everyday shopping searches and took the top 10 products Shopify’s catalog returned for each. That came to 192 products after duplicates. For each one we put Shopify’s version next to the brand’s own description, tags and product page, then checked all 3,546 claims.
What AI agents were told
Most claims matched. Enough didn’t that we stopped calling them edge cases.
14%
of 192 top-ranked products were described to AI agents with a claim that directly contradicts the brand, on health, safety or what’s in it. For baby products: 1 in 5.
89%
had at least one claim the brand never made.
70%
had one about health, safety, certification or what the product does.
How we checked: On Sep 24, 2026 we took the top 10 products Shopify’s catalog returned to AI agents for 20 everyday searches (192 products). AI reviewers checked all 3,546 claims against each brand’s own copy: description, tags and product page. Every example was verified by hand. “Serious” covers health, safety, certification and what a product does. This is one day and the top results only, and listings change. Full method
A contradiction here means Shopify’s summary said the opposite of what the brand said, on something a shopper would care about: who it’s for, whether it’s safe, or what’s in it. A claim the brand never made is anything in the summary we couldn’t find anywhere in the brand’s own copy.
“Essential oil-free and tallow-free formula”
The receipts
Our reviewers flagged every example below, then we checked each one by hand against the brand’s own page. We left the brands’ names out. The products are real.
The brand says
“Tallow from grass-fed cows, naturally-mined baking soda … and coconut oil”
AI agents were told
“Essential oil-free and [tallow-free] formula”
The brand says
“Ages 4+ can take 1 gummy per day … If not chewed properly, this product is a potential choking hazard.”
AI agents were told
“…multivitamin gummies [for kids ages 2 and up]”
The brand’s note removing dosing for ages 2–3 was sitting inside Shopify’s own record.
The brand says
Its FAQ says it “has not been tested for noncomedogenicity.”
AI agents were told
“Lightweight, [non-comedogenic], oil-free, and sulfate-free formula”
The brand says
“Sodium Benzoate: Food–grade preservative. Sodium Dehydroacetate: Organic salt preservative.”
AI agents were told
“Free from: synthetic fragrances … and [preservatives]”
The brand says
“meant for skin of all ages (under 6 months consult your pediatrician)”
AI agents were told
“[Pregnancy safe and suitable for use from birth]”
[also found]
Two [fragrance-free] products whose own ingredient lists include fragrance or essential oils.
How we checked: On Sep 24, 2026 we took the top 10 products Shopify’s catalog returned to AI agents for 20 everyday searches (192 products). AI reviewers checked all 3,546 claims against each brand’s own copy: description, tags and product page. Every example was verified by hand. “Serious” covers health, safety, certification and what a product does. This is one day and the top results only, and listings change. Full method
What this can’t tell you
- It’s one day. We read the catalog on Sep 24, 2026. Listings change, and so do Shopify’s summaries.
- It’s the top results only. Ten products per search, twenty searches. Products further down the list weren’t checked.
- It’s what agents were told, not what they said. We read Shopify’s catalog. Each agent may use that data a little differently.
- AI reviewers did the first pass. Every example on this page was then checked by hand. The percentages come from the first pass.
If you sell on Shopify
Nobody outside Shopify can edit its summary, including us. What you can do is see it, compare it with your own copy, and make your own listing hard to misread. That’s what Hearsay is for.