The Problem With AI Prompt Data: Why Marketers Should Be Skeptical

The Scale Problem: A Drop in the Ocean

Let’s start with some context. WildChat, the largest publicly available dataset of real AI conversations, contains 4.8 million prompts collected over approximately 15 months. Sounds impressive, right? Now compare that to the scale of actual usage. ChatGPT processes over 1 billion queries per day. During those same 15 months, ChatGPT handled an estimated 450 billion queries. WildChat captured 0.001% of them. Now do the math: ChatGPT alone processes over 1 billion queries per day. Over the same 15-month period that WildChat collected data, ChatGPT likely processed approximately 450 billion queries. That means WildChat captured roughly 0.001% of actual ChatGPT usage during that time—and that’s ignoring Claude, Gemini, Perplexity, and dozens of other AI platforms.

Put another way: The dataset represents about 1 in every 100,000 ChatGPT conversations from that period.

Want to explore the dataset yourself? Check out WildVisualizer, an interactive tool for browsing real prompts from the WildChat dataset.

The Self-Selection Bias Problem

Here’s how WildChat collected its data: researchers offered free access to ChatGPT through a Hugging Face-hosted chatbot. In exchange, users had to explicitly consent to having their conversations collected and shared for research.

Source: WildChat: 1M ChatGPT Interaction Logs in the Wild

Who Actually Uses This?

Think about the type of person who:

• Knows what Hugging Face is (already eliminates 99% of consumers)
• Doesn’t have a ChatGPT Plus subscription ($20/month)
• Is willing to trade privacy for free access
• Feels comfortable with their data being collected for research

The Behavior Change Effect

Research shows that people behave differently when they know they’re being observed—a phenomenon called the Hawthorne Effect. Users aware their prompts will be shared for research may:

• Self-censor: Avoid sensitive topics (personal health, finances, relationships)
• Experiment more: Test edge cases rather than solve real problems
• Perform for the dataset: Write “interesting” prompts rather than practical ones
• Avoid proprietary information: Can’t use it for actual work projects

Source: Trust No Bot: Privacy Concerns in WildChat Dataset

What’s Actually Missing: Normal Consumer Behavior

Language and Geographic Bias

WildChat proudly advertises “68+ languages detected” as evidence of diversity. But let’s look at the actual distribution:

Language	Percentage in Dataset	Global Internet Users
English	53%	25.9%
Chinese	13%	19.4%
Russian	12%	2.5%
Spanish	~3%	7.9%
Arabic	~2%	5.2%

Dataset source: WildChat Research Paper | Internet users: Internet World Stats

English is massively overrepresented (2x), while Spanish, Arabic, Hindi, and Portuguese speakers—representing billions of people—are dramatically underrepresented. If you’re marketing globally, this dataset doesn’t reflect your audience.

The Alternative: Browser Extension Data

Some companies claim to have more representative data collected through browser extensions and proxy services. Users install extensions for features like VPNs or SEO tools, and these extensions passively capture AI conversations. Even if ethically collected, browser extension users are still a biased sample:

• Skew younger and more tech-savvy
• Often using free tools (again, price-sensitive)
• Less likely to be in security-conscious corporate environments
• May include more casual or personal use cases vs. professional

So What Should Marketers Do?

This isn’t an argument to ignore prompt data entirely. It’s a call for realistic expectations and proper skepticism.

✓ Good Uses of Prompt Data

• Hypothesis generation: Discover questions or angles you hadn’t considered
• Language patterns: See how people naturally phrase certain types of queries
• Content brainstorming: Identify potential topic areas to explore further
• Supplement other research: Use as one data point among many

✗ Bad Uses of Prompt Data

• Strategic decisions: Basing product roadmap on 0.01% of biased sample
• Assumption of representativeness: Treating it as “what people actually do”
• Competitive intelligence: Competitors’ real users aren’t in these datasets
• Replacing proper market research: It’s no substitute for surveying your actual audience

The Uncomfortable Truth: Only OpenAI Knows

Here’s the reality that nobody in the marketing data business wants to acknowledge: only OpenAI, Anthropic, Google, and the other AI platform providers know what real, representative AI usage looks like. They have:

• 100% of their users’ conversations (not 0.01%)
• Complete demographic and behavioral data
• Enterprise usage patterns alongside consumer behavior
• Paid subscribers alongside free users
• Global representation without self-selection bias

And they’re not sharing it. For good reason—user privacy, competitive advantage, and commercial sensitivity all prevent the release of truly representative data at scale.

A Better Proxy: Search Intent Data

Here’s something most marketers miss while chasing AI prompt data: user intent doesn’t disappear just because the research method changes. People who previously searched Google for “how to fix a leaking faucet” are now asking ChatGPT the same question. The underlying need (fixing their faucet) hasn’t changed. Only the interface has.

Why Google Search Volume Still Matters

While AI is changing how people find information, it’s not fundamentally changing what information they need. Google search data can give you:

• Truly representative scale: Billions of searches with real demographic diversity
• Intent signals: What problems are people trying to solve?
• Trend data: Which topics are growing or declining in interest?
• Seasonality patterns: When do people care about specific topics?
• Geographic distribution: Where is demand concentrated?

The key insight: AI doesn’t eliminate intent—it just changes the expression of it. Someone researching “best CRM for small business” has the same underlying need whether they Google it or ask Claude about it.

Final Thoughts

Public AI prompt datasets are fascinating research artifacts that reveal genuine patterns in how certain types of users interact with AI systems. But they suffer from three fundamental limitations:

1. Scale: They represent a tiny fraction of actual AI usage
2. Selection bias: Only certain types of users contribute to public datasets
3. Data monopoly: Only AI companies themselves have truly representative data, and they’re not sharing it

Until OpenAI, Anthropic, Google, and others release representative samples of their usage data (which seems unlikely for privacy and competitive reasons), marketers must accept that we simply don’t have a complete picture of AI usage at scale. But here’s the good news: You probably don’t need one. The fundamentals of good marketing haven’t change!

Resources & Further Reading