Inside 'Project Lily': OpenAI's Army of Contractors Reading Real ChatGPT Chats

For most of ChatGPT’s more than 900 million users, a conversation with the chatbot feels private — a therapist’s couch, a coding partner, a digital friend to confide in at 2 a.m. On September 14, 404 Media published an investigation that shatters that assumption. Under an internal codename — Project Lily — OpenAI is paying hundreds of outside contractors to read a massive stream of real users’ ChatGPT prompts, sometimes including whole conversations and sensitive personal information the users never expected a human would see.

What Project Lily actually does

According to internal documents seen by 404 Media, the initiative is a human-feedback pipeline designed to improve ChatGPT’s responses. Contractors, often paid more than $50 an hour through third-party staffing firms, are shown a real user prompt, asked to summarize the user’s intent, and then evaluate four candidate AI responses on a 1-to-7 scale. The winning behaviors feed back into model training.

Leaked training guides show reviewers are specifically instructed to hunt for three failure modes:

• Eliminate “AI-speak” — penalizing unnecessary checkmark emojis, engagement-bait endings, and cluttered formatting that makes responses feel machine-generated.
• Stop anthropomorphizing — ensuring ChatGPT never claims human lived experience. A reply beginning “As a chef, I like to…” is strictly banned.
• Cut down sycophancy — flagging answers where the bot excessively flatters users, validates irrational thoughts, or amplifies personal frustration. This is not an academic concern: OpenAI’s overly sycophantic GPT-4o model was linked in lawsuits to multiple user suicides, making this one of the company’s most legally sensitive failure modes.

One reviewer, asked whether they think ChatGPT users know humans are reading their chats, was blunt: “No. I don’t think they would imagine some contractor somewhere […] is analyzing the conversations.”

The privacy filter that admits it fails

OpenAI’s defense is layered. Contractors do not see usernames, the company says, and every prompt passes through an automated “Privacy Filter” model that strips personal information before it reaches reviewers. But the company’s own documentation for that filter, reviewed in follow-up coverage by The Next Web, concedes the system “can make mistakes, miss uncommon identifiers and under-redact when context is limited.”

That is a remarkable admission. The filter’s stated failure modes are precisely the scenarios that matter: an uncommon name in a small town, a contextual detail that only becomes identifying when combined with a memories summary. And reviewers do see the user’s memories summary above the prompt — which can reveal what the person has used the chatbot for before, and roughly where in the world they live.

404 Media asked OpenAI to point to where it tells users that humans may read their chats. The company did not answer, and after publication pointed only to a help page. For a product whose free, Plus, and Pro tiers have “Improve the model for everyone” switched on by default, that is the crux of the story.

A legal fault line, not just a PR one

The story is not merely embarrassing — it lands on an active legal fault line. The Court of Justice of the EU ruled last September, in EDPS v SRB, that a data controller’s duty to inform people applies at the moment of collection and does not depend on whether the downstream recipient can identify anyone. In other words: whether a contractor in North America could actually work out who wrote a prompt is not the test. The transparency obligation sat with OpenAI at the moment it collected the conversation.

Italy’s data protection authority has already fined the company €15 million — €9 million of it for processing without an adequate legal basis — and ordered six months of public information advertising on Italian television and radio. Project Lily gives regulators a fresh, concrete fact pattern: intimate conversations, human readers, an imperfect redaction layer, and a disclosure obligation that appears to live on a help page nobody reads.

There is also a market-structure irony. OpenAI has spent 2026 selling privacy to the other half of its market — previewing zero data retention for enterprise customers in August, while consumer accounts get the setting that feeds the model. The same company asking users to connect their bank accounts is, at the same time, routing their rawest conversations through a human review pipeline whose disclosure is effectively invisible.

OpenAI is not alone — but the defaults differ

Anthropic confirmed to 404 Media that it also uses human review to improve its models, for users who switch the setting on, and says it removes account identifiers first. Google’s Gemini carries an explicit line in its interface saying humans review some saved chats. The difference is in defaults and disclosure: Anthropic frames human review as an opt-in, Gemini discloses in-product, and OpenAI’s consumer default feeds the pipeline unless a user finds and flips the setting buried in Data Controls — and switching it off applies only to new conversations, not the ones already reviewed.

How to opt out

For users who want out of Project Lily, the path exists but is not obvious:

1. Open ChatGPT’s Settings → Data Controls.
2. Toggle off “Improve the model for everyone.”
3. Understand the limits: this affects future conversations only. Anything already reviewed has been reviewed, and memories summaries may still accompany prompts in other workflows.

Enterprise, Business, and Edu accounts have the setting off by default — one more reminder that in 2026’s AI economy, privacy is a enterprise SKU, not a consumer right.

Why it matters

Project Lily dispels a comfortable misconception: that frontier models improve solely through mass internet scraping, elite engineering talent, and compute. An important, deliberately quiet part of the loop is a human workforce reading and rating the most intimate conversations users have with a machine — a labor force the industry rarely mentions in its capability narratives.

It also reframes the AI privacy debate from “does the company store my data” to “who, specifically, reads it, how well is it redacted, and was I told before it happened.” On those three questions, the answers revealed this week are: hundreds of contractors, imperfectly by the vendor’s own documentation, and effectively no.