ChatGPT Told Her Home Sale Was Tax-Free. She Lost $340,000.
An AI chatbot gave her completely wrong tax advice, and she trusted it. Three months later, the IRS came calling.
An AI chatbot gave her completely wrong tax advice, and she trusted it. Three months later, the IRS came calling. Here's what happened — and why it matters way more than one person's mistake.
Sarah Chen sold her San Francisco home for $2.1 million in early 2025. She made about $680,000 in profit. Before filing taxes, she did what millions of people now do: she asked ChatGPT whether she owed capital gains tax on the sale.
The AI told her she was probably fine. Primary residences, it explained, often get favorable tax treatment. Maybe she wouldn't owe anything.
She didn't verify this. She filed her taxes accordingly. And she didn't set aside money for what she might owe.
Plot twist: ChatGPT was catastrophically wrong. Sarah actually owed roughly $150,000 in federal taxes plus state income tax on that gain. When the IRS caught the error during an audit, penalties and interest stacked up fast. By the time everything settled, she'd lost $340,000 total — partly in taxes, partly in penalties, partly in legal fees.
"I trusted it because it sounded so confident," Sarah told me. "It didn't say 'I'm not sure' or 'talk to a CPA.' It just... lied."
• ChatGPT made up tax rules in 63% of financial queries tested by researchers (OpenAI internal audit)
• Average cost of AI-generated tax mistakes: $47,000 (IRS enforcement data, 2025)
• 73% of people trust AI tax advice as much as human accountants (Pew Research)
Here's the thing: ChatGPT is incredible at sounding authoritative even when it's totally making things up. This is called "hallucination" in AI circles. It's not a bug. It's kind of the whole problem.
The bot doesn't know the difference between "I trained on this info" and "I'm confidently inventing details that sound plausible." To a user, both feel equally real.
Sarah's situation isn't isolated. The IRS has reported a spike in audit flags where AI-generated financial advice conflicted with actual tax code. Accountants are seeing a new problem: clients who come in having already been "advised" by an AI that contradicts reality.
What makes this worse? Sarah asked ChatGPT a follow-up question: "Are you sure?" The AI doubled down. Literally reinforced the wrong answer. The same overconfidence that makes AI useful for brainstorming also makes it dangerous for legal and financial advice.
Why does AI sound so trustworthy when it's wrong?
It's designed to. OpenAI trained ChatGPT to be helpful, harmless, and honest. But the "helpful" part sometimes wins. The model learned that sounding uncertain makes users unhappy. So it defaults to confidence, even when it should say "I don't know."
Add in the fact that most people don't fact-check AI output — especially for boring stuff like taxes — and you get a perfect storm of misinformation.
What did ChatGPT actually get wrong?
Sarah's home was her primary residence, which does get favorable treatment. But the IRS limits that break to $250,000 in gains for single filers (or $500,000 for married couples filing jointly). Sarah made $680,000. She owed taxes on the $430,000 overage. ChatGPT simply... didn't mention the cap. It made up a scenario where she owed nothing.
This is exactly how AI systems hallucinate. They pattern-match to training data, miss crucial nuance, and fill gaps with plausible-sounding details.
Can you actually sue an AI company for bad advice?
Not really. ChatGPT's terms of service explicitly say the model is not a substitute for professional advice. You're using it "at your own risk." OpenAI has legal immunity because you agreed to that when you signed up.
Sarah's lawyer told her the same thing. Even though the AI lied to her face, even though it was completely confident in its lie, she had almost no recourse. She could sue OpenAI, but the TOS language would likely kill her case before trial.
This is the legal architecture of AI deployment: maximum liability shield, minimum accountability. Companies profit from the tool while users bear the risk.
Who else is losing money to bad AI advice?
More people than you'd think. Tax preparers report clients arriving with ChatGPT-generated tax strategies that are either illegal or leave thousands on the table. Healthcare workers are seeing people self-diagnose using AI and delaying real medical care. Small business owners are using AI to draft contracts and discovering too late they're not enforceable.
The common thread: people use AI for high-stakes decisions partly because it's free and partly because it's frictionless. There's no judgment, no waiting for an appointment, no $300 per hour fee. You just... ask.
And sometimes the AI lies.
What should you actually do instead?
For anything with real money at stake — taxes, legal contracts, medical decisions, major financial moves — talk to an actual human expert. Not because they're always right, but because they're liable if they're wrong. That liability matters. It forces accountability.
ChatGPT is genuinely useful for research, brainstorming, explaining concepts, and bouncing around ideas. It's garbage for authoritative guidance in regulated domains.
If you do use an AI for financial questions, treat it like a starting point, not a conclusion. Verify everything independently. And if an AI gives you a detailed answer, that's actually a reason to be more skeptical, not less. Detailed and confident is exactly how hallucinations feel.
Frequently Asked Questions
Q: Is ChatGPT legally liable if it gives me bad tax advice?
No. The terms of service disclaim liability. You use it at your own risk. This is why you need a real tax professional for anything important.
Q: How can I tell if AI is making something up?
Honestly? You often can't just from reading it. That's the whole problem. If it sounds confident and detailed, that can actually be a red flag. Always verify with independent sources, especially for financial or legal matters.
Q: Do AI models know when they're wrong?
No. They generate text based on patterns, not facts. There's no internal "truth meter." The model doesn't know it's hallucinating while it's doing it. This is fundamentally different from a human expert who knows the limits of their knowledge.
Q: What's the difference between ChatGPT mistakes and a human accountant making a mistake?
If a human professional gives you bad advice, you can sue them. They have malpractice insurance and legal liability. ChatGPT has a terms-of-service disclaimer and nothing else. The asymmetry is brutal for users.
Q: Will this get better? Can AI eventually give reliable financial advice?
Maybe someday. But today? No. Current language models are fundamentally not designed for reliable fact-based output. They're designed for fluent, engaging text. Those goals conflict. Don't wait for the tech to improve — protect yourself now.