The Uncomfortable Questions#
Metrics are great, but most AI projects fail because nobody asked about the workflow.
If we build this AI and it works perfectly, what actually changes? Who clicks the button? If the client can’t answer, you don’t have a project. You have a science experiment.
You have to ask the awkward questions. Back to our law firm example:
- Who looks at the answer? A paralegal? Or the client directly?
- What if it’s wrong? Does the client get mad, or does a lawyer catch it?
- Do you have time? Is there actually time in the day to review the AI’s work?
- Can we fix it? If the bot hallucinates, how do we correct the record?
Clients get defensive when you ask this. That’s good. It means you’re hitting the real problems.
Your job is to make the tech invisible. We didn’t sell them “RAG Architecture.” We sold them “Answers based on your actual documents.” We didn’t sell “Conversational Agents.” We sold “Asking clients facts instead of making them fill out forms.”
How to Run a PoC That Isn’t a Waste of Time#
The Proof of Concept (PoC) is where you either look like a genius or a fraud. The difference is Scope.
Narrow scope is everything. Every successful PoC I’ve seen does one thing for one team. Every failure tried to do everything for everyone.
For the law firm, we identified five ideas. We built one: The Q&A bot. And we only used it on 50 past cases.
- No Intake Agent.
- No Client Portal.
- Just the Q&A bot.
The “Production Mindset”#
A good PoC is an experiment, not a demo.
- Real Data: Dirty, messy, scanned PDFs. Not clean text files.
- Real Integration: It has to live where they work, not in a separate tab.
- Real Users: Paralegals, not the CTO.
For the law firm, we used their actual messy files. Handwritten notes, coffee stains, the works. If it can’t handle the mess, it won’t work in production.
Rules for Survival#
1. Data Quality > Model Smarts I’ve seen more projects die because the data was missing than because the model wasn’t smart enough. If the data is bad, GPT-5 can’t save you.
2. Pick the Right Tool
- Scikit-learn : If it’s numbers in a spreadsheet.
- Deep Learning : If it’s pictures or text.
- RAG : If it needs to know facts. Don’t use a chainsaw to cut butter.
3. Time-Box It 2 to 6 weeks. That’s it. If you can’t prove value in 6 weeks, the project is too big. Kill it and start smaller.
4. Get the Business Involved If only IT sees the PoC, it failed. You need the actual users testing it. For the law firm, three junior lawyers used the bot every day. Their feedback (“We need a confidence score or we won’t trust it”) saved the project.
The Deliverable is Clarity#
At the end of a PoC, you aren’t just delivering code. You are delivering Truth.
Sometimes the truth is “This won’t work.” And honestly? Clients love that. If you save them 200k by killing a bad project early, they will trust you forever.
Our law firm PoC showed a 30-50% drop in research time. But more importantly, it proved that lawyers would use it—if we added citations. That insight was worth more than the code itself.