A Trail Guide to choosing AI tools without getting lost

There is more AI tooling available to your organization right now than you could evaluate in a year of full-time work. Some of it is genuinely excellent. A great deal of it is a thin wrapper with a confident website. Telling them apart is not actually that hard, but it requires asking different questions than the ones the demo is designed to answer.

Six questions. In order. The last one is the one people forget.

1. What specific task does this remove, and who currently does it?

If you cannot name the task and the person, you are not solving a problem — you are acquiring a capability and hoping a problem shows up for it. That happens more than anyone admits.

Good answer: “It drafts the visit note, which currently takes each clinician about ninety minutes a week after hours.” Bad answer: “It helps with documentation.”

2. What happens when it’s wrong?

Not if. Every one of these tools is wrong sometimes. The question is what the failure costs and who catches it.

A tool that drafts an internal summary and gets a detail wrong costs you thirty seconds of correction. A tool that touches billing, clinical records, or anything a regulator reads has a very different failure profile. Both can be worth using — but the second one needs a human review step designed in from day one, and you need to be honest that the review step eats some of the time savings.

The right question isn’t “how accurate is it?” It’s “what does our process do with the inaccurate ones?”

3. Where does our data go, and can we get it back?

Three sub-questions, and vendors answer them very differently:

• Is our data used to train their models? Get this in writing, not in a sales call.
• If you handle protected health information, financial records or anything similar — will they sign the agreement your compliance obligations require? A vendor who hesitates here has told you something important.
• If you leave in eighteen months, what can you export, in what format, and how much of your accumulated work walks out the door with them?

4. Who on our team owns this after the pilot?

Every tool needs an owner — someone who fields the questions, notices when it stops working, and decides at renewal whether it’s still earning its place. If that person is “whoever has time,” the tool will quietly rot for eleven months and then get renewed automatically because nobody wanted to be the one to kill it.

Name the owner before you buy. If nobody has capacity to own it, that is a real answer, and it means not now.

5. Can we test it on something that doesn’t matter yet?

Find the lowest-stakes version of the real task. Run it there for two to four weeks with two or three people who are willing to tell you the truth — and specifically pick at least one skeptic. Enthusiasts will tell you it’s great. Skeptics will tell you where it breaks, which is the information you are actually buying.

Set the success criteria before the pilot starts, in writing, with a number. “Cuts documentation time by at least a third without increasing correction requests” is a criterion. “See how people like it” is a vibe, and vibes always come back positive at the end of a pilot everyone knew was a pilot.

6. What does this cost at three times our current usage?

Here is the one people miss until it is too late.

Per-seat pricing looks reasonable at twelve seats and different at forty. Usage-based pricing looks free during a pilot and becomes a line item nobody budgeted for once the tool actually gets adopted — which is to say, once it works. Ask for the number at your realistic eighteen-month scale, and ask what the renewal increase has looked like for existing customers.

Then decide whether it’s worth it at that number, not at the pilot number. If the answer is no, you have just saved yourself a painful conversation next spring.

---

The short version

1. Name the task and the person.
2. Design for being wrong.
3. Know where the data goes and how it comes back.
4. Name an owner before you buy.
5. Pilot small, with a skeptic and a number.
6. Price it at three times today.

Run these six and you will say no to most things, which is the correct outcome. The tools that survive all six tend to be the ones still in use two years later — and those are the only ones that ever mattered.