This is for a founder in the sales-call stage with one or more AI development agencies, trying to tell a strong team from a confident pitch. Below are 24 questions grouped into six areas, each with what a good answer sounds like and what a weak one sounds like, so you can score the call in real time.
What questions should I ask an AI agency before hiring them?
Ask questions that require a specific answer, not a confident one. "Have you built something like this before?" invites a yes regardless of truth. "Walk me through the last AI project that didn't go to plan, and what you changed" forces a real story. The groups below cover scoping, technical depth, team and process, cost, ownership, and proof — ask at least two or three from each group in your first real conversation.
Scoping and process questions
- "How do you scope a project before quoting a price?" Good answer: a paid or structured discovery phase that produces a written spec before a fixed quote. Weak answer: "We can give you a ballpark on this call."
- "What does a typical week of working with you look like?" Good answer: specific cadence — daily written updates, weekly demos, a shared project board. Weak answer: "We'll keep you in the loop."
- "How do you handle it when the AI doesn't work as well as expected on our real data?" Good answer: describes an evaluation step early in the build that catches this before full development. Weak answer: "That usually isn't a problem for us."
- "What's the single biggest risk in a project like ours?" Good answer: something specific to your use case (data quality, latency, a tricky integration). Weak answer: a generic "timelines" or "budget" answer that applies to any project.
Technical depth questions
- "Why would you use RAG instead of fine-tuning for this, or vice versa?" Good answer: a reasoned trade-off tied to your data and update frequency. See our own breakdown in RAG vs. fine-tuning as a benchmark for the kind of answer to expect.
- "Which models would you use, and why?" Good answer: names specific models (GPT, Claude, Gemini, open-source options) and a reason tied to cost, latency or capability, not just "the best one."
- "How do you prevent the AI from making things up?" Good answer: mentions grounding responses in real data, evaluation testing, or guardrails — not "we prompt it carefully." See preventing AI hallucinations.
- "How will we know if the AI is actually working well, beyond 'it feels right'?" Good answer: describes a concrete evaluation method with metrics, not vibes. See how to evaluate LLM applications.
- "What happens to this feature if the model provider changes pricing or deprecates the model we're using?" Good answer: describes an architecture that isn't hard-locked to one provider. Weak answer: hasn't thought about it.
Team and delivery questions
- "Who specifically will be writing the code on our project?" Good answer: names the person or role, and confirms it's the person(s) on this call or one you'll meet. Weak answer: "our team" with no further detail.
- "Will the same people who scope this project also build it?" Good answer: yes, or a clear explanation of the handoff. Weak answer: sales team scopes, unnamed delivery team builds.
- "What happens if our main point of contact leaves mid-project?" Good answer: describes documentation and backup coverage. Weak answer: hasn't considered it.
- "Do you subcontract any part of the work?" Good answer: transparent yes/no, and if yes, explains how confidentiality and quality are maintained. A surprising "no, never heard of that happening here" is itself a flag if their size suggests otherwise.
Cost and contract questions
- "Is the model/API usage cost included in your quote, or billed separately?" Good answer: billed separately, ideally directly to your own accounts so you see real-time usage. Weak answer: bundled with no visibility into the breakdown.
- "What's your policy on scope changes once we're underway?" Good answer: a named process and rough rate for change requests. Weak answer: "we'll figure it out."
- "Is this fixed price or time and materials, and why?" Good answer: a reasoned recommendation for your specific project type — see fixed price vs. time and materials. Weak answer: whatever is easiest for them to invoice.
- "What's included in the price after launch — bug fixes, support, monitoring?" Good answer: a named support window (e.g. 30 days) and what happens after it ends. Weak answer: no mention of post-launch at all.
Why does the order you ask these questions matter?
Start with scoping and technical depth before cost and ownership. A vendor's answers to "how do you scope a project" and "why would you choose RAG over fine-tuning" tell you whether they actually know what they're doing; a vendor's answers to cost and ownership questions tell you whether they're being straight with you about terms. If you lead with price, you'll anchor the whole conversation on budget before you've established whether the team is even competent to deliver, which makes it harder to walk away from a cheap but weak offer later. Technical and process questions first, commercial questions second, is the more useful sequence in practice.
What follow-up questions should I ask after a vague answer?
Don't let a vague answer pass — ask it a second way. If "how do you prevent hallucinations" gets "we test thoroughly," follow with "walk me through what that testing actually looks like, step by step, on your last project." If "who writes the code" gets "our team," follow with "which specific person will be in our weekly calls, and is that the person doing the actual engineering." Specific agencies answer specific follow-ups easily because they're describing something real; agencies reciting a sales script tend to repeat the same vague phrasing a second time, which is itself useful information.
Ownership and security questions
- "Who owns the code, prompts and any fine-tuned models at the end of the project?" Good answer: explicit, unambiguous: you do, in writing. See who owns the IP in AI development.
- "Will the code live in our repository from day one, or get handed over at the end?" Good answer: your repository from day one. Weak answer: "we'll transfer it when the project's done" — a real risk if the relationship sours mid-project.
- "What access will you need to our systems, and how is that access removed afterward?" Good answer: least-privilege access with a clear offboarding step. Weak answer: hasn't thought about offboarding.
- "Will you sign an NDA before we share any proprietary data?" Good answer: yes, readily, often before the first detailed call. Weak answer: hesitation or "that's not usually necessary."
Proof and reference questions
- "Can you show me a live product you've built that's similar to ours?" Good answer: a working demo, named client (with permission), and a specific technical detail about how it was built. Weak answer: screenshots only, or a case study with no live link.
- "Can I speak to a past client with a budget similar to mine?" Good answer: yes, promptly. Weak answer: delay, deflection, or only offering their single biggest-name client.
- "What's a project you turned down or walked away from, and why?" Good answer: a specific example, which shows they have standards and say no to bad-fit work. Weak answer: "we try to make every project work," which usually means they take on work they're not suited for.
What do the best agencies have in common when you ask these questions?
They answer quickly, with specifics, and they're comfortable saying "we wouldn't recommend that" when a question touches something they'd actually push back on with you. The strongest signal in a vetting call usually isn't a single great answer — it's a pattern across several questions of genuine, specific knowledge rather than reassurance. A team that says "actually, for your use case, fine-tuning is probably overkill, RAG will get you most of the value for a fraction of the cost" is telling you something true even when it's not what you expected to hear, and that's a far better signal than an agency that agrees with every assumption you walk in with.
How many of these questions should I actually ask?
You don't need all 23 in one call. Pick two or three from each of the six groups — roughly 12–15 questions — for your first real conversation, and save the rest for a follow-up once you're seriously considering the vendor. The goal isn't an interrogation; it's enough specific questions, across enough categories, that a weak agency runs out of specific answers before you run out of questions.
What if an agency can't answer some of these well?
Not every gap is disqualifying. A newer agency may not have a dozen live case studies yet, and that's a reasonable thing to weigh against their technical answers and references rather than treat as an automatic no. What should concern you more is vagueness on ownership, cost transparency, and who's actually doing the work — those are not things that improve with agency size or age, they're things a well-run agency gets right from day one regardless of how new it is. For the full checklist beyond just questions, see how to choose an AI development agency.
How we approach this at Venzpire
We'd rather field all 24 of these than have you find the gaps after signing. On ownership: your code lives in your repository from day one, and you own everything — code, prompts, fine-tuned models — outright. On cost: AI usage is billed to your own accounts at cost, separate from our fees. On team: the people who scope the $1,500 discovery sprint are the people who build, with weekly demos and daily written updates throughout. See our case studies at work or bring your own list of questions to apply.
Building something like this?
I read every application myself and reply within one business day, even when we’re not the right fit.