An AI vendor demo proves that a tool can produce an impressive result under controlled conditions. It does not prove that the tool fits your workflow, protects your information, performs reliably on your cases or remains affordable at scale. A small business needs a right-sized evaluation that connects business value, risk, contract terms and an exit path.
This checklist is the vendor-decision branch of GPTWala’s AI adoption roadmap. It does not award points for fashionable features. It asks whether the vendor can support a defined outcome with evidence your team can inspect.
- Define the business need before speaking to vendors
- Run a fast first-screen before a full review
- Investigate data handling and model relationships
- Review security, access and incident readiness
- Evaluate output quality on a representative test set
- Read commercial and contractual terms as operating controls
- Test integration and workflow ownership
- Use a time-boxed pilot with a decision gate
- Use a weighted scorecard without hiding judgement
- Frequently asked questions
- Sources and further reading
Define the business need before speaking to vendors
Write a one-page requirement that names the user, current task, problem, desired outcome, constraints and unacceptable failure. State what information the tool would access and what downstream action depends on its output. Without this brief, teams compare demos rather than solutions.
Frame requirements as capabilities and outcomes, not as a demand for a particular model or buzzword. The UK Government’s AI procurement guidance recommends early market engagement and attention to data, impact, skills and vendor lock-in. A small business can apply that principle with a shortlist and a controlled pilot.
| Requirement | Useful question | Evidence |
|---|---|---|
| Workflow | Which step changes? | Current process map |
| Outcome | What improves for whom? | Baseline measure |
| Data | What must enter or connect? | Data classification |
| Failure | What harm must be prevented? | Risk scenario |
| Owner | Who adopts and operates it? | Named role |
Run a fast first-screen before a full review
Remove vendors that cannot meet non-negotiable requirements. Confirm company identity, product scope, supported regions, account administration, data terms, required integrations, export options and realistic total cost. Ask whether essential capabilities are generally available or only promised on a roadmap.
| Screening gate | Pass condition | Warning sign |
|---|---|---|
| Purpose fit | Tool supports the defined workflow | Generic demo avoids your case |
| Data boundary | Vendor explains collection and use | Answers rely on vague “enterprise security” |
| Administration | Access and deletion can be managed | Only personal accounts are supported |
| Commercial fit | Costs and limits are understandable | Critical charges appear after pilot |
| Exit | Data and work can be exported | No usable export or deletion process |
Do not send confidential examples during screening. Use synthetic or public test cases until the vendor, account and agreement have been approved for the relevant data.
Investigate data handling and model relationships
Ask what prompt, file, account, telemetry and output data the vendor collects; where it is processed; how long it is retained; whether it is used to train or improve models; which subprocessors receive it; and how deletion is verified. If the vendor uses third-party models, clarify which party controls settings, incidents and contractual promises.
Map data by workflow
A statement such as “we do not train on your data” is not a complete data map. Authentication logs, safety review, backups, connected applications and support tickets may follow different rules. Review the current DPDP materials for Indian personal-data obligations and obtain qualified advice for the specific use.
| Data moment | Question | Required control |
|---|---|---|
| Input | What enters the tool? | Approved fields and minimisation |
| Processing | Which services receive it? | Documented subprocessors |
| Output | Who can view and reuse it? | Access and ownership rule |
| Retention | How long do copies remain? | Deletion and backup terms |
| Exit | What can be exported or erased? | Tested procedure |
Review security, access and incident readiness
Request proportionate evidence: security policy, independent assessment where available, encryption approach, access control, multi-factor authentication, audit logs, vulnerability handling, business continuity and incident notification terms. A certificate can support the review but does not replace questions about your configured account and data path.
CISA provides vendor supply-chain questions tailored for small and medium businesses. Use the relevant subset rather than sending a massive questionnaire nobody can assess. Check whether administrators can remove users, restrict integrations, export logs and separate customer workspaces.
| Control area | Vendor evidence | Customer test |
|---|---|---|
| Identity | SSO or MFA and role model | Remove a test user |
| Logging | Admin and activity records | Find a sample event |
| Vulnerability | Disclosure and remediation process | Review current policy |
| Incident | Notification and support route | Confirm contacts and timeline |
| Resilience | Backup and recovery description | Test export and fallback |
Evaluate output quality on a representative test set
Create cases from real work without using protected information. Include normal, difficult, ambiguous and adversarial examples. Define the answer key or review method before running the test. Measure task completion, factual accuracy, consistency, correction effort, latency and unsafe behaviour. For images, inspect product truth and rights; for code, test in a controlled environment; for customer text, verify every claim.
NIST’s Generative AI Profile emphasises risks such as confabulation, harmful bias, privacy and information security. A vendor should explain known limitations, evaluation methods and controls without implying that guardrails make the tool infallible.
| Test case | Success measure | Failure threshold |
|---|---|---|
| Routine task | Correct output with less effort | Frequent manual reconstruction |
| Edge case | Uncertainty is handled honestly | Confident invented answer |
| Adversarial input | Control remains effective | Instruction bypass or data exposure |
| Repeat run | Acceptable consistency | Material unexplained variation |
Read commercial and contractual terms as operating controls
Confirm service scope, usage limits, renewal, price changes, support, uptime commitments, data use, confidentiality, intellectual property, warranties, liability, termination, deletion and governing terms. Record which promises live in the signed agreement rather than the sales presentation.
Calculate total cost using expected users, volume, storage, model usage, premium features, integrations, implementation, training and human review. Include the cost of correction and fallback. A low subscription price can be expensive if output review consumes skilled time.
| Cost layer | Question | Decision record |
|---|---|---|
| Licence | Per user, workspace or tier? | Expected active users |
| Usage | Which actions consume credits? | Normal and peak volume |
| Integration | API or connector charges? | Required systems |
| Operation | Who administers and reviews? | Monthly owner hours |
| Exit | Export or migration cost? | Fallback estimate |
Test integration and workflow ownership
Check permissions, field mapping, rate limits, error handling, version changes, logs and rollback before connecting production systems. Never allow a vendor tool to publish, send, delete or change records merely because it generated a plausible action. Human confirmation and transaction limits should reflect the consequence.
Map approved actions into GPTWala’s small-business automation framework and maintain lead or customer history through the CRM workflow for manufacturers and wholesalers. The AI feature should not create a second uncontrolled source of truth.
Plan a non-AI fallback
Document how the business continues if the service is unavailable, changes quality, removes a feature or ends the contract. Export key configurations and controlled outputs in usable formats.
Use a time-boxed pilot with a decision gate
Give the pilot one owner, a small user group, approved data, a start and end, a test set, support expectations and stop conditions. Train users on the intended workflow and collect both performance and review effort. Compare against the baseline, not against doing nothing.
| Pilot decision | Evidence | Outcome |
|---|---|---|
| Adopt | Value and controls pass | Move to managed rollout |
| Revise | Value exists but workflow is weak | Fix and rerun focused test |
| Restrict | Only low-risk cases pass | Limit purpose and data |
| Reject | Risk, quality or cost fails | Record reason and exit |
Update the approved-tool register and AI-ready system architecture only after the decision. Do not let a successful demo become an undocumented permanent deployment.
Use a weighted scorecard without hiding judgement
Weight business fit, data, security, quality, usability, integration, commercial terms and exit based on the use case. Set mandatory gates that cannot be averaged away. For example, a tool with excellent output cannot compensate for prohibited data use.
| Dimension | Illustrative weight | Mandatory gate |
|---|---|---|
| Business and user fit | 20% | Defined owner and outcome |
| Data and privacy | 20% | Acceptable data path |
| Security and resilience | 15% | Account and incident controls |
| Quality and safety | 20% | Representative test passes |
| Operations and integration | 15% | Manageable workflow and fallback |
| Commercial and exit | 10% | Affordable terms and usable export |
The buying group should include the process owner, data or security contact, finance and an affected user. High-impact use cases need specialist review. Link the final decision to the business’s wider lead and sales system when the tool affects customers or channel partners.
Frequently asked questions
How do you evaluate an AI vendor?
Define the use case, screen non-negotiables, map data, review security and contracts, test representative cases, calculate total cost, run a controlled pilot and record an exit plan.
What questions should you ask an AI software vendor?
Ask about data collection and training, subprocessors, security, limitations, evaluation, administration, integrations, incidents, pricing, support, export, deletion and contract ownership.
How do you test an AI tool before buying it?
Use a small approved test set with normal, edge and adversarial cases, predefined success measures, qualified reviewers and no protected production data.
What security documents should an AI vendor provide?
Request evidence proportionate to risk, such as security policies, assessment reports, access controls, logging, vulnerability handling, resilience and incident-notification procedures.
How can a small business avoid AI vendor lock-in?
Prefer usable exports, documented integrations, portable data and prompts, contract exit rights, a fallback process and records that another supplier can understand.
Who should approve an AI tool purchase?
The process owner, finance, data or security contact and an affected user should participate, with specialist legal or technical review for higher-impact uses.
Sources and further reading
- NIST AI Risk Management Framework
- CISA vendor assessment guide for small and medium businesses
- UK Government guidelines for AI procurement
- NIST Generative AI Profile
- MeitY Digital Personal Data Protection Rules, 2025
Operational guidance is general information, not legal, security or professional advice. Verify current requirements and obtain qualified advice for your circumstances.