Map the data before the demo

An impressive AI demo does not tell you what happens to your organization's data. List the information the tool would receive: public documents, internal files, customer records, credentials or regulated data. Start with the least sensitive pilot possible. Ask whether inputs and outputs are stored, for how long, in which region and whether they are used to improve a general model. Get the answers in the service terms or a written security document, not just in a sales conversation.

Identify every connection the tool requests. Does it need read-only access to a folder, permission to edit documents or broad access to an entire workspace? Prefer narrowly scoped permissions and a dedicated test account. Confirm that access can be revoked and that the vendor's subcontractors and integrations are disclosed. If the product acts as an agent, ask which actions require confirmation and whether it can make changes that are difficult to undo.

Test controls you can operate

Check single sign-on, role-based access, audit logs, retention settings and export or deletion options. Ask how an administrator can disable the tool quickly. Test the actual user journey: can an employee accidentally share confidential content with a public workspace? Are generated answers marked when they rely on a restricted document? Is there a documented incident-notification process?

Do not reduce the review to a checkbox about certifications. A certification may cover one product scope or time period, while your use case involves a new feature or integration. Look at the applicable report and ask whether your planned workflow is included. For high-stakes or regulated data, involve the appropriate legal and security professionals.

Pilot with a stop rule

Write down acceptable and unacceptable behavior before launch. Use synthetic or approved sample data to test prompt injection, wrong-document retrieval and overbroad access. Keep a log of errors and who resolved them. Decide what event would stop the pilot immediately, such as unauthorized disclosure or an unapproved external action. A good AI purchase is not merely a capable model. It is a tool whose permissions, limits and failure recovery your team can understand and control.