Deployment Labs
Deployment NotesAI governance

AI Data Security: What Happens to Company Data in the AI Tools Staff Use

Deployment LabsUpdated 12 min read

AI data security used to mean stopping staff from pasting secrets into a chatbot nobody had approved. That's no longer where most of the exposure is. In a September 2026 survey of 400 security leaders at US and UK companies with more than 500 employees, 47% said sensitive data had been exposed through an AI tool at their organization, and 77% of those said a company-approved tool was involved. The tools you bought hold your data too, for as long as their settings say, with whatever access you gave them.

What is AI data security?

AI data security is the set of rules, plan choices and settings that decide what company data goes into AI tools, what the vendor may do with it, how long it's kept, and who can reach it through the tool. For most companies of 20 to 1,000 people, that comes down to four tools and three decisions.

This guide is for the people who own those decisions: operations, IT and security leads, and the executive who owns the AI rollout. It covers the workplace plans of ChatGPT, Claude, Microsoft 365 Copilot and Gemini as their vendors describe them in November 2026. It isn't a security product review, and it isn't legal advice.

Where does company data go when staff use AI?

Company data reaches AI tools four ways: staff type or paste it, upload files, connect the tool to company systems, or let an agent act on their behalf. Each path has its own retention, access and training terms, and the last two can move far more data than anyone types.

  • Prompts and pasted text. The obvious path. A contract clause, a customer complaint, a draft board memo.
  • Uploaded files. A spreadsheet of salaries, a client's tax return, a set of interview notes. Files often carry far more than the question needed.
  • Connectors. When an assistant reads email, documents or a CRM, it can pull data no one typed. OpenAI's admin docs say connected apps "use the permissions of the authorized account," and that connected services "have their own retention, logging, access, residency, and compliance requirements."
  • Agents. An agent that books, files or sends things works across systems with a set of permissions. In the same September 2026 survey, 56% of security leaders said autonomous AI agents acting for employees fall outside their existing controls.

Then the data stays somewhere: in chat history, in the vendor's logs, in the workspace's files. That's where retention comes in, and it's the part most companies never set.

Does ChatGPT store your data, and do AI tools train on it?

Yes, ChatGPT stores your data: business workspaces keep chats and files indefinitely unless an Enterprise retention policy says otherwise. On business plans, ChatGPT, Claude, Copilot and Gemini all say they don't train on your data by default. On personal accounts, the employee's own settings decide.

That last line is the one that matters. The training question is settled by the plan, not the brand. Here's what each vendor says in November 2026:

ChatGPT Claude Microsoft 365 Copilot Gemini in Workspace
Trains on business-plan data by default? No (Business, Enterprise, Edu) No (commercial products) No (prompts, responses and Graph data) No, without the customer's permission
Personal accounts User's data controls decide User's Model Improvement setting decides Not covered here Not covered here
How long chats are kept Business: indefinitely. Enterprise: your retention policy Enterprise: indefinitely unless an owner sets a period (minimum 30 days) Stored with your Microsoft 365 content; retention set in Purview Gemini in Workspace: 90 days to indefinite, set by admins. Gemini app: 3, 18 or 36 months (default 18)
Whose access rules apply Connected apps use the connecting account's permissions Workspace roles The user's Microsoft 365 permissions The user's Workspace permissions

A few details behind the table:

  • ChatGPT. OpenAI's pricing page lists "No training on your business data by default" for Business, and puts "Data retention and data residency controls" in Enterprise. Its offboarding page is blunt: "Business workspaces keep chats, files, and canvas documents indefinitely," and "Removing workspace access isn't the same as deleting content." For personal accounts, OpenAI says chat content "may be used to improve OpenAI models if allowed by your ChatGPT data controls."
  • Claude. Anthropic says "By default, we will not use your inputs or outputs from our commercial products" for training. The exceptions are feedback a user submits (the thumbs up or down, kept for up to five years) and data the customer chooses to share; Team and Enterprise owners can switch feedback off with the "Rate chats" setting. Deleted conversations leave the user's history at once and Anthropic's back-end storage within 30 days. On consumer plans, Anthropic's wording is "You choose to allow us to use your chats and coding sessions to improve Claude."
  • Microsoft 365 Copilot. Microsoft says prompts, responses and data accessed through Microsoft Graph "aren't used to train foundation LLMs," and that Copilot services have opted out of the abuse monitoring that includes human review. Web searches Copilot sends to Bing are handled under different terms, with Microsoft as an independent controller.
  • Gemini. Google's privacy hub says "Workspace does not use customer data for training models without customer's prior permission or instruction," and that Gemini app chats won't be reviewed by human reviewers.
Fig. 1What changes when the same prompt goes into a business plan instead of a personal account. Training defaults as of November 2026, from OpenAI, Anthropic, Microsoft and Google's own pages.

The practical gap is the personal account. A Plus or Pro subscription paid on a personal card isn't covered by any company contract. The company can't see it, can't set its retention, can't delete it when the employee leaves, and doesn't control the training setting. That's the case for paying for business seats, and it's the whole argument of our ChatGPT Plus vs Business comparison.

What are the biggest AI data security risks?

The biggest AI data security risks for a mid-size company are personal accounts holding company data, approved assistants surfacing files people shouldn't see, connectors and agents with broad access, and chat history kept indefinitely because nobody set a retention period. Model training, the risk most people worry about first, is the one business plans already settle.

Personal accounts and shadow AI

Staff who don't have an approved tool use their own. Their chats sit outside your contracts and your offboarding. The fix is to give them a sanctioned tool that's at least as good and say clearly what may go into it. Our guide to shadow AI covers how to find this use without punishing it.

Approved tools that see too much

Microsoft is explicit: Copilot "only surfaces organizational data to which individual users have at least view permissions," and it tells customers to use Microsoft 365's permission models "to help ensure the right users or groups have the right access to the right content." Google says the same of Gemini: "If the user doesn't have access to a document or email, Gemini will not retrieve that content." Neither tool breaks your permissions. Both make them easy to search. A salary spreadsheet shared with "everyone in the company" years ago becomes one question away.

AI data leakage through connectors and agents

AI data leakage is data leaving the place it's meant to be through an AI tool: into a vendor's logs, into an answer someone shouldn't see, or out through an agent's action. Connectors and agents are where it scales, because they move data no one typed. The survey above found 56% of security leaders say agents acting for employees sit outside their controls, and only 48% could produce any record of the types of data that left through AI tools.

Retention nobody set

Defaults keep a lot. ChatGPT Business keeps chats and files indefinitely. Claude Enterprise keeps data indefinitely until an owner sets a period. That history is discoverable in a lawsuit, readable by anyone who gets into the account, and still there after the project ends. Decide a period on purpose.

Which AI is best for data security?

No AI tool is secure or insecure by brand. The safest choice is the business plan of a tool you can manage: single sign-on, a retention setting you control, audit logs, and data loss prevention that reaches it. If your company runs on Microsoft 365, Copilot sits inside the permissions, labels and retention you already govern. If it runs on Google Workspace, Gemini does the same there. ChatGPT and Claude make sense where staff need them for the work, on Business or Team plans at least, with Enterprise where you need retention controls.

What changes the answer is regulated data. Health information needs a business associate agreement before it goes into any tool, and the plans that offer one differ by vendor. Our HIPAA compliant AI guide sets them out.

What data rules should a company set?

Set one short data rule that sorts company information into three groups: data that may go into approved AI tools, data that may go in only on specific tools with a reason, and data that never goes in. Staff remember three groups. They don't remember a 12-page classification policy.

A workable starting point:

Group Examples Rule
Open Public information, marketing drafts, general how-to questions, your own writing Any approved AI tool
Restricted Client documents, internal financials, contracts, source code, personnel matters without health details Only approved tools on a business plan, with the purpose written down
Never Passwords and keys, payment card numbers, health records without a BAA, data a client contract keeps out of third-party tools Not into any AI tool
Fig. 2A three-group data rule: what may go into AI tools, what may go in only on business plans with a reason, and what never goes in.

Two details make the rule hold. First, write the "Never" list from your contracts, not from instinct: client agreements and professional rules often say exactly what may go to a third party. Second, tie each group to named tools, so "approved tool" means something specific. The free AI Usage Policy generator writes these rules into a staff-facing policy from your answers.

Data rules are a guardrail, not a ban

Companies that respond to AI data risk by banning everything get the opposite of what they want. Staff keep using AI, on personal accounts the company can't see. The survey found 98% of firms have an AI usage policy, yet 35% have a written list of prohibited inputs with no technical way to enforce it. A rule nobody can follow is a rule nobody follows.

In our framework, the data map is one of the guardrails that let a company say yes quickly: it tells staff what's safe before they ask. The free lesson on the data map walks through building one in an afternoon, and it's the step that makes AI deployment across a company possible without a security review for every request.

Do you need AI data loss prevention?

Most mid-size companies don't need a separate AI data loss prevention product to start. They need the data loss prevention they already pay for switched on for AI, plus business plans that keep AI inside it. Google says existing Workspace DLP controls "will continue to apply to generated output from Gemini," though it notes Gemini Notebook isn't integrated with Workspace DLP yet. Microsoft says Copilot honors Purview sensitivity labels and encryption, and admins can set retention and search Copilot activity in Purview.

A dedicated AI DLP tool earns its cost when staff must use AI tools outside your suite, when you hold regulated data at volume, or when an audit asks for a record of what went into AI. Start with an inventory of what's in use. Our AI inventory template shows what to record for each tool.

How to secure company data in the AI tools staff use

01 · List every AI tool that holds company data

Include personal accounts staff admit to, AI features inside software you already pay for, and connectors. The output is one list with an owner for each tool. Everything else depends on it.

02 · Move company work onto business plans

Buy business seats for the tools people actually use, turn on single sign-on, and tell staff to stop using personal accounts for work. This settles training and puts chats where you can manage them.

03 · Set retention on purpose

For each tool, decide how long chats and files should be kept and set it where the plan allows. Where it doesn't (ChatGPT Business, for example), write the gap down and decide whether it justifies Enterprise.

04 · Tidy permissions before you switch on assistants

Before Copilot or Gemini reads company files, find the folders and sites shared with everyone and fix the obvious ones: pay, personnel, client and board material. Then roll out to a pilot group first.

05 · Publish the three-group data rule and review it quarterly

Name the tools in each group, put the rule where staff will see it, and check it every quarter against new tools, new connectors and any incident. If you want help running this across a company, our AI governance service does it with you.

What's still open

  • Agents. Vendors are adding agents that act across email, files and business systems faster than admin controls are catching up, and the survey above suggests most companies' controls don't reach them yet. Treat each agent as a new tool with its own permissions and owner.
  • Retention defaults. Vendors change plan features often. The retention and training terms in this article are as of November 2026, from each vendor's own pages.
  • Third-party models inside suites. Microsoft now offers Anthropic's Claude models inside Copilot, and says those models are currently excluded from the EU Data Boundary. If you have EU data duties, check which models your tenant allows.

We'll update this article when a vendor changes its training, retention or admin terms for business plans. The date at the top of the page shows when we last checked.

Sources

Questions

Common questions

Start with step one

Find out which AI tools already hold your data.

The free Shadow AI Audit lists the AI your staff use, including personal accounts and the AI features vendors have switched on inside software you already pay for. You can't set data rules for tools you don't know about.