Blog

What is proprietary data, and how do you use it safely with AI?

Learn what proprietary data is and how to use it safely with AI. Find where it leaks and how to govern agent access to live data.

blog-color-sol-yellow-pattern-polar-grid
Alex Patino Alexander Patino Solutions Content Leader Published November 16, 2026 Read time 42 min read

Proprietary data is information a company owns or lawfully controls, keeps confidential, and protects because its value depends on limiting who can see it. Customer records, pricing models, source code, research results, and operational logs are considered proprietary data when the company restricts access and takes reasonable steps to keep them confidential.

Using proprietary data with artificial intelligence (AI) raises three concerns: safety, accuracy, and agent access. 

Safety depends on five exposure points: 

  1. The AI tool tiers your people use

  2. What employees paste into them

  3. How widely files are shared

  4. What agents can reach

  5. Whether the retrieval layer enforces permissions

Accuracy depends on how the data is modeled and governed before an AI system reads it. Agent access depends on the path you build between AI systems and live business data.

Where to start, by goal

Your first action is determined by your goals, and each one has a natural owner:

Your goal

First action

When it comes first

Owner

Keep proprietary data out of vendor model training

Check the training, retention, and human-access terms of every AI tool you’re using 

Terms vary by tier, and one company can run several tiers at once

Procurement and IT

Stop sensitive information leaking through employees

Offer an approved enterprise AI tool and publish an acceptable-use policy

Personal accounts are outside company terms and logging

Security and IT

Launch an AI assistant over internal files

Find and fix overshared files and sites before rollout

Assistants show users whatever existing permissions already allow

IT admins and data owners

Get correct answers from structured business data

Build a semantic layer, a governed set of business definitions in the query path

Generated SQL can run without errors and still compute the wrong metric

Data team with business owners

Let agents act on live operational data

Start read-only and route every agent call through a governed gateway

Agents act on live records continuously, so controls have to apply to every call

Engineering and security

Choose an AI platform

Score vendors on data terms, permission enforcement, portability, and accuracy on your own questions

Public benchmark scores don't predict accuracy on your schemas

Procurement with security and data teams

What is proprietary data?

Information is considered proprietary data when it passes three tests: 

  1. It has commercial value because it's confidential

  2. Its owner takes reasonable measures to keep it confidential

  3. Competitors can't easily recreate it 

The type of data doesn't decide whether information is proprietary. A proprietary dataset, a customer database, proprietary software, or a pricing history can all qualify.

Each test comes from a different source:

  • Value from confidentiality: The World Intellectual Property Organization says a trade secret[1] has to be commercially valuable because it's secret, known only to a limited group of people, and protected by reasonable steps such as confidentiality agreements with employees and business partners.

  • Reasonable confidentiality measures: In U.S. federal law, 50 U.S.C. § 1708(d)(7)[2] defines proprietary information to include bid preparations, negotiating strategies, internal financial data, technical designs, manufacturing processes, and source code, as long as the owner has taken reasonable measures to keep it confidential and the public can't readily obtain it.

  • Hard to replicate: Thomas Davenport and Thomas Redman, writing in Harvard Business Review[3], describe proprietary data as data a company alone holds, or has improved enough that another organization can't easily recreate it.

Proprietary data overlaps with two categories that carry their own rules:

Term

What the term covers

Main test

Proprietary data[3]

Company-controlled information with competitive value

Kept confidential and hard to replicate

Trade secret[1]

An intellectual property right over confidential information, which can be sold or licensed

Value from secrecy, a limited circle of people who know it, and reasonable protective steps

Personal data[4]

Information about an identifiable person

Privacy law gives that person rights over it, whoever holds it

Customer data can be both proprietary data and personal data at once. A purchase history can be an asset the company built and personal information about the customer. EU data protection rules give specific rights[4] to the person the data describes, which leaves the collecting company with control closer to possession than to formal ownership.

Calling data proprietary is a business decision, and it settles no legal question by itself. Whether a dataset is confidential, protected as intellectual property, covered by contractual use rights, or subject to privacy law are separate questions, and one customer table may raise all four.

Before connecting a source system to AI tools, ask what it would cost the business if the information in it spread. Records with a high cost need two protections before an AI system has access to them: 

  1. AI tools whose terms keep inputs out of model training

  2. Permissions that limit who can retrieve them

Take the Operational AI infrastructure readiness assessment

A capable model is only part of an Operational AI system; the data layer underneath it determines whether it performs in production. Take the Operational AI infrastructure readiness assessment to see how your infrastructure measures up across six dimensions and get your readiness report.

Why proprietary data matters

Proprietary data matters because it lets a company make decisions and build products from information competitors can't see, and the international standard for measuring economies now treats data as an asset businesses invest in. The 2025 update to the System of National Accounts, which statistics offices use to compile GDP, records data as a produced asset[5], and national statistics offices are now working out how to apply it.

Experimental estimates give a sense of scale. The OECD's 2022 report on measuring the value of data published these U.S. figures[4]:

Measure of U.S. data value

Amount (USD)

Year

Method

Annual investment in data assets built for own use, business and government (a yearly flow)

$875 billion

2024

Sum of production costs

Accumulated net stock of those data assets after depreciation (a running total)

$1,966 billion

2024

Sum of production costs

The $875 billion figure measures what U.S. organizations spent in one year producing data for their own use, while the $1,966 billion figure is the value of all such assets they’d built up, after depreciation.

Spending isn't the same as return. Data collection, storage, and protection all cost money, and the asset only pays off when the company turns it into decisions, products, or revenue.

For anyone budgeting a data or AI program, invest first in the proprietary data you already use to make money, and treat the rest as storage cost until a use appears.

When proprietary data becomes a durable competitive advantage

Proprietary data gives a company a durable advantage when its own operations renew it, a competitor can't buy or approximate it, and errors in it cost money. A dataset that sits unchanged in an archive gives a much weaker edge, because its value fades while competitors keep collecting their own.

More than half of 20 enterprise-focused venture capitalists said the quality or rarity of proprietary data gives AI startups an edge[6], and one investor singled out customer data that feeds back into the product. Companies holding the same data as everyone else train the same models[3]. And proprietary data loses value as conditions change[7], so only data kept current holds its worth.

Economic research shows that data produces increasing returns[4]: More business activity generates more data, which supports more activity. The edge comes from the process that keeps proprietary data current and puts it to use, and the stored dataset is the latest output of that process.

A four-question test for durable advantage

Score each important dataset against four questions:

Question to ask about a dataset

Signs of a durable advantage

Signs of a short-lived advantage

Do your own operations renew the data?

Every transaction, claim, or sensor reading adds records

A one-time project or purchased dataset that ages

Could a competitor buy, collect, or approximate it?

Requires your customers, contracts, or physical access

Available from data brokers, public sources, or a similar vendor

Does an error in the data cost money?

Wrong values change credit, fraud, pricing, or safety decisions

Approximate answers are good enough

Can you act on new records quickly?

A pipeline feeds fresh data into decisions inside the workflow

Records wait in a warehouse until the next quarterly report

A payments company's transaction history usually passes all four questions. A purchased list of company names and headcounts usually fails the second, because a rival can buy the same list.

Running the test before an AI project shows where proprietary intelligence comes from. Datasets that pass deserve investment in pipelines, data quality, and access controls. Datasets that fail can still be useful, but they shouldn't anchor a strategy or justify heavy protection spending.

Protecting proprietary data with contracts, controls, and law

Companies protect proprietary data with contracts, technical controls, and handling rules, and each layer covers weaknesses the others leave open. No layer works alone, because legal protection for raw data is narrower than legal protection for inventions or creative works.

Each layer has a different job and a different limit:

Protection layer

What it does

Where it falls short

Contracts

Confidentiality agreements, nondisclosure agreements, and noncompete clauses bind employees, contractors, and partners (WIPO[1])

A contract creates a claim after a leak and doesn't stop the leak

Technical controls

Encryption, access control, and regular security audits limit unauthorized access and the damage from a data breach

Controls only protect the systems they're configured on

Handling rules

Documented reasons and review dates for each restriction, plus metadata that records what was withheld (U.S. Geological Survey[8])

Rules depend on people following them consistently

Law

Trade secret law treats misappropriation through espionage, breach of contract, or breach of confidence as an unfair practice (WIPO[1])

Coverage for raw data is limited

The U.S. Geological Survey's handling rules work as a model for any company. The agency applies restrictions to the copy it distributes[8] and never alters or deletes the stored original, and it writes each restriction into the metadata instead of leaving a field blank. Analysts downstream can then tell that information was withheld, which keeps their results honest.

Intellectual property law covers inventions and creative works more fully than facts. A trade secret owner can't stop a competitor[1] that develops the same information independently through its own research or reverse engineering. Copyright usually protects how a database is structured while leaving the facts inside it open[4], and the EU adds a separate database right, so the position varies by jurisdiction.

National governments also penalize theft. The U.S. President can block the property of foreign persons[2] who knowingly benefit from cyber theft of trade secrets or proprietary information owned by U.S. persons. That remedy arrives long after the loss, so it doesn't replace prevention.

Because the legal footing is limited, confidentiality agreements and access controls protect proprietary data more than patents or copyright do. That makes data governance and data security part of daily operations.

Sharing, selling and licensing proprietary data

Proprietary data can be shared, sold, or licensed, usually under confidentiality agreements, but open markets for data remain small. Companies often earn from their data through services built on it instead of selling the data itself.

Direct transfers tend to happen around large transactions such as joint ventures, mergers, and acquisitions[9], where a buyer needs to see the data to value the business. Other than deals like these, trading is rare. In 2025, 33% of EU enterprises analyzed data with their own employees, according to Eurostat. Buying and selling is far less common. In 2019, 15% of European businesses analyzed big data, but only 1.3% bought data and 0.6% sold it[4], even counting licensed access.

These low numbers are due to how data behaves as a product:

  • Resale risk: Copying data costs almost nothing, and weak ownership rights make it hard for a seller to stop a buyer from passing it on.

  • Unknown quality: A buyer can't judge a dataset until after paying for it, which pushes prices toward the average and drives high-quality sellers out.

  • Holdup risk: A company that depends on one external supplier can face sudden price increases, so many firms prefer to produce data themselves.

  • Strategic value: Incumbents hesitate to sell data that would show competitors where their operations are strong or weak.

Companies work around these barriers by selling services instead of datasets[4]. Financial data providers bundle market data with software and terminals, and search engines sell better-targeted advertising based on what they know about users.

If you plan to make money from proprietary data, a service or API usually protects the asset better than handing over a copy.

Three ways AI systems use proprietary data

AI systems use proprietary data in three ways: 

  1. A vendor trains its model on the data 

  2. Your team fine-tunes a model on it

  3. An application retrieves it when someone asks a question

Each path stores the information in a different place, and that place determines whether you can update it, delete it, or limit who sees it.

Path into the AI system

Where the proprietary data ends up

Can you update or delete it later?

Can you restrict it per user?

Typical use

Vendor model training

Inside the vendor's model weights

Not reliably. Removing a record from a training set doesn't remove its influence on a model already trained

No

Happens only when an AI tool's terms allow it

Fine-tuning

Inside the weights of a model you control

Usually through retraining, specialized unlearning, or replacing the model, and none guarantees removal

The serving application can authorize who calls the model, but the weights can't enforce access to individual documents

Adapting behavior, format, terminology, or task performance

Retrieval-augmented generation (RAG)

In a search index or database outside the model

Eventually. Each change has to reach every index, cache, and derived copy, so measure how long that takes

Yes, when the server enforces authorization before content reaches the model

Supplying current, attributable, permission-scoped information

Per-user restriction matters most for proprietary data. A person should be allowed to query a fine-tuned model only if they're authorized to see every document used to tune it[10], but tuning on all of an organization's restricted data could leave almost nobody eligible to use the result. Moreover, retrieval pipelines leak restricted documents into answers when access control lists aren't enforced for everyone involved in an interaction.

The table simplifies both architectures, and production systems often combine them. When a vendor says its assistant "knows your documents," ask which path it uses. The answer tells you how you'll correct a wrong fact, honor a deletion request, or keep one department's files away from another.

Five places proprietary data leaks in AI systems

Proprietary data leaks out of AI systems at five points: 

  1. Vendor training and retention

  2. Employees using unsanctioned tools

  3. Overshared files that AI search makes easy to find

  4. Agents that can be steered into sending data out

  5. Retrieval layers that ignore permissions or mix one customer's records with another's

Each point has a different owner inside a company, so fixing just one leaves the other four open.

Exposure point

What leaks, with examples

Who is usually responsible for the fix

Vendor training and retention

Prompts, files, and outputs an AI provider keeps or learns from

Procurement and IT, through tier choice and contract terms

Employee use of unsanctioned AI tools

Customer data, source code, and financial data pasted into personal accounts

Security and IT, through sanctioned tools and monitoring

Overshared files

Files a user's valid permissions allow, though the owner never meant to share them that widely[11]

Data owners and IT admins, through permission cleanup

Agent exfiltration

Private records that an agent sends outside after processing hidden instructions[12]

Engineering and security, through agent design

Retrieval-layer leakage

Documents an index returns despite the user's permissions[10], or one customer's records appearing in another customer's answers

Engineering, through server-side authorization and tenant isolation

The table also shows why one control rarely covers more than one point. An enterprise contract does nothing about a file shared with the whole company, and a permission cleanup does nothing about an employee pasting a contract into a free chatbot. Give each exposure point a named owner and a review date before any AI rollout reaches proprietary data.

Do AI vendors train on your company's data?

Whether an AI vendor trains on your company's data depends on the product tier your employees use, and one company may run several tiers at once. Business, enterprise, and API tiers from OpenAI, Google, Microsoft, and Anthropic exclude customer data from model training by default, while individual plans may use it depending on the user's settings.

The major vendors draw the line between personal plans and business or API plans:

Vendor

Business, enterprise, and API tiers

Individual plans

OpenAI

Not used for training by default in ChatGPT Business, Enterprise, and Edu[13]. API data isn't used for training by default, and abuse-monitoring logs, which include prompts and responses, are kept for up to 30 days by default[14] 

Free, Plus, and Pro conversations can be used for training[15] unless the user turns off "Improve the model for everyone," and rating a response can bring that conversation into training even after opting out

Google

Workspace customer data[15], including prompts, isn't used to train models without the customer's permission

Gemini app use outside a qualifying Workspace edition[16] may be used for model training and improvement

Microsoft

Microsoft 365 Copilot[17]'s stored interaction data isn't used to train the foundation models behind it 

Microsoft says prompts, responses, and file contents in the Copilot app released August 18, 2026 aren’t used to train foundation models. The older app uses conversations for training unless the user opts out

Anthropic

Team, Enterprise, and API data[18] isn't used to train generative models under the commercial terms 

Since August 28, 2025, Free, Pro, and Max users[18] choose whether their data can be used to improve future models

These terms were checked against each vendor's documentation in October 2026. Vendors change product settings and policies often, so confirm the current terms for the product, endpoint, and configuration you use.

But a no-training commitment is just part of the answer. A business tier can still retain prompts for abuse monitoring, keep them longer under a legal obligation, and reach internal systems through connectors. OpenAI's enterprise privacy commitments make connector control a customer decision[13], which means somebody inside your company has to make it.

Terms also change after you sign. In 2025, Salesforce rewrote Slack's API terms to bar bulk exports, persistent indexes, and the use of Slack data in large language models, and enterprise search vendor Glean told customers the change would limit how they could use their own data[19] with the AI platform they'd chosen. Without negotiated limits, Salesforce could change the terms on notice, and continued use of Slack counted as acceptance.

Take the Operational AI business readiness assessment

The AI outcomes your teams are planning depend on the data layer underneath them. Take the Operational AI business readiness assessment to see whether your infrastructure can support them across six dimensions and get your readiness report.

What to check for every AI tool 

Ask these questions of every tool and tier your company uses:

  • Training default: Does this tier train on prompts and files by default, and who can change the setting?

  • Retention: How long does the vendor keep inputs and outputs, and who sets that period?

  • Exceptions: When does abuse monitoring or a legal requirement extend retention?

  • Connectors: Which internal systems does the tool reach, and who approves new connections?

  • Change terms: Can the vendor amend data terms without your agreement?

Include the personal accounts employees bring to work, and write the answers into contracts where you can.

Employee leakage and shadow AI

Employees regularly paste proprietary data into AI tools, and a large share of it goes through personal accounts that skip single sign-on, central logging, and the company's retention and training terms. It happens during ordinary work, such as debugging code, drafting investor updates, or summarizing a contract.

Two security vendors have measured the scale. Harmonic Security reviewed 1 million prompts and 20,000 uploaded files sent to more than 300 AI tools between April and June 2025, and found sensitive content in 4.37% of prompts and nearly 22% of uploads[20], with 26.3% of sensitive prompts and files going through the free version of ChatGPT. Cyberhaven's 2026 data found that 32.3% of ChatGPT usage and 24.9% of Gemini usage happen in personal accounts[21]. Both firms sell data-loss prevention products and define "sensitive" in their own ways, so take that into account and check each report's sample, timeframe, and definitions.

Files are the biggest problem. Employees upload far fewer files than they type prompts, but one PDF or spreadsheet can hold a customer list or a set of financial information, which is why the sensitive rate for uploads in Harmonic's sample is about five times the rate for prompts.

If employees can't get the same function from a sanctioned tool, they'll keep using personal accounts, so offer an approved AI tool with enterprise terms before you tighten blocks. Pair it with a short acceptable-use policy that names the data classes that never go into any AI tool.

Permission failures in AI search and retrieval

AI search exposes proprietary data through permissions in two ways: 

  1. It shows users files that existing permissions already allow but were not meant to be shared widely

  2. A custom retrieval system returns documents the user's permissions don't allow

The first is a permissions problem that AI makes visible. The second is a defect in the retrieval layer.

AI assistants expose existing oversharing

Microsoft 365 Copilot accesses only data to which each user has at least view permission[22], using the same access controls as the rest of Microsoft 365, and that content encrypted with Microsoft Purview stays hidden unless the user also holds the view usage right.

A salary spreadsheet shared with "everyone" years ago was always open to every employee, but finding it took effort. An assistant that answers questions across SharePoint, OneDrive, and Teams removes that effort, so an old permission mistake turns into an answer to a casual question. Review permissions and oversharing before enabling Copilot[11].

Custom retrieval systems “lose” permissions

When documents are split into chunks and turned into vectors, the source system's permission rules don't travel with them unless engineers attach them. Restricted content leaks through retrieval [10]when access control lists weren't enforced for everyone in the interaction, and every piece of content an AI system uses has to be authorized for all of its recipients. Authorization should narrow the candidate documents before any model processes them[23].

Metadata filters are a reasonable default, but enforcement has to happen on the server, before content reaches the model, and it has to account for inherited permissions and alternate retrieval paths. Retrieval quality and permission correctness also need separate tests, because a system can return the right answer from a document the user should not see.

Copying full permission lists onto every chunk has a cost. One access change on a parent folder means rewriting every chunk beneath it, which at scale looks more like a re-index than an update. A design that scales better tags each chunk with its source document and permission group, then resolves those groups against a permission service at query time, so a folder change updates one entry instead of thousands of chunks. That puts a key-value lookup in the path of every query, which is the workload operational databases such as Aerospike are designed for: predictable low latency at high throughput.

The same rule applies between customers. A software company that serves many clients has to isolate tenants in every layer that stores or processes their data, including indexes, caches, and logs, or one client's proprietary information may appear in another client's answer.

Permission checklist before an AI assistant reaches proprietary data

Work through these steps before rollout:

  1. Find files and sites shared with everyone in the organization, or through open links, and narrow them.

  2. Tag every chunk or record in the retrieval index with its source document and permission group, including inherited permissions.

  3. Resolve those permissions on the server at query time, before any content reaches the model, and check the source system where permissions change often. Never leave the decision to the language model.

  4. Set a target time for permission changes, updates, and deletions to reach every index, cache, and derived copy, measure propagation continuously, alert when the target is missed, and invalidate cached answers built from revoked content.

  5. Isolate tenants in every layer that stores or processes customer data.

  6. Test revoked access: Remove a test user's permission and confirm the content disappears from search, answers, and caches within your target time.

Why retrieval over company documents returns wrong answers

Retrieval-augmented generation reduces hallucinations without stopping them[24]. Retrieval over company documents often returns wrong answers because the retrieval step selects the wrong passages, and company documents are harder to search than the clean samples these systems are usually tested on. When retrieval selects the wrong passages, the system generates a fluent answer from them that reads as correct, but isn't.

Retrieval quality depends on preparation. Sourcing, formatting, and chunking documents before indexing is a project in its own right, and the way documents are split and encoded affects both speed and accuracy. Enterprise files make it harder: tables that run across pages, scanned contracts, and slide decks with the meaning in the diagrams lose structure when they're split into text chunks.

Some questions fail by design. Similarity search returns the handful of passages closest to a question, so it can't reliably answer "how many contracts renewed last quarter" or "summarize every complaint from March," which require counting or scanning sets of records. Questions about counts, totals, or current values belong with structured data and a defined access method.

Retrieval still plays a part in a hybrid pipeline, but a count needs a query or process that covers the defined population. Larger context windows don't remove the need for retrieval, either, because a large document collection exceeds what fits in one prompt, and sending more text with every question adds cost and delay.

Each symptom below points to a likely cause and a first fix:

Symptom in production

Likely cause

First fix to try

Answers cite the wrong document

Poor chunking or missing metadata

Re-chunk by document structure and attach titles, dates, and owners to every chunk

Answers miss existing information 

Keyword-style matching misses conceptual matches

Combine keyword and vector search and test with user questions

Counts and totals are wrong

Similarity search can't aggregate

Route aggregate questions to a structured query over a database

Answers with no support

The system generates text when retrieval returns little

Require citations and let the system say it found nothing

How should structured business data be modeled for AI retrieval and reasoning?

Model structured business data for AI by making its meaning explicit: define each business entity and metric once, describe what every column holds, declare how tables relate, attach permissions and freshness to records, and send each type of question to the access method that can answer it. Generated SQL can run without errors and still compute the wrong metric, so the definitions matter more than the query syntax.

The core tool is a semantic layer: a governed set of entity and metric definitions in the query path, so every AI tool and dashboard computes from the same definitions. Revenue can mean one thing in the finance system, another in the CRM, and a third in operational reporting[7], and a semantic layer settles that once. Its definitions have to be executable: governed metrics the query layer computes, validated relationships between tables, and controls on the SQL that AI tools can generate. A prose definition of revenue won't stop generated SQL from joining orders to a line-item table and counting each order once per line, which inflates the total.

Rules for modeling business data for AI

Apply these rules to each business domain you expose to AI:

  1. Define entities and metrics once. Write one definition each for customer, active account, revenue, and churn, with the formula, source table, owner, the synonyms people use, and a few example questions with verified answers.

  2. Describe columns in business language. Record units, currency, time zone, valid values, and what a blank means.

  3. Make relationships explicit. Declare the keys that connect customers, orders, products, and invoices. Without declared keys, generated SQL can join tables on the wrong columns and produce totals that look plausible but are wrong.

  4. Publish curated views by domain. Give AI tools a sales view or a support view with dozens of well-described columns instead of a raw schema with thousands. Bain recommends anchoring data work to priority business domains.

  5. Carry permissions and freshness on every record. Store access tags and last-updated timestamps with the data, so the query layer filters by user and flags stale answers. Metadata alone doesn't establish authorization or freshness, so define how the query layer enforces each one and how quickly changes propagate.

Not all proprietary operational data is relational. Customer profiles, sessions, device state, and transactions are often stored in key-value or document databases, such as Aerospike, that serve current records at low latency and don't support joins. The same discipline applies in a different form: define the keys and secondary indexes each question needs, keep attributes that are read together in the same record, and send aggregates to a system built for them, such as a warehouse fed from the operational store.

Matching each question type to an access method

Each type of question has an access method that fits it:

Question type

Example

Best access method

Why it fits

Exact lookup

What's the status of order 48213?

Key-based query against the operational database

Returns the current record with no interpretation

Aggregate

What was revenue by region last quarter?

SQL against a defined metric in the semantic layer

Totals need every qualifying record

Open text

What did customers complain about in March?

Retrieval over tickets and documents with date and product filters

The meaning is in prose

Mixed

Which enterprise customers mentioned outages and also renew next month?

Structured filter first, then retrieval inside the filtered set

The filter narrows the records before similarity search runs

If a question needs counting, summing, or a current value, answer it from structured data through a defined metric or key lookup, and use retrieval only for the parts of the question that are stored in text.

As a starting point, take the 20 or so questions your teams ask most often, model the entities and metrics to which they gain access, and expand from there. The same definitions serve dashboards, predictive analytics, and other advanced analytics as well as AI assistants.

What are the best options for creating and maintaining an AI-ready corporate data layer?

A good starting point for many companies is to extend the data platforms they already run with five capabilities: 

  • A semantic layer

  • Permission-aware retrieval

  • A governed access path

  • Continuous data-quality and permission syncing

  • An evaluation harness

The right storage underneath depends on workload, scale, freshness, consistency, latency, and cost. Whichever products you choose, the system has to keep authorization and data correctness intact as records change and AI workloads grow.

In a 2024 Gartner survey of data management leaders, 63% said their organizations either didn't have the right data management practices for AI or weren't sure whether they did[25]. Gartner's advice was to build on existing data management by adding AI-specific capabilities such as vector stores, chunking, embedding, and retrieval integration, and to treat AI-ready data as an ongoing practice.

The five components of an AI-ready data layer

Each component prevents a specific failure:

Component

What it does

The problem it prevents

Semantic layer

Defines entities, metrics, synonyms, and example questions once for every AI tool

Each assistant computes revenue or customer counts its own way

Permission-aware retrieval indexes

Stores chunks and vectors with source references, permission groups, metadata, and timestamps

Users see restricted content, or answers cite stale records

Governed access path

Routes every AI query or agent call through one gateway with a registry of approved tools, logging, and audit

Agent data access unspecified 

Continuous operations and permission syncing

Runs automated quality checks, lineage tracking, and permission re-syncs as data changes

Indexes drift from source systems within hours under heavy write volume

Evaluation harness

Tests AI answers against a fixed set of real questions with known answers before each release

Users see wrong answers

Bain recommends that governance needs to be built into systems and pipelines through automated quality checks, real-time lineage, and continuous access management, and it describes registries of approved agents and tools[7], a governed gateway, and test suites that agents must pass before release. Gartner's guidance adds active metadata management[25], pipelines that feed live data to production AI systems, and data observability.

Build options for the storage underneath

Four common architectures each make a different tradeoff:

Build option

Best for

Main strength

Main limitation

Extend the relational database you already run, such as PostgreSQL with the open-source pgvector extension

Teams whose AI questions need joins and filters over existing business tables

One system to secure, with vectors stored next to the records they describe

AI reads compete with transactional traffic on your system of record unless you add replicas, and write scaling is largely vertical

Add a dedicated vector database

Very large document collections where similarity search is the main workload

Purpose-built indexing and search features

A second system to synchronize, secure, patch, monitor, and maintain 

Use a managed AI search or assistant platform

Fast starts on document search across many SaaS tools

Little engineering to launch

Data access depends on the vendor's terms and the APIs of each source system

Use a distributed operational database, such as Aerospike

Permission lookups and key-value access to live customer, product, and transaction records at high concurrency and a strict latency target

Scales horizontally while keeping latency predictable at high throughput

No joins or ad hoc analytics, so aggregates and text or similarity search run in another system

Compare candidates on data volume, filters, update rate, concurrency, recall, tail latency, operational burden, and total cost, using your own workload. Keep the semantic layer, indexes, and access logs in systems you control, so a change in a SaaS vendor's API terms can't cut off your data layer or your ability to switch AI vendors.

What to measure before choosing storage

The database and indexing architecture determine how the data layer behaves under load, so define these requirements first and test them with production-like traffic:

  • Latency under fan-out: How many lookups one user request triggers once multi-step and multi-agent workflows run, and the p99.9 latency of each lookup at realistic concurrency

  • Freshness: How long a source change takes to reach indexes and caches, and the maximum staleness each use case tolerates

  • Consistency: Whether an agent needs a consistent view across several reads in one task, or only read-your-own-writes within one request; many systems offer the second by default, and needing the first changes the architecture

  • Availability: How each component fails and what users see when it does

  • Write and ingest load: Permission re-syncs, re-embedding after a model change, and agent logs add writes, and sizing only for query volume doesn’t account for this maintenance traffic

  • Growth: How fast records, embeddings, and logs grow, and when that will force a change of architecture.

  • Recovery: How permissions, indexes, caches, and derived copies get reconciled after a failure

  • Cost: Database operations, embedding generation, retrieval, model inference, and ongoing synchronization

Why agent fan-out changes the latency target

Fan-out turns rare slow calls into common slow answers. If one user request triggers 50 lookups and each lookup has a 1% chance of landing in the slow tail, the chance that at least one of them does is about 40% (1 minus 0.99 to the 50th power). The user waits for the slowest call, so per-call p99 latency stops being a useful target.

Measure p99.9 for each lookup and the end-to-end latency of complete agent requests, and test both at the concurrency you expect in production. A data store that looks fast on average can still make most multi-step answers slow, so choose storage that keeps tail latency predictable under concurrent load.

What happens when part of the data layer fails

Plan for each component of the data layer to fail separately, because each one fails differently and affects a different set of users. Decide in advance whether a degraded answer is acceptable or whether the system should decline to answer and say why.

Component that fails

What users see

What it affects

Recovery and fallback

Operational database or system of record

Lookups and agent actions fail or time out

Every agent and assistant that reads live records

Fail over to replicas, and have agents return a clear error

Retrieval index

Missing or stale passages, and answers without citations

Document questions and any agent step that searches

Rebuild from source, and fall back to keyword search or decline to answer

Embedding service

New and changed content stops being indexed, so freshness drifts 

Every index fed by that service

Queue changes, replay them on recovery, and alert on a growing backlog

Model provider

No answers, or much slower ones

Every AI feature that uses the provider

Route to a secondary model where policy allows, or show a clear outage message

Permission sync pipeline

Revoked access still works, or new access doesn't

Every retrieval path that relies on synced permissions

Fail closed by denying access when permissions can't be confirmed, and alert

Fail closed on permissions to hide records the system can't confirm a user may see. For answers that drive decisions, such as pricing, credit, or customer commitments, return nothing with a clear message when data is stale or partial. For low-stakes reference questions, a partial answer labeled as partial may be acceptable.

How to maintain the data layer

Maintenance runs continuously once AI systems depend on the layer:

  1. Re-sync permissions and content from source systems whenever they change, and remove deleted records from every index, cache, and derived copy.

  2. Run data quality checks in the pipeline itself, before records reach an AI system.

  3. Track lineage so any AI answer can be traced to the records and definitions behind it.

  4. Re-run the evaluation set after every model, prompt, schema, or index change.

  5. Review metric definitions with their business owners on a fixed schedule.

A data layer like this is a lot of work, but it’s worth it when more than one assistant or agent depends on it. Bain notes that companies that make this investment early find that each later agent deployment gets faster and cheaper.

How do you connect AI agents to live customer, product, transaction, or operational data?

Connect AI agents to live business data through a governed access path that gives each agent the narrowest access it needs, enforces authorization outside the model, and logs every call. Agents act on current records continuously, so controls have to apply to every call.

Agents make decisions every minute on live data[7], while traditional governance checks data on a monthly or quarterly cycle. 

The model context protocol (MCP), an open standard, gives agents a common interface to tools and data sources. With a common interface, you can put one governed gateway in front of every tool. The same permissions and logging then apply to each call, and no integration has to handle them its own way. In December 2025, Anthropic donated MCP to the Agentic AI Foundation, a directed fund under the Linux Foundation, to keep the standard open and vendor-neutral. MCP doesn’t decide what an agent is allowed to reach, so the gateway has to enforce authorization.

Explore Aerospike Voyager

Aerospike Voyager ships with an embedded model context protocol (MCP) server, so your AI coding agent connects to the same cluster you do and writes queries in the same expression language your application ships. Token-based authentication, read-only and full-access profiles, and configurable timeouts keep exploration safe at scale.

Three ways to connect agents to live data

Agents can reach live data through three approaches, each with a different freshness and risk profile:

Connection approach

How it works

How fresh the data is

Main risk

Best for

Query an indexed copy

The agent searches a synchronized index of records and documents

As fresh as the last sync, which can range from seconds with streaming change capture to a day with nightly batch jobs

Answers built on stale or revoked data

Reference questions where the sync interval is acceptable

Call governed tools or APIs, such as through MCP servers

The agent calls predefined functions, such as one that returns an order's status, through a gateway

Live

Tools with broader access than the task needs, and prompt injection

Lookups and approved actions on current customer, product, or transaction records

Generate queries against the database

The agent writes SQL against curated views

Live

Queries that look right and return wrong results, accidental writes, and unbounded scans that overload the production database and run up per-read costs

Ad hoc analysis by trained users who check the query

A sensible design mixes the three: governed tools for live lookups and actions, an index for documents, and generated queries only where a person reviews the result. Governed tools put a low-latency lookup behind every agent step, which is why the fan-out and tail-latency requirements in the data layer apply here, too.

Check every agent for the lethal trifecta

The “lethal trifecta for AI agents[12]” is the combination of access to private data, exposure to untrusted content, and the ability to communicate externally. When all three are present, instructions hidden in content an agent processes can direct it to send private data to an attacker. GitHub's official MCP server combined all three. Treats guardrail products that claim to block 95% of such attacks as inadequate, because in application security a 95% success rate is a failing grade.

A customer support agent shows how easily the three conditions line up. It processes inbound customer emails, which anyone can write; it looks up account history in the CRM, which holds customer data and financial information; and it can send email. Removing or tightly restricting one condition reduces the vulnerability: the agent can draft replies for a person to approve instead of sending them, or it can receive only structured ticket fields instead of raw email text.

The trifecta is a heuristic for spotting dangerous combinations, and other disclosure paths remain after you remove one condition. An agent with no outbound channel can still reveal data in its answer to the wrong user, and a read-only agent can still expose records or run up query and token costs.

Controls for agents with access to live systems

Apply these controls before an agent gets access at scale:

  1. Start with least privilege. Begin with read-only access in most cases, and separately limit how much data each call can return, how many queries an agent can run, how long each query may take, and how much it can spend. Isolate agent queries from customer-facing workloads. Require a person to approve writes, refunds, record changes, and outbound messages until the agent has a track record, then write the policy that lets low-risk actions run automatically.

  2. Enforce authorization outside the model. Check every tool call and data request against an explicit policy. Where an agent acts for a user, apply that user's delegated permissions. For authorized background tasks, use a narrowly scoped service identity with explicit rules for the resources and operations it may use.

  3. Expose narrow tools and validate them. Give agents functions with defined inputs and outputs over curated views, validate those inputs and outputs in code outside the model, and keep raw database credentials out of agent reach.

  4. Restrict outbound data flows. Allow only approved destinations for any message, request, or file an agent can send.

  5. Route every call through one gateway. Use a gateway with a registry of approved agents and tools, where policy, telemetry, and audit trails apply to every call. Log enough to reconstruct the identity, the data accessed, the policy decision, and the action behind each call.

  6. Prove behavior before granting authority. Run new agents in shadow mode alongside existing processes, then require them to pass automated promotion gates and a fixed test suite before release. Include hostile documents with planted instructions in that suite.

  7. Contain chain reactions. In multi-agent workflows, an error in one agent can trigger actions in the next, so set limits on how far an action travels without review.

  8. Measure the load. Track end-to-end request latency, per-call p99.9 latency, fan-out, and cost on the systems that serve customers while agents run against production-like traffic before you grant access at scale.

Building an agent is about 20% of the work, evaluations about 60%, and ongoing monitoring the remaining 20%. Treat the split as a planning heuristic. Token costs become a material line item at enterprise scale, so budget for the controls above, and the queries they generate, from the start.

Evaluating AI tools that will use proprietary data

Evaluate AI tools that will use proprietary data on six criteria: 

  1. Data-use terms

  2. Permission enforcement

  3. Portability

  4. Accuracy on your own questions

  5. Auditability

  6. Cost at scale

Test each one with your own data. Public benchmark scores and vendor test sets come from other organizations' data, so they don't predict how a tool handles your schemas, permissions, and phrasing.

For each criterion, ask a specific question and require evidence:

Evaluation criterion

Question to ask 

Evidence to request

Red flag

Data-use terms

Does each tier we'll use train on our data, how long is data kept, and who has access to it?

Written terms for every tier, including any personal or trial accounts

Answers that cover only the top tier

Permission enforcement

Are source-system permissions checked at query time for each user?

A live test with a restricted user account

Permissions checked only when documents are indexed

Portability

Can we export our indexes, definitions, and logs, and what happens if a source API changes?

Export formats and the contract's change clause

Product depends on one SaaS API with no fallback

Accuracy on our data

How does the tool score on our real questions?

Results on 50 to 100 of your questions with verified answers

Only public benchmark scores

Auditability

Can we see which records and definitions produced each answer?

Per-answer source traces and access logs

No way to trace an answer to its data

Cost at scale

How do costs grow with users, queries, and tokens?

Pricing for the tiers that include the security and governance features you need

Required controls sold only as add-ons

How to run a fair evaluation

Run the same test against every candidate:

  1. Collect 50 to 100 real questions, with verified answers, from the teams who'll use the tool. Treat that number as a starting point and increase the set over time.

  2. Include lookups, aggregates, and open-text questions so you test both structured and document data.

  3. Score retrieval quality and permission correctness separately, because a system can return the right answer from a document the user should not see.

  4. Add test users with restricted access, revoke a permission partway through, and confirm the content disappears within your freshness target.

  5. Plant hostile instructions in test documents and confirm the system doesn't follow them.

  6. Score each answer for correctness, correct citation, and whether the tool reports when it has no answer.

  7. Repeat the same test after every change to the model, prompts, schema, or index.

Keep the question set after the evaluation. It becomes the evaluation harness for your data layer, and it gives security, legal, and business stakeholders the same evidence to review.

Templates for using proprietary data safely with AI

Three documents turn these controls into everyday rules: 

  1. An AI acceptable-use policy

  2. A data classification guide with AI rules for each level

  3. Updated confidentiality agreement clauses

Each one takes a few minutes to read and applies to everyone who handles proprietary data.

The data classes to name come from what leaks. Source code, credentials, and proprietary algorithms made up about a third of sensitive exposures[20], alongside M&A plans, financial projections, and customer records.

AI acceptable-use policy

The policy should name approved tools and tiers, list the data that never goes into any AI tool, and explain how to request a new tool. Here’s an example:

## Approved AI tools
Use only the company workspace of [approved tools]. Personal accounts are not
approved for company work, including free and trial accounts.

## Never enter into any AI tool
- Source code from private repositories
- Passwords, API keys, and access tokens
- Customer personal information and payment data
- Unreleased financial results, forecasts, and M&A documents

## Requesting a new AI tool
Submit the tool name, tier, intended data, and business owner to [security contact].

Marketing and sales teams need this policy as much as engineers do, because campaign results, customer lists, and pricing are proprietary data that marketers handle. 

Data classification guide with AI rules

Each classification level gets its own AI rule:

Classification level

Examples

AI rule

Public

Published pages and press releases

Any approved AI tool

Internal

Process documents and internal wikis

Approved enterprise tiers only

Confidential

Customer data, contracts, pricing, and financial data

Approved enterprise tiers with permission-aware retrieval and logging

Restricted

Source code, credentials, unreleased results, and trade secrets

No AI use without a documented exception

Treat these rules as a conservative default. The right rule for each level depends on your threat model, where the AI system runs, your legal obligations, and the controls you have in place. Record the reason for each restriction and a date to review it, so classifications don't harden into permanent blocks.

Confidentiality agreement clauses for AI

Confidentiality agreements with employees and partners protect trade secrets[1]. Agreements signed before generative AI tools were common may not say whether a recipient can put disclosed information into an AI tool. Clauses to cover include whether AI processing is permitted, which tiers and training settings are acceptable, retention and deletion requirements, and whom to tell if information enters a tool by mistake.

Have legal counsel review these clauses, the acceptable-use policy, and the data classification guide before use, and check that your public privacy policy covers any customer personal information you send to AI vendors. Assign each document an owner responsible for compliance reviews, and include links to all three in employee onboarding.

Aerospike and proprietary data

Safely deploying AI over proprietary data requires day-one decisions on tool tiers, permissions, and agent access, but as your rollout scales, agent workflows will trigger massive query spikes that overwhelm standard architecture. Aerospike is the database engineered to solve this operational bottleneck, acting as the high-speed engine behind governed AI agents to deliver live customer records, transactions, and real-time permission checks with predictable, ultra-low latency. For your underlying data layer to enforce security and return fresh values at enterprise scale, talk to an Aerospike engineer about your performance targets or try Aerospike against your own workload.

FAQs

Frequently asked questions about proprietary data

Proprietary data is information a company owns or lawfully controls, keeps confidential, and protects because its value depends on limiting who sees it. Customer records, source code, pricing models, research results, and operational logs are common examples.

No. A trade secret[1] is a legal intellectual property right over confidential information that meets specific tests, including commercial value from secrecy and reasonable steps to keep it secret. Proprietary data is a broader business term, and some of it doesn’t qualify for trade secret protection.

It depends on the tier. OpenAI, Google, Microsoft, and Anthropic exclude data in their business, enterprise, and API tiers from model training by default. Individual plans from OpenAI[15], Google[16], and Anthropic[18] may use conversations for training depending on the user's settings.

No. Enterprise tiers usually exclude customer data from training, but they may still retain prompts for abuse monitoring, keep data longer when the law requires it, and reach overshared files through connectors and company search.

Only into approved tools and tiers, and only for data classes your policy allows. Harmonic Security found sensitive content in 4.37% of prompts and nearly 22% of uploaded files[20] in its 2025 sample, much of it through free accounts.

No. Retrieval-augmented generation grounds answers in your documents and reduces hallucinations, but a system can still produce unsupported answers[24] when retrieval returns weak or irrelevant passages.

No. A large document collection exceeds what fits in one prompt, and sending more text with every question adds cost and delay, so retrieval still has to choose which content reaches the model.

Yes, usually under confidentiality agreements or licensing contracts. Open data markets remain small: in 2019, only 1.3% of European businesses bought data, and 0.6% sold it[4].

Define business entities and metrics once in a semantic layer, describe what every column means, make relationships between tables explicit, and attach permissions and freshness to each record. Then send counts and totals to structured queries, and open-text questions to retrieval.

A good starting point is to extend the data platforms you already run with a semantic layer, permission-aware retrieval indexes, a governed access path, continuous quality and permission syncing, and an evaluation harness. Gartner describes AI-ready data as an ongoing practice that needs continued investment in metadata management and data observability[25].

Give agents narrow, least-privilege tools through a governed gateway, enforce authorization outside the model with the user's delegated permissions or a narrowly scoped service identity, and require a person to approve writes until you've written a policy for automated actions. Check each agent for the lethal trifecta[12]: private data, untrusted content, and a way to send information out.

Take the Operational AI assessment

Your AI is only as current as the data beneath it, and when the operating loop falls behind, every decision runs on a version of reality that no longer exists. Find out where your infrastructure stands: take the technical readiness assessment or the business readiness assessment and get a clear picture of whether your data layer is ready for Operational AI.

Footnotes

  1. World Intellectual Property Organization. “Trade Secrets.” https://www.wipo.int/en/web/trade-secrets

  2. Cornell Law School, Legal Information Institute. “50 U.S. Code § 1708 - Actions to address economic or industrial espionage in cyberspace.” https://www.law.cornell.edu/uscode/text/50/1708

  3. Harvard Business Review. “Your Organization Needs a Proprietary Data Strategy.” May 2020. https://hbr.org/2020/05/your-organization-needs-a-proprietary-data-strategy

  4. OECD. “Measuring the value of data and data flows.” OECD Digital Economy Papers, no. 345, 2022. https://doi.org/10.1787/923230a6-en

  5. United Nations Statistics Division. “SNA News and Notes,” no. 43, June 2024. https://unstats.un.org/UNSD/nationalaccount/sna/nn43-en.pdf

  6. Rebecca Szkutak. “VCs say AI companies need proprietary data to stand out from the pack.” TechCrunch, January 10, 2025. https://techcrunch.com/2025/01/10/vcs-say-ai-companies-need-proprietary-data-to-stand-out-from-the-pack

  7. Bain and Company. “Turn Artificial Intelligence into Proprietary Intelligence.” https://www.bain.com/insights/solutions/turn-artificial-intelligence-into-proprietary-intelligence/decision-3-proprietary-data/

  8. U.S. Geological Survey. “Proprietary and Sensitive Data.” USGS Data Management. https://www.usgs.gov/data-management/proprietary-and-sensitive-data

  9. Enverus. “Proprietary Data.” Enverus Glossary. https://www.enverus.com/glossary/proprietary-data/

  10. Shashank Shreedhar Bhatt et al. “Enterprise AI Must Enforce Participant-Aware Access Control.” arXiv:2509.14608, September 2025. https://arxiv.org/abs/2509.14608

  11. Microsoft. “Get ready for Microsoft 365 Copilot and agents with SharePoint Advanced Management.” Microsoft Learn. https://learn.microsoft.com/ar-sa/sharepoint/sharepoint-copilot-best-practices/

  12. Simon Willison. “The lethal trifecta for AI agents: private data, untrusted content, and external communication.” Simon Willison’s Weblog, June 16, 2025. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

  13. OpenAI. “Enterprise privacy at OpenAI.” Updated January 8, 2026. https://openai.com/enterprise-privacy/

  14. OpenAI. “Data controls in the OpenAI platform.” OpenAI Developers. https://developers.openai.com/api/docs/guides/your-data

  15. OpenAI. “How your data is used to improve model performance.” OpenAI Help Center. https://help.openai.com/en/articles/5722486-how-openai-uses-your-data

  16. Google. “Learn how Gemini in Gmail, Calendar, Chat, Docs, Drive, Sheets, Slides, Meet, Vids, and Pics protects your data.” Google Help. https://support.google.com/mail/answer/14615114

  17. Microsoft. “Data, Privacy, and Security for Microsoft Copilot for Microsoft 365.” Microsoft Learn. https://learn.microsoft.com/DeployOffice/privacy/microsoft-365-copilot

  18. Anthropic. “Data usage.” Claude Code Docs. https://docs.claude.com/en/docs/claude-code/data-usage

  19. Hunton Andrews Kurth. “Salesforce Locks Down Slack Data: Time to Review Your Slack API Terms.” Legal update. https://www.hunton.com/insights/legal/salesforce-locks-down-slack-data-time-to-review-your-slack-api-terms

  20. Harmonic Security. “GenAI Data Exposure: What GenAI Usage Is Really Costing Enterprises.” July 31, 2025. https://www.harmonic.security/blog-posts/genai-data-exposure-report

  21. Cyberhaven. “Sensitive Enterprise Data Is Flowing Into AI Tools at Scale.” February 11, 2026. https://www.cyberhaven.com/blog/sensitive-data-flowing-into-ai-tools

  22. Microsoft. “Examine how Microsoft 365 Copilot protects sensitive business data.” In “Examine data security and compliance in Microsoft 365 Copilot.” Microsoft Learn. https://learn.microsoft.com/en-us/training/modules/examine-data-security-microsoft-365-copilot/3-examine-copilot-protects-sensitive-business-data

  23. Rohith Namboothiri. “Authorization-First Retrieval: Enforcing Least Privilege in Multi-Agent RAG Systems.” Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026). Association for Computational Linguistics, July 2026. https://aclanthology.org/2026.trustnlp-main.15/

  24. Kyle Wiggers. “Why RAG won’t solve generative AI’s hallucination problem.” TechCrunch, May 4, 2024. https://techcrunch.com/2024/05/04/why-rag-wont-solve-generative-ais-hallucination-problem/

  25. Gartner. “Lack of AI-Ready Data Puts AI Projects at Risk.” February 26, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk

  26. Model Context Protocol. “MCP joins the Agentic AI Foundation.” Model Context Protocol Blog, December 9, 2025. https://blog.modelcontextprotocol.io/posts/2025-12-09-mcp-joins-agentic-ai-foundation/