Data, Storage, Use, and the Decision Loop

This opening chapter draws the whole map before Part I zooms in. It moves through four questions in turn — where business data comes from, how it is stored, how it is used, and how those pieces connect into a single data-to-decision loop. The order matters. A manager should meet the modern business data system first, and the methods later, as tools that serve specific decisions inside that system.

Where Data Comes From

The executive question: what business activity generated this data?

The first managerial skill is not choosing a model. It is asking what the data is a trace of. A sales row is a trace of a completed transaction, not of all customer demand. A click is a trace of attention inside one interface, not of preference in general. A support ticket is a trace of a problem serious enough to report, not of every problem customers experienced. A prompt log is a trace of an AI workflow being used, not proof that the workflow helped.

This matters because every data source carries the bias of the process that created it. If the process changes, the data changes even when the underlying business does not. If the app redesign makes the return button harder to find, return requests may fall while dissatisfaction rises. If a chatbot deflects simple questions, the support tickets that remain will look more severe. If a store starts scanning loyalty IDs more consistently, "repeat customer" metrics may jump without any real change in loyalty.

Figure 1 gives the broad map. Six source families cover most data a modern manager will encounter: customer behavior, business operations, digital systems, the physical world, human language, and AI workflows. The source family matters because it tells you what kind of claim the data can support.

A day in the life of data

Consider one Bean & Basket customer on a Tuesday morning.

At 7:48 a.m., Maya searches the mobile app for "oat latte." That creates a search event. At 7:49, the app shows her a seasonal drink recommendation. That creates an impression. She taps it, adds a pastry, applies a loyalty reward, and checks out. That creates cart, transaction, payment, promotion, and loyalty records. Her order is prepared at Store 104, where the point-of-sale system updates inventory and the kitchen display records fulfillment time. Her phone location confirms she picked up the order. At 9:10, she rates the experience four stars and writes, "Great drink, long wait." That creates a rating and review text. Later, the operations team uses that review in a text dashboard, the marketing team uses the transaction in a churn model, the product team uses the search event to tune recommendations, and the regional manager sees the wait-time issue in a KPI dashboard.

One morning produced many records. None of them is "the customer" in full. Each is a trace from a specific system, with a specific purpose and a specific blind spot.

The lesson is practical: do not call these records "customer data" as if they were interchangeable. The search event, transaction, review, wait-time record, and AI log are different slices of the customer's interaction with the firm. They become powerful only when the manager knows which slice is being used.

The source changes the claim

The same managerial topic can look different depending on where the data came from.

Take customer satisfaction. A firm might measure it through star ratings, review text, support tickets, refund requests, call transcripts, social posts, survey responses, churn, or repeat purchase. These are not redundant measures of the same thing. They capture different moments in the customer journey.

• Star ratings are easy to monitor but shallow.
• Review text explains reasons but overrepresents people willing to write.
• Support tickets reveal operational problems but only after the customer escalates.
• Refunds and returns capture costly dissatisfaction but miss silent disappointment.
• Churn is a final outcome, often too late for diagnosis.
• AI summaries can scale interpretation but must be evaluated against source evidence.

The managerial question is not "which data source is best?" The question is: which source is closest to the decision we need to make, and what bias does its generation process introduce?

Three generation traps

Trap 1: activity bias. Data overrepresents people who act inside the measured system. App data overrepresents app users. Reviews overrepresent people motivated to write. Loyalty data overrepresents identified customers. The unmeasured population may behave differently.

Trap 2: workflow bias. Data changes when the business process changes. A new refund policy, a redesigned app, a chatbot handoff rule, or a new sales script can change recorded behavior without changing underlying demand or satisfaction.

Trap 3: AI feedback bias. AI workflows create new records and change the behavior that future models learn from. If an AI support assistant routes some complaints away from human agents, the remaining ticket data no longer represents the full complaint mix. If a recommender shows the same products repeatedly, future purchase data reflects exposure as much as preference.

These traps are not reasons to avoid data. They are reasons to read data as a product of its generating process.

A trace only becomes usable once it lands somewhere. That makes the next question structural: where does the data live, and what was that system built to do?

How Data Is Stored

The word "database" hides too much. The system that records a customer's payment is not built for the same job as the system that scans five years of transactions for a pricing analysis. The place that stores raw app logs is not the same as the place that supports semantic search over policy documents. A manager does not need to administer these systems, but does need to understand their roles. Otherwise every data conversation becomes vague: "Can we get the data?" Which data? From which system? For what decision? At what latency? With what quality contract?

The executive question: what job is this data system doing?

Modern firms usually store data in several layers. Each layer optimizes for a different job.

An operational database records the next event correctly: a payment, an order, a login, a shipment, a service case. It is built for reliability, identity, permissions, and fast small updates. An analytical database scans many past events: a year of transactions, a panel of stores, a customer cohort, a product assortment, a marketing funnel. It is built for aggregation, history, and comparison. The distinction is not technical trivia. It determines whether the system is meant to run the business or analyze the business.

Figure 2 is the practical map. Operational SQL, NoSQL, lakes, warehouses, local analytical engines, vector databases, graph stores, and search indexes are not competing names for the same thing. They are specialized pieces of a workflow that moves from source activity to decision.

Transactional versus analytical

The most important distinction is transactional versus analytical.

A transactional system answers: can we record and retrieve one business event correctly right now? The point-of-sale system must know the price, charge the customer, update inventory, and create a receipt. The CRM must record a sales interaction. The app database must know which user is logged in. Mistakes here interrupt the business.

An analytical system answers: what pattern emerges across many business events? The warehouse computes weekly revenue by region, demand by product, churn by cohort, margin by promotion, and service quality by store. It is not trying to record the next transaction. It is trying to make history comparable.

Managers feel this distinction in ordinary meetings. When the CFO asks for margin by promotion over the past six quarters, the answer should not come from the live checkout database. When customer support needs the current status of an order, the answer should not wait for the nightly warehouse refresh. Each system can be excellent and still be wrong for the job.

The major storage roles

SQL operational databases store structured app and business records: customers, orders, payments, products, subscriptions, tickets. They usually enforce relationships and consistency. If one customer has many orders, SQL is good at keeping that relationship explicit.

NoSQL systems serve flexible or high-scale application data: product catalogs, session state, user profiles, event payloads, documents, and other records whose structure changes often. They are often useful when the application needs fast reads and writes over flexible objects.

Data lakes and object storage keep raw or semi-raw assets: logs, vendor files, parquet tables, documents, images, audio, and historical extracts. The lake is useful when the firm wants to preserve data before every analytical use is known.

Warehouses and lakehouses make history analyzable. Systems such as Snowflake, BigQuery, and Databricks-style lakehouses are used to scan large historical datasets, join source systems, define metrics, and support dashboards, notebooks, and model training.

DuckDB-style local analytics gives analysts a fast, lightweight way to work with serious data on a laptop or in a reproducible script. This is useful for teaching, prototyping, case packs, and focused investigation before work becomes shared infrastructure.

Search, vector, and graph systems support retrieval and relationships. Keyword search finds exact or near-exact terms. Vector databases store embeddings so workflows can retrieve semantically related documents, products, customers, or images. Graph stores represent relationships such as referrals, product co-purchases, supply chains, account networks, and organizational structures.

The point is not to memorize product names. The point is to ask which system is doing which job.

Batch, streaming, and freshness

Data also differs by freshness.

Some workflows are fine with a nightly refresh. A weekly executive KPI dashboard, a monthly pricing review, or a quarterly market expansion analysis does not need every transaction within seconds. Other workflows need near-real-time data: fraud detection, stockout alerts, delivery routing, ad bidding, anomaly detection, or a customer-facing recommendation shown during a session.

Freshness has a cost. Real-time systems are harder to build, harder to monitor, and easier to over-trust. A manager should ask: what decision becomes better if this is refreshed sooner? If the action is weekly, minute-level freshness may only create noise.

How storage affects methods

The rest of the book repeatedly depends on storage choices.

• Dashboards need stable metric tables, not ad hoc extracts.
• Causal analysis needs historical data at the right grain, not only summary reports.
• Prediction needs labels, features, and timestamps aligned in a feature table.
• Recommenders need exposure logs as well as purchase logs, or they confuse preference with what the system happened to show.
• Retrieval-augmented generation needs a document store, a search or vector index, source metadata, and a way to evaluate retrieval quality.
• Governance needs lineage: where the data came from, who owns it, how fresh it is, and what changed since last time.

Storage tells us what is possible to ask. The next question is what the firm actually does with the answer.

How Data Is Used

Data creates value only when it changes a workflow. A dashboard no one reviews is a report, not a management system. A prediction score that triggers no action is a number, not intelligence. A recommendation system that cannot learn from exposure and feedback is guesswork with software around it. A language model that summarizes documents without evaluation is convenience without control. The practical question is always the same: what job is the data doing?

The executive question: what decision workflow will this evidence improve?

Modern data use falls into a small number of recurring workflow families. The names change by industry, but the managerial logic is stable.

Some workflows monitor the business: revenue, margin, churn, conversion, service time, stockouts, quality, risk. Some diagnose where a metric moved: by customer segment, geography, product, cohort, channel, store, or time period. Some learn causally: did a price change, campaign, policy, or process change cause an outcome? Some predict: which customers will churn, which orders are risky, how much demand should we expect? Some rank and recommend: what should be shown first, who should be contacted first, which action should be suggested next? Some read unstructured work: tickets, calls, documents, images, contracts, resumes, invoices, policy manuals. Some optimize a constrained action: staffing, inventory, routing, pricing, media spend, assortment, or scheduling.

Figure 3 is one of the book's central habits. Before debating tools, route the question.

Monitoring: what is happening?

Monitoring is the most familiar use of data. The firm defines metrics, refreshes them on a cadence, and watches for movement. KPI dashboards, scorecards, alerts, and operating reviews all live here.

Monitoring is useful when:

• the metric is clearly defined;
• the owner knows what action they can take;
• the refresh cadence matches the action cadence;
• the dashboard separates normal variation from signals that require attention;
• the view supports drilldown when a metric moves.

Monitoring fails when the dashboard becomes a collection of charts without a decision path. A useful dashboard answers three questions in sequence: what moved, where did it move, and what should we inspect or do next?

Diagnosis: where, for whom, and why might it be happening?

Diagnosis starts after monitoring notices movement. If weekly margin falls, the manager needs to know whether the issue is a product, store, region, customer segment, campaign, channel, or supply problem. This is where segmentation, cohorts, funnels, small multiples, maps, and drilldowns matter.

Diagnosis does not prove causality. It narrows the search. It tells the team where to investigate and what comparison might be useful.

Strategic decisions: where should the firm place its bets?

Strategy uses data differently from daily operations. The evidence is often less fresh, more aggregated, and more uncertain. Market expansion, pricing architecture, product portfolio, customer segment focus, channel strategy, capacity investment, and acquisition screening all require a blend of historical data, external context, assumptions, and judgment.

The practical discipline is to separate three things:

1. Facts from the current business. What do our customers, products, stores, channels, and margins show?
2. Assumptions about the future. What must be true for the strategy to work?
3. Tests and signals. What data would tell us early that the strategy is working or failing?

This is why the book returns to decision memos. Strategic decisions need evidence, but they also need an explicit threshold for acting and a plan for learning after action.

Causal learning: did the action work?

Many business questions are causal: did the email cause incremental sales, did the discount lift profit, did the new onboarding flow reduce churn, did the policy change reduce risk? Historical data alone often makes these questions look easier than they are.

The key issue is the counterfactual: what would have happened without the action? Experiments, A/B tests, difference-in-differences, synthetic control, regression with credible identification, and other designs are ways of constructing a comparison that earns the word "caused."

Causal learning is not always required. If the question is only "which stores are currently above target?", a dashboard is enough. If the question is "should we roll out this promotion nationally?", a dashboard is not enough.

Prediction, ranking, and recommendation

Prediction asks what is likely to happen next. Churn models, demand forecasts, fraud scores, lead scores, risk models, and delivery-time predictions all live here. The model does not need to know what caused the outcome to be useful, but it does need a clear action attached to the score.

Ranking and recommendation go one step further. They order choices: which product to show, which customer to contact, which ticket to escalate, which loan to review, which store to visit, which document to retrieve. Ranking systems need extra care because they shape the future data they observe. If a product is never shown, the system cannot learn whether customers would have liked it.

Generative AI and unstructured workflows

Many modern workflows use data that does not look like rows and columns: support tickets, product reviews, policy documents, sales calls, PDFs, images, screenshots, contracts, invoices, code, slides, and emails. AI workflows help classify, extract, summarize, retrieve, draft, route, and monitor this work.

The mature version is not "ask the model." It is a designed workflow:

1. collect the source material;
2. retrieve or select relevant context;
3. ask the model for a bounded task;
4. require structured output when the result must feed another system;
5. evaluate accuracy, grounding, bias, privacy, and refusal behavior;
6. route uncertain or high-risk cases to human review;
7. monitor the workflow after deployment.

AI is powerful because it makes language, documents, and images operational. It is risky for the same reason: it can make weak evidence look fluent.

Optimization: what should we do under constraints?

Optimization turns predictions, rules, and business constraints into an action plan. How many employees should be scheduled? Which stores should receive inventory? How should delivery routes be assigned? Which media channels should receive budget? Which price should be offered under margin and fairness constraints?

Optimization is often where analytics becomes real. It also exposes hidden objectives. Are we optimizing revenue, margin, customer satisfaction, utilization, fairness, retention, or risk? If the objective is wrong, the optimized answer is wrong with confidence.

Monitoring, diagnosis, causal proof, prediction, and AI workflows are not separate islands. They are stages of one loop — the loop that connects every part of this book.

The Data-to-Decision Loop

The modern data system is a loop. Human and machine activity creates records. Records move into storage. Storage feeds transformations, metrics, charts, models, search indexes, and AI workflows. Those evidence assets inform decisions. Decisions change customer experience, operations, pricing, policy, product, staffing, or automation. Those changes create new data. The loop starts again.

The executive question: where are we in the data-to-decision loop?

Most analytics mistakes come from losing track of the loop. A team starts with a model but has not named the action. A dashboard monitors a metric but has no threshold. A causal analysis estimates an effect but does not connect to a decision cadence. An AI workflow answers questions but has no evaluation or escalation path. Each failure is a broken link between data and action.

Figure 4 is the front-door operating model for the book. Part I begins in the source record. Part II turns records into visual evidence. Part III asks whether an action caused an outcome. Part IV predicts and ranks future cases. Part V brings text, documents, images, embeddings, and language models into the loop. Part VI asks how the organization runs the loop repeatedly without losing ownership, quality, or governance.

Data-driven versus data-decorated

A decision is data-driven only when three things are named:

1. The action. A specific lever someone can pull.
2. The comparison. What would happen if the firm did not act or acted differently.
3. The threshold. The signal, effect size, ROI, quality level, or risk standard required to act.

Anything missing one of these may still be useful description. It may be a good dashboard, a good analysis, or a promising model. But it is not yet a decision.

The decision ladder

The book's evidence languages climb a ladder of business questions. The lower rungs do not disappear when the upper rungs arrive. A prediction model still depends on a clean target. A causal design still depends on a well-defined unit, timing, and outcome. A language model workflow still needs source data, storage, evaluation, monitoring, and human review.

The ladder is a routing device:

• What happened? Read the data and define the metric.
• Where and for whom? Visualize, segment, compare, and diagnose.
• What caused it? Construct a credible counterfactual.
• How much does the lever matter? Estimate effects, elasticity, and heterogeneity.
• What is likely next? Predict, rank, and evaluate.
• What does the text, image, or document say? Use unstructured data and AI workflows.
• How do we operate this? Monitor, govern, communicate, and learn.

Six evidence languages

The important point is not the numbering. It is the discipline of choosing the evidence language that fits the decision. A dashboard should not be asked to prove causality. A predictive model should not be treated as an intervention. A language model should not be trusted because it is fluent. A causal estimate should not ship if no one knows what action it changes.

The artifacts that survive the work

The book does not end each method with "and now you know the method." It ends with an artifact that a firm can reuse, audit, and refresh.

The artifact family matters because modern analytics is not a sequence of one-off clever analyses. It is infrastructure for repeated decisions. A metric card can be reused in a dashboard. A predictive task contract can be reused by a modeling team. A model card can be used by risk, legal, product, and operations. An AI workflow card can be audited when the workflow changes. A decision memo can show what evidence led to action and how the firm will learn afterward.

The cases

One through-line company, Bean & Basket Coffee, appears throughout the book. Standalone cases add real empirical grounding where a specific method needs a richer dataset.

The purpose of the cases is not to decorate chapters. It is to make the loop concrete. Every case asks: what activity generated the data, where is it stored, what evidence language fits the decision, what artifact should survive, and how would the organization monitor what happens next?

Where Part I begins

Part I now begins after the full system is visible. We zoom in from the operating loop to the basic object inside it: a dataset. The first question becomes almost physical: what does one row mean? That question sounds small, but it controls everything that follows. Grain, structure, variable type, joins, reshaping, metrics, and data quality are the mechanics that make the larger loop trustworthy.