Anticipate the four standard ways enterprise data lies to you
Choose between reading from source systems and building a staging layer, and defend the choice
Apply ontology thinking: model the business's things, not the systems' tables
Build pipelines that fail loudly and partially instead of silently and totally
The discovery lesson taught you to audit data before promising anything. This lesson is about what happens after the audit, when you have to actually build on what you found.
Every FDE engagement eventually reaches the same scene: five systems that disagree about something as basic as what a customer is, and your job is to build one working thing on top of all of them. Palantir's engineers spend a large share of their field time on precisely this, and their platform's central idea, the ontology, came out of that experience.
Enterprise data lies in patterns. Knowing the patterns turns a month of surprises into a checklist.
The same word means different things. Revenue in the sales system books at contract signing; revenue in finance books at invoice. Both columns are called revenue. Join them and you get numbers that are confidently, invisibly wrong.
The same thing has different names. One physical customer exists as three IDs across CRM, billing, and support, spelled two ways. Until you resolve identity, every per-customer metric is fiction.
The field exists but the values do not. The audit taught you to count nulls. Now add the sneakier version: placeholder values. A date column where 1900-01-01 means unknown. A status where 99 means whatever the retired system needed it to mean.
Your Reflection
Saves automatically
What’s one thing you learned? What’s still confusing?
The pipeline exists but the freshness is theatre. The dashboard says the warehouse syncs nightly. The sync job has been failing silently on one table since a schema change in the spring, and the numbers on that table are from April.
What Do You Think?
You join claims data to customer data and present total claims per customer. The operations lead says the numbers look roughly double reality for the biggest accounts. What is the most likely cause?
The first architectural fork in every integration: query the customer's systems live, or copy data into a layer you control and build on that.
Reading live is fast to start and honours freshness by definition. It is also fragile in someone else's hands: their maintenance windows break your demo, their rate limits throttle your pipeline, and every query you run lands on systems that other people's jobs depend on. Nothing ends goodwill faster than your prototype slowing down the claims team's core system on a Monday.
A staging layer costs setup time and introduces staleness you now own. In exchange you get isolation, reproducibility, and a place to fix the four lies once instead of in every query. Eval sets also need frozen inputs, which live systems cannot give you.
The field default is boring and correct: a small staging layer, refreshed on a schedule you publish, holding only the tables your scoped workflow needs. Not a data lake. Not a platform. The minimum copy that makes your one workflow reliable, with its refresh time printed on every screen that shows its data.
Palantir's core product insight, distilled: model the business's things, and map each system's tables onto those things, rather than building directly against tables.
Concretely, you define the small set of concepts your workflow actually reasons about: Customer, Claim, Policy, Adjuster. For each, you write down where its truth lives, which is usually a precedence rule across systems: identity from the CRM, balances from billing, contact history merged from both, conflicts resolved by recency. Your pipeline and your prompts then speak only ontology language, never table language.
The payoff is leverage. When the customer replaces their CRM next year, you remap one layer instead of rewriting every query and prompt. When an LLM answers questions about a claim, it reasons over one clean object instead of five raw rows. And the precedence rules, written down, become the first honest documentation of the customer's data that anyone there has seen. Sharing that document builds more credibility than any demo.
Field pipelines run in hostile territory: upstream schemas change without notice, sources go down for maintenance nobody mentioned, and the volume triples during month-end close. Two properties keep them alive.
Loud failure: when an assumption breaks, the pipeline says so, specifically, to a human, at the moment of breakage. The silent alternative is how April data ends up in a July decision. Cheap validation at every boundary: row counts within expected range, nulls below threshold, no dates from the future, checked on every run, alerting on violation.
Partial failure: one bad source degrades one slice of the output, clearly marked, instead of taking the whole system down. The claims feed being late should not stop yesterday's processed claims from being visible. Design the output so each slice carries its own freshness, and stale slices say so on the screen.
Both properties echo the eval-first discipline from the last lesson. In hostile territory, the systems that survive are the ones that measure themselves.
Quick Check1 / 4
Sales and finance both have a column called revenue, with different booking rules. What integration failure does this create if joined naively?