Resources
Preparing data before any analysis
Preparation takes up the largest share of analysis work, and it decides the validity of everything that follows. Sources, formats, missing values, duplicates, consistency: each point is handled by a documented decision.
Every cleaning decision is documented
Replacing a missing value with a mean, discarding an outlier, merging two customer reference sets: these acts change the result, and whoever reads the analysis in six months must be able to find them. Traceability of transformations is therefore a methodological requirement as much as a convenience: it allows the preparation to be replayed on new data, two periods to be compared, and an objection to be answered by pointing at the exact line.
- An inventory of sources, with their owner and update frequency
- One single format per value type: date, amount, identifier
- An explicit decision on each missing value, applied uniformly
- A written deduplication rule, tested on known cases
- One reference set per business object, aligned across systems
The order of operations
- 01
Inventory and date
Where each column comes from, who produces it, how often. This inventory prevents impossible joins.
- 02
Standardise formats
Dates, amounts and identifiers brought to a single notation. Half the discrepancies vanish here.
- 03
Handle the gaps
One written decision per column. A missing value is sometimes the most interesting information in the set.
- 04
Deduplicate and reconcile
A rule tested on known cases, then applied. Cross-system reconciliation is verified by sampling.
- 05
Document and replay
Preparation becomes a replayable routine. That turns a one-off analysis into regular monitoring.
The weight of preparation in the catalogue
- hours on data quality
- 200
- hours of data engineering
- 240
- hours in the decision-makers' certificate
- 40
Dedicated block of the business intelligence programme.
Extraction, transformation, quality, orchestration.
The same subject, reduced to its operational core.
Questions about preparation
What share of the time does preparation take?
The largest, and that is a design fact rather than a surprise: a data project's schedule therefore books that time explicitly. A project that underestimates it delivers an analysis whose figures nobody can defend.
What should I do with a half-empty column?
Examine it before treating it: absence often follows a logic — an optional form, a period before a system went live. That logic is sometimes the most interesting information in the set, and it is documented just like the decision taken.
How do I reconcile two customer reference sets?
With a written rule, first tested on cases whose answer you know, then verified by sampling on the rest. The data quality block builds that rule and its traceability, on real datasets.
Explore next
- DashboardsThe starting question, chart choice, visual hierarchy, thresholds and alerts, distribution: building a dashboard that triggers decisions.
- Misleading analysisSelection effects, chosen denominators, correlation versus causation, truncated axes: the four checks to run before accepting a figure.
- Machine learningRegression, classification, cross-validation, model selection and interpretability: what a model actually does and how it is validated.
- Models in productionInference service, drift monitoring, retraining, running cost and the trade-off between local and cloud processing.
- Data governanceData ownership, life cycle, hosting, retention, reversibility and vendor dependence: framing governance.
- Personal dataPurpose, legal basis, minimisation, retention, security and individual rights: the processing register in practice.