C26 1 hr 30 min
Data pipeline: acquisition to mining
Acquire, cleanse, transform, and mine data so it becomes useful.
The four stages
| Stage | What happens |
|---|---|
| Acquisition | Collect raw data from sensors, logs, or APIs |
| Cleansing | Remove duplicates and fix missing or invalid values |
| Transformation | Normalise units, derive fields, aggregate |
| Mining | Find patterns, thresholds, and relationships |
Why order matters
Dirty data produces misleading results. Cleansing before transformation prevents unit errors from spreading, and transformation before mining makes patterns comparable.
Example
Raw temperature readings might arrive as strings with missing values. Cleansing converts them to numbers and drops gaps; transformation converts Fahrenheit to Celsius; mining finds periods above a threshold.
Check your understanding
- Explain the difference between cleansing and transformation.
- What could go wrong if you mine before cleansing?