A Practical Guide to Historical Data Analysis

Historical data analysis is the disciplined description of what has already been recorded: how values were distributed, how they changed over time, and how one period or group differs from another. Most of the work is not statistical in the narrow sense. It consists of establishing what the data actually contain, aligning them to a consistent calendar, and making comparisons that are fair. Errors at these early stages propagate silently into every chart and summary that follows.

This guide is organized as a sequence an analyst can follow on any time-stamped dataset. It closes with a section on interpretation, including the point most often misunderstood in practice: describing past frequencies is not the same as predicting future outcomes.

Before computing a single statistic, profile the dataset. The goal is to know its boundaries and its defects, not to fix everything immediately.

1. Coverage: what are the first and last dates? Do they match what the source claims?
2. Completeness: how many calendar days in the range have at least one observation? A simple ratio of days with data to calendar days reveals gaps quickly.
3. Duplicates: are there repeated identifiers, or identical rows that were loaded twice?
4. Valid ranges: are there negative values where only positive values make sense, or values orders of magnitude away from the rest?
5. Units and precision: is the unit consistent over time? A switch from seconds to minutes, or from one decimal to none, will show up as an abrupt shift in level or in the number of distinct values.
6. Category labels: are the same groups spelled consistently ("North", "north", "N")? Inconsistent labels split one group into several.

For example, a 90-day range with observations on 84 distinct days has coverage 84 / 90 ≈ 93.3%. That number alone does not say whether the gaps matter. The next step is to find out where they are.

Missing data are rarely missing at random. Sensors fail during storms, reporting stops on public holidays, systems lose records during migrations. When the probability of a value being missing depends on the value itself or on something related to it, summaries computed from the remaining data are biased.

Worked example: missing weekends

Take a daily demand series in which weekdays average 500 MWh and weekend days average 460 MWh (8% lower). Over four complete weeks there are 20 weekdays and 8 weekend days:

Now suppose four of the eight weekend days are missing. The mean of the remaining 24 days is (20 × 500 + 4 × 460) / 24 = 11,840 / 24 ≈ 493.3 MWh, about 4.8 MWh higher. Nothing about demand changed; only the composition of the sample did. Coverage of 24 / 28 ≈ 85.7% looks acceptable, yet the mean is biased upward by roughly 1%.

Practical responses, in order of preference:

• Report the gap and analyse the complete periods separately.
• Compare like with like: weekday means with weekday means, or use a weekday × month breakdown so that composition is held fixed.
• Impute only with a stated method, and never mix imputed and observed values without marking them. Descriptive work should normally show what was observed.

A date is only meaningful relative to a time zone. An event recorded at 22:30 on 31 March 2024 in São Paulo (UTC−3) occurred at 01:30 UTC on 1 April 2024. Depending on which convention the pipeline uses, it belongs to March and Q1 or to April and Q2. When records from several systems are combined, inconsistent conventions shift observations across day, month and quarter boundaries.

Insider Magazin stores every observation as a UTC calendar day (ISO format YYYY-MM-DD) and performs all date arithmetic in UTC, independent of the viewer's local time zone. If your source records local timestamps, decide on the conversion before the data are loaded, and document it.

Calendar effects

• Months differ in length. February 2024 had 29 days, March 31, and a 31-day month has about 11% more days than a 28-day February. Monthly totals differ by that much for this reason alone; monthly means per observation do not.
• Weekday composition differs. A month with five weekends has a different mix of days than one with four. For data with a weekly cycle, this moves monthly means.
• Same date, different weekday. 1 March 2024 was a Friday; 1 March 2025 was a Saturday. A day-by-day year-over-year comparison compares different weekdays.
• ISO weeks cross year boundaries. 1 January 2023 was a Sunday, so it belongs to the ISO week starting Monday 26 December 2022. Weekly aggregates at the edges of a range are often partial weeks.
• Leap years. 2024 had 366 days. Annual counts are not directly comparable to 365-day years without normalizing.

Periods are almost never the same size. Quarters have 90 to 92 days, data collection may have started midway through a year, and gaps reduce some periods more than others. Comparing raw counts across such periods confounds the size of the period with the distribution of values within it.

Suppose response times are grouped into four classes. Period A contains 90 observations, Period B only 45 because of a data gap.

By raw count, the slowest class looks unchanged (9 in both periods). By relative frequency, its share doubled from 10% to 20%. The relative view is the correct one for comparing distributions. The raw counts remain useful as a reminder of how much data supports each figure: 9 out of 45 is a less stable share than 18 out of 90 would be.

Two further rules keep period comparisons fair. First, use identical class edges for every period; bins computed separately for each period cannot be compared row by row. Second, for summary statistics, pair the difference in means with a measure of spread. A standardized effect size such as Cohen's d (difference in means divided by the pooled standard deviation) puts the difference in context. It is descriptive, not a significance test.

Historical data describe what was recorded, which is not always what happened. Selection bias arises when the process that put records into the dataset is related to the values being studied.

• Survivorship bias: only entities that still exist are in the data. A table of currently active products, branches or funds excludes those that were discontinued, usually because they performed poorly, so historical averages look better than reality.
• Reporting thresholds: some systems only record events above a size limit or after a status change. Small or cancelled events are absent, which shifts the distribution.
• Changing definitions: when the definition of a category changes, the series before and after the change describe different populations even if the label is the same.
• Sampling convenience: data collected where collection was easiest (one channel, one region, business hours) do not represent the rest.

No statistic can correct for data that were never collected. The defence is documentation: knowing how each record came to exist, and stating the population your summaries describe.

Historical frequencies describe the past. Whether they say anything about the future depends entirely on the process that generates the data. Where outcomes are driven by persistent structure, such as seasonal demand or a long-term trend, past patterns can be informative, although extrapolating them is a forecasting task with its own methods and uncertainty. Where outcomes are independent random events, past frequencies carry no information about the next outcome at all.

Independence and the gambler's fallacy

For independent events, the probability of the next outcome is unchanged by any sequence of previous outcomes. A fair coin that has landed heads five times in a row still lands heads on the sixth toss with probability 1/2. The probability of six heads in a row, judged before any toss, is (1/2)⁶ = 1/64 ≈ 1.6%, but once five heads have occurred, that history is fixed and irrelevant to the sixth toss. Believing that a value is "due" because it has appeared less often, or "hot" because it has appeared more often, is the gambler's fallacy.

The law of large numbers is often misread as a balancing mechanism. It is not. If a fair coin shows 10 heads in its first 10 tosses, the expected number of heads in the next 990 tosses is still 495. The expected total is 505 out of 1,000, or 50.5%. The proportion approaches one half because the early excess is diluted by many new tosses, not because later tosses compensate for it.

How large is ordinary variation?

Even for perfectly uniform independent outcomes, observed frequencies differ. If a fair six-sided die is rolled 600 times, each face is expected 100 times, and the count for any one face has standard deviation √(600 × 1/6 × 5/6) ≈ 9.1. Counts of 88 or 112 for some faces are therefore unremarkable, and they say nothing about which face will appear next.

Insider Magazin is built on this distinction. It is descriptive software: it summarizes, compares and visualizes recorded data, and it does not forecast outcomes or suggest that past patterns determine future events.

The workspace supports each step of this checklist. The overview shows total observations and dataset coverage for the selected range. The time series and weekday × month heatmap reveal gaps and calendar effects. The frequency-by-period heatmap reports relative frequencies, so each year or quarter sums to 100%. The comparison view puts two periods or categories side by side on identical bins, with mean difference, relative change and Cohen's d.

Explore these views in the interactive demo, review the date conventions and formulas in the methodology, and see the documentation for a description of each panel.