here are some notes about sampling, especially where it helps us map the terrain we're exploring, and where it fails us. It all comes down to the terrain and its "mappability".
Think of sampling as collecting partial views of a whole (the terrain) and synthesizing it into informative data points about this whole: mean, median, range, standard deviation, correlation, etc.
In this discovery process, inherent characteristics of the object being discovered (a.k.a. the terrain) determine the value or futility of our sampling.
terrain 1: fully visible
This is the realm of Probability 101 in college. The structure of the terrain is fully visible and symmetric, and the distribution can be known in advance through inspection and deduction.
Sampling, therefore, is not needed. Examples: flipping coins, rolling dice, playing roulette.
terrain 2: invisible, mild, and immutable
The typical example of this terrain is the urn that contains colored marbles in unknown proportion. Sampling helps us here: we draw a marble, record its color, return it, shake the urn, and repeat.
Due to the stability and tractability of the terrain, after a number of draws, the observed frequency converges to the true proportion of the contents of the urn.
terrain 3: invisible, mild, but altered by the observer
Here we take the urn from terrain 2 but we draw marbles without placing them back in. In this case, every draw changes the urn, and the color distribution of the remaining marbles depends on the prior draws.
Things are becoming more interesting... but not dangerous. Sampling is still useful in this terrain. In fact, because we are the only ones "messing" with the urn and have a record of what's been drawn, we can achieve higher confidence than on terrain 2, as each draw reduces the number of remaining marbles.
In theory, because the terrain is finite, sampling in terrain 3 terminates in certainty because the urn will be empty.
terrain 4: invisible but unstable
Here the composition or characteristics of the terrain change while we sample. This time not through our recorded removals, but because the terrain itself is unstable.
Financial markets and complex systems live in this terrain. The forces that drive the economy are many and their relative importance and interconnectedness ever-changing. Imagine trying to forecast inflation in the 1970s.
Beware of sampling here.
terrain 5: invisible and wild
Terrain 5 can be stable or unstable, but its defining characteristic is fat and wild tails. The middle of the distribution is quiet. Sampling will show you easy terrain until one single observation from the tails moves your sample mean by 10x your expectation.
Consider 100 samples of 1000 people, with net worth as the element in question. After averaging people's net worth across all samples, you estimate $150K as the mean with, say +/- $50K as standard deviation. You confidently make the following bet: the next sample's mean will be less than $300K. You gave yourself 3 standard deviations of "protection". At 999 people, the sample mean is right at $150K... then a random billionaire enters the sample. It's just one guy, just one observation. Let's say his net worth is $5B.
What happens to the mean? It moves from $150K at 999 folks to ~$5.15 million. You missed by 34 times and you thought you were well protected with three standard deviations.
Financial markets live here too. Wild and unstable.
Sampling here is much less informative than our intuition suggests.
terrain recognition and margin of safety
I believe the very first step should be to form a hypothesis about the type of terrain we are stepping into. This is imperative before sampling even enters the picture.
The more uncertain the terrain, the less we can rely on sampling/inference and the more structural (not statistical) protection we need: our friend margin of safety, position size limits, drawdown simulations in excess of observed data.