But George, you've clearly identified the weakness with pure index investing: there are factors beyond simple market cap
in play, and thus, indexes are actively managed.
As far as I can see, the main other factor used by S & P is liquidity: float, corporate form, etc, etc.
But we've known that for a long time. Indeed, private equity -- perhaps the least liquid form of investment by our pension plans -- is considered a separate asset category, quite different from public equity. Companies omitted by S & P mostly fall somewhere along the spectrum between General Electric and Dick's Sporting Goods.
Your contention is that in aggregate, all investors receive the market return. But which market are we talking about?
A very good question.
My view is that there is not just one market, but rather multiple markets: publicly traded equities, private equity, REITs, preferred, bonds and other forms of fixed income, commodities, art and other collectibles, etc. Each investor defines his or her reference market as a combination of these. How widely should I go? The trade-off is the following: A wider definition of market means more diversification. But to get there I have to go into investments that are less liquid and with higher transaction costs (think of the bid-ask spread on paintings).
So when I write about "indexing" and beating (or not) the index, I'm really talking about just one asset category, i.e. equities that are publicly traded on organized (more or less) exchanges. Personally, I adopt the definition used by VT:
The investment seeks to track the performance of a benchmark index that measures the investment return of stocks of companies located in developed and emerging markets around the world. The fund employs an indexing investment approach designed to track the performance of the FTSE Global All Cap Index, a free-float-adjusted, market-capitalization-weighted index designed to measure the market performance of large-, mid-, and small-capitalization stocks of companies located around the world.
Clearly more diversification would be desirable. But I'm a very small individual investor, and my investment amounts cannot justify the expense of, say doing due diligence on private equity. Only the big boys can do that effectively.
Active management may or may not do better by diversifying more widely. Recent experience of hedge funds suggests otherwise, but that's an empirical question, and the answer may change going forward. However, if active management limits itself to the asset category "publicly traded equity", it is simple arithmetic that active management as a whole cannot have resu8lts much different from the index (before expenses).
The plural of anecdote is NOT data.