Beating the Market: Yes, it can be done

By The Economist online

On October 13th it was announced that the 2013 prize in economic sciences in memory of Alfred Nobel had been awarded to the American economists Eugene Fama, Lars Peter Hansen and Robert Shiller. All three were awarded the prize for their work on the “empirical analysis of asset prices”. Mr Fama and Mr Shiller were first mentioned together in The Economist in an article in 1992, which we republish below.

The Economist | Dec 5th 1992

For nearly 40 years economists have preached that investors can earn above-average returns only by taking extra risks or by striking lucky. Recent evidence suggests that clever investors can outwit the market after all

The death, when it was announced this year, sent shock waves along Wall Street. Life would never be the same again, wailed the obituaries. Had some star analyst passed away, or the head of a top investment bank? Or maybe another crooked company boss had slipped from his yacht? In fact, the deceased was not a he or a she, but an it. “Beta”, screamed the papers, was dead.

Beta? Why would the average Wall-Streeter mourn the second letter of the Greek alphabet? Beta is one of the best-known measures of risk used by investors. It plays a key role in lots of tricky financial activities, from portfolio management to pricing derivatives, working out the cost of capital and deciding if a firm should go ahead with its investment plans, calculating or even setting top executives’ pay. Without beta, these tasks are hard indeed.

It now seems that the reports of beta’s death were much exaggerated. The press based its sensational headlines on a study of share-price changes by two well-know financial economists, Eugene Fama and Kenneth French, of Chicago University*. But, says the academic grapevine, several other economists are soon to publish work contradicting Messrs Fama and French. Beta lives on. Yet it is wounded: Wall Street’s faith in it has been shaken.

Investors may soon be grappling with even bigger doubts. Beta is the most familiar product of what economists call the “theory of efficient markets”. Disputes about beta are just part of a growing debate about that whole theory. Many economists now question whether financial markets can any longer be called efficient.

Welcome to the real world, many people will conclude. To them, inefficient markets are a plain fact. What, after all, could be less efficient than the crashing stockmarkets and lurching exchange rates that are now all too familiar? This misses the point, however. When economists talk about efficiency, they use the word in a precise sense quite different from what most people mean by it. And until recently the economists’ definition explained real-world events pretty well. It also underpinned much of what investors do.

Walking at random

Efficient-market theory was born in 1953. Maurice Kendall, a British statistician, found that, instead of behaving in predictable ways, share and commodity prices followed a “random walk”. At any moment it was impossible to tell what prices would be a moment later. This, said theorists, is because prices are “efficient” –they reflect all available facts. Future prices differ from current prices only if buyers or sellers get new information. This, by definition, is unpredictable (or “random”).

But why should prices be efficient? Put simply, if they are not, it means the market is ignoring price-sensitive information. But this gives whoever has that information a chance to make big profits by trading on it. As soon as he does so, the overlooked information is incorporated in the price. This will make it “efficient”.

This led to the theory’s main claim. If prices are unpredictable, investors should not be able consistently to beat the market. Share-picking and foreign-exchange speculation should not pay. Put more precisely, one investor should be able to earn higher returns than another one only by taking bigger risks—markets reward risk-taking—or if he is lucky (and luck rarely lasts). After allowing for differences in asset-riskiness, no investor would regularly earn higher (“excess”) returns than the herd.

By the end of the 1960s, the theory had been refined. In principle, all markets could be efficient, whatever the product traded. In practice some markets are more efficient than others. Liquidity—how easy it is to buy and sell—boosts efficiency. So do visible, up-to-date prices, which help people to spot wrong prices fast. So does the availability of many similar products, so that relative prices can be compared. Financial markets enjoy all these advantages, and so are more likely to be efficient than are most.

Economists also graded and defined the efficiency of markets according to the information that prices reflected. “Strong” efficiency means that all relevant information—whether published or not—is reflected in prices. With “semi-strong” efficiency, prices reflect all information that is publicly available. With “weak” efficiency, the current price reflects, at least, all information contained in earlier prices of that asset. This last kind means that looking at past and current prices should tell you nothing about what will happen to prices in the future. If price changes follow predictable trends, the market is not even weakly efficient.

Economists soon learned that strong efficiency was to be found rarely, if at all. But numerous studies backed up Mr Kendall’s results, finding that financial markets displayed both weak and semi-strong efficiency. Efficient-market theory was built into a host of practical theories and investment strategies. The best-known are the capital-asset-pricing model (which created beta), the arbitrage-pricing theory and the Black-Scholes option-pricing model. The effect of such theories was huge. By the mid 1980s they had transformed the way people (or, at least, professional investors and financiers) understood and used financial markets. It was they that drove the rapid growth in futures and options markets and swaps, for instance.

Calendar effects and other flaws

While the influence of efficient-market theories soared on Wall Street, however, academics started to have doubts. Faster, cheaper computers and bigger databases made possible more and better research. This had a simple aim: to unearth trading strategies that would, in the past, have produced what economists call “excess returns”— returns that are above average even after allowing for the investment being riskier (or less risky) than average.

A first wave of evidence, in the mid-1980s, cast doubt on semi-strong efficiency, finding cases where information was publicly available yet returns were predictably high (or low). “Calendar effects” were the most striking: at certain times of the year and on certain days of the week, prices predictably rise (or fall) by more (or less) than on the average day, and the related returns vary correspondingly.

One example is the “January effect”: shares of small companies beat the market in January. There is a “turn-of-the-month effect”: all prices tend to rise by more than average on the last trading day of the month and on the first three of the new month. They do the same on Friday, but fall on Monday—the “weekend effect”. There is a “holiday effect”: prices rise more than average on the day before a public holiday.

Robert Shiller, of Yale University, looked at one sort of information that affects share prices—changes in dividends. He found that the market overreacted to these. It was far more volatile than efficient-market theory would predict. Others have looked for simple trading rules that make extra profits. In stockmarkets, for instance, it has long paid to buy shares with a below-average ratio of market price to book value. This “value investing” was practised by skilled investors like Warren Buffet well before economists rumbled it.

In a new study** based on many years’ data, Ken Froot, of Harvard University, shows that short-term interest rates forecast returns in foreign-exchange, stock, bond and commodity markets at the same time. His simple rule? A fall of one percentage point in (annualised) short rates is usually associated with an extra three percentage points in (annualised) excess returns to investors trading in accordance with that interest-rate change.

Others have found profitable trading rules that are based solely on tracking changes in the price of an asset over time. These suggest that markets are not even weakly efficient. Stephen Taylor, of Lancaster University, found that several rules based solely on trends in past exchange rates produced above-average returns in foreign-exchange markets over the ten years to December 1991. Take, for instance, the “double moving average” rule. The trader uses a short and a long moving average. He sells when the shorter average falls below the long-term average, and buys when the short-term average exceeds the longer one. This rule produced average annual returns of 14.2%, compared with an 8% average return on American Treasury bills.

Most of the work on weak efficiency has been far from simple. The current fad among finance economists is to use complicated computer models that churn long series of prices until they find predictable patterns. Trading on the basis of these patterns, only possible with a big, fast computer, would have brought investors excess returns.

Academics have also found that market volatility varies predictably. If prices are jumpy one day, they will probably be jumpy the next day too; if they are flat one day, they will probably be flat the next.

Reassuringly, some of the more blatant effects weakened after academics drew attention to them. But enough evidence remains to make a strong case against the efficient-market theory. Armed with the right rules or computer software, investors can, it seems, earn juicy excess returns—as can (and do) many of the academics who discover them, whether by trading directly or taking jobs as investment advisers.

Rethinking risk

Yet few economists are prepared to abandon the efficient market. As Charles Goodhart, of the London School of Economics, points out, no one has thought up a better theory. Instead, academics have tried to reinterpret the awkward evidence in less threatening ways. They have shown that, at times, trading costs would have wiped out apparent excess. And, more basically, they have rethought risk.

The simple efficient-market rule, remember, is that investors should not be able to earn excess profits after account is taken of the riskiness of their investment relative to the average. Most studies assume that the “price of risk”—how much extra profit an investor needs to expect before he will take extra risk—never changes: offer an investor an identically risky investment at two different times and he will want the same amount of extra profit each time.

But what if the price of risk varies over time? For instance, as people get richer they may be more willing to take risks; in a recession, less so. If the price of risk varies, what studies show as “inefficient” excess returns might, if they were earned, say, during a recession, turn out really to have been “efficient” average return; the returns would indeed be higher, but only because the risk component in them was larger.

Academic journals have been filled to bursting over recent years with papers explaining away excess returns by varying the price of risk with papers arguing the opposite. Alas, no clear conclusion has emerged. That may be inevitable. Measuring changes in the price of risk is mostly guesswork, Yet these guesses largely determine a study’s results.

This may, then, be a blind alley. Indeed, so may much of the empirical research on both sides of the efficient-market debate. In a recent paper, Fischer Black, co-author of option-pricing theory and now with Goldman Sachs, launched a fierce attack*** on “data-mining”: people digging into data until they find what looks like a trend. For instance, some academics have claimed that shares of small firms consistently outperform those of big firms, an apparent inefficiency.

To Mr Black, this “sounds like people searched over thousands of rules till they found one that worked in the past. Then they reported it, as if past performance were indicative of future performance. As we might expect, in real life the rule did not work any more”. Needless to say, Mr Black prefers theory.

The hunt for losers

Can efficient-market theory and evidence of inefficiency be reconciled? Many economists now think so. If some people gain from market inefficiency, they point out, others must lose from it. And though it is obvious why winners stay in the market, it is not clear why the losers do so. So theorists should find out who the losers are, and try to understand why it is that they are willing to lose. In short, allow that the way markets work is efficient, but that some of the people investing in markets are not.

This may make efficient-market theory a more useful tool than it was when used simply as a rod to beat inefficiency-finding heretics. For instance, show persistent losers that—and why—they are losers, and there is a chance they may do something about it.

This approach means changes to the theory, though. Economists (like bookmakers long before them) now distinguish two types of punter: “smart”, informed traders and “noise” traders. Smart traders consistently earn excess profits, because noise traders consistently lose. In early versions of efficient-market theory, arbitrage—trading by smart investors—would have driven noise traders out of the market and kept prices efficient. Now economists think complete arbitrage unlikely, not least because the smart money available is not unlimited.

This has a curious knock-on effect. Without complete arbitrage, there is no reason to expect smart traders to keep prices rational. If they know that noise traders are unduly bullish, it may pay them to jump on the bandwagon rather than bid prices down. This produces an added complication: an efficient price after smart trading may now look to economists like an inefficient one.

The search for persistent losers has only just begun. Though some likely candidates have been lined up—central banks, individual investors, company pension funds—most studies are hotly disputed. The problem is to find plausible explanations why particular groups should lose.

One tack is to identify institutional factors that may lead investors to take bad decisions. For example, most central banks are affected by political pressures. They are probably the main long-term losers in the foreign-exchange markets, reckons Lancaster University’s Mr Taylor. Huge losses by the Bank of England on this September’s “Black Wednesday”—when sterling had to drop out of the European exchange-rate mechanism—seem to confirm this, though the evidence is blurred by central banks’ reluctance to reveal their blunders.

Another sort of pressure may distort investment by company pension funds. The sponsor of the fund hires professional managers to do this investing, and reviews their performance each year. But even good managers are likely to do badly in one year or another. So—argues Bill Sharpe, the Nobel-prize-winning author of the capital-asset-pricing model—the managers may over-invest in those shares that they can most easily justify to the sponsors. These shares may be overpriced, so making underperformance more likely; but if the fund does underperform, it is a lot less embarrassing to meet sponsors with a fistful of Microsoft shares than with shares in unknown firms.

Psychology may explain some losers. Consider closed-end mutual funds (known in Britain as investment trusts). These invest in shares, and the price of their own shares should—you might think—almost exactly reflect the market value of those underlying assets. In fact, though, they usually trade at a discount. How can one account for this blatant market inefficiency? Richard Thaler, of Cornell University has an answer. Investors in closed-end funds are almost all individuals; the discount is a good indicator of their collective optimism (or pessimism) reckons Mr Thaler. Shifts in the discounts of closed-end funds are paralleled by changes in the price of other assets in which individuals invest heavily. In other words, investment by individuals may be mood-driven, rather than having much to do with the merits of what they invest in.

Onward to greater-efficiency

The world, then, is not as simple as economists once thought. The possibility of making long-term profits by playing the foreign-exchange markets or by picking stocks can no longer be ruled out on the grounds of principle. It may even be that the trendy, high-tech trading software, based on “neural networks” and “artificial intelligence”, that is now selling well on Wall Street, is not as gimmicky as it seems. But that does not mean that anything goes. Far from it. Rather than dismissing markets as inefficient, studies are likely to pay growing attention to losers, with the possibility that these losers will learn and adapt.

Nor does proof of inefficiency in markets mean, as some people believe, that governments should intervene, especially at times of market turbulence. Government action will probably just create more noise. Black Wednesday shows how efficiently markets can mug inefficient governments and their lackeys.

One last point. Though evidence of market inefficiency has soared in recent years, that does not mean markets have become less efficient. More likely, it means academics have got better computers and bigger databases, and that there are more of them data-crunching. In fact, most economists believe financial markets are more efficient now than ever before, thanks not least to shrinking official intervention. Theorists may argue about beta or market efficiency, but investors should not imagine that making money will get any easier.

* “The cross-section of expected stock returns”, by Eugene Fama and Kenneth French,

The Journal of Finance

, June 1992.** “Short rates and expected asset returns”, by Ken Froot,

The Journal of Finance

, forthcoming.*** “Estimating expected return”, by Fischer Black, Goldman Sachs.