Wednesday, June 27, 2012

Is It Broken?


Note: Sorry in advance for the long post.  I hope you find it worth your reading time!



A reader posted this terrific comment: "I'm curious about your take on the performance so far. Is it a question of the system being misaligned with the modus operandi of the current market, bad money management, of something else?"

This is an outstanding question, because we've all been there - you are in the midst of trading a supposed "winning" system, yet losing money doing so!

This question is like an onion, with many layers.  So, let's start peeling...


First, the reader is asking because my contest system is losing money.  That losing can't be denied.  But notice how he asks the question - he is thinking/wondering something is wrong (misalignment, bad money management).  Makes total sense, of course.  If it isn't making money, something must be wrong, correct?

Well, maybe something is wrong, and maybe not.  As I write this, I honestly don't know the answer. We'll examine that together later.

But for now, that thinking brings us to the second layer on the onion.  Most people, myself included, usually assume something is wrong with a trading system when it loses money.  And once you convince yourself that something is wrong, the normal human response is to try to "fix" it.  

Change parameters, change instruments, change stops, add a filter etc. Just change SOMETHING!  Get it working right!

Usually, that is the wrong thing to do (unless the system is indeed "broken").  Making fixes as you go usually ends up like being on a hamster wheel - you go round and round, but never get anywhere.  You are chasing performance, and you end up making bad decisions because of it.  So the lesson here is to be aware of the questions you ask yourself - you may be assuming things which will lead you astray.

Why is trying to implement a fix wrong?  The answer is in the third layer of the onion - probabilities.  Many people don't realize that even a historically profitable system may have a prolonged losing streak in the future.  For any system, there is a chance - a non-zero probability - that you will lose most/all your money trading it.  

In the long run (years and years) the system might be profitable with 99% probability, but in the short run ANYTHING CAN HAPPEN.  And the shorter the time frame, the more likely it is that something bad will occur.  You have to give the probabilities enough time to kick in.  Think of a casino.  On any one spin of the roulette wheel, the casino may lose money.  But since the odds are in the house's favor, over many tables, and many spins, they (the casino) are almost guaranteed to win.


OK, so what does this actually mean?  Here is a real world example.  Unfortunately for me, I'll use my losing contest account.  This analysis will answer: "Is Kevin's contest system broken?"

For my contest system, I had previously determined through testing that on "average" (keyword here) should get something like this:

Avg Profit Per Trade = $206
# Trades per Year = 101
Avg Profit Per Year = $20,806

At this point, I should be up roughly $10,000.  Instead, I'm down around $2,000.

"Is this kind of actual performance possible, given these historical statistics?"

To answer the question, I'll use what is called Monte Carlo analysis.  Basically, you put all the trade results in a hat, and randomly pick one after another, until you've pulled 50 trades (in my case) out of the hat.  Adding these values up, you build an equity curve.

Obviously, if you do this one time, you might get lucky, pick all winning trades, and have a great equity curve.  Or, you might pull all losers, and have a terrible equity curve.  There is a low probability of each of these possible equity curves happening.  But, the point is ANYTHING CAN HAPPEN when you do this process once.

So, to get meaningful results, you repeat this process 100 or 1000 or 10,000 times.  Then, you can see some interesting things.

I built a Microsoft Excel spreadsheet to do this.  It is based on the simulator I offer free on my website.

Here are the results, with my system...


1.  Average Performance [Dark Gray line]  The chart below shows, on average, how my contest account should be doing.  After 6 months (50 trades), I should have about $10,000 in profit.  That agrees with the earlier analysis.  The line is a little wavy, due to the random nature of the simulation.







2.  Actual Performance [Blue line, with dots]  Here is my actual performance, overlaid on the same graph.  Yuk!  But, it is a great visual check of where I "should" be and where I am actually at.  Based on this, maybe there is a problem...







3.  95% Best Case Performance [Green Line]  If I run the simulation 100 times, some of the equity curves will look great.  This line is the 95% percentile line, meaning that there is only a 5% chance that the performance will be better than this.  If my performance was above this line, I'd be happy, but I'd also be wondering if something was "wrong."  Good "wrong," but still a concern.







4.  5% Worst Case Performance [Red Line]  Again, running the simulation multiple times gives you this curve.  It says there is a 5% chance that my performance would be below this line.  I'd be worried if it was below the line - something could be broken.





At this point, you might be thinking "All this analysis is ridiculous!  You still aren't making money!!!  What does all this goofy analysis tell you?!?"


Here is what I learned:

A.  My current performance is within the range of what is expected (above 5% line, below 95% line).

B.  My actual performance is at the 13% percentile. Meaning, I am currently pretty unlucky, and my antenna should be up that something may be wrong with the system.  Not panic time yet, though.  Not time to change the system.  But, I want to be above 50%, and until I get there, I'll be closely watching.

C.  If I want a "guaranteed" profit, I have to trade this system for at least 100 trades (imagine when the red 5% line crosses zero).  In other words, I have to be patient, and give the probabilities time to work.

D.  At this point, I am in the "major disappointment" zone, not in the "panic" zone.





I hope you found this long post useful.  It was useful to me!




10 comments:

  1. Long post! But worth the read : ) I like the annotation on your last chart, "What I Hope For" "Mild to Major Disappointment" and "Panic!"

    Kevin, the trades your system has taken, would you have made the same exact trades if you were manning the controls?

    ReplyDelete
  2. Thanks for the comment!

    Every time I try to "man the controls" via a discretionary method - and I've tried a bunch of different method over the years - I've run the car right off the road and crashed into a tree!

    But to answer the question: No, I would not have taken some of the trades I did, nor would I have stayed in some as long as I did. But would I be better off? Maybe, but I kinda doubt it.

    Up to this point in my trading life, mechanical or automated systems are the only thing that have really worked for me. But I hear another discretionary approach calling, with its siren song, so I know I will try it again eventually.

    Hopefully the next time I'll be much better at "manning the controls."

    ReplyDelete
  3. In the charts above, are you showing the results for 50 randomly chosen trades from the simulated system results or are you picking 50 random trades from your actual results?

    ReplyDelete
  4. Thanks for the question.

    For the Monte Carlo simulated curves, I used all the system historical trade data up until Dec 31, 2011. That trade data was based on in-sample and walk forward results. This represents the baseline of what the system should have done in 2012, which is why I used it.

    The blue curve with the dots represent the actual trade data from 2012.

    Really, what my analysis does is answer the question "are 2012 results from the same population/system as historical results?" There are other statistical tests that can be run to answer this question, too.

    ReplyDelete
  5. That answers my question, thanks. Really interesting stuff.

    ReplyDelete
  6. Thanks for answering my questions Kevin. Excuse my ignorance when it comes automated trading,or pretty much everything, I assumed automated trading was a traders method programmed into their trading platform.

    As for manning the controls, I have questioned my own ability not let my emotions take the car off the cliff. I like to refer to it as falling off the wagon.

    Personally I am working on a automated system that would take the same trades I do not have the patience or time to sit and wait to develop, the kind of trades I would have taken if I was manning the controls in a while in a good trading zone.

    ReplyDelete
  7. Hi Trin -

    You are right - automated systems are trader's methods (strategies) programmed into their trading platform.

    Mechanical systems are the same methods, but not necessarily automated.

    So mechanical systems can be automated or not automated, but the idea is still to follow the prescribed rules 100%.

    Many strategies (such as my contest system) are mechanical, but not automated.

    There are pros and cons to both automated and non-automated, even if both follow the strategy 100%.

    So your thinking was right on!

    ReplyDelete
  8. Hi Kevin,

    Fantastic, thought-provoking post! The "probabilistic (word?!) mindset" is very difficult to cultivate, at least it was for me...if one doesn't get that aspect of trading down pat, they'll always struggle IMHO.

    Is your system subject to high variance? That is, does it have a lower win rate with a higher reward:risk ratio? Running X number of trades through a simulator a given number of times should produce possible equity curves that are closer together when a lower variance system (high WR, low RR...) is sampled compared to one with higher variance.

    Thanks

    MM

    ReplyDelete
  9. Excellent point, MM! You are correct, the variance of trades in my system is probably pretty high.

    The historical stats for my system, over 600+ trades:

    win rate = 56.2%
    avg trade = $206
    avg win = $869
    avg loss = -$644
    std dev of trades = $1490
    Reward/Risk (avg win/avg loss) = 1.35
    expectancy = .32
    Van Tharp SQN (100 trades) = 1.38


    The killer here is the variability, or the standard deviation. It is high, and that causes a lot of scatter in potential results.

    Just a thought: has anyone out there ever tried to optimize on standard deviation?

    ReplyDelete
  10. Hi Kevin,

    A really fantastic post and very much appreciate the honesty in this blog. Your process is some what different from mine but I'll have to reserve my questions until I've gotten up to speed with the story from the start. Thanks for posting.

    1Lot

    ReplyDelete