Line Your Own Pockets

Michael: 00:00

Hello, everyone, and welcome to another episode of Line Your Own Pockets. Today's gonna be a fun one, because it's gonna be... We're buzz kills, I think I think we're gonna be today. And this is this is kind of what do you do when you run a back test and it just looks amazing. It looks absolutely perfect.

Michael: 00:18

You have like a profit factor of 10, and the line straight up with no pullbacks at all. And you're like, I hacked the matrix. I I'm sure you have as well, Dave. In my early years, I would get so excited. Now I get I get bummed, because I know there's something wrong, and I gotta go figure out what it is.

Michael: 00:35

But we'll give you some ideas of things to look at when you've tested something that's a little bit too good to be true.

Dave: 00:42

Yeah. I think we should probably call this one... I think Buzzkill's a good one, or... So I was... One of the guys I was coaching showed me Backtest, and I sort of burst his bubble, and he and he said, Dave, thank you for bursting my bubble.

Dave: 00:59

Like, so it's... It is a very frustrating thing because you often look at a backtest, and you'll see, wow, this this curve looks amazing. And a lot of times, it has, like, a profit factor of 10, but oftentimes, it's way more subtle than that. It, like, looks reasonable, so it's really hard. And this is a very common problem for for people to start backtesting because it's so easy to fall into the trap of several things we're gonna mention here.

Dave: 01:29

And the earlier you can find out about the issue, the better off you're gonna be.

Michael: 01:35

Well, and, you know, I I think we kinda covered one of the benefits I wanted to talk about of just doing this for a long time, is you'll only get super disappointed because you think you've hit something right a couple times. So if you're in that stage where you look at the equity curve and you go, yes, I figured it out, that will only happen two or three times until you get like us, where you're like, oh, man, it looks too good to be true. And that's, you know, I always want to to start with with that, and just to go, it's okay. You just need to kind of dispel from your brain pretty quickly that you've like hacked the matrix, and and you have this perfect algorithm that predicts what the market's gonna do next. And then, I I find it way less disappointing then.

Michael: 02:21

Because again, the moment I see it, the moment I see that, you know, equity curve, you know, come out, it's like, oh, this is just absolutely amazing. I'm like, okay. I know there's gotta be something I missed, there's gonna be something that's going on. So it's a it's a very sad and disappointing problem, but for like two times that you do it. And then every time after that, you're like, okay, I've got an idea of of what's what's happening now.

Michael: 02:45

And hopefully, if we can give you this checklist of things to go through and take a look at, one of those things helps you out and then you can start, you know, you you won't run into that problem again, you know, because everything you start doing, you start learning and saying, the next test I do, let's make sure that I'm not gonna really upset myself in the future by by missing this up... This step. Right?

Dave: 03:09

Yeah. And this is one of the things that... You know, when I say that to be a good trader, you really have to be an optimist, this is probably, you're like, on, like, something that happens where you need to be able to realize that you've done something wrong, or realize you've made a mistake, and be glad that you discovered it as quickly as possible. Mhmm. Know that you're gonna be a better trader as a result of it, but yeah, you're gonna be very disappointed at some points.

Dave: 03:36

I remember years ago, I remember doing something in a Backtest. I think I was using some new machine learning thing, and I was so excited. I went for a bike ride, and I got like a mile down the road, and here I am, like, riding as joyfully as I could, and then I realized, oh crap. Like, dawned on me what I had done wrong. I was like, ah.

Dave: 04:03

So it's a really bad feeling to have, but overall, it's a very... It makes you better.

Michael: 04:10

And it's funny because I always I always remember the study, right, of just like a happiness scale. Like, they give someone $20, and they they judge what the increase in happiness to those people are. Then they take the $20 back, and it's the decrease in... Because in theory, you're just no better or worse off. You're probably actually a little better off because if the problem is not really catastrophic, you probably have a good strategy somewhere.

Michael: 04:37

It's just not know, gonna turn you into a trillionaire in five years kinda kinda strategy. But the the feeling that you get is, like, way worse than even when you started even though you're Yeah. Closer to your end goal. But just because you you got really really high, and then you got knocked down more than one peg, like five or six pegs, and then you're bummed afterwards.

Dave: 04:58

Yeah. So there's probably some software developers that are listeners, know there are. And there is a really good analogy with software development. So when you're a developer, the thing that you're working for is to get your code in production to out to users so they can... You know, their lives are better from something you created.

Dave: 05:17

Right? So you're you're writing code. You you you do a PR, which is like a promotion request. Like, okay. Get my code in production.

Dave: 05:27

And you can set up what are called linters or things that can be caught at the very end of the process. And it's like, you know, you've done all this work, you've... Maybe you've done it in the wrong style, and then all of sudden, there's a style checker at the very end of the process, and you're like, oh, I'm so close. You know, the code works. Why do I have to go make these silly changes?

Dave: 05:49

It's kinda like that same process where, you know, when I was CTO at Trade Ideas, I move... You can move all those checks as early as possible in the process. So as you're writing the code, you can have linters set up and enforce a styling guide then, where it's much easier than, like, right at the last minute where everybody's ready for this code to go to production, and they're like, oh, this... You know, it's not in the right style. You need to go fix it.

Dave: 06:14

And like, well, it works. Like, you're so close. So, yeah, earlier in the process you can discover this, the better off you're gonna be. Just like... And I hope...

Dave: 06:23

My goal is at the end of this episode, people will have a good blueprint of things to check for so that they can discover earlier in their back testing process what's wrong, and that can lead to not making those mistakes and having a better and more efficient process for themselves.

Michael: 06:41

Well, alright. Well, let's just get into it. So step one, you know, so you you know, you put yourself right at the beginning, you've run the backtest, the system, whatever you're using, has given you the equity curve. You know that you haven't hacked the matrix, so it looks too good to be true. What's the first thing that you go and and say, alright, I gotta do I gotta do this one thing to make sure it's not like the dumbest, like the easiest problem to solve.

Michael: 07:07

Because I always look at it like, you know, if you're like trying to fix your dishwasher, I say, because I had to fix my dishwasher recently. Right? It's like you're running through, okay, what are the simplest things? Can I clean out the little trap thing? Can I?

Michael: 07:19

You know, what's the... It doesn't cost me anything. I don't have to buy anything. I don't have to figure anything out. The the very simplest, and then we'll go into maybe the more complex and and harder to deal with fixes after that.

Dave: 07:31

Okay. Well, this one this one can get complicated, but I wanna try... But it's by far the most common, and I wanna try to present it in the simplest way that I can. The the most common problem is your your... There's a future leak in your backtest somehow that you don't realize.

Dave: 07:53

So you're using a, you know, you're using a price in the future for your entry price that you can't know about at the time that you entered the trade, or there's many different ways that you could be looking into the future accidentally, and it's by far the most common. The the real... The main point to think about here is anything that you use in your backtest, any rule you apply, any entry price you assign to the trade, any of that stuff should be based on settled data. So data that will not change as the candle completes. Right?

Dave: 08:40

So for example, you could be... You can use the open of the bar, but the close of the bar during that bar, the lifetime of the the... Of that candle, you don't know where it's gonna close. You know where it's gonna open, but you don't know where it's gonna close. So the close is not settled during the lifetime of a bar, but the open is settled.

Dave: 09:04

So you always wanna be relying on settled data when you... In your logic. So that's... I think the... I always have to go back and even, like, relearn this sometimes because it it sounds super easy, but it does get complicated, especially when you're looking at a backtest, and then you're, like, thinking through what's gonna be happening in real time.

Dave: 09:26

It's very important to... And and I've finally gone back, and I I even wrote an explainer about this because it's so frequent. And I'm gonna be writing to the... My newsletter, some articles about this, because it's a surprisingly complicated topic. But if you just boil down to, I'm only writing my logic on settled data, that that is what you can rest on.

Michael: 09:48

Yeah. And, you know, for for, I'd say, swing trading, this one's a little bit easier than than day trading because a lot of what we're doing is close of bar. Right? And and that's the main thing that you want to look at is that, are you doing things based off a live real time price? I mean, it's fine.

Michael: 10:07

We'd... You know, I do a lot of that as well, but you have to be much more careful. If you're brand new to this, I would always say the way to go about doing it is even if you're doing day trading, is if you could do one minute bar, but do close of, like you mentioned, I'm gonna enter on the opening of the next bar every time. It really helps you out. But even sneakier than that, sometimes indicators, if you're plotting those and there's a little error in your code or something, the indicator value as some, you know, some people don't even think about how they're calculated.

Michael: 10:39

So like, you know, an RSI being calculated on a close or a Bollinger Band being calculated on a close or something like that. There's a lot of things out there that could look. And then when you start trading, it... You notice that there's discrepancies between the indicator value and and what you think it should be, and that's because, right, you're taking that that kind of close value each time. So was gonna be my first one as well because that is the one that will make...

Michael: 11:04

Not that it's the easiest to solve, but it will make the prettiest looking equity curve every time. Because if you just know the future, then all of a sudden, like, the rest of the the optimization becomes way easier. And it's it's funny in the cruncher, I've seen this a couple times where I just... I have columns in there like MFAE and MAE that I've added myself. And if I don't forget to remove those before I put it into the cruncher, of course, they're at the top.

Michael: 11:29

Right? Because like, oh, you just, you know, you just set your maximum favorable excursion really high and then all of a sudden, everything works out great. It's like, oh, good for me, you know?

Dave: 11:39

Oh, I'm glad you mentioned that because I've... For a lot of these, I've added checks into the cruncher where I can so that people are warned of this. So when you, you know, it used to be that if you put an MFE column in there that, you know, wasn't a standard name, it would show up as important, but now it knows, like, it'll detect that situation and give you a warning and say, hey, We we excluded these columns because they're they're too close to profit. So... Yeah.

Dave: 12:10

And there's been... There's several things I've added. So just, trying to give the user feedback as early on as the... In the process to be able to, you know, discover what's going on. And, you know, you you said you can make a backtest look really good.

Dave: 12:26

That's true. But it's it's even more subtle, and it's very... I remember seeing this for the very first time, and it was so shocking to me how realistically good you could make a backtest look by just looking... Doing something very subtle to look into the future. It...

Dave: 12:45

And that's really where it gets difficult because it looked... The curve looks realistic. It's not like it got a profit factor of 10. It's got something that looks reasonable. And then you go to optimize, you continue, and you...

Dave: 12:58

Then you get all the way through the process, go live, and then it doesn't measure up. You try to... Like, it's a really hard problem to figure out what's going on, and that's where your your reconciliation process is gonna be really helpful. When you have that dialed in, it's it's your backstop. So you you can know when this problem happens by having a good reconciliation process, and you can see this.

Dave: 13:23

So that that's kinda your backstop for all these sort of problems.

Michael: 13:27

Yeah. I would... I, you know, I think it's your backstop for almost everything in in errors of programming and and thinking... And as a non programmer, I have a lot of errors in programming in the beginning. And then also just errors in fill assumption, which I guess would be another one that you you thought there.

Michael: 13:47

It is it is the whole, you know, you have to brush your teeth and exercise. You have to reconcile back to your backdust. You just absolutely have there's no way around it. There's no way to prevent it. And thank God that AI exists because there's no excuse not to.

Michael: 14:03

Right? There is, you know Yeah. My bot wakes up every night, and it says... It downloads automatically, the trades and the down... It's like...

Michael: 14:10

So there's no excuse not to, but you're right. If there's any of these errors that pop up, we hope that us telling you about these will, you know, help you kind of figure them out earlier. But just know that anything that we don't talk about or anything that you can't find as an error, that's where it comes up. And you want to find them first because you don't want to have to trade it for a week or so, and then look back at the data and say, okay, everything wrong. Ideally, you catch it before you you put any money into it.

Michael: 14:37

But yeah, this is your absolute, you know, this is your saving. So just just do it. If anyone's not doing it right now, just

Dave: 14:43

do it. I mean, I I've gone for well more than two weeks one time with this problem, because it was just so subtle, and, you know, I was sort of punted looking at the details, because, you know, what you'll see in the Backtest, or what you'll see in your live trading, is trades that you take that don't appear in the backtest or the opposite, and you're like, wait, what happened here? And it's it's like the indicator value that you were using in your rule is slightly different. So that that takes a lot of debugging to figure out what's going on, like, what's happening. So...

Dave: 15:18

Yeah. Just the the the more curious you are, the more detail oriented you are about that reconcile process and see, okay, why is it not measuring up? The the better off you're gonna be, and there's more that you can learn from from doing that. So yeah, you're totally right.

Michael: 15:35

I think I yell at it so much because like a past self, like I'm not a detail oriented kind of person. I don't, you know, it's I don't like doing this stuff. I don't like the journaling side of things. I don't like the whatever, but it is it is a must. Right?

Michael: 15:50

It's like I just... I have to I have to do it just like I I don't like doing cardio. I like lifting weights, but I know I will die if I don't do some cardio. So it's like you just... You have to do it, whether it's...

Michael: 16:01

You like or not, you gotta do it. Right?

Dave: 16:03

Non negotiable. Yeah. Let's go to the number number two. So this is a pretty common, fairly common situation. So you got your back test.

Dave: 16:16

You're... You have an entry bar. Let's say you're doing it on the one minute bars. And you have to make a decision about which bar is the first bar to stop out the trade. If you do it on the entry bar, then that's gonna be a little bit pessimistic, depending on how your strategy is set up.

Dave: 16:42

So think about it. You've got one minute bars, that's your lowest resolution that you're backtesting on. You have an entry, and your stop is some distance away. Mhmm. On some of the entry bars, the...

Dave: 16:57

Often, the stop is gonna be... Or the low of that entry bar is gonna be below the stop. Mhmm. So you got a you got a problem there. Okay.

Dave: 17:06

What do you do? Did you... Would you have taken that trade or not taken that trade? Do you stop out on that first bar or not? And the way this gets people in trouble is they choose not to stop out on the first bar, or they don't even realize that they're not stopping out on the first bar.

Dave: 17:22

And, you know, they got a lot of sides. Maybe they're trade... You know, they're entering on the open, and those trades are gonna stop out in real trading, but they don't stop out in the backtest. So that can create a situation that you don't realize that can make the the backtest look way more optimistic than live trading is gonna be.

Michael: 17:49

Well, I'll give an example of the same thing that I ran into with day trading with real tests and and daily bars, and that's, you know, I... A lot of my day trading systems, I still run, I know, a couple years, Dave, on on daily bars with real tests. But the problem is with a daily bar, if you put out... If you're just doing a limit for entry or a stop limit for entry and and a stop loss, then that's usually fine. But as soon as you add a profit target into it, now you have three orders that exist in one bar.

Michael: 18:21

And it... Because you're just using high, low, open, close, you don't know which of those is is hit first. Right? So, you know, say you have a... I'm gonna buy a stock at ten, and I'm gonna try to take a profit at twelve, and my stop loss at nine.

Michael: 18:34

And all of those prices are hit throughout the day, it's just no way of knowing what order it is. Now, in some back testers, like I know real tests, it always assumes worst case. Right? That you got stopped out before you you hit the profit target. And that is...

Michael: 18:48

It's inefficient, but at least it's a good way to do it. But you need to know how these things work because you're right. When you... When you're doing everything in trebar, which happens the tighter and tighter that you're trying to be, it gets it gets really, really problematic. And the only way that I know to kinda solve it is either, you know, widen your stuff out enough that it's not that big of a deal, or you gotta dial in even deeper, which for the daily bar was a pretty easy solution.

Michael: 19:17

Right? I went to Ami Broker, and I I grabbed the one minute data for as far back as I could go, and and and worked that out that way. When you're... You you know, for day traders, when you're already in it, there's... Other than like looking at the tape, and then you're...

Michael: 19:31

I think the the smaller in you get there, you're increasing the margin, potential margin for error so much that you could take a strategy that looks amazing and be... Not have any obvious bugs in it, like Dave was talking about. But just through random noise in the market, it's gonna degrade that so much. So, yeah, you have to be really, really careful when you're doing things that tight, based off of whatever bar that you're testing, if that makes sense.

Dave: 19:57

Yeah. So there is... So in the trade client, the Mabe kit trade client, I've added a few months ago, a way for it to automatically cancel the entire order if the stop gets hit first before you enter. Okay. So that can help the situation and give you another lever to pull so that, you know, the stop so it's got some logic in there to do that if you'd like to.

Dave: 20:24

So... I mean, the other thing to think about here, this is... And we did a whole episode on this. This is probably the most or one of the most important reasons to do backtesting without stops at the beginning. Mhmm.

Dave: 20:38

Because it removes this problem entirely. That's half the reason, probably close to half the reason that you would want to backtest without stops, is because you were removing the situation entirely. So you want to be able to... You're not creating a... An exaggerated situation of using tight stops that is just that's gonna look good in the backtest, but it's not gonna pan out in reality.

Michael: 21:06

Yeah. And it's same with, you know, profit targets because that's like just the opposite problem. Right? If if you think you're entering and and then taking a profit target just immediately after, It's the same thing. So just figuring out whether or not...

Michael: 21:20

Because you could take any trading strategy out there, and I bet you if you got tight enough with those filters in the right way, you would take any random trading strategy and you could make it look amazing. And I'm specifically thinking, like, there was one strategy I did, not with stops, but with profit targets way back on trade ideas. And I thought I'd figured out because I was I was... Yeah. I thought I figured out the world.

Michael: 21:43

Because trading views... Or or... Sorry. Trade views use one minute bars. So every trade, it was like 98% of trades were profit because it was showing an entry and an...

Michael: 21:53

And a limit because I had a wider stop loss and I had a really tight limit. I'm like, oh, man, if I'm only trying to make a couple cents a trade, it's super easy. I'm like, well, move over, Citadel. I've got this, I've got I've got all this figured out. And then very quickly on day one, I'm like, oh, okay.

Dave: 22:12

Yeah.

Michael: 22:12

I I I get what's going on here. And, it... You know, you just... So yeah, test it naked first, test it without anything, you know, make sure that there is some signal, even just a minute or two later. If there is something there a minute or two later, then great.

Michael: 22:28

You're... You know, start Spend your time after that, right, going a little bit deeper down the the rabbit hole.

Dave: 22:34

Yeah. And it's really important to be, you know, really dig deep into what your back tester's doing, and the limitations. Like, you mentioned the trade ideas. That's a really good example. Mhmm.

Dave: 22:49

It would enter on ticks, but it would use for the timed exit and the stops, would use the one minute bars. At one point, it was using the five minute bars. I was like, no. We gotta move this down to one. Mhmm.

Dave: 23:01

But once you understand the limitations and what it's actually doing, you're you're in a lot better position to come up with something that works. A lot of... I I remember getting a lot of, you know, feedback about, you know, somebody would see this and go through the exact situation you did, running a backtest that appears to be... Look good. They would go trade it, and it wouldn't work, and they would blame the backtest.

Dave: 23:24

Right? That's not the right person to blame. You know? The only person to blame is yourself. So the the quicker you can come to that conclusion and be in sort of a...

Dave: 23:33

You know, have an optimistic, positive mindset about this, and that, you know, really ingrain the fact that I have learned something here that's gonna make me better even though I'm really disappointed right now, is really helpful.

Michael: 23:48

Well, and, you know, I was working there, and I didn't fully understand that limitation right away. But the beauty of, like, what we talked about reconciling back, it just took one day for for me to... You know, was a shitty day. I, you know, I I got, you know, like, 10 trades, and I think probably all of them, you know, maybe one or two of them, got the the out that I wanted to. And it was a negative day.

Michael: 24:11

And then I just went and ran the back test the the next day, once all the data had kinda caught up. And I was like, oh, I should have done this. Instead, I did this. It doesn't work.

Dave: 24:19

Yeah.

Michael: 24:19

So, you know, if you if you skip that step, then that bad day would have been a bad couple days or a bad week or until I had done that work. So, you know, again, just to to cycle back, you you have to. Right? Thankfully, it only lasts one day. And then I think I went and I asked people in the office, and they just kinda laughed.

Michael: 24:39

They're like, yeah, this is this is why it's doing that. But at least, again, it wasn't one day, and also shows why you don't go full hog into Absolutely. A new a new trading strategy. Know, if I was risking a thousand bucks a day, it'd be real... Or trade, it'd be a real shitty day.

Michael: 24:53

But instead, was risking, I think, like $20 or something, I lost like a $100, like,

Dave: 24:57

You you can you can learn these lessons for very cheap if you put your ego aside and use some really tiny size. That's a that's a great point. Alright. So the next one is... You've alluded to this.

Dave: 25:12

Are the fills realistic? So that's a more subtle one, but yeah, that's you you can recognize this, but it does take some experience and intuition to recognize it at that point, so... And that'll come with time. But yeah, so tell me more about the realistic fills or unrealistic fills that you were alluding to earlier.

Michael: 25:38

Yeah. Well, you know, a couple ways that... For realistic fills, like, is just volume. Right? Look, make sure the volume you're doing.

Michael: 25:44

But, yeah, these were these were essentially exit prices, and sometimes they were happening. Going back when I was looking at it, another way to easily check it out, as I should have looked at the time stamps of all the trades, was another way to look at these realistic. Because sometimes it was an entry and then an exit a second later or a half a second later or even Yeah. I think sometimes it maybe trade ideas did it at the very same time because it was using that kind of one minute bars. So, you know, there there are some ways to deal with this that we've talked about with, like, marking on open orders and and things like that.

Michael: 26:22

And limits, most of the time, are gonna be okay, unless the rare time that the limit goes right to your thing to the tick and then reverses, that would be a problem. But hopefully, if you're trading something with enough volume, that's a small one. For me, often, was stop limits and stops that became not as realistic because you're sending market orders out into the market, and very quickly those market orders can be off not just by a little bit from what you've expected, they can be off a lot. Depending on how frequently the the system trades, if it's off a couple cents here or there many many many times, then it can it can change the strategy entirely. So something that's way more, I think, a problem with very active day trading than swing trading.

Michael: 27:08

And the more frequently, you know, you're off by 5¢, right, the more frequently that happens, the worse the the strategy degrades over time.

Dave: 27:17

Yeah. I think using limit orders is a great idea, and using, using them with a pretty conservative price relative to your your strategy. So, like, make it pretty close to your entry price. What that does is it... It's gonna cause a situation that sounds like it's bad, but it's actually good.

Dave: 27:42

So you're going to miss some trades because you're not gonna get filled on a price like that in some strategies. And I... That that is a lot better, I believe, than getting filled with a market order at a worse price. Yeah. Because you're getting all those trades, and then you're like, okay.

Dave: 28:03

Yeah. I'm not missing any trades. That's good. But it's sort of harder to get to the bottom of the the situation by examining the slippage. When you use a, you know, conservative limit price so that you miss these trades, it kinda makes it way more stark what's happening, and it's easier to track down that way, because...

Dave: 28:27

And it's gonna be more motivating because you're you're gonna compare the backtest. Your backtest is gonna have made these trades that you missed, and you're like, all of a sudden, you're really super motivated to figure out what's going on. And that's a lot better than, oh, I got, you know, I got a little different price on this one. Why? Like, that's...

Dave: 28:44

It's a less urgent problem, and and this kinda amps the urgency, and it's safer to do anyway.

Michael: 28:50

Well, and you can learn a lot from it. So I could see two different scenarios where when your when your limit's close to the the trigger price, there's a... There's one world in which you're missing all the good trades. There's another world where you're actually missing some of the some of the bad trades. And it's just one of those you don't know until you kinda get in there and then do that reconciliation back.

Michael: 29:13

Because, yeah, if you are missing all of the good trades because it's, you know, the good trades hit the thing and then rip right away without you, then yeah, that... You'll be very very motivated to solve that problem. But I could see another world where, you know, maybe the ones that move fast, depending on when you're cancelling the order after a missed trade, there's information to be had there. So like a lot of these problems, if looked at the right way, are just more information to kinda gather in the trading strategy. Because, yeah, I could see two two worlds, now it's probably just luck and and the way the market moves that you're missing all the best ones.

Michael: 29:51

But, right, there is a world where the opposite is true, and and again, you you only know from this kind of back checking and reconciliation process.

Dave: 30:00

Pretty rare, I mean, most of the time most of the time, you're gonna get all the losing trades and you're gonna miss some of the winners. So... There's gotta be

Michael: 30:08

a, like, one Occam's razor kind of thing like that, right, where you get all the... We always used to joke and I remember in the prop firm that if someone complained that they only got filled a certain amount of shares that they wanted, that was gonna be the best trade of the day. Like, if you heard a guy like mad, he's like, man, I was trying to get like 5,000 shares and I just got like a 100. We're like, man, that thing's gonna rip today. That's gonna be a great trade.

Michael: 30:34

But yeah, you you... These are the problems that you gotta figure out. Right? Step one is having the good backtest, and then step two is trying to figure out a way to to trade the thing. Right?

Dave: 30:42

Yeah. Alright. So let's go to number four. Is your position sizing reasonable? So this is somewhat related to the Yep.

Dave: 30:58

All these are kinda related. But are you using position sizing that is gonna be reasonable in real life? So... And probably the easiest way to detect this, and what I suggest everybody do, anytime you get a... You run a Backtest, sort the trade list by profit, and look at the very best trades, and look at the very worst trades and see if those are what you expect.

Dave: 31:29

A lot of times, you'll see a backtest that, you know, 95% of the trades, even more, 99% of the trades look reasonable. And then there's, like, one trade that's, like, $20,000,000 of profit or $20,000,000 loss hiding in there, and... But you don't look at that because you're looking at the total profit or the p, you know, the p and l or the profit factor, so you don't see that. So that's the first thing you should do, do that. Sort by profit, descending and ascending, sort by position size, descending and ascending.

Dave: 32:02

A lot of times, your logic will cause these outliers, and you don't even really realize it.

Michael: 32:09

Well, and to to take an example there to the swing trading world, trying to find... Look at where your trades are clustered, because it's the same thing. I've built some strategies that just coincidentally, I bought like every stock in the world in the COVID bottom. And it, you know, it looks like an amazing strategy. And even the equity curve looks great because if you just look at the equity curve and you're not looking at the time access on the bottom, it's a nice smooth equity curve, then it just doesn't make any trades.

Michael: 32:37

And just, you know, I... What I've ended up doing is buying a whole bunch of stuff when the crash low. So one example of... Like a real example is we've had a couple crashes like this recently. We had the whole, I'm gonna wipe it, civilization off the map or whatever it was in 2026, and we had the tariff tantrum 2025.

Michael: 32:56

And then we had COVID before that, and there's all these like incredible v bottoms in the market. So I had built one strategy that looked out to be amazing, but it just turned out that it was, you know, it was buying these really kind of climactic bottoms and just doing so in a whole bunch of different stocks. Like, to go on the theme of that was more information, I didn't abandon that strategy entirely. In fact, it's now something that just scalps the S and P 500 when this rare event type of thing occurs. But, yeah, you need to look at the...

Michael: 33:27

This is, you know, the equity curve and all the charts and stuff, get really good visualizations, I think of this. But just trying to find out where the profits and where the losses are clustered, not only if, you know, you have one kind of lucky trade that makes up the the whole thing, but also are you are you clustering your trades around individual events? And then if so, you're not really... It's not alpha, it's just kind of leverage beta at that point. You're just...

Michael: 33:53

You're do... You're you're doing something that the market is doing, and ends up working, which again isn't necessarily the worst thing in the world. But you just have to know that, that it's not like a trading strategy, you're just taking advantage of when the market gets smoked.

Dave: 34:07

Yeah. It's sort of like you're accidentally creating a strategy that'll trade every time COVID happens, which Right. Probably not gonna happen Right?

Michael: 34:16

I was gonna say, you better knock on something, but hopefully, it's not gonna it's not gonna happen again. And that I find is is very common for what people, they just take Yeah. It's a rare one off event, but because if you were just trading the index on these rare... Not one off events, you know, they happen three or four times in the last ten years. But if you were just looking at the index, it would be very obvious because you would've just gotten like five trades.

Michael: 34:42

If you were looking at individual equities, you could look at it and say, well, I have hundreds and hundreds or or thousands of trades here. It's like, yeah, but they all happened on, you know, like, four days. So it's still just one trade, and it's because, you know, there's the old saying that in market crashes, correlations turn to one. Right? Everything moved together at that one moment.

Michael: 35:00

It didn't matter what you bought. It just... It was... You could've thrown a dart at a board, and you would've gotten the same result.

Dave: 35:06

Right. And I've... Yeah. You're right. This...

Dave: 35:10

We could probably break that out into a separate one, like I call it the spread outedness of the trades. That's one thing that I've added something to the cruncher called the Mabe kit score, and it takes that into account, among But other it helps you not just select for the few days where there was a whole bunch of trades where this situation occurred, which is a common problem. I mean, it's... Sometimes it's easy to see in the equity curve, but sometimes it's not. So...

Dave: 35:43

And there's things you could do about it with a strategy like that earlier in the process, so you can make... Select some filters such that you're not just selecting for the big days.

Michael: 35:53

Well, yeah. And I I think for intraday traders, a good example would be like FOMC day. You always get some craziness every time FOMC comes out. If you have a strategy that periodically spikes in trades and spikes in profits during those days, it could just be that it's, you know, it's it's taking some action that the spy is doing, but you've just taken it on a 150 stocks, so your trade count looks okay. But it's just spiking on the days that there's this intraday at, like, lunchtime or whenever they they talk.

Michael: 36:24

Like, that that volatility spike that happens intraday, that is a rare event. And again, doesn't mean that it's like the... You just throw out the strategy. Because if you found a way to trade FOMC that's profitable, then great. Maybe maybe you wanna look at that.

Michael: 36:36

But maybe instead of taking a thousand trades, you look to take five or six, and it just becomes this one strategy that runs during FOMC days or... Like, so there... It doesn't necessarily mean you throw it out. It just means that that's not a consistently profitable trading strategy. It is a rare event edge that maybe you want to exploit.

Michael: 36:55

But, yeah, understanding that cluster of profits, I think, is is huge.

Dave: 37:00

Yeah. Understanding the why of some of these things that you see as you Mhmm. As you backtest and optimize is and that's sort of... When you back up, that's kind of what we're doing. Like, understanding the why is gonna help you, in the future, come up with a strategy that avoids some of these pitfalls and situations that you you come across.

Michael: 37:20

Well, it also shows, you know, I I get a question sometimes that, you know, if systematic trading is the best way to do it, why aren't, like, mathematicians and and stuff like that win? Then I always point them to Renaissance Capital. Right? It's like the greatest money making, and he only hired mathematicians, but anyway. But the other thing is that that is, like, that whole FOMC thing that I said, that is a trading...

Michael: 37:42

You know, if you took someone who's just a mathematician, that's one of the millions of things that you learn just by watching charts, like, you know, what the heck happened on this random Wednesday? And then you go look at it and you say, oh, right, this, you know, I have these guys, they want their moment in the sun. So he comes out at 03:30 eastern or whatever every couple Wednesday or every couple Wednesday, and he wants to talk about some crap, and everyone cares about it, the market goes crazy for a bit and goes off. And it's like, you just need to know these things occur, and it's like the mathematician may not, but the trader who's in the seat watching the charts, well, he would, and then it would just be for him to spot this mistake without having like five FOMC events go on before he tried to figure out what was happening to his strategy.

Dave: 38:29

Yeah. Yeah. You mentioned mathematicians. I I've been thinking a lot about this lately, but I I think intelligence is probably overrated for trading. We'll see.

Dave: 38:39

I'm not definite about that, but...

Michael: 38:42

Okay. Everybody put your IQ scores in the comments. Yeah. We'll see.

Dave: 38:47

It's like, there's a whole lot of smart people out here now. Always has been. Mhmm. And yet, everybody, you know, we all have money in our accounts. It's not all...

Dave: 38:58

So, you know, hasn't been one person to suck up all the money. So... Yeah. Well, there

Michael: 39:01

was the famous... Did did you hear about what was sir Isaac Newton? Did you ever follow his trading? He he blew up in the tulip bubble. Like, he was like one of the smartest dudes in the move.

Michael: 39:13

I think it was it was either the tulip or the East Indian company or whatever. And you can actually find his trades out there where, like, this thing had been ripping for years. He had bought early, and then he sold a little bit. And then it just kept ripping for years and years and years, and he FOMO'd in at the top, and he just blew up. It's like, well Awesome.

Michael: 39:30

If this is like the smartest dude in the world at that time, and he just got crushed, then yeah, maybe it's got nothing to do with that.

Dave: 39:37

Wow. I'm not... I don't think I've heard that story, that's great.

Michael: 39:41

I'll send it to you on Slack later.

Dave: 39:42

Yeah, that's good. Alright. So the last one that I... That's on my list, this is a softball for you, Michael, because I I heard... I think I think I've heard from you over the last, know, the time we've been doing this, that you swing trade, is that right?

Michael: 39:57

Yeah, sometimes. So

Dave: 40:01

the last one is survivorship bias. So I'm gonna lob this up for you, let you hit this one out of the park.

Michael: 40:09

Yeah. This is this is a common one. Right? Again, not so much when... It's funny, I was I was programming and I was doing some stuff in Amu Broker, that that stop loss thing for the the day trades that I was talking about earlier.

Michael: 40:22

And it kept... I'm like, okay, go to Amu Broker and and, you know, test this thing, and then we're trying to use MFE to find a good thing that I couldn't with real tests. And it kept coming back. It's like, man, I can't find the survivorship bias in IQ feed. And I had to, like, tell it, like, five times.

Michael: 40:37

It's okay. It's it's a day... So this is really just for the swing traders. Because for day trades, we actually ended up doing a study. Maybe I'll, like, pretty that up and then send that out to you guys.

Michael: 40:48

But of the differences, because I have all of the data for all the survivorship bias for this one system on both, just to see what the difference was, and it was because it was a day trading only system. It was negligible. Yeah. Nothing that you really cared about. But for swing trading, and especially for some strategies of swing trading, survivorship bias is huge.

Michael: 41:09

And the examples that I'll point to is, you know, all of the SPACs that have gone bankrupt. Remember the electric scooter company in San Diego? Well, that's gone. Bird Scooters or whatever, that's bankrupt. Uh-huh.

Michael: 41:21

There's all these SPAC companies that that despaced and went down and went to zero. And Bed Bath and Beyond and all these as well. Well, if you have a mean reversion strategy, and it buys extreme dips, well, just over the last couple years, we've had the SPACs, we've had Bed Bath and Beyond, we had that regional bank crisis where a bunch of those went under. And if you run that system without having those old stocks in there, it will automatically look better because, you know, you bought Silvergate Bank or whatever, and you would have lost all your money the next day. But in the Backtest, because that does no longer exist, you wouldn't.

Michael: 42:02

So for the different... Right? For that different swing trading strategies things. And I would say specifically mean reversion, but I would include all of it. Just make sure you have a data source with with survivorship data.

Michael: 42:16

Norgate is what I use for that, just to preempt that question. But, yeah, you need it. I think you need it.

Dave: 42:22

Yeah. And so how would it show... So what if you naively did that, and and and didn't have... Didn't think about survivorship bias, you went to trade something with a strategy that appears to look good. Mhmm.

Dave: 42:38

How would you notice it? So... Because it would be a while before you would notice it.

Michael: 42:42

Right? It would be a while, and it would be catastrophic. Would be the only way, and that's the unfortunate side of it. Where sometimes, like, survivorship bias can cut both ways. The only other way that it kind of cuts to the good side is buyouts.

Michael: 43:02

Right? If you are buying stocks and they get bought out, they're no longer gonna be part of that subset of data, so you would forget those. But that's a good thing. Right? Generally speaking, you're getting bought out for a nice premium over whatever the last day's closes.

Michael: 43:15

So you're not too mad about that. It would be rare events, and it would be rarer, I guess, depending on what you're trading, and it would be catastrophic. Like, the WFF we just talked about in the green room before we started. This stock went from $2 to, like, $60, and then back to 14¢, 15¢. And it's probably it's probably a zero in a couple months.

Michael: 43:42

So if you have a trading strategy that's looking to buy dips, and you would have bought this thing at, like, $23 as it dropped from 60 to $23. Yeah. It just wouldn't show up in your backtest. Yeah. Because it it...

Michael: 43:55

It's likely... I don't know. It's likely going to be a zero in in the next couple months. And that's why this potential is is so bad is because if you have any percentage of your account in that, it's it's gone forever. You can't recover it.

Michael: 44:10

No stop loss can get you out of it, anything like that. So that's the only way is it would show is you would end up running the backtest later and having lost a whole bunch of money in this stock that no longer exists, and it's just gone from the backtest. And... Yeah. So this is why this is one of those that is I I very much pound the table on.

Michael: 44:31

Not because it should be a huge deal. Right? There's a whole saying that no stock has gone bankrupt above its two hundred day moving average. Right? There's there's ways to kind of filter out these kind of junky names.

Michael: 44:43

But if you get caught in one, it is it is disastrous. So it's it's a a small mistake that could be a a really huge deal.

Dave: 44:52

Yeah. And I guess the I guess the the big deals would be rare, but you'd still have plenty of losing trades that you're skipping in the backtest, right, that you're gonna get live. So you would... It's not like you're looking for the disasters. You're also gonna be excluding accidentally a bunch of bad trades, because these are typically gonna go down and hit your stop loss.

Michael: 45:20

So Yeah.

Dave: 45:21

It's not just the disasters, it's like pretty cons... Probably pretty consistently...

Michael: 45:26

Well, and and clusters of them too is is important to point out. Like, during the regional bank crisis, was that like... I'm old. Is that like two years ago now? Where was what First National?

Michael: 45:36

There's a whole bunch of year... We don't have that problem in The States, so it's... Or in Canada, so it's hard for me to to... We've got like six banks. They're all huge.

Michael: 45:43

But you had, like, every day you'd wake up, and there'd be this other regional bank that I never heard of that just completely disappeared and and went to zero. And those events are going to cluster. So imagine not just, oh, I lost, you know, I had 5% of my account in this crappy penny stock and went to zero, and I lost 5%. It's like I bought 17 regional banks and they all went to zero, that's I'm wiped out. Or in the year 2000, I bought pets.com and all of these other ones, and they all went to zero, and they wiped out.

Michael: 46:14

So these these events are going to happen in baskets. So there's a way that you're not just putting all your money into some BS stock, and you still lose it all. That's, again, why I hammer on it so much, is that it's a a solvable, a very solvable problem.

Dave: 46:31

Yeah.

Michael: 46:31

And it's just, you need to make sure that you have that data if you're holding these things overnight.

Dave: 46:35

Yeah. It's a good... It's a really good explanation of it. Yeah. That's...

Dave: 46:40

I like that. I think people get a lot out of that. But... So so I'm sure there's other things that we haven't... That aren't on our list that people know.

Dave: 46:53

Listeners, write in and let us know if we've forgotten some obvious ones, but we probably have. One of the things I've thought about with this... So I've thought about on my mailing list, or maybe I'll do this in the mastermind, where I wanna have a contest to come up with the worst... Like, the best looking backtest that is actually not reality. Because I think it'd be really good to have a whole bunch of examples of things that appear to look good that aren't gonna pan out, and I think just, like, having, like, a database of those would be really great.

Michael: 47:30

You'd have to be careful not to crash some people's systems. Right? Because what happens if you get into, like, the quintillions of of dollars in in profits? Right? There's there's definitely a...

Michael: 47:41

An issue with that. And that's another one thing that I wanted to leave people with with long term backtests and compounding. I'm not generally opposed to to compounding in your backtests, but, for example, right, I've well, got that's a fight for another day. But I I've got some systems that, you know, they're generating 30, 40% a year. And, you know, if I go back to the year 2000, I test those with a $100,000 account, it ends up being billions of of dollars throughout.

Michael: 48:14

So you need to, at some point in your backtest, make sure you're not applying more weight to those more recent trades than the older trades because there's no way you're getting filled on those amounts. And then also just desensitize yourself to the dollar numbers. It's why, you know, I think both me and Dave preached so much, which is looking at the equity curve. The equity curve shouldn't change if you're looking at, like, a percentage basis, whether or not it's it's compounding or not, if you're looking at it logarithmically instead of arithmetically. But, you know, make sure you're you're understanding that because, you know, there are times where if I if I combine all the systems together and the return number gets a little bit too crazy, you are talking like trillions of dollars by the end.

Michael: 48:58

And it could weight those last couple trades insanely highly. Oh, yeah. Even though there's just no way that you could end up getting Yeah. That fill ever, and you're putting too much weight on a trade that just couldn't happen, couldn't make any sense.

Dave: 49:12

Yeah. Hopefully, what we... Hopefully, the one we talked about earlier where you're sorting by profit and sorting by position size and your backtest when you get done with it, you would catch that and understand that that's what's going on. But, yeah, compounding is, like, very... It's it's another one that is pretty common in real real test backtest, people do that, seems to be by default, but

Michael: 49:34

It defaults to It defaults to comp banning. And there's simple ways just for the real test people to go out, is if you apply instead of, because real test defaults to compounding, and also a common way that people do swing trading as opposed to sizing per dollar risk is by putting a percentage of your account in every trade. It's more common in... You know, because we want to utilize all of our capital. So you might say, okay, 5% of my account goes in every trade.

Michael: 50:01

Well, that will change over time. Instead, if you just turn that to a dollar amount, right, if you have a $100,000 account, and then you turn that into a dollar amount, so 5%... $5, I'm gonna put in every single trade, then it just it gets rid of it right away. So there's some easy ways to do it. There's some ways inside of RealTest to have it withdraw capital, like monthly, things like that.

Michael: 50:21

There's tons of ways you can do it. Just know that it's a it's a thing and it's something that you should you should focus on.

Dave: 50:27

Yeah. I keep... All my backtests are, like, constant sizing. Like, I wanna separate the backtest from sizing decisions that I would make live when I'm trading it. So it just it just keeps things...

Dave: 50:41

To me, it keeps it way cleaner, way easier to

Michael: 50:43

think about. It's a number you can play with later. Right? You can go... And if you're saying, I'm gonna risk a $100 per trade, that's gonna be the default throughout.

Michael: 50:50

And then if you wanna... You know, you're worried about blow up risk or something, which I know is something people will stress test and say, at any point, would these drawdowns have hurt me? Well, you can just play with that number in the backtest and then expand it and contract it and and move it around or whatever. But first, make sure all optimization decisions and all everything is just based off of something standard, and then start to to kinda play from there. But Yeah.

Michael: 51:14

Yeah, that's that's one to that's one to know for sure.

Dave: 51:17

Alright. Well, I think this I think this is a good one. Please write it in, comment if you think of some situation that you've gotten into that we didn't mention here, I'd love to know about it, and we'll add it to the list.

Michael: 51:31

Yeah. And if you think you've ever done dumb stuff trading, just Google that Sir Isaac Newton thing. I... It made me feel better. I think that's why I I originally looked it up, but...

Dave: 51:41

We gotta put that in the show notes.

Michael: 51:43

Yes. I will. I I'm gonna send you the Reddit post now, and then, yeah, you'll put that in the in the show notes for sure. But as always, I'm Michael Nauss.

Dave: 51:51

And I'm Dave Mabe. Talk to you next week on Line Your Own Pockets.