Line Your Own Pockets

Michael: 00:00

Welcome back, everyone, to another episode of Line Your Own Pockets. Last week, we had a nice interview with Trent from IQ Feed talking about, you know, data and the background process of, you know, all the data that we need to do what we do, and it was kinda cool to see a little bit behind the curtain. Today, we're gonna talk a little bit more about that and then also an email that Dave got just kinda talking about different data sources and why one over the other and advantages, disadvantages. This is one of those conversations that's not might not be, like, as as sexy as our AI chat, but is more, like, the most important, probably, part of what we do. Because if you got crap data, you could be the best systems designer in the world.

Michael: 00:41

It's not gonna do anything for you. Right?

Dave: 00:44

Yeah. Data is really important. And, you know, sometimes people you know, I'll tell people, hey. I'm a trader. And they're like, oh, yeah.

Dave: 00:53

It sounds really cool. Let me you know, can you tell me about it? And, you know, what do I need to do to get into it? And I kinda talk them out of it because it's way harder than you think it's gonna be. Mhmm.

Dave: 01:05

A lot of it's because you're gonna have to deal with data a lot. You're gonna have to, you know, dive in and figure out why something is not showing up in your backtest when you think it should. And to do that, you're gonna have to go in and really pull some spreadsheets and start really looking at data very closely. If you're not comfortable doing that, then training's probably not for you. Right?

Michael: 01:31

Yeah. So Well, and not just the market data, but then you gotta you spend half your time looking at that, and then half the time is looking at your own data, and half the time, you know, trying to figure out, you know, this trade that I took, why did I get that slippage or why I didn't. And because for me, that's like the sticking point. Like, I love the backtesting side. I love that data side of things, but for some reason, I really hate my personal data side of things, like doing the reconciliation back.

Michael: 02:03

Maybe it's because I had, you know, a really bad time in accounting class, and it's gonna bring me back to that. But, yeah, so you're dealing with data coming in, and then you're making your own data, and you have to deal and then the two of those together make another dataset that you have to deal with. And so keeping all this together and keeping it clean is, know, hugely important, and it's just not the fun thing that people think about when they think about trading. They think about, you know, hitting a button, and then they're right, and they get to shout and and have a good time. But the it's the other side.

Michael: 02:37

It's it's the boring part, but it's the part that you just could not replace even with AI doing coding for you and things like that. At the end of the day, the, right, the data is is everything.

Dave: 02:48

Yeah. Yeah. The data is everything. And I'll I'll think back on that episode from last week with Trent. I mean, the most kind of important thing he said or when I asked him, hey, why are you even in business?

Dave: 03:02

Why should people pay money for data when they get it for free often with their broker? And I thought his answer was really good there. It's totally true. You're gonna you will wake up one day or come across a trade one day and you'll see, okay, yeah, the data is not right here. The free data that I thought was just standard across every service is not right.

Dave: 03:28

Let me compare that to another source and then you see, okay, the other source is showing different data, which one's right. So it really becomes pretty important when you're basing decisions based on the state. You're setting up strategies, making rules based on data being accurate or reasonably accurate. So it's pretty jarring when you come across something like, wow, okay, I thought this data was the same, but it's not.

Michael: 03:59

Well, and that to me was, as someone who had a fair amount of understanding of of the background of it, but hearing from someone who's behind the curtain, just talk about kind of what a nightmare just the back end of our stock market in general is. How, you know, it's this kinda old dated piece of technology we're kinda holding together. And you you talked about how, you know, it's amazing that it works as well as it does. Yeah. But then, you know, getting the data in and cleaning it up, I think a lot of people think that there's just, like, one pipe.

Michael: 04:32

Oh, you just you plug into the exchange, and it's gonna spit out all the correct data for you, and you're good to go. But there's a lot of, you know, cleaning and merging and and things like that that have to happen behind the scenes. And without that kind of stuff, you can get really kinda wonky, maybe back tests that don't make any sense. You know, if there's if it's assuming that you're getting fills at at prices that are unrealistic and bad ticks and and things like that. So we talked a lot about that.

Michael: 04:59

And I you know, it's again, it's it's huge is because if I'm running a mean reverting system, but all of a sudden it's assuming I'm getting filled on these bad ticks that we all see all the time on charts that happen, like, 20% out of the money, well, that back test is gonna look absolutely amazing. And, you know, without the cleanness of the data coming in that at least removes those or something within my code itself that that removes those errant trades, you could take something that it looks like you've, you know, hacked the matrix, and you're gonna be a trillionaire in no time to something that's just the worst trading strategy that you could ever do.

Dave: 05:34

Yeah. So it reminds me of a lot of traders will get to the end of a backtest. They've optimized it. It looks great. Right?

Dave: 05:44

And they're going to trade it live thinking they're gonna you know, the work's done. Mhmm. Time to turn the money machine on. And they're really primed for disappointment right there. Like, you you you're on this high.

Dave: 06:01

You think the work is done. Yep. And then you're humbled by the market or, like, well, it didn't quite work out the way you thought. Here's and you have to dig into why. That's sort of where the work starts.

Dave: 06:13

I see a similar thing when people want to get into back testing. They're like, Okay, I bought Ambroker. Or, Okay, I've made the decision to do this. I'm ready to start the backtest and do all this work. And then there's lots of steps to get your data foundation in order even to even to make one backtest.

Dave: 06:38

So even to get to the starting line to be able to create a base backtest, there's a lot of steps involved in, and it's harder than most people think. So

Michael: 06:50

And I think that brings you to the brings us to the email, that you that you showed me here that that someone had emailed you. And we can talk a little bit about that while you bring it up. And it essentially just talking about there is a there's a a wide amount of options available. Right? And and you summarize the email just a little bit better, but the the user was kind of looking for input on why, you know, why of all these data types, which one should I use?

Michael: 07:19

What's the advantage of them? What's the you know, what do I look for? Kind of like if you're shopping for a car, what do you do you need? There's a million of them out there. So when it comes to data, kind of, this email just seemed a little bit confused about where to start to kinda get their data from.

Dave: 07:35

Yeah. And there's he's got some very specific questions here that may not apply exactly, but but this is this email is sort of emblematic of or in a category I see a lot of people asking about, which is that they've just started they wanna start backtesting, and they're, like, fumbling around with where to get the data. So Mhmm. I think it's probably good that we should go over the options for getting data into the various tools that we would use to, to backtest. So we could start with Amber Broker.

Dave: 08:12

There are some probably some free you you can probably find some free data sources for it. But like we've talked about before, like, you know, free commissions, you kinda get what you pay for. You're get what you pay for. And it's good to think about and and go ahead and do the work, spend the money for a high quality data provider so you know that the the data you're getting is as accurate as possible.

Michael: 08:45

Well, and that, you know, outside of even accuracy, that was the experience that I had, you know, that we talked about on the podcast and earlier when I was doing my learning amnibroker on top of RealTest was I initially just went to the kinda cheapest option. I'm like, oh, it's $20 a month for what do they call it now? Massive or something? Yeah. Polygon.

Dave: 09:07

Massive. Yeah.

Michael: 09:07

And, yeah, just very quickly kinda wasted, like, a week or two of my life and just trying to, you know, figure out ways to, like, merge things and put them in and and do all that. Where if I had just gone straight to IQ Feed that has a more kind of direct link and plug in directly to Amity Broker, then that's gonna be it was obviously the the right move and the thing that I should have done with to begin with. So that's kind of where I would I would start is by looking at your broker or looking at whatever back testing engine you're using and saying, okay. What data has already been kind of built to plug into that right away? As the simplest tool, it might not be kind of the best off the off the rip, but at least you've you've jumped a couple steps.

Michael: 09:56

Because for me to do a polygon or massive, there was gonna be, like, four or five steps more that I was gonna have to do. And then when it was time for real time data, I was gonna have to figure out something in entirely different. So I would just say if there is whatever you're using, a couple options that can just be plugged in, probably better to spend the extra money and just go with that and save yourself the hassle of trying to come up with Python scripts to do all these different steps.

Dave: 10:22

Yeah. And so when I first started doing this, Polygon was not an option, and I was using the data from Interactive Brokers. And the problem there is, like, sometimes you'll do some quick testing, and it seems to to work okay. So you go with it. But then once you you know, it's maybe a few days later or once you get something live or you, you know, you get something live, but then a big day comes along where you've got a lot of candidates to sort through and all of a sudden your data feeds maxed out because you can only backfill a certain amount of days.

Dave: 11:07

You can only backfill a certain amount of symbols and you're not going to notice that in your testing and like, you know, your one off stuff. You're just like, okay, this seems to work only when you get something to scale. Do you realize, okay, well, I've got a big problem here. And a lot of times those days are the ones that you need to be prepared for because potentially those are the days you can make the most money on.

Michael: 11:30

Yep. And, yeah, it will always be the generally speaking, and the you know, this could just be, you know, me seeing shapes and clouds. But generally speaking, data errors will work against you. There's usually not a time that you're like, oh, you know, you get a data error and you end up making a ton of money, and then you look back and you go, oh, man. I shouldn't take any of those trades.

Michael: 11:52

It's kind of like when, you know, you're if you're more discretionary trader, when you take, like, revenge trades or emotional trades, they they never they never work out in in the long run. It's the same with that. I don't know if I ever gotten a data glitch that's made me a ton of money, but I've definitely had a whole bunch of bad days that's lost me a ton of money.

Dave: 12:11

Yeah. Sometimes it happens, but it's it's wanna be careful that you take that exactly the right way, which is it's a mistake. You made money, and it's a mistake, and it's still a problem worth getting to the bottom of.

Michael: 12:28

Yeah. Absolutely. And I think that kinda leads as I'm looking through this email to the the next question that I think the the user kind of asked himself is that, what is the ultimate goal of your trading? Because I see you mentioned Norgate, IBKR, IQ feed. Well so for example, Norgate, I think, is an amazing data source for daily data and long term swing trading.

Michael: 12:56

They do an amazing job with, you know, removing long or removing delisted stocks, bought out stocks, that kind of thing. Something that I don't see the other ones do. But if your goal is to do intraday trading exclusively, you know, or or you think even in the future, you're going to need, you know, some sort of intraday candle, a one minute candle, a thirty minute candle, whatever it is, just doesn't do it. Right? So there's there's a you know, it's one of those things that we can't really give a this is the best data source for this reason.

Michael: 13:30

It is here is what I'm working with and what I'm looking to do when you have to find one that fits that. So, you know, if you were using real test and only looking at daily data, I think Norgate's amazing. But then if you're looking at intraday data in Amibroker, well, then it's probably IQ Feed. And there's you have to first create a this is what I'm looking to do now and in the future, and then try to shop, I think, for your data source after that as opposed to the other way around where you're looking at all the different data sources and trying to find one that you think will fit. So start with, a, you know, a framework, I think, of goals, which would make a lot of sense.

Dave: 14:06

Yeah. I think that's a good way to do it. And yeah. You're right. That that's this is one of the first things I ask.

Dave: 14:12

Are you using daily data now, or are you ever gonna use intraday in the future? Because sometimes it is hard to predict, and, it it's definitely a dividing line. And, yeah, I agree that Norgate's a really good data provider for daily data for sure. I love the way they clean the data. And, you know, there's there's groups that have, you know, quant trading firms.

Dave: 14:40

They'll have a whole group of engineers just to work on the data. Mhmm. So it's not an easy thing to be able to tackle. And and it's it's more it's just way more complicated than you think. You know, you think that, okay, there's a bunch of trades that happen in the market.

Dave: 14:59

Yeah. Let's see what the price is, how much the volume is, what the bid and they ask was at the time. Easy game. Right? How how hard can this possibly be?

Dave: 15:08

But, yeah, it turns out there's a lot of things that that go into that. And Well even just like which ticks are allowed or should be put on a chart, like, that that would encompass the chart to to make up the candle data, that's a whole there's a whole set of attributes that go on every tick that come from the exchange that say, okay. Yeah. This is chartable. This should go toward counting toward the high of the day or the low of the day if if so downstream services can look at that and take that into account.

Dave: 15:40

So it's it's just more complicated than you think.

Michael: 15:44

Well and it kinda leads me to, I think, the next thing is that really knowing the time frame in which you need the data, I think, is is just kind of everything. Because the faster that you need data, the more important all of this is. You know, for someone like Norgate, I'd say their job is significantly easier because they're giving you the data, you know, fifteen minutes delayed, and it's got they can build systems to clean it up. And then that end of day print is gonna be pretty much decided at the high, low, open, close. Those prints are gonna be easier than if you're doing this on a a tick by tick basis or a one minute basis where, you know, you'd have to deal with these things more in real time when they when they occur.

Michael: 16:29

So, you know, the shorter term nature that you go in, the more that I think it's it's paramount that you're not skimping on good data. If you were doing you know, if you were someone, for example, who was just doing weekly, right, that was your you need the weekly high, low, open, close, and, you know, maybe more investment type systems. I would almost even say you could probably, like, experiment with a a freer version or or something like that, a little bit more fringe because you probably have the whole weekend for the data to get, you know, put in your system and cleaned up and make your decisions for the next week. That's that's a different beast entirely. So you could even probably grab something for your broker for that.

Michael: 17:06

But the shorter term that you're looking at, the more these decisions have to be made by a really good data provider with a lot of experience of what to let in and what to what to kick out and and to make sure your prices are very clean in the moment. So, yeah, the the faster you're planning on trading just it makes this, you know, exponentially even more important.

Dave: 17:25

That's a great point. Totally agree. I mean, if if you if you, you know, zoom out far enough and your time frame's large enough, then you don't even need real time data. Just use delayed data. It's totally fine.

Dave: 17:39

Right?

Michael: 17:40

Mhmm. Yeah. Because well, and it's the difference. When I'm building swing trading systems, my slippage worries are just significantly lower than when I'm building day trading systems. Because if I'm planning on holding something for a week and and when I'm right, I'm gonna make 10%, and when I'm wrong, I'm gonna lose 5%, then the that that difference in the in the difference of of what did I get filled a couple ticks higher or lower is is way less.

Michael: 18:06

But if you are looking to make, you know, a quarter of a percent and and lose a quarter of a percent, you can see that every individual tick becomes incrementally more important the the the tighter down in time frame you go. It's why these HFT shops that you're talking about, they they can't make a single mistake on data ever because the edge they're looking for is is so much thinner. And it's it also just becomes harder to, I think, to to reconcile the trades and to go in and to look at that data because as soon as you get into that intraday time frame, if you're talking one minute candles, well, everything has been multiplied by 360, right, minutes per day as opposed to four data points. Now you're dealing with four data points time every single minute that's happened throughout the day.

Dave: 18:55

Yeah. I like the way you're talking about this because, you know, you could you could take this to the nth degree. A lot of traders want to do that. They want to a lot of traders come from kind of an engineering background. They got an engineering mindset.

Dave: 19:09

They wanna over engineer something. So a lot of them will might hear this and say, okay. I need I need level one data for all the trades I make in my back testing database. I need level two data. What you know, I've got the means to do it.

Dave: 19:25

Why don't I just do it? And there is a bit of a sweet spot where if you go too far down, you're you're competing with some very well capitalized firms, HFT firms that like, this is what they do. And but there is some limit to where the kind of strategies you trade are gonna just pit you against some really good trading firms where this is exactly what they do. This is their sweet spot. Yep.

Dave: 20:04

So you don't wanna go that far, but there is this I think of it like, a bit of a continuum where you can day trade, but minute bar data should be all you really need for most strategies. Mhmm. Not for necessarily everything you could possibly think of, but for the vast majority of that there's plenty of edge you can find looking at just minute bar data. And, you know, the the the need you feel or a lot of engineers feel to try to overengineer solution and and get data less than that, you might you you should resist that because there's so much you could do with minute data even though, you know, it's it's this minute bar data.

Michael: 20:51

And that's, yeah, that's, you know, very human. That, right, we think, okay, if we can get faster and smaller and lighter, then things will work out better. But you're right. There's a a certain world that you zoom into, and then now all a sudden, you're sitting next to Ken Griffin and the billions of dollars that Citadel has in order to to to do tick by tick, and it's just not a a game that you wanna play. And I know that there's a lot of day traders out there that that was the reason to kinda transition to swing trading or at least, you know, instead of scalping, holding for hours and minutes and things just to say, I'm gonna avoid you entirely, and I'm gonna look for for kind of longer term trends.

Michael: 21:32

I also think it was the rise of of really small cap trading as well. Just looking for places that they're not having this kind of competition anymore. And just like I mentioned that when you go from daily data to one minute data, you're multiplying the complexity a crap ton. If you go from one minute data to level two data, you're you're it's probably even crazier because that level two could change 10,000 times in a minute. Right?

Michael: 22:01

There could be an order there that's gone, that's moved, that's and all of that happens within a one minute bar. So, yeah, as you're increasing kind of complexity on the way down, you just have to find your sweet spot is when am I allowing in enough data that I can find some amount of edge to explore, but not so much data that I'm just spending my entire life trying to build, like, an HFT system from the ground up to compete against, you know, your citadels and your market makers and all those kind of guys in the world, which is you're not gonna do well.

Dave: 22:34

Yeah. I mean, there's there are some back testers that I use that have tick by tick data. Like, for example, S and B has a really good one. But there there there's so many people back testing there now since the advent of Claude. There's, like all of a sudden, there's this explosion of back testing.

Dave: 23:01

And because it does tick by tick and there's a way to do level two day or level one data, I mean, the some of the backtests just take very long. So there's a bit of a bottleneck there because of that. So it's it's it's definitely you wanna resist the urge to do it. And certainly, you're a retail trader, then, yeah, you want you definitely wanna resist the urge to do that. There's just so much more that you can get, and you don't wanna get in the situation where it takes all weekend to run a back test.

Michael: 23:42

Yeah. Yeah. How many how many weeks are you putting up? And and one example in this will probably only hit with, like, a couple other gym bros like me out there. But, you know, there's a you know, you you go to the gym, you can get 90% of everything that you need and be in great shape and and look great or whatever with a just a barbell.

Michael: 24:03

Right? And that's it. Barbell and some weights. You'll do fine. Now you had a a dumbbell and a couple machines.

Michael: 24:10

Maybe, you know, maybe the barbell got 80% of the way there. But the dumbbell and the couple machines, that gets you, like, 95% of the way there. And then eventually, if you keep going, you're that, you know, the crazy guy doing, like, the weird exercises that you see on TikTok where you're, like, balancing on a ball and you got, like, stuff. Like, there is a point in which you're, like, you should have just stuck at, like, the 90%. You got most of it done.

Michael: 24:32

You get everything. You're in and out in a half hour, and you're fine. And then, you know, eventually, you can't fit in any of your T shirts anymore because you're doing, like, all kinds of weird peptides and crazy. And it's the same way. It's like, what can I do?

Michael: 24:44

You know, boil it down to like the eighty twenty principle. What can I do to get the the large majority of what I need covered covered? And then this little fringe stuff, I'll just leave to other people with more resources and time and staff and then whatever. Because I'm probably never like, in my life, I'm probably never going to get to the point where a barbell I can't fit enough weight on a barbell in order to to lift it to be to be good. And it's probably the same way.

Michael: 25:11

You'll probably never get to the point where you've exhausted every available option for one minute data, know, including other indicators or you could bring in other data sources or or whatever it is, but you you'll probably never get there in your life. And if you do, you're probably already making a ton of money, so maybe just be be okay with it for a

Dave: 25:30

bit. So so it reminds me, you know, I ride with I ride bikes with some some fast guys. And there is some of them have sort of an arms race for getting the the latest, greatest bike, the most expensive bike, the lightest bike. They'll they have you know, they weigh all their components to make sure they're getting the lightest component for their bike, you know, the the water bottle bottle holder. Like, they'll weigh that and make sure they're getting the lightest one.

Michael: 26:01

Yeah.

Dave: 26:01

I it's just, like, there's I I kinda look at this and laugh just because you spend so much time thinking about that. And, you know, as an engineer, I could see it's kinda fun. So you could easily get trapped in that. But, hey, why don't we just go ride bikes and, you know, have one that is a lot less expensive and ride it for longer than everybody else does that they're they've long since replaced bikes that I got mine at the same time. So Well, it's

Michael: 26:33

just more fun. Is everything else perfectly dialed in? Right? So, like, before the person put that, like, lighter weight, whatever on the on his bike, did he have cheesecake the night before? You know, like, there's there's so many other parts of your life that unless you have completely kinda maxed out and dialed in, it doesn't make sense to do this additional step.

Michael: 26:56

And I'd say it's the same with with with trading. Do you are you looking to build a level to optimize back tester? Well, have you built one strategy that can make any amount of money using, like, daily charts? Like, start, you know, start there. And then if that's built and optimized, well, then maybe you you kinda zoom in and and you go from there.

Michael: 27:18

And if you've ever gotten to the point where you have now, you know, every moment of your day, your buying power is completely utilized with a whole bunch of different systems that's making you a ton of money, and you've got some spare time and you wanna incorporate something else, well, go nuts. But if you haven't optimized all the way down to that level, it's like you're you're kinda skipping everything. Be like a fat guy trying to get like, you know, lighten up his bike as much as he can. This is it's not gonna work. You're gonna lose some weight first.

Michael: 27:47

That's gonna be, you know?

Dave: 27:49

There's other things that are the that are the priority now. Right?

Michael: 27:51

Before this. Yeah. If you're if you're unprofitable now, you don't need the level two data. Right? You need one strategy that makes some amount of money and then start to get crazier after that.

Dave: 28:03

Yeah. I like the way you put that. That's good. So so let's talk about Amibroker and backfilling Amibroker because that's often what where people struggle with. And that's you know, we had a couple episodes on that, you know, using Polygon versus IQ Feed to get data in there.

Dave: 28:23

One thing about IQ Feed that a lot of people don't realize is the there's a free trial to IQ Feed. So you could get the free trial. During the free trial, you cannot backfill all the data because what Trent told me outside of the conversation we had on the podcast was a lot of people were taking the free trial, downloading all the data for years and years and years and then canceling the trial. Now, I think from IQ Feed's perspective, might be okay. But the NYSE does not like that at all.

Dave: 29:08

So Yeah. That they put a stop to that real quick. So it's not very apparent when you take the free trial to IQ Feed that that's the case. So you go to to, do a backfill in Amber Broker to to feed your database with data. If you're on the free trial, it's not gonna work.

Dave: 29:28

It and it's gonna and it's not there's not gonna be a message that says, hey, this is why it's not working. It's kind of this nebulous situation that it's not really clear why, like maybe it's something on your computer. No. That that's the reason. So, yeah.

Dave: 29:44

For that reason, it's it's more difficult than it should be. And and I'm working on some tools that I think will make this whole process better. Because a lot of times you'll do a backfill with IQ Feed. And like I had a couple users today, they they were they get to the a 100% mark, get to the very end of just the backfilling and it would just hang at a 100%. So it seemed like it was done.

Dave: 30:11

It seems like it was done, but it just still won't complete for whatever reason. So, it's just there's all sorts of snags you can run into.

Michael: 30:23

Yeah.

Dave: 30:24

And I'm working on a better process now to to that I think will be a lot cleaner. It'll allow you to see the process of backfilling way better. You can see exactly what's been done, exactly what's remaining, exactly what symbols it's stuck on, exactly the data that it downloaded. You can go back and verify all this. So, yeah, I'm excited about this because I can tell that people struggle with this and it's it's the point where, you know, they've already made the decisions to they they can see the potential in starting to do this.

Dave: 30:54

And you got all these steps just to get the foundation right to even be able to get to the starting line to even do this. So I'm I'm optimistic I can make this a lot better.

Michael: 31:06

Yeah. That'd be great. Because, like, one thing with Amity Broker and especially the data as as I fought with it for so long, the user friendliness is certainly not the, not the forefront. But, you know, at the end of the day, I I kinda looked at it, and and this is when I was battling with Polygon and even when I was trying with, like, the free trial of IQ feed and everything. I'm like, it's a $100 a month.

Michael: 31:29

Right? At the end of the day, it's like, you know, you spend I think it's like I'm spending like a 150 or or something a month. It's like, if you're serious about it, that should be like one good day trade is is the difference between between that and not. So there are things to be cheap about. Right?

Michael: 31:46

There are things to kinda save your money on here or there. It's just data is just not not one of them. Right? It's one of those things that you just really do need something that is going to be reliable, and it's gonna be something that tests. And then you don't need to spend your time doing that.

Michael: 32:05

It's like our whole conversation with the AI agents is what are we offloading to something else? Right? We mentioned why not build your own back tester with Claude? Like, yeah, for a couple $100, someone who's already done the work. And same with this with data.

Michael: 32:18

Do I try to skimp out on data, do I just pay the guy a $150 to have some assurance that it's clean and it's accurate and it's arriving on time and and all of these things and just spend my time and mental energy doing things that are just way more important on the other end of it.

Dave: 32:34

Yeah. I totally agree. And it's it becomes clear when you I mean, I I remember specifically some trades that I, a couple days that I took when I first started automated trading and systematically trading I was going back and reconciling so I would run a big backtest afterwards using my backtesting database and then compare it to the fills and what I was seeing in real time and they weren't matching up. Like I would see something in the backtest like a big winning trade and such and such. I didn't see that when I was trading real time.

Dave: 33:14

It's like, okay, what the heck's going on here? That can't happen. Like what is going on? I need to at least understand what's going on here. And when it comes down to it, a lot of times you'll see it's a data issue where the one data provider is showing one set of data and another data provider is showing another one.

Dave: 33:36

So it's good to make those consistent. And it's just it's something you see over and over again. And you you kinda count this up to slippage, like data issues like this. And sometimes you'll get to the bottom of it and it won't be a solvable problem. It'll just be, well, this is just a data discrepancy that you're gonna have to live with.

Dave: 33:56

There's nothing you can actually do to make this better. So sometimes you'll get to that point and but but you realize during that whole process just how valuable it is when you're doing systematic trading at at the level we're talking about.

Michael: 34:13

Well, yeah. And and, you know, the difference between that $150, it could be more money that you're making, but like you mentioned, it could just be money that you're not losing. Like, if you're, you know, if you're going to pay that money somewhere else. And that and that, you know, having the same data provider as as the backtest, as the real time trading, that was one of the things that really got me into IQ Feed. Because right now I'm I'm doing like a little bit of workaround with with TradingView and stuff, but eventually, I'm gonna undo that and go right to Amni Broker.

Michael: 34:43

And that to me makes the most sense because you're you're laying another layer of complexity on top of it if you have this is my back testing data provider, and this is my real time data provider. It's there could be these little minute differences that you just don't come across. But if you're using the same company for both, regardless what that company ends up being, that's gonna be greatly diminished, and it's certainly gonna help you do that switch because that's one of the most nerve wracking part of this when you've gone through back testing world and you flip it over. And like you were mentioning, Dave, it will never be exactly perfect. There'll always be something that comes up, and that's why the reconciliation process is so important.

Michael: 35:24

But the you're lowering that kind of worrisomeness just a bit saying, well, at least it's the same data that I've tested on that's coming in. It's not just completely different feed from, you know, god knows what happened after that.

Dave: 35:40

Yeah. And I think I think another thing to really think about, especially for somebody that's new getting into this, there's a big difference between backfilling historical data and a real time data feed. And a lot of times, even with IQ Feed, even from their support, I'll see some users that say, hey, Dave, you can't do you can't backfill more than 500 symbols. I need to have 500 symbols or more than 500 symbols in my IQ feed. And what people even sometimes their support doesn't realize is it's 500 symbols for 500 level one quote feeds coming back to you in real time real time quote feeds.

Michael: 36:27

Mhmm.

Dave: 36:28

That's very different than a backfill. You could do a backfill anytime. You can, you could do that after hours. You could do it on holiday. Like, it doesn't matter necessarily how quickly you can back to you know, you can backfill your back testing database.

Dave: 36:45

Time is not of the essence. You're not you you don't need real time data to do a back test.

Michael: 36:50

Mhmm.

Dave: 36:51

So a lot of times, even the providers themselves kinda get confused there because they assume people want a level one feed for for a whole bunch of symbols at once, and that's not what you're doing when you're back testing.

Michael: 37:04

No. And and I could see that in and of itself breaking a system. Right? You get your backtest. It takes forever to go in.

Michael: 37:13

But then when you're designing your your script or your system in real time, if it's if it's either ingesting a bunch of data it doesn't need or looking at things that it doesn't need, that can incorporate slippage in and of itself. And, you know, that would be, again, another reason to make sure you're doing a bit of the same one because, yeah, with I could have kept going with Polygon, especially Claude Code came out since then. I could I could just tell it to brute force, kinda pull everything in and put it together and throw it in there, and it it would figure it out. It would be fine. But that that speed of especially for the intraday strategies, you can't replace that.

Michael: 37:49

So you need to make sure that that data is coming in as instantaneously as you can. Because like we talked about, the lower your time frame, the more significant all this ends up becoming.

Dave: 37:59

Yeah. We should probably do a whole episode on the real time part of trading with with Amber Broker. I mean, that's that's it's very different to back testing. Time is of the essence, and there's a specific sequence of steps to do to get the workflow working where you can actually scan the entire market using basically, you use trade ideas to filter your universe. And then in Amabroker, you're not scan you're not actually scanning the entire market.

Dave: 38:36

You're scanning that universe, the trade ideas is scanned, you know, filtered for you. Yeah. So it's a really efficient way to do it. And, you know, I even I I even hear from Tomas, the the creator of Amber Broker, that it's just impossible to do. I see him reacting that way on some of these forums to traders that's that want to do something like this, but it is possible.

Dave: 39:01

There's just this very specific workflow to be able to do it.

Michael: 39:04

Well, good. Again, we always I think at least one time an episode, we say, oh, we've just as we're chatting, have have come up with a idea for another episode. So there you go. So if you guys are interested in that, make sure you're you're tuning in whenever we're coming out with these weekly and, right, subscribing wherever you're seeing it and all that kind of stuff. But as always, I'm Michael Nauss.

Dave: 39:26

And I'm Dave Mabe. Talk to you next week on Line Your Own Pockets.