Questions about gaps in simulation data from pausing in the debugger

[carnitron]

When I am debugging my AddOn in VS2022 with simulated data and I stop in the debugger and then resume, the series data I get next has a gap in it from what I saw before I entered the debugger. Unsurprisingly, the gap is about as long as the time I was stopped in the debugger.

I've traced this back to NT8 itself -- when I resume from the debugger, it picks up with what the time is now, without generating any of the intervening data; the BarsSeries has a gap and I can't be clever about adjusting the MinIndex to go back earlier to reclaim the data that was missed while I was paused in the debugger.

I've also tried rekicking the BarsRequest with sufficient backfillCount to get the missing data, but that doesn't return it either.

Is this expected?

I could see arguments for generating the data upon resume, but can understand why that might not have been implemented. The issue is that it means I have to make various parts of my code resilient against possible data gaps. I do not love this, b/c it sprinkles complexity throughout my codebase. OTOH, perhaps this is a fact of life for all trading apps and I absolutely SHOULD be making my code resilient to gaps, b/c this is an ongoing issue for all trading apps at all times; it's the internet, and sometimes you don't get your data.

Can you clarify?

If I CAN actually fill in the missing data, what is the best practice for doing so?

Thanks in advance, for any help you can provide.

Cheers,
Peter

[NinjaTrader_ChelseaB]

Hello Peter,

This would be expected. You are stopping the NinjaTrader process which ceases all activity. This will create a gap in market data.

With the Simulated Data Feed, this does not provide historical data and provides self-generated fake data in real-time only.
To correct any gaps you would need to connect to a data feed that provides historical and real-time data, and then right-click the chart and select Reload All Historical Data.

[carnitron]

Ok, thanks for clarification, Chelsea.

So to be clear, NT8 doesn't HAVE to work this way. It could detect the timing gap and backfill the missing data before continuing. The upside to this would be cleaner client code.

The larger question is whether I can expect gaps when running live.

If I should expect gaps, then the cost of this problem is 0, b/c I have to make my code resilient to gaps anyway. If there aren't gaps with historical or real-time data, then it means that the NT8 developer has to put in mitigations only for simulated data, which kind of damages that feature.

The fix on the NT8 side might be relatively simple and in one place, whereas the mitigations on the client side can be in many places. (I call this bleeding complexity.) So I'd vote for logging an enhancement request. But it hasn't become super costly for me yet, and I'm set to move on to the real thing soon, so not high priority.

. . .

Also. When you mention selecting Reload All, is this in the middle of my debugging session? Does it somehow know not to send all the prior data up until the breakpoint back into my app a 2nd time while I'm still running?

[NinjaTrader_ChelseaB]

Hello carnitron,

If you are halting the NinjaTrader thread when running live this would also introduce gaps. If there is a connection loss this would also introduce gaps.

NinjaTrader needs to be running to process incoming data. I wouldn't expect the debugger to be attached when trading live, but connection losses can still occur with a weak connection.

Reloading historical data will redownload all data selected in the Data Series Days to load and would be done once you have detached the debugger.

[carnitron]

Thank you for the further clarification. I have Google Fiber so the connection is top tier. Sounds like losing data should not normally occur, which is great news.

I have since realized I can just fill in the data on my own, so that is what I have done. The theory was that I wouldn't have to wait for any fix from NT8 and I would remove the mitigating complexity I was introducing into my code.

This basically works.

However, it means my algo can trigger on fake data, which produces orders that fail. So I have to mitigate against that instead. That feels less complicated, tbh, since the number of entry and exit locations is smaller than all the places where I ingest the price stream.

It would be easier for people using simulated data if NT8 handled filling in the gaps with actionable data. That way debugging with breakpoints would be transparent to code execution, not the distorting force it is now. Using breakpoints for debugging is obviously a core useage scenario.

I was able to implement something simple on my end in less than a day. Is this something that could be added to an enhancement request list somewhere?

Cheers,
Peter

[NinjaTrader_ChelseaB]

Hello Peter,

I have submitted your request for the development team to consider. Once I have a tracking ID for this request I will post in this thread for future reference.

[carnitron]

Awesome, thank you kindly.

Cheers,
Peter