Memory Collapse while working with Strategy

[zhiga]

Hello,
I'm running my strategy on a base period value of 30-60 seconds on 3-5 days of data. Doesn't matter on the chart or not. In the strategy I use my 3 custom indicators. I define the indicators in State == State.Configure, and I also add here AddDataSeries(Data.BarsPeriodType.Tick, 1);
AddDataSeries(BarsPeriodType.Second,1).
I define the series in State == State.DataLoaded like so:
= new Series<double>(this, MaximumBarsLookBack.Infinite);
= new Series<double>(this);
I simplified and optimized the code as much as I could. I am not a programmer.
All indicators used in the strategy are loaded onto the chart at the same time quite quickly and do not load memory.
Problem: When I turn on the strategy on one tool, there is a jump of 2-3 GB of download in RAM. When I turn off the strategy, there is also a jump in RAM consumption. For 2-3 instruments, memory consumption is about 10-15GB.
It also clogs disk space. About 40 GB of free memory on the C drive has become filled with something. Once, when all the memory on the disk was full, the computer turned off, and after turning on the free disk space, it became as much as it was. As soon as I started enabling the strategies again, the disk space began to decrease again.
Please tell me what should I pay attention to first of all in order to remove possible errors?

[QuantKey_Bruce]

How much physical RAM does this computer have? And what instruments are your 2-3 instruments for which you are loading 1-tick bars for 3-5 days of data? Your description sounds like you are running out of RAM and the computer is starting to swap.

You should also track down what is using all your space in Windows when it is being used. One idea would be this: https://www.wikihow.com/Find-the-Lar...-in-Windows-10 or https://www.mindgems.com/article/find-largest-folders/. Armed with this information you will be in a better position to understand the problem.

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your post.

The impact NinjaTrader has on your PC will be dependent on how much NinjaTrader has to calculate

What are the instruments that you are loading in the script?

Do the indicators you are referencing in the script also add an additional 1-tick series?

You could consider setting the MaximumLookBack property or your custom series to MaximumLookBack.TwoHundredFiftySix instead of MaximumLookBack.Infinite to conserve resources. When set to MaximumLookBack.TwoHundredFiftySix, only the last 256 values of the series object will be stored in memory and be accessible for reference. This results in significant memory savings when using multiple series objects.

Adding an additional 1-Tick series will increase the amount of resources the PC will need to use to process data.

We typically advise the following as tips to improve platform performance.

1. Configure charts so their Days To Load is an absolute minimum, just enough for your indicators to function properly
2. Configure indicators to use Calculate.OnBarClose when possible as opposed to Calculate.OnPriceChange and Calculate.OnEachTick.
3. Use less charts and less complex indicators
4. Avoid using Tick Replay.

See this help guide page for information about Performance Tips: https://ninjatrader.com/support/help...ance_tips2.htm

MaximumBarsLookBack: https://ninjatrader.com/support/help...rslookback.htm

You may also wish to use the NinjaScript Utilization Monitor (Control Center > New > Output window > Right click > NinjaScript Utilization Monitor) This tool will tell you which components in your workspace are creating the most impact on your system.

After you leave this utility running, the items on the top will be those that are taking the most CPU time. Reducing/eliminating these items will be bring the most performance improvement. Please note that some NinjaScripts, like ChartStyles, require drawing every visible bar with every render pass. Having a ChartStyle at the top of your utilization Monitor would be normal.

I also suggest using the Windows Task Manager to view other programs that are using your CPU, and to close resource-intensive programs that are taking juice away that NinjaTrader would otherwise be using.

[zhiga]

How much physical RAM does this computer have? And what instruments are your 2-3 instruments for which you are loading 1-tick bars for 3-5 days of data? Your description sounds like you are running out of RAM and the computer is starting to swap.

You should also track down what is using all your space in Windows when it is being used. One idea would be this: https://www.wikihow.com/Find-the-Lar...-in-Windows-10 or https://www.mindgems.com/article/find-largest-folders/. Armed with this information you will be in a better position to understand the problem.

RAM is 48 GB. But hard Disk is small. It was about 45-50 GB free space before this issue . Mostly I test on GC, NQ and CL.

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your notes.

This could possibly be caused by the historical data on your PC. Please clear the historical data on your PC by opening a File Explorer window, navigating to the Documents\NinjaTrader 8\db folder, and deleting the contents inside of the day, minute, and tick folders (do not delete to folders themselves).

After clearing the historical data, close all workspaces on your platform and test the 'SampleIntrabarBacktest' reference sample from the help guide on a new blank chart.

SampleIntrabarBacktest: https://ninjatrader.com/support/help...ipt_strate.htm

Does the same behavior occur after clearing all historical data, closing all workspaces, and testing the SampleIntrabarBacktest reference sample?

[zhiga]

Hello zhiga,

Thanks for your notes.

This could possibly be caused by the historical data on your PC. Please clear the historical data on your PC by opening a File Explorer window, navigating to the Documents\NinjaTrader 8\db folder, and deleting the contents inside of the day, minute, and tick folders (do not delete to folders themselves).

After clearing the historical data, close all workspaces on your platform and test the 'SampleIntrabarBacktest' reference sample from the help guide on a new blank chart.

SampleIntrabarBacktest: https://ninjatrader.com/support/help...ipt_strate.htm

Does the same behavior occur after clearing all historical data, closing all workspaces, and testing the SampleIntrabarBacktest reference sample?

Hello Brandon
Thanks, I'll try this option.

The question is not related to this case. There is one indicator that works with TickReplay Chart, there is another indicator that works without TickReplay. How to use them together in the 3d script (indicator or strategy)?

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your notes.

You would have to enable Tick Replay for the data series if you are using an indicator that requires Tick Replay within a strategy.

There would be no means for enabling Tick Replay for only a single indicator being used in the strategy.

TickReplay: https://ninjatrader.com/support/help...ick_replay.htm

Developing for Tick Replay: https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm?zoom_highlightsub= developing+for+tick+replay

[zhiga]

Hello zhiga,

Thanks for your notes.

You would have to enable Tick Replay for the data series if you are using an indicator that requires Tick Replay within a strategy.

There would be no means for enabling Tick Replay for only a single indicator being used in the strategy.

TickReplay: https://ninjatrader.com/support/help...ick_replay.htm

Developing for Tick Replay: https://ninjatrader.com/support/helpGuides/nt8/developing_for__tick_replay.htm?zoom_highlightsub= developing+for+tick+replay

Thank you, Brandon

Can you clarify the next point.
in BlockVolume Indicator I replaced:
else if (State == State.Configure)
AddDataSeries(BarsPeriodType.Tick, 1);
to
else if (State == State.Configure)
AddDataSeries(BarsPeriodType.Second, 1);

I did not change the logic of the calculation.
But the new indicator is not updated on the Chart. Only after F5 it draws bars. Why? How to fix?

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your note.

This would change the indicator so that there is an added 1-second data series instead of an added 1-tick data series used for calculations.

Do you see any error messages in the Log tab of the Control Center?

Are you connected to a live data feed connection and see realtime chart bars populating?

Please send me an exported copy of the script so that I may test it on my end. To export a script, go to Tools > Export > NinjaScript AddOn.

Exporting: https://ninjatrader.com/support/help...tAsSourceFiles

I look forward to assisting further.

[zhiga]

Hello zhiga,

Thanks for your note.

This would change the indicator so that there is an added 1-second data series instead of an added 1-tick data series used for calculations.

Do you see any error messages in the Log tab of the Control Center?

Are you connected to a live data feed connection and see realtime chart bars populating?

Please send me an exported copy of the script so that I may test it on my end. To export a script, go to Tools > Export > NinjaScript AddOn.

Exporting: https://ninjatrader.com/support/help...tAsSourceFiles

I look forward to assisting further.

Brandon,

I'd connected to live dara through IQFeed.
Tested on 1-min and 30-sec Chart.
There are no error in the Log tab of the Control Center.
Please find in attachments.

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your notes.

I see you did actually change the logic of the script by adding BlockSize_min and BlockSize_max and using those values for calculations in the CalculateBlockVolume section of code.

When testing the script I see that the indicator plot is showing a value of say 34. Then, when you Reload NinjaScript, the value of that plot bar changes to 705. This behavior is likely being cause by your custom calculations in the script.

You would have to debug the logic by adding prints in the script to understand exactly what is causing the unexpected behavior to occur.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

[zhiga]

Hello zhiga,

Thanks for your notes.

I see you did actually change the logic of the script by adding BlockSize_min and BlockSize_max and using those values for calculations in the CalculateBlockVolume section of code.

When testing the script I see that the indicator plot is showing a value of say 34. Then, when you Reload NinjaScript, the value of that plot bar changes to 705. This behavior is likely being cause by your custom calculations in the script.

You would have to debug the logic by adding prints in the script to understand exactly what is causing the unexpected behavior to occur.

Below is a link to a forum post that demonstrates how to use prints to understand behavior.
https://ninjatrader.com/support/foru...121#post791121

The first version that was it only with a replacement DataSeries (AddDataSeries(BarsPeriodType.Second, 1);).
BlockSize_min is BlockSize in BlockVolume indicator.
BlockSize_max is upper level.
If You take it away from one place in code, and leave everything as in BlockVolume Indicator, it won't work either.

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your notes.

After further investigation, we have found that the BlockVolume indicator that comes with NinjaTrader has some logic in the code of the indicator that requires an added 1-Tick series added to the script. This logic likely will not work with an added 1-Second series.

if (!inTransition && hasCarriedOverTransitionTick && !forceCurrentBar && Calculate == Calculate.OnBarClose)
CalculateBlockVolume(true);

Since this indicator logic requires a 1-Tick series to calculate values properly, it will not function correctly if you use an added 1-Second series instead of an added 1-Tick series. You would need to leave the AddDataSeries(BarsPeriodType.Tick, 1); code in State.Configure.

[zhiga]

Hello zhiga,

Thanks for your notes.

After further investigation, we have found that the BlockVolume indicator that comes with NinjaTrader has some logic in the code of the indicator that requires an added 1-Tick series added to the script. This logic likely will not work with an added 1-Second series.

if (!inTransition && hasCarriedOverTransitionTick && !forceCurrentBar && Calculate == Calculate.OnBarClose)
CalculateBlockVolume(true);

Since this indicator logic requires a 1-Tick series to calculate values properly, it will not function correctly if you use an added 1-Second series instead of an added 1-Tick series. You would need to leave the AddDataSeries(BarsPeriodType.Tick, 1); code in State.Configure.

Thank you Brandon for your answer.
do you mean ,the nuance is in the calculation of the variable "inTransition"?
Can you explain in more detail why this works correctly only with the tick series?

[NinjaTrader_BrandonH]

Hello zhiga,

Thanks for your notes.

The BlockVolume indicator code requires that a 1-Tick data series is added to the script in order for the script to properly calculate block volume and plot values. In CalculateBlockVolume(bool forceCurrentBar), inTransition and hasCarriedOverTransitionTick needs tick data to detect individual ticks that occur.

Since it requires an added 1-Tick data series, you must leave the AddDataSeries(BarsPeriodType.Tick, 1); code in the indicator for it to properly function.