NinjaTrader crashes while loading historical data

[Harry]

NT 8.0.2.0 connected to Kinetick Live Data Feed

Tried to update my historical FOREX charts via the Historical Data Manager. For updating I selected the instrument list "FOREX", which contains the following instruments

- AUDUSD
- EURCHF
- EURGBP
- EURJPY
- EURUSD
- GBPUSD
- USDCAD
- USDCHF
- USDJPY

System.OutOfMemoryException was unhandled
Message: An unhandled exception of type 'System.OutOfMemoryException' occurred in NinjaTrader.Core.dll

Details in the trace file shown were:

2016-12-10 23:47:10:282 *************** unhandled exception trapped ***************
2016-12-10 23:47:10:282 Exception of type 'System.OutOfMemoryException' was thrown.
2016-12-10 23:47:11:363 ERROR: Failed to call 'Add' method: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at NinjaTrader.Data.Chunk..ctor(Int32 lengthOfChunk, Double tickSize) at NinjaTrader.Data.BarsSeries.AddBar(Double open, Double high, Double low, Double close, DateTime time, Int64 volume, Double tickSizeIn, Double bid, Double ask) at NinjaTrader.Data.Bars.AddBar(Double open, Double high, Double low, Double close, DateTime time, Int64 volume, Double tickSizeIn, Double bid, Double ask) at NinjaTrader.Data.BarsType.AddBar(Bars bars, Double open, Double high, Double low, Double close, DateTime time, Int64 volume, Double bid, Double ask) at NinjaTrader.NinjaScript.BarsTypes.TickBarsType.OnD ataPoint(Bars bars, Double open, Double high, Double low, Double close, DateTime time, Int64 volume, Boolean isBar, Double bid, Double ask) at NinjaTrader.Data.BarsSeries.Add(Bars bars, Double open, Double high, Double low, Double close, DateTime time, Int64 volume, Double tickSize, Boolean isBar, Double bid, Double ask)
2016-12-10 23:47:12:431 System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at NinjaTrader.Data.BarsBytes.GetFileName(DateTime time, Instrument instrument, BarsPeriodType periodType, MarketDataType marketDataType)
at NinjaTrader.Data.BarsSeries.Save()
at NinjaTrader.Data.Bars.Save()
at NinjaTrader.Data.BarsSeries.<>c__DisplayClass120_2 .<DownloadFromProvider>b__4(Bars b, ErrorCode c, String m, Object s)
at NinjaTrader.Cbi.Instrument.<>c__DisplayClass202_0. <RequestBars>b__1(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCal lback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(Exec utionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionCon text executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System. Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWa itCallback()

Physical memory used was only 4.3 GByte from 8.3 Gbyte of total physical memory of my PC. However, I could observe that the RAM went up by 1GByte during the loading of historical data.

NinjaTrader was entirely frozen and I had to close via the Process Explorer.

Afterwards I got an error message: "Unable to recover NinjaTrader custom assembly. Please reinstall NinjaTrader."

All this happened just, because I started downloading historical data.

[Harry]

I have further enquired into this behaviour.

I have an even larger instrument list of futures contracts, and when I download historical data - tick, minute and daily data all selected - I observe the following

- RAM utilization is increased when data is loaded
- the instrument which is loaded is shown at the bottom of the HIstorical Data window
- sometimes when an instrument has been loaded the RAM is liberated, sometimes it cumulates data from several contracts
- in general it cuts back RAM usage before causing any problems

However, when loading FOREX data things are different.

Loading single FOREX contracts works. But the RAM remains high after loading the data, which shows that it is still sitting in the RAM after it has been written to the historical data base.

EURGBP RAM goes up to 1.060 MByte
EURCHF RAM continues to go up to 1.400 MByte (in between dropping back to 1.000 MByte)
AUDUSD RAM NinjaTrader crashes at around 1.400 MByte

At this stage there is stil 4.000 MByte physical memory available.

This is simply not working, because the memory is not managed here.

To reproduce, please

-> create an instrument list with the 9 FOREX pairs mentioned below
-> connect to Kinetick live data feed
-> open the Historical Data window -> Load
-> select all tick, minute and daily data
-> load data from November 1 to December 10 by selecting the instrument list

My machine is a solid HP server with 8 GByte of RAM. The Process Explorer shows that there are at least 4GByte of RAM available when NinjaTrader crashes.

[NinjaTrader_PatrickG]

I could not reproduce this behavior, however, I suspect this may be partly due to the fact that Kinetick limits tick data requests to 8 historical days during RTH hours. Because of this, I cannot request all the historical data which you had requested in your provided steps.

I will test this again after 4:30PM Eastern time and follow up with the results.

[Harry]

I could not reproduce this behavior, however, I suspect this may be parlty due to the fact that Kinetick limits tick data requests to 8 historical days during RTH hours. Because of this, I cannot request all the historical data which you had requested in your provided steps.

I will test this again after 4:30PM Eastern time and follow up with the results.

Thank you for your answer. Indeed I tried to update my historical data over the weekend, when tick requests were not limited to 8 days.

I was basically doing the same thing that I always did with NinjaTrader 7.

Please find attached the trace file that shows the problem. In order to obtain a small trace file, I have shut down NinjaTrader, then restarted it and downloaded historical data.

[NinjaTrader_PatrickG]

I was able to recreate the crash when using NinjaTrader 8 32-bit. Testing the exact same steps with NinjaTrader 7 32-bit does not produce a crash. I have submitted my findings to the Development Team and I will follow up when I have additional information. This is being tracked internally as NTEIGHT-10905.

Thanks for bringing this to our attention.

[Harry]

Did you also test the 64-bit version?

[karlomacariola]

I'm experiencing the same issue. Thank you for raising this.

[NinjaTrader_PatrickG]

Did you also test the 64-bit version?

Yes, I tested the 64-bit version of NinjaTrader 7 and NinjaTrader 8. These tests completed as expected.

[NinjaTrader_Matthew]

Did you also test the 64-bit version?

We were able to run further analysis and found that the memory usage in this scenario is expected due to the size of the forex tick files, and was not much worse than NinjaTrader 7.

We are not aware of any issues using the 64-bit version.

Is there a specific reason you were using the 32-bit version?

[Harry]

We were able to run further analysis and found that the memory usage in this scenario is expected due to the size of the forex tick files, and was not much worse than NinjaTrader 7.

We are not aware of any issues using the 64-bit version.

Is there a specific reason you were using the 32-bit version?

However, the main problem here is how NinjaTrader 8 handles the data.

When an instrument list, say of 6 FOREX instruments is loaded, NinjaTrader does not liberate the RAM once the data of the first instrument has been loaded. In the case I observed data for the first and the second FOREX instrument was correctly loaded. The crash only occured after the 3rd instrument was loaded, because NinjaTrader 8 did not liberate the RAM. RAM usage is cumulated and therefore quickyl hits the 3 GByte limit imposed by 32-bit applications.

If this was done correctly, NinjaTrader would

(1) download tick data to the RAM
(2) save that data to the historical data base
(3) then liberate the RAM before data for the next instrument is downloaded

However, the third step is omitted and FOREX tick data from several instruments is cumulated in the RAM until the application crashes.

[Harry]

With NinjaTrader 7 my default list contains both futures and FOREX contracts. For NinjaTrader 8 I have separated them into two instrument lists, one for futures contracts and one for FOREX data.

I have now downloaded data for the aggregate instrument list via the Historical Data Manager of NinjaTrader 7 (32-bit). This list is much larger than the FOREX instrument list that I wanted to download with NinjaTrader 8.


Result:

NinjaTrader 7(32-bit) does not use more than 670 MByte of RAM (this was the maximum value before it dropped back to 410 MByte)

NinjaTrader 8(32-bit) crashes when loading data for the 3rd instrument with RAM running at 1.400 MByte.

In both cases I have downloaded data from November 1 to Dec 14 (Dec 12).


NinjaTrader 7 and NinjaTrader 8 are definitely not behaving in a similar way.

[NinjaTrader_Matthew]

However, the main problem here is how NinjaTrader 8 handles the data.

When an instrument list, say of 6 FOREX instruments is loaded, NinjaTrader does not liberate the RAM once the data of the first instrument has been loaded. In the case I observed data for the first and the second FOREX instrument was correctly loaded. The crash only occured after the 3rd instrument was loaded, because NinjaTrader 8 did not liberate the RAM. RAM usage is cumulated and therefore quickyl hits the 3 GByte limit imposed by 32-bit applications.

If this was done correctly, NinjaTrader would

(1) download tick data to the RAM
(2) save that data to the historical data base
(3) then liberate the RAM before data for the next instrument is downloaded

However, the third step is omitted and FOREX tick data from several instruments is cumulated in the RAM until the application crashes.

This is definitely true - NinjaTrader 8 data load logic has been expanded to load data asynchronously which results in faster load times. This also uses more memory during load. The memory should be cleared up as soon as the requests are done, but they normally finish around the same time which is the cause for this behavior you have observed.

We do see a potential to give customers an option for either behavior so we have logged an SFT in response. I will follow up with the ID # for that request when it has been processed.

With NinjaTrader 7 my default list contains both futures and FOREX contracts. For NinjaTrader 8 I have separated them into two instrument lists, one for futures contracts and one for FOREX data.

I have now downloaded data for the aggregate instrument list via the Historical Data Manager of NinjaTrader 7 (32-bit). This list is much larger than the FOREX instrument list that I wanted to download with NinjaTrader 8.


Result:

NinjaTrader 7(32-bit) does not use more than 670 MByte of RAM (this was the maximum value before it dropped back to 410 MByte)

NinjaTrader 8(32-bit) crashes when loading data for the 3rd instrument with RAM running at 1.400 MByte.

In both cases, I have downloaded data from November 1 to Dec 14 (Dec 12).


NinjaTrader 7 and NinjaTrader 8 are definitely not behaving in a similar way.

I am not so sure we observe the same results then.

In testing, it was documented in this scenario that NinjaTrader 7 would use up to 1.8GB, where NinjaTrader 8 would cap out at 1.3GB before it crashes. The difference in handling is attributed to NT7 being large memory address aware which we are currently debating adding back to NInjaTrader 8 if we see significant demand for 32-bit support. (If you can follow up per my last post, that would be helpful in our decision-making process).

If your tests show NinjaTrader 7 never uses more than 670 MByte our tests must be different, and there are some variables I would need to work out before I can comment further.

There are a few reasons that would be the case:

- NinjaTrader 7 defaults to full pip
- NinjaTrader 8 defaults to tenth pip
- NinjaTrader 8 now stores "ask/bid" values with the last file, so the implicit size of memory is greater in NinjaTrader 8.

The memory usage between version would never be exactly the same due to the tick data storage changes, but should only account for a few hundred MB's in size. If you are seeing a difference of more than 20-50% in NinjaTrader 8, we would be curious what the cause would be.

To make sure tests are apples to apples as possible, please double check your NinjaTrader 7 is configured to show "tenth pip" from Tools-> Options-> Quote currencies in Tenth Pip

Lastly - Right now, forex tick data is limited from data servers as previously mentioned. As a result, the memory consumption at the current time will be much lower (i.e., in a recent test this morning, NinjaTrader 8 used ~450MB, while NinjaTrader 7 used ~300MB, which is significantly less than our tests showed earlier this week)

[NinjaTrader_Matthew]

We do see a potential to give customers an option for either behavior so we have logged an SFT in response. I will follow up with the ID # for that request when it has been processed.

This is logged under SFT-1806 : Data loading memory management options

We will be doing more tests to ensure this area is working as best as possible.

[Harry]

I have now tested the behaviour with the 64-bit version of NinjaTrader 8 and did not have any problems.

Actually I only discovered this bug by accident. After installing NinjaTrader 8.0.2.0, I had not yet attached the NinjaTrader 8 (64-bit) icon to the task bar and therefore accidentally started the 32-bit version.

The 64-bit version behaves as expected. I think that by now most of the users have 64-bit machines.

[D Trader]

I can confirm that the ever-increasing memory use when downloading historical data is also an issue for the 64-bit version of NT8 -- at least when it is running on Windows 7.

On each successive symbol or date range that is downloaded, the memory required by NT8 is taken from the system and added to the memory already reserved for the prior download. This cycle repeats until the system literally runs out of RAM. As a result, NT8 freezes up (as does any other app running on Windows) or becomes unstable. At that point, the only way out is to kill the program in Task Manager as the Control Center no longer responds to user input.

A work-around is to monitor memory use while downloading historical data, and restart NT8 whenever total RAM used approaches the system's limit. Also, try using smaller date ranges to give more opportunity to control memory consumption.

One final observation -- if I wait about 4-5 minutes between historical downloads, the memory reserved for the last download is released back to the OS! So apparently there is a mechanism to free up memory, but it just takes too long before it kicks in.

All of the above can be monitored by simply observing the NinjaTrader.exe process in Windows Task Manager while performing historical downloads.

Hope this helps.