Market Analyzer Crashes if loading all SP500 instruments with samplestreamwriter ind

[wallsteetking]

I am having some issues with the market analyzer with samplestreamwriter. Every time I load all SP500 instruments with indicator samplestreamwriter. Ninjatrader crashes. Here are the steps to replicated the error. I have attached 3 ninjatrader indicators. The first one is SampleStreamWriter, the second is is Spytest and third a indicator called ky1.

Step 1: Go to market Analyzer > click columns indicators > samplestreamwriter > DataSeries 1 day > # of bars to look back 3001.

Step 2: Load all sp500 instruments.

Step 3: Wait 15-30 minutes while some of the data is being loaded. After it gets to about 100 stocks it crashes.

I tried it on 3 different computers with 3 different operating system and still ninjatrader crashes. Please advise how to fix this issue. I even tried it on the cloud with high memory ram and processing power but issues still occurs.

Please advise how to fix this issue. Thank you

[NinjaTrader_Jesse]

Hello wallsteetking,

I wanted to check, are you receiving log errors relating to the text file not being able to be accessed? The provided sample is really set up to be used as a single instance on a single instrument as just an example of the concept. In this case all of the instruments would be trying to access the same file at once which should cause the script to fail.

Are you trying to write all information per instrument to a single file? Or are you trying to record each instrument as an individual file? In the second case there would not be any problems writing to file as each instrument should have its own file, but in the first case there would be lost data per the logic in the script each time it fails to write while in use.

I look forward to being of further assistance.

[wallsteetking]

Hello Jesse,

I am trying to record each instrument as an individual file. Which would mean I would have about 500 files in that folder. But with Spytest indicator on, which is a multi-instrument indicator( i believe the error comes from the multi-instrument that i am using on the indicator) Ninjatrader crashes. The error disappears when I have no indicators with multi-instrument in them. I hope this helps. Thank you

[NinjaTrader_Jesse]

Hello,

To confirm the error is happening with the spytest and not the sample, Is this correct? If you are unsure which is causing the crash, I would suggest to only apply one indicator at a time to determine if that is the cause.

The SpyTest does contain an error, one Item I can see you would need to change would be to add a check to ensure there are bars to process. When running this I can see errors being printed to the Tools -> Output window.

You could add in the following statement before Plotting to prevent the error.

if(CurrentBars[0] < 1 || CurrentBars[1] < 1) return;
Plot0.Set(((Closes[1][0] / Opens[1][0])-1)*100) ;

I look forward to being of further assistance.

[wallsteetking]

Hello Jesse,

I have added the updated code and I still get the error. What I know so far about the error. First, if I uninstall Ninjatrader and just load only SPYTest into samplestreamwriter then it works. BUT, if I add SpyTest and Ky1 indicator ninjatrader crashes.

what I tried so far:

1) I tried renamed the db file to OLDNinjaTrader.SDF = Failed
2) I uninstalled ninjatrader completely and reinstalled it = Failed
3) I tried Ninjatrader 8 = Failed
4) Deleted all workspace = Failed
5) Ran it on older operation machine = Failed



I am able to make it work only if I load 100 stocks into market analyzer anything higher than 112 stocks Ninjatrader crashes.

I really need to fix this issues. I been trying for months and I am out of ideas. Thank you

[NinjaTrader_Jesse]

Hello,

I wanted to clarify the steps you are using.

In the first post you noted you were have 3 indicators but only noted the SampleStreamWriter in the instructions.

Are all three indicators needed to see the crash? if so what are the settings you are using for the other two? could you include the total setup steps you are using to reproduce the crash?

The SpyTest indicator specifically was having errors, after making the changes had you created a new Market analyzer window or had you reloaded the existing instance? I would suggest to try a new column and remove the existing column after making this change.

Also are you able to load the entire list without the indicators applied?

I look forward to being of further assistance.

[wallsteetking]

Yes total of 3 indicators

1) SampleStreamWriter - indicator
2) SPYTest - Indicator
3) ky1 - indicator

I put SpyTest and ky1 into SampleStreamWriter. Then I go to Market Analyzer click on columns --> indicators and load SampleStreamWriter. Then I load all SP500 stocks.

Are all three indicators needed to see the crash?
Yes, because SampleStreamWriter has both SPYTEST and ky1 inside the code.

Good NEWS! We fixed the problem. By NOT removing ASKPRICE/BIDPRICE/LASTPRICE (like u said columns and Not reloading existing instance) from market analyzer now it works.

Thank you this made my day. A++++ Phenomenal job

[wallsteetking]

.....................

[wallsteetking]

Ok I think I got it.

in Market analyzer I have to load all columns with the indicators. for example

Market Analyzer
=============
Instrument / Ask Price/ Bid Price/ Last Price/ SpyTest / Ky1/ SampleStreamWriter.

Then and only then it works.

[traderthomas]

Everything crashes on Ninjatrader unless you have Texas size memory. Seems old fashioned You can run bigger programs on a phone without them crashing.

[wallsteetking]

True

[NinjaTrader_Jesse]

Hello,

I wanted to reply back on this item.

I can see this working as expected on my end but only after you correct the errors we had previously discussed. Additionally I spotted one more error in the sample you provided.

In the SampleStreamWriter's OnTermination you have sw.Dispose(); outside of the if(sw == null) check. This does cause an exception for every instrument and would cause a lot of logging.

I do see this also can take a large toll on the CPU while initially processing, additionally this would hold a large amount of ram open as you are opening a stream writer for each instrument in the list. In the case the computers performance suffers because of this I would suggest to load less instruments at a time or a stronger PC may be required for this type of logic.

To recap, the spytest indicator would need to be modified to check if there are enough bars, I would also suggest to limit its plotting to the BarsInProgress 0:

if(CurrentBars[0] < 1 || CurrentBars[1] < 1) return;
if (BarsInProgress == 0)
{	
     Plot0.Set(((Closes[1][0] / Opens[1][0])-1)*100) ;
}

You would also need to edit the SampleStreamWriter to only have the following in OnTermination:

protected override void OnTermination() 
{
	if (sw != null)
	{
		sw.Dispose();
		sw = null;
	}
}

Please let me know if I may be of addional assistance.

[wallsteetking]

Hey Jesse,

I have tried those updates, it only works when I add the same indicators to market analyzer that are inside the samplestreamwriter with the same amount of bars to look back. However, if I run samplestreamwriter by it self on Market Analyzer it fails again.

furthermore, in your last post you wrote

To recap, the spytest indicator would need to be modified to check if there are enough bars

I added BarsRequired = 3001; inside initialize() on all 3 indicators and
MaximumBarsLookBack = MaximumBarsLookBack.Infinite;

Still fails. The only time it works is if I do 100 stocks.

would it be possible to fix this issue with the next update?