Writing to CSV file during Optimization in Strategy Analyzer

[cb4gusto22]

Hello,

My strategy makes final calculations during OnTermination and outputs to various CSV files. The strategy works fine when applied to a chart or backtested in Strategy Analyzer.

However, the problem arises when I attempt to run it through the optimizer. It seems like it runs through once, but when it moves to the next iteration, one of the CSV files is still in use. Therefore, only one iteration of the optimization occurs and no trades are generated for the rest of the iterations.

I believe it is because the file is still in use from the first run and future runs aren't able to access the file. Any ideas on how to either write concurrently (if that is even possible?) or to wait and then write when available. It's important that this be able to be done from the Optimizer in the Strategy Analyzer. Thanks for any insight!

[NinjaTrader_PatrickH]

Hello cb4gusto22,

Thank you for your post.

Are you using StreamWriter or System.IO to write to the external file?

[cb4gusto22]

Hi Patrick,

Using StreamWriter I believe. I declare a private variable like this:

private System.IO.StreamWriter sw2

and then create the file in OnBarUpdate if it isn't already created:

if (! File.Exists(file))
{
sw2 = File.AppendText(file);
sw2.WriteLine("column headers");
}
else
sw2 = File.AppendText(file);

Lastly, I write to the file during OnTermination where calculations are performed:

try
{
sw2.WriteLine(calculations);
}
catch (Exception e)
{
Log("Error on file OnTermination output.", NinjaTrader.Cbi.LogLevel.Error);
throw;
}

Need it to run correctly in the Optimizer so that the calculation for each run is written to the file. Thank you!

[NinjaTrader_PatrickH]

Hello cb4gusto22,

Thank you for your response.

I may not fully understand the issue here. So the file is only saving one run but you do not see any errors, correct? Is the run that is saved the most recent or the first?

[cb4gusto22]

Hi Patrick,

It is only saving the first run, then there are errors in the NT log/output window as I believe the file is still in use as the calculation from another run(s) is trying to write to the file as well. Therefore, only the calculation from the first run is properly written and none of the other runs actually produce trades (or write to the file).

[cb4gusto22]

Hi Patrick,

For more info, please see the following thread that I originally posted to:

http://ninjatrader.com/support/forum/showthread.php?p=450117#post450117

Thank you!

[bltdavid]

Hi Patrick,

It is only saving the first run, then there are errors in the NT log/output window as I believe the file is still in use as the calculation from another run(s) is trying to write to the file as well. Therefore, only the calculation from the first run is properly written and none of the other runs actually produce trades (or write to the file).

In essence, all optimization runs are trying to update the same filename?

That means the file is a shared resource and multiple threads are
trying to access it all at once.

Have you tried putting a lock() around this code?

Study why & when to use lock() here:

http://stackoverflow.com/questions/59590/lock-keyword-in-c-sharp

Using lock() is simple, see example here:

http://stackoverflow.com/questions/42505/thread-safe-use-of-a-singletons-members/42517#42517

[cb4gusto22]

Great suggestion, bltdavid!

After playing around for a while, I was able to get a solution that works for 10 runs. However, when I try to up it to say 20 runs, there is problems accessing the file again and errors occur. Any thoughts on that part? Thank you very much!

[bltdavid]

... there is problems accessing the file again and errors occur. Any thoughts on that part?

Nope. I can't see your monitor.

What problems?
What errors?

Thank you very much!

My pleasure, glad it's coming along.

[cb4gusto22]

Touche! I actually wrote out a better description but then as I was working on it concurrently *believe* I found a solution. Here was what I had wrote:

************************************************** ***********************************
Let me explain further. In the variables section, I create a lock:

private object edgeLock = new object();

Nothing relevant to this issue occurs until OnTermination where the streamwriter is created and the relevant calculation is made (eRatio):

string edge = Cbi.Core.UserDataDir.ToString() + "PPDailyTimeBased-Edge.csv"; // name of the file path
lock(edgeLock) // CREATE THE FILE IF IT DOESN'T EXIST, OTHERWISE APPEND
{
	try
	{
		if (! File.Exists(edge)) 
		{
			sw2 = File.AppendText(edge);
			sw2.WriteLine("Time," + "NDays Exit," + "Proximity," + "Edge Ratio"); // column headers
		}
		else 
			sw2 = File.AppendText(edge);
	}
	catch (Exception e)
	{
		// Outputs the error to the log
		Log("Error on StreamWriter Edge Startup.", NinjaTrader.Cbi.LogLevel.Error);
		throw;
	}
}
lock(edgeLock) // WRITE THE NECESSARY INFO TO THE FILE
{
	try
	{
		sw2.WriteLine(DateTime.Now + "," + NDays + "," + Proximity + "," + eRatio);
	}
	catch (Exception e)
	{
		Log("Error on Edge OnTermination output.", NinjaTrader.Cbi.LogLevel.Error);
		throw;
	}
	if (sw2 != null) // GET RID OF RESOURCES WHEN IT'S FINISHED
	{
		sw2.Close();
		sw2.Dispose();
		sw2 = null;
	}
}

Yesterday, when I ran the script through the optimizer and incremented "NDays" from 1 to 10 by 1, it worked - meaning all the calculations were performed and written to the file. When I tried 1 to 20 by 1, it had errors. Now today, even 1 to 10 also has errors on the first try, but not on others, so it is inherently unstable.

The errors are:
1. Not all calculations are saved to the file, while sometimes one run might be saved twice. For instance, NDays = 5 was saved twice while NDays = 7, 9, 10 were omitted.

2. Per the error handling in the code above, the log has two errors: "Error on StreamWriter Edge Startup" and an error calling OnTermination (cannot access the file because it is being used by another process).
************************************************** ************************************************** *

However, I got rid of virtually all of the problems by adding one more lock containing a termination of the streamwriter resources at the very end of OnTermination. No more missing calculations or error messages in the log, and I was able to do 150 runs! The only minor issue is that the run related to the highest optimized performance metric is written to the file twice (with the second coming at the very end). I believe this is due to the optimizer holding the best performing run in memory until it gets knocked off or until the optimization is complete. Can NT or someone else confirm that this is the case? Thank you!

[koganam]

Hi Patrick,

It is only saving the first run, then there are errors in the NT log/output window as I believe the file is still in use as the calculation from another run(s) is trying to write to the file as well. Therefore, only the calculation from the first run is properly written and none of the other runs actually produce trades (or write to the file).

Dispose() of the sw2 StreamWriter when you complete writing, then close the file on each run.

[cb4gusto22]

Thanks, Koganam, that's what I'm doing along with a lock() and it now works. Would you be able to shed any light on the following?

The only minor issue is that the run related to the highest optimized performance metric is written to the file twice (with the second coming at the very end). I believe this is due to the optimizer holding the best performing run in memory until it gets knocked off or until the optimization is complete. Can NT or someone else confirm that this is the case?

Thanks for any insight!

[NinjaTrader_PatrickH]

Hello cb4gusto22,

how are you writing it? What are you looking for (the condition) and what actions are taken? This will give us a better idea on what to look for.

[cb4gusto22]

Hi Patrick,

Exactly what I am doing is described in Posts #1 and #10. Essentially I am using the optimizer to backtest a strategy while incrementing one parameter and making a calculation. Each parameter and related calculation is written to a file so that at the end of the optimization, I have a file with each parameter/calculation. However, one of the parameter/calculations is always written twice - it happens to be the one that "scores" the highest in the backtest (e.g. profit factor or whatever I'm optimizing for).

[bltdavid]

Hi Patrick,

Exactly what I am doing is described in Posts #1 and #10. Essentially I am using the optimizer to backtest a strategy while incrementing one parameter and making a calculation. Each parameter and related calculation is written to a file so that at the end of the optimization, I have a file with each parameter/calculation. However, one of the parameter/calculations is always written twice - it happens to be the one that "scores" the highest in the backtest (e.g. profit factor or whatever I'm optimizing for).

Don't be fooled by the optimizer ... if you click on the Trades tab or some of the other tabs, such as the Chart tab, I'm pretty sure the analyzer will re-run your strategy with the specific settings of that ONE specific run, so that it can generate (for example) trade details for the Trades tab and trade markers for the Chart tab.

So, for this run, too, the results are also being written to your results file.

Voila, you now have a duplicate in your results file!

Try to design some experiments to test this theory. Start with doing nothing immediately after all optimizing runs are finished, make sure the optimizer ends on the Summary tab (I don't recall if the data from the Summary tabs need to be re-generated) and check your results file. Now click around to view results of the other runs -- but make only one mouse click at a time, and then re-check your results file after each click.

Better idea: put some Print statements in OnStartUp & OnTermination. Then, after the optimizer shows you its top ten list (aka, it appears finished), carefully watch the OutputWindow as you do these clicking around experiments.

My point is: I don't think the optimizer caches every single item of output data it needs to display when you click on the various tabs to get specific details of each of the top ten runs -- I think the optimizer just re-runs the strategy on-the-fly, with the specific settings for that run, to get the trade details, and then it displays those newly generated results nicely formatted in whatever tab you clicked on.

It's a mouthful to explain, but the on-the-fly strategy execution to regenerate trade details to display is probably messing up your results file.