NinjaTrader - Writting to Memory then to Disk

[cutzpr]

I would like to make the below code more efficient by writing all text(CSV) to memory before writing to disk. The CSV file could have upwards of 100,000 or rows. With the current code, it opens and closes the file for R/W each time. There must be a better way. I am calling the WriteToCSV on each OnBarUpdate. The file on disk after it runs is approx. 100MB so there is no issue storing the entire contents in memory before writing to disk. I run this code with historical data and it takes a long time because of all in I/O operations R/W to disk

The loops are to generate the signal code and are necessary. The signal code is what is going to be eventually written to disk. Below is an example of a row ZZZZZZZZZZ-ZZZZZZZZZZ-ZZZZZZZZZZ-BBAAAABBBB-ZBAAAAAAZZ-CCBCCXYZZZ-XXCXXYYZZZ-BBBBBBBBCC 1.1748 1.1748 1.17475 1.17475 10/5/2020 0:02'

private void WriteToCSV()
{

if(CurrentBar <= BarsRequiredToPlot + LookBack) return;

StringBuilder header = new StringBuilder();
StringBuilder logEntry = new StringBuilder();
StringBuilder signalCode = new StringBuilder();

string fullPath = System.IO.Path.Combine(filePath,fileName);

header.Append("Index Key,");
header.Append("CurrentBar,");
header.Append("SignalCode,");
header.Append("Open,");
header.Append("High,");
header.Append("Low,");
header.Append("Close,");
header.Append("BarCloseTime");

//INPUT1
for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input1_1_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input1_2_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input1_3_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input1_4_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input2_1_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input2_2_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input2_3_trendchar[i]);
if(i == LookBack) signalCode.Append("-");
}

for(int i = 1; i <= LookBack; i++)
{
signalCode.Append(input2_4_trendchar[i]);
}

logEntry.Append(indexKey);
logEntry.Append(",");
logEntry.Append(CurrentBar-1);
logEntry.Append(",");
logEntry.Append(signalCode.ToString());
logEntry.Append(",");
logEntry.Append(Open[1]);
logEntry.Append(",");
logEntry.Append(High[1]);
logEntry.Append(",");
logEntry.Append(Low[1]);
logEntry.Append(",");
logEntry.Append(Close[1]);
logEntry.Append(",");
logEntry.Append(Time[1]);
indexKey++;

try
{
if (File.Exists(fullPath) == false)
{
using (StreamWriter sw = new StreamWriter(fullPath, true))
{
sw.WriteLine(header); // If file doesnt exist, create it and add the Header
sw.WriteLine(logEntry); // Append a new line to the file
sw.Close(); // Close the file to allow future calls to access the file again.
}
}
else //File Does Exisit
{
if (IsFileLocked(fullPath) == false) //If file is not locked for editing
{
using (StreamWriter sw = new StreamWriter(fullPath, true))
{
sw.WriteLine(logEntry); // Append a new line to the file
sw.Close(); // Close the file to allow future calls to access the file again.
}
}
}
}
catch (Exception e)
{
// Outputs the error to the log
//Log(uniqueStrategy + ": ExecutionLog - cannot write and read at the same time.", NinjaTrader.Cbi.LogLevel.Error);
Print("ERROR WRITTING SignalLog");
}

}

[bltdavid]

Why not first try something more simple.

Have you tried?
Open and close the file just once.

For example, in OnStateChange, manage the open/close
of the file there, performing open/close operations just once.

For example,
During State.SetDefaults: Open the file
During State.Termination: Close the file

I run this code with historical data and it takes a long time because of all in I/O operations R/W to disk

It runs slowly because your code is preventing Windows from doing any
kind of memory buffering for the I/O operations -- because you close the
file after each write.

You want to invent your own code to 'buffer' your output in memory?
That is not very smart, don't do that -- I'd bet $10,000 it would probably
be a big waste. Why not let Windows do the buffering for you?

It would be a lot smarter to better understand how Windows (actually,
how all operating systems) perform automatic memory buffering of I/O
operations.

How does Windows do this?
Just open the file once somewhere at the beginning, let each OnBarUpdate
write to the file using that same StreamWriter reference, then close that
reference just once at the end of processing. That's it!

Guess what happens? Dramatic speed improvements due to the builtin
I/O buffering automatically performed by Windows -- which you get for
free by making a few relatively minor changes to your current code.

Under the hood, Windows does it's own buffering of your output. Only
when enough output has been collected (again, all under the hood) does
Windows perform a real write-to-disk. This is the exact effect you wish
to invent, but you'd get all that beautiful fully-debugged Microsoft code
for free if you'd just re-structure your own code to take advantage of it.

By closing the file in between each of your write operations, you have
effectively disabled all Windows attempts to do this buffering for you.

Make sense?

[NinjaTrader_ChelseaB]

Hello cutzpr,

While I cannot comment on saving to memory directly, I did want to ask, wouldn't saving the information to a string, and appending the string until you are ready to write all of the information to a file, be keeping it in memory?
The string variable is pointing to a memory location with that string.. You can write to file once you have all the data appended to the string.

[cutzpr]

Thank you both for your input. I was able to dramatically make it more efficient by creating models and a list of those models to export to CSV using CSVhelper. https://joshclose.github.io/CsvHelper/

I am posting this code here as a reference to help others.

using CsvHelper;
using System.Globalization;

public List<CsvRecord> CsvRecordsList;
private bool cSV_Written = false;

else if (State == State.Configure)
{
CsvRecordsList = new List<CsvRecord>();
}

public class CsvRecord
{
public string IndexKey { get; set; }
public string CurrentBAR { get; set; }
public string SignalCode { get; set; }
public string OpenPrice { get; set; }
public string HighPrice { get; set; }
public string LowPrice { get; set; }
public string ClosePrice { get; set; }
public string BarTimeClose { get; set; }
}

//Wiithin OnBarUpdate

CsvRecord csvRecord = new CsvRecord{
IndexKey = indexKey.ToString(),
CurrentBAR = (CurrentBar-1).ToString(),
SignalCode = signalCode.ToString(),
OpenPrice = Open[1].ToString(),
HighPrice = High[1].ToString(),
LowPrice = Low[1].ToString(),
ClosePrice = Close[1].ToString(),
BarTimeClose = Time[1].ToString(),
};

CsvRecordsList.Add(csvRecord);
indexKey++;


if(State == State.Realtime && cSV_Written == false) // Only write when I get to real time to disk
{
cSV_Written = true;

try
{
if (File.Exists(fullPath) == false)
{
using (StreamWriter sw = new StreamWriter(fullPath, true))
using (var csv = new CsvWriter(sw,CultureInfo.InvariantCulture))
{
csv.WriteRecords(CsvRecordsList);
csv.Flush();
}

}
}
catch (Exception e)
{
// Outputs the error to the log
//Log("ERROR Writing Historical SignalLog "", NinjaTrader.Cbi.LogLevel.Error);
Print("ERROR Writing Historical SignalLog " + e);
}
finally
{
CsvRecordsList.Clear();
}
}
else if(State == State.Realtime) // Only write when I get to real time to disk
{
try
{
using (StreamWriter sw = new StreamWriter(fullPath, true))
using (var csv = new CsvWriter(sw,CultureInfo.InvariantCulture))
{
csv.WriteRecord(csvRecord);
csv.NextRecord();
csv.Flush();
}
}
catch (Exception e)
{
// Outputs the error to the log
//Log("ERROR Writing RealTime SignalLog "", NinjaTrader.Cbi.LogLevel.Error);
Print("ERROR Writting Realtime SignalLog " + e);
}
}
}

[cutzpr]

I also cut down in the loops and made it easier to read.

var trendchars = new[] // let's make an array of it ...
{
input1_1_trendchar,
input1_2_trendchar,
input1_3_trendchar,
input1_4_trendchar,
input2_1_trendchar,
input2_2_trendchar,
input2_3_trendchar,
input2_4_trendchar
};

foreach (var trendchar in trendchars) // ... and process it in a loop
{

for (int i = 1; i <= LookBack; i++)
{
signalCode.Append(trendchar[i]);
}
signalCode.Append("-");
}

signalCode.Remove(signalCode.Length - 1, 1); // cut the last redundant "-"

[Lance El Camino]

I'm trying to reference CsvHelper and can't get it right. I'm a total hack with C# so any help is appreciated. I've added the reference but still get an error:

"The type System.Object is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51."

"The type System.IDisposable is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51."

"The type System.IO.TextReader is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51."

"The type System.Globalization.CultureInfo is defined in an assembly that is not referenced. You must add a reference to assembly 'netstandard, Version=2.1.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51."

[Lance El Camino]

Here's the declarations:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Text;
using System.Windows.Media;
using System.Xml.Serialization;
using System.IO;
using CsvHelper;
using System.Globalization;
​

And here's where I try to create the readers:

sr = new System.IO.StreamReader(path);
            csv = new CsvReader(sr, CultureInfo.InvariantCulture);​

[NinjaTrader_Emily]

Hello Lance El Camino,

Thank you for your notes.

The error messages you are receiving are related to missing references for the CsvHelper script that you are trying to use in NinjaTrader. Since this is from a 3rd party, the resolution is outside the scope of support we are able to offer. You might be able to do an internet search for more information about the netstandard reference that is mentioned in the errors and once you have the necessary file on your PC, you could try adding that reference in NinjaTrader to see if it resolves the errors. This thread is also open for anyone from the NinjaTrader forum community to assist you with the errors.

Thank you for your patience. Please don't hesitate to reach out with any NinjaTrader items we may assist you with.

[eleven]

Hello Lance El Camino,

Where you able to load and reference CsvHelper.dll without any errors? I'm trying to use CsvHelper in my indicator code.

UPDATE: I figured it out. I used a different version of the dll.

Thanks,
eleven