How much data can market replay play safely?

[kaywai]

Hi,

I ran 3 30 min charts of $EURUSD with an indicator on each chart concurrently as I was testing the indicator. After running about a month's worth of data, my whole PC grinded to a halt! Market Replay couldn't run anymore!

I checked the log and trace files and the message is

1/12/2013 2:49:22 AM|2|4|Unable to load server side config file: The operation has timed out
1/12/2013 4:48:49 AM|2|4|Unable to load server side config file: The operation has timed out
1/12/2013 5:48:45 AM|2|4|Unable to load server side config file: The operation has timed out
1/12/2013 6:48:46 AM|2|4|Unable to load server side config file: The operation has timed out
1/12/2013 7:48:42 AM|2|4|Unable to load server side config file: The operation has timed out
1/12/2013 8:49:01 AM|2|4|Unable to load server side config file: The operation has timed out

The trace file had an additional message:-

2013-01-12 12:00:27:832 Connection to NinjaTrader data server lost: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 size, SocketFlags socketFlags)
at NinjaTrader.Server.Client.Receive()
at NinjaTrader.Server.Client.Loop()
2013-01-12 12:00:28:074 ERROR: Connection to NinjaTrader historical market data server lost: An existing connection was forcibly closed by the remote host
2013-01-12 12:10:39:794 Server.Client.Loop0-1: 64.202.118.181/31654
2013-01-12 12:18:20:258 Server.Client.Loop0-2: 64.202.118.181/31654
2013-01-12 12:45:29:900 Server.Client.Loop1-2
2013-01-12 12:45:33:232 Connection to NinjaTrader data server lost: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 size, SocketFlags socketFlags)
at NinjaTrader.Server.Client.Receive()
at NinjaTrader.Server.Client.Loop()
2013-01-12 12:45:33:300 ERROR: Connection to NinjaTrader historical market data server lost: An existing connection was forcibly closed by the remote host
2013-01-12 12:53:51:043 Server.Client.Loop0-3: 64.202.118.181/31654
2013-01-12 12:53:52:785 Server.Client.Loop1-3

I currently run a Core 2 Quad 2.4Ghz with 6Gb of RAM on 64bit Win7.

Can someone tell me where the problem lies? And if possible, offer a solution.

Thanks and Regards

Kay Wai

[kaywai]

is it normal for NT7 to hog so much resources? Please see attach. Kay Wai

[kaywai]

this is what happens after running 2 charts for 3 hours. please see attached. NT7 is grinding to a halt! Help!

[marcow]

this is a known problem with market replay, I also posted a test strategy to reproduce this slowdown very fast.

see this thread:

http://www.ninjatrader.com/support/forum/showthread.php?t=52458

at post#29 NT support say they can reproduce the slowdown and will fix this in the next major release (don't ask for the "W" word )

Marco

[kaywai]

here is the status of running market replay with 1 30 minute chart with 1 indicator for 8 hours - crashed.

[marcow]

Do you only run an indicator on the charts, or also a strategy ? What indicator(s) is it ?
Does this problem also occur with other NT-default indicators ?

I'm asking because memory usage of 5GB seems a bit excessive indeed.

As mentioned before I also get these slowdowns of market replay when running strategies which make a lot of trades. What sometimes seems to help is to reset the Replay101 account and the database (warning! you'll lose historical trade data )

Marco

[kaywai]

Hi Marco,

I only ran 1 indicator on 3 charts with similar instruments concurrently. Then I reduced it to 2 charts and then 1 chart. Hate to think what would happen if I ran an automated strategy too!

Well everytime it hangs or slows to a snail's pace (at 500x, from making 3 prints over 10 to 15 minutes of data on a 30 minute bar chart to making 3 prints on every minute of data), I shut NT7 down and look at the resource monitor memory section and wait for NT7 to disappear before running it again.

I'm too fearful to run an automated strategy "live" if I'm going to face such problems. NT support have been pretty silent so far this weekend so I'm not sure what to make of it.

I would rather not mention the indicator I'm using. Certainly the code written for it isn't small in size (about 59k at the moment).

Kay Wai

[marcow]

Well everytime it hangs or slows to a snail's pace (at 500x, from making 3 prints over 10 to 15 minutes of data on a 30 minute bar chart to making 3 prints on every minute of data), I shut NT7 down and look at the resource monitor memory section and wait for NT7 to disappear before running it again.

-with all respect- to me it seems there is something wrong with the indicator you're using. The memory usage you're experiencing is very excessive and not normal.

What happens if you run the same replay sequence with only a default NT indicator, like MACD ?

Does the indicator draw a lot of lines/objects or filled regions in the chart ?
I would suggest to make a copy of the indicator and then strip it from all the draw instructions, and see what happens. In my experience, such excessive memory usage often is graphics related.

Marco

[kaywai]

Hi Marcow,

The indicator does draw quite a few lines and objects. In addition to that calculateonbarclose is false.

Thanks for your advice. Appreciate it. Will see what i can do at my end.

Best regards

Kay Wai

[koganam]

Hi Marcow,

The indicator does draw quite a few lines and objects. In addition to that calculateonbarclose is false.

Thanks for your advice. Appreciate it. Will see what i can do at my end.

Best regards

Kay Wai

Are you properly deleting the many text numbers that you are drawing when a T.Setup fails, and needs to be removed? That indicator already draws a lot of objects: you need to be careful to remove those that should not be there because the rules say that they must be removed.

[kaywai]

Hi Koganam,

Think the key phrase here is "properly deleting". Is the removal of the text numbers from the screen considered 'properly deleting'?

I started running 3 versions of the indicator together, 2 with COBC = true and 1 with COBC = false. Then i ran 2 versions of 1 each. And after that 1 version at the time. Seems the issue could lie somewhere between market replay and COBC = false.

Your advice is much appreciated.

Kay Wai

[NinjaTrader_JC]

Hello Kay Wai,

Could you clarify how you are removing the text from the screen?

Are you using the RemoveDrawObject() method?

Happy to be of further assistance.

[kaywai]

Hi JC,

Yes, the code is using the RemoveDrawObject method.

You could try running the code if you like but I won't be posting it in the forum.

Regards

Kay Wai

[NinjaTrader_JC]

Hello Kay Wai,

You can either post the snippet of how you are drawing (note that the logic should not be needed) and removing the drawing objects, or send an email to support [at] ninjatrader [dot] com with the subject line ATTN: JC. Inside the body of the email please reference this thread and let me know if the scenario that you are testing this in.

[marcow]

Can you guys please report back on this issue if you found the cause of these slowdowns/high memory usage, this might be helpfull for other readers too ?

Marco