Sim101 and Market Replay

[Vasia.Pupkin]

I have tested my strategy on with ES 09-15 using Sim101 account and after that I downloaded market replay data using NinjaTrader (with L1 & L2 data) and tested the same strategy with Market Replay.

Sim account is with NinjaTraderBrokerage (from https://www.ninjatraderbrokerage.com...s_trading_demo ). All the delays ("Delay comm." & "Delay exchange") were set to 0 for Sim101 account.

When I tested with "enforce immediate fills" checked, then the results were about the same with both Replay and Sim.

But when I tested with "enforce immediate fills" un-checked, then the results were different. It seems that limit orders are filled slower with Sim account than with Replay account.

I believed that Market replay works in the same way as Sim account, but with additional delays. And because I switched off all the delays then I would expect similar results from Market replay and Sim. Could you explan why there is a difference?

[NinjaTrader_ZacharyG]

Hello Vasia.Pupkin,

Thank you for your inquiry.

Please note that it is expected that you may get different fill results between the Sim account and on Market Replay due to how the simulation engine works.

Please note that the Delay comm. and Delay exchange settings are not used when connected to Market Replay, as stated in the help guide: http://ninjatrader.com/support/helpG...ulator_tab.htm

[Vasia.Pupkin]

Yes, I understand all these ( that delays are not used in Replay & that there are different results) - but it is not an answer to my question

And I also read the answers at this forum when your co-workers said that they cannot tell implementation details of the simulation engine works.

Could you please answer these two questions:

1. Is there a difference between Replay and Sim101 that happens because Replay uses localy recorded data that Ninjatrader can use without a delay, and Sim101 uses real time data from CQG and there is some lag with this data.

(Please note: I do not ask how it is implemented. I just ask if it is affected with the network latency between CQG and my local Ninjatrader)

2. If 'yes' then can it help to impove the results of the simulation, if I run Ninjatrader for example here: http://www.speedytradingservers.com/ ?

[NinjaTrader_ZacharyG]

Hello Vasia.Pupkin,

Just to clarify the points on your first post:

How are you comparing the results of the Replay and the live trades on the Sim101 account? Can you provide an image of what you are seeing so I may look into this further?

To send a screenshot with Windows 7 or newer I would recommend using Window's Snipping Tool.

Click here for instructions

Alternatively to send a screenshot press Alt + PRINT SCREEN to take a screen shot of the selected window. Then go to Start--> Accessories--> Paint, and press CTRL + V to paste the image. Lastly, save as a jpeg file and send the file as an attachment.

Click here for detailed instruction

To answer your questions:

1. There may be slight delays because the information on the live feed must be sent from CQG's servers to your computer, whereas the Market Replay data is already on your computer
   
2. Can you please clarify what you mean by "improve the results of the simulation"?

[Vasia.Pupkin]

>How are you comparing the results of the Replay and the live trades on the Sim101 account?


I just compared the profits. When I set 0 to "Delay comm. (msec)" and "Delay exchange" and run my strategy in Sim 101 account. The profit was less than $0. Later, I downloaded replay data, and run my strategy at the same day and time for the same amount of time, but using Market Replay account. The profit was greater than zero.

And for me it looks like the difference between the Sim & M.replay results happens because orders are filled slower in Sim 101 than in Market Replay.

In order to show what I mean I did the following test:

I sent a short limit order for 1 contract ES 09-15 with SubmitOrder function and saved time, OrderState & last price (Close[0]), later when the order changed its state I also saved time, state & price and so on.

Note:
I saved everything to arrays and saved the arrays to file only after the test.
I repeated this test firstly for Sim 101 and later for market replay - in the same time as it was done in Sim 101
I used timeGetTime function in order to get time. It retrieves the system time, in milliseconds. The system time is the time elapsed since Windows was started.
I also substracted the time of SubmitOrder from all other times (so "Time (ms)" column shows time since order was submitted)

The results:

Sim101

N       Time(ms)         State   Price    Comment
1              0   SubmitOrder 2072.25    <- SubmitOrder call
3            203 PendingSubmit 2072.25    <- Order state is "PendingSubmit" in OnOrderUpdate
4            203      Accepted 2072.25
5            203       Working 2072.25 
17         17906   ChangeOrder 2072.00    <- I called ChangeOrder in OnBarupdate (~18 seconds since order was submitted)
18         17906 PendingChange 2072.00    <- Order state is "PendingChange" in OnOrderUpdate
19         17906      Accepted 2072.00
20         17906       Working 2072.00
21         18469        Filled 2072.00    <- Order state is "Filled" in OnExecution 

Market Replay

N      Time(ms)         State   Price
1             0   SubmitOrder 2072.25
3             6 PendingSubmit 2072.25
4             7      Accepted 2072.25
5             7       Working 2072.25
9            97        Filled 2072.25 

So with the same data, at the same time, a short limit order was filled in 97 milliseconds at Market Replay and in almost 18.47 seconds at Sim 101.

What I try to understand is: why does this difference happen? Is it possible to fix it? (make Sim 101 work faster) if 'yes' then how? How live account will behave?

>There may be slight delays because the information on the live feed must be sent from CQG's servers to your computer, whereas the Market Replay data is already on your computer

Yes, I mean this. Thanks. But it is just 10 ms (please check below if I measured it correctly), and I am not sure if this delay is why the orders behaves differently.


I checked the delay in this way:

When my Ninjatrader is connected to live demo, then this command

C:\>netstat -n

prints this:

Proto  Local Address          Foreign Address        State
....
  TCP    ...                    208.48.16.202:443      ESTABLISHED
  TCP    ...                    64.208.51.202:2823     ESTABLISHED
  TCP    ...                    64.208.51.210:2823     ESTABLISHED
  TCP    ...                    208.48.16.202:2823     ESTABLISHED
.... 

And these addresses are in the list "Demo and Simulator Environment IP Addresses" (see page 5 http://www.cqg.com/Docs/CQGTraderTechSpec.pdf)

And this command:

C:\>ping 64.208.51.210

Prints this:

Pinging 64.208.51.210 with 32 bytes of data:
Reply from 64.208.51.210: bytes=32 time=10ms TTL=118
Reply from 64.208.51.210: bytes=32 time=10ms TTL=118
Reply from 64.208.51.210: bytes=32 time=10ms TTL=118
Reply from 64.208.51.210: bytes=32 time=10ms TTL=118

Ping statistics for 64.208.51.210:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 10ms, Maximum = 10ms, Average = 10ms 

Ping of other addresses is similar ~10-11 ms)

[NinjaTrader_ZacharyG]

Hello Vasia.Pupkin,

For further testing, I would suggest using the built-in functions of NinjaScript to see when your orders are being executed and compare between Market Replay and Sim in this way.

Example:

protected override void OnExecution(IExecution execution)
{
    if (entryOrder != null && entryOrder == execution.Order)
         Print(execution.ToString());
}

protected override void OnOrderUpdate(IOrder order)
{
    if (entryOrder != null && entryOrder == order)
         Print(order.Time + " " + order.ToString());
}

Ensure that the Ouput Window is open before running your strategy. To do this, you will click on Tools -> Output Window in the NinjaTrader Control Center.

Please provide the results you receive once you have done this.