Announcement

Collapse
No announcement yet.

Partner 728x90

Collapse

Can I access Bid and Ask data that has been imported into the historical database

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Can I access Bid and Ask data that has been imported into the historical database

    I have imported Last, Bid and Ask data into the historical database.

    Click image for larger version  Name:	historicalDatabase.jpg Views:	0 Size:	179.9 KB ID:	1279311

    I can view this data on a chart and I've confirmed the data on the chart matches the data in the historical database.

    Click image for larger version  Name:	chart.jpg Views:	0 Size:	214.7 KB ID:	1279312

    That works as I would expect, but when I run a strategy on the same instrument that is already loaded into a chart, data in the chart is reloaded via the strategy running and then the Bid and Ask data in the chart no longer matches the Bid and Ask data in the historical database as it did before I ran the strategy. Though, unlike the Bid and Ask values that no longer match the historical database data, after running the strategy, the "Last" data values on the chart do still match the historical database data “Last” values. After running a strategy, all the OHLC values for every Bid and Ask bars are one penny apart as can be seen in the Data Box screenshot.

    Click image for larger version  Name:	dataBox.jpg Views:	0 Size:	76.1 KB ID:	1279310

    Is this a bug? I see on this help page:

    https://ninjatrader.com/support/help...ick_replay.htm
    It states:
    Tick Replay ONLY replays the Last market data event, and only stores the best inside bid/ask price at the time of the last trade event. You can think of this as the equivalent of the bid/ask price at the time a trade was reported. As such, historical bid/ask market data events (i..e, bid/ask volume) DO NOT work with Tick Replay. To obtain those values, you need to use a historical bid/ask series separately from Tick Replay through OnBarUpdate()​
    But I've tried both tick replay and a historical bid/ask series as seen in below coding examples which yield the results as described above where the Bid and Ask OHLC values are one penny apart and do not match the Bid and Ask values that are stored in the historical database.

    Note: I'm running the strategy via Playback via the "Historical" radio button option as seen in the attached screenshots.

    Click image for larger version  Name:	playback.jpg Views:	0 Size:	16.5 KB ID:	1279309

    I've tried the following code examples which all yielded data that does not exist in the historical database:

    example 1: with Tick Replay turned off
    Per documentation:

    https://ninjatrader.com/support/help...nt8/getbid.htm
    Code:
    protected override void OnBarUpdate()
    {            
        Print("output    " + Close[0] + " " + Bars.GetBid(CurrentBar) + "    " + Bars.GetAsk(CurrentBar) +
                "     " + Cbi.Connection.PlaybackConnection.Now.ToString("HH :mm:ss.fff") );
    }​​

    example 2: with Tick Replay checked on for the strategy
    Per documentation:
    https://ninjatrader.com/support/help...ick_replay.htm
    &
    https://ninjatrader.com/support/help...aeventargs.htm
    Critical: If used with TickReplay, please keep in mind Tick Replay ONLY replays the Last market data event, and only stores the best inside bid/ask price at the time of the last trade event. You can think of this as the equivalent of the bid/ask price at the time a trade was reported​
    What is this best inside bid/ask price at the time of last trade event? Whatever it is, it doesn't match any data in the historical database for bid/ask values, and this mismatching of data values is not isolated to TickReplay.
    Code:
    protected override void OnMarketData(MarketDataEventArgs marketDataUpdate)
    {
        // Print some data to the Output window
        if (marketDataUpdate.MarketDataType == MarketDataType.Last)
        {
            Print("Last = " + marketDataUpdate.Price + "    " + Cbi.Connection.PlaybackConnection.Now.ToString("HH:mm:ss.fff") );
        }
        else if (marketDataUpdate.MarketDataType == MarketDataType.Ask)
        {
            Print("Ask = " + marketDataUpdate.Price + "    " + Cbi.Connection.PlaybackConnection.Now.ToString("HH:mm:ss.fff") );
        }
        else if (marketDataUpdate.MarketDataType == MarketDataType.Bid)
        {
            Print("Bid = " + marketDataUpdate.Price + "    " + Cbi.Connection.PlaybackConnection.Now.ToString("HH:mm:ss.fff") );
        }
    }​

    example 3: Using Historical Bid/Ask Series with Tick Replay turned off
    Per documentation:
    https://ninjatrader.com/support/help..._ask_serie.htm
    Code:
    else if (State == State.Configure)
    {
        AddDataSeries("ADBE", BarsPeriodType.Minute, 1, MarketDataType.Bid);
        AddDataSeries("ADBE", BarsPeriodType.Minute, 1, MarketDataType.Ask);
    }
    protected override void OnBarUpdate()
    {    
        if(BarsInProgress == 0)
        {
            Print("Last   " + BarsInProgress + "    " + Open[0] + "    " + High[0] + "    " + Low[0] + "    " + Close[0] +
                     "    " +      Cbi.Connection.PlaybackConnection.Now.ToString("HH:mm:ss.fff") );
        }
        else if(BarsInProgress == 1)
        {
            Print("Bid   " + BarsInProgress + "    " + Open[0] + "    " + High[0] + "    " + Low[0] + "    " + Close[0] +
                     "    " + Cbi.Connection.PlaybackConnection.Now.ToString("HH:mm:ss.fff") );
         }
         else
         {
             Print("Ask   " + BarsInProgress + "    " + Open[0] + "    " + High[0] + "    " + Low[0] + "    " + Close[0] +
                      "    " + Cbi.Connection.PlaybackConnection.Now.ToString("HH:mm:ss.fff") );
          }​
    }
    What I'd really like to know is how do I access the Bid/Ask values that are stored in the Historical Database via NinjaScript code as per all the coding examples I've provided I cannot find a way to do so. It seems as if the only time the Bid/Ask values from the historical database are accurately displayed is when plotting a chart without a strategy running, but not for actual strategies. It seems that anytime a strategy is running the Bid/Ask values are generated from the closing price of the "Last" value that is stored in the Historical Database. Is this the case? Is this a bug? Maybe I'm missing something? Is there no way to get the Bid/Ask values that are stored in the Historical Database via Ninjascript code?
    Last edited by Jason Fornkohl; 11-25-2023, 10:53 PM.

    #2
    Hello Jason Fornkohl,

    I would suggest trying to clear the cache and then re test using a chart. You can use the steps below to clear the cache. To access the historical data you would use AddDataSeries from your script to access that data.

    TickReplay is not going to match your Bid/Ask series, TickReplay only provides the bid and ask prices at the time of the last event and not the full granularity of bid/ask data. That test should be excluded as it would produce differences. To use bid/ask data historically you would need to use AddDataSeries which would match the series you add to a chart which also use that same data.

    Another note is that it would not be suggested to use Bars.GetBid and GetAsk if you have historical ask and bid data added. You can use Closes[1][0] and Closes[2][0] to get the current values from the two secondary series.

    To reset the cache:
    1. Exit NinjaTrader
    2. Open the folder: Documents\NinjaTrader 8\db
    3. Delete the cache folder
    4. Restart NinjaTrader and open a chart and load the 3 series.

    JesseNinjaTrader Customer Service

    Comment


      #3
      I've experimented some more, and I have discovered that I can access Bid/Ask values with a Ninjascript strategy when using the Strategy Analyzer, but I cannot access the Bid/Ask values when using Playback Connection with the Historical radio button option checked. I'm hoping to access the Bid/Ask data per the Playback Connection with the radio button Historical option checked.

      This is what I have discovered:

      I have no Connections open.
      With no Connections open, I add a 1 tick Bid chart.

      Click image for larger version

Name:	bidChartWithNoPlayBack.jpg
Views:	210
Size:	163.7 KB
ID:	1279566

      The chart displays the correct Bid prices that are in the Historical Database.

      Click image for larger version

Name:	bid.jpg
Views:	196
Size:	135.7 KB
ID:	1279567

      I now open a Playback Connection and check the Historical radio button option.

      Click image for larger version

Name:	playBackConnection.jpg
Views:	192
Size:	18.1 KB
ID:	1279568

      I click the play button and the Bid chart populates with data. (note: I am not using any strategies here. I'm simply showing that by using the Playback option with the Historical radio button checked that the chart does not display the Bid data that is in the Historical database, but it actually displays the wrong data. It displays the "Last" data that is in the Historical database.

      Click image for larger version

Name:	bidChartWithPlayBack.jpg
Views:	195
Size:	162.5 KB
ID:	1279569

      The chart data does not match the Historical Bid data, but rather it matches the Historical "Last" data in the Historical Database.

      Click image for larger version

Name:	last.jpg
Views:	192
Size:	129.7 KB
ID:	1279570

      This incorrect data reporting is the same thing that is happening when I try to access Bid/Ask values via Ninjascript code when running strategies per the Playback Connection with the Historical radio button checked. Is there any known way to access the Bid/Ask values per Ninjascript code from a strategy while using the Playback Connection with the Historical radio button checked?

      Comment


        #4
        Another follow up:

        It appears to me that the Playback connection loops through the BarsInProgress in a manner similar to having a live data feed which is different than how the Strategy Analyzer loops through the BarsInProgess. The Strategy Analyzer loops through the BarsInProgress sequentially per the data in the Historical Database. The Playback connection loops through BarsInProgress according to the time in which they are received. If two bars have the same timestamp, then I think the Playback connection randomly chooses one, so it is not always sequential like the Strategy Analyzer is. I think this prevents accessing Bid/Ask data in the way it is possible to do so with the Strategy Analyzer. Based on my findings, at this point, I'm assuming it is not possible to access Bid/Ask data from the Historical Database when using the Playback Connection with the Historical radio button checked. Can anyone confirm this is correct information?

        See comments in the code below

        Code:
                    else if (State == State.Configure)
                    {
                        AddDataSeries("ADBE", BarsPeriodType.Tick, 1, MarketDataType.Bid);
                        AddDataSeries("ADBE", BarsPeriodType.Tick, 1, MarketDataType.Ask);
                    }
                }
        
                protected override void OnBarUpdate()
                {                
                    //While testing with the Strategy Analyzer this code is working perfectly, all data matches data in the Historical Database and all BarsInProgess are always sequentially 0,1,2 everytime without skipping over any bars while looping through OnBarUpdate()
                    //note: this code only gives correct data with the Strategy Analyzer.  It does not work with Playback Historical data.
                    //With Playback Historical data the BarsInProgress do not run sequentially either which is a hint as to why this code does not match the data in the Historical Database
        
                    if(BarsInProgress == 0)
                    {
                        Print("BarsInProgress    " + BarsInProgress + "    " + Closes[0][0] + "    " + Closes[1][0] + "    " + Closes[2][0] + "    " + Time[0].ToString("yyyy-MM-dd HH:mm:ss.fff"));
                    }
                    else
                    {
                        //this provides a visual print out to confirm bars are printing out sequentially
                        Print("XXX    " + BarsInProgress);
                    }
                }
        Last edited by Jason Fornkohl; 11-28-2023, 11:29 AM.

        Comment


          #5
          Hello Jason Fornkohl,

          Regarding the bid price being the last price, I would need to look into that. Are you currently using 8.1.2.0 version NinjaTrader? If not please update and retest the situation before we continue.

          Regarding the events in playback, because you are playing back the data it would not have a specific sequence like a backtest. The BarsInProgress is determined by the series calling OnbarUpdate. In your use case you will have 0 1 and 2 for the indexes that call OnBarUpdate in no specific order. Playback plays the data in the same order it was received.

          If you are coding your strategy based on the strategy analyzer and how it processes you should avoid that and instead code it for realtime based on how events will actually be observed. In most cases you would program a strategy to work in realtime first and then if you want to backtest it you would add additional logic to accommodate processing in backtesting determined by your logic and use case.



          JesseNinjaTrader Customer Service

          Comment


            #6
            I've updated my Ninjatrader to the current version 8.2.2.1, and I'm still getting the same results as described above, but I noticed this when clicking on the Historical Playback option this time:

            Click image for larger version  Name:	noLevel2.jpg Views:	0 Size:	14.5 KB ID:	1279728

            I quess this makes sense that Bid/Ask data is not working correctly in this mode as that is Level II data.

            "If you are coding your strategy based on the strategy analyzer and how it processes you should avoid that and instead code it for realtime based on how events will actually be observed. In most cases you would program a strategy to work in realtime first and then if you want to backtest it you would add additional logic to accommodate processing in backtesting determined by your logic and use case."

            I've learned that the hard way, and it is the process that I'm doing now and it is what has brought me to my current questions asked here. I originally coded for live testing by using Playback of Market Replay data. I was going to backtest further with historical data with the Playback Historical data option which led me into the current issues of getting the Bid/Ask data. I believe I should be able to add the additional logic to accommodate for processing in a backtest on historical data by using the Strategy Analyzer instead of attempting to do so per the Playback Historical data option in order to get the Bid/Ask data I'm seeking.

            Thanks for your assistance Jesse.
            Last edited by Jason Fornkohl; 11-28-2023, 10:27 PM.

            Comment

            Latest Posts

            Collapse

            Topics Statistics Last Post
            Started by Nightmaregpu, Yesterday, 10:55 AM
            5 responses
            29 views
            0 likes
            Last Post Nightmaregpu  
            Started by stevepmay, Today, 09:00 AM
            0 responses
            2 views
            0 likes
            Last Post stevepmay  
            Started by TraderTimmy71, Today, 07:05 AM
            2 responses
            12 views
            0 likes
            Last Post TraderTimmy71  
            Started by ntbone, Today, 01:50 AM
            4 responses
            19 views
            0 likes
            Last Post bltdavid  
            Started by michelz, Today, 08:30 AM
            0 responses
            5 views
            0 likes
            Last Post michelz
            by michelz
             
            Working...
            X