charlescara / datastreamdsws2r Goto Github PK

I've asked this question on the Refinitiv forum and understood that timeSeriesRequest calls the make.name function to create column names.

However, I do find this not useful as I do use the original names (eg XMUJ (NAV)) to find out historical time series of a particular NAV of a security. Is this behavior changeable to allow invalid characters (and spaces)?

Thanks.

Hi there

I keep getting the following error when trying to access datastream through R. Is this an issue with the package or my DS credentials?

Thanks
Abigail

In the code below, the NSR instrument returns "NA" while the other two return valid dates. However, the entire request fails due to the "NA" for NSR.

mydsws$snapshotRequest(c("MYI","NEX","NSR"), datatype = 'TIME', requestDate = '0D')
Error in charToDate(x) : 
  character string is not in a standard unambiguous format

I am using DatastreamDSWS2R version 1.7.2 in R version 4.0.2.

Is it not possible to replace the "NA" character string with NA_character_ so that it can be converted to an object of class Date?

Received for the first time this error message. This should be handled automatically by requesting a new token, and continuing to process the request.

AuthenticationFailed 
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:[GUID] Time:2019-11-15T02:41:27.9533371Z Request date header too old: 'Fri, 15 Nov 2019 02:25:42 GMT' Return code: 403

Is there a reason that the maximum number of instruments that can be queried at one time is set to 20 (both in snapshotRequest and timeSeriesRequest)? This is quite restrictive and I typically have to call timeSeriesRequest repeatedly within a loop to get all the data that I need.

I have used this code:

myData <- mydsws$snapshotRequest(instrument = c("ABF","RIO","WPP"), expression = "PCH#(XXXX,3M)", requestDate = "0D")
The returned data does't have instruments.

Instrument X

NA -7.48
NA -10.11
NA 1.88

For certain tickers timeSeriesRequest returns no data, even though when searching for the data on Excel I can see it is available. For example,

raw_output = mydsws$timeSeriesRequest(
      instrument = "USAUD3F",
      datatype = "EB",
      startDate = "1999-12-31",
      endDate = Sys.Date()-1,
      frequency = "D"
) 
length(raw_output)

returns 0. Similarly, for other forward exchange rates no data is returned. With the deprecated package it used to work. Any idea what the issue may be?

Many thanks!

There seems to be a problem extracting data with timeSeriesRequest for certain RICs. For example, consider Naspers (mnemonic = R:NPNJ and RIC = NPNJn.J) on the Johannesburg Stock Exchange (JSE). Requesting the price series using the RIC returns NAs:

mydsws$timeSeriesRequest(instrument = '<NPNJn.J>', datatype = 'P', startDate = '-1Y', endDate = '0D')

But requesting the same data using the Reuters mnemonic does work:

mydsws$timeSeriesRequest(instrument = 'R:NPNJ', datatype = 'P', startDate = '-1Y', endDate = '0D')

I can, however, get the price data by RIC using snapshotRequest:

mydsws$snapshotRequest(instrument = '<NPNJn.J>', datatype = 'P', requestDate = '0D') # works!

I have observed the same behaviour for Naspers' spin-off company Prosus (mnemonic = R:PRXJ and RIC = PRXJn.J).

It might be worth noting that the mnemonics and RICs for stocks on the JSE are typically of the form R:---J and ---J.J, respectively. Naspers and Prosus are exceptions in that the RIC is of the form ---Jn.J (note the little "n"). These are the only two stocks that I have found where timeSeriesRequest returns NAs if you use their RICs, and it so happens that their RICs are unusual.

An empty request which returns the datastream key, which should technically be character. If so, detected empty series or any other kind of foul return would be easier.

Currently the behavior is as follows

xx <- ds$timeSeriesRequest(
  instrument = "SWHWWAR",
  startDate = "-30Y",
  endDate = "-0D",
  frequency = "D"
)

class(xx)
# returns 
[1] "xts" "zoo"

Shouldn't this be changed? Any other suggestions to detect foul returns easier in a list of returns?

For example: <WAFGn.DE>:

JSON request to DSWS server is:

{"DataRequest":{"DataTypes":[{"Properties":{},"Value":["ISIN"]},{"Properties":{},"Value":["MED#(X(VA)~U$, -1Y, 0D)"]}],"Date":{"End":["2019-06-07"],"Frequency":["D"],"Kind":[0],"Start":["2019-06-07"]},"Instrument":{"Properties":{},"Value":["<WAFGN.DE>"]},"Tag":{}},"Properties":{"Properties":{}},"TokenValue":["xxx"]}

when a mnemonic has NA value for a Date field, it caused error. See below example. Separately each mnemonic returns a value; when we mix one mnemonic that returns NA with another one that returns a non-NA value, the function returns but with the Date in Numeric (that is fine we can convert it to Date late). However when we mix the one mnemonic that have NA value with two mnemonics that have non-NA value, the function failed.

staticRequest(dwei,DSCode=c("BDUN%TOTQ", "IDUN%TOT", "MSARGTL")[1],Expression='UPDATE')
Instrument UPDATE
1 BDUN%TOTQ 2020-06-03
staticRequest(dwei,DSCode=c("BDUN%TOTQ", "IDUN%TOT", "MSARGTL")[2],Expression='UPDATE')
Instrument UPDATE
1 IDUN%TOT 2020-04-06
staticRequest(dwei,DSCode=c("BDUN%TOTQ", "IDUN%TOT", "MSARGTL")[3],Expression='UPDATE')
Instrument UPDATE
1 MSARGTL NA
staticRequest(dwei,DSCode=c("BDUN%TOTQ", "IDUN%TOT", "MSARGTL")[2:3],Expression='UPDATE')
Instrument UPDATE
1 IDUN%TOT 18358
2 MSARGTL NA
staticRequest(dwei,DSCode=c("BDUN%TOTQ", "IDUN%TOT", "MSARGTL")[1:3],Expression='UPDATE')
Error in charToDate(x) :
character string is not in a standard unambiguous format

Prepare for release:

• git pull
• Check current CRAN check results
• Polish NEWS
• urlchecker::url_check()
• devtools::check(remote = TRUE, manual = TRUE)
• devtools::check_win_devel()
• rhub::check_for_cran()
• revdepcheck::revdep_check(num_workers = 4)
• Update cran-comments.md
• git push

Submit to CRAN:

• usethis::use_version('patch')
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• git push
• usethis::use_github_release()
• usethis::use_dev_version()
• git push

Error when requesting list of all dead calls:

library(DatastreamDSWS2R)
mydsws <- dsws$new()
static <- mydsws$listRequest(instrument = "LOPTBNPEDC", 
                               datatype = c("BDATE"), 
                               requestDate = "0D")

Gives

Error in charToDate(x) : 
  character string is not in a standard unambiguous format

Prepare for release:

• git pull
• Check current CRAN check results
• Polish NEWS
• urlchecker::url_check()
• devtools::check(remote = TRUE, manual = TRUE)
• devtools::check_win_devel()
• rhub::check_for_cran()
• revdepcheck::revdep_check(num_workers = 4)
• Update cran-comments.md
• git push

Submit to CRAN:

• usethis::use_version('patch')
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• git push
• usethis::use_github_release()
• usethis::use_dev_version()
• git push

Hi there

I am trying to pull through a time series of a number of datatypes for a list and I keep getting the following error.

Command in R: testfords <- mydsws$timeSeriesListRequest(instrument="WSUS1",datatype = c("WC18198","WC01084"),startDate = "-2Y",endDate="0Y",frequency = "Y")

I have been assured by Refinitiv that this should be possible, am I doing something wrong?

Thanks,
Abigail

The "INF" string such as the mnemonic of INFORMA is incorrectly converted to Inf.
Example:

result.df <- mydsws$listRequest(instrument="INF", datatype=c("NAME","MNEM","ISIN","RIC"), requestDate="-0D")

result.df

>  Instrument    NAME MNEM         ISIN   RIC
>1        Inf INFORMA  Inf GB00BMJ6DW54 INF.L

Running a request gave this message. In theory simpleErrors should be caught before checking if it is an http_error.

Response is not able to be parsed: Error message was:
Operation was aborted by an application callback
Warning: Error in UseMethod: no applicable method for 'http_error' applied to an object of class "c('simpleError', 'error', 'condition')"
  78: httr::http_error
  77: .self$.makeRequest
  76: .self$.basicRequestTSChunk

Frm Monika S:

mnem <- 'LASX200I'
idate <- '2019-12-01' #input in YYYY-MM-DD format
mmyy <- paste0(substr(idate,6,7),substr(idate,3,4))
cdata <- mydsws$listRequest(instrument = paste0(mnem,mmyy),
                            datatype = c("NAME","MNEM","P","WTIDX","TIME"),
                            requestDate="2019-12-01")

cdata contains latest prices not prices from Dec 2019.

Hello Charles,

I am still going through the tests and I am stuck on the first test in test-common.R:

 jsonDate <- "/Date(1449218172199)/"
 rDate <- DatastreamDSWS2R:::.convert_JSON_Datetime(jsonDate)
 expect_equal(rDate, as.POSIXlt("2015-12-04 08:36:12 GMT"), tolerance = 0.5)

this test fails on my computer that runs on CET (and not GMT) .
The values on my PC for
rDate is

> rDate
[1] "2015-12-04 09:36:12 CET"

the test result shows:

`rDate` not equal to as.POSIXlt("2015-12-04 08:36:12 GMT").
1/1 mismatches
[1] 2015-12-04 09:36:12 - 2015-12-04 08:36:12 == 1 hours

I have tried to read in the Jsondatetime string story but this not quite uniformly defined.
If I then check the documentation of .convert_JSON_Datetime shows that it delivers a POSIXct (POSIX central time) object. While in the test you are comparing rData (the result) with a POSIXlt (POSIX local time) object.

In the function.convert_JSON_Datetime itself I found the following thing:
structure(as.numeric(1449218172199)/1000, class = c("POSIXct", "POSIXt"))
[1] "2015-12-04 09:36:12 CET"

while at my pc the following command failed
as.POSIXct(1449218172199/1000)
Error in as.POSIXct.numeric(1449218172199/1000) :
'origin' must be supplied

When I then look up the origin in the documentation of ``as.POSIX*`{base}

'''
origin : a date-time object, or something which can be coerced by as.POSIXct(tz = "GMT") to such an object.
'''

I am therefore wondering what the correct behavior of .convert_JSON_Datetime should be:

1. Delivering results in GMT --> then the test should not fail, and the function should be changed
2. Delivering time in local system timezone or in my case CET --> then the test is wrong and should be changed accordingly.

please advise

thanks,

Laurens

When the request should return a vector of NA then the request fails:
xtsData <- mydsws$timeSeriesRequest(instrument = "KO:SGL",
datatype = "AX",
startDate = "2018-05-07",
endDate = "2018-05-09",
frequency = "D")

On the contrary, if a numeric value is included in the result then the request
returns correctly:
xtsData <- mydsws$timeSeriesRequest(instrument = "KO:SGL",
datatype = "AX",
startDate = "2018-05-03",
endDate = "2018-05-07",
frequency = "D")

Hello, I have tried to install the DatastreamDSWS2R package by downloading the zip file first and then calling install("Myfilepath/DatastreamDSWS2R-master"). When I do this, I get the following error (see screenshot). It says there is no package called 'DatastreamDSWS2R'. Could you help me out with this, please? Any help would be appreciated.

PS: I wasn't able to install the package via the devtools function "install_github(CharlesCara/DatastreamDSWS2R)" either.

I was wondering if it would be possible to add the functionality to allow users to pull data for a specified set of dates (not necessarily evenly spaced)... something mid-way between snapshotRequest and timeSeriesRequest. The simplest implementation might be to allow snapshotRequest to accept a vector of dates for its requestDate parameter.

Do you have any plans to put this on CRAN and give this nice package a little more exposure?
If you need help with that... I am in.

Hi Charles!
I am an experienced R user, however with way too limited dev abilities!
I am trying to install DatastreamDSWS2R thru the usual command line and receiving the following error:

install_github('CharlesCara/DatastreamDSWS2R')
Downloading GitHub repo CharlesCara/DatastreamDSWS2R@master
Error in utils::download.file(url, path, method = method, quiet = quiet, :
cannot open URL 'https://api.github.com/repos/CharlesCara/DatastreamDSWS2R/tarball/master'

Could you help me please?!

Thanks!!

From my current process, I would need to go from entering SEDOL into the eikonapir to retrieve the RIC, which will then be used with the package to retrieve the mnemonic. Is there a way to skip the eikonapir step and have the code recognize a SEDOL string and return the appropriate mnemonic?

The reason why I asked about this is because of the fact that there are some securities which does not have corresponding RICs in Eikon & DSWS. For example, ITA.Z is the primary RIC for the SEDOL B14MG44 in Eikon, while if you search the same SEDOL up in DS, the RIC is ITA instead.

Thanks!

I find that my requests (using timeSeriesRequest) sometimes timeout, particular when I am requesting many, long time series. This is frustrating when trying to set up automated pipelines, as sometimes it works and sometimes it doesn't. Is there a way to set the timeout value (longer than 30s)? If not, I would like to request that this please be added as an optional parameter.

Dear Charles,

Thanks a lot for your Github project. I have been using it for a while and I am very satisfied with the performance. As I wanted to check the package I did a cran check with it on rstudio. I had to make a number of textual changes to the documentation, Desription file and datafiles in order to get it running. You can find these here: https://github.com/GreenGrassBlueOcean/DatastreamDSWS2R
For example the tests can not be run because they require authentication. This can be solved at a later stage by maybe adding some data for tests and excluding some other tests from the package.

However when the Cran checks the r code itself it gave the following warning:

> checking R code for possible problems ... NOTE
  .getJSONValue: no visible global function definition for
    '.convert_JSON_Boolean'
  UCTSAppend: no visible global function definition for
    'make.index.unique'
  staticListRequestSet: no visible binding for global variable
    'iExpression'
  staticRequestSet: no visible binding for global variable 'iExpression'
  Undefined global functions or variables:
    .convert_JSON_Boolean iExpression make.index.unique

When I then check the code it looks like these function are not present at all in the package?
Could you please help me with this?

I have used this code:
myData <- mydsws$snapshotRequest(instrument = "STATS", datatype = c("DS.USERSTATS"), requestDate = Sys.Date())

The returned data is:

Instrument User

STATS ZEIK339

I have tested timeSeriesListRequest with LFTSE100.

xtsData <- mydsws$timeSeriesListRequest(instrument = "LFTSE100", datatype = "P", startDate = "-30D", endDate = "-0D", frequency = "D")

It returns:
Error requesting data. HTTP message was: Server error : Internal Server Error : Server error: (500) Internal Server Error
​

Is it possible to download the usage stats as described in page 2 of the PDF linked below inside of R? I have tried the following, but could not get back a valid response:

mydsws$snapshotRequest(instrument = "STATS", datatype = "DS.USERSTATS", requestDate = "0D")

https://developers.refinitiv.com/sites/default/files/DSWS%20for%20Desktop%20-%20User%20stats%20and%20limits_0.pdf

Hi, I have been using your package without problems until 2 weeks ago. I did not change username or password for Datastream, but this morning, suddenly this error appeared while trying to donwload a time series. How could I solve it? did something change in the function in the package? Thanks!

code:

mydsws <- dsws$new()
xtsData <- mydsws$timeSeriesRequest(instrument = "MKS",
datatype = "MV",
startDate = "-30D",
endDate = "-0D",
frequency = "D")

Console:

Error requesting json...
Error: Forbidden

GenericCurlError:
Forbidden

ForbiddenHTTPErrorsimpleErrorerrorcondition
Error in rjson::fromJSON(json_str = myDataResponse) :
STRING_ELT() can only be applied to a 'character vector', not a 'list'

Hi Charles, would it please be possible to add an option to use RIC codes instead of mnemonics? In Excel there is an checkbox for this. Perhaps it's an idea to add an parameter 'RIC = TRUE' to the function? Thanks!

Would it be possible to add an optional parameter to return data (e.g. via a timeSeriesRequest) as a data frame, rather than an xts object? I find that I almost always wrap my requests in as.data.frame, because this format is more useful for subsequent manipulations (e.g. performing joins to other data). The data frame is also a standard format that most R users will be familiar with it, while xts is not as well known.

The issue emerges when initializing dsws object with a know token value/date like this:

mydsws <- dsws$new(
  token = list(
    TokenValue = "...",
    TokenExpiry = as.POSIXct("2020-01-24 13:00:00 CET")
    )
)

Error in is.timeBased(token$TokenExpiry) : 
  could not find function "is.timeBased"

The static request with composite datatypes doesn’t work in R. Datastream has a number of these datatypes which when used display a number of data items. For example QTEALL provides all listing details for a security. For each related quote, the following items are returned: CD01, MN01, MJ01, PQ01, MK01, GL01.

When requested in Datastream Web Service they have following output:

However when I made the same snapshot request in RStudio I got just one item returned.

mydsws <- dsws$new()
data<- mydsws$snapshotRequest(instrument = 'VOD',
                              datatype = "QTEALL",
                              requestDate = "0D")

Hi Charles,

I noticed you make use of QTEALL, a datatype not many know! Since it returns CD01, MN01, MJ01, etc, I was wondering if you can alter the output of this type?

Since I also need the listing DATE of each of these associated stocks

Hope you can help :)

(I am doing research on reverse cross listing for my university)

I am in a corporate environment where users are not supposed to user credentials directly, but only with a token given from local server.

Currently, DatastreamDSWS2R requires credentials eventhough Refinitiv API does not.

A potential workaround may be to give it some fake credentials.

I am willing to make a PR with proper handling of the case, but it may require some refactoring of the code base, I assume.