mathpaquette / iqfeed.csharpapiclient Goto Github PK

Python is very popular these days and having the examples translated in would be useful for many.

The task is to translate these examples:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/tree/master/src/IQFeed.CSharpApiClient.Examples

We can merge them one by one so if you are welling to help, pick one and raise a PR.
Thanks

I'm not too fond of putting my IQfeed login and password in code like appconfig nor do I want to add it to the environment. It seems easier to use what IQFeed already stores in the Registry. Also the Product ID can be stored there. The following replacement for IQFeedLauncher.cs

using System;
using System.Configuration;
using System.Diagnostics;
using System.Threading;
using IQFeed.CSharpApiClient.Extensions;
using IQFeed.CSharpApiClient.Socket;
using IQFeed.CSharpApiClient.Streaming.Admin;
using IQFeed.CSharpApiClient.Streaming.Admin.Messages;
using Microsoft.Win32;

namespace IQFeed.CSharpApiClient
{
// ReSharper disable once InconsistentNaming
public static class IQFeedLauncher
{
public static void Start(string login = null, string password = null, string productId = null, string productVersion = null, int connectionTimeoutMs = 100, int retry = 50)
{
var appSettings = ConfigurationManager.AppSettings;

        // Add <PackageReference Include="Microsoft.Win32.Registry" Version="4.4.0" /> to csproj
        RegistryKey key = Registry.CurrentUser.OpenSubKey("Software\\DTN\\IQFeed\\Startup");

        login = login ??
                key.GetValue("LOGIN").ToString() ??

                Environment.GetEnvironmentVariable("IQCONNECT_LOGIN") ??
                appSettings["IQConnect:login"].NullIfEmpty() ??
                throw new Exception("Unable to find IQConnect login from environment variable or app.config");

        password = password ??
                    key.GetValue("PASSWORD").ToString() ??
                   Environment.GetEnvironmentVariable("IQCONNECT_PASSWORD") ??
                   appSettings["IQConnect:password"].NullIfEmpty() ??
                   throw new Exception("Unable to find IQConnect password from environment variable or app.config");

        productId = productId ??
                    key.GetValue("PRODUCT").ToString() ??
                    Environment.GetEnvironmentVariable("IQCONNECT_PRODUCT_ID") ??
                    appSettings["IQConnect:product_id"].NullIfEmpty() ??
                    throw new Exception("Unable to find IQConnect product ID from environment variable or app.config");

        productVersion = productVersion ??
                         Environment.GetEnvironmentVariable("IQCONNECT_PRODUCT_VERSION") ??
                         appSettings["IQConnect:product_version"].NullIfEmpty() ??
                         "1.0.0.0";

        var iqConnectParameters = $"-product {productId} -version {productVersion} -login {login} -password {password} -autoconnect";
        Process.Start("IQConnect.exe", iqConnectParameters);

        WaitForAdminPortReady(connectionTimeoutMs, retry);
        WaitForServerConnectedStatus(IQFeedDefault.Hostname, IQFeedDefault.AdminPort);
    }

    public static void Terminate()
    {
        foreach (var process in Process.GetProcessesByName("IQConnect"))
        {
            process.Kill();
        }
    }

    private static void WaitForAdminPortReady(int connectionTimeoutMs, int retry)
    {
        var adminPortReady = SocketDiagnostic.IsPortOpen(IQFeedDefault.Hostname, IQFeedDefault.AdminPort, connectionTimeoutMs, retry);
        if (!adminPortReady)
            throw new Exception($"Can't establish TCP connection with host: {IQFeedDefault.Hostname}:{IQFeedDefault.AdminPort}");
    }

    private static void WaitForServerConnectedStatus(string host, int port, int timeoutMs = 10000)
    {
        var manualResetEvent = new ManualResetEvent(false);
        var adminClient = AdminClientFactory.CreateNew(host, port);

        adminClient.Stats += AdminClientOnStats;
        adminClient.Connect();

        var connected = manualResetEvent.WaitOne(timeoutMs);
        if (!connected)
            throw new Exception($"Haven't received connected status with host: {host}:{port}");

        adminClient.Stats -= AdminClientOnStats;
        adminClient.Disconnect();

        void AdminClientOnStats(StatsMessage message)
        {
            if (message.Status == StatsStatusType.Connected)
                manualResetEvent.Set();
        }
    }
}

}

Hi

Firstly thank you for your excellent library, it's been a godsend!

I'm currently using the library from python to download historical data, using pythonnet as per your examples. I've managed to achieve a certain amount of parallelism using multiple connections (up to 6 seems to work) and a multiprocessing pool. However, I've not found a way to use the async variants (GetHistory*Async) on the HistoricalFacade as I believe this would require some interop between python asyncio and C# tasks which I don't think is supported according to this .

Are you aware of any solutions to this? I would really like to be able to get 50 requests/sec inflight instead of being limited to 6. Downloading options data takes forever... ;-)

Thanks
Andy

System.Net.WebException : The request was aborted: Could not create SSL/TLS secure channel.
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
at IQFeed.CSharpApiClient.Lookup.Symbol.MarketSymbols.MarketSymbolDownloader.GetMarketSymbolsFile(String downloadPath, String marketSymbolsUrl)
at IQFeed.CSharpApiClient.Lookup.Symbol.SymbolFacade.GetAllMarketSymbols(String downloadPath, String marketSymbolsArchiveUrl)
at IQFeed.CSharpApiClient.Tests.Integration.Lookup.Symbol.SymbolFacadeTests.Should_Return_MarketSymbols_From_Sample_Archive_File()

Would be nice to have a built in support for socket disconnected event for streaming clients so that the user will be notified when the underlying socket is closed and the updates will stop. Now until you make another request you won't know about the problem.

Hi!

I am using this code below but it takes forever to get the data. Its taking line by line, and if I take a request of 1 million likes it should take several hours. Is there any work aroud this?

def create1min():
    lookupClient.Connect()
    timeframe_1min = lookupClient.Historical.GetHistoryIntervalDays("AAPL", 60, 2000000, 10000000)
    for timeframe_1mi in timeframe_1min:
        timeframe = str(timeframe_1min.get_Timestamp())
        high = timeframe_1min.get_High()
        low = timeframe_1min.get_Low()
        open = timeframe_1min.get_Open()
        close = timeframe_1min.get_Close()

    empresas = Empresas1min(timeframe, high, low, open, close)
    db.session.add(empresas)
    db.session.commit()

Thank you!

Hello, yesterday it seems IqFeed sent wrong data which caused the following unhandled exception to be generated:

Application: IqFeedManager.exe
CoreCLR Version: 5.0.20.51904
.NET Version: 5.0.0
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OverflowException: Value was either too large or too small for an Int32.
   at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
   at System.Number.ParseInt32(ReadOnlySpan`1 value, NumberStyles styles, NumberFormatInfo info)
   at IQFeed.CSharpApiClient.Lookup.Historical.Messages.TickMessage.Parse(String message)
   at IQFeed.CSharpApiClient.Lookup.Common.BaseLookupMessageHandler.ProcessMessages[T](Func`2 parserFunc, Func`2 errorParserFunc, Byte[] message, Int32 count)
   at IQFeed.CSharpApiClient.Lookup.Historical.HistoricalMessageHandler.GetTickMessages(Byte[] message, Int32 count)
   at IQFeed.CSharpApiClient.Lookup.Common.BaseLookupFacade.<>c__DisplayClass4_0`1.<GetMessagesAsync>g__SocketClientOnMessageReceived|1(Object sender, SocketMessageEventArgs args)
   at IQFeed.CSharpApiClient.Socket.SocketClient.ProcessReceive(SocketAsyncEventArgs e)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pNativeOverlapped)

The exception has been generated in the _IOCompletionCallback thread and wasn't handled in that thread, so it became unhandled exception and crashed the service.

Wrapping the API call which generated the IQFeed call above doesn't catch this exception:

try {
 ... await lookupClient.Historical.GetHistoryTickDatapointsAsync(...
} catch (Exception ex) {
...
 // wasn't called
}

obviously because there is no any code which marshals this exception from the _IOCompletionCallback thread to the context of calling function.

There is no way to know if a client is connected, disconnected or suddenly lost connection.
There is a heavy need for:

• Event for connection state change.
• Property for current connection state.

Current workaround for knowing if client is connected and is working.

a.1. accessing private field _socketClient of type SocketClient and then _clientSocket of type Socket via reflection to send a poll request.

b.1. Disabling timestamps feed (enabled by default) Level1Client.ReqTimestamps(false);
b.2. Sending a timestamp request with 200ms timeout via Level1Client.ReqTimestamp();
b.3. If no response came in or exception was thrown - not connected.

When method 'a' and 'b' are true, socket is connected.

Looking at Level1MessageHandler, every message that comes in is first parsed and only then it's related event is checked if it there is any delegates registered to it.
Skipping the parsing will actually improve performance on heavy applications.
note: this might be relevant to all message handlers.

Example for a fix:

    private void ProcessUpdateMessage(string msg)
    {
      Action<UpdateSummaryMessage> update = this.Update; //copy locally for multithreading purposes.
      if (update == null)
        return;
      update(UpdateSummaryMessage.Parse(msg));
    }

@mathpaquette, Let me know if you want a PR.

Using .net Core 3.1 the library hangs during IQFeedLauncher.Start()

It looks like there's an issue with using dynamic fieldsets. Sending the following requests works
level1Client.SelectUpdateFieldName(DynamicFieldset.MostRecentTrade,
DynamicFieldset.MostRecentTradeSize,
DynamicFieldset.MostRecentTradeTime);
level1Client.ReqWatch("@ESU18");
but an exception is thrown trying to parse the update message because it assumes all current update fieldnames are present. After the message split there's then an array bounds exception.
See IQFeed.CSharpApiClient.Streaming.Level1.Messages.UpdateSummaryMessage.Parse

During publishing my app in a virtual server that is located in France, it failed during parsing TickMessage at the following commented line/s:

public static TickMessage CreateTickMessage(string[] values) {
    return new TickMessage(
        DateTime.Parse(values[0]),
        float.Parse(values[1]),
        int.Parse(values[2]),
        int.Parse(values[3]),
        float.Parse(values[4]), //this line threw FormatException: Input string was not in a correct format.
        float.Parse(values[5]),
        long.Parse(values[6]),
        char.Parse(values[7]),
        int.Parse(values[8]),
        values[9]);
}

It happened because the machine's system language is french therefore the culture in the code is fr-FR
and some European countries write fraction number with comma instead of period so that line tries to parse "14,5" where in fact the value is "14.5".

Fixing it possible by specifying invariant culture like this in any line that uses Parse.

float.Parse(values[4], CultureInfo.InvariantCulture)

Edit:
The exception is thrown in any kind of parse, even in DateTime .

hi, I am trying to use this library as a way to get streaming L2 data from iqfeed via python as a dll. When I try to request watch on a Level2Client I get the error below. It would be good to escalate in the exception message the actual unhandled message as a string.
Many thanks.

Unhandled Exception: System.Exception: Unknown type of level 2 message received.
at IQFeed.CSharpApiClient.Streaming.Level2.Level2MessageHandler.ProcessMessages(Byte[] messageBytes, Int32 count)
at IQFeed.CSharpApiClient.Extensions.EventExtensions.RaiseEvent[T](EventHandler`1 event, Object sender, T e)
at IQFeed.CSharpApiClient.Socket.SocketClient.ProcessReceive(SocketAsyncEventArgs e)
at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs e)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)
at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

Today, I received "E,Too many simultaneous history requests" in response to an HTD request.
I called support and they said it is a recent change, and they will be posting it on the developer forum soon.
The change was explained to me as:
Basically each user has a limit of 30 history requests per second, and every 1/30 of a second an existing request "drops off" and it decreases by one. As long you stay below an average of 30 request/second (and there is a little padding) you'll be fine.

Just wanted to pass it along.

HIT request for 1/1/2007 to current throws System.OverflowException: 'Value was either too large or too small for an Int32.'

Anyhow, when i look up the Options Chain for AMZN on 20180731, this is the error i get (Just wanted to share)

lookupClient.Historical.ReqHistoryIntervalTimeframeAsync(
"NIK.X",
60,
startDateTime,
endDateTime)

startDateTime is new Datetime(2007, 1, 1)
endDateTime is DateTime.Now

Going to investigate this further on my own...not sure what is going on here. Requests are identical to requests that are returning correct interval data, however symbols containing ".X" are returning daily data at around the original request time.

Hi & thanks for all your hard work!

I'm using VS 2017 and, on first compile I get the one and only message:

Severity Code Description Project File Line Suppression State
Warning NU5125 The 'licenseUrl' element will be deprecated. Consider using the 'license' element instead. IQFeed.CSharpApiClient C:\Program Files\dotnet\sdk\2.1.507\Sdks\NuGet.Build.Tasks.Pack\buildCrossTargeting\NuGet.Build.Tasks.Pack.targets 202

Only a warning. Obviously not a huge deal, but thought I'd mention it

Best regards,
Merlin
[email protected]

Summary

The value type for variables that hold a balance/portfolio or price/bid/ask should be decimal.

Pros

• High accuracy when handling calculations with money (this is decimal's purpose and what it was created for).
• decimal is the standard value type for handling money in all libraries I've worked with by now. (QuantConnect/Lean, sonvister/Binance, sefbkn/gdax.netcore)

Cons

• Breaking Change that will require move to version 2.0.
• There is a performance impact when using decimal, see reference.
• A Decimal takes 16 bytes vs Double that takes 8 bytes. essentially doubling the memory usage.

My Opinion
The performance impact isn't small but tolerable.
The accuracy is indeed required.
The changes are breaking, therefore it will have to be released in a new major version release.
Unless we plan to break the market's standard by being a special snowflake, I do not see a reason.

Would love to hear your opinions.

There should be a resharper solution/projects configuration for the refactoring rules based on the current styling to make contribution easier.
This is essential if there will be contribution guidelines in the future.

Are there plans to support the following recent DTN change?

Last weekend, IQFeed released a server-side update to our historical data servers which changed the method that we use to rate limit requests. This change was necessary to protect
our servers from rogue software and to help standardize and improve the performance of our history servers for all customers. This change ONLY applies to our history servers.

Previously, customers were allowed to have 15 simultaneous requests active at any given time. While this was a good rate limiting method when it was originally developed,
advances in machine hardware and reduced latency to our servers over the internet had effectively removed this limit entirely. Customers were able to make nearly unlimited
requests and that was never the intent for our service. This method of request rate limiting was confusing for new customers. It was dependent upon latency to our servers
so each customer had a different limit that they could achieve. This was also never intended, and led to mixed results and customer experiences. For years we were able to
operate this way using an informal "honor system", by closely monitoring our server performance and reaching out to customers to adjust their software as needed. However,
this approach is no longer feasible.

The new method is a straightforward "requests per second" rate limiting method. It is a bucket limit system similar to the Linux kernel's iptables hashlimit. As a result, you
start with a full bucket of request credits. Each request consumes 1 credit and credits are re-allocated every few milliseconds unless you are already at max. If you send a request
when you are out of requests, you will get an error stating too many requests (the error message is the same as before). This new method allows all customers to have the same
limit. Also, customers can monitor the limit themselves, and know their real limit without trial and error

With all that said, the new limit for requests is 50/second (with one new credit added each 20ms). For the vast majority of customers, this change will not affect their
IQFeed performance at all. Unfortunately, for a few of you who are downloading a large symbol list as fast as possible, you will notice this change (if you haven't already).
Almost all of you who fall into this category can still get the data you need, but it will take you a bit more time to make the requests. If you fall into this category,
we urge you to reach out to our dev support team with a description of the data you need and the requests you are currently trying to use to download it. Our support
staff will help you figure out the best way to get this data from the feed.

Could the Raw.RequestHistory, which seems to be intended to create files have an option to include a header?

In the past, I've done it like this:
sRequest = String.Format("HDX,{0},{1},{2},{3},{4}\r\n", txtSymbol.Text, txtStrike.Text, txtDirection.Text, txtRequestID.Text, txtDatapointsPerSend.Text);
header = "Time Stamp,Last,High,Low,Open,Close,Volume,Open Interest";
for each type of request.

There should be SerializableAttribute to every POCO that represents data received from IQFeed (UpdateSummaryMessage, FundamentalMessage and so on).
Adding this attribute does not create any overhead whatsoever.

Motivation

Allow (de)serialization from various external libraries like netserializer that require just this attribute to (de)serialize a class.

Looks like it is using lookup client instead of level1 client.
FROM CLIENT LookupRequest 15136 1 2020-06-05 12:42:05 HTT,NEPT,20200401 040000,20200403 200000,,,,0,,

TO CLIENT LookupError 15136 1 2020-06-05 12:42:05 E,!NO_DATA!,,

Per IQ Feed:

Response is HLOC and both message types are parsing as HLCO. Will be creating PR for fix shortly.

System.FormatException: 'String was not recognized as a valid DateTime.' is thrown when requestId is passed to ReqHistoryIntervalTimeframeAsync.

Since we targeting both frameworks, appveyor wont use appveyor.yml configuration for .net core
example:

test:
assemblies:
- IQFeed.CSharpApiClient.Tests.dll

even with this, on the .net core build, integration tests will be run.

expected:
be able to run only unit tests for all targets.

The IQFeed launcher will use the PW and user id from the registry. That seems like a more secure method than including it in app.config where one has to remember to remove it when the code is shared.

Could that be done in the next version?

You can request say 1 minute bars that are updated every 5 seconds. Currently it just ignores the update interval of 5 seconds.

The request is incorrectly formatted in DerivativeRequestFormatter.ReqBarWatch. The developer documentation says the format should be:

BW,[Symbol],[Interval],[BeginDate BeginTime],[MaxDaysOfDatapoints],[MaxDatapoints],[BeginFilterTime],[EndFilterTime],[RequestID],[Interval Type],[Reserved],[UpdateInterval]

The reserved field is not taken into account, so it should be:

$"BW,{symbol.ToUpper()},{interval},{beginDate?.ToString(DerivativeDatetimeFormat)},{maxDaysOfDatapoints},{maxDatapoints},{beginFilterTime?.ToString(DerivativeTimeFormat)},{endFilterTime?.ToString(DerivativeTimeFormat)},{requestId},{intervalType?.ToString().ToLower()},,{updateInterval}{IQFeedDefault.ProtocolTerminatingCharacters}";

Also there shouldn't be a comma before "{IQFeedDefault.ProtocolTerminatingCharacters}" for all the request types in DerivativeRequestFormatter.

Hi @mathpaquette,

Are you interested in a PR to support 6.1?

I've got a completely working version, although because MarketSummary is in beta, and subject to change, it's handled semi-dynamically, which obviously is a little slower, but no reflection. Otherwise it fits pretty well with everything that you've done previously, architecturally speaking.

I've also written some, but obviously not enough, tests to go along with it.

I'm sure there are some improvements to be made, and I'll probably get to them in a bit, but I had a hard need to get 6.1 functional for the project I'm doing.

Nich

Calling ReqWatchList produces exceptions:

Unhandled exception. [System.ArgumentException: Offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of the source collection.
   at System.IO.MemoryStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at IQFeed.CSharpApiClient.Socket.SocketMessageHandler.TryRead(Byte[]& output) 

It happens when there are a lot of symbols. I tried 1300 options symbols.
After some debugging, I figured out that the size of the buffer is not enough.

Increasing _readBytes solves the issue, quick fix:

_readBytes = new byte[bufferSize * 3];

When starting up, the library can try to bind to the admin port of the client using IPv6 depending on how your network/computer is set up. The library should be changed to filter out any IPv6 addresses.

Not sure if that is the expected behavior, but if you call any of these snapshot methods:

• GetFundamentalSnapshotAsync
• GetUpdateSummarySnapshotAsync

on a client while there was an ongoing ReqWatch on the same symbol, the updates will stop since both of them are calling ReqUnwatch on completion.

From the user perspective I didn't expect that. I though I'd report it here, FYI.

As a library, .net-framework version should be supported at lowest version possible.
Therefore .net-framework version should be downgraded or even better also support .net45,
Below .net45 there will be needed heavier changes related to TPL and Tasks.

In .net45 there is no System.Configuration.ConfigurationManager,
Use mscorlib's assembly System.Configuration instead.

Historical Options do not seem to include volume and open interest. Can they be made available?

Hey,
During accessing history data of msft-nasdaq, in parsing of TickMessage inside the constructor (see code below),
the following line was received:
"2018-04-17 17:51:22.000000,96.0700,1061909,0,0.0000,0.0000,4145784264,O,19,143A"

The parsing code inside constructor:

var values = msg.Split(',');
return new TickMessage(
    DateTime.Parse(values[0]),
    float.Parse(values[1]),
    int.Parse(values[2]),
    int.Parse(values[3]),
    float.Parse(values[4]),
    float.Parse(values[5]),
    int.Parse(values[6]), //this line is tickId that attempts to parse 4145784264.
    char.Parse(values[7]),
    int.Parse(values[8]),
    values[9]);

Fix:

Change TickId to ulong (or long if there is a possibility to receive -1).

What is the product id parameter and where to get it from ?

HistoricalRequestFormatter HistoricalDataTimeFormat is currently hhmmss, but the IQ Feed api documentation states the format should be HHmmSS. This is causing any filter date past 12p to be invalid.

Currently TickMessage.TotalVolume is of type int.
I've received the following tick:

Might I suggest upgrading LastSize to 'long' as-well.

Dynamic Level1 Streaming, trying use the field "Type" to identify trades vs updates. When I add "Type" to my list of fields it causes all the other data (Bid,Ask,Last etc) to come back weird (1s, and 2s,).

Pic1: Type commented out -- everything works just fine

Pic 2: Type included -- everything comes back weird