It looks like #80 (4.3.4) introduced a problem where TCP connections are not being disposed, or at least are still in memory. Customers report that the TCP connection count on listening tentacles increase.

When I tested it, the issue is not present in 4.3.3, but is in 4.3.4 and 4.3.15.

Steps to Reproduce

1. In the Halibut project set SampleClient.Program.Main to:

Log.Logger = new LoggerConfiguration()
    .WriteTo.ColoredConsole()
    .CreateLogger();

Console.Title = "Halibut Client";
var certificate = new X509Certificate2("HalibutClient.pfx");

var hostName = args.FirstOrDefault() ?? "localhost";
var port = args.Skip(1).FirstOrDefault() ?? "8433";
while (true)
{
    using (var runtime = new HalibutRuntime(certificate))
    {
        var calculator = runtime.CreateClient<ICalculatorService>("https://" + hostName + ":" + port + "/", "EF3A7A69AFE0D13130370B44A228F5CD15C069BC");
    
        try
        {
            var result = calculator.Add(12, 18);
            Console.WriteLine("12 + 18 = " + result);
            Thread.Sleep(100);
        }
        catch (Exception ex)
        {
            Console.Write(ex.Message);
        }
    }
}

2. SampleServer.Program.Main to:

Log.Logger = new LoggerConfiguration()
    .WriteTo.ColoredConsole()
    .MinimumLevel.Warning()
    .CreateLogger();

Console.Title = "Halibut Server";
var certificate = new X509Certificate2("HalibutServer.pfx");

var endPoint = new IPEndPoint(IPAddress.IPv6Any, 8433);

var services = new DelegateServiceFactory();
services.Register<ICalculatorService>(() => new CalculatorService());

using (var server = new HalibutRuntime(services, certificate))
{
    server.Listen(endPoint);
    server.Trust("2074529C99D93D5955FEECA859AEAC6092741205");

    Console.WriteLine("Server listening on port 8433. Type 'exit' to quit, or 'cls' to clear...");
    while (true)
    {
        var line = Console.ReadLine();
        if (string.Equals("cls", line, StringComparison.OrdinalIgnoreCase))
        {
            Console.Clear();
            continue;
        }

        if (string.Equals("q", line, StringComparison.OrdinalIgnoreCase))
            return;

        if (string.Equals("exit", line, StringComparison.OrdinalIgnoreCase))
            return;

        Console.WriteLine("Unknown command. Enter 'q' to quit.");
}

3. Run SampleServer under dotMemory
4. Run SampleClient from the command line or IDE
5. Take snapshots in dotMemory about 15 seconds appart
6. Compare the snapshots and filter down to the type TcpClient. Note that there are no Dead Objects in the affected versions.

Attached are the dotMemory snapshots produced using the above procedure
Halibut.SampleServer 4.3.3.dmw.zip
Halibut.SampleServer 4.3.4.dmw.zip

When a client is unreachable, it logs a full exception stacktrace, making it appear this is a completely unexpected situation. We're already specifically catching this and throwing a custom exception - we're just not handling it at the spot where we log.

https://example.com:10933/      128  Unexpected exception executing transaction.
Halibut.HalibutClientException: The client was unable to establish the initial connection within 00:01:00
   at Halibut.Transport.TcpClientExtensions.ConnectWithTimeout(TcpClient client, String host, Int32 port, TimeSpan timeout)
   at Halibut.Transport.TcpClientExtensions.ConnectWithTimeout(TcpClient client, Uri remoteUri, TimeSpan timeout)
   at Halibut.Transport.TcpConnectionFactory.CreateConnectedTcpClient(ServiceEndPoint endPoint, ILog log)
   at Halibut.Transport.TcpConnectionFactory.EstablishNewConnection(ServiceEndPoint serviceEndpoint, ILog log)
   at Halibut.Transport.ConnectionManager.<>c__DisplayClass8_0.<CreateNewConnection>b__0()
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.LazyInitValue()
   at System.Lazy`1.get_Value()
   at Halibut.Transport.ConnectionManager.<>c__DisplayClass8_0.<CreateNewConnection>b__1()
   at Halibut.Transport.ConnectionManager.AcquireConnection(IConnectionFactory connectionFactory, ServiceEndPoint serviceEndpoint, ILog log)
   at Halibut.Transport.SecureListeningClient.ExecuteTransaction(Action`1 protocolHandler)

We should not log the stack trace here, nor should we call it an "unexpected exception".

I've been trying to tackle an issue for about two months now.
I'm not sure if this would be a Halibut issue, or just Windows, or something different, but am hoping this sounds familiar to you or you'd be able to point out ways to debug this better.

We've got Halibut setup in listening mode on quite a couple of machines. This used to work perfectly fine, until an ISP in between us and some machines was changed. Since then (exactly on that date), we started seeing a weird saturation-like issue, where the sites where the ISP has changed suddenly start disconnecting after a certain time period.

Interestingly, restarting the applications on both ends does not resolve this issue but even more interesting, restarting our server (aka the Halibut Client connecting to the listening Halibut Servers) does resolve the issue. Though this issue only occurs on those machines that are behind the ISP change, restarting those does not affect anything but restarting our server does resolve the issue, no other change needed.

Things that I've seen, that might or might not be related:

• Executing a 'health/are-you-up' call seems to make this occur less often, though it does not eliminate
• On the Halibut Server side, behind the ISP, netstat sometimes shows ~5 Established connections to our client (our serverse) while those show 1. Sometimes it shows 0 on each side but a connection still will timeout/fail when trying.
• We run in Microsoft Azure but have set the TCP squeezing to 30 minutes, while the health calls execute once a minute.
• It does often (not sure if always) try to EstablishNewConnection within the SecureClient, and either fail at ConnectWithTimeout but most often at System.Net.Security.SslState.EndProcessAuthentication with a timeout
• Stopping our halibut client (our servers) does not immediately remove Established connections on halibut server. Waiting for a certain period (5-15 minutes) does stop them. Starting the halibut client again after all are closed has no effect (still fails)
• When starting up again (halibut client) the connection shows as Established on both ends, until it fails the System.Net.Security.SslState.EndProcessAuthentication. It then keeps the connection open on the halibut server but the client (our server) does not have the connection anymore. This essentially repeats for each connection attempt and presumably causes more connections to open up to the client.
• It occurs about one time a day, sometimes a bit more, sometimes a bit less.
• We run the connections to the listeners over two machines on our end and it does not happen at both at the same time

I know this is super vague, I've been breaking my head over it for quite some time. The restart of our server solving it is the most annoying bit, as netstat does not show any indication anything still being open, our services on both ends stopped not having any effect. I've been unable to replicate this locally, it's only on those sites. The ISP has been extremely unhelpful in even giving access to the routers (I was thinking it could possibly be some kind of protection setup in the router that kicks in, looking to turn it off). Any ideas to debug, suggestions in any way, would be much appreciated.

Team

• I've assigned a team label to this issue

What happened?

When a deployment to a polling tentacle that has recently gone offline is cancelled, then RPC retries may continue to execute until the RPC retry time limit is reached.

Reproduction

• Ensure RPC Retries is on
• Deploy successfully to a Polling Tentacle that is online
• Take Polling Tentacle offline
• Retry the deployment
• Wait for RPC retry message to start, then cancel

At this point, RPC retry messages will continue to stream in, even though we have cancelled.

Error and Stacktrace

No response

More Information

What appears to be happening is that the cancellation is happening correctly.

But since it has gotten past get capabilities, and has called 'start script', it doesn't know if the Tentacle actually received the message or not. So now it has to call Cancel on the Tentacle just in case.

But because the Tentacle is offline, the Cancel RPC call never gets through.

Workaround

No response

Team

• I've assigned a team label to this issue

What happened?

I'm implementing a service daemon that hosts a TCP listener and perform a few long monitoring tasks and calculations when requested and I really like Halibut programming model using request/response.

However using this approach there is no session and each call ends up to be stateless so I would like to know if the library supports or implements any mechanism that allows to report the execution progress to the client when executing long tasks at the server side? Or how can that be achievable using Halibut?

Reproduction

N/A

Error and Stacktrace

No response

More Information

No response

Workaround

No response

Your recent change to the TypeRegistry allowing polymorphism is a great addition, but we find its contraints a bit too limiting.

The current implementation requires types to be either abstract or interefaces in order to pick up derived types, but this is not always feasible. My suggestion is to introduce a HalibutPolymorphicAttribute and optionally HalibutDerivedTypeAttribute similar to how microsoft did it with System.Text.Json

Hi, we are running some existing application code inside Halibut services remotely. We catch specific exceptions in order to cleanup during failures.

You have done a great job passing over the stacktrace of remote exceptions with HalibutClientException. However, it does not expose the actual InnerException which makes it hard for us to do automatic handling.

Would it be possible for you to extend the ServerError class with an InnerException property so that it can be passed on to the client?

Currently the client proxy interface is permitted to not supply a HalibutProxyRequestOptions, which means the caller does not provide a CancellationToken so the code must generate CancellationToken.None and the caller has no way of cancelling that token.

This also lead to situations in which the code needs to do null checking to see if the HalibutProxyRequestOptions are passed or not.

To complete this we should enforce that HalibutProxyRequestOptions must be provided for every RPC call e.g. the client service contracts must support this parameter and the Proxy enforces it is not null.

See details here #525 (comment)

A SocketException is thrown when Halibut attempts to create connections but IPv6 is disabled:

System.Net.Sockets.SocketException (97): Address family not supported by protocol
 at System.Net.Sockets.Socket..ctor(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
 at System.Net.Sockets.TcpClient..ctor(AddressFamily family)
 at Halibut.Transport.TcpConnectionFactory.CreateTcpClient()
 at Halibut.Transport.TcpConnectionFactory.CreateConnectedTcpClient(ServiceEndPoint endPoint, ILog log)
 at Halibut.Transport.TcpConnectionFactory.EstablishNewConnection(ServiceEndPoint serviceEndpoint, ILog log)
 at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
 at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
 at System.Lazy`1.CreateValue()
 at Halibut.Transport.ConnectionManager.<>c__DisplayClass8_0.<CreateNewConnection>b__1()
 at Halibut.Transport.ConnectionManager.AcquireConnection(IConnectionFactory connectionFactory, ServiceEndPoint serviceEndpoint, ILog log)
 at Halibut.Transport.SecureClient.ExecuteTransaction(Action`1 protocolHandler)

This issue was reported by a customer attempted to use Linux Tentacle: https://octopususergroup.slack.com/archives/CFG3BFMB3/p1568009170003500

Currently there doesn't seem to be any way of notifying the HalibutRuntime consumer when an unauthorized users attempts to connect (other than a debug log message).

I think it might be useful for some applications to make the users or admins aware of these attempts. Looking at the code, it doesn't look too difficult to add a callback from the SecureListener and SecureWebSocketListener's Authorize method.

PR on the way.

I'm investigating adding support for the MS Generic hosting environment, and have gotten quite far. One issue I have found though, is that the abstraction hides access to the runtime, which makes adding and removing trusted thumbprints difficult.

What are your thoughts on adding a ITrustProvider (and a default implementation to use when not provided), and a constructor on HalibutRuntime to take one in.

Something like this :

public interface ITrustProvider 
{
  void Add(string clientThumbprint);
  void Remove(string clientThumbprint);
  bool IsTrusted(string clientThumbprint);
}

public class HalibutRuntime : IHalibutRuntime
{
        public HalibutRuntime(IServiceFactory serviceFactory, X509Certificate2 serverCertificate, ITrustProvider trustProvider)

}

This would allow us to inject the ITrustProvider elsewhere in our application to manage the trust without needing access to the Halibut runtime (which in my case would be contained in an IHostedService instance (which would then take the ITrustProvider in it's constructor).

Our team is looking into integrating a Halibut server into an existing Asp Net application. The hosting model of this application is a bit restrictive so we would like to use the existing iis server if possible. I have created a fork and tested accepting websockets from the Asp Net Owin pipeline, but I would like to keep Halibut updated without maintaining a fork.

Would you consider changing the internal HalibutRuntime ctor, ExchangeProtocolBuilder, HandleMessage, HandleMessageAsync to protected? Then I could simply inherit the HalibutRuntime class and maintain type safety. I could work around this by using reflection, but that seems a bit to brittle for me

Team

• I've assigned a team label to this issue

What happened?

Providing a thumbprint with different casing on the client and server-side gives the following exception:

The server at https://localhost:8080/ presented an unexpected security certificate. We expected the server to present a certificate with the thumbprint 'a7b12a038945afdee39661bcf59cf1600c811122'. Instead, it presented a certificate with a thumbprint of 'A7B12A038945AFDEE39661BCF59CF1600C811122' and subject ''.

As you can see, the thumbprint is correct, but differs in casing. It expected lowercase, but got uppercase. Certificate thumbprints are a hex-string representation of SHA-based hashes and shouldn't be treated as equal whether they are upper- or lower case.

Reproduction

Provide a thumbprint with different casing on the client and server-side.

Workaround

Aligning the thumbprints with the same case fixes the problem.

Hello,

First of all, we've been using Halibut for a long time and we love it! (See our PR regarding routing)

We've recently switched to using WebSockets as it enable us sharing the port with other components running on the same machine easily.

From that moment, we've experienced some strange issues where our Polling Client would suddenly lose connectivity (sometimes having 100% CPU).

Here's what our setup looks like:

The Tentacle is a windows service running under net 4.8 (running latest standard Halibut, no routing)
The Octopus is a windows service running under netcoreapp3.1 (running our Halibut routing)

Logs and dump/trace analysis on the Tentacle suggests that 3 of the PollingThread seems stuck forever (no mention of reconnection or activity in the logs).

We're wondering if it's something you've also experienced since you're using Halibut on such a large infrastructure?

We've also noticed the following issue that looks very similar to what we're experiencing but are unsure if this is applicable to our setup: dotnet/runtime#16810

We do have dumps, logs & dotTrace exports to help if needed, we've tried a lot of things but couldn't figure out where this come from with certitude.

I have a very simple service which in my application is talking around 1300ms for the call (listen mode, client and server on the same machine).

public interface IAgentRegistrationService
{
	string Register(string hostname,);
}

When I transplant this service to the sample app it takes 1.5ms! I've paired the service back so the implementation is as simple as it can get it, no DI (using the DelegateServiceFactory), the implementation is just a simple class with 1 method, the method takes a string, returns the same string back

So I've ruled out the service implementation as the cause of performance issue, but I'm still none the wiser. The application uses microsoft.hosting.extensions and asp.net (with httpsys), ie self hosted. It also uses serilog, autofac, targets netcoreapp3.0 (I needed something that was not in 2.2), so I suspected net core 3.0 might be the cause, but the sample server targeting netcoreapp3.0 does not exhibit this problem.

I tried setting Serlilog MinimumLevel to Error on the client and server, as I saw a reference to this on an earlier issue, but that didn't make any difference.

Any ideas? I'm desperate to resolve this as Halibut so far is the best replacement for WCF that I have found.

We have been using the "listening service model" for a while. And we want to try out “polling service mode” mode, but we found a concerning scenario regarding request routing during our testing.

Given a client runtime which has only trusted fingerprint "xxx" and a server subscribing to the queue "server-1" with fingerprint "xxx" we would expect the following:

• A service request to ServiceEndpoint("poll://server-1", "xxx") should be routed to the server subscribing to “server-1”.
• A service request to ServiceEndpoint("poll://server-1", "yyy") should fail.

However, both scenarios end up routing to the server subscribing to “server-1”. Is this intentional to allow a “competing consumers” pattern or is it a bug?

If it is intentional do you have a suggestion to how we can achieve a strict and secure 1 to 1 communication in polling mode?

In my app I dynamically have a couple of tentacles, who have a port connected and can be either polling or pushing.
I expected the listen call to check if it is already listening, if not to start listening. What it actually does is it created a SecureListener regardless and starts it.

I'd propose a check to see if we're already listening on that endpoint. If not, we add a listener, if so we return the port directly.

Let me know if you guys would appreciate this or if I need to keep an extra state in my own app to make sure I don't do a double-listen.

The issue can be replicated with the following:

• An agreed upon service that sends a DataStream.
• The remote (tentacle) is setup as polling.
• The server sends a DataStream that will throw for example: new DataStream(10000, stream => throw new Exception("Oh noes"))

The result appears to be that the remote wont service any more requests and the CPU will go to 100%.

The issue also happens when the remote is listening, the CPU will go above 100% until some thread limit is exhausted and then will stop servicing any more requests.

It looks like #144 has caused some timeout issues.

This was seen in the octopus server build.

A similar issues appears to occur when the buffer size is set to 40MB, that results in test StreamsCanBeSentToPolling() never terminating. To reproduce on windows disable net462.

Hello,
This article show how to use a custom ClientCertificateValidator but it looks not compatible with actual Library. Do you have a new sample please ?
http://www.codeproject.com/Articles/554692/Halibut-a-secure-communication-stack-for-NET-Mono

It would be nice to do something like this

 var host = new HostBuilder()
        .UseHalibut(options => 
         { 
            options.Port = 9999;
            options.Certificate = @"c:\mycert.pfx";
         })
        .Build();

Obviously it would need to be a separate nuget package as it's a .net core 2.2+ thing. I'm happy to have a crack at it, as I'm about to use Halibut in an app that uses the generic host and want to keep my startup code nice an neat!

Team

• I've assigned a team label to this issue

What happened?

Upgrading System.Drawing.Common in Tentacle requires us to also update Halibut to remove all old versions of the library - see this issue

Reproduction

N/A

Error and Stacktrace

No response

More Information

No response

Workaround

No response

We're very keen on having routing functionality implemented into Halibut for various reasons (see #65 (comment)).

We've started work towards this and would love to have your feedback about simple changes we've made that could enable routing functionality.

POC can be found here: https://github.com/willemda/Halibut/tree/feature/routing/source (Run SampleClient, SampleProxyServer & SampleServer)

Changes can be summarized by this single file changes (HalibutRuntime.cs) :
willemda@018f2ce#diff-e9557d3eda420c227f8e7efdeac71162

• When sending a message, we make use of the routeTable dictionary to check if we need to route the message through another endpoint.

• When receiving a message, we check if the destination of the message matches the current endpoint, if not, we try and send this message over (can be routed again potentially). If yes, we process as usual

Our goal is to implement this in a way that would suit everyone so that we can contribute those changes back into Halibut.

Team

• I've assigned a team label to this issue

What happened?

When passing null to an asynchronous service method accepting a Nullable<T> parameter, Halibut will not consider it a match and fails with the following exception:

Halibut.Exceptions.AmbiguousMethodMatchHalibutClientException : Could not decide which candidate to call out of the following methods:
 - System.Threading.Tasks.Task`1[System.Int32] IncrementAsync(System.Threading.CancellationToken)
 - System.Threading.Tasks.Task`1[System.Int32] IncrementAsync(System.Nullable`1[System.Int32], System.Threading.CancellationToken)
The request arguments were:
<null>

Reproduction

Define an async service method like this:

    public interface ICountingService
    {
        public int Increment(int? number);
    }

    public interface IAsyncCountingService
    {
        public Task<int> IncrementAsync(int? number, CancellationToken cancellationToken);
    }

And invoke it like this:

[Test]
public async Task AsyncInvokeWithNullableParamsOnAsyncService()
{
    var serviceFactory = new ServiceFactoryBuilder()
        .WithConventionVerificationDisabled()
        .WithService<ICountingService, IAsyncCountingService>(() => new AsyncCountingService())
        .Build();

    var sut = new ServiceInvoker(serviceFactory);
    var request = new RequestMessage()
    {
        ServiceName = nameof(ICountingService),
        MethodName = nameof(ICountingService.Increment),
        Params = new object[] { null! },
    };

    var response = await sut.InvokeAsync(request);
    response.Result.Should().Be(1);
}

Error and Stacktrace

Halibut.Exceptions.AmbiguousMethodMatchHalibutClientException : Could not decide which candidate to call out of the following methods:
 - System.Threading.Tasks.Task`1[System.Int32] IncrementAsync(System.Threading.CancellationToken)
 - System.Threading.Tasks.Task`1[System.Int32] IncrementAsync(System.Nullable`1[System.Int32], System.Threading.CancellationToken)
The request arguments were:
<null>

More Information

I have written a test and patched the bug, will submit a PR ASAP.

Workaround

No response

The .nuspec dependency for Newtonsoft.Json is (6.0.0,).

However, the project itself seems to use v7.0.1 based on the packages.config. When building out a POC with the project from NuGet, I get this error, which would indicate that the nuspec probably needs to get updated to match the actual usage.

if i start samples with mono :
mono.exe Halibut.SampleServer.exe
then
mono.exe Halibut.SampleClient.exe
I get this exception on client side
Unhandled Exception:
Halibut.HalibutClientException: An error occurred when sending a request to 'https://localhost:8433/', after the request began: CreateZStream

Halibut logs attempts by unauthorized polling Tentacles to connect to the Octopus Server. These logs were mainly added as part of the reliable comms pod work in 2023, and while potentially still useful as a way to detect abandoned polling Tentacles, they are causing extra noise in logs.

This can be useful information, but it isn't usually actionable as the refused client may not be one managed by the owner of the logs.

I see the latest there is 2.5.0.

Hello,
First of all, great job that you did :-)

I'm building and running all on the same 32 bits computer.

I have a service with this function
int Ping(int a);

I have this exception:
Server exception:
System.Reflection.AmbiguousMatchException: Could not decide which candidate to call out of the following methods:

• Int32 Ping(Int32)
  The request arguments were:
  Int64

  à Halibut.ServiceModel.ServiceInvoker.SelectMethod(IList1 methods, RequestMessage requestMessage) dans y:\work\f70967c2fadadfb5\source\Halibut\ServiceModel\ServiceInvoker.cs:ligne 100 à Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage) dans y:\work\f70967c2fadadfb5\source\Halibut\ServiceModel\ServiceInvoker.cs:ligne 29 à Halibut.HalibutRuntime.HandleIncomingRequest(RequestMessage request) dans y:\work\f70967c2fadadfb5\source\Halibut\HalibutRuntime.cs:ligne 125 à Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func2 incomingRequestProcessor) dans y:\work\f70967c2fadadfb5\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:ligne 139
  0 logs remaining

i tried with that but result is the same.
Int32 Ping(Int32 a);

Do you have any idea ?

Thank you

Hi,
Is there any performance comparison? I have sent 2K RPC calls and it took 29-32 seconds and RPC calls are just given in the eg: CalculatorService methods Add and Substract.

Thanks

Team

• I've assigned a team label to this issue

What happened?

After upgrading from v4.4.793 to v4.5.8, we are getting a messageEnvelope is null error when sending a Halibut request. Any ideas what could be the cause of this error?

Reproduction

This happens when we call a method with the following signature:

public void OnMessageList(Guid asyncCallId, IList<Message> messageList, bool processImmediately = false)

The public properties of Message are:

• long Id
• long? ParentId
• MessageStatusType MessageType
• string Text
• DateTime? TimeStamp
• bool? AdditionalMessagesExist
• bool? TailCountPlusMessagesExist
• int Depth
• bool IsExternal

where MessageStatusType is an enum.

Error and Stacktrace

An error occurred when sending a request to 'https://servername:9000/', after the request began: messageEnvelope is null

   at Halibut.Transport.SecureListeningClient.HandleError(Exception lastError, Boolean retryAllowed)
   at Halibut.Transport.SecureListeningClient.ExecuteTransaction(ExchangeAction protocolHandler, CancellationToken cancellationToken)
   at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request, CancellationToken cancellationToken)
   at Halibut.HalibutRuntime.<SendOutgoingRequestAsync>d__44.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Halibut.HalibutRuntime.SendOutgoingRequest(RequestMessage request, CancellationToken cancellationToken)
   at Halibut.ServiceModel.HalibutProxy.Invoke(MethodInfo targetMethod, Object[] args)
   at generatedProxy_2.OnMessageList(Guid , IList`1 , Boolean )

More Information

No response

Workaround

Downgrade to v4.4.793

Team

• I've assigned a team label to this issue

What happened?

Occasionally errors such as:

                    |     System.IO.InvalidDataException: The archive entry was compressed using an unsupported compression 

or timeouts seem to occur in halibut.

Reproduction

This tents to happen when halibut is running under NET6 and either, both server and client are under load or hosted away from each other e.g. server is in the cloud and client is on a home internet connection.

This only occurs with polling tentacles (so far).

Error and Stacktrace

|     System.IO.InvalidDataException: The archive entry was compressed using an unsupported compression method.
                    |     at System.IO.Compression.Inflater.Inflate(FlushCode flushCode)
                    |     at System.IO.Compression.Inflater.ReadInflateOutput(Byte* bufPtr, Int32 length, FlushCode flushCode, Int32& bytesRead)
                    |     at System.IO.Compression.Inflater.ReadOutput(Byte* bufPtr, Int32 length, Int32& bytesRead)
                    |     at System.IO.Compression.Inflater.InflateVerified(Byte* bufPtr, Int32 length)
                    |     at System.IO.Compression.DeflateStream.ReadCore(Span`1 buffer)
                    |     at System.IO.BinaryReader.InternalRead(Int32 numBytes)
                    |     at System.IO.BinaryReader.ReadInt32()
                    |     at Newtonsoft.Json.Bson.BsonDataReader.ReadNormal()
                    |     at Newtonsoft.Json.Bson.BsonDataReader.Read()

### More Information

None.

### Workaround

None.

I've found that the 'tentacle' always logs an error after the server is gone.
For example, my server opens a connection to the tentacle whenever I receive a message for it from a queue. The tentacle is (obviously) always listening on the port.

Once the method is executed and the result returned, the tentacle logs an error (see below) about 10 minutes later. It's an obvious error, where it somehow expects the stream to stay open even though the server is long gone. It doesn't impair the usability in any way, but since it basically always occurs and doesn't seem to actually offer any benefit I was wondering if it could be changed so that, when a connection is closed, it doesn't log this. More than happy to put in the work if you guys agree.

System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 96
   at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 67
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
   at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 126

I'm currently using the timers to manage the retries. In the setup I use it, the retries are managed out of Halibut already. It would be nice to set the RetryCountLimit as with the other items in HalibutLimits.

More than happy to make a pull request for this!
If so, let me know if you'd like to relocate this one HalibutLimits too, or keep it in the clients (or make to different ones for each in HalibutLimits).

Hi

Looking at possibly using this as a replacement for wcf.

I'm interested to hear if anyone has integrated this with autofac or another DI container?

A quick look at the code suggests it should be possible, by implementing a custom IServiceFactory?

Halibut tests only test communication between the same dotnet version (e.g. net6 to net6 or net48 to net48).

WebSockets services are only supported in net48 (while the client is supported in both), which means websock tests only run when testing net48 to net48. This means we have no tests for websocket code in net6.

We ought to test that a net6 client can communicate with a net48 service to cover this gap.

Since the CancellationToken is not passed, if an RPC call is cancelled the cancellation of the cancellation token wont result in the write being cancelled resulting in Halibut taking longer to cancel the call.

This can be worked around by using one of the other DataStream constructor methods.

Note that stream timeouts still apply, so Halibut can not be stuck forever in the write.

The issue is in the code below.

Hello, this is a question rather than a problem / issue, but i didn't know where else to ask.

I think that halibut can be used to transfer files, is this correct?
I read in doc: "Requests and responses can also contain streams of arbitrary length".
Is there any code sample on how to transfer a file (with progress report)?
Is there any doc on how octopus actually transfers packages? (I know about octodiff, but i think this is just for patch generation without containing any file-transfer logic, right?).

thanks in advance

Hey,
Awesome project.

An error occurred when sending a request to 'https://localhost:44344/', after the request began: Error resolving type specified in JSON 'Program+ExampleResult, ℛ℘-d0bc25e5-da34-475a-8b08-3768bb4c383f-2'. Path 'Message.result.$type'.

It would be nice that Halibut would be able to deserialize unknown types to dynamic or some kind of proxy with dictionary, when client doesn't recognize the type.

Exception example code:
https://gist.github.com/SlowLogicBoy/cad889cc3db28be0e20794c5fae733ef

Migrating from wcf and hit an issue. In our wcf services, we used the ambient OperationContext.Current to get the calling address, however I cannot find a way to do this in halibut.

Any ideas on how to implement this? I've been looking at the code, and the required info is present in the listener ExecuteRequest method, but it's a major rabit hole to pass that down to the servicefactory (where it could be injected by a custom servicefactory).

Let me know if you'd like me to remove this.

KeepAlives are a useful way of keeping open TCP connections and detecting if a TCP connection is broken, including in the case of a non graceful TCP termination. We have enabled TCP keepalives for listening and polling tentacles as described in OctopusDeploy/OctopusTentacle#692 . Doing so allows Halibut to detect failed connections allowing the caller to retry sooner and in the case of polling tentacles allows for the Polling tentacle to establish a new working connection sooner.

Currently we don't support TCP keepalives for Polling WebSockets as dicussed in this PR, we found that keep alives were only working with TCP connections, not web sockets.

Ideally we should support TCP keepalives with websocket, in this way TCP connectoons that are not gracefully terminated can be detected earlier allowing for callers to be told earlier that the RPC has failed allowing them to retry sooner.