Hi folks,

We are working on a new feature of our product which has a dependency on Release 4.4 of this artifact. Can you please advise its timeline if any, and then we can plan our development & release accordingly?

Thanks.

Basic information

• Axon Framework version: axon-bom:4.9.4'
• JDK version: 21
• Spring Cloud Extension version:
• Complete executable reproducer if available (e.g. GitHub Repo): None
• Running in Application in K8s environment with 4 Application pods connecting to Eureka discovery service running in a separate pod.

Steps to reproduce

1. Run Axon Application with more than three nodes
2. Make sure all the nodes are Registered to Eureka successfully
3. Ingest at least 50K commands to the Axon command gateway
4. Kill all the instances at the same time.
5. Watch the number of Registered instances with Eureka at: http://eureka-host:8761
6. There will be at least one instance still registered with Eureka in emergency state (self-preservation-mode)
7. Other nodes will be routing commands to this node

Expected behavior

As soon as the Axon application dies it should deregister itself with Eureka Service and never re-register until the next heartbeat event.

Actual behavior

When an application shuts down, it de-registers itself with the Eureka server with the status DOWN and registers itself again without having the status UP with the application. This is causing the dead application instance to still be in the list of members to which commands can be routed for some time and eventually removed. The application should immediately de-register and never register in case of an application shutdown event.

In issue #1079, the option was introduced to add application specific exception details back to another node.
The Spring Cloud extension should be adapted to utilize this mechanic too.

The registration of a new ServiceInstance on Eureka fails due to a NullPointerException while retrieving instance URI.

The issue occurred using Spring boot 2.1.2.RELEASE within Spring cloud 2.0.1.RELEASE.

Original Message

Using Consul, the possibility to use the SpringCloudHttpBackupCommandRouter could be error prone and takes additional overhead to determine the supported commands of a serviceInstance.

I think in a lot of cases, filtering a certain serviceInstance by using serviceInstanceFilter on the SpringCloudCommandRouter.Builder should be enough to be sure, that the instance supports all needed Commands.

Proposal: Add a property

useAcceptAllCommandMessageFilter

to the SpringCloudCommandRouter.Builder. It sets the AcceptAll CommandMessageFilter to the MessageRoutingInformation.

What do you think?

Proposed Approach

The gist of the problem, is the way we map a ServiceInstance to it's command handling capabilities, the MessageRoutingInformation. In essence, that could opt to making the SpringCloudCommandRouter#getMessageRoutingInformation(ServiceInstance) method configurable through the builder, instead of a method to override if another implementation is necessary.

The default configured operation would thus use the existing approach, which for the SpringCloudCommandRouter means retrieval from the ServiceInstance#getMetadata field and for the SpringCloudHttpBackupCommandRouter means retrieval by making a REST call. In your application, you could instead adjust that to directly returning a MessageRoutingInformation which accepts all incoming commands.

I'm trying to switch from using Spring Cloud Netflix (Eureka) to Spring Cloud Kubernetes as we are running on OpenShift. Only problem is that KubernetesRegistration.getMetadata() returns null. This causes a NullPointerException in SpringCloudCommandRouter.updateMembership() when invoked from SpringCloudHttpBackupCommandRouter.updateMembership().

I should add that I'm using Axon v3.3.5 and Spring Cloud Kubernetes v1.0.1.RELEASE. Also, when I say that KubernetesRegistration.getMetadata() returns null, I mean it is explicitly returning null. From their github:

public class KubernetesRegistration implements Registration, Closeable {
    .
    .
    .
    @Override
    public Map<String, String> getMetadata() {
        return null;
    }
    .
    .
    .
}

Any suggestions here?

As time has proven the original approach of the SpringCloudCommandRouter, thus using ServiceInstance metadata to communicate command handling capabilities, no longer suffices for a multitude of Spring Cloud Discovery mechanism implementations. This follows from the fact that as far as we know currently no implementation allows continued updates on this metadata in a live set up.

That leaves the SpringCloudHttpBackupCommandRouter as the sole reasonable solution, with a rather awkward naming attached to it. As it's no longer a back-up, but the only solution.
We feel an overhaul on the SpringCloudCommandRouter is thus in place.

Instead of the current set up we should introduce the capability to configure a (for a lack of better word) DiscoveryMode on the SpringCloudCommandRouter.

This DiscoveryMode would provide the implementation on how to receive and send command routing information from one instance to another. We think the following implementations would be reasonable to introduce:

• Dynamic-HTTP -> Essentially the current implementation of the SpringCloudHttpBackupCommandRouter. This should be the default.
• Simple / AcceptAll -> In a "homogeneous command handling"-system, each node can handle everything. Thus there is no real need to share this knowledge at all. Kuddos to @dhermanns in #22 for sharing this idea.

Hi Axon Team!

If someone would like to use Consul right now you will have to use the SpringCloudHttpBackupCommandRouter. It introduces a special message endpoint.

This can add additional complexity and pitfalls e.g. if you have connection timeouts when calling this endpoint.

In our setup all Axon nodes are able to execute all existing commandmessages. So there is no need for this special feature. If you like I can contribute our so called SimpleSpringCloudRouter which doesn't care about commandmessage filtering.

This is why it's running fine on Consul without the need for a special backup messaging HTTP-Endpoint. The blacklisting feature is removed too since every Axon node discovered is able to handle all incoming Axon commands. This makes it for us more robust: Before this, aggregates needs to be "moved" from one node to the other just because of connection timeouts of the messaging endpoint.

When using the SpringCloudCommandRouter, the SpringCloudCommandRouter#resetLocalMembership(InstanceRegisteredEvent) method will clear out the local instance.
This is done, and required, as otherwise the local instance's name and URI will never be set correctly.

Forcefully removing the membership, as is done currently, can however introduce a period of time wherein a SpringCloudCommandRouter isn't aware of any memberships at all until the first HeartbeatEvent is catched to update the memberships.

It would be better to mark the local member in some fashion, so that it can be overwritten upon a HeartbeatEvent, instead of removing it entirely.

Note that I've encountered this behaviour when using Spring Cloud Kubernetes and Spring Cloud Eureka. Unsure whether other implementation, like Spring Cloud Consul for example, introduce the same behaviour, as it depends on how the local Registration is implemented.

Marking this issue as a "Could", as the time window is typically no longer than 30 seconds (the default heartbeat time interval), minimizing the risk overall.

Hi,
Below is the axon stack I'm using:

Axon: 4.2
axonframework.extensions.kafka : 4.0-RC3
axon-distributed-commandbus-springcloud : 4.0-RC1

I've 2 applications(A and B).
Application A has a saga event handler.
Application B has Aggregate and Aggregate command handlers.

Test case is to issue a command from saga event handler(in A) to update aggregate state(in B), in case no aggregate is found then AggregateNotFoundException should be thrown and the same Exception should be caught in the saga event handler.

To achieve this I've configured MessageHandlerInterceptor which will throw CommandExecutionException with actual Exception attached in details object.
And an ErrorHandler to extract the actual exception from CommandExecutionException. (Refer below snippets)

Now in case of AggregateNotFoundException I'm getting Below error in application A, after which the EventProccessor remains in WAITING state and application needs to be restarted.

ThreadDump:

Basic information

• Axon Framework version: 4.6.8
• JDK version: 11
• Spring Cloud Extension version: 4.6.0
• Complete executable reproducer if available (e.g. GitHub Repo): None, but described below

Steps to reproduce

Have an application using the SpringCloudCommandRouter with the RestCapabilityDiscoveryMode which sends commands, but does not have any command handlers.

Expected behaviour

I'd expect this class to be able to send commands to other applications registered with the SpringCloudCommandRouter.

Actual behaviour

We get a "No node known to accept command" error when attempting to send any command from the application with no handlers.

We see the following message in our logs:

o.a.e.s.c.m.RestCapabilityDiscoveryMode : Failed to receive the capabilities from ServiceInstance [<application name>] under host [<ip address>] and port [<port number>]. Will temporarily set this instance to deny all incoming messages.

for each instance the application finds through service discovery. This explains why it cannot send commands; the application has an exception when attempting to discover capabilities on any other application, and so records them as not having any capabilities.

Enabling debug logging allows us to see the stacktrace from the RestCapabilityDiscoveryMode:

RestCapabilityDiscoveryMode  : Service Instance [<instance info>] is denying all messages due to the following exception: 

java.lang.NullPointerException: null
	at org.axonframework.extensions.springcloud.commandhandling.mode.RestCapabilityDiscoveryMode.isLocalServiceInstance(RestCapabilityDiscoveryMode.java:123)
	at org.axonframework.extensions.springcloud.commandhandling.mode.RestCapabilityDiscoveryMode.requestMessageRoutingInformation(RestCapabilityDiscoveryMode.java:107)
	at org.axonframework.extensions.springcloud.commandhandling.mode.RestCapabilityDiscoveryMode.capabilities(RestCapabilityDiscoveryMode.java:91)
	at org.axonframework.extensions.springcloud.commandhandling.SpringCloudCommandRouter.updateMemberships(SpringCloudCommandRouter.java:223)
	at org.axonframework.extensions.springcloud.commandhandling.SpringCloudCommandRouter.resetLocalMembership(SpringCloudCommandRouter.java:183)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...

According to the stacktrace the error appears to be coming from this method:

    private boolean isLocalServiceInstance(ServiceInstance serviceInstance) {
        return Objects.equals(serviceInstance, localInstance.get())
                || Objects.equals(serviceInstance.getUri(), localInstance.get().getUri());
    }

Debugging through it confirms that localInstance is an empty AtomicReference, and so the NPE comes from calling .get().getUri() on it.

Extra info

It looks like the RestCapabilityDiscoveryMode expects AbstractCapabilityDiscoveryMode#updateLocalCapabilities to be called at least once, as it has the side effect of setting the reference for the localInstance. But it seems that since the localCapabilities are already correct by default if there are no command handlers on the local instance, that method never gets called and thus the side effect never occurs.

This appears to be very similar to #1; maybe you could call it a regression, although that issue was for the old Metadata based approach, so that's maybe unfair.

Workarounds

Adding a class like this to the application:

@Service
public class FooCommandHandler {
    @CommandHandler
    public void handle(final Foo command) {}

    public static class Foo {

    }
}

causes the updateLocalCapabilities method to be called and gets rid of the NullPointerException thus allowing commands to be routed properly.

A (slightly?) more graceful workaround we've found is to just call

commandBus.updateLoadFactor(commandBus.getLoadFactor());

in the method where we configure the DistributedCommandBus bean, which also forces the updateLocalCapabilities method to be called and allows command routing.

Feature Description

Axon Framework 4.6.0 has seen the introduction of integrated tracing through Axon Framework issue #2294.
This issue did not cover the SpringHttpCommandBusConnector though.

Current Behaviour

The integrated tracing support is not in place for the SpringHttpCommandBusConnector.

Wanted Behaviour

Introduce the SpanFactory in the SpringHttpCommandBusConnector to trace its processes.

Hallo people, I saw that Axon 4.3 is released and the changes for 4.3 in the extension are also in the master branch. But I didn't find any a 4.3 release of extension-springcloud in maven repo. do you have a plan to release 4.3 soon?

thx. a lot.

Currently, the SpringCloudHttpBackupCommandRouter's knowledge of MessageRoutingInformation will only be used if no ServiceInstance metadata is present.

However, through some testing me and @corradom figured out that more recent versions of Eureka support metadata adjustment of the local Registration, but this knowledge is only propagated towards the DiscoveryClient once. Due to this, the global ServiceInstance metadata will be stuck on the command filter DenyAll, making it so that Eureka is currently unusable.

If we can enforce the MessageRoutingInformation from the SpringCloudHttpBackupCommandRouter to be used when such a discrepancy is in place, or if we enforce the usage of MessageRoutingInformation in favor of the ServiceInstance metadata, then we can alleviate this issue.

In order to introduce some adjustments to the extension (like #12) we should first move to the latest version of Spring Cloud Commons.

The serializer used by the SpringCloudCommandRouter is hard-coded.

protected final XStreamSerializer serializer = XStreamSerializer.builder().build();

So far, I have not found a way to provide my own serializer. I would like to do so, since the hardcoded one produces this warning: "Security framework of XStream not initialized, XStream is probably vulnerable."

Hi.
Can Axon framework provide publish events directly to Spring cloud stream?
Or we should use spring-amqp in event-driven microservices based on Spring cloud?
Thanx.

Hey there!

I noticed that if a service wants to use DistributedCommandBus with SpringCloudCommandRouter it's not really possible to do so. SpringCloudCommandRouter will throw a
new IllegalStateException("There should be no scenario where the local member does not exist."
when there haven't been any local member (which would happen when subscribing for a local @CommandHandler?).

It is a perfectly valid scenario that a service would like to dispatch a method to another one, whilst it itself doesn't have any command handlers.

There's no obvious workaround currently, as things are declared as private, so can't really override stuff without copy-pasting the whole class.

Thanks for your help!

It looks to me like a once blacklisted service will stay blacklisted. It will be removed in class SpringCloudCommandRouter here:

    private void cleanBlackList(List<ServiceInstance> instances) {
        blackListedServiceInstances.removeIf(
                blackListedInstance -> instances.stream().noneMatch(instance -> equals(instance, blackListedInstance))
        );
    }

only, if the serviceInstance isn't discovered anymore at all.

This could be problematic, if you are using the SpringCloudHttpBackupCommandRouter.
It blacklists a service just because the method:

private Optional<MessageRoutingInformation> requestMessageRoutingInformation(ServiceInstance serviceInstance) {

failed. This could be the case during the bootup phase: Consul already discovers the service, but the HTTP Endpoint isn't reachable. In our case, the service was blacklisted for all times as a result.

Suggestion: Reevaluate the blacklisted services from time to time. That would work around the race condition during service startup. If you like, I could add a solution to my already existing PR #4.