brightercommand / brighter Goto Github PK

As well as amqp and RestMS it would be worth supporting 0MQ to 'get the set'

Celery does this and it looks like a fairly common requirement. http://celery.readthedocs.org/en/latest/getting-started/brokers/redis.html

We might want to wait on Disque https://github.com/antirez/disque/blob/master/README.md as it builds a proper in-memory distributed work queue via broker topology on Redis. That fits our needs better as today we rely on our broker for an HA infrastructure for messaging.

Given we have defined the Brighter message format we can use work queues to become platform independent i.e. send from a producer written in one languages to a consumer in another language. This is the most useful initial set for Huddle.

In essence they need to replace ServiceActivator, but idiomatically for their own runtime

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

The command processor post method has the writing to the message store and the writing to Message Gateway wrapped in a single Retry and Circuit breaker.

If the writing to the message store is successful and writing to the gateway fails and the the retry is triggered, the writing to the message store will fail on the retry.

Writing to the Message Store should be an idempotent operation.

We want both command line and web based monitoring and control of service activator instances. We want to support a view of the message store(s) and sending messages to active instances via a control bus.

The web site should be self-hosted and allow multiple message stores to be viewed in one location.

The control bus ought to accept over-HTTP control inputs. It needs to publish it's statistics. We envisage using the RestMS as a messaging hub to send messages to Brighter, and to publish stats. Those stats could update the message store web interface to show who consumed messages and ack/reject/requeue.

A simple command line like Curl and WGet could then be used to manage instances. In particular we want to report the passage of messages through, and be able to start and stop individual performers. It would be nice to be able to introduce new performers to handle load. Because it might be a pain to keep formatting and sending RestMS control messages we envisage a python script being developed to help automate this.

The web site could then provide a 'nicer' interface but similar functionality

The goal here is to produce something like Celery management and monitoring tools: http://celery.readthedocs.org/en/latest/userguide/monitoring.html

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Appear to be some issues

Get the core libraries building with dnx

Current blocker is System.Configuration, which will not be implemented in the coreclr.
A new configuration library is been built

• IConfiguration
  http://whereslou.com/2014/05/23/asp-net-vnext-moving-parts-iconfiguration/

I've created a microservice setup that uses TopShelf to host a OWIN self host service which starts a WebApp... pretty standard setup. This microservice accepts a command, handles it, packages it up into an event which gets published to a RabbitMQ exchange.

My command processor setup seems trivial:

        RmqMessageProducer gateway = new RmqMessageProducer(logger);

            var commandProcessorBuilder = CommandProcessorBuilder
                .With()
                .Handlers(new HandlerConfiguration(BuildSubscriberRegistry(),
                    new AutofacCommandHandlerFactory(c.Resolve<IComponentContext>())))
                //.NoPolicy()
                .Policies(BuildPolicyRegistry())
                .Logger(logger)
                //.NoTaskQueues()
                .TaskQueues(new MessagingConfiguration(couchbaseMessageStore,
                    gateway, BuildMessageMapperRegistry(c.Resolve<IComponentContext>())))
                .RequestContextFactory(new InMemoryRequestContextFactory());

When I start the service and immediately stop it everything shuts down fine.

If I start it and actually send in a command the command is handled properly but when I shut down the service the service stops but the window does not close. Visual Studio indicates that there is still a process running. My investigation seems to point to the RmqMessageProducer not getting cleaned up. If I turn off the policies and the task queues (bypassing the queue code) the service will shut down correctly. Something is keeping the process running despite the Stop method being called on my service.

Any suggestions of what I can look into?

We don't have an automated build process, just some batch scripts. Use FAKE by preference.
The hard part is building the NuGet packages, particularly the symbol files, which need to understand the files to include, as well as ensuring dependencies to internal projects are aligned.
We also need a deploy script to push to NuGet

Implement a Python equivalent to ServiceActivator, allowing us to send messages from components running one one platform to components running on another.

1: Replace the broker in the Tasks example with the RestMS broker
2: Implement persistent repository; needs to support distribution i.e.multiple broker nodes (Raven?) via AP over CP. This requires the async pipeline from commandprocessor
3: Add a user key api, to generate new user id and key (API not UI based for now)
4: Documentation - stress this is not production quality
5: Support HTTPS

By default unacceptable messages (e.g. messages that failed to be read from message infrastructure)
are acknowledged and drop.

A configuration setting is needed to override this behaviour and allow a threshold to be set.
When the threshold is reached, messages should stop being read by the Connection consumers.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

channel.ExchangeDeclare(configuration.Exchange.Name, ExchangeType.Direct, true);

See the restms_support branch

It should be possible to use Azure Service Fabric for hosting services instead of TopShelf Windows Services and IIS hosted APIs. I'm going to do a proof of concept, for part of an upcoming Microservices talk.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Similar to the consumers issue, but the idea here is to allow you to push messages supporting the Brighter message format on the queue and have them consumed by a consumer in a different format i.e. a golang http server pushes messages onto a queue that are handled by a .NET consumer.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

In Progress on the restms_support branch

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Once we have the gateway working for this application layer, it may make sense to put it into it's own repository. It's not part of the Command Dispatcher/Task Queue project, it's a way to implement it. It also has it's own potential backlog to make a version with persistent queues, HA etc.

The only real advantage of keeping it with the project is that it could be used as the server for the examples, which prevents people from needing to have RabbitMQ installed to run them. One of our goals is to make running the examples easy, because it is often difficult in OSS projects to spin up the examples and see how they work. This is one reason we embed the examples in the project

See http://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
We need to add the headers to our message if someone intends to delay.
I suggest we add an optional parameter to Post with a default value of -1, that indicates the number of seconds to delay (-1 == no delay).

Other platforms i.e. RESTMS will need to figure out their own mechanism to support a delay

It may require a slight re-write to make it simple for the tests to inject the client, into the message producer and consumer. It may also require a slight modification to allow us to pull the server configuration into the HttpServer set up, out of the actual OWIN endpoint.

We use Policy to provide Circuit Breaker, but we don't provide a way to fall back to something other than the call, leaving that to the implementer. Hystrix advice is fail fast, fail silent, static, stubbed, cached via network. This involves raising a new command that is queued or serviced by cache in case those elements are also defective.

https://github.com/Netflix/Hystrix/wiki/How-To-Use

Should we offer more support for fallback options?

The main issue here is that Hystix commands return results. We have strict CQS and so don't do this. So our only model for resilience is to queue for eventual execution. It could suggest that we need to provide a Brighter Query library that supports all these options...

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

Similar to the consumers issue, but the idea here is to allow you to push messages supporting the Brighter message format on the queue and have them consumed by a consumer in a different format i.e. a python django server pushes messages onto a queue that are handled by a .NET consumer.

https://github.com/helios-io/helios

https://github.com/damianh/LibLog

Hi,

I've seen the Not just layers! What can pipelines and events do for you? by Ian Cooper.

In the presentation Ian uses a framework to build a Pipes and Filters pipeline.

I was wondering if I can use Brighter to create pipelines of chained commands.

Since the RequestHandler has a Successor property I've been able to run chained commands like this:

var pump = new PumpCommandHandler(logger);
var filter1 = new Filter1CommandHandler(logger);
var filter2 = new Filter2CommandHandler(logger);  
var sink = new SinkCommandHandler(logger);
pump.Successor = filter1;
filter1.Successor = filter2;
filter2.Successor = sink;

I'm wondering if I'm supposed to use Brighter to do this... or if I should implement a Pipes and Filters pipeline instead.

Can you advise?

Thanks in advance

Today we just constrain both of these to an IRequest. But our semantics imply that Send is one handler only for a Command and Publish is zero or more handlers for an event. Should this be explicit?

The fluent configuration needs to work with SubscriberRegistry and register the first item in the chain. In order to support fluent and attributes we might to parse all handlers in the chain for attributes and insert into the chain where there are attributes even if using fluent.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

I took notice that my handler factory Release was not being called for my command handler. Digging into the Brighter code I realized that the handler is not added to the LifetimeScope unless the handler implements IDisposable. Since a handler that doesn't implement IDisposable is never added to the LifetimeScope, Release will not be called for the handler. I suppose if I have no real cleanup to do in my handler not calling Release would be OK and that it would eventually be disposed.

I thought I'd tell you because based on the documentation I fully expected Release to be called all the time. So I don't know if this is really a code issue or a documentation issue.

We use Ghost to provide initial comments. We want to ensure these are up to date, and then publish to our project web site

As late as 02/27 1:46 PM EST NuGet pulled down version 3.0.94 with all the LibLog goodness and other changes. When I pulled from NuGet at 02/27 6:36 PM EST I'm back at 2.0.1.0 with Common logging, etc.

Dazed and confused...

Low Priority...

In the ServiceActivator app.config.install.xdt

<section name="serviceActivatorConnections" type="paramore.brighter.serviceactivator.ServiceActivatorConfiguration.ServiceActivatorConfigurationSection, paramore.brighter.serviceactivator" allowLocation="true" allowDefinition="Everywhere" xdt:Transform="Insert"/>

should be InsertIfMissing maybe? Every time I upgrade I get the section even if it is already present.

At an attribute of the form

[FeatureSwitch(""MyCoolNewFeature)]

That looks for a FeatureSwitch config setting, in a custom config section with an entry for the feature and a yes/no as to whether the feature is enabled or disabled.

Throw a FeatureNotEnabled exception on trying to access a non-enabled feature.

Default to throw i.e. we forgot to add the config setting

I'm newbie about distributed systems.

IMHO, The ideal world is working together Paramore (https://github.com/iancooper/Paramore/) and Wolfpack (https://wolfpack.codeplex.com/) and another like:

Asimov Deploy https://github.com/asimov-deploy by Torkel Ödegaard (@torkelo).

https://periscope.codeplex.com/Wikipage?ProjectName=periscope

https://github.com/chucknorris/dropkick
https://github.com/mmooney/Sriracha.Deploy
https://github.com/asimov-deploy/asimov-deploy-winagent
http://presto.codeplex.com/

https://wolfpack.codeplex.com/

From all the good answers provided by everyone here I am now getting a better understanding of appropriate use of this topic and IMHO other programmers will definitely benifit from the

knowledge sharing here. Some of the implementations that I had worked in the past could have been better implemented using this knowledge.
IMHO, better samples for minimize learning curve are real applications with full source code and good patterns

Now this is the kind of invetigative reporting, that I am interested in. Real world problems, with real world solutions. Most real world solutions start with kick ass debugging.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

in essence we want to implement TODO MVC equivalent, with us as the server side: http://todomvc.com/

If we want to support multiple application layer protocols we need to separate header and body for the message. The header needs things like id, routing key, retries, delivery date, and content type (JSON, BSON, XML, YAML, pickle, protobuf etc.). The body contains the content in that content-type.

Once messages have an agreed format we can become independent of the application layer protocol used to transport them

Memoization might optimize the pipeline for returning types determined at runtime

The advantage is that this leads towards interop between .NET producers and consumers with other Celery clients and consumers. So it becomes a way to easily platform decouple a solution

The concept behind Brighter being with Paramore was that Paramore was to provide a fuller example of building a hexagonal architecture service using Brighter. I still think that is a valid goal, but it may become questionable for them to live in the same repository over time.

The Taskmailer example will give pipeline errors when you use more than one performer