Doco to explain you shouldn't expose admin interface to public

Explain the level of XSS protection offered to plugins and to suggest you write plugins cognitive of this.

Cougar has a dependency on jcl-over-log4j, which causes problems when co-existing on classpath with commons-logging (since both expose same classes). We should exclude commons-logging when depending on libraries that have transitive dependencies to it.

We should not be accepting it as a parameter value.

We have an issue with cougar platform and could use your assistance.

When running any of our (non-Cougar) applications, we need to override cougar’s com.betfair.cougar.util.configuration.PropertyConfigurer or the application will not run.

The problem we are facing is related to the defaultConfig setter (PropertyConfigurer.setDefaultConfig). This setter is used by Spring to inject the resource in four different xml files, in which the resource is contained in a sub-folder named “conf”. As the path is given as a String starting at the root of the jar (e.g. ), the path is resolved as a ServletContextResource (due to the ‘/’), instead of a ClassPathResource. In light of this, the application fails during startup as a FileNotFoundException is thrown (i.e. such ServletContextResource does not exist).

In order to fix this problem, we are overriding the class (PropertyConfigurer) and altering the setDefaultConfig method to take a String (instead of a Resource), resolving the String ourselves (i.e. this.defaultConfig = new DefaultResourceLoader().getResource(defaultConfig)). This solution does not require any changes in the xml files.

A different (possible) solution to this problem would be to include a “classpath:” prefix on the four xml files (in the defaultConfig property location):
(cougar-core-impl-2.9.6.jar) conf.core-config-cougar-core-impl.xml
(cougar-client-2.9.6.jar) conf.cougar-application-spring.xml
(cougar-marshalling-impl-2.9.6.jar) module-config-cougar-marshalling.xml
(cougar-standalone-ev-2.9.6.jar) module-config-cougar-standalone-ev.xml.

Various requests merge into one:

• We are looking for ways to implement a mechanism for rejecting requests that are coming for non-essential flows when we detect that a service has an out of ordinary load/response time. Impl by having a fully automatic mechanism that based on some rules to take the decision of blocking calls or to limit the number of calls based on flow priorities and/or agreed SLA’s.

Thoughts on how to implement:

Have a list of monitors (or a status aggregator) with thresholds that cause a trigger (seperate to impact on monitoring status), and then based on that execute pieces of logic to process a request at the entry to the EV (and maybe to it's Q), to allow rejection of requests with a reason.

Fails ~40% of travis builds.

See: https://gist.github.com/neothemachine/4060735

Add a note in the public doco regarding the trustworthiness of the data source, and it's potential as an attack vector

Requires https://github.com/betfair/cougar-archetypes/issues?milestone=1&state=open to be complete

At the moment the maven-test-process-plugin in this project doesn't specify -B, so the logs are spammed with download progress lines. Just add -B to each process command.

The issue was that Cougar was returning the error code from a server dependency as the error code from the server rather than an internal error.

The chain:
Client -> Server -> ServerDependency

Responses:
Client gets NotFound <- Server gets NotFound <- ServerDepenedency returns NotFound

What would be nice:
Client gets InternalError <- Server gets NotFound <- ServerDependency returns NotFound
And the Server logs the response it gets from its dependency

Looking into Cougar source code (EventPublisherRunnable class from JMSEventTransportImpl.java, sonic-transport module) I saw that the MessageProducer objects are never closed:
public void run() {
try {
Session session = sessionManager.get();
Destination destination = createDestination(session, destinationName);
MessageProducer messageProducer = session.createProducer(destination);
TextMessage textMessage = getEventMarshaller().marshallEvent(descriptor, event, session);
messageProducer.send(textMessage, textMessage.getJMSDeliveryMode(), textMessage.getJMSPriority(), textMessage.getJMSExpiration());
success = true;
} catch (CougarFrameworkException cfe) { // Catch possible exception thrown from session creation
success = false;
error = cfe;
} catch (JMSException ex) { // Catch any other exception thrown during message publication
success = false;
error = ex;
} finally {
unlock();
}
}

Required https://github.com/betfair/cougar-documentation/issues?milestone=1&state=open to be complete

While we were testing our app's binary connection to another , we configured it with the wrong cert for the service. This results in the following message being logged about 30 times a second or something crazy on our app:

2013-09-20 15:34:52.292: com.betfair.cougar.client.socket.IoSessionFactory INFO - Connected to ***:9003
2013-09-20 15:34:52.293: com.betfair.cougar.netutil.nio.RequestResponseManagerImpl INFO - Notified 0 outstanding requests for session 0023870
2013-09-20 15:34:52.335: com.betfair.cougar.client.socket.ExecutionVenueNioClient WARN - Unexpected exception from session 0023871
javax.net.ssl.SSLHandshakeException: SSL handshake failed.
        at org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:416)
        at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:299)
        at org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53)
        at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:648)

Because the SSL stuff is started after the connection is initiated because the original protocol didn’t support SSL, then it’s an exception handler in the client which triggers a reconnect. So there is no delay like the backoff and retry that occurs with the ReconnectTask in IoSessionFactory.

It would be hard to add one because there’s no context of the original connect request when the exception comes through. We could potentially keep a map of last attempted connect time by address (perhaps with a ttl on entries so we don’t end up with loads) and then implement the backoff logic based on that, but like I said, it’s not going to be an easy fix..

an IDL Must have at least one dataType definition. Now that we have primitive or void return types this validation step is no longer required

We added defensive coding as part of the open sourcing effort, but didn't add unit tests to ensure it works.

Currently it's done on the EV pool, which is rather wasteful.

This predates the introduction of slf4j.

Add capability for a per-op time limit on the server side after which the EV processing thread should be interrupted.

Currently maven-idltods-plugin generates the code by following the Java beans pattern. A builder pattern would enable the created objects to be immutable and provide a more elegant solution for their construction.

http://betfair.github.io/cougar/maven/dependency-convergence.html

Including only ever setting passwords via overrides.properties

Cert strength:
Mention that Cougar supports whatever JSSE supports by default. Pointer to doco on how to change the supported algo's within JSSE

From #19:

Client timestamps in calls, EV answers with suitable code if max-age/expiry time is reached (effectively drops internal requests that have sat in the surge queue or the send buf (bin)) for too long
Allow clients to specify an inbound operation time that the client would specify – that way, we don’t bother to accept a call if it has already aged while it has been in the surge queue

http://sonarqube.15.x6.nabble.com/sonar-dev-Adding-a-FOSS-project-to-Nemo-td5006841.html

If Tornjak is in use on the client (normally because it's another Cougar service), then expose extra client events into kpis, for example:

• response size
• response time

This is waiting on jacoco/jacoco#119

we would like cougar client to be modified to return a sensible error when it receives an HTTP 413 (HEAD full) error from Jetty.

At the moment it returns:
DSC-0026, Server is NOT identified as a Cougar 2 Server

This error can be simulated by passing a very long URI Rescript request

need to decide what to do when breached (shun or surge)

Current activemq transport was cobbled together at the point of open sourcing to provide a concrete jms transport impl.

This should be finished off and integrated into the integration tests.

Add easier way to specify things such as:

• Binding requires SSL
• Binding requires 2-way ssl
• Internal ips only

Could allow whitelist of trusted ip addresses, or trust it because from a non-repudiated client.

The initial build does an install and runs no tests, and then the second build runs all tests (including unit and integration).

We could combine these two builds, which might result in a faster build.

The downside being that it might take longer for the committer to become aware of a compilation failure.

1. As an internal developer
   I want APIs to be easy to code against
   So that my dev/maintenance tasks are easier, and produce cleaner code

2. As a internal/external developer/tester
   I want to be able to exercise APIs with a browser where possible
   So that I have low barriers to integration and testing

At the moment we have (2) at the expense of (1), e.g. one call has 21 parameters, so a call can look like this:

service.doSomething(null, null, null, something, null, null, null, ... (and so on).

To facilitate (1) and (2) simultaneously, allow GETs for complex types.

Possible implementations already discussed somewhat:

a) All items in non-aggregator types must be stringable and not have any naming collisions (or be qualified appropriately)
b) Put JSON object on query string

Also:

Not very readable (subjective):

service.executeQuery(true, 43843, “1234”, null, null, OrderType.EXECUTABLE, null, null);

Much more readable:

            Query req = new Query();
            req.setAllTransactions(true);
            req.setAccountId(43843);
            req.setReferenceNumber(“1234”)
            req.setOrderType(OrderType.EXECUTABLE);
            service.executeQuery(req);

Can be wrapped with sugar:

            service.executeQuery(
                        new Query()
                                    .allTransactions()
                                    .forAccountId(43843)
                                    .withReferenceNumber(“1234”)
                                    .ofOrderType(OrderType.EXECUTABLE)
                                    .build()
                        )
            );

Request objects can also be constructed in Spring, cloned, passed around easily, and so on.

Request objects also don’t suffer from the major (IMO) problem of breaking generated clients at compile time if I add an optional parameter. The way we are doing it, all clients have to add a ‘null’ onto all their calls.

Why don’t we use request objects in our IDDs I wonder? Is there a good reason?

The flat signature in an IDD is often so it can still be called in a browser easily, we have no way to have complex request objects represented in a GET request at the moment.

It would be nice to remove that limitation. Instead of generating a synthetic request for a flat IDD signature, I wonder whether it would be better to make a more OO IDD and use dot notation for specifying object properties on the URI:

?request.thing=1&request.otherThing=1,2,3,4&request.nestedThing.property=foo

Perhaps even to make this backwards compatible it could assume that if you don’t specify a full path that matches anything it should scan the hierarchy and insert the value in any property that matches by name (so thing=1 would still get injected into request).

Messages like these make up 95% (guessing) of the build output.

[INFO] Writing MapOfComplexOperation using template rescript/requestDataType.ftl to file /home/travis/build/betfair/cougar/baseline-client/client-idd/target/generated-sources/java/com/betfair/baseline/v2/rescript/MapOfComplexOperationRequest.java

I've never gained any value from these messages, and their volume makes troubleshooting builds un-necessarily harder. Proposal: suppress them unless mvn is invoked with -X. This will have a small but appreciated effects for folks that have to look at Cougar/Cougar-application build logs on a regular basis.

Add binding toggles by property so that all bindings for a particular protocol can be enabled/disabled at start time.

Currently, when a connection which has heap subscriptions is closed (for whatever reason), all heap subs are terminated and need to be manually reconstructed by the user of the client.

We want to add the capability to automatically failover to another server. This would necessitate:

• Notifying the sub listeners that we're reconnecting
• Resending the sub request (possibly the same used in #38) to a new server
• Repopulating the heap state based on the new initial state
• Notifying the sub listeners that we're reconnected

It would be desirable to be able to perform reconnection on a set of subs at one time, although we need to ensure that the service is given the opportunity to reject these reconnections. We need to ensure that we log the restart of this sub.

There is a danger with a batch reconnect that we overload the server in question since identity resolution will occur for each request and cached identities are perhaps unlikely to exist.

It's perhaps possible that we'd want to maintain a heap with >1 server at all times (in contrast to #38) so that we don't need to receive a new heap at a busy failure point, but only need to worry about the sub creation (on both client and server) - a kind of passive heap that's kept waiting for HA purposes if you will.

We want to add the capability such that a client can choose to migrate a subscription to a particular heap uri to a different server. This situation might occur if a request to subscribe comes in, the uri returned by the server matches one that another server has already returned, and so the client would prefer to receive and process only one stream of updates rather than many copies of the same thing.

We would probably want to do this as part of the subscription process, and should be toggleable.

It's also likely that the client would want to be able to direct the initial subscription request to the same server where the parameters (or a certain subset of parameters) for a particular CO method match previous requests - essentially some kind of subscription request to server affinity.

We'd like to be able to "alias" a path so that a request for say "/webping" actually gets re-routed to "/customContext/health/v3/summary"

So that a particular interface's exposure on a protocol can be enabled/disabled by config params. Related to #32.

Add the ability to return the missing mandatory parameters when a DSC-0018 error is raised.

Also cleanup the upload of the test output when they fail to S3, and stop that running when testing pull requests (we could perhaps cat the output of the failed tests then?)

Cougar uses "," (comma) as separator while rendering collections. This leads to request logs of variable field length which will break any downstream log parser tools, such as ones for importing into a database.

We should use the pipe character as the default.

It should be a non breaking change to add parameters to a BSIDL operation response.

Functionally, that is the case and everything works as it should, but there is a performance impact... on older clients who do not know about the type of the new field, the Hessian deserializer gets asked for its class, 'misses' and throws an Exception. This Exception is then caught and logged.

This happens every time a call is made that returns a type that the client doesn't know about. In the case of a high volume service call, that means every single request/response will throw (and catch, and log and swallow) an exception, which is far from desirable, and has been measured at around 25 times slower than the happy path.

An enhancement is needed to introduce a negative cache, which will avoid the lookup cost and associated exception throwing and logging behavior.

The enhancement has been implemented already for Cougar 2.7, 2.8 and 2.9. I will submit a pull request at some point with the 3.0 changes.