choreos / choreos_middleware Goto Github PK

The REST resource classes should be better documented, since they define details of the REST API.

If two or more services have the same "group" specification, they must be deployed on the same node.

The rationale to this feature was a requirement to collocate some SAs in the old Petals DSB. With Easy ESB maybe this feature is unnecessary.

At the end of the /tmp/chef-client.log:

FATAL: Chef::Exceptions::Exec: package[tomcat6] (tomcat::default line 37) had an error: Chef::Exceptions::Exec: apt-get -q -y install tomcat6=6.0.32-5ubuntu1.3 returned 100, expected 0

Currently the user can write a new CloudProvider or NodeSelector.
But if he write the name of his custom class in the properties file, DeploymentManager will not be able to select his class.
We must enable it.
We must also ensure the user can use his own custom ContextSender and RecipeBuilder.

When 50 VMs are created by POST /nodes, usually one of the threads get blocked (in exact which point?).

Write the LICENSE file and the license headers on each source-code file.

Chef-client is a heavy operation that decreases our scalability.

Currently, if a node receives N services, there will be N chef-client executions. However, all the services will be likely deployed in the first two chef-client executions, so the rest of the executions is an unnecessary over-head.

This situation can be improved if the Deployment Manager can receive multiple service specifications in a single POST /services invocation. So there will be just one "node upgrade" invocation, and therefore just one chef-client execution per node per choreography.

Just after invoking awsCloudProvider.destroy(nodeId), invoking awsCloudProvider.getNode(nodeId) return the node instead throwing the NodeNotFoundException.

It must be renamed to choreographyServiceUUID (so it becomes clear which how attributes must to match).

Each project has its own resources folder. But when project A uses project B, and the JVM is started in the context of the project A, classes in project B will search for resources in the project A resources folder!

Currently we are accessing resources with the classLoader.getSystemResource() method.

Two points where this causes problems:

1. ChefNodeNameRetriever (Commons project) is searching my_chef_name.sh in the Deployment Manager project. Currently this is handled with the duplication of the my_chef_name.sh resource in both projects.

2. Choreography Deployer integration tests should the Deployment Manager, but this step is not working because the way like the RecipeBuilder access the resources. The workaround now is manually starting the Deployment Manager server and after running the Choreography Deployer test. But the right fix would be making the RecipeBuilder properly accessing the resources.

I'm not sure about the best solution, but in a Python project, Eduardo helped me to find the following solution: the source code knows the resource relative path and access it through the absolute path, joining the source file absolute path to the relative path. However some research is necessary to find the best Java way to do this.

A common base class, JCloudsCloudProvider, would avoid some code replication.

The RollingFileAppender limits the log file size.

it is on my desktop : )

If a service deployment has failed, the client must issue a new request POST /chors/{id}/enactment, and the new execution must try to redeploy just the previously failed deployments.

ContextCasterTest must become an unit test using rehearsal mock (after do this, delete the integration test).

The chef bootstrap take so long because it is necessary install the ruby and knife stuff.
This first step could be skipped, since it is possible to create VMs with this stuff already installed.
In our dev environment, we can make a VM snapshot with this things already installed.
So, the bootstrap would have only to deal with server configuration and chef files/configuration on the node.
However, we do not know yet how to tell bootstrap command that chef is already installed. We will have likely to create our own bootstrap script, with the end of the regular chef bootstrap.

Trocar o nome do projeto "EnactmentEngine" por "ChoreographyDeployer"
Trocar o nome do projeto "ServiceDeployer" por "DeploymentManager"
No projeto DeploymentManager, trocar o nome do pacote "serviceDeployer" por "services" e o "npm" por "nodes"; estas duas mudanças são em referência aos nomes recursos REST.

IdlePool could be generalized to a pool of Objects (the new generalized version could stay at the Commons project).

For each available owner, the chordeployer.properties must have an OWNER.DEPLOYMENT_MANAGER_URI entry.

So, the Choreography Deployer will use these entries to deploy a service in the correspondent owner infrastructure. If owner is not specified, the default Deployment Manager is used, just as now.

Concurrent requests can trigger concurrent bootstraps on the same node!
In a situation of concurrent bootstraps, the node got no bootstrapped at all
and NPM did not detect it!
We should avoid the attempt of concurrent bootstraps in the same node.

When a folder already exists under DeploymentManager/src/main/resources/chef/recipes named as the recipe name, the DeploymentManager does not create a new recipe.
It should be possible turn off this cache through config file.

If so, throw a proper exception.

Besides the jar, inside the tar.gz there may be config files.
This would enable the EE user to properly manage the build/release/run cycle by decoupling code from configuration.

Obs: a service must be removed only in this case.
In a choreography update, old services must continue to run.

Currently this verification is very weak, not robust.
We wait for "18" appear on the log.

http://www.a2sdeveloper.com/page-working-with-the-one-second-rule.html
Currently we are relying on the client to respect the limit

Third party library should log only ERROR or SEVERE messages in our log files.

If the Choreography Deployer is restarted it can not lose information about the running choreographies, so we must transform the static map that stores choreographies information in a database.

This is not necessary to the Deployment Manager, since the available nodes can be retrieved from the cloud gateway, and the installed services in each node can be retrieved from Chef.

It must be possible to update a choreography. For now, we will not handle subtle details of dynamic updates.

Request:
URI: choreographydeployer/chors/{id}
Método: PUT
Body: ChorSpec XML

Successful response:
Status code: 200 OK
Header “location”: enactmentengine/chors/{id}
Body: Choreography XML

Failure responses:
400 BAD REQUEST
404 NOT FOUND
500 INTERNAL SERVER ERROR

This is a just-registration operation. Actual job is performed in the next invocation to POST /chors/{id}/enactment. Just the services whose serviceSpec.version had increased must be re-deployed.

Obs: the old version must not to be undeployed.

Throwing an exception would be better than returning null, since not founding a node is really an exceptional situation.

Jar file names get very ugly!
It could have the original name, as in the server.

Check by name and version.

We must verify if the "checking by name and version" overhead is worthwhile when comparing with the time to upload a cookbook (I think it is).

Se não tem ninguém usando o EE, deixa o pool com 0 máquinas. Se começam a chegar requisições, faz o pool ter sempre duas máquinas; se o volume de requisições por minuto é "grande" (?), passa para 3 ou 4 máquinas. E faz aquele esquema em que, depois de algum tempo sem uso (tipo 50 minutos) a máquina morre, pra não ficar pagando à toa.

Currently the EE client that wants to deploy a web service that uses some database needs first deploy by its own the database, and pack the database configuration within the service artifact.

Maybe there will be need some synergy with the StorageFactory to handle this issue.

Bootstrap log:

23.20.114.23 There was a network error connecting to the Chef Server:
23.20.114.23 Error connecting to http://aguia1.ime.usp.br:4000/clients - getaddrinfo: Temporary failure in name resolution
23.20.114.23 
23.20.114.23 Relevant Config Settings:
23.20.114.23 -------------------------
23.20.114.23 chef_server_url  "http://aguia1.ime.usp.br:4000"
23.20.114.23 
23.20.114.23 If your chef_server_url is correct, your network could be down.

After the bootstrap, I logged in the VM (EC2) and it had Internet access (ping to uol worked)

Maybe this option has no sense anymore, since the cookbooks are created with the service UUID as part of their names.

And make unit tests!

In the EE there are several points with external calls.

Each call has a "number of trials" and a "timeout for each trial".

It would be better if all these values would be set in a config file.

So, it will be easier to make experiments changing such values to find the more suitable values.

it breaks bamboo

Create different log messages to the following situations:

• service did not receive yet a request to setInvocationAddress
• Airline service is not accessible

Currently, both situations trigger the message NOT POSSIBLE TO BUY NOW.

This change will help in debugging error situations.

Consider http://www.12factor.net/build-release-run
Check Capistrano rollback

Obs: make #7 (Choreography Update) first