cmr-exchange / dev-env-manager Goto Github PK
View Code? Open in Web Editor NEWAn Alternate Development Environment Manager for the CMR
License: Apache License 2.0
An Alternate Development Environment Manager for the CMR
License: Apache License 2.0
Right now, the Hawk watch objects are thrown away; we should track them so that we can unwatch and close during component shutdown.
There are lots of different ways to fix this ... need to identify the one that will be efficient for core.async
.
With the latest changes to the code, (run)
now induces very high CPU load. Need to find out why, and then fix that.
To duplicate:
lein repl
(run)
-- ~one CPU core will jump a little, due to the new service starting up, and then it will settle down.(reload)
-- all CPU cores will jump, and then 1-2 will stay maxed at 100%; subsequent, repeated (reload)
s will cause all CPU cores to peg out at 100%.Depends on #3.
This will be another docker service.
cmr.dev.env.manager.watcher.impl.linux
cmr.dev.env.manager.components.common
cmr.dev.env.manager.components.checks.common
Leaning towards the first one, since:
cmr.dev.env.manager.components
and cmr.dev.env.manager.components.dem
cmr.dev.env.manager.components.checks
health pulls in ns's from cmr.dev.env.manager.components
and cmr.dev.env.manager.components.dem
... and we wouldn't want to cross the streams ...We can use the pubsub component/capability once #4 is done.
timing
functions created in #19timing/timer
function to accept a parameter to set the looping timeThis can be fixed by ensuring that the docker id file is only deleted if it actually exists.
Important Links:
Epic features:
MXBean info in Java requires a sample period of time to get data; this is a perfect fit for a timer function that would update the system state atom in the REPL.
Info on extracting stats: https://stackoverflow.com/a/13034111
cmr.dev.env.manager.watcher.impl.nio
See:
We need to be able to run functions periodically, every given interval. This will be used initially to support health checks of running services/processes.
Dev tasks:
core.async
examples of thisWe may need to implement components for various services that are used in CMR's dev-system. Possible candidates:
batch-subscribe
functionbatch-subscribe
functionDefaultSubscribers
componentDefaultSubscribers
to systemcmr.dev.env.manager.components.system
cmr.dev.env.manager.components.common.docker
(depends on #14)dev-system
Update Watcher component to:
:code-change? true
, :successful-recompile? false
:topic :unsuccessful-recompile
:message {... :service-key service-key}
:code-change? true
, :successful-recompile? true
:topic :successful-recompile
:message {... :service-key service-key}
Tasks:
core.async
timer functionality - see #19cmr.dev.env.manager.components.dem.health
cmr.dev.env.manager.components.system
cmr.dev.env.manager.components.system
Data structure might be something like this:
[{:name :mock-echo :type :tcp :frequency 30000}
{:name :mock-echo :type :http :frequency 5000}
{:name :elasticsearch :type :docker :frequency 10000}
Depends on #21
cmr.dev.env.manager.components.common.docker
cmr.dev.env.manager.watcher.impl.darwin
From cmr.dev.env.manager.config
to cmr.dev.env.manager.components.dem.config
.
Use trifl.
cmr.dev.env.manager.watcher.impl.hawk
This might be all we really need ... seems like it supports Linux well via Java WatchService API and on Mac OS X, the BarbaryWatchService.
Also wrap the lein
function project/read
so that we can memoize that, too (it's expensive).
This should decrease startup
times.
Tasks:
project/read
default-config
build
System should shutdown so that there are no remaining child/spawned processes (e.g., service process should be exited, as should running docker container processes).
The core functionality for this is provided by the core.async/unsub
function:
(unsub p topic ch)
However, this requires that we hold onto the channel that gets subscribed to a publisher. The pubsub messaging impl needs to be updated to track subscribers. These should probably be named, too -- which means a handful of functions need to be updated. We should also provide API fns for getting all subscribers, one subscriber, unsubscribing all subscribers, and unsubscribing one.
This will mean we can update the shutdown for the default-subscriber component to unsub its own and then update the messaging component to unsubscribe all. The timer component can unsubscribe its own, too.
Add a lein
profile for running with jmx instrumentation enabled.
Right now, any component that needs to send or receive messages is creating and managing its own set of core.async channels. It would be better if there was a pubsub component that allowed any component to publish to topics, and any component to subscribe to topics.
In cmr.dev.env.manager.components.*
, remove the name from the constructor function symbol that is redundant with the namespace.
For every type of supported health check, we will need a function to perform the check. So far, these are:
:tcp
- a simple TCP "ping-like" functionality (e.g., TCP CONNECT
-only) - connect to host and port:http
- an HTTP health check - do an HTTP GET
against a specified endpoint and parse results:docker
- check the status of a running Docker container:cpu
- check the CPU usage of a process-based service:mem
- check the memory usage of a process-based serviceUltimately, these would be used like so at the system-component-level:
[{:name :mock-echo :type :tcp :frequency 30000}
{:name :mock-echo :type :http :frequency 5000}
{:name :elasticsearch :type :docker :frequency 10000}
But directly, these functions would probably be utilized by a new health
method supported by service components -- those would be responsible for setting/configuring the proper hosts, ports, endpoints, containers, etc.
As such, dev tasks might be this:
{:connects? <bool>}
GET
ter with response body-parsing and HTTP status code-checking ... maybe return {:status <int> :value <parsed body>}
{:container-up? <bool>}
{:cpu <float>}
ProcessRunner
component to support a health
methodDockerRunner
component to support a health
methodhealth
methods for each service type to call the appropriate generic health functions, dispatched on typeA declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.