cmr-exchange / dev-env-manager Goto Github PK

Right now, the Hawk watch objects are thrown away; we should track them so that we can unwatch and close during component shutdown.

There are lots of different ways to fix this ... need to identify the one that will be efficient for core.async.

With the latest changes to the code, (run) now induces very high CPU load. Need to find out why, and then fix that.

To duplicate:

1. Start REPL: lein repl
2. Start the system: (run) -- ~one CPU core will jump a little, due to the new service starting up, and then it will settle down.
3. Reload the system: (reload) -- all CPU cores will jump, and then 1-2 will stay maxed at 100%; subsequent, repeated (reload)s will cause all CPU cores to peg out at 100%.

Depends on #3.

This will be another docker service.

cmr.dev.env.manager.watcher.impl.linux

• Option 1: cmr.dev.env.manager.components.common
• Option 2: cmr.dev.env.manager.components.checks.common

Leaning towards the first one, since:

1. the ns's that are using it will be in cmr.dev.env.manager.components and cmr.dev.env.manager.components.dem
2. the ns cmr.dev.env.manager.components.checks health pulls in ns's from cmr.dev.env.manager.components and cmr.dev.env.manager.components.dem ... and we wouldn't want to cross the streams ...

We can use the pubsub component/capability once #4 is done.

• Use the timing functions created in #19
• Create timer subscribers that subscribe to timer intervals
• Update timing/timer function to accept a parameter to set the looping time
• Have the timer component set the looping time

This can be fixed by ensuring that the docker id file is only deleted if it actually exists.

• Define states
• Define state transitions
• Only allow legal transitions between two given states
• Implement start for whole system
• Implement stop for whole system
• Implement restart for whole system

Important Links:

• Problem Statement
• High-level requirements
• Project Design Goals
• Areas to Explore

Epic features:

• System
  • Add component infrastructure
  • Implement whole-system state management - see #11
  • Add component messaging capability - see #4
  • Implement dev-env-only components - IN PROGRESS - see #9
• CMR Services
  • Implement service components - IN PROGRESS - see #8
  • Implement per-service state management - see #10
• Monitoring
  • Create fs watcher protocol/API - see #32
  • Add file system watcher component - see #38
  • Manage code changes - see #35
  • Support auto-restarting services (optional)
  • Support auto-reloading changed files (optional)
  • Add service monitor (health check) component - IN PROGRESS - see #17
  • Support auto-killing/restarting of services with lost connections (optional)
• Performance
  • Add JMX instrumentation - see #3
  • Add time-tracking to state management - see #12

MXBean info in Java requires a sample period of time to get data; this is a perfect fit for a timer function that would update the system state atom in the REPL.

Info on extracting stats: https://stackoverflow.com/a/13034111

cmr.dev.env.manager.watcher.impl.nio

See:

• https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystem.html#newWatchService()
• https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
• https://docs.oracle.com/javase/7/docs/api/java/nio/file/Path.html#register(java.nio.file.WatchService,%20java.nio.file.WatchEvent.Kind...)
• https://docs.oracle.com/javase/7/docs/api/java/nio/file/StandardWatchEventKinds.html
• http://markusjais.com/file-system-events-with-java-7/

We need to be able to run functions periodically, every given interval. This will be used initially to support health checks of running services/processes.

Dev tasks:

• Investigate core.async examples of this
• Implement best fit for our needs
• Parameterize appropriately for use by components

We may need to implement components for various services that are used in CMR's dev-system. Possible candidates:

• elasticsearch - IN PROGRESS - see #15
• in-memory db
• external db
• in-memory queue
• external queue
• aws queue

• Create messenger-oriented batch-subscribe function
• Create system-oriented batch-subscribe function
• Create DefaultSubscribers component
• Add DefaultSubscribers to system

• Create component in cmr.dev.env.manager.components.system
• Use cmr.dev.env.manager.components.common.docker (depends on #14)
• Identify the correct version to use and the corresponding docker container to use
• Add elasticsearch-head support (see #24)
• Ensure that it's configured like the one started by CMR's dev-system

• Add in-REPL state tracking for each component
• Add support for starting a single component
• Add support for stopping a single component
• Add support for restarting a single component
• Add support for tracking start time of a single component
• Add support for tracking end time of a single component

Update Watcher component to:

• Perform "ubercompile" and check for errors
• Upon ubercompile error
  • Update state of service: :code-change? true, :successful-recompile? false
  • Log an error message
  • Publish message :topic :unsuccessful-recompile :message {... :service-key service-key}
  • Go no further in the code change management process
• Upon successful recompile
  • Log success message to debug
  • Update state :code-change? true, :successful-recompile? true
  • Publish message :topic :successful-recompile :message {... :service-key service-key}

Tasks:

• Create generic health check functions - see #18
• Create core.async timer functionality - see #19
• Create timer component - see #21
• Create cmr.dev.env.manager.components.dem.health
  • Ensure individual service state gets updated upon checks
• Create new component instance in cmr.dev.env.manager.components.system
• Add data structure for all desired / currently enabled (or supported) health checks in cmr.dev.env.manager.components.system
• Perform overall system checks, too
  • Ensure system state gets updated upon checks

Data structure might be something like this:

[{:name :mock-echo :type :tcp :frequency 30000}
 {:name :mock-echo :type :http :frequency 5000}
 {:name :elasticsearch :type :docker :frequency 10000}

Depends on #21

• Create a namespace that won't get reloaded
• Define atoms for holding time data:
  • service start times (moved to #10)
  • system start times
  • service stop times (moved to #10)
  • system stop times
• Define convenience functions for extracting data
  • Get last start/stop/restart time
  • Get average start/stop/restart time

Depends upon: #10, #11

cmr.dev.env.manager.watcher.core

• Develop at least one implementation

Depends on one of:

• Mac impl - #30
• Linux impl - #31
• Java NIO impl - #33
• Clojure Hawk impl - #34

• Create namespace cmr.dev.env.manager.components.common.docker
• Support non-daemon process fork
• Capture the container ID and track this in the component's data
• Provide a means of checking the container's status
• Use the container ID to shutdown the correct container

cmr.dev.env.manager.watcher.impl.darwin

From cmr.dev.env.manager.config to cmr.dev.env.manager.components.dem.config.

• Add ltest as a dep
• Run all tests
• Run just unit tests
• Run just integration tests
• Run tests matching regex
• Run test namespace
• Run a single test case

Use trifl.

cmr.dev.env.manager.watcher.impl.hawk

This might be all we really need ... seems like it supports Linux well via Java WatchService API and on Mac OS X, the BarbaryWatchService.

See: https://github.com/wkf/hawk

Also wrap the lein function project/read so that we can memoize that, too (it's expensive).

This should decrease startup times.

Tasks:

• Memoize project/read
• Memoize default-config
• Memoize build

System should shutdown so that there are no remaining child/spawned processes (e.g., service process should be exited, as should running docker container processes).

• mock-echo (done in 783818a)
• cubby (done in 6640171)
• metadata-db
• access-control
• index-set
• indexer
• ingest
• search
• virtual-product
• bootstrap

The core functionality for this is provided by the core.async/unsub function:

(unsub p topic ch)

However, this requires that we hold onto the channel that gets subscribed to a publisher. The pubsub messaging impl needs to be updated to track subscribers. These should probably be named, too -- which means a handful of functions need to be updated. We should also provide API fns for getting all subscribers, one subscriber, unsubscribing all subscribers, and unsubscribing one.

This will mean we can update the shutdown for the default-subscriber component to unsub its own and then update the messaging component to unsubscribe all. The timer component can unsubscribe its own, too.

Add a lein profile for running with jmx instrumentation enabled.

Right now, any component that needs to send or receive messages is creating and managing its own set of core.async channels. It would be better if there was a pubsub component that allowed any component to publish to topics, and any component to subscribe to topics.

In cmr.dev.env.manager.components.*, remove the name from the constructor function symbol that is redundant with the namespace.

For every type of supported health check, we will need a function to perform the check. So far, these are:

• :tcp - a simple TCP "ping-like" functionality (e.g., TCP CONNECT-only) - connect to host and port
• :http - an HTTP health check - do an HTTP GET against a specified endpoint and parse results
• :docker - check the status of a running Docker container
• :cpu - check the CPU usage of a process-based service
• :mem - check the memory usage of a process-based service

Ultimately, these would be used like so at the system-component-level:

[{:name :mock-echo :type :tcp :frequency 30000}
 {:name :mock-echo :type :http :frequency 5000}
 {:name :elasticsearch :type :docker :frequency 10000}

But directly, these functions would probably be utilized by a new health method supported by service components -- those would be responsible for setting/configuring the proper hosts, ports, endpoints, containers, etc.

As such, dev tasks might be this:

• Create generic TCP ping function ... maybe return {:connects? <bool>}
• Create generic HTTP GETter with response body-parsing and HTTP status code-checking ... maybe return {:status <int> :value <parsed body>}
• Create generic Docker container checker ... maybe return {:container-up? <bool>}
• Create a generic process CPU checker ... maybe return {:cpu <float>}
• Update ProcessRunner component to support a health method
• Update DockerRunner component to support a health method
• Implement health methods for each service type to call the appropriate generic health functions, dispatched on type