Code Monkey home page Code Monkey logo

cedar's Introduction

cedar -- Build Data Warehouse

Overview

Cedar is a collection of data-focused tools for data generated by and used in Evergreen builds. It is not a part of evergreen, but a companion service and tool kit. In particular, Cedar focuses on the "offline" data processing cases, and providing access to data to support external user interfaces (e.g. cli tools, data visualizations, etc.)

Build operations generate data, and Evergreen stores most of this data directly: the outcome of tasks, the outcomes of specific tests, artifacts produced by builds. There's other data, specific to particular projects, data that is less naturally structured or less specifically linked to specific execution events. Additionally, cedar provides a platform to collect and analyze data about an evergreen deployment in the large.

Components

Cedar has three core components:

  • a simple JSON-only REST web service to data storage and access. (See the rest sub-package.)
  • a command line tool that provides a REST client for common operations, as well as independent data collection and exploration tools. (See the operations sub-package.)
  • an offline task processing framework based on amboy for application-level data aggregation operations. (See the units sub-package.)

Data Types

  • a simple log storage system, as a proof of concept. Logs stream from the client to the cedar application which stores metadata about the logs and then saves data to S3. Out of band, jobs can process log data.
  • MongoDB Core Server internal dependency data which stores data produced by the scons dagger target.
  • Cost spending data, which is collected from Evergreen and hosting providers' APIs (or static information as needed.)

Use

  1. Download and compile cedar. I prefer something like:

    go get github.com/evergreen-ci/cedar
    cd $GOPATH/src/github.com/evergreen-ci/cedar
    make build
    pushd /usr/local/bin
    ln -s $GOPATH/src/github.com/evergreen-ci/cedar/build/cedar
    popd
    
  2. Explore the help menus in the cedar binary:

    cedar --help
    cedar cost --help
    cedar dagger --help
    cedar service --help
    cedar client --help
    cedar worker --help
    
  3. Enjoy!

Development

The cedar project uses a makefile to coordinate testing. Use the following command to build the cedar binary:

make build

The artifact is at build/cedar. The makefile provides the following targets:

test
Runs all tests, sequentially, for all packages.
test-<package>
Runs all tests for a specific package
race, race-<package>
As with their test counterpart, these targets run tests with the race detector enabled.
lint, lint-<package>
Installs and runs the gometaliter with appropriate settings to lint the project.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.