Code Monkey home page Code Monkey logo

swiv's Introduction

Build Status

Swiv

Swiv is no longer actively supported or maintained. For something that is under active development, you might be interested in turnilo.

Swiv is a web-based exploratory visualization UI for Druid built on top of Plywood.

Swiv is best used with Druid. Internal and external APIs may change with little notice.

Contribution

This is a community owned and operated project. Contributions are welcome and all interaction is done via this git repository. File issues, make pull requests and review them all here please.

Features

Drag-and-drop UI

Drag to Split

Contextual exploration

Time Highlight

Comparisons

Time Highlight

Usage

Ensure that you have an up-to-date node

Make sure you have node (>= 4.x.x) installed. On MacOS with homebrew you can do:

brew update
brew install node

Install

Next simply run:

npm i -g yahoo-swiv

That's it. You are ready to Swiv.

Example

Start off by running an example (static) dataset:

swiv --examples

Run with Druid

Next connect Swiv to your broker by simply pointing it to your broker host

swiv --druid your.druid.broker.host:8082

Swiv will automatically introspect your Druid cluster and figure out your dimensions and measures.

Note: if Swiv starts up and gives you a query error it is most likely because it could not properly introspect your schema. You probably have some hyperUnique column that Swiv is trying to SUM over. You will have to provide Swiv with a config file as in the nest section.

Create a config

In general Swiv will never know your schema as well as you. To get a better experience you should create a config and provide it to Swiv. The fastest way to create a config is to have Swiv do it for you.

swiv --druid your.druid.broker.host:8082 --print-config --with-comments > config.yaml

The --print-config option will make Swiv run through its regular introspection and then, instead of tarting a server, dump the YAML onto the stdout and exit.

swiv --config config.yaml

Now open the config in your favorite editor and adjust to taste. Make sure to read through the documentation about the possible configuration options.

Development

Here are the steps to clone Swiv and run it as a developer.

Firstly make sure you have the latest node (>= 5.5.x) and gulp installed:

npm i -g gulp

Clone the project

git clone [email protected]:yahoo/swiv.git
cd swiv

Inside the swiv folder run:

npm install
gulp

Finally you have to create a config.yaml file. (or use the sample)

./bin/swiv --druid your.druid.broker.host:8082 --print-config --with-comments > config.yaml

The --with-comments flag adds docs about what goes into the config.

Then you are ready to

./bin/swiv --config config.yaml

We use WebStorm 2016.1 to develop Swiv and the checked in .idea directory contains all of the auto formatting and code styles. You are free to use any editor as all the build scripts are editor agnostic.

Running gulp watch will build the project and start all the automated watchers.

Roadmap

Recent improvements:

  • Exclusion filters
  • Full support of Druid 0.9.1
  • Swiv can connect to multiple clusters, also Postgres and MySQL
  • Continuous dimension filtering and splitting
  • Support for Druid Theta sketches (for countDistinct())
  • Horizontal bars in Table
  • Side panel resizing
  • Ability to define custom granularities for bucketing
  • Timezone support

For a full list of changes see our CHANGELOG

We will be working on:

  • Additional visualizations (geo, heatmap)
  • String / RegExp filters
  • Removing strict limits on queries
  • Bookmarks and dashboarding features
  • Various additions, improvements and fixes to make the app more complete

Questions & Support

Please file bugs and feature requests by opening an issue on GitHub, also questions can be asked via GitHub issues.

swiv's People

Contributors

asherbitter avatar cheddar avatar djfwan avatar evasomething avatar fjy avatar gianm avatar hollow avatar joshwalters avatar lorem--ipsum avatar monitorjbl avatar pjain1 avatar typpo avatar vogievetsky avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

swiv's Issues

Unable to create cluster with auto creation of data cubes

I am trying to add a cluster which has a data cube in it.
When I try to add a cluster, I get 0 data cubes present. If I use the same cluster with pivot 0.10.27, I get proper cubes to add.

I verified using network requests and seems like swiv is not making few requests to get cubes as well as verifying the cluster host:port.

Am I doing something wrong or is there something missing. Any help is much appreciated.
Thanks in advance.

How to get accurate distinct count ?

I am trying to use this tool for some demo. I need distinct count and then calculate average basket size. ( Total Sales / Distinct Orders ).

Is there any way we can do that given that dimension selected can be changed.

Version discrepancy

Hi,

I see that package version defined at package.json is 0.9.39, lower than the version published in the npm package yahoo-swiv (0.9.40).

What are the differences between these versions?

Support for druid 0.9.2 specific features

We should plan to have a release aligned with every release of druid version ensuring the compatibility and making use of new capabilities or enhancements released in the new version of druid on the existing features of swiv.

Thoughts ?

PG10 support

Hi!
First of all, awesome work! I would like to use Swiv with PG and it seems PG10 is not supported at the moment:
Setting up sourceReintrospect timer in cluster 'pg' (every 120000ms)
Cluster 'pg' is running postgres@10beta3
Introspecting all sources in cluster 'pg'
Scanning cluster 'pg' for new sources
Cluster 'pg' making external for 'app_versions'
Fatal settings load error: invalid version 10beta3
Error: invalid version 10beta3
at PostgresExternal.External (/home/swiv/node_modules/swiv-plywood/build/plywood.js:2823:23)
at PostgresExternal.SQLExternal (/home/swiv/node_modules/swiv-plywood/build/plywood.js:6067:16)
at new PostgresExternal (/home/swiv/node_modules/swiv-plywood/build/plywood.js:6251:16)
at Function.exports.External.External.fromValue (/home/swiv/node_modules/swiv-plywood/build/plywood.js:3137:16)
at Cluster.makeExternalFromSourceName (/home/swiv/build/common/models/cluster/cluster.js:65:40)
at /home/swiv/build/server/utils/cluster-manager/cluster-manager.js:248:44
at Array.forEach (native)
at /home/swiv/build/server/utils/cluster-manager/cluster-manager.js:234:21
at _fulfilled (/home/swiv/node_modules/q/q.js:834:54)
at self.promiseDispatch.done (/home/swiv/node_modules/q/q.js:863:30)
Is this PG10 support scheduled in the next versions?
Many thanks,
Flo

Error when save cluster data

When i save cluster data in Settings I am receiving an error:

swiv-main.c2b004db9c5ff65c4130.js:79127 Uncaught TypeError: Cannot read property 'bind' of undefined at ClusterEdit.renderCreateCubesModal (swiv-main.c2b004db9c5ff65c4130.js:79127) at ClusterEdit.render (swiv-main.c2b004db9c5ff65c4130.js:79185) at ReactCompositeComponentWrapper._renderValidatedComponentWithoutOwnerOrContext (swiv.js?v=0.9.42:16535) at ReactCompositeComponentWrapper._renderValidatedComponent (swiv.js?v=0.9.42:16562) at ReactCompositeComponentWrapper._updateRenderedComponent (swiv.js?v=0.9.42:16485) at ReactCompositeComponentWrapper._performComponentUpdate (swiv.js?v=0.9.42:16456) at ReactCompositeComponentWrapper.updateComponent (swiv.js?v=0.9.42:16373) at ReactCompositeComponentWrapper.performUpdateIfNecessary (swiv.js?v=0.9.42:16281) at Object.performUpdateIfNecessary (swiv.js?v=0.9.42:8225) at runBatchedUpdates (swiv.js?v=0.9.42:7817)

Datasource name is missed from the UI

Datasource name is missed from the UI.
swiv

although i can see it on the console.

Introspecting all sources in cluster 'druid'
Scanning cluster 'druid' for new sources
Got the latest time for 'basefact_network' (2014-11-01T00:00:00.000Z)
Got the latest time for 'druid_tpcds_ss_sold_time_2' (1999-09-16T00:00:00.000Z)
Got the latest time for 'druid_tpcds_ss_sold_time_subset' (1999-11-30T00:00:00.000Z)
Got the latest time for 'druid_hive_table' (2016-06-01T02:00:00.000Z)
Got the latest time for 'druid_tpcds_ss_sold_time' (2003-01-02T00:00:00.000Z)

Duplicate dimension name error when introspecting druid

Hey,

we have several dimensions like d_a & d_A or d_key & d_Key which is considered duplicate and Swiv errors out, would it be possible to filter out the duplicates and log error instead of crashing?

Fatal settings load error: duplicate dimension name 'd_a' found in data cube: 'gwiq-daily-p'
Error: duplicate dimension name 'd_a' found in data cube: 'gwiq-daily-p'
    at /usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:39:23
    at List.__iterate (/usr/local/lib/node_modules/yahoo-swiv/node_modules/immutable/dist/immutable.js:2208:13)
    at List.forEach (/usr/local/lib/node_modules/yahoo-swiv/node_modules/immutable/dist/immutable.js:4383:19)
    at checkUnique (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:36:20)
    at new DataCube (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:137:9)
    at DataCube.addAttributes (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:728:16)
    at DataCube.updateWithExternal (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:497:21)
    at Function.DataCube.fromClusterAndExternal (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:164:48)
    at SettingsManager.onExternalChange (/usr/local/lib/node_modules/yahoo-swiv/build/server/utils/settings-manager/settings-manager.js:237:41)
    at ClusterManager.updateManagedExternal (/usr/local/lib/node_modules/yahoo-swiv/build/server/utils/cluster-manager/cluster-manager.js:75:23)

I don't know swiv much, so I cannot really tell why it is trying to avoid duplicate dimensions and metrics and as I've never used JS or TS, I'd have hard times to submit a fix.

Split Query Limitation -- Plywood

is there any limitation in the plywood query that only we can split upto 5 dimensions? I'm getting error if i add more that 5 in the split query. Please help

Generating swiv urls

Hi, I'd like to be able to generate swiv URLs outside of swiv itself so I can deep link to a specific search. Is there any documentation outline how that URL is formatted?

Increasing default timeout for druid introspection

Hey,

I'm trying to do this :

swiv --druid your.druid.broker.host:8082 --print-config --with-comments > config.yaml

But as one of the datasources has low granularity and it takes more than 2 minutes to retrieve the metadata, Swiv is timing out on this datasource and repeatedly trying to do so again infinitely which ultimately overloads historical nodes :

problem

Is it possible to change this timeout without building swiv ?

Can not start swiv Cannot find module 'fs-extra'

After a fresh install of swiv i am getting the following error

 swiv --druid 52.33.252.237:8888
module.js:472
    throw err;
    ^

Error: Cannot find module 'fs-extra'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/yahoo-swiv/node_modules/fs-promise/index.js:4:15)

you can see the versions of my stack here

[sbouguerra@HW13470 ~] npm i -g yahoo-swiv
/usr/local/bin/swiv -> /usr/local/lib/node_modules/yahoo-swiv/bin/swiv
/usr/local/lib
└─┬ [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  └── [email protected]

[sbouguerra@HW13470 ~] swiv --druid 52.33.252.237:8888
module.js:472
    throw err;
    ^

Error: Cannot find module 'fs-extra'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/yahoo-swiv/node_modules/fs-promise/index.js:4:15)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
[sbouguerra@HW13470 ~] swiv --example
module.js:472
    throw err;
    ^

Error: Cannot find module 'fs-extra'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/yahoo-swiv/node_modules/fs-promise/index.js:4:15)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)

Documentation for connection to Postgres

Please provide the documentation for connecting to Postgres from swiv.

Is any way to introspect postgres?
How to configure swiv to show data from tables in Postgres Database?

Getting compilation error while running gulp command

I am getting this error while running gulp.

/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(179,11): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(180,12): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(181,13): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(178,10): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(179,11): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(180,12): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(181,13): error TS1005 ',' expected.

Any suggestion would help?

I have

Typescript does not compile

I don't do a lot with Node and I've never tried to use Typescript before, so it's entirely possible that I'm just doing something wrong. However, I've tried just about everything I can think of and this project simply will not build.

This is what I do to attempt a build:

npm install -g gulp
npm install
gulp

I then get

[00:43:58] Using gulpfile ~/Projects/swiv/gulpfile.js
[00:43:58] Starting 'all'...
[00:43:58] Starting 'clean'...
[00:43:59] Finished 'clean' after 145 ms
[00:43:59] Starting 'style'...
[00:43:59] Starting 'icons'...
[00:44:00] Finished 'icons' after 1.05 s
[00:44:01] Finished 'style' after 2.31 s
[00:44:01] Starting 'server:tsc'...
[00:44:01] Starting 'client:tsc'...
node_modules/@types/express-serve-static-core/index.d.ts(242,37): error TS1110 Type expected.
node_modules/@types/express-serve-static-core/index.d.ts(243,39): error TS1110 Type expected.

There are dozens more Typescript errors similar to that in the log. Based on the research I've done, the *.d.ts files are apparently being treated as normal Typescript files in the compilation process. The type definition files are not in Typescript syntax and so they understandably throw these compilation errors.

I've done a bit with Gulp, so I tried to see if I could fix it. However, the build is using Imply's laborer package to compile the typescript, which is completely undocumented. Seriously, even by OSS standards that project is egregious.

Anyway, the last build in Travis appears to have worked, so I tried building this in an Ubuntu Docker container with matching major versions of the node/npm:

root@dfd7a351076f:/swiv# node --version
v6.11.1
root@dfd7a351076f:/swiv# npm --version
3.10.10
root@dfd7a351076f:/swiv# gulp --version
[05:01:11] CLI version 3.9.1
[05:01:11] Local version 3.9.1

Same problem with the latest commit (e4085f4) and latest tag (v0.9.42):

root@dfd7a351076f:/swiv# ./travis-test 
[05:01:52] Using gulpfile /swiv/gulpfile.js
[05:01:52] Starting 'all'...
[05:01:52] Starting 'clean'...
[05:01:52] Finished 'clean' after 121 ms
[05:01:52] Starting 'style'...
[05:01:53] Starting 'icons'...
[05:01:54] Finished 'icons' after 1.49 s
[05:01:55] Finished 'style' after 2.76 s
[05:01:55] Starting 'server:tsc'...
[05:01:55] Starting 'client:tsc'...
/swiv/node_modules/@types/express-serve-static-core/index.d.ts(242,37): error TS1110 Type expected.
/swiv/node_modules/@types/express-serve-static-core/index.d.ts(243,39): error TS1110 Type expected.

Is this happening for others? I don't see any other way that I could build this project, but given the complete lack of documentation for laborer perhaps I'm missing something.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.