yahooarchive / swiv Goto Github PK

Hi,

I see that package version defined at package.json is 0.9.39, lower than the version published in the npm package yahoo-swiv (0.9.40).

What are the differences between these versions?

Hi!
First of all, awesome work! I would like to use Swiv with PG and it seems PG10 is not supported at the moment:
Setting up sourceReintrospect timer in cluster 'pg' (every 120000ms)
Cluster 'pg' is running postgres@10beta3
Introspecting all sources in cluster 'pg'
Scanning cluster 'pg' for new sources
Cluster 'pg' making external for 'app_versions'
Fatal settings load error: invalid version 10beta3
Error: invalid version 10beta3
at PostgresExternal.External (/home/swiv/node_modules/swiv-plywood/build/plywood.js:2823:23)
at PostgresExternal.SQLExternal (/home/swiv/node_modules/swiv-plywood/build/plywood.js:6067:16)
at new PostgresExternal (/home/swiv/node_modules/swiv-plywood/build/plywood.js:6251:16)
at Function.exports.External.External.fromValue (/home/swiv/node_modules/swiv-plywood/build/plywood.js:3137:16)
at Cluster.makeExternalFromSourceName (/home/swiv/build/common/models/cluster/cluster.js:65:40)
at /home/swiv/build/server/utils/cluster-manager/cluster-manager.js:248:44
at Array.forEach (native)
at /home/swiv/build/server/utils/cluster-manager/cluster-manager.js:234:21
at _fulfilled (/home/swiv/node_modules/q/q.js:834:54)
at self.promiseDispatch.done (/home/swiv/node_modules/q/q.js:863:30)
Is this PG10 support scheduled in the next versions?
Many thanks,
Flo

After a fresh install of swiv i am getting the following error

 swiv --druid 52.33.252.237:8888
module.js:472
    throw err;
    ^

Error: Cannot find module 'fs-extra'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/yahoo-swiv/node_modules/fs-promise/index.js:4:15)

you can see the versions of my stack here

[sbouguerra@HW13470 ~] npm i -g yahoo-swiv
/usr/local/bin/swiv -> /usr/local/lib/node_modules/yahoo-swiv/bin/swiv
/usr/local/lib
└─┬ [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├── [email protected]
  └── [email protected]

[sbouguerra@HW13470 ~] swiv --druid 52.33.252.237:8888
module.js:472
    throw err;
    ^

Error: Cannot find module 'fs-extra'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/yahoo-swiv/node_modules/fs-promise/index.js:4:15)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
[sbouguerra@HW13470 ~] swiv --example
module.js:472
    throw err;
    ^

Error: Cannot find module 'fs-extra'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/local/lib/node_modules/yahoo-swiv/node_modules/fs-promise/index.js:4:15)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)

Kindly let me know whether there is any active development or support for swiv?

Please provide the documentation for connecting to Postgres from swiv.

Is any way to introspect postgres?
How to configure swiv to show data from tables in Postgres Database?

Is there a plan to have login module incorporated into swiv?

Can you provide a comparable solution for swiv? Also let me know Swiv was integrate with any other applications?

Hi, I'd like to be able to generate swiv URLs outside of swiv itself so I can deep link to a specific search. Is there any documentation outline how that URL is formatted?

Hey,

I'm trying to do this :

swiv --druid your.druid.broker.host:8082 --print-config --with-comments > config.yaml

But as one of the datasources has low granularity and it takes more than 2 minutes to retrieve the metadata, Swiv is timing out on this datasource and repeatedly trying to do so again infinitely which ultimately overloads historical nodes :

Is it possible to change this timeout without building swiv ?

Datasource name is missed from the UI.

although i can see it on the console.

Introspecting all sources in cluster 'druid'
Scanning cluster 'druid' for new sources
Got the latest time for 'basefact_network' (2014-11-01T00:00:00.000Z)
Got the latest time for 'druid_tpcds_ss_sold_time_2' (1999-09-16T00:00:00.000Z)
Got the latest time for 'druid_tpcds_ss_sold_time_subset' (1999-11-30T00:00:00.000Z)
Got the latest time for 'druid_hive_table' (2016-06-01T02:00:00.000Z)
Got the latest time for 'druid_tpcds_ss_sold_time' (2003-01-02T00:00:00.000Z)

I am trying to use this tool for some demo. I need distinct count and then calculate average basket size. ( Total Sales / Distinct Orders ).

Is there any way we can do that given that dimension selected can be changed.

Hey,

we have several dimensions like d_a & d_A or d_key & d_Key which is considered duplicate and Swiv errors out, would it be possible to filter out the duplicates and log error instead of crashing?

Fatal settings load error: duplicate dimension name 'd_a' found in data cube: 'gwiq-daily-p'
Error: duplicate dimension name 'd_a' found in data cube: 'gwiq-daily-p'
    at /usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:39:23
    at List.__iterate (/usr/local/lib/node_modules/yahoo-swiv/node_modules/immutable/dist/immutable.js:2208:13)
    at List.forEach (/usr/local/lib/node_modules/yahoo-swiv/node_modules/immutable/dist/immutable.js:4383:19)
    at checkUnique (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:36:20)
    at new DataCube (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:137:9)
    at DataCube.addAttributes (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:728:16)
    at DataCube.updateWithExternal (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:497:21)
    at Function.DataCube.fromClusterAndExternal (/usr/local/lib/node_modules/yahoo-swiv/build/common/models/data-cube/data-cube.js:164:48)
    at SettingsManager.onExternalChange (/usr/local/lib/node_modules/yahoo-swiv/build/server/utils/settings-manager/settings-manager.js:237:41)
    at ClusterManager.updateManagedExternal (/usr/local/lib/node_modules/yahoo-swiv/build/server/utils/cluster-manager/cluster-manager.js:75:23)

I don't know swiv much, so I cannot really tell why it is trying to avoid duplicate dimensions and metrics and as I've never used JS or TS, I'd have hard times to submit a fix.

When i save cluster data in Settings I am receiving an error:

swiv-main.c2b004db9c5ff65c4130.js:79127 Uncaught TypeError: Cannot read property 'bind' of undefined at ClusterEdit.renderCreateCubesModal (swiv-main.c2b004db9c5ff65c4130.js:79127) at ClusterEdit.render (swiv-main.c2b004db9c5ff65c4130.js:79185) at ReactCompositeComponentWrapper._renderValidatedComponentWithoutOwnerOrContext (swiv.js?v=0.9.42:16535) at ReactCompositeComponentWrapper._renderValidatedComponent (swiv.js?v=0.9.42:16562) at ReactCompositeComponentWrapper._updateRenderedComponent (swiv.js?v=0.9.42:16485) at ReactCompositeComponentWrapper._performComponentUpdate (swiv.js?v=0.9.42:16456) at ReactCompositeComponentWrapper.updateComponent (swiv.js?v=0.9.42:16373) at ReactCompositeComponentWrapper.performUpdateIfNecessary (swiv.js?v=0.9.42:16281) at Object.performUpdateIfNecessary (swiv.js?v=0.9.42:8225) at runBatchedUpdates (swiv.js?v=0.9.42:7817)

I am trying to add a cluster which has a data cube in it.
When I try to add a cluster, I get 0 data cubes present. If I use the same cluster with pivot 0.10.27, I get proper cubes to add.

I verified using network requests and seems like swiv is not making few requests to get cubes as well as verifying the cluster host:port.

Am I doing something wrong or is there something missing. Any help is much appreciated.
Thanks in advance.

I am getting this error while running gulp.

/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(179,11): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(180,12): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(181,13): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(178,10): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(179,11): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(180,12): error TS1005 ',' expected.
/Users/k0m00qd/projects/swiv-master/node_modules/@types/express-serve-static-core/index.d.ts(181,13): error TS1005 ',' expected.

Any suggestion would help?

I have

[email protected]
[email protected]
[email protected]

I don't do a lot with Node and I've never tried to use Typescript before, so it's entirely possible that I'm just doing something wrong. However, I've tried just about everything I can think of and this project simply will not build.

This is what I do to attempt a build:

npm install -g gulp
npm install
gulp

I then get

[00:43:58] Using gulpfile ~/Projects/swiv/gulpfile.js
[00:43:58] Starting 'all'...
[00:43:58] Starting 'clean'...
[00:43:59] Finished 'clean' after 145 ms
[00:43:59] Starting 'style'...
[00:43:59] Starting 'icons'...
[00:44:00] Finished 'icons' after 1.05 s
[00:44:01] Finished 'style' after 2.31 s
[00:44:01] Starting 'server:tsc'...
[00:44:01] Starting 'client:tsc'...
node_modules/@types/express-serve-static-core/index.d.ts(242,37): error TS1110 Type expected.
node_modules/@types/express-serve-static-core/index.d.ts(243,39): error TS1110 Type expected.

There are dozens more Typescript errors similar to that in the log. Based on the research I've done, the *.d.ts files are apparently being treated as normal Typescript files in the compilation process. The type definition files are not in Typescript syntax and so they understandably throw these compilation errors.

I've done a bit with Gulp, so I tried to see if I could fix it. However, the build is using Imply's laborer package to compile the typescript, which is completely undocumented. Seriously, even by OSS standards that project is egregious.

Anyway, the last build in Travis appears to have worked, so I tried building this in an Ubuntu Docker container with matching major versions of the node/npm:

root@dfd7a351076f:/swiv# node --version
v6.11.1
root@dfd7a351076f:/swiv# npm --version
3.10.10
root@dfd7a351076f:/swiv# gulp --version
[05:01:11] CLI version 3.9.1
[05:01:11] Local version 3.9.1

Same problem with the latest commit (e4085f4) and latest tag (v0.9.42):

root@dfd7a351076f:/swiv# ./travis-test 
[05:01:52] Using gulpfile /swiv/gulpfile.js
[05:01:52] Starting 'all'...
[05:01:52] Starting 'clean'...
[05:01:52] Finished 'clean' after 121 ms
[05:01:52] Starting 'style'...
[05:01:53] Starting 'icons'...
[05:01:54] Finished 'icons' after 1.49 s
[05:01:55] Finished 'style' after 2.76 s
[05:01:55] Starting 'server:tsc'...
[05:01:55] Starting 'client:tsc'...
/swiv/node_modules/@types/express-serve-static-core/index.d.ts(242,37): error TS1110 Type expected.
/swiv/node_modules/@types/express-serve-static-core/index.d.ts(243,39): error TS1110 Type expected.

Is this happening for others? I don't see any other way that I could build this project, but given the complete lack of documentation for laborer perhaps I'm missing something.

is there any limitation in the plywood query that only we can split upto 5 dimensions? I'm getting error if i add more that 5 in the split query. Please help

How can we show actual complete number rather than 0.1k and all.

We should plan to have a release aligned with every release of druid version ensuring the compatibility and making use of new capabilities or enhancements released in the new version of druid on the existing features of swiv.

Thoughts ?

As per implydata/pivot#61