data-forge / data-forge-js Goto Github PK

View Code? Open in Web Editor NEW

142.0 13.0 11.0 2.14 MB

JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.

License: MIT License

JavaScript 99.99% Batchfile 0.01%

data-forge data-analysis data-wrangling data linq pandas

data-forge-js's People

Contributors

Stargazers

Watchers

Forkers

ashleydavis awesomehumphrey jgabriellima woodyzhou bcolloran younes-barhouni theheimes mmalewski matyunya antonmelbardis david0718

data-forge-js's Issues

Not issue - question. Merge to DFs based on list of column names.

I'm wondering is there any way to merge two (or more) DFs based on multiple column names, similar to what pandas dataframe can do.

Thanks.

Equivalent of Pandas pivot_table

What is the equivalent function of python pandas pivot_table? I see that you have pivot(columns, values), but it doesn't seem to do the same as as python pandas pivot_table one.

Import into Angular 2 project.

Hello, I think it might be also actual to projects with React, Ionic, Typescript etc.

The error I'm getting:

ReferenceError: require is not defined(…)

Simple solution is to create data-forje.bundle.js file that will contain all required('*') modules.

Thanks again for this great package.

.average() returns NaN

while both max and min works fine on the series, if i call median or average on the same series it returns NaN. i've tried .where(v => isNaN(v)).count(), but it returns 0, so this seems to be a real bug.

Failed to minify in create-react-app

Hi there,

When this module is added to a create-react-app in the process of building the application the next message is shown:

"
Failed to minify the code from this file:

./node_modules/data-forge/src/series.js:2235

Read more here: http://bit.ly/2tRViJ9
"
Basically it seems that the series.js contains es6 code when the create-react-app expects all the modules to be in es5 code ..

Would it be possible to get a package build with full es5 support?

Thanks

Unable to install

I have a VueJS project built with vue-cli that I'm struggling to use data-forge-js in. I did the following:

npm install --save data-forge

and added this import to my component:

import dataForge from 'data-forge';

In my component I attempt to create a simple DataFrame:

const dataFrame = new dataForge.DataFrame({ columnNames: ['Col1'], values: [ [1], [5], [10], ], });

Upon executing npm run dev I see the following errors and warning (there are many, I've posted a subset):

WARNING in ./node_modules/require_optional/index.js
82:18-42 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/require_optional/index.js
 @ ./node_modules/mongodb-core/lib/auth/gssapi.js
 @ ./node_modules/mongodb-core/index.js
 @ ./node_modules/promised-mongo/index.js
 @ ./node_modules/data-forge/src/dataframe.js
 @ ./node_modules/data-forge/src/series.js
 @ ./node_modules/data-forge/src/concat-series.js
 @ ./node_modules/data-forge/index.js
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/TrainerHome.vue
 @ ./src/components/TrainerHome.vue
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/MemberHome.vue
 @ ./src/components/MemberHome.vue
 @ ./src/router/index.js
 @ ./src/main.js
 @ multi ./build/dev-client ./src/main.js

WARNING in ./node_modules/require_optional/index.js
90:20-44 Critical dependency: the request of a dependency is an expression
 @ ./node_modules/require_optional/index.js
 @ ./node_modules/mongodb-core/lib/auth/gssapi.js
 @ ./node_modules/mongodb-core/index.js
 @ ./node_modules/promised-mongo/index.js
 @ ./node_modules/data-forge/src/dataframe.js
 @ ./node_modules/data-forge/src/series.js
 @ ./node_modules/data-forge/src/concat-series.js
 @ ./node_modules/data-forge/index.js
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/TrainerHome.vue
 @ ./src/components/TrainerHome.vue
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/MemberHome.vue
 @ ./src/components/MemberHome.vue
 @ ./src/router/index.js
 @ ./src/main.js
 @ multi ./build/dev-client ./src/main.js

...

ERROR in ./node_modules/harmony-proxy/index.js
Module not found: Error: Can't resolve './lib-cov' in 'C:\Users\chris\data\work\move2thrive\move2thrive_web\node_modules\harmony-proxy'
 @ ./node_modules/harmony-proxy/index.js 2:4-24
 @ ./node_modules/promised-mongo/index.js
 @ ./node_modules/data-forge/src/dataframe.js
 @ ./node_modules/data-forge/src/series.js
 @ ./node_modules/data-forge/src/concat-series.js
 @ ./node_modules/data-forge/index.js
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/TrainerHome.vue
 @ ./src/components/TrainerHome.vue
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/MemberHome.vue
 @ ./src/components/MemberHome.vue
 @ ./src/router/index.js
 @ ./src/main.js
 @ multi ./build/dev-client ./src/main.js

ERROR in ./node_modules/babyparse/babyparse.js
Module not found: Error: Can't resolve 'fs' in 'C:\Users\chris\data\work\move2thrive\move2thrive_web\node_modules\babyparse'
 @ ./node_modules/babyparse/babyparse.js 41:16-29
 @ ./node_modules/data-forge/src/dataframe.js
 @ ./node_modules/data-forge/src/series.js
 @ ./node_modules/data-forge/src/concat-series.js
 @ ./node_modules/data-forge/index.js
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/TrainerHome.vue
 @ ./src/components/TrainerHome.vue
 @ ./node_modules/babel-loader/lib!./node_modules/vue-loader/lib/selector.js?type=script&index=0!./src/components/MemberHome.vue
 @ ./src/components/MemberHome.vue
 @ ./src/router/index.js
 @ ./src/main.js
 @ multi ./build/dev-client ./src/main.js
....

I've been able to install and import many other npm packages without an issue. I've done a lot of searching for these errors and warnings but, thus far, I've been unable to get it to work. I'd really like use data-forge-js in my project. Any assistance provided would be much appreciated.

right value of join is null

Hi am having an issue with join() and joinOuterLeft()
I can run the join example here just fine...
https://www.npmjs.com/package/data-forge#join
However when I add my own data the join fails with an error that right is null
TypeError: Cannot read property 'adjust' of null

I suspect that I am missing something very obvious as opposed to is being a bug in your library, but the code appears identical to the example. Any help would be appreciated.

const dataB = [
    { "_id" : "5a651b801d6bd40443c98f38", "datetime" : new Date("2018-01-22T00:00:00Z"), "adjust" : 2 },
    { "_id" : "5a651b801d6bd40443c98f39", "datetime" : new Date("2018-01-23T00:00:00Z"), "adjust" : 2 },
    { "_id" : "5a651b801d6bd40443c98f40", "datetime" : new Date("2018-01-24T00:00:00Z"), "adjust" : 1 },
    { "_id" : "5a651b801d6bd40443c98f41", "datetime" : new Date("2018-01-25T00:00:00Z"), "adjust" : 1 }
];

const dataA = [
    {
        "datetime": new Date("2018-01-21T00:00:00.000Z"),
        "indexType": "stock",
        "vol": 5180900,
        "open": 7500,
        "high": 7579,
        "low": 7463,
        "close": 7579
    },
    {
        "datetime": new Date("2018-01-22T00:00:00.000Z"),
        "indexType": "stock",
        "vol": 6180900,
        "open": 7700,
        "high": 7706,
        "low": 7663,
        "close": 7679
    }
];

var df_a = new dataForge.DataFrame(dataA);
var df_b = new dataForge.DataFrame(dataB);

        var df_merged = df_a.joinOuterLeft(
            df_b,
            left => left.datetime,
            right => right.datetime,
            (left, right) => {
                return {
                    datetime: left.datetime,
                    vol: left.vol,
                    adjust: right.adjust
                };
            }
            )
        ;

pivot API?

hello,

this is an invaluable dataframe library in javascript!

One thing I needed to self-implement was Pivot calculation, which simply counts the occurrence of records along interested two axis. See this example for more clarity, written for Apache Spark: https://databricks.com/blog/2016/02/09/reshaping-data-with-pivot-in-apache-spark.html

I managed to do the same thing in this data-forge but not using the built-in pivot. Looks like the built-in method is not designed for this purpose. I am willing to deliver a pull request but wanted to first confirm if my understanding is correct. And also a suggested name for a new API so that it does not conflict with an existing one.

Thanks,

Junji

add "materialize" method to dataframe to force evaluation and return a new materialized dataframe object

We found performance issue when we drive more dataframe from a common source dataframe.
The iterator is repeating on the common part.
we need something like, so it won't start from beginning.

function materialize(){ 
   return new dataForge.DataFrame({
        	values: this.toArray()
})};

.groupBy.aggregate only returns the first grouping

Hi Ashley, first of all I really admire what you are trying to do with data-forge! here are issues I encountered when trying to use it.

first off, your canonical example for groupby+aggregate seems broken:

// Group by client.
var summarized = salesData
    .groupBy(row => row.ClientName);
    .select(group => ({
        ClientName: group.first().ClientName,

        // ERROR: deflate is not a function of group
        Amount: group.deflate(row => row.Sales).sum(),
    }))
    .inflate() // Series -> dataframe.
    .toArray(); // Convert to regular JS array.

i havent dug through your various versions but it seems that you may not have implemented deflate yet? the example doesnt even actually use aggregate.

now when i try to roll my own aggregation:

var summary = dataFrame
            .groupBy(row => row.account) // i know there are multiple groups
            // // .skip(1)
            .aggregate(seed, 
                (agg, row) => { //only returns the first group
                    return {
                        account: row.toArray().map(x=> x.account), 
                        series: row.toArray().map((x)=>{return {date:x.date, count:x.actcount}}),
                    };
                }
            );

just to be clear, what this code is supposed to be doing is groupBy account name, and then display a pairing of unique account name, alongside a dataframe of corresponding dates and actcounts for each account. When i run this I only get a result for the first matching group and then the other results are not returned, which makes me think i need some sort of map somewhere but I can't figure out where to put it!

am i missing some forEach or map somewhere? i cant figure out how to implement. I'm relatively new to node but have messed around with data wrangling in d3.

Any chance on providing a types entry on DefinitelyTyped?

Not sure if this issue belongs here or on the data-forge-ts project.

https://github.com/DefinitelyTyped/DefinitelyTyped

I have this in my .ts file for my angular-cli project:

`import * as df from 'data-forge';

And WebStorm is giving me the error that it can't resolve the definitions for that module.

Funny thing is that I can still run the app no problem and it functions correctly.

gaps data example has the wrong var name

minor thing

var moment = require('moment');

var dailyShareDataWithGaps = ...

var gapExists = function (pairA, pairB) {
    // Return true if there is a gap longer than a day.
    var startDate = pairA[0];
    var endDate = pairB[0];
    var gapSize = moment(endDate).diff(moment(startDate), 'days');
    return gapSize > 1;
};

var gapFiller = function (pairA, pairB) {
    // Fill values forward.
    var startDate = pairA[0];
    var endDate = pairB[0];
    var gapSize = moment(endDate).diff(moment(startDate), 'days');
    var numEntries = gapSize - 1;

    var startValue = pairA[1];
    var newEntries = [];

    for (var entryIndex = 0; entryIndex < numEntries; ++entryIndex) {
        newEntries.push([
            moment(pairA[0]).add(entryIndex + 1, 'days').toDate(), // New index
            startValue // New value, copy the start value forward to fill the gaps. 
        ]);
    }   

    return newEntries;
}

var sequenceWithoutGaps = sequenceWithGaps.fillGaps(gapExists, gapFiller);

sequenceWithGaps isnt defined

separately - feature request - would be great to have the ability to add Series or Dataframes together