Comments (1)
All the type detection is done in detect_types
right now.
How to store the results is a different question. For now I'd recommend sticking to the existing structure until I come up with something better.
One thing that would be quite easy to add would be adding detection for duplicate rows and duplicate columns to detect_types
- or potentially to clean
if that makes more sense. If two continuous features or two categorical features are perfectly correlated (even with different categories) we should detect that and mark one as "useless".
That's a bit dirty, and maybe doing it in clean
would be nicer?
If one column is a monotonous transformation of another column, we might also want to warn, but maybe not drop.
It would be nice to have an option for dropping this, as for tree-based models we could drop it.
Let me know if this clarifies things.
from dabl.
Related Issues (20)
- Showing Alley with high F value seems suspicious in ames housing data
- Plot missing categorical values in classification plots HOT 1
- Use spearman correlation for determining features for regression plots
- ValueError: too many values to unpack (expected 2) with dabl.SimpleClassifier().fit() HOT 2
- add sankey diagram HOT 1
- use to_numeric, infer_objects and convert_dtypes to remove some hacks
- do something on birdmaps
- fix detect_types on empty dataframe HOT 4
- Add example for energy star
- SimpleRegressor and clean don't drop NaNs in target for regression HOT 1
- allow global configuration of what kind of plots are preferred
- bytes type in pandas dataframes leads to errors in type detection
- preserve knowledge of low_card_int in plot
- URGENT: ImportError Issue with `detect_types` Module from `dabl` Library HOT 1
- no useful visualization of parity5_plus_5
- covertype visualization goes awry
- optionally don't do pca, lda
- test_plot_classification_continuous flaky
- dabl.clean changes the values of booleans fields
- dabl.plot dislikes categorical features
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dabl.