Comments (4)
So to follow up on this:
For the documentation, I don't have much guidance at this time. If someone has some proposals of what they'd like to see, I can provide some feedback.
As for nan
vs exception, I think the behavior I'd most like is for an individual featurize() function (for one object) to throw an exception if something goes wrong. For example, if the featurizer requires cations and anions (needs + and - charged sites) but you feed in a metal where all the charges are zero - to throw a meaningful exception like RuntimeError("no cation site found"). Hopefully not just a random failure of some line in the code unless the user tries something really crazy (e.g., if they throw in a crystal structure to a composition object, a weird failure is fine).
Note that I am saying this because the featurize_dataframe()
method includes an ignore_errors
key, so I am assuming that if 999 of my entries are good and 1 entry is incompatible with a featurizer, I can still call featurize_dataframe and get data for the 999 and nan
for the 1 without everything crashing. Subsequently, I can use pandas to fill nan
values with 0 or some other value I like.
from matminer.
For nan
vs exception, I found the most efficient workflow for featurizing a lot (all ~60k MP mats) of (potentially) error prone data was to ignore_errors = True and filter mostly nan features with pandas nan threshold afterwards. But I can also see why throwing exceptions is good, in the case of accidentally putting in a structure for a comp or something, because if you had ignore_errors on, you would be essentially eliminating a good feature and not even know it.
Maybe in the future we can throw a warning to the user for each feature that has 90+% (or some parameter percentage) nan values. That way, even if you have a df with 300+ columns, you might not see which columns are mostly nan (and possibly due to a user error) unless you explicitly check, but you would get a warning saying "Look at feature X to make sure all these nan values are actually due to some crazy error, not due to accidentally putting in an invalid argument."
from matminer.
@ardunn @WardLT Should this issue be considered closed?
Note that for 90% nan
errors, I think we should just have a separate utility function for that OR just put that kind of filtering in our example notebooks using standard pandas tools (rather than build it into the featurize_dataframe).
from matminer.
@computron This issue is closed. Also note my comments above are woefully outdated. I think the capabilities of matminer now (ie having both return_errors and ignore_errors) is sufficient for a user to figure out how a featurization went awry. I agree that the 90% thing should be taken care of by pandas once the df is returned.
from matminer.
Related Issues (20)
- Materials Project time split dataset - `load_data_from_json` returns `None` during debugging (conditionally)
- `matminer.datasets.utils._validate_dataset()` flaky on Windows? HOT 5
- [FEATURE REQUEST] SkipAtom compositional featurizer
- AttributeError: 'DensityFeatures' object has no attribute 'desired_features' HOT 1
- SOAP features HOT 2
- Suggestion: OPTIMADE data retriever HOT 1
- Fail to approach MPData HOT 4
- Handling NaNs from ElementProperty HOT 3
- CI failing due to broken mongo service
- Fixing matminer's multiprocessing problem HOT 1
- New release 0.9.0? HOT 2
- WenAlloy wrong valence electron counts
- mp-api for MPDataRetrieval needs upgrade badly HOT 1
- Missing compatibility with pandas v2 HOT 11
- Issue link to matsci.org broken
- compatibility request: pandas-2+ HOT 2
- Error in import composition HOT 6
- Simple composition-based featurization fails due to an upgrade in pymatgen HOT 2
- Re-enable tests that are skipped in CI HOT 3
- when I import matminer,mistake as following:ValueError: Unexpected atomic number Z=119。 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from matminer.