Comments (3)
I do think missing values ought to be handled in a preprocessing step and the README/docs recommend tips to run event detection, including filling missing data. However, this burden is on the caller. There are too many types of signals and data to account for all the possible ways a given time series could be filled. In some cases, it comes down to personal preference.
I would certainly question the events returned, so a warning may be justified, but I don't think event detection should fill values on the behalf of the user.
from hydrotools.
The filters seem robust again NaN values. The algorithm completes and returns data if NaNs are present. I ran the example below using a 7-day (larger than the gap) and a 30-minute (smaller than the gap) window. The NaNs will definitely influence the deviation from the median signal and effect where the start and end event time land. It's still not recommended to run event detection on time series with missing data, but it won't shut down the process and at least in this case return consistent (if not sensible) "events."
# Import tools to retrieve data and detect events
from hydrotools.nwis_client.iv import IVDataService
from hydrotools.events.event_detection import decomposition as ev
import matplotlib.pyplot as plt
# Retrieve streamflow observations for two sites
service = IVDataService(
value_time_label="value_time"
)
observations = service.get(
sites='01360640',
startDT='2021-07-01',
endDT='2021-08-01'
)
# Drop extra columns to be more efficient
observations = observations[[
'value_time',
'value'
]]
# Check for duplicate time series, keep first by default
observations = observations.drop_duplicates(
subset=['value_time']
)
# Resample to hourly, keep first measurement in each 1-hour bin
observations = observations.set_index("value_time")
observations = observations.resample("15min").nearest(limit=1)
# Detect events
events = ev.list_events(
observations['value'],
halflife='6H',
window='7D',
start_radius="24H"
)
# Print event list
print(events)
# Plot
observations.plot(logy=True)
observations.loc[events.start, "value"].plot(style="o", ax=plt.gca())
observations.loc[events.end, "value"].plot(style="o", ax=plt.gca())
plt.show()
from hydrotools.
I agree with your decision here (not that it matters). You should be able to use the tool improperly. I think there is sufficient documentation and citations to feel comfortable about this issue.
from hydrotools.
Related Issues (20)
- NWIS IV Client `FutureWarning` HOT 3
- NWM Client New Test Failure: AttributeError: 'EntryPoints' object has no attribute 'get' HOT 5
- Pandas >= 2.0.0 package compliance audit HOT 4
- `nwis_client` "sqlite3.OperationalError: database is locked" HOT 6
- Move `hydrotools` namespace packages to separate repositories HOT 3
- "Run Slow Unit Tests" Action has been failing for some time HOT 2
- 3.7 Tests failing: xarray EntryPoints has no attribute get HOT 6
- DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace HOT 1
- AWS Retrospective HOT 10
- SVI Client slow unit tests failing HOT 8
- nwm_client_new documentation is incomplete for private servers. HOT 1
- nwm_client_new `get` methods fails with custom Parquet Store
- Consider supporting MS Azure (`nwm_client_new`) HOT 1
- Determine feasibility of _restclient's continued dependence on `aiohttp_cache_client` HOT 5
- SVI Client get method failing due to Pydantic>2 issue HOT 1
- New version of `_restclient` cannot be pushed to PyPI b.c. namespace packages with leading `_` in package name cannot be uploaded HOT 1
- Add some basic information about the NWM operational configuration to the `nwm_client_new` package. HOT 1
- Event Detection methods are raising `FutureWarning` HOT 3
- question about update cycle for hydrotools HOT 3
- NWPS API Available HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hydrotools.