As user, I would like event detection to consider when a station is temporarily discontinued about hydrotools HOT 9 CLOSED

Good morning Alex:

Can you post your code so we can duplicate the issue?

from hydrotools.

I am using the evaluation manager calling the file decomposition.py

here is the code:

        for nws_lid in sites:
            # Slice on lid
            site_df = (forecasts_df.loc[forecasts_df['nws_lid'] == nws_lid,
                       :])

            # Extract location details
            usgs_site_code = site_df['usgs_site_code'].values[0]

            # Slice out forecasts
            for issue_time in site_df['forecast_issue_time'].unique():

                # Slice on issued time
                issued_df = (site_df.loc[site_df['forecast_issue_time'] == issue_time,
                                         ['dateTimeUTC', 'value']])

                # Set index
                issued_df = issued_df.set_index('dateTimeUTC')

                # Resample
                issued_df = issued_df.resample('1H').nearest()

                # Define event
                #issued_el = self.list_events(issued_df, baseflow_window='36H')
                issued_el = self.list_events(issued_df, halflife='6H', window='7D')

from hydrotools.

So far, I'm unable to duplicate this issue. A working example is below. However you'll notice some lines of code are commented out. These lines are preprocessing steps you'll want to include somewhere in your code before calling event detection directly like this. The event detection algorithm expects a continuous time series with a monotonically increasing datetime index.

If you need an example of how to run event detection on multiple sites, there's an example in the events README located here.

Note: you can now install hydrotools from PyPI using pip like this:

pip install -U pip
pip install hydrotools.nwis_client
pip install hydrotools.events

# Import tools
from hydrotools.nwis_client.iv import IVDataService
from hydrotools.events.event_detection import decomposition as ev
# import numpy as np

# Get observations for last 10 days
observations = IVDataService.get(
    sites='02064000',
    period='P10D'
)

# Check for duplicate timestamps in case NWIS returned multiple time series
# observations = observations.drop_duplicates(
#     subset=['value_date']
# )

# Set index to 'value_date'
observations = observations.set_index('value_date')

# Handle missing or negative values, transform to hourly data
# observations.loc[observations['value'] < 0.0, 'value'] = np.nan
# observations = observations.resample('H').nearest().ffill().bfill()

# Detect events on 'value' time series
events = ev.list_events(
    observations['value'],
    halflife='6H',
    window='7D'
)

# Print events
print(events)

                 start                 end
0  2021-04-30 21:15:00 2021-04-30 21:15:00
1  2021-05-01 00:30:00 2021-05-01 00:45:00
2  2021-05-01 01:15:00 2021-05-01 01:15:00
3  2021-05-01 01:45:00 2021-05-01 01:45:00
4  2021-05-01 02:15:00 2021-05-01 02:30:00
5  2021-05-01 06:45:00 2021-05-01 07:00:00
6  2021-05-01 12:00:00 2021-05-01 13:15:00
7  2021-05-01 17:30:00 2021-05-01 17:45:00
8  2021-05-01 18:30:00 2021-05-01 19:00:00
9  2021-05-02 00:00:00 2021-05-02 00:00:00
10 2021-05-02 00:30:00 2021-05-02 00:45:00
11 2021-05-02 01:45:00 2021-05-02 01:45:00
12 2021-05-02 04:00:00 2021-05-02 04:00:00
13 2021-05-02 05:30:00 2021-05-02 05:30:00
14 2021-05-02 06:00:00 2021-05-02 06:00:00
15 2021-05-02 06:45:00 2021-05-02 06:45:00
16 2021-05-02 07:15:00 2021-05-02 08:45:00
17 2021-05-02 09:45:00 2021-05-02 09:45:00
18 2021-05-02 10:30:00 2021-05-02 10:30:00
19 2021-05-02 11:30:00 2021-05-02 11:30:00
20 2021-05-02 12:00:00 2021-05-02 12:00:00
21 2021-05-02 12:30:00 2021-05-02 13:00:00
22 2021-05-02 13:30:00 2021-05-02 13:45:00
23 2021-05-02 14:30:00 2021-05-02 15:00:00
24 2021-05-02 17:00:00 2021-05-02 17:00:00
25 2021-05-02 21:00:00 2021-05-02 21:00:00
26 2021-05-02 22:45:00 2021-05-02 22:45:00
27 2021-05-03 00:45:00 2021-05-03 00:45:00
28 2021-05-03 02:45:00 2021-05-03 02:45:00
29 2021-05-03 05:00:00 2021-05-03 05:00:00
30 2021-05-03 10:45:00 2021-05-03 10:45:00
31 2021-05-03 13:15:00 2021-05-03 13:15:00
32 2021-05-03 18:15:00 2021-05-03 18:15:00
33 2021-05-03 21:00:00 2021-05-03 21:00:00
34 2021-05-04 01:15:00 2021-05-04 01:15:00
35 2021-05-04 03:45:00 2021-05-04 03:45:00
36 2021-05-04 05:15:00 2021-05-04 05:15:00
37 2021-05-04 07:30:00 2021-05-04 07:45:00
38 2021-05-08 08:30:00 2021-05-10 12:15:00

from hydrotools.

Humm -
I will check why is not running on my end. The list seems to be suspicious. Are there events on that list that are with no duration or 15 minutes long?

from hydrotools.

Humm -
I will check why is not running on my end. The list seems to be suspicios. Are there events on that list that are with no duration or 15 minutes long?

That's correct. The method does not filter by default. You'll need to implement your own post-processing or set the minimum_event_duration parameter. You may also want to set the start_radius parameter to shift start times to a local minimum. You can find documentation for event detection here.

Example code:

# Detect events with optional parameters
events = ev.list_events(
    observations['value'],
    halflife='6H',
    window='7D',
    minimum_event_duration='15min',
    start_radius='1H'
)

from hydrotools.

It seems that all those events between 4/20 and 5/4 were calculated when the station was down. It would be nice if the event detection was already filtering the data. The response from nwis is telling us that there was a problem with the equipment.

https://waterdata.usgs.gov/nwis/uv?cb_00060=on&cb_00065=on&format=html&site_no=02064000&period=&begin_date=2021-05-01&end_date=2021-05-11

from hydrotools.

But agree, this is not an event detection problem. It needs to be filtered before enters the method.

from hydrotools.

Is this a request for the event detection algorithm or our internal event detection service? The event detection method hosted here is just a dumb signal processing algorithm that identifies interesting time series features. It doesn't know anything about NWIS or the source signal. The user is the best judge of how to handle their data and whether this particular algorithm is suited to their data and/or their concept of "events."

from hydrotools.

Thank you Jason,
We can close this ticket.

from hydrotools.