watttime / pyiso Goto Github PK

Args {"market": "RT5M", "latest": true} at 2016-06-10 10:32:00.084118+00:00

 File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 15, in get_generation
    data = c.get_generation(**kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/caiso.py", line 92, in get_generation
    return self._generation_latest()
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/caiso.py", line 752, in _generation_latest
    soup = self.fetch_todays_outlook_renewables()
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/caiso.py", line 708, in fetch_todays_outlook_renewables
    return BeautifulSoup(response.content)
AttributeError: 'NoneType' object has no attribute 'content'

This method may be most analogous to parse_oasis_demand_forecast (https://github.com/WattTime/pyiso/blob/master/pyiso/caiso.py#L377).

It should take a raw_data argument that is the Beautiful Soup output from fetch_oasis.

It should return a list of dictionaries.

Each dictionary should have these keys:

• timestamp
• freq
• market
• ba_name

It should also have one or more keys that are relevant to LMP data.

Tests should be added to tests/test_caiso.py

args {"market": "DAHR", "forecast": true, "freq": "1hr"} at 2016-06-10 22:05:09.354772+00:00

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 30, in get_load
    data = c.get_load(**kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/ercot.py", line 112, in get_load
    df = self._request_report('load_7day')
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/ercot.py", line 30, in _request_report
    params=params).content
AttributeError: 'NoneType' object has no attribute 'content'

Is using Entso-e as a data source on the roadmap for pyiso? Is it something that you would be open to accepting PRs for?

data available http://mis.nyiso.com/public/P-63list.htm
eg http://mis.nyiso.com/public/csv/rtfuelmix/20160526rtfuelmix.csv

CAISO is responding slowly this afternoon, with frequent errors of:
HTTPConnectionPool(host='oasis.caiso.com', port=80): Read timed out. (read timeout=20)

BaseClient.request should catch these and retry a small number of times.

Not all queries support all markets, but the code should support HASP where it's available.

These changes should mostly be in construct_oasis_payload (https://github.com/WattTime/pyiso/blob/master/pyiso/caiso.py#L158).

args {"market": "RT5M", "latest": true} at 2016-06-13 18:41:00.194257+00:00

 File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 30, in get_load
    data = c.get_load(**kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/pjm.py", line 184, in get_load
    load_ts, load_val = self.fetch_edata_point('InstantaneousLoad', 'PJM RTO Total', 'MW')
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/pjm.py", line 56, in fetch_edata_point
    response = self.request(url)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/pjm.py", line 97, in request
    if response.status_code == 400:
AttributeError: 'NoneType' object has no attribute 'status_code'

only rudimentary sphinx autogen output now--make it not suck!

suggested by @florianfp

For ISONE get_load and get_lmp, get data from these excel files if no data is available from the regular ISONE API (eg if the date is before Aug 2014), or if the user requests hourly data (market='RTHR'):

http://www.iso-ne.com/isoexpress/web/reports/load-and-demand/-/tree/zone-info

The Alberta Electric System Operator (AESO) has legislative obligations to make their data public. You can get a sense for some of it at: http://ets.aeso.ca/

It's rough, but CSV data is available at: http://ets.aeso.ca/ets_web/ip/Market/Reports/CSDReportServlet?contentType=csv

We could easily add latest support for get_generation(...) using the fuels, get_trade(...) using Net Actual Interchange, and get_load(...) using Alberta Internal Load (AIL).

An AESOClient would provide value, since currently the EIAClient lists the AESO in its canada_mexico list, meaning that data is not supported.

Hi

I am using Pyiso to understand the CO2 associated to the production of electric energy. I would like to do this for California
This is the script I have

from pyiso import client_factory
import pandas as pd

caiso = client_factory("CAISO")
data_caiso = caiso.get_generation(latest = True)
df_caiso = pd.DataFrame(data_caiso)
print df_caiso

and this is the results I get

  ba_name freq  fuel_name    gen_MW market                  timestamp
0   CAISO  10m  renewable   1706.00   RT5M  2015-05-27 17:00:00+00:00
1   CAISO  10m      solar   5342.00   RT5M  2015-05-27 17:00:00+00:00
2   CAISO  10m       wind    939.00   RT5M  2015-05-27 17:00:00+00:00
3   CAISO  10m      other  21534.32   RT5M  2015-05-27 17:00:00+00:00

Unfortunately I'd like to have more information and not only other in terms of fuel description.

I tried with other ISOs ("ISONE", "ERCOT" and "PJM") in order to see if they had more info.
Here's an example of script

from pyiso import client_factory
import pandas as pd
from datetime import datetime

isone = client_factory("ISONE")
data_isone = isone.get_generation(latest = True)
df_isone = pd.DataFrame(data_isone)
print df_isone

however I get this error

No handlers could be found for logger "pyiso.base"
Traceback (most recent call last):
  File "get_carbon_footprint.py", line 10, in <module>
    data_isone = isone.get_generation(latest = True)
  File "/usr/local/lib/python2.7/dist-packages/pyiso/isone.py", line 63, in get_generation
    data = self.fetch_data(endpoint, self.auth)
  File "/usr/local/lib/python2.7/dist-packages/pyiso/isone.py", line 35, in fetch_data
    return response.json()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 819, in json
    return json.loads(self.text, **kwargs)
  File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 413, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/dist-packages/simplejson/decoder.py", line 420, in raw_decode
    raise JSONDecodeError("No JSON object could be decoded", s, idx)
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

any suggestions?

Thanks

Users should not be confused when ISONE tests fail because they don't have an account.

The link to get an account is http://www.iso-ne.com/participate/applications-status-changes/access-software-systems#data-feeds

CAISO now limits access to 1 request every 5 seconds:
http://www.caiso.com/Documents/OASIS_ReleaseNotes_v1_1Clean_Fall2015Release.pdf

Requests should retry with a 429 status code

EIA Electric System Operating Data has an API for the hourly load (actual and forecast), generation, and imports/exports from all balancing authorities in the lower 48 states! This is an amazing complement to the data provided from the ISOs themselves.

To implement, create a file pyiso/eia_esod.py and add a new client class with the standard methods get_generation, get_load, and get_trade. Each method should take a balancing authority abbreviation as an argument, similar to how some get_lmp methods take a node name. All returned data should be in the RTHR and DAHR markets, and generation should be fuel other.

The query browser will be useful for writing the parsers, eg http://www.eia.gov/opendata/qb.cfm?category=2122628&sdid=EBA.CISO-ALL.D.H

Also check out their amazing docs! http://www.eia.gov/beta/realtime_grid/docs/UserGuideAndKnownIssues.pdf

You are using
lxml = 3.6.1
as package prerequisite, but 3.6.1 has some problems on windows (compile error). I changed your source code and tried 3.6.4 and it worked.
Therefore I suggest maybe we could change the requirement to
lxml>=3.6.1

As requested by @jinhw1989 in issue #6.

When trying to run this script

from pyiso import client_factory
import pandas as pd

isone = client_factory("ISONE")

data_isone = isone.get_generation(yestarday = True)

I get this error

Traceback (most recent call last):
  File "get_carbon_footprint.py", line 10, in <module>
    data_isone = isone.get_generation(yestarday = True)
  File "/usr/local/lib/python2.7/dist-packages/pyiso/isone.py", line 51, in get_generation
    raise ValueError('Either latest must be True, or start_at and end_at must both be provided.')
ValueError: Either latest must be True, or start_at and end_at must both be provided.

if trying

from pyiso import caiso
ca = caiso.CAISOClient()
ca.get_generation(yesterday = True)

I get

/usr/local/lib/python2.7/dist-packages/pandas/index.so in pandas.index.IndexEngine.get_loc (pandas/index.c:3824)()

/usr/local/lib/python2.7/dist-packages/pandas/index.so in pandas.index.IndexEngine.get_loc (pandas/index.c:3704)()

/usr/local/lib/python2.7/dist-packages/pandas/hashtable.so in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12349)()

/usr/local/lib/python2.7/dist-packages/pandas/hashtable.so in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12300)()

KeyError: 'Hour'

• prices (by node), DAM / RTM
• AS prices

It looks like intro.rst should contain a list of balancing authorities:

Specifically, here are the included balancing authorities and their respective data sources:

But the list is nowhere to be found. :) I'm trying to add US statistics to electricitymap, so knowing this would be super helpful.

It's not much, but support for latest=True could be added for get_load(...) using SaskPower's system load feed here:

http://www.saskpower.com/spfeeds.nsf/671fbe1718238db706257fd9006b36d9?OpenView

The sysloadJSON feed contains the current system load the timestamp of the information.

This new method should be analogous to get_load (https://github.com/WattTime/pyiso/blob/master/pyiso/caiso.py#L70).

It should call these methods in this order:

1. handle_options
2. construct_oasis_payload('xx')
3. fetch_oasis
4. parse_oasis_lmp

All the other stuff should be essentially copy and paste.

Tests should be added to a new test file, tests/test_lmp.py

I want to be able to pull 15-minute LMP data. I am able to modify the code to accomplish this, but I recommend adding it as a feature. Here is the CAISO OASIS URL that accomplishes my task:

http://oasis.caiso.com/oasisapi/SingleZip?queryname=PRC_RTPD_LMP&startdatetime=20150901T00:00-0700&enddatetime=20151001T00:00-0700&market_run_id=RTPD&version=1&resultformat=6

The key differences from the 5-minute market are "PRC_RTPD_LMP" and "RTPD"

Daylight savings time switchover issue

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 30, in get_load
data = c.get_load(**kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 142, in get_load
df.index = self.utcify_index(df.index)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/base.py", line 335, in utcify_index
aware_utc_index = aware_local_index.tz_convert('UTC')
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/tseries/index.py", line 1502, in tz_convert
raise TypeError('Cannot convert tz-naive timestamps, use '
TypeError: Cannot convert tz-naive timestamps, use tz_localize to localize

This method may be most analogous to parse_oasis_demand_forecast (https://github.com/WattTime/pyiso/blob/master/pyiso/caiso.py#L377).

It should take a raw_data argument that is the Beautiful Soup output from fetch_oasis.

It should return a list of dictionaries.

Each dictionary should have these keys:

• timestamp
• freq
• market
• ba_name

It should also have one or more keys that are relevant to ancillary services data.

Tests should be added to tests/test_caiso.py

{"market": "RTHR", "latest": true} 2016-06-13 13:05:00.647676+00:00

  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 30, in get_load
    data = c.get_load(**kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nvenergy.py", line 41, in get_load
    df, mode = self.fetch_df(this_date)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nvenergy.py", line 123, in fetch_df
    dfs = pd.read_html(response.content, index_col=0)
AttributeError: 'NoneType' object has no attribute 'content'

Traceback:

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 83, in get_trade
    df = self.get_any('ExternalLimitsFlows', self.parse_trade)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 144, in get_any
    dates_list = self.dates()
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/base.py", line 458, in dates
    local_now = self.local_now()
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/base.py", line 449, in local_now
    return pytz.utc.localize(datetime.utcnow()).astimezone(pytz.timezone(self.TZ_NAME))
AttributeError: 'America/New_York' object has no attribute 'fromutc'

At 2014-10-27 13:48:

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 15, in get_generation
data = c.get_generation(*_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 122, in get_generation
df = self.fetcher()()
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 73, in fetch_recent
index_col=0, parse_dates=True, usecols=cols)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/base.py", line 296, in parse_to_df
df = pd.read_csv(filelike, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 400, in parser_f
return _read(filepath_or_buffer, kwds)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 198, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 479, in init
self._make_engine(self.engine)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 586, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 957, in init
self._reader = _parser.TextReader(src, **kwds)
File "parser.pyx", line 477, in pandas.parser.TextReader.cinit (pandas/parser.c:4434)
File "parser.pyx", line 686, in pandas.parser.TextReader._get_header (pandas/parser.c:6545)
ValueError: Passed header names mismatches usecols

Revert portions of 8409031

http://www.nyiso.com/public/markets_operations/market_data/graphs/index.jsp
click on Real-Time Fuel Mix

csvs:

• daily like http://mis.nyiso.com/public/csv/rtfuelmix/20160119rtfuelmix.csv
• zipped monthly like http://mis.nyiso.com/public/csv/rtfuelmix/20160101rtfuelmix_csv.zip

get_lmp could be returning congestion price, etc in addition to LMP. Add a kwarg all_data=False that, if set to True, returns all the data types instead of just LMP.

Observed during 10:10-10:20am Pacific time on April 22, 2016:

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 67, in get_load
    df = self.get_any('pal', self.parse_load_rtm)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 148, in get_any
    for csv in self.fetch_csvs(date, label):
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 180, in fetch_csvs
    if response.status_code == 200:
AttributeError: 'NoneType' object has no attribute 'status_code'

This caused a type error to occur in code at work that assumed the return value is always a DataFrame instance, and not the DataFrame class object.

In attempting to use pyiso to get CAISO data, I receive the error below. Using Python 3.5 on ubuntu. It appears that _base.py should now be just base.py

https://groups.google.com/forum/#!topic/beautifulsoup/JO_9ufJB4UM

`---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
----> 1 caiso = client_factory('CAISO')

/home/dan/anaconda3/lib/python3.5/site-packages/pyiso/init.py in client_factory(client_name, **kwargs)
67 # load
68 try:
---> 69 mod = imp.load_module(module_name, fp, pathname, description)
70 finally:
71 # Since we may exit via an exception, close fp explicitly.

/home/dan/anaconda3/lib/python3.5/imp.py in load_module(name, file, filename, details)
232 raise ValueError(msg)
233 elif type_ == PY_SOURCE:
--> 234 return load_source(name, filename, file)
235 elif type_ == PY_COMPILED:
236 return load_compiled(name, filename, file)

/home/dan/anaconda3/lib/python3.5/imp.py in load_source(name, pathname, file)
170 module = _exec(spec, sys.modules[name])
171 else:
--> 172 module = _load(spec)
173 # To allow reloading to potentially work, use a non-hacked loader which
174 # won't rely on a now-closed file object.

/home/dan/anaconda3/lib/python3.5/importlib/_bootstrap.py in _load(spec)

/home/dan/anaconda3/lib/python3.5/importlib/_bootstrap.py in _load_unlocked(spec)

/home/dan/anaconda3/lib/python3.5/importlib/_bootstrap_external.py in exec_module(self, module)

/home/dan/anaconda3/lib/python3.5/importlib/_bootstrap.py in _call_with_frames_removed(f, _args, *_kwds)

/home/dan/anaconda3/lib/python3.5/site-packages/pyiso/caiso.py in ()
4 import copy
5 import re
----> 6 from bs4 import BeautifulSoup
7 from io import BytesIO, StringIO
8 import pandas as pd

/home/dan/anaconda3/lib/python3.5/site-packages/bs4/init.py in ()
28 import warnings
29
---> 30 from .builder import builder_registry, ParserRejectedMarkup
31 from .dammit import UnicodeDammit
32 from .element import (

/home/dan/anaconda3/lib/python3.5/site-packages/bs4/builder/init.py in ()
312 register_treebuilders_from(_htmlparser)
313 try:
--> 314 from . import _html5lib
315 register_treebuilders_from(_html5lib)
316 except ImportError:

/home/dan/anaconda3/lib/python3.5/site-packages/bs4/builder/_html5lib.py in ()
68
69
---> 70 class TreeBuilderForHtml5lib(html5lib.treebuilders._base.TreeBuilder):
71
72 def init(self, soup, namespaceHTMLElements):

AttributeError: module 'html5lib.treebuilders' has no attribute '_base'

`

SVERIClient.get_generation(latest=True) returns no data after switching to the new url.

This new method should be analogous to get_load (https://github.com/WattTime/pyiso/blob/master/pyiso/caiso.py#L70).

It should call these methods in this order:

1. handle_options
2. construct_oasis_payload('xx')
3. fetch_oasis
4. parse_oasis_ancillary

All the other stuff should be essentially copy and paste.

Tests should be added to a new test file, tests/test_ancillary.py

at https://github.com/WattTime/pyiso/blob/master/pyiso/bpa.py#L31

don't allow this error to be raised:

Task pyiso.tasks.get_generation with id e5eed057-fccc-4523-a149-c8aa58a0a376 raised exception:
'TypeError("'NoneType' object is not iterable",)'

Task was called with args: ('BPA',) kwargs: {'latest': True, 'market': 'RT5M'}.

The contents of the full traceback was:

Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
R = retval = fun(_args, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.16.0.12/newrelic/hooks/application_celery.py", line 66, in wrapper
return wrapped(_args, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 416, in protected_call
return self.run(_args, _kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 15, in get_generation
data = c.get_generation(_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 118, in get_generation
raw_data += self._request_tsv(url, latest, start_at, end_at)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 43, in _request_tsv
vals = self._preprocess(row)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 31, in _preprocess
vals[0] = self._utcify(dateutil_parse(vals[0]))
File "/app/.heroku/python/lib/python2.7/site-packages/dateutil/parser.py", line 748, in parse
return DEFAULTPARSER.parse(timestr, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/dateutil/parser.py", line 310, in parse
res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

Traceback:

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 67, in get_load
    df = self.get_any('pal', self.parse_load_rtm)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/nyiso.py", line 155, in get_any
    df = pd.concat(pieces)
  File "/app/.heroku/python/lib/python2.7/site-packages/pandas/tools/merge.py", line 812, in concat
    copy=copy)
  File "/app/.heroku/python/lib/python2.7/site-packages/pandas/tools/merge.py", line 845, in __init__
    raise ValueError('No objects to concatenate')
ValueError: No objects to concatenate

File "pandas/tslib.pyx", line 4352, in pandas.tslib.tz_localize_to_utc (pandas/tslib.c:75236)
pytz.exceptions.AmbiguousTimeError: There are 29 dst switches when there should only be 1.

I received the following error(s) when trying to pull PJM forecasts:

In [78]: pjm1.get_load(latest=True, forecast=True)
C:\Anaconda\lib\site-packages\requests\packages\urllib3\connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):

  File "<ipython-input-78-e344816073a0>", line 1, in <module>
    pjm1.get_load(latest=True, forecast=True)

  File "C:\Anaconda\lib\site-packages\pyiso\pjm.py", line 151, in get_load
    df = self.fetch_edata_series('ForecastedLoadHistory', {'name': 'PJM RTO Total'})

  File "C:\Anaconda\lib\site-packages\pyiso\pjm.py", line 86, in fetch_edata_series
    dfs = pd.read_html(response.content, header=0, index_col=0)

  File "C:\Anaconda\lib\site-packages\pandas\io\html.py", line 896, in read_html
    keep_default_na=keep_default_na)

  File "C:\Anaconda\lib\site-packages\pandas\io\html.py", line 733, in _parse
    raise_with_traceback(retained)

  File "C:\Anaconda\lib\site-packages\pandas\io\html.py", line 727, in _parse
    tables = p.parse_tables()

  File "C:\Anaconda\lib\site-packages\pandas\io\html.py", line 196, in parse_tables
    tables = self._parse_tables(self._build_doc(), self.match, self.attrs)

  File "C:\Anaconda\lib\site-packages\pandas\io\html.py", line 451, in _build_doc
    from_encoding=self.encoding)

  File "C:\Anaconda\lib\site-packages\bs4\__init__.py", line 228, in __init__
    self._feed()

  File "C:\Anaconda\lib\site-packages\bs4\__init__.py", line 289, in _feed
    self.builder.feed(self.markup)

  File "C:\Anaconda\lib\site-packages\bs4\builder\_html5lib.py", line 68, in feed
    doc = parser.parse(markup, **extra_kwargs)

  File "C:\Anaconda\lib\site-packages\html5lib\html5parser.py", line 235, in parse
    self._parse(stream, False, None, *args, **kwargs)

  File "C:\Anaconda\lib\site-packages\html5lib\html5parser.py", line 85, in _parse
    self.tokenizer = _tokenizer.HTMLTokenizer(stream, parser=self, **kwargs)

  File "C:\Anaconda\lib\site-packages\html5lib\_tokenizer.py", line 36, in __init__
    self.stream = HTMLInputStream(stream, **kwargs)

  File "C:\Anaconda\lib\site-packages\html5lib\_inputstream.py", line 151, in HTMLInputStream
    return HTMLBinaryInputStream(source, **kwargs)

TypeError: __init__() got an unexpected keyword argument 'encoding'

It looks like our old friend bs4 and html5lib that pandas.io is using again...

Lucas

Seems like when trying to call caiso.get_load, caiso,get_lmp and caiso.get_trade with latest=True results in a TypeError.

Should be an easy fix, changed datetime.utcnow() to datetime.utcnow().strftime(self.oasis_request_time_format) and it worked.

Changes should be done on lines 119, 151 and 415 in CAISO if I'm not mistaken.

Error at "latest": true, "market": "RT5M", 2016-06-06 20:43:03.894154+00:00

  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 30, in get_load
    data = c.get_load(**kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 145, in get_load
    df = self.fetcher()()
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 73, in fetch_recent
    df = self.parse_to_df(response.text, skiprows=6, header=0, delimiter='\t',
AttributeError: 'NoneType' object has no attribute 'text'

The docs have a link that is supposed to go to the Watttime Impact API. Doesn't look like that GitHub repo exists.

Far down the call chain, one value in CAISOClient.get_generation(yesterday=True) returned the string #NAME when it should have returned a float:

ValueError: could not convert string to float: #NAME?

At 2014-10-27 18:23:

File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 30, in get_load
data = c.get_load(*_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 139, in get_load
df = self.fetcher()()
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/bpa.py", line 73, in fetch_recent
index_col=0, parse_dates=True, usecols=cols)
File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/base.py", line 296, in parse_to_df
df = pd.read_csv(filelike, *_kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 400, in parser_f
return _read(filepath_or_buffer, kwds)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 198, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 479, in init
self._make_engine(self.engine)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 586, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/app/.heroku/python/lib/python2.7/site-packages/pandas/io/parsers.py", line 957, in init
self._reader = _parser.TextReader(src, **kwds)
File "parser.pyx", line 477, in pandas.parser.TextReader.cinit (pandas/parser.c:4434)
File "parser.pyx", line 599, in pandas.parser.TextReader._get_header (pandas/parser.c:5831)
CParserError: Passed header=0 but only 0 lines in file

The Transmissions and System Operator division of NB Power makes some of its data publicly available online. Data for latest=True calls for get_load(...) and get_trade(...) could be scraped from its realtime System Information page at: https://tso.nbpower.com/Public/en/SystemInformation_realtime.asp

Forecast information for get_load(...) is available from: https://tso.nbpower.com/Public/en/op/market/report_list.aspx?path=\load%20forecast\hourly

Hi

when running the following script

from datetime import datetime
from pyiso import caiso
ca = caiso.CAISOClient()
ca.get_generation(start_at = datetime(2015,1,1), end_at = datetime(2015,1,4))

I get an error similar to #23

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-13-e26d8b8a16f3> in <module>()
----> 1 ca.get_generation(start_at = datetime(2015,1,1), end_at = datetime(2015,1,4))

/usr/local/lib/python2.7/dist-packages/pyiso/caiso.pyc in get_generation(self, latest, yesterday, start_at, end_at, **kwargs)
     79             return self._generation_forecast()
     80         else:
---> 81             return self._generation_historical()
     82 
     83     def get_load(self, latest=False,

/usr/local/lib/python2.7/dist-packages/pyiso/caiso.pyc in _generation_historical(self)
    384 
    385                 # combine date with hours to index
--> 386                 indexed = self.set_dt_index(df, this_date, df['Hour'])
    387 
    388                 # original header is fuel names

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in __getitem__(self, key)
   1789             return self._getitem_multilevel(key)
   1790         else:
-> 1791             return self._getitem_column(key)
   1792 
   1793     def _getitem_column(self, key):

/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in _getitem_column(self, key)
   1796         # get column
   1797         if self.columns.is_unique:
-> 1798             return self._get_item_cache(key)
   1799 
   1800         # duplicate columns & possible reduce dimensionaility

/usr/local/lib/python2.7/dist-packages/pandas/core/generic.pyc in _get_item_cache(self, item)
   1082         res = cache.get(item)
   1083         if res is None:
-> 1084             values = self._data.get(item)
   1085             res = self._box_item_values(item, values)
   1086             cache[item] = res

/usr/local/lib/python2.7/dist-packages/pandas/core/internals.pyc in get(self, item, fastpath)
   2849 
   2850             if not isnull(item):
-> 2851                 loc = self.items.get_loc(item)
   2852             else:
   2853                 indexer = np.arange(len(self.items))[isnull(self.items)]

/usr/local/lib/python2.7/dist-packages/pandas/core/index.pyc in get_loc(self, key, method)
   1576         """
   1577         if method is None:
-> 1578             return self._engine.get_loc(_values_from_object(key))
   1579 
   1580         indexer = self.get_indexer([key], method=method)

/usr/local/lib/python2.7/dist-packages/pandas/index.so in pandas.index.IndexEngine.get_loc (pandas/index.c:3824)()

/usr/local/lib/python2.7/dist-packages/pandas/index.so in pandas.index.IndexEngine.get_loc (pandas/index.c:3704)()

/usr/local/lib/python2.7/dist-packages/pandas/hashtable.so in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12349)()

/usr/local/lib/python2.7/dist-packages/pandas/hashtable.so in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12300)()

KeyError: 'Hour'

Error for ISONEClient.get_generation(latest=True) at 2016-06-01 05:08:00.162154+00:00

  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/tasks.py", line 15, in get_generation
    data = c.get_generation(**kwargs)
  File "/app/.heroku/python/lib/python2.7/site-packages/pyiso/isone.py", line 61, in get_generation
    raw_data += data['GenFuelMixes']['GenFuelMix']
KeyError: 'GenFuelMixes'

When start_at and end_at are provided, times after end_at are returned too.

sample args:
'start_at': '2015-12-29 05:00:00+00:00', 'end_at': '2015-12-29 05:10:00+00:00', 'market': 'RT5M', 'latest': False