Code Monkey home page Code Monkey logo

howto-pandas's Introduction

Build status Build Status

HowTo Solcast - Python, Pandas, MatPlotLib, and PvLib

NOTE: You will need to register your user account to obtain an API key https://solcast.com.au/api/register. Without an API key you will not be able to successfully obtain valid API results.

Windows setup Solcast API Key

Setup System/User environment variable. Details on advanced editing StackOverflow superuser walkthrough

WinKey + R

Copy and Paste the following text to the Open: input text box for the Run Dialog

%windir%\System32\rundll32.exe sysdm.cpl,EditEnvironmentVariables

That will present this screen

Run Windows Environment Editor

Add a user or system environment variable to hold the Solcast API key. User environment variables will only be available to your particular user, system environment variables are shared for all users on the system

Add Windows User Solcast API key

After you have added the environment variable you will see the key listed in the current variables

Added Windows User Solcast API key

NOTE: To reference this key you will need to reopen your shell prompt to read these variables again from the system (cmd, command.com, powershell, etc)

Linux / mac OS

Open a terminal prompt

  • mac OS: Spotlight search for terminal
  • Linux: Open bash
nano .bash_profile

If you do not have nano it is simpler text editor than vi. Use your package manager to download and install or use vi. The preferred package manager for mac OS is Homebrew and once installed on your system you can issue similar commands to Linux apt-get and yum with the brew package manager.

Add the Solcast API Key to your user profile variables.

Added mac OS User Solcast API key

This how to demonstrates working with using the Solcast API to load pandas data frames and then plot different chart data. Tested and built with Python 3.6.2

This is a demonstration project that shows an example of how to use the solcast-py library to load a Pandas data frame and then display a single field graphically using MatPlotLib. A familiarity with Python is not required.

Clone or download this project

Install all the require dependencies

If you are not familiar with the python module pip, please refer to this PIP for Python

pip install --upgrade -r requirements.txt

Alternatively you may need to use sudo the -H instructs sudo to install dependencies to local users home directory instead of system

sudo -H pip install --upgrade -r requirements.txt

If you are receiving an error when installing the netcdf4 package on Mac such as:

ValueError: did not find netCDF version 4 headers

or

ValueError: did not find hdf5 version 4 headers

These two packages need to be installed onto your Mac with homebrew. Click this link to install homebrew here. These packages are located in the homebrew/science section. You will need to "tap" this section to access it. More information is found here.

NOTE: Installing hdf5 on mac will take a long time. Give it time.

Latitude and Longitude

  • First as stated above you will need an API key to make valid API requests to the Solcast system.
  • Second for all current library calls you will need a valid Lat/Lng coordinate in the EPSG:4326 format. If you are familiar with modern web maps you most likely have used the expected format or a decimal point that expresses a position on the Earth.

Clarification as I often forget the coordinate planes of Latitude and Longitude along with bounds. Lat/Lng

Credits - Learner.org

The Solcast API expects West for Longitude and South for Latitude to be expressed as a negative numbers

Example Locations on the Globe

Name Latitude Longitude
Sydney, Australia -33.865143 151.209900
Mumbai, India โ€Ž 19.228825 72.854118
Tokyo, Japan 35.6895 139.69171
Paris, France 48.864716 2.349014
Los Angeles, USA 34.052235 -118.243683

Power/Radiation library method interaction

In the file main.py is a minimal set of commands to obtain the following charts. The break in the continuity between the Estimated Actuals and the Forecast predicted values on the time scale is due to that the time period falls between an in place Estimated Actual result being recorded and a next period available Forecast. Currently all period results are broken into 30 minute timestamp periods. This howto uses the period_end fields as the indexed field for the DataFrames (x - axis on the following charts)

Within this project workspace is folder solcast_frames that provides some solcast-py library helper translators to pandas DataFrames. In the main.py file these helper classes are referenced below.

# This line is only for using the matplotlib
import matplotlib.pyplot as plt
from solcast_frames.latlng import LatLng
from solcast_frames.radiationframehandler import RadiationFrameHandler
from solcast_frames.powerframehandler import PowerFrameHandler

plt.interactive(False) # Turn this off to create plots

These following python classes are now available to use LatLng, RadiationFrameHandler, PowerFrameHandler

First you should create a location to obtain results for solcast forecasts. Example below.

location = LatLng(lat=-33.86785, lng=151.215256, name="Sydney", tag="No wild Koalas", timezone="Australia/Sydney")

The LatLng helper class if no keyword arguments are supplied will set your location to (0,0) on the globe

The LatLng class contains a desc() method to give you a string output of its current details if needed i.e. print(location.desc())

To obtain and plot the current estimates and forecast from the Solcast API for PvPower use the following code

# Plot the power `pv_estimate`
# Required fields latlng valid position, capacity as integer > 0
# Adding in an optional keyword argument of an azimuth of 0
# The following optional keyword arguments are recognized by the solcast-py library
# azimuth | range: [-180 to 180] default: 0 in Southern Hemisphere, 180 in Northern Hemisphere
# tilt | range: [0 to 90] | default: 23
# install_date | format: yyyyMMdd Will be ignored if a loss_factor is supplied
# latest | [True, False] | default False
power_estimated_actuals = PowerFrameHandler.estimated_actuals(location, 5000, azimuth=0)
power_estimated_actuals.pv_estimate.plot()

# Plot the power `pv_estimate`
# Required fields latlng valid position, capacity as integer > 0
# The following optional keyword arguments are recognized by the solcast-py library
# azimuth | range: [-180 to 180] default: 0 in Southern Hemisphere, 180 in Northern Hemisphere
# tilt | range: [0 to 90] | default: 23
# install_date | format: yyyyMMdd Will be ignored if a loss_factor is supplied
# loss_factor | [0 to 1] | default 0.9
fx_solcast_power = PowerFrameHandler.forecast(location, 5000)
fx_solcast_power.pv_estimate.plot()
plt.show()

Power Estimated Actuals with Forecast (pv_estimate over period_end) Power Graph

To obtain and plot the current estimates and forecast from the Solcast API for Radiation use the following code

# Plot the radiation `ghi` field
# Required fields latlng valid position
# The following optional keyword arguments are recognized by the solcast-py library
# latest | [True, False] | default False
radiation_estimated_actuals = RadiationFrameHandler.estimated_actuals(location)
radiation_estimated_actuals.ghi.plot()
# Plot the radiation `ghi` field
# Required fields latlng valid position
fx_solcast_radiation = RadiationFrameHandler.forecast(location)
fx_solcast_radiation.ghi.plot()
plt.show()

Radiation Estimated Actuals with Forecast (ghi over period_end) Radiation Graph

Details of Solcast Python Client Library Solcast Python API client library

The Solcast Python library is designed as a synchronous web service and does not include the graphing or advanced data options of Pandas/MatPlotLib.

The PowerFrameHandler.py and RadiationFrameHandler.py classes in this howto under solcast_frames example convert the list value resultsets into pandas dataframes aligned and indexed by the period_end DateTime field (all datetimes are expressed in UTC timezone).

Example data results for Power DataFrames with columns available:

Power DataType

Power DataType

Power DataFrame contents

                            period  pv_estimate
period_end                                     
2017-10-02 19:30:00+00:00 00:30:00     0.000000
2017-10-02 20:00:00+00:00 00:30:00    27.628064
2017-10-02 20:30:00+00:00 00:30:00   243.625359

Example data results for Radiation DataFrames with columns available:

Radiation Estimated Actuals DataType

Radiation Estimated Actuals DataType

Estimated Actuals Radiation DataFrame contents

                           cloud_opacity  dhi  dni  ebh  ghi   period
period_end                                                           
2017-10-02 13:00:00+00:00             26    0    0    0    0 00:30:00
2017-10-02 12:30:00+00:00             31    0    0    0    0 00:30:00
2017-10-02 12:00:00+00:00              0    0    0    0    0 00:30:00

Radiation Forecasts DataType

Radiation Forecasts DataType

Forecast Radiation DataFrame contents

                           air_temp  azimuth  cloud_opacity  dhi  dni  dni10  dni90  ebh  ghi  ghi10  ghi90   period  zenith  
period_end                                                                                                                    
2017-10-02 19:30:00+00:00        17      -97             17    1    0      0      0    0    1      1      1 00:30:00      94  
2017-10-02 20:00:00+00:00        17      -93             33   13   10      4     27    2   15     12     19 00:30:00      88  
2017-10-02 20:30:00+00:00        17      -88              7   45  196     43    275   33   79     57     86 00:30:00      82

Integrating with pvlib

In the file main.py is a minimal set of commands to obtain the following charts. The pvlib library is a powerful open source tool that is used in computing and forecasting photovoltaic solar cell arrays. There are numerous options and details that can be configured, the following example compares pvlib forecasts with Solcast forecasts and how to obtain datasets that can be compared and graphed.

# This line is only for using the matplotlib
import matplotlib.pyplot as plt
from solcast_frames.latlng import LatLng
from solcast_frames.radiationframehandler import RadiationFrameHandler
import timeit
import pandas as pd
import datetime
# import pvlib forecast models
from pvlib.forecast import GFS, NAM, NDFD, HRRR, RAP

plt.interactive(False) # Turn this off to create plots

#
# Following code is from http://pvlib-python.readthedocs.io/en/latest/forecasts.html
#

# Changing the location pvlib default model uses Tucson for examples
location = LatLng(lat=32.2, lng=-110.9, name="Tucson", tag="Cactus Land", timezone="US/Arizona")
print(location.desc())

start_time = timeit.default_timer()
radiationForecast = RadiationFrameHandler.forecast(location)
elapsed = timeit.default_timer() - start_time
print("Solcast Radiation Forecast Location: %s Time: %s (seconds)" % (location.name, '%.6f' % elapsed))

# specify time range with timezone
start = pd.Timestamp(datetime.date.today(), tz=location.timezone)
end = start + pd.Timedelta(days=7)

start_time = timeit.default_timer()
# fx is a common abbreviation for forecast
fx_model = GFS() # From Forecast models http://pvlib-python.readthedocs.io/en/latest/api.html#forecast-models
fx_data = fx_model.get_processed_data(location.lat, location.lng, start, end)
elapsed = timeit.default_timer() - start_time
print("pvlib (GFS) Radiation Forecast Location: %s Time: %s (seconds)" % (location.name, '%.6f' % elapsed))


plt.plot(fx_data.ghi, label="ghi - PVLIB (GFS)")
plt.plot(radiationForecast.ghi, label="ghi - Solcast", linestyle='dashdot')

plt.plot(fx_data.dhi, label="dhi - PVLIB (GFS)")
plt.plot(radiationForecast.dhi, label="dhi - Solcast", linestyle='dashdot')

plt.plot(fx_data.dni, label="dni - PVLIB (GFS)")
plt.plot(radiationForecast.dni, label="dni - Solcast", linestyle='dashdot')

plt.legend()
plt.show()

Radiation Forecast pvlib vs Solcast ghi dhi, dni Radiation Compare pvlib/solcast

howto-pandas's People

Contributors

siliconrob avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

howto-pandas's Issues

Add travis ci or circle ci build status

Have the appveyor windows build status ready and would like some help with getting travis ci or equivalent to work. Current problem I am having is the numpy+mkl requirement from the requirements.txt file so anyone with experience on scipy and linux or how to do alternate requirements or setup the travis build to be conditional.

``missing 1 required argument: 'api_key'`` while experimenting basic API call

Hi,
Following the tutorial, I am using Ubuntu, exported the SOLCAST_API_KEY in shell, and in the python environment of execution of the script, if this makes any difference.

I encountred the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-16263a0dce87> in <module>
      7 # install_date | format: yyyyMMdd Will be ignored if a loss_factor is supplied
      8 # latest | [True, False] | default False
----> 9 power_estimated_actuals = PowerFrameHandler.estimated_actuals(location, 5000, azimuth=0)
     10 power_estimated_actuals.pv_estimate.plot()
     11 

~/Bureau/Untitled Folder 1/solcast_frames/powerframehandler.py in estimated_actuals(lat_Lng, capacity, **kwargs)
     19         :rtype: pd.DataFrame
     20         """
---> 21         data = sc.PvPowerEstimatedActuals(lat_Lng.lat, lat_Lng.lng, capacity, **kwargs).content["estimated_actuals"]
     22         new_frame = pd.DataFrame(data)
     23         new_frame.set_index(['period_end'], inplace=True)

~/.conda/envs/tf/lib/python3.6/site-packages/solcast/pv_power_estimated_actuals.py in __init__(self, latitude, longitude, capacity, *args, **kwargs)
     35             self.end_point = self.end_point + '/latest'
     36 
---> 37         self._get(*args, **kwargs)
     38 
     39         if self.ok:

~/.conda/envs/tf/lib/python3.6/site-packages/solcast/base.py in _get(self, api_key, **kwargs)
     23         if api_key == None:
     24             raise TypeError('{type}() missing 1 required argument: \'api_key\''\
---> 25                             .format(type=type(self)))
     26 
     27         logger = logging.getLogger()

TypeError: <class 'solcast.pv_power_estimated_actuals.PvPowerEstimatedActuals'>() missing 1 required argument: 'api_key'

Thank you.

Add radiation pytest for location

As the value returned from the Solcast API can vary dependent on weather conditions try to do the following

  • Look for a result set
  • Set your position to location that is dark when you make the request i.e. something like if the current UTC time is 14:00 then that is AEDT 1:00 so it will be dark at lat/lng โ€Ž-33.865143, 151.215256 (Sydney). This is a bit complicated as you will need to find dark time positions for every UTC zone or so or perhaps a minimum of 6 positions on the earth that will be in darkness at any UTC time.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.