Rooftop PV production forecast

This project supports an extensive set of production forecasts for PV rooftop installations. Various weather data sources, PV modeling algorithms and storage methods for results can be used.

The project has been developped on Python3 and runs on Raspberries and integrates with solaranzeige (see Solaranzeige Integration)

Generally, functionality is configured through a configuration file (default: .\config.ini, .\solcast_light_config.ini, a different location can be provided with -c command line option)

Two main scripts are provided:

The following description of the full script is relatively complex. Hence, this section describes the minimalisic needs to only run SolCast forecasting:

1. prepare Python to run the script:
   1. Basic Installation
   2. add some Python modules
2. update solcast_light_config.ini to your SolCast registration
   1. SolCast Configuration
   2. Influx Storage Configuration
   3. (depricated): SolCast Tuning Configuration
3. if integration with Solaranzeige is desired, read this
4. Install and run script

A couple of more options can be configured, but are left out of this brief description. They would become obvious when reading the full text below.

• Introduction
• SolCastLight: Minimalistic Installation
• Main Functionality and Configuration
  • Forecast Sources
  • Solcast Configuration
  • OWM configuration
  • MOSMIX configuration
  • Forecast models
    • Convert Weather Data to Irradation Data
    • Convert Irradiation Data to PV Output Power
  • Data Storage
    • SQLite Storage
    • Influx Storage
  • .csv File Storage
  • Solcast Tuning
  • Solaranzeige Integration
• Installation
  • The Basics
  • Minimal Requirements
  • Full Installation
  • Optional
• Running the Script
• To Do
• Disclaimer
• License

Table of contents generated with markdown-toc

The reminder of this ReadMe file is meant to describe the full script configuration.

.\config.ini is a configuration file parsed with python's configparser. Most importantly:

• items can be promoted to the [DEFAULT] section if same key = value pair is used in multiple sections
• inline comments are configured to start with #
• multi-line values are not allowed
• out-commented key = value pairs show the respective default options, which could be changed as needed

The config.ini file provided with the distribution contains a few site specific values, which cannot be pre-configured. The are shown as <xx>. This file should be read alongside the below text for best understanding.

[Forecasts]                       # enable / disable certain forecasts
    Solcast           = 1         
    OWM               = 0         # OpenWeatherMap.org
    MOSMIX_L          = 0         # single station file, updated every 6h
    MOSMIX_S          = 0         # all stations,        updated hourly download

Forecast Sources can be (dis-)enabled with 0 or 1. Any number of sources can be enabled simultaneously.

Depending on the data source, various forecast algorithms are available. The configuration happens in the respective sections described below.

[SolCast]
    resource_id       = <resource_id_from_solcast.com>
    # resource_id_2   = <second_resource_id_from_solcast.com>
    api_key           = <api_id_from_solcast.com>
    # post            = 0         # enable posting and tuning (depricated)
    # interval        = 60        # interval at which SolCast is read (during daylight only)
    Latitude          = 51.8
    Longitude         =  6.1

Solcast allows for the free registration of a residental rooftop PV installation of up to 1MWp and allows for up to 20 API calls/day. The registration process provides a 12-digit resource_id (xxxx-xxxx-xxxx-xxxx) and a 32 character API key.

To stay within the limits of 20 API calls/day, the API is only called with an interval = 60 minutes between sunrise and sunset only. That's the sole use of Latitude and Longitude parameters (which maybe better placed in [Default] section, if weather based forecasts, as described in the following sections, are also calculated)

Solcast directly provides a PV forecast (in kW) for 30min intervals, with 10% and 90% confidence level. Hence, no further modelling is needed.

Since tuning was depricated, Solcast allows the definition of a second rooftop site to support split-array setups. In such a situation, a resource_id and resource_id_2 can be provided. Both will be queried at the same times. Individual and total forecast results will be stored in the database(s)

[OpenWeatherMap]
    api_key           = <api_id_from_openweathermap.org>
    Irradiance        = all       # irrandiance model (for OWM)

OpenWeatherMap offers free access to their API to regularly download weather forecasts. The registration process provides a 32 character API key.

The weather forecast consists of approx. 10 parameters, including cloud coverage, which can be modelled to a PV forecast (see Forecast Models)

[DWD]
    DWD_URL_L         = https://opendata.dwd.de/weather/local_forecasts/mos/MOSMIX_L/single_stations/
    DWD_URL_S         = https://opendata.dwd.de/weather/local_forecasts/mos/MOSMIX_S/all_stations/kml/
    DWDStation        = <station_number>    # Station number
    Irradiance        = all       # irradiance model (for MOSMIX)
    storeKMZ          = 0         # store downloaded .kmz files (.kml compressed as .zip)
    # keepKMZ_S       = 0         # keep MOSMIX_S original file after downloading     

Unlike modern APIs, Deutscher Wetterdienst (DWD) allows only file download for what they call MOSMIX data. Hence, the software described here has to accomodate for the associated complications.

Two download schemes (as described above) exist. Keys DWD_URL_L and DWD_URL_S provide the respective stems of download links. The station abreviation needs be taken from their website

storeKMZ: The files downloaded are named *.kmz which is inadequate in two ways: First, the files are simple .zip files (so, why are they not called that way?) and second, a .zip file is meant to contain multiple files, which clearly the .kmz files never do. Hence, with storeKMZ = 1, downloaded data is stored in the more adequate .gz format. For MOSMIX_L, the downloaded files for the selected station are stored. For MOSMIX_S, an extract for the selected station is stored in a self-contained compressed .xml file. That format is very similar to MOSMIX_L files.

keepKMZ_S: in case of downloading the (huge) MOSMIX_S file, they can be stored by enabling this option. Note that approx. 900MByte/day of storage space will be consumed!

Data from OWM and MOSMIX do not directly contain PV output power. This needs be modelled using functionality provided by pvlib. Multiple modelling approaches are supported, selected by the Irradiance parameter seen above.

Essentially, the modelling consists of a two-step approach:

1. convert weather data to irradiation data (GHI, DNI, DHI)
2. convert such irradiation data into PV output power

Where needed, DHI is calculated from GHI and DNI using the fundamental equation DNI = (GHI - DHI)/cos(Z) where Z is the solar zenith angle (see eg. Best Practices Handbook)

Secondary weather parameters considered in above models include:

pvlib supports two modelling strategies for a PV system:

1. model system with actual component parameters based on a CEC database provided with pvlib
2. simplified PVWatts model

Both approaches are supported and selected based on Model

[PVSystem]
    # Model            = CEC      # modeling strategy for PV: 'CEC' or 'PVWatts' 
    # TemperatureModel = open_rack_glass_glass
    # clearsky_model   = simplified_solis
    
    # --------------------------- physical definition of PV System, using CEC database
    # based on .csv files at .../lib/python3.8/site-packages/pvlib/data, special characters to be replaced by '_'
    ModuleName        = LG_Electronics_Inc__LG325N1W_V5
    InverterName      = SMA_America__SB10000TL_US__240V_
    NumStrings        =   2       # number of strings 
    NumPanels         =  15       # number of panels per string
    
    # --------------------------- PVWatts definition
    InverterPower     = 10000     # name-plate inverter max. power
    NominalEfficiency = 0.965     # nominal European inverter efficiency
    SystemPower       =  9750     # system power [Wp]
    TemperatureCoeff  = -0.0036   # temperature coefficient (efficiency loss per 1C)

The .csv are stored whereever pvlib installs on your system. A good place to start searching is in /usr/local/lib/python3.7/dist-packages/pvlib/data or ~/.local/lib/python3.8/site-packages/pvlib/data. In that directory should be two files sam-library-cec-inverters-2019-03-05.csv and sam-library-cec-modules-2019-03-05.csv for inverters and modules respectively.

If the (default) CEC approach is used, the selected model should at a minimum match the nameplate power of the installed panels (eg. 325Wp). The selected inverter is uncritical as long as the nameplate power is same or higher as installed inverter (eg. 10kW) - the modeling of inverters is relatively poor in pvlib, considering only a NominalEfficency.

The PVWatts model considers the following inefficiencies (which is less than PVWatts defaults, which are meant to model monthly or annual output):

    pvwatts_losses   = { 'soiling'          : 0,   
                         'shading'          : 0, 
                         'snow'             : 0, 
                         'mismatch'         : 0, 
                         'wiring'           : 2, 
                         'connections'      : 0.5, 
                         'lid'              : 0, 
                         'nameplate_rating' : 0, 
                         'age'              : 0, 
                         'availability'     : 0 }

pvlib models panel temperature (and related efficiency loss) based on TemperatureModel and weather parameter temp_air.

clearsky_model is used for irradiation model clearsky. ineichen and simplified_solis are supported, haurwitz is not.

Both models also need basic parameters of the system location and orientation:

    Latitude          = 51.8
    Longitude         =  6.1
    Altitude          =  74       # altitude of system (above sea level)
    Tilt              =  30
    Azimuth           = 127       # 270=West, 180=South, 90=East

Since latitude and longitude parameters are also needed by Solcast to calculate sunrise and sunset, it is efficient to put these two parameters into the [Default] section of the configuration file.

Forecasting PV output power would be pointless, if the resulting data wouldn't be stored anywhere. The application supports two main storage models:

1. SQLite (file based relational database)
2. Influx

The following configuration parameters control what is stored where and can be configured separately in sections [SolCast], [OpenWeatherMap], [DWD] or commonly in section [Default] (0 = disable, 1 = enable)

[DBRepo]
    dbName  = pvforecasts.db      # SQLite database name (at 'storePath')

An SQLite database is dynamically created with above defined name at storePath. It will contain a (subset of) the following tables, depending on what models have been calculated:

All tables contain IssueTime (when forecast was issued) and PeriodEnd (end time of forecast period). Date from previous IssueTime are not deleted to allow analysis of accuracy of forecasts over different forecast horizons.

All times are in UTC.

Note that if the configuration file is changed on a running system, more or less data maybe calculated:

• newly needed tables are created on-the-fly
• dropped fields are simply left empty (which SQLite handles relatively efficiently)
• however, new fields are not added dynamically - it is advised to drop the old database in such cases, which causes dynamic creation of a new one.

[Influx]
    host              = <your_hostname>         # default: localhost
    # port            = 8086
    database          = <your_influx_db_name>   
    power_field       = PV.Gesamtleistung

This will create the following measurements (akin tables) in the defined Influx database:

where <model> refers to one of the irradiance models calculated

power_field will be discussed in Solcast Tuning below

The database must pre-exist in Influx. If it does not, the following manual operations can create it:

~ $ influx
> show databases
> create database <your_influx_db_name>
> show databases
> quit
~ $

Config sections [DWD] and [PVSystem] support an option storeCSV = 1 to store output in .csv files at storePath.

Depricated by SolCast Solcast previously allowed to post PV performance data to tune forecast to eg. local shadowing conditions, etc.

[SolCast]
    post = 1

in the configuration file. But of course, it requires that such performance data is available locally.

The script assumes that performance data is available in the same Influx database as configured for forecast data storage. Saying

[Influx]
    database    = <your_influx_db_name>
    power_field = PV.Gesamtleistung

assumes that <your_influx_db_name> contains a measurement (table) PV with a field Gesamtleistung which has regular recordings of the PV generated power.

It is assumed that

• this field has at least a time resolution of 5 minutes or less
• power is in W
• Influx stores times internally always as UTC (this is not actually an assumption, rather a fact, which the application storing power data must be aware of)

This application is designed to run seamlessly alongside solaranzeige. Hence, if installed on the same host, the [Influx] configuration section discussed in previous section may very well look like this:

[Influx]
    # host            = localhost             # default: localhost
    database          = solaranzeige   
    power_field       = PV.Gesamtleistung

This will add the discussed measurements to the solaranzeige database and make them immediatly available for display in Grafana.

The power_field is dependent on the inverter: Most have either a field Gesamtleistung or Leistung, reflecting the PV DC power.

Installation mainly ensures that all necessary python modules are available. We assume a Raspberry host here - although the instructions are probably quite generic.

It is assumed that Python 3.x is available and pandas, numpy installed. This can be checked with the following commands:

~ $ python3
>>> import pandas as pd
>>> import numpy as np
>>> pd.__version__
>>> np.__version__
>>> quit()
~ $

If errors are seen, checkout pandas installation instructions and use

sudo apt install python3-pandas

pandas versions from 1.1.2 and numpy 1.19.2 are known to work. Earlier versions might need an upgrade. Try the script and see what happens before you upgrade though. Some googling might be needed as on Raspberries the upgrade path is not always as linear.

In case Influx Storage is desired, but Influx is not yet available, installation instructions can be found here

sudo pip3 install pysolcast                  # enables access to SolCast
sudo pip3 install astral                     # provides sunrise, sunset
sudo pip3 install influxdb                   # provides access to InfluxDB

With this we are able to run SolCastLight.py, which is limited to supporting Solcast as the only forecast source

Other forecast sources require modelling a photovoltaic system, which is acheived using pvlib. Unfortunatly, this library is not always straight forward to install - especially on 32-bit OS such as Raspbian. The default install command

sudo pip3 install pvlib[optional]

will likely fail - if it succeeds, you are the lucky guy. But what if not?

Some parts of pvlib require nrel-pysam, which only runs on 64-bit versions of Python, which we don't have on Raspbian ... so we might get something like

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-7twe_oqh/nrel-pysam/

What worked for me was following instructions here and do:

sudo apt-get install libhdf5-dev
sudo apt-get install libhdf5-serial-dev      # seems redundant, might be skipped
sudo python3 -m pip install h5py
sudo apt-get install netcdf-bin libnetcdf-dev
sudo python3 -m pip install netcdf4          # take a while ...

# siphon and tables are also needed by pvlib ...
sudo pip3 install siphon
sudo pip3 install tables
# ... and now it should work
sudo pip3 install pvlib

Finally, we might need (if not already installed by default) elementpath to handle MOSMIX .xml files:

sudo pip3 install elementpath
sudo pip3 install beautifulsoup4

If SQLite storage is configured, you'll end up with an SQLite database which you might want inspect. A great way (but by far not the only one) to do that is with SQLite Browser

sudo apt-get install sqlitebrowser

After downloading the script from Github, into a directory of your choosing (eg. \home\pi\PV), you should have these files (and some more):

./PVForecasts.py
./SolCastLight.py
./config.ini
./solcast_light_config.ini
./PVForecast/*.py        # approx. 9 .py files

• update the config file (config.ini or solcast_light_config.ini, depending which version you want to run
• try it out ...: python3 PVForecast.py or python3 SolCastLight.py
• install it in cron

A typical crontab entry can look like so (assuming you have downloaded into \home\pi\PV):

0 * * * * cd /home/pi/PV && /usr/bin/python3 SolCastLight.py >> /home/pi/PV/err.txt 2>&1

which would run the script every hour. Replace SolCastLight.py with the PVFirecast.py to run the full script.

A great explanation of cron is here. Crontab entries are made with crontab -e and checked with crontab -l.

Note: The script doesn't do much in terms of housekeeping (eg., limit size of SQLite database or err.txt file used above to redirect error messages).

• PV system modeling (using config section [PVSystem]) does nothing to model systems with panels in multiple orientations. The basic structure is prepared to handle this, but it is not implemented as of now.

The software pulls weather data from various weather sources. It is the users responsability to adhere to the use conditions of these sources.

The author cannot provide any warranty concerning the availability, accessability or correctness of such weather data and/or the correct computation of derieved data for any specific use case or purpose.

Further warranty limitations are implied by the license

Distributed under the terms of the GNU General Public License v3.