green-software-foundation / carbon-aware-sdk Goto Github PK
View Code? Open in Web Editor NEWCarbon-Aware SDK
Home Page: https://carbon-aware-sdk.greensoftware.foundation/
License: MIT License
Carbon-Aware SDK
Home Page: https://carbon-aware-sdk.greensoftware.foundation/
License: MIT License
This feature contribution will add further WattTime plugin support, including Forcasting.
Adding first versions of Integration Tests for both JSON and WattTime plugins (aka dataSources), using WireMock for mocking WattTime.
Includes updates of file folders and structures to match dotNet standards.
All SDK interface methods MUST have automated test coverage (with related test data) to ensure quality of the SDK.
Build out a REST API that exposes the SDK.
Needs to be container based for easy deployment.
Needs to leverage command line configuration.
To help people get started, we would like to start create guided content through video and updated docs.
This would include (by order of priority):
Link to existing resources where applicable (especially regarding learning more about what other working groups have covered)
Rename projects to namespace/package names so they are easy to discover and have hierarchical package names i.e. CarbonAware.Plugins. vs CarbonAwarePlugin
As a Green Developer I want to have my development environment in a devcontainer so that I can have confidence start immediately without having to install anything
This feature contribution will add WattTime plugin support and various improvements to the Carbon Aware SDK WebAPI and CLI. The specific details have been discussed with UBS, GSF and Microsoft.
As a developer I want to be able to manage all text in a unified location so that I can update it easily
It will be easier for other libs or application to consume or implement the sdk by referencing with published nuget package.
The CLI works currently through a variety of command line parameters, and while this works for the current array of options, it is unlikely to cater for future needs.
With a dynamic plugin based architecture, plugins will require a variety of custom configurations that can not be predetermined by the command line.
To handle this we need to abstract this complexity of "how" the SDK is configured from the command line parameters.
No response
The carbone-sdk/getting started md files is missing some key information to be able to run the cli (and likely the webapi) such as :
CLI
N/A
There is a need to map a named location i.e. westus
for a datacentre, but this needs to be mapped to a geo-location or watttime region name or electricitymap etc to send to the live carbon data services for querying. This also needs an approach for mapping back in the other direction.
Example we need to be able to consistently map the following ...
azure dc name <-> wattime geolocation
aws dc name <-> wattime geolocation
gcp dc name <-> wattime geolocation
ntt dc name <-> wattime geolocation
... your own DC... <-> xx service geolocation
azure dc name <-> electricitymap geolocation
aws dc name <-> electricitymap geolocation
gcp dc name <-> electricitymap geolocation
All architecture decisions should have an architecture decision record. This is so we can track the rationale behind why we made decisions, but also give us a way to discuss and finalise on proposed architecture decisions.
Extract command line configuration into common so it can be leveraged by other consumers (including web api #20)
We are looking to build out a good demo in python for the SDK. Demo would be using the REST endpoint or the CLI, with the python doing [something pythony(TIM)]
[something pythony(TIM)] is yet to be defined. It could be a machine learning scenario, or maybe it is a common scenario in the python developer community that we think would benefit.
Skills needed:
If you are interested let us know below.
This feature contribution will add further WattTime data source support, including endpoints for batch historical forecast, batch historical multi-day actuals, and batch average carbon intensity.
Also added updates to readmes and docstrings to better reflect endpoints and various use cases. Includes a handful of bug fixes such as improving defaulting/validation logic, the rolling average code, and adding support for app insights instrumentation key.
With the plugin based architecture, the CLI no longer has a build depdendency on the BasicJsonPlugin project due to abstraction/IoC.
As a baseline the plugin should copy across to the CLI/plugins folder.
Sampe data is copied already due to this data being moved into the CA.Data.Sample project.
Documentation needs to be created for the new service configuration (carbon-aware.json).
The WattTime API can be used as one of the sources for determining carbon index of different regions. We will also need to later add support for other data-sources - e.g. Electricity map.
To use it with the CarbonAware SDK, we need to:
C# WattTime API development workflow:
The objective is to simplify adoption of the SDK by providing a website for documentation, demo, use cases, etc.
Proposal for solutions:
An initial version can be found in : https://green-software-foundation.github.io/carbon-aware-sdk/
We want to build out a web demo using the SDK. While we have some ideas on what to do, we're looking for:
Please reply here as to what you could provide, and we can start to shape it up!
Add full test converage for the SDK API
We need help writing the Carbon Aware SDK Story
Reply below and we can start kicking off streams
No response
[Bug]: Lowest returns a single result in the WebApi - this should be an array since multiple regions and times may match the lowest emissions. The decision of which region/time to use should sit with the consumer of the information.
For example, the lowest emissions may be now
and 1 hour from now
and the application to decide whether to use the soonest for expedience or delay an hour for better use of time.
WebAPI (Default)
No response
Set up github actions automated build agent
This feature contribution will add electricityMap plugin support and various improvements to the Carbon Aware SDK WebAPI and CLI. This plugin should be similar to Watttime plugin.
Electricity map supports more countries' data including many Asia such as India, China and others which Watttime doesn't support. (#67 for details)
As discussed in a weekly meeting (5th July), target to merge into v1.1.
The CloudProvider context is not something that is relevant for users of a product and specifically web users. I'd like to provide a way to submit a latitude and longitude value directly to the Carbon Aware SDK so that I can get the intensity value of that location. The LocationEnums file seems to imply that CloudProvider is required so maybe we create an option for "OnPrem" cloud provider types that can support direct latitude and longitude values.
As a user of the SDK
I want to easily set my local settings
So that I can get started quickly and still be able to get latest without any conflict
Local settings file as non tracked changes.
This should allow setting up the local machine without creating local changes and pull latest easily.
The local settings file would be called appsettings.local.json and be added to the .gitignore .
A template file would be added with placeholder for all settings needed.
User would just have to copy and rename the file and update the value.
GitHub/Slack: Willmish
When running the auto-generated client for Python and calling get_emissions_data_for_location_by_time
(which polls GET /emissions/bylocation
endpoint I believe), the program sends the request, receives a response and crashes on parsing the response, when converting the type of duration
parameter. (From a string JSON value to TimeSpan Object)
openapi_client.exceptions.ApiTypeError: Invalid type for variable 'duration'. Required value type is TimeSpan and passed type was str at ['received_data'][0]['duration']
Script used (With the WebAPI SDK deployed locally and running on localhost:5073:
import time
import openapi_client
from pprint import pprint
from openapi_client.api import carbon_aware_api
from openapi_client.model.emissions_data import EmissionsData
from dateutil.parser import parse
# Defining the host is optional and defaults to http://localhost
# See configuration.py for a list of all supported configuration parameters.
configuration = openapi_client.Configuration(
host = "http://localhost:5073"
)
# Enter a context with an instance of the API client
with openapi_client.ApiClient(configuration) as api_client:
# Create an instance of the API class
api_instance = carbon_aware_api.CarbonAwareApi(api_client)
location = "westus" # str | (optional)
time = parse('2022-07-22T10:30:00.00Z') # datetime | (optional)
to_time = parse('2022-07-22T11:00:00.00Z') # datetime | (optional)
duration_minutes = 0 # int | (optional) (default to 0)
try:
api_response = api_instance.get_emissions_data_for_location_by_time(location=location, time=time, to_time=to_time, duration_minutes=duration_minutes)
pprint(api_response)
except openapi_client.ApiException as e:
print("Exception when calling CarbonAwareApi->emissions_bylocation_get: %s\n" % e)
WebAPI (Default)
File "carbon-aware-sdk/src/clients/python/openapi_client/model_utils.py", line 1459, in attempt_convert_item
raise get_type_error(input_value, path_to_item, valid_classes,
openapi_client.exceptions.ApiTypeError: Invalid type for variable 'duration'. Required value type is TimeSpan and passed type was str at ['received_data'][0]['duration']
As part of documentation, there is value in expressing that carbon aware software should also take demand side into consideration.
"At times of higher demand, like 6-8pm or in the afternoons on a very hot day, the grid is more strained and often more dirty,"
No response
We should consider renaming the time
parameter to something like fromTime
- better description.
Also, examples for client generated libraries won't be broken as well - they normally create a variable called time for values of the time
parameter, most languages have a library/package called time and although some languages allow renaming variables to package names (e.g. Python), this breaks any references to the time library package.
WebAPI (Default)
No response
GitHub/Slack: Willmish
The CLI does not change the output to a single, best location and time, but simply omits the parameter (parses it, but doesn't do anything with it), see the diff file with some changes I tried making to fix it (currently CarbonAwareCOnstants.best props is set to true by default, but never used later on - only _aggregator.GetEmissionsDataAsync()
is called but never _aggregator.GetBestEmissionsDataAsync()
diff --git a/src/CarbonAware.CLI/src/CarbonAwareCLI.cs b/src/CarbonAware.CLI/src/CarbonAwareCLI.cs
index 350639d..db2ff3d 100644
--- a/src/CarbonAware.CLI/src/CarbonAwareCLI.cs
+++ b/src/CarbonAware.CLI/src/CarbonAwareCLI.cs
@@ -63,16 +63,23 @@ public class CarbonAwareCLI
{ CarbonAwareConstants.MultipleLocations, locations },
{ CarbonAwareConstants.Start, _state.Time },
{ CarbonAwareConstants.End, _state.ToTime },
- { CarbonAwareConstants.Best, true }
1 diff --git a/src/CarbonAware.CLI/src/CarbonAwareCLI.cs b/src/CarbonAware.CLI/src/CarbonAwareCLI.cs
+ { CarbonAwareConstants.Best, _state.Lowest }
};
return await GetEmissionsDataAsync(props);
}
private async Task<IEnumerable<EmissionsData>> GetEmissionsDataAsync(Dictionary<string, object> props)
{
- IEnumerable<EmissionsData> e = await _aggregator.GetEmissionsDataAsync(props);
+ if ((bool)props[CarbonAwareConstants.Best])
+ {
+ EmissionsData e = await _aggregator.GetBestEmissionsDataAsync(props);
+ return new IEnumerable<EmissionsData> {e};
+ }
+ else
+ {
+ return await _aggregator.GetEmissionsDataAsync(props);
+ }
- return await _aggregator.GetEmissionsDataAsync(props);
}
public void OutputEmissionsData(IEnumerable<EmissionsData> emissions)
CLI
No response
GitHub/Slack: Willmish
/emissions/bylocation
endpoint, and only providing Location and not a time frame, by default the SDK requests data for a 7 days in 5 minute intervals (hundreds of data).time
and toTime
, SDK will respond with 2 values: before time
and after toTime
This doesn't seem like a healthy default for the SDK.
Instead, I suggest:
toTime
, only up to - this way we will also receive a single value when time
is the same as toTime
Example Swagger UI query which produces a lot of data:
WebAPI (Default)
No response
There is a configuration mismatch between .net 6 and Visual studio 2019 and hence the current solution does not compile with Visual Studio 2019.
Working on seeing if the proposed workarounds -https://docs.microsoft.com/en-us/answers/questions/620284/how-to-target-net-60-with-visual-studio-2019.html solve the issue
Add file format validation error handling to CarbonAwareStaticJsonDataService.GetData
There is a need to have some core tenets laid out to ensre we have a clear agreed and unified perspective on the engineering approach for the SDK.
These will be kept on the Wiki for now. This issue is to track discussion and track progress. NOTE: Discussions can still happen in discussions tab (a tool for th ejob), just link them from here, and link back to this issue please.
Initial list of things to discuss:
This feature will update the CLI and WebAPI Dockerfiles to follow production-ready best practices and add documentation & examples for how to build and deploy the docker container to a container registry.
the duration being a timespan creates too much complexity for little to zero gain.
Moving it to an integer primitive creates better interoperability, with a default of zero providing "at this point in time" results as desired.
GitHub/Slack: Willmish
The API has a non standardised output for carbon intensity units, since WattTime uses lbs CO2 per MWh, while other providers such as ElectricityMap use grams CO2 per kWh. The response should either provide the units for the reading as well or automatically convert it to a single standard unit (preferred) - ideally using a combination of SI units (grams CO2 per kWh).
Other issue with the output is that the response only contains the WattTime (or other carbon intensity provider) region index, so when querying multiple locations (on the GET /emissions/bylocations
or GET /emissions/bylocations/best
) it requires additional work to identify which data centre region corresponds to which WattTime region. Instead, the API should provide both the WattTime region and the matching data centre region name in the response.
TO DISCUSS:
WebAPI (Default)
No response
No response
Tracking officially creating Release 1.0
Steps:
WebAPI (Default)
No response
No response
If you get the latest dev branch (with 0.1-preview changes) and try and run the cli with the below command, the cli errors.
dotnet run -l eastus westus australiaeast northeurope -t 2021-12-28 --toTime 2021-12-30 -v --lowest
Expected:
Using the default parameters, the cli should be able to come back with a default response...
Solution:
Comparing to the WebApi which does work as described above, it must be missing some default values
CLI
info: CarbonAware.Aggregators.CarbonAware.CarbonAwareAggregator[0]
Aggregator getting carbon intensity from data source
info: CarbonAware.DataSources.WattTime.WattTimeDataSource[0]
Getting carbon intensity for locations {"locationType":0,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"eastus","displayName":"Not Provided"}, {"locationType":0,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"westus","displayName":"Not Provided"}, {"locationType":0,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"australiaeast","displayName":"Not Provided"}, {"locationType":0,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"northeurope","displayName":"Not Provided"} for period 12/28/2021 00:00:00 +00:00 to 12/30/2021 00:00:00 +00:00.
info: CarbonAware.DataSources.WattTime.WattTimeDataSource[0]
Getting carbon intensity for location {"locationType":0,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"eastus","displayName":"Not Provided"} for period 28/12/2021 00:00:00 +00:00 to 30/12/2021 00:00:00 +00:00.
fail: CarbonAware.DataSources.WattTime.WattTimeDataSource[0]
Failed to convert the location {"locationType":0,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"eastus","displayName":"Not Provided"} into a Balancying Authority.
CarbonAware.Exceptions.LocationConversionException: Location '' cannot be converted to Geoposition.
at CarbonAware.LocationSources.Azure.AzureLocationSource.ToGeopositionLocationAsync(Location location) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.LocationSources/CarbonAware.LocationSources.Azure/src/AzureLocationSource.cs:line 55
at CarbonAware.DataSources.WattTime.WattTimeDataSource.GetBalancingAuthority(Location location, Activity activity) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.DataSources/CarbonAware.DataSources.WattTime/src/WattTimeDataSource.cs:line 151
Unhandled exception. CarbonAware.Exceptions.LocationConversionException: Location '' cannot be converted to Geoposition.
at CarbonAware.LocationSources.Azure.AzureLocationSource.ToGeopositionLocationAsync(Location location) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.LocationSources/CarbonAware.LocationSources.Azure/src/AzureLocationSource.cs:line 55
at CarbonAware.DataSources.WattTime.WattTimeDataSource.GetBalancingAuthority(Location location, Activity activity) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.DataSources/CarbonAware.DataSources.WattTime/src/WattTimeDataSource.cs:line 151
at CarbonAware.DataSources.WattTime.WattTimeDataSource.GetCarbonIntensityAsync(Location location, DateTimeOffset periodStartTime, DateTimeOffset periodEndTime) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.DataSources/CarbonAware.DataSources.WattTime/src/WattTimeDataSource.cs:line 98
at CarbonAware.DataSources.WattTime.WattTimeDataSource.GetCarbonIntensityAsync(IEnumerable`1 locations, DateTimeOffset periodStartTime, DateTimeOffset periodEndTime) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.DataSources/CarbonAware.DataSources.WattTime/src/WattTimeDataSource.cs:line 57
at CarbonAware.Aggregators.CarbonAware.CarbonAwareAggregator.GetEmissionsDataAsync(IDictionary props) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.Aggregators/src/CarbonAware/CarbonAwareAggregator.cs:line 36
at CarbonAware.CLI.CarbonAwareCLI.GetEmissionsDataAsync(Dictionary`2 props) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.CLI/src/CarbonAwareCLI.cs:line 73
at CarbonAware.CLI.CarbonAwareCLI.GetEmissions() in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.CLI/src/CarbonAwareCLI.cs:line 68
at CarbonAware.CLI.Program.GetEmissionsAsync(String[] args, ICarbonAwareAggregator aggregator, ILogger`1 logger) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.CLI/src/Program.cs:line 41
at CarbonAware.CLI.Program.Main(String[] args) in /Users/dan.benitah/projects/ava/gsf/carbon-aware-sdk/src/CarbonAware.CLI/src/Program.cs:line 15
at CarbonAware.CLI.Program.<Main>(String[] args)
for using simply in CICD pipeline
Based on usage and common patterns of other cloud based CLI's, the console/tabular output option should be removed. "json" will continue to be supported.
mark.oates
Tried with location variable as WestUS, this failed, tried again with westus, this worked fine.
Fail: dotnet run -l WestUS -t 2022-09-22T00:00 --toTime 2022-09-22T01:00
Success: dotnet run -l westus -t 2022-09-22T00:00 --toTime 2022-09-22T01:00
CLI
Getting carbon intensity for location {"locationType":2,"latitude":null,"longitude":null,"cloudProvider":null,"regionName":"WestUS","displayName":"WestUS"} for period 9/22/2022 12:00:00 AM +00:00 to 9/22/2022 1:00:00 AM +00:00.
Unhandled exception. System.ArgumentException: Unknown region: Region name 'WestUS' not found
We are looking to build out a good demo in Java for the SDK. Demo would be using the REST endpoint or the CLI, with the scenario and demo being something common in Java.
The scenario is yet to be defined. Please let us know if you have ideas below.
Skills needed:
If you are interested let us know below.
Update usage documentation to reflect new command line parameters.
--data-file was removed
-c --config was added
As a Green Developer I want to know the carbon emissions across data centres that I deploy in on multiple clouds so that I can make the greenest software decisions.
I believe these files were added in a replicated folder name by accident. Files in this location to be moved to root folder.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.