Currently, it is being done manually. Instead, we should try to create a DataFrame and write to csv.

Summary

We need an input describing what flows are free, and we chose to use GML.

Basic Example

Here is an example of GML file:

graph [
	comment "This is a sample graph"
	directed 1
	id 42
	label "Hello, I am a graph"
	node [
		id 1
		label "node 1"
		thisIsASampleAttribute 42
	]
	node [
		id 2
		label "node 2"
		thisIsASampleAttribute 43
	]
	node [
		id 3
		label "node 3"
		thisIsASampleAttribute 44
	]
	edge [
		source 1
		target 2
		label "Edge from node 1 to node 2"
	]
	edge [
		source 2
		target 3
		label "Edge from node 2 to node 3"
	]
	edge [
		source 3
		target 1
		label "Edge from node 3 to node 1"
	]
]

Drawbacks

It looks like it doesn't support data to be attached to nodes and edges (@suvayu, do you know?). If that is the case, we might have to find something else.

@datejada, is there anything else apart from the profile that is attached to an edge?

Reference Issues

• #103

Create the data structure that the model consumes.

• #118
• #103
• #104
• #105
• #106
• #261
• #262
• #263

Like a high-level TODO list, or a collection of user stories.

• What is this
• How to use
• API
• Tutorial

Add to documentation

Summary

Add explicit permissions in the workflow files. The default will be changed to read and write by #39, so it is safer to be explicit.
All unspecified permissions are set to no access (https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token).

Basic Example

Add some code like

permissions:
    contents: read

Drawbacks

None that I can see.

Reference Issues

No response

Maybe something like https://medium.com/@dave_lunny/sane-github-labels-c5d2e6004b63 but on a smaller scale?

For instance, check https://github.com/stevemao/github-issue-templates/tree/master/.github/ISSUE_TEMPLATE but much shorter.

Probably something for

• Bug reports
• Feature request

It runs daily and it is failing. I did something similar here: abelsiqueira/ProbabilityThing.jl#1 and possibly something has to be enabled in the settings. Haven't checked yet.

• #35
• #113
• #114
• Discuss invariance of solutions (something akin to property based testing) (Get back to @suvayu on this)
• #115
• #491
• Create function to filter output
• #191
• Create the function to convert to ESDL (possibly in Python, possibly part of #88)
• #295
• #352

Discussed in #56

^{Originally posted by datejada August 22, 2023}
I can foresee three data files in the model that need to be considered as inputs and outputs for the model; see Figure below.

1. ESDL files (.esdl): These files will contain the available information that can be stored in this format. Generally speaking, it is capacities, efficiencies, investment costs, operational costs, and commission and decommissions dates.
2. Complementary information files (.csv?): Here we have two categories

• Extra information for assets: Information not available in the ESDL format, e.g., unit-commitment parameters.
• Configuration parameters: setting parameters such as the years to run, type of simulation (only dispatch or with investment), etc.

3. Profiles database (.sqllite?): This file/DB will contain a large amount of input data (e.g., availability profiles) and results (e.g., production profiles for each asset).

Maybe we need more types of files; I'm not sure. In addition, the pre-processing and post-processing can be either in Julia or Python; these steps should be able to link the information in all the data sources with the optimization model.

I hope these comments and general overview help in the discussion for next week's meeting to talk about the design of the model.

Such as new issues being labeled "Todo" and issues moving to "In Progress" if someone is assigned or there is a Pull Request referencing it. And labeled "Done" when closed with a Pull request.

README should have important at-a-glance information for potential users, and some information for people that know what they want. A possible structure is as follows:

• Brief description for users who don't know if they are in the right place.
• Brief description for specialized users.
• Installation instructions.
• 3-5 lines example and link to documentation for further explanation.
• Directions for contributors.
• License and link to LICENSE.

Summary

We should publish the package ASAP so we can dibs the name.

Basic Example

Do it, quick!

Drawbacks

We might start seeing responses from the "community."
We have to start dealing with version releases.

Reference Issues

No response

Lazy Loading is loading only the data needed for the current run, not ALL data.
This method is for saving RAM.

This file should describe how to setup the environment to develop the package. For instance, installing pre-commit.
Check https://github.com/abelsiqueira/PlaygroundWithGit.jl for example.

EditorConfig is configured to use 4 spaces, but YAML and JSON files were created with 2 spaces.
Since the formatting of these files is not checked - only correctness (hopefully) - these are not enforced.
However, when using an editor, especially when "format on save" is enabled, the editor tries to force 4 spaces.

We should try to find a formatter/linter for yaml and json. Either way, we should fix the behaviour with one of two options:

Option 1:

• Configure .editorconfig to use 2 spaces for yaml and json

Option 2

• Update all yaml and json files to use 4 spaces

Dicts are

D = Dict(:a => 1, :b => 2)

and NamedTuples are

N = (a = 1, b = 2)

You can access named tuples via ., like N.a.

NamedTuples are like structures without name. They don't use strings and they are immutable, so they might be more efficient.

Also, it makes more sense, since they are parameters that won't be changed within the code.

Codecov integration is not complete. I believe it needs to be enabled at https://app.codecov.io by an owner.
Although I could do it, it might be interesting for someone from TNO to check it out.

The Benchmark workflow is supposed to run the benchmarks and comment the resources in the PR. However, it needs permissions that can't be granted if the PR is from a fork.

The "straightforward but not simple solution" would be to change our workflow entirely and use branches in the main repo instead of forks.

@suvayu also posted the idea below:

IMO this should be a bot, and a cron job. Bot to manually trigger it in a PR, and cron that checks performance regression, say, once a week.

Originally posted by @suvayu in #50 (comment)

@suvayu, do you know how to set that up, do you know if it works without permissions?

Change the model formulation to include the following:

• #107
• #108
• #135
• #109
• #110
• #217
  • views with index arrays?
  • any other paradigm (ask @lsoucasse)?
• #123

Test cases:
See #88

This would go on all new issues to help keep track of what still needs to be discussed.

Check https://github.com/abelsiqueira/PlaygroundWithGit.jl, for instance.

Compare output to expected output, i.e., test that the output of the solver is correct.

For instance, using https://bit.ly/cffinit.

In VSCode (or similar) you can change your settings so that the Julia Formatter runs when you save. We should tell people to enable this in the README.dev.
This makes sure your code works in the Julia Formatter before running pre-commit, etc.

Description

The linting is failing for the following files after the update in the spacing format of the YML files:

-ADDITION-REQUEST.yml
-BUG-REPORT.yml
-IMPROVEMENT-REQUEST.yml

Reproduction steps

1. Go to git bash
2. then run the command "pre-commit run -a"
3. The error message is shown in the Logs section.

Logs

prettier.................................................................Failed
- hook id: prettier
- files were modified by this hook

.github/ISSUE_TEMPLATE/ADDITION-REQUEST.yml
.github/ISSUE_TEMPLATE/BUG-REPORT.yml
.github/ISSUE_TEMPLATE/IMPROVEMENT-REQUEST.yml

OS

No response

The Julia files don't follow the Julia Formatter definitions. One option to update them all is running the pre-commit hook and committing the changes in the files.

The current README.dev.md includes a contributing workflow without git commands. It would be handy, especially for beginners like us, to have the git commands also included there.

I propose to add the commands from the notes https://hackmd.io/l1XUZ2n-QfSIbRl74FWk6g.

Possibly add to a scripts folder.

https://github.com/JuliaCI/BenchmarkTools.jl

We need to add the stable documentation of the README, as right now there is no file linked to this, and an error will occur in the pre-commit when not fixed.

Summary

For internal use, the issue templates are too constraining.

Basic Example

Epic is an example of issue that doesn't match what we want.

Drawbacks

We don't want other people to be able to create epic issues (maybe that can be enforced?)

Reference Issues

No response

We should have a function to create the model that directly receives the parameters, and another function that reads from the data. This will be useful for benchmarking as well.

On August 16 we came up with different options to think about:

• FlexBloom
• EnergyBloom
• BloomEn

Summary

Define the model in sets, indices, constraints, variables, etc.

• ADD ISSUES HERE

Epics Overview
1. ESDL round trip
2. Data structure
3. Model definition
4. Output comparison

Summary

Change the names to register the package in the future in Julia.

Basic Example

• Repository: Tulipa Energy Model
• Organization: Tulipa Energy

Drawbacks

NA

Reference Issues

No response

Description

The Badge on README is pointing to CI.yml, which does not exist anymore on main.
The fix is to remove it and add new badges for Test, Lint, and Docs.

Reproduction steps

Check the action tab.

Logs

No response

OS

No response

Ensure compatibility between ESDL asset/flows attributes with the model formulation.

The strategy is to play with the map editor to understand how parameters are assigned to flows/assets in ESDL. Then, compare with formulation and see if something has to be changed either on the ESDL side or on the model side.

Currently, the model creates dictionaries to obtain the parameter values.
For instance,

p_variable_cost   = Dict((row.a)                => row.pVarCost for row in eachrow(assets_df))

This is needed because the model is indexed by the values of the sets. For instance,

A = assets_df.a
...
@variable(model, 0 <= v_float[A,...])

Instead, we can index via the numerical index of the set:

num_assets = length(assets_df.a)
A = 1:num_assets
...
p_variable_cost = assets_df.pVarCost

However, to do this, the parameters p_demand and p_availability would need some extra attention, since they are not, technically, 1-dimensional.
If it is possible to create this version, a comparison will help decide which strategy to follow.

Summary

Part of the README references "TulipaBulb.jl" which has since been changed to "TulipaEnergyModel." This should be updated.

Basic Example

Drawbacks

None!

Reference Issues

#41

Summary

This key is used by the TagBot and can be used by CompatHelper. It allows them to make commits, pull requests, etc., using a user's SSH key.

Basic Example

To generate use

pkg> activate --temp
pkg> add DocumenterTools
julia> using DocumenterTools
julia> DocumenterTools.genkeys(user="TulipaEnergy", repo="TulipaEnergyModel.jl")

Then follow the instructions.

Drawbacks

None.

Reference Issues

No response

For instance, only run tests if .jl files have changed.

What are the rules for when to assign yourself an issue, when is something labeled as "in progress?"
Write a draft of this to see if others agree.

This file should have the pull request template.
Check https://github.com/abelsiqueira/PlaygroundWithGit.jl for example.

For consistency and also because it already is inside the workflows file.

See Suvayu's email for the correct setup

Update the formulation such that the flow variable is from asset $a$ to assets $a'$ like this: $v_{flow}(a,a',rp,k)$.
There are at least changes in the following parts:

• Input data files
• Parameter definitions
• Variables definitions
• Constraints

Go through the ESDL file and map ports to connections to create a graph in Julia.

The esdl ports link to a hash that later must be scanned to find the right connection.

Also, obtain the assets and flows data

Removed the following because this will happen before GML+CSV are created, i.e., before Julia is involved.
Possible implementation

• represent graph data structure using GraphIO.jl

Convert ESDL to GML + CSV and back
(Most likely a python script on another repository.)

• #89
• #91
• Handle aggregated attributes
  • e.g. variable cost can be a sum of different costs
• #102
  • (maybe part of #91?)

See also #63 for more info/discussion.

Test cases:

• #100
• #111
• #101
• #112