genxproject / genx Goto Github PK

Attempting to run GenX example from commit 8848444 on the main branch I get an error when reading input files.

Excerpt of the run:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.3.1 (2019-12-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(GenX) pkg> activate .
Activating environment at `~/Documents/qvist/GenXProject/GenX/Project.toml`

(GenX) pkg> instantiate
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed SentinelArrays ─ v1.3.5
  Updating `~/Documents/qvist/GenXProject/GenX/Project.toml`
  [6e4b80f9] + BenchmarkTools v1.1.1
  [336ed68f] + CSV v0.8.5
  [9961bab8] + Cbc v0.8.0
  [e2554f3b] + Clp v0.8.4
  [aaaa29a8] + Clustering v0.14.2
  [861a8166] + Combinatorics v1.0.2
  [a93c6f00] + DataFrames v1.2.1
  [864edb3b] + DataStructures v0.17.20
  [41bf760c] + DiffEqSensitivity v6.33.0
  [b4f34e82] + Distances v0.10.3
  [e30172f5] + Documenter v0.27.4
  [35a29f4d] + DocumenterTools v0.1.13
  [60bf3e95] + GLPK v0.14.12
  [2e9cd046] + Gurobi v0.9.14
  [b6b21f68] + Ipopt v0.7.0
  [4076af6c] + JuMP v0.21.4
  [fdba3010] + MathProgBase v0.7.8
  [1dea7af3] + OrdinaryDiffEq v5.42.3
  [8a4e6c94] + QuasiMonteCarlo v0.2.3
  [2913bbd2] + StatsBase v0.33.8
  [ddb6d928] + YAML v0.4.7
  [ade2ca70] + Dates 
  [37e2e46d] + LinearAlgebra 
  [10745b16] + Statistics 
  Updating `~/Documents/qvist/GenXProject/GenX/Manifest.toml`
  [a4c015fc] + ANSIColoredPrinters v0.0.1
  [ae81ac8f] + ASL_jll v0.1.2+0
  [621f4979] + AbstractFFTs v1.0.1
  [1520ce14] + AbstractTrees v0.3.4
  [79e6a3ab] + Adapt v2.4.0
  [ec485272] + ArnoldiMethod v0.1.0
  [4fba245c] + ArrayInterface v2.14.17
  [4c555306] + ArrayLayouts v0.1.5
  [56f22d72] + Artifacts v1.3.0
  [6e4b80f9] + BenchmarkTools v1.1.1
  [b99e7846] + BinaryProvider v0.5.10
  [6e34b625] + Bzip2_jll v1.0.7+0
  [fa961155] + CEnum v0.4.1
  [336ed68f] + CSV v0.8.5
  [49dc2e85] + Calculus v0.5.1
  [9961bab8] + Cbc v0.8.0
  [38041ee0] + Cbc_jll v2.10.5+2
  [3830e938] + Cgl_jll v0.60.2+6
  [d360d2e6] + ChainRulesCore v0.9.45
  [e2554f3b] + Clp v0.8.4
  [06985876] + Clp_jll v1.17.6+7
  [aaaa29a8] + Clustering v0.14.2
  [523fee87] + CodecBzip2 v0.7.2
  [944b1d66] + CodecZlib v0.7.0
  [be027038] + CoinUtils_jll v2.11.3+4
  [861a8166] + Combinatorics v1.0.2
  [38540f10] + CommonSolve v0.2.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v3.31.0
  [e66e0078] + CompilerSupportLibraries_jll v0.3.4+0
  [88cd18e8] + ConsoleProgressMonitor v0.1.2
  [adafc99b] + CpuId v0.2.2
  [a8cc5b0e] + Crayons v4.0.4
  [9a962f9c] + DataAPI v1.7.0
  [a93c6f00] + DataFrames v1.2.1
  [864edb3b] + DataStructures v0.17.20
  [e2d170a0] + DataValueInterfaces v1.0.0
  [2b5f629d] + DiffEqBase v6.44.3
  [459566f4] + DiffEqCallbacks v2.13.5
  [c894b116] + DiffEqJump v6.10.0
  [77a26b50] + DiffEqNoiseProcess v5.8.0
  [41bf760c] + DiffEqSensitivity v6.33.0
  [163ba53b] + DiffResults v1.0.3
  [b552c78f] + DiffRules v1.0.2
  [b4f34e82] + Distances v0.10.3
  [31c24e10] + Distributions v0.23.8
  [ffbed154] + DocStringExtensions v0.8.5
  [e30172f5] + Documenter v0.27.4
  [35a29f4d] + DocumenterTools v0.1.13
  [d4d017d3] + ExponentialUtilities v1.8.2
  [7a1cc6ca] + FFTW v1.3.2
  [f5851436] + FFTW_jll v3.3.9+8
  [9aa1b823] + FastClosures v0.3.2
  [1a297f60] + FillArrays v0.8.14
  [6a86dc24] + FiniteDiff v2.8.1
  [59287772] + Formatting v0.4.2
  [f6369f11] + ForwardDiff v0.10.18
  [069b7b12] + FunctionWrappers v1.1.2
  [60bf3e95] + GLPK v0.14.12
  [e8aa6df9] + GLPK_jll v5.0.0+0
  [781609d7] + GMP_jll v6.1.2+6
  [01680d73] + GenericSVD v0.3.0
  [708ec375] + Gumbo v0.8.0
  [528830af] + Gumbo_jll v0.10.1+1
  [2e9cd046] + Gurobi v0.9.14
  [cd3eb016] + HTTP v0.9.12
  [b5f81e59] + IOCapture v0.2.2
  [7869d1d1] + IRTools v0.3.3
  [d25df0c9] + Inflate v0.1.2
  [83e8ac13] + IniFile v0.5.0
  [1d5cc7b8] + IntelOpenMP_jll v2018.0.3+2
  [41ab1584] + InvertedIndices v1.0.0
  [b6b21f68] + Ipopt v0.7.0
  [9cc047cb] + Ipopt_jll v3.13.4+2
  [42fd0dbc] + IterativeSolvers v0.8.5
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [682c06a0] + JSON v0.21.1
  [7d188eb4] + JSONSchema v0.3.3
  [4076af6c] + JuMP v0.21.4
  [2ee39098] + LabelledArrays v1.6.2
  [a5e1c1ea] + LatinHypercubeSampling v1.8.0
  [73f95e8e] + LatticeRules v0.0.1
  [4af54fe1] + LazyArtifacts v1.3.0
  [1d6d02ad] + LeftChildRightSiblingTrees v0.1.2
  [94ce4f54] + Libiconv_jll v1.16.0+8
  [093fc24a] + LightGraphs v1.3.5
  [d3d80556] + LineSearches v7.1.1
  [2ab3a3ac] + LogExpFunctions v0.2.5
  [e6f89c97] + LoggingExtras v0.4.7
  [bdcacae8] + LoopVectorization v0.8.26
  [d00139f3] + METIS_jll v5.1.0+5
  [856f044c] + MKL_jll v2021.1.1+1
  [d7ed1dd3] + MUMPS_seq_jll v5.2.1+4
  [1914dd2f] + MacroTools v0.5.6
  [b8f27783] + MathOptInterface v0.9.22
  [fdba3010] + MathProgBase v0.7.8
  [739be429] + MbedTLS v1.0.3
  [c8ffd9c3] + MbedTLS_jll v2.16.8+1
  [e1d29d7a] + Missings v1.0.0
  [46d2c3a1] + MuladdMacro v0.2.2
  [d8a4904e] + MutableArithmetics v0.2.20
  [d41bc354] + NLSolversBase v7.7.1
  [2774e3e8] + NLsolve v4.5.1
  [872c559c] + NNlib v0.6.6
  [77ba4419] + NaNMath v0.3.5
  [b8a86587] + NearestNeighbors v0.4.9
  [ca575930] + NetworkOptions v1.2.0
  [6fe1bfb0] + OffsetArrays v1.10.3
  [656ef2d0] + OpenBLAS32_jll v0.3.9+4
  [efe28fd5] + OpenSpecFun_jll v0.5.3+4
  [429524aa] + Optim v1.2.4
  [bac558e1] + OrderedCollections v1.4.1
  [1dea7af3] + OrdinaryDiffEq v5.42.3
  [7da25872] + Osi_jll v0.108.5+4
  [90014a1f] + PDMats v0.10.1
  [d96e819e] + Parameters v0.12.2
  [69de0a69] + Parsers v1.1.0
  [e409e4f3] + PoissonRandom v0.4.0
  [2dfb63ee] + PooledArrays v1.2.1
  [85a6dd25] + PositiveFactorizations v0.2.4
  [21216c6a] + Preferences v1.2.2
  [08abe8d2] + PrettyTables v1.1.0
  [33c8b6b6] + ProgressLogging v0.1.4
  [92933f4c] + ProgressMeter v1.7.1
  [1fd47b50] + QuadGK v2.4.1
  [8a4e6c94] + QuasiMonteCarlo v0.2.3
  [74087812] + Random123 v1.3.1
  [e6cf234a] + RandomNumbers v1.4.0
  [3cdcf5f2] + RecipesBase v1.1.1
  [731186ca] + RecursiveArrayTools v2.11.4
  [f2c3362d] + RecursiveFactorization v0.1.8
  [189a3867] + Reexport v0.2.0
  [ae029012] + Requires v1.1.3
  [ae5879a3] + ResettableStacks v1.1.1
  [37e2e3b7] + ReverseDiff v1.9.0
  [79098fc4] + Rmath v0.6.1
  [f50d1b31] + Rmath_jll v0.2.2+2
  [f2b01f46] + Roots v1.0.11
  [21efa798] + SIMDPirates v0.8.26
  [476501e8] + SLEEFPirates v0.5.5
  [322a6be2] + Sass v0.2.0
  [91c51154] + SentinelArrays v1.3.5
  [699a6c99] + SimpleTraits v0.9.4
  [ed01d8cd] + Sobol v1.5.0
  [a2af1166] + SortingAlgorithms v1.0.1
  [47a9eef4] + SparseDiffTools v1.9.1
  [276daf66] + SpecialFunctions v0.10.3
  [860ef19b] + StableRNGs v1.0.0
  [90137ffa] + StaticArrays v0.12.5
  [82ae8749] + StatsAPI v1.0.0
  [2913bbd2] + StatsBase v0.33.8
  [4c63d2b9] + StatsFuns v0.9.8
  [789caeaf] + StochasticDiffEq v6.25.0
  [69024149] + StringEncodings v0.3.5
  [fa267f1f] + TOML v1.0.3
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.4.4
  [5d786b92] + TerminalLoggers v0.1.4
  [9f7883ad] + Tracker v0.2.16
  [3bb67fe8] + TranscodingStreams v0.9.5
  [a2a6695c] + TreeViews v0.3.0
  [5c2747f8] + URIs v1.3.0
  [3a884ed6] + UnPack v1.0.2
  [3d5dd08c] + VectorizationBase v0.12.33
  [19fa3120] + VertexSafeGraphs v0.1.2
  [ddb6d928] + YAML v0.4.7
  [a5390f91] + ZipFile v0.9.3
  [83775a58] + Zlib_jll v1.2.11+18
  [e88e6eb3] + Zygote v0.4.12
  [700de1a5] + ZygoteRules v0.2.1
  [47bcb7c8] + libsass_jll v3.5.5+0
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8bb1440f] + DelimitedFiles 
  [8ba89e20] + Distributed 
  [7b1f6079] + FileWatching 
  [9fa8497b] + Future 
  [b77e0a4c] + InteractiveUtils 
  [76f85450] + LibGit2 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [44cfe95a] + Pkg 
  [de0858da] + Printf 
  [3fa0cd96] + REPL 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [1a1011a3] + SharedArrays 
  [6462fe0b] + Sockets 
  [2f01184e] + SparseArrays 
  [10745b16] + Statistics 
  [4607b0f0] + SuiteSparse 
  [8dfed614] + Test 
  [cf7118a7] + UUIDs 
  [4ec0a83e] + Unicode 

(GenX) pkg> st
Project GenX v1.0.0
    Status `~/Documents/qvist/GenXProject/GenX/Project.toml`
  [6e4b80f9] BenchmarkTools v1.1.1
  [336ed68f] CSV v0.8.5
  [9961bab8] Cbc v0.8.0
  [e2554f3b] Clp v0.8.4
  [aaaa29a8] Clustering v0.14.2
  [861a8166] Combinatorics v1.0.2
  [a93c6f00] DataFrames v1.2.1
  [864edb3b] DataStructures v0.17.20
  [41bf760c] DiffEqSensitivity v6.33.0
  [b4f34e82] Distances v0.10.3
  [e30172f5] Documenter v0.27.4
  [35a29f4d] DocumenterTools v0.1.13
  [60bf3e95] GLPK v0.14.12
  [2e9cd046] Gurobi v0.9.14
  [b6b21f68] Ipopt v0.7.0
  [4076af6c] JuMP v0.21.4
  [fdba3010] MathProgBase v0.7.8
  [1dea7af3] OrdinaryDiffEq v5.42.3
  [8a4e6c94] QuasiMonteCarlo v0.2.3
  [2913bbd2] StatsBase v0.33.8
  [ddb6d928] YAML v0.4.7
  [ade2ca70] Dates 
  [37e2e46d] LinearAlgebra 
  [10745b16] Statistics 

julia> include("Example_Systems/RealSystemExample/ISONE_Singlezone/Run.jl")
Loading packages
[ Info: Precompiling GenX [5d317b1e-30ec-4ed6-a8ce-8d2d88d7cfac]
Configuring Solver
Loading Inputs
Reading Input CSV Files
ERROR: LoadError: BoundsError: attempt to access 0-element Array{Int64,1} at index [1]
Stacktrace:
 [1] getindex at ./array.jl:744 [inlined]
 [2] load_load_data(::Dict{Any,Any}, ::String, ::String, ::Dict{Any,Any}) at /home/anton/Documents/qvist/GenXProject/GenX/src/load_inputs/load_load_data.jl:78
 [3] load_inputs(::Dict{Any,Any}, ::String) at /home/anton/Documents/qvist/GenXProject/GenX/src/load_inputs/load_inputs.jl:54
 [4] top-level scope at /home/anton/Documents/qvist/GenXProject/GenX/Example_Systems/RealSystemExample/ISONE_Singlezone/Run.jl:60
 [5] include at ./boot.jl:328 [inlined]
 [6] include_relative(::Module, ::String) at ./loading.jl:1105
 [7] include(::Module, ::String) at ./Base.jl:31
 [8] include(::String) at ./client.jl:424
 [9] top-level scope at REPL[5]:1
in expression starting at /home/anton/Documents/qvist/GenXProject/GenX/Example_Systems/RealSystemExample/ISONE_Singlezone/Run.jl:60

Potential solution?
By changing the line in Project.toml from: CSV = "0.6.0, 0.8" to CSV = "0.6.0", I can run the example.

If it matters, I am running 64-bit Ubuntu 20.04.2 LTS.

Superb project!
/Anton

Minimizing friction for new users is good.
Recently I was playing around with trying to get GenX to run example cases external to the GenX directory (so that the git status remains unchanged and so I'm more confident that I'm not going to break something). I realized that the instructions

~ > cd GenX
~ > julia --project="GenX"
(GenX) pkg> activate .

are somewhat confusing and that as written, the second line's command is superfluous. Specifically, step 2 of Method 1 in the readme says that after starting julia "You will see that you're already within the (GenX) environment.". However, this is not really true. If you type (GenX) pkg> st you'll see that the project loader thinks it has an empty project, which just happens to be named GenX:

(GenX) pkg> st
      Status `~/GenX/GenX/Project.toml` (empty project)

it's not till you type activate . that the correct toml file is loaded.

Solutions / other information gleaned

Here's a few way to start a julia REPL with in the correct environment. These assume my installation is in ~/GenX. Basically, you can use --project=[...] from the shell or pkg > activate [...] from a REPL; as far as I can tell they do the same thing. See also https://bkamins.github.io/julialang/2020/05/10/julia-project-environments.html .

Three lines

~ > cd GenX
~/GenX > julia
]
(@v1.6) pkg > activate .

Two lines:

~ > cd GenX
~/GenX > julia --project=.

or

~ > julia
]
(@v1.6) pkg > activate GenX

or

/any/directory > julia
]
(@v1.6) pkg > activate /path/to/my/GenX

note that you can't activate ~/GenX because the ~ does not get expanded.

One line:

~ > julia --project=GenX

or

/any/directory > julia --project=/path/to/my/GenX

When running an example (13 regions, 163 resources, 1 year with 10 representative periods) with Cbc, it took me about 2 hours to write results. I have identified that ternary statements such as the one included below from write_costs() (line 70) take several seconds for each region-resource pair. However when using Clp, results writing is almost instantaneous. I will continue to use Clp for now, but I include this for the group's information.

tempCFix = tempCFix +
(y in inputs["STOR_ALL"] ? value.(EP[:eCFixEnergy])[y] : 0) +
(y in inputs["STOR_ASYMMETRIC"] ? value.(EP[:eCFixCharge])[y] : 0) +
value.(EP[:eCFix])[y]

Lines 59-62 of write_power_balance.jl are commented out, but it's not clear why. Immediately above it looks like some rows are initialized to zero.

If the code is defunct it should be removed; if it's optional somehow there should be an explanation for why it's currently commented-out and when one might activate it - or better, there should be a setting.

          dfTemp1[t+rowoffset,8] = 0
          dfTemp1[t+rowoffset,9] = 0
          #=
              if setup["NetworkExpansion"] == 1
                  dfTemp1[t+rowoffset,8] = value(EP[:ePowerBalanceNetExportFlows][t,z])
                  dfTemp1[t+rowoffset,9] = -1/2 * value(EP[:eLosses_By_Zone][z,t])
              end
          =#

Generators_data.csv has several columns that aren't always needed. One example is all of the Charge_Cost columns, which are only used if a resource is STOR ==2. When assembling data for a model it would be great if I could omit these columns.

One solution would be to check if any resources have STOR values of 1 or 2 and then add the MWh/MWhyr or Charge column names to a list of all columns that need to be scaled (in this section).

If someone gives me a general guide for how to do this in Julia I can try to implement and submit a PR.

There might be some broader implications for the NEW/RET_CAP_CHARGE/ENERGY lines here -- I haven't followed the code to see where inputs_gen goes yet.

The docs correctly say that the Solver parameter in genx_settings.yml is case sensitive. But the comments in the documentation and the example file give conflicting (and incorrect) examples of how to specify the CLP solver -- one says CLP, the other says clp, and the code expects Clp. It would be more user-friendly to make the parameter case-insensitive. I'm not sure what the julia method for making a string lowercase is, but in python I'd do something like this:

...
elif solver.lower() == "clp":
...

https://github.com/GenXProject/GenX/blob/2a264cc1ac5b17e96c6f31718a014784121df140/Example_Systems/SmallNewEngland/OneZone/Run_test.jl#L74

User-callback calls 1905, time in user-callback 0.01 sec
LP solved for primal
Writing Output
ERROR: LoadError: MethodError: no method matching ndims(::Type{DataFrames.LazyNewColDataFrame{Int64}})
Closest candidates are:
  ndims(::Type{MutableArithmetics.Zero}) at C:\Users\Weiqi Ji\.julia\packages\MutableArithmetics\6pKCK\src\rewrite.jl:89
  ndims(::Base.Generator) at generator.jl:53
  ndims(::Number) at number.jl:67
  ...
Stacktrace:
 [1] Base.Broadcast.BroadcastStyle(::Type{DataFrames.LazyNewColDataFrame{Int64}}) at .\broadcast.jl:103
 [2] combine_styles(::DataFrames.LazyNewColDataFrame{Int64}) at .\broadcast.jl:420
 [3] combine_styles(::DataFrames.LazyNewColDataFrame{Int64}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Float64}}) at .\broadcast.jl:421
 [4] materialize!(::DataFrames.LazyNewColDataFrame{Int64}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Float64}}) at .\broadcast.jl:845
 [5] write_power(::String, ::String, ::Dict{Any,Any}, ::Dict{Any,Any}, ::Model) at C:\Users\Weiqi Ji\Dropbox (MIT)\Project\OpenEnergy\GenX\GenX\src\write_outputs\write_power.jl:46
 [6] write_outputs(::Model, ::String, ::Dict{Any,Any}, ::Dict{Any,Any}) at C:\Users\Weiqi Ji\Dropbox (MIT)\Project\OpenEnergy\GenX\GenX\src\write_outputs\write_outputs.jl:63
 [7] top-level scope at C:\Users\Weiqi Ji\Dropbox (MIT)\Project\OpenEnergy\GenX\GenX\Example_Systems\SmallNewEngland\OneZone\Run_test.jl:74
 [8] include(::String) at .\client.jl:457
 [9] top-level scope at REPL[33]:1
in expression starting at C:\Users\Weiqi Ji\Dropbox (MIT)\Project\OpenEnergy\GenX\GenX\Example_Systems\SmallNewEngland\OneZone\Run_test.jl:74

In the Requirements section of the Documentation, it refers to the file " juliaenv.jl in the parent directory lists all of the packages and their versions needed to run GenX " . I believe this file is now named "julenv.jl".

I came across a modeling bug when modeling thermal generators with Up_time/Down_time >= hours_per_subperiod for each representative period. In this case, the implemented minimum up and down time constraints look at commitment status of thermal generators in subsequent periods rather than in the same period. This results in the model reporting a key error for the last period.

I think we should update the documentation and tell the user they must use longer subperiods than the longest min up/down time if modeling UC

To reproduce,

Turn OperationWrapping = 1 in the SmallNewEngland/OneZone example, and examine the resulting power_balance.csv. It appears that there are large amounts of NSE.

Wilson suggested that for some reason with OperationWrapping on the NSE costs were not being properly accounted. I haven't looked into this yet, but he suggested it's because this is a full year problem.

The new LDS changes in Dev are leading to issues writing results when OperationWrapping is turned off in settings. For example in storage_all.jl, the cSoCBalStart constraint is not implemented when OperationWrapping = 0. I think all references to OperationWrapping should be removed after the section in load_load_data.jl where it is used to set the number of representative periods to 1. @sambuddhac @nspatank

Information is output to the log.
This was probably from print-statement debugging:

Hours per subperiod type
Int64
Type of s
Int64
Max flexible demand delay type
Int64
Hours per subperiod type
Int64
Type of s
Int64

probably this should be removed.

Maybe these got deleted?

Info

#CPLEX = "a076750e-1247-5638-91d2-ce28b192dca0"
#Gurobi = "2e9cd046-0924-5485-92f1-d5272153d98b"

[compat]
#CPLEX = "0.7.7"
#Gurobi = "0.9.14"

I set up a system where Generators_data.csv had columns ESR_1, ESR_2 and Energy_shares_requirement.csv had the order reversed (ESR_2, ESR_1), which gave me the error below. Reversing the order in Energy_shares_requirement.csv resolved the error but it was a lucky guess on my part. It isn't clear why the column order should make a difference.

Energy Share Requirement Policies Module
ERROR: LoadError: BoundsError: attempt to access 0×0 Matrix{Float64} at index [1:0, 1]
Stacktrace:
  [1] throw_boundserror(A::Matrix{Float64}, I::Tuple{Base.Slice{Base.OneTo{Int64}}, Int64})
    @ Base ./abstractarray.jl:651
  [2] checkbounds
    @ ./abstractarray.jl:616 [inlined]
  [3] _getindex
    @ ./multidimensional.jl:831 [inlined]
  [4] getindex(::Matrix{Float64}, ::Function, ::Int64)
    @ Base ./abstractarray.jl:1170
  [5] macro expansion
    @ ~/.julia/packages/MutableArithmetics/8xkW3/src/rewrite.jl:279 [inlined]
  [6] macro expansion
    @ ~/.julia/packages/JuMP/b3hGi/src/macros.jl:676 [inlined]
  [7] (::GenX.var"#695#699"{JuMP.Model, Dict{Any, Any}, Dict{Any, Any}, Int64, DataFrames.DataFrame})(ESR::Int64)
    @ GenX ~/.julia/packages/JuMP/b3hGi/src/Containers/macro.jl:194
  [8] #36
    @ ~/.julia/packages/JuMP/b3hGi/src/Containers/container.jl:72 [inlined]
  [9] iterate
    @ ./generator.jl:47 [inlined]
 [10] collect(itr::Base.Generator{JuMP.Containers.VectorizedProductIterator{Tuple{Base.OneTo{Int64}}}, JuMP.Containers.var"#36#37"{GenX.var"#695#699"{JuMP.Model, Dict{Any, Any}, Dict{Any, Any}, Int64, DataFrames.DataFrame}}})
    @ Base ./array.jl:678
 [11] map(f::Function, A::JuMP.Containers.VectorizedProductIterator{Tuple{Base.OneTo{Int64}}})
    @ Base ./abstractarray.jl:2323
 [12] container
    @ ~/.julia/packages/JuMP/b3hGi/src/Containers/container.jl:72 [inlined]
 [13] container(f::Function, indices::JuMP.Containers.VectorizedProductIterator{Tuple{Base.OneTo{Int64}}})
    @ JuMP.Containers ~/.julia/packages/JuMP/b3hGi/src/Containers/container.jl:66
 [14] macro expansion
    @ ~/.julia/packages/JuMP/b3hGi/src/macros.jl:142 [inlined]
 [15] energy_share_requirement(EP::JuMP.Model, inputs::Dict{Any, Any}, setup::Dict{Any, Any})
    @ GenX ~/Documents/github/GenX/src/model/policies/energy_share_requirement.jl:46
 [16] generate_model(setup::Dict{Any, Any}, inputs::Dict{Any, Any}, OPTIMIZER::MathOptInterface.OptimizerWithAttributes, modeloutput::Nothing)
    @ GenX ~/Documents/github/GenX/src/model/generate_model.jl:169
 [17] generate_model(setup::Dict{Any, Any}, inputs::Dict{Any, Any}, OPTIMIZER::MathOptInterface.OptimizerWithAttributes)
    @ GenX ~/Documents/github/GenX/src/model/generate_model.jl:92
 [18] top-level scope
    @ ~/Documents/github/GenX/Example_Systems/pg_test/Run.jl:62
 [19] include(fname::String)
    @ Base.MainInclude ./client.jl:444
 [20] top-level scope
    @ REPL[5]:1
in expression starting at /Users/greg/Documents/github/GenX/Example_Systems/pg_test/Run.jl:62

Hi there.

Trying to compile GenX with Gurobi 9.5, which then requires julia version of gurobi to be bumped from the current 0.9.14 to 0.10.1 (https://github.com/jump-dev/Gurobi.jl).

So I changed this specification in the Project.toml and went ahead trying to build GenX:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.4 (2021-11-19)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(GenX) pkg> activate .
  Activating environment at `~/GenXProject/GenX/Project.toml`

(GenX) pkg> instantiate
ERROR: Unsatisfiable requirements detected for package MathOptInterface [b8f27783]:
 MathOptInterface [b8f27783] log:
 ├─possible versions are: 0.5.0-0.10.5 or uninstalled
 ├─restricted by compatibility requirements with Gurobi [2e9cd046] to versions: 0.10.0-0.10.5
 │ └─Gurobi [2e9cd046] log:
 │   ├─possible versions are: 0.5.0-0.10.1 or uninstalled
 │   └─restricted to versions 0.10.1-0.10 by GenX [5d317b1e], leaving only versions 0.10.1
 │     └─GenX [5d317b1e] log:
 │       ├─possible versions are: 0.1.0 or uninstalled
 │       └─GenX [5d317b1e] is fixed to version 0.1.0
 ├─restricted by compatibility requirements with JuMP [4076af6c] to versions: 0.10.4-0.10.5
 │ └─JuMP [4076af6c] log:
 │   ├─possible versions are: 0.18.3-0.22.0 or uninstalled
 │   └─restricted to versions 0.22 by GenX [5d317b1e], leaving only versions 0.22.0
 │     └─GenX [5d317b1e] log: see above
 └─restricted by compatibility requirements with Cbc [9961bab8] to versions: 0.9.7-0.9.22 — no versions left
   └─Cbc [9961bab8] log:
     ├─possible versions are: 0.4.0-0.9.0 or uninstalled
     └─restricted to versions 0.8 by GenX [5d317b1e], leaving only versions 0.8.0-0.8.1
       └─GenX [5d317b1e] log: see above

Could be I missed something? Any help is appreciated.

Best,
Anton

GenX.jl runs on Julia v1.3.0 and JuMP v0.21.3...
on l. 26

Note that if you have not already installed the required Julia packages, you are using a version of JuMP other than v0.21.4, or you do not have a valid Gurobi license on your host machine, you will receive an error message and Run.jl will not run to completion.
on l. 70 (and 78)

Looking at Project.toml, it looks like all references should be to 0.21.3, unless there's different version compatibility when using Gurobi/CPLEX? I didn't see the relevant lines in the [deps] and [compat] in the Project.toml file to be able to uncomment for Gurbobi/CPLEX (l. 73).

After setting up the GenX model as suggested in the documentation, I'm trying to run the OneZone test case study. All works until the moment when it needs to start writing the outputs. Any hint on this? Any step I'm missing?

Writing Output
ERROR: LoadError: MethodError: no method matching ndims(::Type{DataFrames.LazyNewColDataFrame{Int64}})
Closest candidates are:
ndims(::Type{MutableArithmetics.Zero}) at C:\Julia.julia\packages\MutableArithmetics\6pKCK\src\rewrite.jl:89
ndims(::Base.Generator) at generator.jl:53
ndims(::Number) at number.jl:67
...
Stacktrace:
[1] Base.Broadcast.BroadcastStyle(::Type{DataFrames.LazyNewColDataFrame{Int64}}) at .\broadcast.jl:103
[2] combine_styles(::DataFrames.LazyNewColDataFrame{Int64}) at .\broadcast.jl:420
[3] combine_styles(::DataFrames.LazyNewColDataFrame{Int64}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Float64}}) at .\broadcast.jl:421
[4] materialize!(::DataFrames.LazyNewColDataFrame{Int64}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0},Nothing,typeof(identity),Tuple{Float64}}) at .\broadcast.jl:845
[5] write_power(::String, ::String, ::Dict{Any,Any}, ::Dict{Any,Any}, ::JuMP.Model) at C:\GenX-main\src\write_outputs\write_power.jl:46
[6] write_outputs(::JuMP.Model, ::String, ::Dict{Any,Any}, ::Dict{Any,Any}) at C:\GenX-main\src\write_outputs\write_outputs.jl:63
[7] top-level scope at C:\GenX-main\Example_Systems\SmallNewEngland\OneZone\Run.jl:73
[8] include(::String) at .\client.jl:457
[9] top-level scope at REPL[11]:1
in expression starting at C:\GenX-main\Example_Systems\SmallNewEngland\OneZone\Run.jl:73

By constructing paths to CSV files using the built-in function joinpath (which returns a String) we can remove the sep variable and its being passed as an argument to each load_xxx function. This makes the code simpler.

I am attempting to run the new release (6d2bdb3) using 'Method 2', but I get the following error at precompilation.

Loading packages
[ Info: Precompiling GenX [top-level]
ERROR: LoadError: ArgumentError: Package RecursiveArrayTools not found in current path:
- Run `import Pkg; Pkg.add("RecursiveArrayTools")` to install the RecursiveArrayTools package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:893
 [2] include
   @ ./Base.jl:384 [inlined]
 [3] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1235
 [4] top-level scope
   @ none:1
 [5] eval
   @ ./boot.jl:360 [inlined]
 [6] eval(x::Expr)
   @ Base.MainInclude ./client.jl:446
 [7] top-level scope
   @ none:1
in expression starting at /home/jupyter-anton/GenXProject/GenX/src/GenX.jl:17
ERROR: LoadError: Failed to precompile GenX [top-level] to /home/jupyter-anton/.julia/compiled/v1.6/jl_UNztK7.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:33
 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::Base.TTY, internal_stdout::Base.TTY, ignore_loaded_modules::Bool)
   @ Base ./loading.jl:1385
 [3] compilecache(pkg::Base.PkgId, path::String)
   @ Base ./loading.jl:1329
 [4] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1043
 [5] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:936
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:923
 [7] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [8] top-level scope
   @ REPL[2]:1
in expression starting at /home/jupyter-anton/GenXProject/GenX/Example_Systems/SmallNewEngland/OneZone/Run.jl:30

By adding a line Pkg.add(Pkg.PackageSpec(name="RecursiveArrayTools")) in the julenv.jl makes the error go away.

Best,
Anton

Apologies but I cannot figure out how to run GenX without Gurobi/CPLEX licenses. The instructions state that GenX can be run with Clp/Cbc, but I continue to get errors when trying to run a case that point back to CPLEX. From the instructions, the note below appears in several places, I cannot see how to follow steps that would allow me to run without CPLEX as is indicated. Thanks for any help.

"Note that if you have not already installed the required Julia packages, you are using a version of JuMP other than v0.21.4, or you do not have a valid Gurobi license on your host machine, you will receive an error message and Run.jl will not run to completion."

(in the Dev branch) Line 98 of load_network_data.jl attempts to load the :WACC column from the dataframe associated with Network.csv. This column is generally not present except in the multi-stage examples, and so causes an error.

inputs_nw["transmission_WACC"]= convert(Array{Float64}, collect(skipmissing(network_var[!,:WACC])))

This line should be behind an if statement, probably if setup["MultiStage"]==1 && setup["NetworkExpansion"]>=1 or similar.

In the functionload_cap_reserve_margin_trans from load_cap_reserve_margin.jl, the columns CapRes_* are referenced. PowerGenome no longer creates these columns in the Network.csv file and they don't appear to be used anywhere in GenX. Can someone confirm they aren't used and remove them? Also need to remove from the documentation.

The first term of the objective has

\Pi^{invest} x \overline{\Omega}^{size} x \Omega

but the notation has
\Pi^{invest} : $/(MW year) (a parameter)
\overline {\Omega}^{size} : MW (a parameter)
\Omega : MWh (a decision variable).

Together this would mean that the expression has dimensions of cost * energy / time .

The second term has a different problem,

\Pi^{FOM} : $/(MW year) (parameter)
\overline {\Omega}^{size} : MW (parameter)
\Delta^{total} : MW (decision variable)
this puts the product in $ MW / year, or cost * power / time.

Whereas the third term is

\Pi^{INVEST, energy} : $/(MWh year)
\Omega^{energy} : MWh
so the product is $/year

I think the units of $/year is correct since we're minimizing annual costs.

Example issues:

1. Small hydro technology had no technology flag. As a result, the technology capacity was unconstrained.
2. Settings the THERM and MUST_RUN flags ON for Nuclear produced erroneous results
3. Setting the THERM flag was ON and MUST_RUN flag was OFF for biomass and geothermal technologies produced erroneous results.
4. Setting the STOR flag for small hydro to 2 instead of 1 produced erroneous results.

configure_gurobi.jl does not include "Crossover" in the settings listed at line 68. This means that changing the "Crossover" tag in gurobi_settings.yml currently does nothing.

Lines 63-69 of write_storagedual.jl are

        if setup["ParameterScale"]==1
            dual_values[y,:] = x1[y,:]./inputs["omega"] *ModelScalingFactor
        else
            dual_values[y,:] = x1[y,:]./inputs["omega"] *ModelScalingFactor
        end

The two lines are identical; surely the second should not be scaled?

The heat rate of some NG resources in the "GenX\Example_Systems\RealSystemExample\Generator_data.csv" looks weird. For example, natural gas combustion turbine (R_ID = 7 and 29) has a heat rate of 4.7, which is physically impossible for such a technology. In addition, R_ID = 8 has a heat rate of 0, which is also impossible.

I've been running the SmallNewEngland/OneZone example using commit 773b0d8.

Unless I misunderstand what AnnualSum means here, there's an issue: the AnnualSum column is 0 and but the Nonserved_Energy
column is very non-zero.

See the attached file..

power_balance.csv

We should update solver settings example files to include all options showing default values (copied from the relevant configure_solver/ .jl files for each solve) but with all commented out. And provide commented instructions that if a user wishes to modify these settings, they can uncomment and modify the parameter value. If they do not wish to modify, they can leave them commented out and the default settings will be used.

@nspatank or @sambuddhac please make this update. Thanks.

Updated to julia 1.6.2 and attempting to run new release of GenX example from commit 46b0baf on the main branch I get an error when writing output files.

Excerpt of the run:

➜  GenX git:(main) julia --project="GenX"
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.2 (2021-07-14)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(GenX) pkg> activate .
  Activating environment at `~/Documents/qvist/GenXProject/GenX/Project.toml`

(GenX) pkg> instantiate
Precompiling project...
...
   Installed Reexport ──────────────────── v0.2.0
   Installed TableTraits ───────────────── v1.0.1
   Installed LineSearches ──────────────── v7.1.1
   Installed CSV ───────────────────────── v0.8.5
  Downloaded artifact: FFTW
  Downloaded artifact: ASL
  Downloaded artifact: CoinUtils
  Downloaded artifact: IntelOpenMP
  Downloaded artifact: Rmath
  Downloaded artifact: Clp
  Downloaded artifact: MUMPS_seq
  Downloaded artifact: Bzip2
  Downloaded artifact: Ipopt
  Downloaded artifact: OpenBLAS32
  Downloaded artifact: OpenSpecFun
  Downloaded artifact: Gumbo
  Downloaded artifact: METIS
  Downloaded artifact: libsass
  Downloaded artifact: Cbc
  Downloaded artifact: GLPK
  Downloaded artifact: Cgl
  Downloaded artifact: Osi
  Downloaded artifact: Libiconv
    Building Cbc ─────────→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/8da071e2f96de95decbd551d54db9ac82c0bf4f6/build.log`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1530
  Downloaded artifact: Bzip2
  Downloaded artifact: OpenBLAS32
    Building FFTW ────────→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/1b48dbde42f307e48685fa9213d8b9f8c0d87594/build.log`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1530
    Building NNlib ───────→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/d9f196d911f55aeaff11b11f681b135980783824/build.log`
    Building Clp ─────────→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/3df260c4a5764858f312ec2a17f5925624099f3a/build.log`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1530
  Downloaded artifact: Clp
  Downloaded artifact: MUMPS_seq
  Downloaded artifact: CoinUtils
  Downloaded artifact: Osi
    Building SLEEFPirates → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/67ae90a18aa8c22bf159318300e765fbd89ddf6e/build.log`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1530
ERROR: Unsatisfiable requirements detected for package SIMDPirates [21efa798]:
 SIMDPirates [21efa798] log:
 ├─possible versions are: 0.1.0-0.8.26 or uninstalled
 ├─restricted to versions 0.8.1-0.8 by an explicit requirement, leaving only versions 0.8.1-0.8.26
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left


julia> include("Example_Systems/RealSystemExample/ISONE_Singlezone/Run.jl")
Loading packages
Configuring Solver
Loading Inputs
Reading Input CSV Files
Load_data.csv Successfully Read!
Fuels_data.csv Successfully Read!
Generators_data.csv Successfully Read!
Generators_variability.csv Successfully Read!
About to read Capacity_reserve_margin.csv
Capacity_reserve_margin.csv Successfully Read!
Minimum_capacity_requirement.csv Successfully Read!
Energy_share_requirement.csv Successfully Read!
CO2_cap.csv Successfully Read!
CSV Files Successfully Read In From /home/anton/Documents/qvist/GenXProject/GenX/Example_Systems/RealSystemExample/ISONE_Singlezone/
Generating the Optimization Model
Discharge Module
Non-served Energy Module
Investment Discharge Module
Unit Commitment Module
Dispatchable Resources Module
Storage Resources Module
Storage Investment Module
Storage Core Resources Module
Storage Resources with Symmetric Charge/Discharge Capacity Module
Hydro Reservoir Core Resources Module
Flexible Demand Resources Module
Thermal (Unit Commitment) Resources Module
C02 Policies Module
Energy Share Requirement Policies Module
Capacity Reserve Margin Policies Module
Minimum Capacity Requirement Module
Solving Model
Coin0506I Presolve 2435709 (-936952) rows, 1200916 (-297112) columns and 9662112 (-3702922) elements
Clp0006I 0  Obj 3141.734 Primal inf 349629.83 (62086) Dual inf 76.842493 (512)
Clp0006I 1000  Obj 2127.6447 Primal inf 4950050.5 (71935)
Clp0006I 2000  Obj 2127.6448 Primal inf 4760138.3 (72592)
Clp0006I 3000  Obj 2127.6449 Primal inf 4601195.2 (73361)
.
.
.
Clp0006I 25200  Obj 3458.7899 Dual inf 0.0013071488 (850)
Clp0006I 25400  Obj 3458.7899 Dual inf 0.0014543915 (487)
Clp0006I 25600  Obj 3458.7898 Dual inf 0.00020648187 (69)
Clp0006I 25702  Obj 3458.7898
Clp0000I Optimal - objective value 3458.7898
Clp0032I Optimal objective 3458.789795 - 490919 iterations time 1631.542, Presolve 8.06
LP solved for primal
Writing Output
[3, 16, 36, 44, 52]
1
ERROR: LoadError: ArgumentError: broadcasting requires an assigned BroadcastStyle
Stacktrace:
 [1] copy(bc::Base.Broadcast.Broadcasted{Base.Broadcast.Unknown, Tuple{Base.OneTo{Int64}}, typeof(*), Tuple{Vector{Float64}, JuMP.Containers.DenseAxisArray{Float64, 1, Tuple{Base.OneTo{Int64}}, Tuple{Dict{Int64, Int64}}}}})
   @ Base.Broadcast ./broadcast.jl:899
 [2] materialize(bc::Base.Broadcast.Broadcasted{Base.Broadcast.Unknown, Nothing, typeof(*), Tuple{Vector{Float64}, JuMP.Containers.DenseAxisArray{Float64, 1, Tuple{Base.OneTo{Int64}}, Tuple{Dict{Int64, Int64}}}}})
   @ Base.Broadcast ./broadcast.jl:883
 [3] write_net_revenue(path::String, sep::String, inputs::Dict{Any, Any}, setup::Dict{Any, Any}, EP::JuMP.Model, dfCap::DataFrames.DataFrame, dfESRRev::DataFrames.DataFrame, dfResRevenue::DataFrames.DataFrame, dfChargingcost::DataFrames.DataFrame, dfPower::DataFrames.DataFrame, dfEnergyRevenue::DataFrames.DataFrame, dfSubRevenue::DataFrames.DataFrame, dfRegSubRevenue::DataFrames.DataFrame)
   @ GenX ~/Documents/qvist/GenXProject/GenX/src/write_outputs/write_net_revenue.jl:70
 [4] write_outputs(EP::JuMP.Model, path::String, setup::Dict{Any, Any}, inputs::Dict{Any, Any})
   @ GenX ~/Documents/qvist/GenXProject/GenX/src/write_outputs/write_outputs.jl:136
 [5] top-level scope
   @ ~/Documents/qvist/GenXProject/GenX/Example_Systems/RealSystemExample/ISONE_Singlezone/Run.jl:75
 [6] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [7] top-level scope
   @ REPL[3]:1
in expression starting at /home/anton/Documents/qvist/GenXProject/GenX/Example_Systems/RealSystemExample/ISONE_Singlezone/Run.jl:75

There was also an error setting up the package SIMDPirates:

  Downloaded artifact: Osi
    Building SLEEFPirates → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/67ae90a18aa8c22bf159318300e765fbd89ddf6e/build.log`
┌ Warning: Could not use exact versions of packages in manifest, re-resolving
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1530
ERROR: Unsatisfiable requirements detected for package SIMDPirates [21efa798]:
 SIMDPirates [21efa798] log:
 ├─possible versions are: 0.1.0-0.8.26 or uninstalled
 ├─restricted to versions 0.8.1-0.8 by an explicit requirement, leaving only versions 0.8.1-0.8.26
 └─restricted by julia compatibility requirements to versions: uninstalled — no versions left

Maybe the two errors are related?

I am running 64-bit Ubuntu 20.04.2 LTS.

/Anton

https://github.com/GenXProject/GenX/blob/2a264cc1ac5b17e96c6f31718a014784121df140/julenv.jl#L17-L37

Most of my experience is setting those dependences in the Project.tml so that they will be installed with GenX together. Any consideration to create a new environment?

Especially for the Example_Systems/SmallNewEngland/ThreeZones_MultiStage/ results,
can we use glob patterns to ignore Results directories? See https://linuxize.com/post/gitignore-ignoring-files-in-git/ for how it's done. Or are specific files in those directories that should be included?

Hi,

I am looking at
https://genxproject.github.io/GenX/dev/policies/#Energy-Share-Requirement

On the right-hand side of the equation, there is $\mu^{ESR}{p,z}$ which I couldn't find in the "Notation" section. It seems to be $\mu^{RPS}{p,z}$ instead. Please check.

-Rahman

I've been trying to understand why some of the output functions are very slow.

I think much of it has to do with adding together vectors of different types. For example, in write_net_revenue, about 85% of the time is spent on one line which is simply adding together vectors,

dfNetRevenue.Cost = dfNetRevenue.Inv_cost_MW 
+ dfNetRevenue.Inv_cost_MWh 
+ dfNetRevenue.Fixed_OM_cost_MW 
+ dfNetRevenue.Fixed_OM_cost_MWh 
+ dfNetRevenue.Var_OM_cost_out 
+ dfNetRevenue.Var_OM_cost_in 
+ dfNetRevenue.Fuel_cost 
+ dfNetRevenue.Charge_cost 
+ dfNetRevenue.EmissionsCost 
+ dfNetRevenue.StartCost

It turns out that all of these things are Vector{Float64} except for the third from last, dfNetRevenue.Charge_cost, which is a Vector{Union{Missing, Float32}}.

These arrays with types Missing are initialized in places like write_rsv.jl:

dfRsv = DataFrame(Resource = vcat(inputs["RESOURCES"], "unmet"), Zone = vcat(dfGen[!,:Zone], "na"), Sum = Array{Union{Missing,Float32}    }(undef, G+1))

and it's not obvious to me what the benefit of an array supporting Missing is here. Are we ensuring that any csv file inputs which were not specified show up as missing, which would alert us to an input problem?

This uses black text on a transparent background, so when looking at a browser in Dark Mode it's not readable.
(I'll submit a PR to fix this shortly)

Attempting a clean install of GenX and Julia on December 9th 2021.

When I attempt to initiate Julia to work with GenX I get the following error when trying to install the required packages with instantiate:

One made it into the root directory of the dev branch.

The Network.csv file is supposed to have a column Network_zones. According to the documentation it is a mandatory column with values "Specified as z* where * is a number of the zone."

Based on the code it looks like the z* format isn't necessary. The number of zones is specified entirely by the length of this column array.

The rest of the data in this file deals with transmission lines between zones, each of which is represented by a single row. Adding a column that is unrelated to the other data in each row is potentially confusing and a pain to create.

Is it possible to determine the number of zones based on the number of z* column names instead of the Network_zones column?

I just ran into an error while running GenX with an new case study that relates to missing column "region" in Generators_data.csv - apparently this is used in write_energy_revenue.jl. However, this column is not described in the documentation. Same for "cluster" column in Generators_data.csv

Similar to capacity reserve margin, it would be useful to output the shadow price of the operating reserve requirement constraints (spinning and regulation) and include the revenues from these constraints in the "NetRevenue.csv" file.

I'm having trouble getting the SmallNewEngland/ThreeZones example to work. The other two SmallNewEngland examples do work. I'm running GenX version b0ae69f (HEAD, tag: v0.1.0) using Julia version 1.6.1.
The issue is the same on the most recent main commit, 46a495b.

I think the error might be related to missing data? It comes with an error:
Reading Input CSV Files
┌ Warning: thread = 1 warning: only found 14 / 15 columns around data row: 4. Filling remaining columns with missing
└ @ CSV ~/.julia/packages/CSV/Zl2ww/src/file.jl:612
Network.csv Successfully Read!
[...]
CSV Files Successfully Read In From /home/jacob/GenX/Example_Systems/SmallNewEngland/ThreeZones/
ERROR: LoadError: MethodError: no method matching rename!(::DataFrames.DataFrame, ::Vector{Any})
Closest candidates are: [...]

in expression starting at /home/jacob/GenX/Example_Systems/SmallNewEngland/ThreeZones/Run.jl:40
I'm using the Clp optimizer.

Possible issue in the costs.csv output:

There seems to be an issue with adding up the zone-specific costs. When I run the SmallNewEngland/OneZone example (with all default settings, on the current main branch 935f19efce8afe7344efe615bf2a19c39c68883c or on the Dev branch aa6fd65e063ac1f3b5d7677b682671a80e43d214) I get

Costs,      Total, Zone1
cTotal,    2.46e10, 1.46e10
cFix,      2.46e10, 2.46e10
cVar,       1.39e7,  1.39e7
cNSE,0.0,0.0
cStart,0.0,0.0
cUnmetRsv,0.0,-
cNetworkExp,0.0,-

(some decimals have been truncated and formatted to protect the innocent)
The Zone1 cost total is lower than the fixed cost, which seems wrong. It does look like the (Total, cTotal) cost is the correct summation.

Wilson was able to reproduce this output.

Hi,

Thanks for posting GenX as OpenSource, it is really a fantastic resource. I am setting up an ultra-simple model with only one zone, one CCGT power plant and onshore wind. This is just to build the simplest model to then gradually include more realistic features. My setup is Ubuntu and Clp. I have turned off all "extras" from the genx config file (ie. no CO2, no unit commitment, etc.) and accordingly only use the "required" input files for the model.

The model runs fine and an optimal solution is found (actually the status.csv is written and shows and OPTIMAL outcome). However, the code finds an error when writing the other output files and particularly cost.csv as it is apparently unable to find eTotalCVarIn in the model. My guess is this is due to the model not having any storage (I am assuming is CVarIn is the cost of charging storage devices). I attach a sample output of the GenX instance below. The output writing code is stopped there and only status.csv is written.

As a matter of fact, other elements in the optimization seem to be working fine. I did some pre-debugging deleting the references to eTotalCVarIn in write_cost.jl and the model runs fine and prints all the outputs correctly when that is done.

Not sure if this is just my setup or it may be something worth investigating. I understand most models will be run with some form of storage anyway, but I thought it may be worth mentioning it.

Best

Daniel

Sample output follows....

Configuring Solver
Loading Inputs
Reading Input CSV Files
Network.csv Successfully Read!
Load_data.csv Successfully Read!
Fuels_data.csv Successfully Read!
Generators_data.csv Successfully Read!
Generators_variability.csv Successfully Read!
CSV Files Successfully Read In From /home/daniel/Documents/ProjectGENX/GenX/Example_Systems/Daniel_Trial_Systems/Fifth_Ireland/
Generating the Optimization Model
Discharge Module
Non-served Energy Module
Investment Discharge Module
Dispatchable Resources Module
Thermal (No Unit Commitment) Resources Module
C02 Policies Module
Solving Model
Coin0506I Presolve 52560 (-17522) rows, 26284 (-1) columns and 175200 (-17522) elements
Clp0006I 0 Obj 437.43131 Primal inf 29799.729 (17520) Dual inf 790.1623 (2)
Clp0006I 487 Obj 0.34767904 Primal inf 26764.806 (8760)
[...]
Clp0006I 19185 Obj 451.22294
Clp0000I Optimal - objective value 451.22294
Coin0511I After Postsolve, objective 451.22294, infeasibilities - dual 0 (0), primal 1.1281284e-06 (3)
Coin0512I Presolved model was optimal, full model needs cleaning up
Clp0006I 0 Obj 451.22294 Primal inf 7.915044e-07 (3)
Clp0006I 3 Obj 451.22294
Clp0000I Optimal - objective value 451.22294
Clp0032I Optimal objective 451.2229419 - 19188 iterations time 1.842, Presolve 0.05
LP solved for primal
Writing Output
ERROR: LoadError: KeyError: key :eTotalCVarIn not found
Stacktrace:
[1] write_costs(::String, ::String, ::Dict{Any,Any}, ::Dict{Any,Any}, ::JuMP.Model) at /home/daniel/.julia/packages/JuMP/YXK4e/src/JuMP.jl:936
[2] write_outputs(::JuMP.Model, ::String, ::Dict{Any,Any}, ::Dict{Any,Any}) at /home/daniel/Documents/ProjectGENX/GenX/src/write_outputs/write_outputs.jl:70
[3] top-level scope at /home/daniel/Documents/ProjectGENX/GenX/Example_Systems/Daniel_Trial_Systems/Fith_Ireland/Run.jl:73
[4] include at ./boot.jl:328 [inlined]
[5] include_relative(::Module, ::String) at ./loading.jl:1105
[6] include(::Module, ::String) at ./Base.jl:31
[7] include(::String) at ./client.jl:424
[8] top-level scope at REPL[1]:1
in expression starting at /home/daniel/Documents/ProjectGENX/GenX/Example_Systems/Daniel_Trial_Systems/Fith_Ireland/Run.jl:73

On the Dev branch, this causes runs to fail if they were not configured with MultiStage in mind. There should be a default setting in configure_settings.jl, probably 0.

I start the julia repl and run

include("Run.jl")

I get an error:

julia> include("Run.jl")
Loading packages
[ Info: Precompiling GenX [top-level]
Clustering Time Series Data...
Reading Input CSV Files
ERROR: LoadError: BoundsError: attempt to access 0-element Array{Int64,1} at index [1]
Stacktrace:
 [1] getindex at ./array.jl:744 [inlined]
 [2] load_network_data(::Dict{Any,Any}, ::String, ::String, ::Dict{Any,Any}) at /home/[...]/GenX/src/load_inputs/load_network_data.jl:35
 [3] load_inputs(::Dict{Any,Any}, ::String) at /home/[...]/GenX/src/load_inputs/load_inputs.jl:45
 [4] cluster_inputs(::String, ::Dict{Any,Any}, ::Bool) at /home/[...]/GenX/src/time_domain_reduction/time_domain_reduction.jl:489
 [5] cluster_inputs(::String, ::Dict{Any,Any}) at /home/[...]/GenX/src/time_domain_reduction/time_domain_reduction.jl:458
 [6] top-level scope at /home/[...]/GenX/Example_Systems/SmallNewEngland/OneZone/Run.jl:42
 [7] include at ./boot.jl:328 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1105
 [9] include(::Module, ::String) at ./Base.jl:31
 [10] include(::String) at ./client.jl:424
 [11] top-level scope at REPL[1]:1
in expression starting at /home/[...]/GenX/Example_Systems/SmallNewEngland/OneZone/Run.jl:39

The same procedure in the two other SmallNewEngland examples works fine, generates Results, etc.

My environment: Ubuntu 20.10, Julia 1.3.1, if I start the repl in this directory:

v1.3) pkg> st
    Status `~/.julia/environments/v1.3/Project.toml`
  [336ed68f] CSV v0.8.5
  [e2554f3b] Clp v0.8.4
  [aaaa29a8] Clustering v0.14.2
  [861a8166] Combinatorics v1.0.2
  [a93c6f00] DataFrames v1.1.1
  [b4f34e82] Distances v0.10.3
  [e30172f5] Documenter v0.26.3
  [60bf3e95] GLPK v0.14.11
  [2e9cd046] Gurobi v0.9.13
  [7073ff75] IJulia v1.23.2
  [b6b21f68] Ipopt v0.6.5
  [4076af6c] JuMP v0.21.3
  [fdba3010] MathProgBase v0.7.8
  [91a5bcdd] Plots v1.6.12
  [2913bbd2] StatsBase v0.33.8
  [ddb6d928] YAML v0.4.7
  [ade2ca70] Dates
  [37e2e46d] LinearAlgebra

I noticed that there's one line in Run.jl which was commented out here #include(environment_path) ... which was not commented in the two other working examples. Things work when I uncomment it.

In all the example Run.jl scripts, the line myinputs = Dict() does nothing because it is immediately
overwritten on the next line by the load_inputs function. We can simplify the scripts by eliminating this line.

Looks like a workflow file that automatically updating and deploys the document website is missing. Something like this example. I may be able to help with that if needed.

Update: maybe it is not necessary.

Windows 10 Pro
Version : 10.0.19042 Build 1902
Julia 1.6.1
CSV v0.6.1
Cbc v0.8.0
Clp v0.8.4
Clustering v0.14.2
Combinatorics v1.0.2
DataFrames v0.20.2
DataStructures v0.17.20
Distances v0.10.3
Documenter v0.24.11
DocumenterTools v0.1.11
GLPK v0.14.12
Ipopt v0.6.5
JuMP v0.21.4
MathProgBase v0.7.8
StatsBase v0.33.8
YAML v0.4.7
Dates
LinearAlgebra

How to Replicate?

1. Julia navigates to the root file of the cloned GenX project.
2. julia> include(“/Run.jl”)
3. And then I get the error below

Error report

ERROR: LoadError: MethodError: no method matching ndims(::Type{DataFrames.LazyNewColDataFrame{Int64}})
Closest candidates are:
ndims(::LinearAlgebra.UniformScaling) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\LinearAlgebra\src\uniformscaling.jl:87
ndims(::CartesianIndices) at multidimensional.jl:356
ndims(::DataFrames.AbstractDataFrame) at C:\Users\PereEscortell.julia\packages\DataFrames\S3ZFo\src\abstractdataframe\abstractdataframe.jl:29
...
Stacktrace:
[1] Base.Broadcast.BroadcastStyle(#unused#::Type{DataFrames.LazyNewColDataFrame{Int64}})
@ Base.Broadcast .\broadcast.jl:103
[2] combine_styles(c::DataFrames.LazyNewColDataFrame{Int64})
@ Base.Broadcast .\broadcast.jl:420
[3] combine_styles(c1::DataFrames.LazyNewColDataFrame{Int64}, c2::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0}, Nothing, typeof(identity), Tuple{Float64}})
@ Base.Broadcast .\broadcast.jl:421
[4] materialize!(dest::DataFrames.LazyNewColDataFrame{Int64}, bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{0}, Nothing, typeof(identity), Tuple{Float64}})
@ Base.Broadcast .\broadcast.jl:891
[5] write_power(path::String, sep::String, inputs::Dict{Any, Any}, setup::Dict{Any, Any}, EP::JuMP.Model)
@ GenX C:\Users\PereEscortell.julia\environments\GenX\GenX\src\write_outputs\write_power.jl:46
[6] write_outputs(EP::JuMP.Model, path::String, setup::Dict{Any, Any}, inputs::Dict{Any, Any})
@ GenX C:\Users\PereEscortell.julia\environments\GenX\GenX\src\write_outputs\write_outputs.jl:63
[7] top-level scope
@ C:\Users\PereEscortell.julia\environments\GenX\GenX\Example_Systems\SmallNewEngland\Onezone\Run.jl:73
[8] include(fname::String)
@ Base.MainInclude .\client.jl:444
[9] top-level scope
@ REPL[130]:1
in expression starting at C:\Users\PereEscortell.julia\environments\GenX\GenX\Example_Systems\SmallNewEngland\Onezone\Run.jl:73

We have long-duration storage constraints for hydrogen storage as well as for hydro storage. We need to be able to turn ON the LDS flag for each technology separately. One way to do it is to add STOR = 3 for hydrogen storage and HYDRO = 2 for hydro storage with long-duration constraints.

configure_settings is designed to apply some default settings in case they are not listed in genx_settings.yml files.
It is defined in configure_settings.jl, and is exported and included in GenX.jl. However, that's it -- it never actually gets used. I think it could become useful.

The typical Run.jl files have

genx_settings = joinpath(settings_path, "genx_settings.yml") #Settings YAML file path
mysetup = YAML.load(open(genx_settings)) # mysetup dictionary stores settings and GenX-specific parameters

but if we instead were to use
mysetup = configure_settings(genx_settings)
then the function would be applied. Yea? Nay?