nrel / rev Goto Github PK

Bug Description
reV collect fails with a point index error.

Full Traceback
Traceback (most recent call last):
File "/home/twillia2/.conda-envs/rev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/twillia2/github/reV/reV/handlers/cli_collect.py", line 304, in
main(obj={})
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/twillia2/github/reV/reV/handlers/cli_collect.py", line 161, in collect
file_prefix=file_prefix)
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 583, in collect
mem_util_lim=mem_util_lim)
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 299, in collect_dset
dc._collect()
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 273, in _collect
log_mem(logger, log_level='DEBUG')
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 271, in _collect
f_out, f_source, fp)
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 271, in _collect
f_out, f_source, fp)
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 238, in _collect_chunk
out_slice = self._get_gid_slice(self._gids, source_gids)
File "/home/twillia2/github/reV/reV/handlers/collection.py", line 177, in _get_gid_slice
raise HandlerRuntimeError(emsg)
reV.utilities.exceptions.HandlerRuntimeError: GID indices for source file are not sequential in destination file!

To Reproduce
Steps to reproduce the problem behavior

1. On eagle, go to '/projects/rev/new_projects/nsrdb_ensemble/smallrun/nsrdb_sf0'
   run 'reV -c config_collect.json' collect

Expected behavior
Chunked HDF files are combined into single yearly HDF files.

Wish to have exclusion multipliers between 0-1.

Implement PySAM (deprecate PySSC). This seems to be the way SAM is moving and the interface is much more pythonic.

Currently the pipeline status is keyed by cli command, which assumes each module is only run once. collect might be have to run twice.

@grantbuster
The processing algorithm to produce the representative profiles should consider the number of pixels within a region when calculating the representative profile.

If you want to see how it was done in reV V1, please see: https://github.nrel.gov/reV/reV/blob/dev/reV/rep_profiles.py#L103

Just to be clear, the selected "representative profile" should be native and unchanged from the NSRDB / WTK. Only the "meanoid" should have weighting considered.

• ORCA/offshore integration
• Cost-based exclusions
• Windfarm saturation and spatial proximity

Gotta make sure reV can run CSP. I think it should just include setting up the SAM handlers to pull the correct data and setting the resource data structs (hopefully) the same as PV.

When multi-year collection is run from the CLI, an empty directory is made in the directory where the cli was run from. The directory name is the output file name.

Multi-year collection job: /lustre/eaglefs/scratch/gbuster/rev/ri_my_mean_test_data

Weird directories created:

Would be best to allow ExclusionsMask to take a None argument and return all 1's or True's.

Source all variables form 5-min .h5 files

We might be able to significantly streamline our cli by using ctx.forward

Something to look into.

I'd like to remove xarray as a dependancy if possible. It doesn't install properly (it doesn't seem to have dask as a required dependency, which then causes certain functions to fail.

Can we do this when we migrate from geotiffs to hdf5 files for the exclusions?

Possible that json is more effective for project points and (econ) site data inputs.

I followed Option 1 (https://github.com/NREL/reV/wiki/Installation-Guide#option-1-pip-install-the-most-recent-version-of-master-recommended-for-analysts) from the wiki, and immediately received the error below when running:
reV --help

Traceback (most recent call last):
  File "/home/ngrue/.conda-envs/rev/bin/reV", line 6, in <module>
    from reV.cli import main
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/cli.py", line 7, in <module>
    from reV.batch.cli_batch import from_config as run_batch_from_config
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/batch/cli_batch.py", line 6, in <module>
    from reV.generation.cli_gen import main
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/generation/cli_gen.py", line 15, in <module>
    from reV.generation.generation import Gen
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/generation/generation.py", line 13, in <module>
    from reV.SAM.generation import PV, CSP, LandBasedWind, OffshoreWind
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/SAM/generation.py", line 19, in <module>
    from reV.SAM.econ import LCOE, SingleOwner
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/SAM/econ.py", line 13, in <module>
    from reV.SAM.windbos import WindBos
  File "/home/ngrue/.conda-envs/rev/lib/python3.7/site-packages/reV/SAM/windbos.py", line 7, in <module>
    from PySAM.PySSC import ssc_sim_from_dict
ModuleNotFoundError: No module named 'PySAM'

Drop return_obj kwarg and always return for simplisity

Currently $/MW

• Use economies of scale to compute $/MW
• Compute capital costs as a function of capacity

In config_aggregation.json there is a static input variable "power_density". Can you please add (open up) this variable to the "project_points.csv" so that it can vary in space?

Need to feed Balance of System cost to SAM on a site by site basis

Can you please add the ability to incorporate regional capital cost multipliers? These would be applied pre LCOE calculation.

Example multipliers below.

SAM has some outputs that produce arrays that are not "full" (8760 multiple) timeseries.

For example Cash Flow Revenue (cf_total_revenue) is of shape (analysis_period, sites)

Bug Description
Collect fails on master branch. Had worked on gb/misc_features before recent merge.

Full Traceback
cat collect_nsrdb_sf0_gen_2015_1841883.e

Traceback (most recent call last):
File "/home/twillia2/.conda-envs/rev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/twillia2/github/reV/reV/handlers/cli_collect.py", line 305, in
main(obj={})
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/twillia2/.conda-envs/rev/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/twillia2/github/reV/reV/handlers/cli_collect.py", line 161, in collect
file_prefix=file_prefix, low_mem=True)
TypeError: collect() got an unexpected keyword argument 'low_mem'

To Reproduce
Steps to reproduce the problem behavior

1. On eagle, go to '/projects/rev/new_projects/nsrdb_ensemble/smallrun/nsrdb_sf0'
2. run 'reV -c config_collect.json' collect

Expected behavior
Chunked HDF files are combined into single yearly HDF files.

@grantbuster @MRossol
When running generation, is there a list of the variables I can get out in addition to the usual cf_mean and cf_profile for the output_request?

For example, I'll want average wind speed, but not sure how to state that. For PV, we'd want average GHI (ghi_mean). What is the equivalent multi-year mean wind speed?

It sounds like we will need resource means in the SC_Table for ReEDS, can we pass this through by default (I assume it is computed by default in gen?)

Why this feature is necessary:
We have had many cases where users want to pass through resource data (ghi, wind speed, etc...) for downstream analysis. It makes sense that the generation module would make this possible. Even though this is feature drift, the generation architecture should make this easy to implement.

A possible solution is:
In the generation module, copy the requested resource data to the outs dictionary. Use a method similar to the resource mean computation.

I have considered the following alternatives:
Scripts. Gen architecture is better.

Collect CLI would remove node files that get collected. Flag option to keep files.

The nsrdb_ensemble runs (fixed vs tracking, all years, full extent) successfully wrote output files with correct dimensions and datasets, but failed when running collect to combine the yearly files. The files initialized fine, but failed with a memory error.

How should we go about fixing this issue, and how do we determine memory requirements?

Enable the ability to run Gen and Econ from a dictionary instead of only from a json path

@grantbuster I think we should update the logging defaults for the CLIs to always print to Stdout and then to add printing to a file_handler if a log_file is supplied as mentioned in reVX

Need to update the NSRDB handler or create a 5min handler for 2018 + data

@grantbuster
We currently can't run Gen or Econ using HSDS as there is no way to pass the hsds kwarg through Gen.reV_run or Econ.reV_run (or SAM).

Resource class:

Gen.reV_run error:

Any chance you can hotfix this ASAP as I need it for a NARIS Mexico deliverable.

Why this feature is necessary:
When we open source the repo, users should be able to access examples of how to use common reV class frameworks in a similar fashion to Pandas.

A possible solution is:
Document use case examples in class doc strings for commonly used classes (handlers, gen, econ, SAM, etc...). The Pandas doc string examples look like this:

I have considered the following alternatives:
Our current example folder is good but it's always nice to have examples in the source code.

Charge code
WEPG.10348.08.01.02

Make it so reV will attempt to export unknown SAM variables.

• remove slots dict output thing
• Set default output shape (assume array of scalars?)
• set default dtype (float32).

Pipeline monitoring is structured primarily for Eagle runs right now. Make more generic for local runs for AWS.

The current multi-year collection group framework will make it difficult for downstream modules (SC aggregation) to parse data using our resource handlers. Currently, it is easiest if the Gen module computes LCOE so everything is in the same file. We should consider some other options:

1. If multi-year is collecting from gen and econ files, check to see if the datasets can be part of the same group, and then if so write all datasets to the top h5 level (no group).
2. Implement a group handling method for the resource handler. This could be as simple has having the dsets property return the group/dset names recursively so the downstream modules can easily select and extract datasets within groups.

Make updates to reV to enable open-sourcing:

• #2 Exclusions in .h5
• Upload exclusions to AWS
• #22 and #25 Local pipeline and testing on AWS
• #1 PySSC to PySAM
• Add AWS examples to docs as well as cleaner docs page in general

Json's drive me crazy sometimes....

Save current geotif exclusion mask in a single .h5 file.

.h5 file format:
CONUS_Exclusions.h5
/meta (group)

• attr['profile'] = geotif profile
  /lat_v (n, 1)
  /lat_a (n, m)
  /lon_v (m, 1)
  /lon_a(n, m)
  /exclusions (group)
  /landcover (n, m)
• dtype = unint8
• attrs = units, source, creation_date, owner
  /tech_maps (group)
  /nsrdb (n, m)
  /wtk (n, m)

At the supply curve point level:

• If reV builds a point, we want to limit or eliminate the up-wind resource (exclude up-wind SC points)
• Downwind di-incentive (economic hit, derate AEP)
• Have a "predominant" direction layer from the power rose (Atlas)
• When do we implement, how are "farms" (supply curve points) prioritized
• Use site based LCOE
• Use 8 cardinal directions or a fan for NN search
• Distance threshold (used to computer k for NN search)

@grantbuster
@WilliamsTravis is trying to run the full extent for NSRDB, but the generation module is either timing-out or crashing at consistent iterations.

2 things I noticed are:

1. Several runs are terminating at the 108th iteration (which is roughly 10% of the total # of iterations for this configuration of 1086 iterations)
2. Other runs that make it past the 108th iteration are timing out, which isn't an error, but generation seems like it is going very slow...?

Please reference the project logs at:

/projects/rev/new_projects/nsrdb_ensemble/set1_sf0/stdout

Referencing the above list:

1. Log "set1_sf0_gen_2018_04_1781814.o" is an example of a job crashing at the 108th iteration. "set1_sf0_gen_2006_02_1781750.o" is another example.

2. Log "set1_sf0_gen_1998_03_1781710.o" doesn't fail at the 108th iteration, but times out after 2 hours only reaching iteration 213 of 1086. It seems like only making it to the 213th iteration in 2 hours seems slow...?

Any thoughts on these?

Refactor Supply Curve

Remove supply curve points during aggregation that are below an available capacity threshold

combine run_direct and run_smart so that whenever nworkers is set to 1 serial runs are used

Can you please enable "legacy" exclusion types? Effectively, this means that we will have areas that are 100% excluded and areas that are 50% excluded.

FYI - @nickwg03

Need to update supply curve to handle offshore points:

• Add offshore flag to sc table
• Skip aggregation for offshore points
• Update LCOE with offshore LCOT
• Onshore tie-in points will be added to trans_table as sc_gids

Improve the default mem request vs partition request to handle the issue that SLURM limits node memory to the memory request integer.

@grantbuster this is related to the run last week where the memory filled up before the job failed. I had a similar issue even after what I thought would have fixed the memory.

The job is at: /scratch/ngrue/reeds_runs/landwind_v2_fix_attempt

The memory looks like it keeps swelling before failing:

INFO - 2019-11-15 18:23:53,577 [execution.py:721] : Parallel run at iteration 1115. Memory utilization is 90.773 GB out of 99.647 GB total (91.1% used, limit of 100.0%)
INFO - 2019-11-15 18:29:11,039 [execution.py:721] : Parallel run at iteration 1151. Memory utilization is 93.724 GB out of 99.647 GB total (94.1% used, limit of 100.0%)
INFO - 2019-11-15 18:34:35,956 [execution.py:721] : Parallel run at iteration 1187. Memory utilization is 96.025 GB out of 99.647 GB total (96.4% used, limit of 100.0%)
ERROR - 2019-11-15 18:34:40,042 [generation.py:1119] : reV generation failed!
Traceback (most recent call last):
  File "/home/ngrue/reV/reV/generation/generation.py", line 1117, in reV_run
    mem_util_lim=1.0, **kwargs)
  File "/home/ngrue/reV/reV/utilities/execution.py", line 807, in execute
    manager.run(**kwargs)
  File "/home/ngrue/reV/reV/utilities/execution.py", line 768, in run
    futures = self.gather_and_flush(i, futures)
  File "/home/ngrue/reV/reV/utilities/execution.py", line 711, in gather_and_flush
    self.obj.out = [future.result() for future in futures]
  File "/home/ngrue/reV/reV/utilities/execution.py", line 711, in <listcomp>
    self.obj.out = [future.result() for future in futures]
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Why this feature is necessary:
If someone accidentally collects an incomplete list of dsets (misses some gen outputs), the node files will be purged and some data could be lost.

A possible solution is:
Raise error if collection dsets != gen dsets.
Do not purge if collection dsets != gen dsets.

I have considered the following alternatives:
Always running with the do not purge option but this seems annoying.