pyomo / mpi-sppy Goto Github PK

Toward 100 iterations or so, cross scenario cuts being injected into the sub-problems yields negative gaps and other very odd/wrong behaviors.

To start, I am running the latest main branch (presently 52cc831).

In Pyomo it is sometimes convenient to introduce arbitrary decision variables and just let the solver ignore them. I have done this in the attached example -- the variable sales[0] is arbitrary, and I introduced it solely to avoid needing special logic for setting up the root node in the scenario tree.

inventory.tar.gz

When I execute run.sh, I encounter an issue in the Update_W function of spbase.py:

https://github.com/Pyomo/mpi-sppy/blob/main/mpisppy/phbase.py#L224

The issue is that the arbitrary decision variable is not assigned a value by the solver, so the code attempts to subtract a float from None. It is evident to me that somebody else has already encountered this issue given the commented print lines preceding the subtraction.

Intuitively, if I force sales[0] == 0 (a constraint that is commented out in inventory.py), then there is no issue. Unintuitively, the issue also goes away if I change --bundles-per-rank=0 to --bundles-per-rank=1 or --bundles-per-rank=3.

It took me a while to figure out why mpi-sppy was complaining. I suppose it would be nice for such a condition to be automatically detected and handled. Alternatively, it would probably save someone some time down the road if the resulting TypeError were captured and given some additional messaging. Something like "Check that your instance does not have arbitrary decision variables."

[#13 in old repository]

DLW: We should convert almost all output to logging to enable:

standardized inclusion of who is doing the output, and
easier control over verbosity
wrap TicTocTimer for easier logging

In preparation for Pyomo 6, we need to import pysp as its own package.

Ben: Probably should be driven by the length of the nonant vector

We should attach all the objects we currently attach to scenario subproblems to a single block on those subproblems, perhaps named mpisppy.

This will ensure we have exactly one name clash to worry about, and the internal mpisppy names can be less verbose.

We should try out and then support the not-yet-released automated persistent interface in Pyomo.

...to pass in a scenario name and then arbitrary kwargs.

To not run on forks, and also to run on PRs from forks

Thanks for this great library!

I'm interested in modeling multi-stage problems as ConcreteModels so that I have full flexibility to modify
them in python.

Thanks to the examples I have been able to adjust newsvendor and farmer to my needs. However this are
only two-stage problems and I'm now stuck trying to convert the financial planning example to a direct model.

I have studied the docs but not everything is clear yet, an example for a concrete multi-stage model would help me a lot. Is there such a model in the examples that I overlooked or can someone share such an example?

Thanks

An issue for the code review of phbase is that we need to re-work PH_Prep. It has evolved to a strange state with argument names that are misleading and behavior that is not what one would expect. Unfortunately, there is code using the names and relying on the behavior (APH, and various spokes).

Attempting to directly solve the extensive form of the hydro example (three stages) fails while making the comms (even though this is inherently serial).

import hydro
from mpisppy.opt.ef import ExtensiveForm

options = {"solver": "Gurobi"}
num_scenarios = 9
cb_data = [3, 3]
all_scenario_names = [f"Scen{i+1}" for i in range(num_scenarios)]
scenario_creator_options = {"cb_data": cb_data}
ef = ExtensiveForm(
    options,
    all_scenario_names,
    hydro.scenario_creator,
    scenario_creator_options=scenario_creator_options,
)

The issue is caused by the inadequate knowledge of the tree structure within EF:

Traceback (most recent call last):
  File "test.py", line 16, in <module>
    scenario_creator_options=scenario_creator_options,
  File <path>/mpisppy/opt/ef.py", line 57, in __init__
    cb_data=cb_data,
  File "<path>/mpisppy/spbase.py", line 111, in __init__
    self.create_communicators()
  File "<path>/mpisppy/spbase.py", line 329, in create_communicators
    raise RuntimeError(f"Tree node '{nodename}' not in node list {self.all_nodenames}")
RuntimeError: Tree node 'ROOT_0' not in node list ['ROOT']

The parameter scen_name_list of mpisppy.scenario_tree.ScenarioNode.__init__ is said to be optional and no longer used, but cannot have a default value and be followed by positional parameters. Constructing a ScenarioNode gives TypeError: __init__() missing 1 required positional argument: 'scen_name_list' unless you supply a value. Is there any reason not to change the constructor signature ASAP?

This should not reference the main branch as it doesn't exist. We should probably be using a released version.

[#56 in old repository]

JPW: Basic logic should be moved to sputils.py, and the reset logic should be implemented in L-shaped and (maybe) EF, at a minimum.

These should exclusively be in the user-interface now.

This trouble is sporadic, but sometimes can be seen with netdes 10-30-L-01 with cross-scenario)

We need to centralize and expand (perhaps drawing from Egret) how we check for the existence of solutions and a sane termination condition. The conditional in phbase.py is getting rather un-weidly. At a minimum, it should be moved to a util module, and we should carefully consider the full range of returns conditions that we should require.

In preparation for Pyomo 6, we need to write our own timer. For one thing, we need to be able to have it go silent.

is there an indicator for the benders object that tells the user whether it converged or not?

Right now, incumbent spokes store the nonanticipative variables when a new solution is found. They should really store all the Vars on the Pyomo model to ensure that the original solution found can be recovered easily. This may be useful for certain classes of problems, such as those with integer recourse actions.

As-is, we fail out for certain sub-optimal termination conditions (like time limit).

Ideally the replacement for both would

1. Inherit from SPOpt (not PHBase like XhatTryer does)
2. Be compatible with both MMW and the spoke incumbent finders (xhatspecific, xhatshufflelooper, xhatlooper)

LShapedCutGenerator and LShaped need to rename some attributes due to Pyomo/pyomo#1828.

[#15 in old repository]

DLW: After everything is up and running, we should probably add a feature that allows for transmission of one vector per bundle to the spokes. With large bundles, this could be much more efficient than sending all scenarios.

Currently mpi-sppy handles multi-stage models in a somewhat brittle way which requires the scenario and node names to line up just-so.

We only use sympy.factorint in a single file: https://github.com/Pyomo/mpi-sppy/blob/main/mpisppy/confidence_intervals/ciutils.py.

We should find an alternative to eliminate this dependency.

[#57 in old repository]

DLW: lagranger_bounder.py shares code with LagrangianOuterBound; this probably should be factored. Related: we should think about "elevating" the rho-scaling code to ph_base so that extensions and/or other hubs could use it.

Could use some work -- it's basically only valid when each subproblem has its own rank.

Multistage problems could be bundled to look like a two-stage problem, enabling better use of non-multistage code.

We need a hierarchy of solver options and a service function to manage the options. E.g., an EF could look first for EF_solvername but if it is not found, use solvername, etc. DLW has some ideas about this.

Also: many examples (particularly older examples) have something like
options["subsolvedirectives"] = None
but if it was not None, nothing would change. The examples should probably be updated.

PHBase.PHoptions["bundles_per_rank"] should be (optionally) calculated based on an easier-to-understand number like total number of bundles or bundle size.

...in mpi-sppy classes for options as well as for helping set up spokes/hubs (e.g., baseparsers.py).

https://pyomo.readthedocs.io/en/latest/developer_reference/config.html

Observed on Safari and Firefox. Each bullet appears as its own paragraph. For instance, see the page on the scenario creator function.

I am encountering the runtime error from the issue title as the progressive hedging algorithm attempts to terminate. The file below contains the text I see written to the terminal.

winterization.stdout.txt

The particular section of code that raises the error is here.

In those lines of code, I see the comment saying "this should be feasible here; if not we've done something wrong". This check was added a little more than a year ago. Have the developers since encountered this issue? Is it possible that the settings I specify are causing this issue?

We don't have mpi4py as a dependency, which forces a manual installation. Which can annoy end-users.

[#14 in old repository]

DLW: We need a four-stage test

Hi,

I've been wracking my brain trying to figure out why my nonant constraints aren't being respected. So far the best explanation I have is that it is due to some of the nonant variables in the same node having different names in different scenarios. I've looked through the codebase but couldn't quite determine for sure whether this would be an issue or not.

Making all the names match could potentially require quite a large rewrite for me, so I wanted to make sure I am actually on to something before I try it.

Any help is appreciated!

• CVaR
• Chance constraints
• Enhanced slamming heuristics (var by var up/down min/max over scenarios or to bounds)
• Linearized proximal term (PR #43)
• Cycle detection for PH (what to do about it? Slam, perturb w, etc?)

Hi there, a quick question, is there a way to get value for variables that not only subject to non-anticipitivity? Please help... Thank you

I am not sure whose bug this is (pyomo? Cplex? mpi-sppy?), but it could well be mpi-sspy but it might also be cplex, or numeric troubles in Egret.

in examples/acopf
mpiexec -np 2 python -m mpi4py ccopf2wood.py 2 3 2 0 cplex
works, but cplex_persistent does not; I kind of think it works with xpress_persistent.

In the Pyomo config system version, we should better support command line setting of spoke_sleep_time (note that is a keyword
argument for the spoke constructor); also, the heuristic in spcommunicator might be ignoring.

• New user function for SPBase called write_scenario_solution (like create_scenario).
• A new service for Hubs which selects the best incumbent and instructs the associated Spoke opt object to call the write_scenario_solution function on SPBase.
• Default implementation of write_scenario_solution which blurts out the Pyomo model (look at PySP)
• Create functionality examples

Some code:

in PHBase.Compute_Xbar

This would weaken mpi-sppy's dependence on PySP to a single module: mpisppy.utils.pysp_model.