Refer to #25 and #26 for discussions on tank data structures and variables, respectively. Similar to junctions, tanks must satisfy flow conservation constraints at each time step. The starting value for the water surface elevation of a tank should be elevation + initlevel and should be constrained with equality to this value at time step n = 0. Otherwise, for n greater than zero, the temporal difference in water surface elevation should be a function of the incoming and outgoing flow, multiplied by the (constant) time step and cross-sectional wetted area of the tank. For cylindrical tanks with constant diameters, these constraints should be easy to derive. For tanks where volcurve is nonempty, this may require the addition of new variables and separate constraints to relate the change in water surface elevation to a change in tank volume. Refer to this paper for a detailed tank model that includes binary variables.

Additionally, we can likely assume that the head at the inlet of a tank is greater than or equal to the head at the outlet of a tank. Separate constraints also may need to be added to ensure minimum volume constraints are satisfied (especially when dealing with tanks with nonempty volcurve values).

We have a software engineering challenge centered around having essentially three different modeling choices

1. Mathematical Model, i.e. MINLP or MISCOP, etc.

2. Direction, i.e. whether or not the flow directions are fixed or not

3. Pipeflow physics, i.e. Hazen-Williams or Darcy-Weichbach

At the moment, the code is controlling these choices via if/then/else statements. I suggest we control these choices based on typing the the model. GasModels has already done this for the equivalents of 1) and 2) using types and union operators - see https://github.com/lanl-ansi/GasModels.jl

Thus, I suggest the following hierarchy

AbstractWaterFormulation - top level formulation object, all common constraints/variables/etc. are defined at this level, equivalent of AbstractGasFormulation in https://github.com/lanl-ansi/GasModels.jl/blob/master/src/core/base.jl

AbstractDirectedWaterFormulation <: AbstractWaterFormulation and AbstractUndirectedWaterFormulation <: AbstractWaterFormulation - defines formulation types for directed vs. undirected, This is like AbstractDirectedGasFormulation and AbstractUndirectedGasFormulation in https://github.com/lanl-ansi/GasModels.jl/blob/master/src/core/base.jl

AbstractDarcyWeisbachWaterFormulation <: AbstractWaterFormulation and AbstractHazenWilliamsWaterFormulation <: AbstractWaterFormulation - defines formulation types for Hazen Williams and Darcy Weisbach

Now, I suggest we use the union operator to combine the formulations into types

i.e. AbstractHazenWilliamsDirectedForm = Union{AbstractHazenWilliamsWaterFormulation, AbstractDirectedWaterFormulation}

This will then allow you defined the mathematical objects on the form of the model, i.e. for MINLPs, you will have things like

""
@compat abstract type AbstractMINLPHazenWilliamsDirectedForm <: AbstractHazenWilliamsDirectedWaterFormulation end

""
@compat abstract type StandardMINLPHazenWilliamsDirectedForm <: AbstractMINLPHazenWilliamsDirectedForm end

See https://github.com/lanl-ansi/GasModels.jl/blob/master/src/form/minlp.jl for an example.

If you run into situations where combinations of the modeling choices share common code, you can use the union operator to combine types.

See for example AbstractMINLPForms = Union{AbstractMINLPDirectedForm, AbstractMINLPForm} in GasModels. - this is used for MINLP features that don't care if the model is directed or undirected.

At the end of the day, the only thing that really differs across all three combinations of modeling choices is the pipeflow equations, and this will allow us to switch between them through the definition of the mathematical form.

• the network id need not be specified explicitly for single network only models
• conditionals of the form T <: Union{AbstractMICPForm, AbstractNCNLPForm} should be avoided in favor of function dispatch / overloading.

The function _get_node_id_by_name has a worst-case quadratic runtime when called inside _add_link_ids! use of a Dict for lookup will be expected linear time.

Add these helper functions to prepare for a multi-infrastructure implementation.

We should try to avoid things like, enumerate(ref(wm, n, :resistance, a)) in favor of things like ref(wm, n, :resistance).

The REQUIRE should be broken into two files. The package root REQUIRE is for the bare minimum needed for the package. The test REQUIRE adds extra dependencies needed for running Pkg.test.

Future versions of WaterModels will require the ability to export design and control decisions to an EPANET file. As such, the current input parsing methods that convert characters to lowercase should be replaced to ensure internal data (e.g., source_id) are instead consistently cased. Additionally, data fields that are usually present in EPANET are also converted to lowercase (e.g., [PIPES] becomes "pipes" and then eventually ref[:pipes]). Internally, lowercase labeling of component types seems to be the most readable option and I think should be retained. This may add additional complexity upon the construction of an output templater, but it is probably worth it.

At the least this block needs an integer specifying how many items each array will have. It would also benefit from a time_elapsed parameter and any other metadata that might come out of the source data.

Refer to #29 for a discussion on check valve data structures. Obviously, the variables that already exist for pipe objects will remain for pipes with check valves. However, each check valve will require the addition of a binary variable, wm.var[:nw][n][:x_check][a], which defines if there is flow within pipe a (1) or not (0) at time n.

@JuliaRegistrator register()

The data dict should be compatible with InfrastructureModels.print_summary. The use of nothing seems to be an issue, there may be a bug in InfrastructureModels.

WaterModels should also extend the print_summary to provide some package specific options. For example see, https://github.com/lanl-ansi/PowerModels.jl/blob/master/src/core/data.jl#L178

See the Open Water Analytics definition of check valves and EPANET's description of check valves in Section 3.1. Note that shutoff and check valves are not considered to be [VALVES] but are instead parts of [PIPES]. Pipe objects with status flags equal to cv should be considered check valves. A :check_valve key within ref should be constructed to easily filter links that are pipes with check valves. Since [PIPES] objects are already parsed in src/io/epanet.jl, implementing these data structures should be easy. Furthermore, when check valves are detected, the corresponding flow_direction attribute should be set appropriately (e.g., to POSITIVE).

See the Open Water Analytics definition of pumps and EPANET's description of pumps on page 32. Pumps are links that increase head across the flow direction. Pumps in the initial implementation should contain the following data:

Note that pumps are associated with unidirectional flow.

To ensure interoperability with EPANET (e.g., for further testing of design and control decisions), a template processor must be developed that converts solution data from the internal data model to a file that conforms to the EPANET format.

This will ensure greater conformity with PowerModels.jl and GasModels.jl.

I suggest breaking this out into into two functions one for hazen williams and one darcy weisbach. Then, these functions get called depending on the pipeflow equation that is added. In general, we don't store calculated values in the data unless it is an O(n) operation (or worse).... this prevents inevitable bugs where a calculated value gets out of sync with the data used to compute the calculated value. (see how we compute conductance and suscpetance in PowerModels and friction factors in GasModels)

@JuliaRegistrator register()

In many of the formulations, flow variables along an arc are decomposed into two separate directed flow variables (i.e., one in the positive direction and one in the negative direction). Furthermore, for network expansion models, flow variables are repeated per possible resistance value that can be selected. Formulating the problems in this way is mathematically elegant, but it discourages the easy retrieval of final solution data (e.g., flows along an arc). For these formulations, this can be remedied by introducing expressions that perform a reduction on the multiple flow variables (e.g., qp and qn) into a single expression q = qp - qn without having to introduce variables into the JuMP model. All formulations that use nonstandard physical variables should "roll up" into these placeholder JuMP expressions representing the original physical variables of interest, when possible.

These should be ints, not strings. If the source format enforces int names then those are used, otherwise the component ids should be from 1-to-n based on the order they appear in the file and the name field and source_id can be used to save the string id.

Refer to #25 for a discussion on tank data structures. Since tanks are most generally defined by volume versus height relationships, the water surface elevation within a tank of index i at time n should be defined by the variable wm.var[:nw][n][:h][i], which represents the water surface elevation (or hydraulic head) at time n. The lower bound on this variable should be elevation + minlevel, and the upper bound should be elevation + maxlevel. The starting value should be elevation + initlevel, and the variable should be constrained with equality to this value at time step n = 0. When volcurve is nonempty, the formulation may have to include additional variables and/or approximations that relate the change in height to a change in volume. Otherwise, for a cylindrical tank, the volume will be a simple function of the change in water surface elevation and the (constant) tank diameter. Refer to this paper for a detailed tank model that includes binary variables.

It looks like you want 3 mathematical forms

MINLP - the mixed integer non linear programming form
MICP - the mixed integer convex programming form (relaxation)
MILP - the mixed integer linear programing form (the piecewise linear relaxation of the MICP)

Then, in the implementations of the various variable creation functions, the necessary variables and constraints can be added. For example, in GasModels, there is a function variable_flow. If the mathematical form is undirected, the implementation of this function creates direction variables. If the mathematical form is directed, the implementation does not.

ie.

function variable_flow{T <: AbstractMIForms}(gm::GenericGasModel{T}, n::Int=gm.cnw; bounded::Bool = true)
variable_mass_flow(gm,n; bounded=bounded)
variable_connection_direction(gm,n)
end

vs.

function variable_flow{T <: AbstractMIDirectedForms}(gm::GenericGasModel{T}, n::Int=gm.cnw; bounded::Bool = true)
variable_mass_flow(gm,n; bounded=bounded)
end

Eventually we will want to include the NLP form that encodes the sign (abs) without resorting to direction variables, but that will be much later

I recommend that you bump the min version of Memento to 0.8, then you can remove this line,

setlevel!(LOGGER, "info")

which has a non-ideal hack.

It looks like the conversion to SI units is happening in base.jl. Rather than doing the conversion there, I suggest doing the conversion in the io packages, since that input units could vary depending on the source data. So, the epanet parser will do the conversion to SI units depending on how the epanet file defines it units, the foo parser will do the conversion to SI depending on how some arbitrary foo type file defines it, etc.

We need some basic tests of this formulation.

Merge this into the node component data.

The non-dimensionalization (per unit) conversion should live in data.jl (see how PowerModels.jl and GasModels.jl handles the automation of conversion between per unit and mixed units (PowerModels) or SI units (GasModels)

Type signatures of Dict{String, Any} should be updated to Dict{String, <:Any} to accommodate strong typed inputs.

Move the type definitions into core/types.jl

This should only be defined in SI or non-dimensionalized units. We should assume that after the io, all units are converted into an internal standard and all further code can assume that the standard is followed.

Currently, there are no tests to ensure that multinetwork data are handled properly when reading inputs, building models, solving models, and storing solutions. Tests should be implemented to ensure proper coverage of this functionality.

For water flow (wf) problem formulations, networks should be replicated and combined into a single water flow problem. In this case, solutions obtained (e.g., flow and head values) per network replica should be identical. For network expansion (ne) problem formulations, similar replicas of the initial data should be made. In this case, solutions obtained (e.g., selected resistances) per network replica should also be identical, and the optimal objective should be a multiple of the number of replicas.

To assist in building this functionality, see how PowerModels.jl replicates network data and the corresponding multinetwork tests.

I saw the request to Install AppVeyor. Do you have a specific need for windows CI? If not, I would skip it. If so, is Travis and option? There is basic support for windows as far as I know.

The current formulations either use the full nonconvex physics or outer-approximations of the head loss relationship. A piecewise linear relaxation of the head loss relationship should also be implemented to form mixed-integer linear programming (MILP) variants of the existing core problem specifications (i.e., water flow and network expansion). The MILP formulation, with a standard convex combination rewriting of the piecewise linear constraints, seems to be common throughout the literature and is thus important to include in WaterModels. Further note that binary variables representing the active portion of a head loss curve can be shared across shared flow variables along an arc, e.g., in network expansion formulations. (That is, only one binary variable per arc is needed if the same number of breakpoints are used per discrete head loss curve.)

Currently, all component IDs are assumed to be integer upon the reading of network data. These IDs are then used identically throughout various model building and solution routines. However, many EPANET input files contain component IDs that are non-integer (i.e., they contain combinations of characters and numbers). For this reason, WaterModels.jl should support network encodings that assume string-like component IDs.

Internally, in the modeling routines and the ref structure, corresponding integer IDs should be used that map to each component. The method for assigning these integer IDs should return deterministic mappings when repeatedly called. When the solution to the model is obtained, solution data should be printed with respect to the original (string-like) component IDs.

This could be most easily accomplished through the addition of an integer index field to the network components when reading in data (e.g., in [src/io/epanet.jl]). A string id field could then be used to provide a mapping from the integer to the string. However, reading in data that modifies the network may be more tricky. In this case, the initial network (e.g., from an EPANET input file) should always be constructed first. Then, modifications can be applied on top of the existing network to ensure changes are always made to the correct components.

Discuss if OrderedDicts are required of if we can get by with Array and Dict.

Pipes with shutoff valves are bidirectional and must satisfy the same head loss relationships as other pipes. However, when the shutoff valve is closed, the head difference between connecting nodes must become decoupled. (That is, the head at one end of a pipe with a shutoff valve, when closed, should not depend on the head at the other end of the pipe.) To accomplish this, the head loss equation must be relaxed with big-M constraints that depend on the binary variable describing the state of the shutoff valve.

Interestingly, though, consider the mixed-integer convex relaxation (MICP) and linear outer-approximation (MILPR) formulations already present within WaterModels. In these formulations, the head loss relationships are already similarly relaxed (decoupled). However, the relaxations are upper-bounded by a linear constraint that depends on the flow upper bound. Thus, in these formulations, when a check valve is closed, the relaxation upper bounds should instead become nonbinding, i.e., the upper bounds should be replaced with some larger big-M or simply excluded. Right now, I'm not quite sure how such a big-M should be derived, but it should represent the absolute largest head difference that could exist between adjacent nodes.

We should explore if it is possible to drop the registered functions in core/function.jl.

See the Open Water Analytics definition of control valves and EPANET's description of control valves on page 176. Note that shutoff and check valves are not considered to be [VALVES] (i.e., control valves) but are instead parts of [PIPES]. Control valves in the initial implementation should include the following data:

Valve types and settings should be modeled as tuples of enumerated types and setting types, respectively. Valve types and settings include the following data:

Refer to #28 for a discussion on shutoff valve data structures. Obviously, the variables that already exist for pipe objects will remain for pipes with shutoff valves. However, each shutoff valve will require the addition of a binary variable, wm.var[:nw][n][:x_so][a], which defines if the shutoff valve within pipe a has been opened (1) or closed (0) by the operator at time n.

I am wondering if we can omit the start/end_node_name node names from the pipes as these will be captured in explicit node objects.

Functions in data.jl should only depend on the data dict. Functions that require GenericWaterModel should be moved into a new file base/ref.jl.

These can help in isolating the key mathematical differences in the formulations by collecting all of the data extraction and processing into a common location.

the epanet parser isn't loaded by default when runninginclude("src/WaterModels_linux.jl"). Also, the parser appears to be nonfunctional for anything but a basic '.inp' file.

See the Open Water Analytics definition of tanks and EPANET's description of tanks in Section 3.1. Tanks in the initial implementation should include the following data:

First, note that the total water surface elevation is the bottom elevation (elevation) plus the water level. Also note that, for cylindrical tanks, computing the water level as a function of volume is simple. Otherwise, if a nonempty volcurve is defined, the appropriate [CURVES] data structure should be referenced, which defines a volume versus height relationship.

When reading in network data (e.g., in src/io/epanet.jl), it makes sense to have [TANKS] objects separated from [CURVES] objects. Moreover, an initial implementation should focus only on the functionality of cylindrical tanks. However, when a nonempty volcurve is defined, it is unclear what the data structure held in ref should look like. Should the associated curve be attached to the tank object, or should the two types remain decoupled?

• Add node-based arc/link lookup lists so that flow balance constraints are not quadratic
• filter out inactive components (i.e. status values)
• update to improved build_ref design, lanl-ansi/PowerModels.jl#505

A lot of data is being unnecessarily precomputed and stored in ref. The data is later used when constructing individual constraints. Much of this should instead be done within src/core/constraint_template.jl.

See the Open Water Analytics definition of shutoff valves and EPANET's description of shutoff valves in Section 3.1. Note that shutoff and check valves are not considered to be [VALVES] but are instead parts of [PIPES]. Although this does not appear to be documented, as far as I can tell, pipe objects with empty status flags should be considered shutoff valves. A :shutoff_valve key within ref should be constructed to easily filter links that are pipes with shutoff valves. Since [PIPES] objects are already parsed in src/io/epanet.jl, implementing these data structures should be easy.

PowerModels has standardized around the post-fix values _fr and _to for edge-like components. Following this we might change f_id/t_id to node_fr/node_to.

Pipes with check valves are unidirectional and must satisfy the same head loss relationships as other pipes. Right now, I think this could be most cleanly handled by simply ensuring lower bounds of zero are placed on flow variables for pipes that contain check valves (assuming that flow will travel from the node1 to node2 indices specified in the original EPANET file). We should also ensure that flow_direction is properly set (to POSITIVE) upon reading the EPANET file.