User guide - operational setup

This guide outlines how MPC's optimisation is embedded in an operational system. To a large extent, the workflow is the same as for an operational system with forecast models. The difference is in the optimisation on top of the forecast models, which feeds back the optimised setpoints to the client's SCADA system. For the sake of a full overview, all steps - including those related to data preparation and forecasting - are listed below.

MPC needs a repository of time series (both for input time series and result time series), computational power, and a way of scheduling data transfer and model executions. This can be accomplished either by using MIKE OPERATIONS or DIMS.CORE on a dedicated server or by a cloud service that fetches data, stores it, and executes the models.

Overall there are two ways:

1. Either the models are executed on an on-premises server, which is dedicated to data collection and model execution. A buffer database might be present between tha SCADA system and the server that executes the models (not shown in the below figure).

2. Or client data is transferred via sftp to a virtual machine outside of the client's network, which executes the models and returns the result via sftp.

It is assumed that the client collects data from sensors in a SCADA system, and that the sensors as well as the controllable devices in the system interact with the SCADA system through PLCs.

In the on-premises solution, MPC is installed on a server inside the client's network. This means that DHI needs access to the client's network to get access to the server, both through the configuration phase and for later troubleshooting. This server will be used both for collecting data from the SCADA system and for scheduling the recurrent model execution of MPC models.

In the off-premises solution, MPC is installed on a server outside of the client's network or alternatively as a cloud-based workflow. As the above figures suggest, this imposes an extra layer of data transfer, as fetching the data first requires the data to be sent to an sftp server, and then to be picked up by MPC's time series repository. However, the solution is less intrusive than installing on premises, and this makes it well suited for pilot projects. Inside the client's network, it only requires a workflow that sends data to sftp. This workflow may be hosted by an on-premises MIKE OPERATIONS or DIMS.CORE installation, but any service that can fetch data from the SCADA system and post it to sftp will work. The downside, compared to the on-premises solution, is that data has to be transferred in and out of the client's network.

Tasks in operational schedule

Overview - list of tasks

• Get raw data from SCADA system
• Prepare data for MPC
• Execute MPC models
  • Predict boundaries
  • Optimise setpoints for controlled flows
  • Detailed prediction of system evolution. Use a Hi-Fi model to simulate the effect of using the setpoints proposed by the optimisation (optional)
• Return results
  • setpoints for controlled flows
  • predicted boundaries (optional)
  • predicted states, that is levels/volumes/flows (optional)
    • either predicted by MPC's simplified dynamics or by Hi-Fi model or both
    • processed to quantities and units that fits the end user
• Diagnostics (optional)
  • Accuracy of boundary prediction
  • Accuracy of state prediction

Get raw data from SCADA system

Raw data is a tainted label...here we use it about data in the condition that the model server receives it from the SCADA system. It depends on the client's configuration of the SCADA system what processing the data has undergone before it is transferred to MPC's calculation workflow.

MIKE OPERATIONS and DIMS.CORE both support connection to SCADA systems and integration of data from different sources into one database. Integration of data from different sources is for instance needed if MPC's optimisation is based on rainfall forecast calulated by a meteorology institution.

Decide on a naming convention for the raw data. A common approach is to use the SCADA tag names (and folder structure, if the SCADA data is organised in a hierarchy). Observe that this approach runs short, when/if the SCADA tag names are changed.

Prepare data for MPC

MPC only needs a time window of the most recent data, and it is adviced to cut out the required time slice, process the slice and copy it to a dedicated input folder, from which the MPC models can pick it up. The processed input data may be located either as files on the disk or stored in the time series repository's database.

The processing should cover

• removal of outliers, at least by a range check
• resampling, if the raw data has finer time step than the time step used by the models
• conversion of units to SI
• renaming from raw data name (possibly the SCADA tag) to a shorter, human readable name, which is used by the models.

Be aware of the time stamp - is it UTC or local time? If the time stamp is local time, does it reflect standard time or daylight saving time?

The mapping between raw data names and short names must be unique in both directions.

Execute MPC models

First execute the boundary forecast. The model for boundary prediction depends on the type of the water network:

• For a network of rivers and reservoirs, the boundary forecast is the catchment runoff, predicted by a cathment model (like NAM) driven by rainfall forecast (provided by a meteorological service)
• For a water supply network the boundary forecast is the predicted demand, which might be based on a seasonal model or a machine learning model.
• For a combined sewer system, the boundary forecast is a combination of a catchtment load model (for the dry-weather flow) and a catchment runoff model for the rainfall.

For river networks and sewer systems, the boundary forecast may even be calculated by a HiFi model (e.g. MIKE 11 or MIKE+), which calculates the flow from upstream (uncontrollable) parts of the system down to the points that are used as boundaries for the optimisation model.

Then execute MPC's optimisation. The optimisation needs the boundary prediction and the initial state of the system.

Make sure that the time resolution of the boundary prediction is greater than or equal to the time step of the system dynamics in the optimisation. The optimisation can interpolate the boundaries with greater time step, but its resampling procedure does not cater for resampling from smaller time steps.

The initial condition is either taken from observations or from simulations. Common problems about initial conditions are

• Observations might be flawed, leading for instance to mass balance errors if used as initial condition.
• Observations might not cover all system states (not even in the simplified dynamics for the optimisation)
• Initial condition from a simulation has system-wide coverage and no mass-balance errors - but might not agree with observations of for instance volumes in the subparts of the system.

The future system state when using the optimised flow setpoints is an additional output of MPC's optimisation. This is the outlook as seen from the perspective of the simplified dynamics used by the optimisation.

Additionally, the effect of using the flow setpoints may be simulated by a HiFi model. As the HiFi model is a more detailed model of the system, its prediction is expected to be more accurate than that made by the Optimisation's surrogate model.

Return results

The primary result is time series of flow setpoints for all gates/valves/pumps that are controlled by MPC. However, the boundary prediction and the prediction of system state might also be valuable information to return to the client's system.

The simulation results might need to be processed before returning, especially

• Convert time stamp. If the time stamp has been converted from e.g. local time to UTC, it should be converted back before returning the time series
• Convert state variables. If e.g. depth has been converted to level, it should be converted back.
• Convert units from SI units back to what the receiver expects (e.g. m^3/s --> l/s)

Diagnostics

Diagnostics covers the "self"-monitoring of how the models perform. As we work with forecasts, we are first able to quantify the accuracy of a prediction, when observations become available. An example of monitoring forecast performance could be a graph that compares the forecasted boundary flow to the observed boundary flow for different lead times. Such a graph might reveal if there is a pronounced bias in (some of) the boundary flows or if the discrepancy between simulated and observed values has an increasing trend.

The timing problem

MPC uses an optimisation approach, which makes the optimisation calculation fast, so fast that it only takes a fraction of the time step between optimisation runs. In theory this means that the "waiting time" to get the optimised flow set points is so low that is can be neglected, implying that the first time step in the optimised setpoint time series can be applied right away with a delay of only a small fraction of the time step. In practice, other factors than sheer calculation time of the optimisation influence when the set points are ready to send. Consider again the workflow:

• Get raw data from SCADA system
• Prepare data for MPC
• Execute MPC models
  • Predict boundaries
  • Optimise setpoints
  • Detailed prediction of system evolution (optional)

We will use the case with one-hour resolution of the optimised setpoint time series as well as one hour between optimisations as example. Every hour, on the hour, the workflow starts by retrieving the latest data from the SCADA system. The tasks included in the transfer have to be scheduled in a way that allows for the data processing in the SCADA system to have prepared the data for pick-up (by DIMS.CORE or MIKE OPERATIONS) or for push to a shared buffer (e.g. sftp or a light-weight database).

A number of tasks have to be finished, before the raw data is ready on the model server. So if the data transfer is initiated on the hour, the time stamp of the latest measurement value will be "some time before" the hour. Then the data travels through a number of relays before reaching the model server. Each relay needs to be scheduled so there is ample time. After having retrieved the raw data to the model server, it must be processed to create input for the model that predict boundaries for the optimisation's dynamic model. Then the results from the boundary prediction, as well as the relevant observations must be processed for inputting to the optimisation. So, the optimisation does not start until "some time" after the hour.

This encapsulates "the timing problem": the optimisation assumes that the first time step (in this case the first hour) is a decision variable, but while all prerequisites for the optimisation are being put into place the clock is ticking, and the setpoint during this time interval is not determined by the optimisation. Consequently, this gap between the last observation and the time when the optimisation's decision of the setpoint is ready has to be adressed.