- Email: [email protected]

Water Supply Optimization: An IPA Approach ? Constantinos Heracleous, Zinon Zinonos, Christos G. Panayiotou KIOS Research Center for Intelligent Systems and Networks, and Dept. of Electrical and Computer Eng., University of Cyprus Email: [email protected], [email protected], [email protected] Abstract: In this paper we address the problem of deciding whether a water source should be used by a water utility such that the production cost as well as the penalty cost due to water shortages is minimized while certain constraints are satisfied. The problem is modeled using composition of multiple open hybrid automata while the decision logic depends on certain parameters that need to be optimized. Subsequently, infinitesimal perturbation analysis (IPA) is used to optimize these parameters. The proposed approach is non-parametric in the sense that it does not depend on any assumptions on the stochastic processes that drive the system dynamics and it can be used online to continuously adjust the control parameters even when the input processes are not stationary. Keywords: Water systems, hybrid systems, infinitesimal perturbation analysis (IPA). 1. INTRODUCTION In this paper we consider a water utility that needs to optimize its operation by deciding when certain water source(s) should be used in order to satisfy its customer demand. In general, a water utility may have several sources where it can obtain water, e.g, from a river, a reservoir, an underground well or a desalination plant. The water availability and production cost among the various sources varies significantly. For example, water from a dam or river may be cheaper to use, while its availability depends on the weather conditions. On the other hand, water from desalination plants is significantly more expensive, but availability is more predictable. At the same time, the water utility’s decision should take into consideration quality of service and other environmental constraints. For example, customers should have a continuous supply of water while the water volume of the reservoir should remain above a certain threshold to limit possible environmental consequences. To address this problem, currently utilities rely on decision support systems (DSS). In the literature, several approaches have been proposed that try to solve the water resource management problem. Most approaches rely on linear programming and stochastic dynamic programming. Using linear programming, Kenneth and Richard (1982) and Rani and Moreira (2010) determine optimal reservoir operations while stochastic programming is used in Kim et al. (2007) to derive optimal policies for multi-reservoir systems. In these approaches, an important challenge is to determine the parameters to be used in the formulation. ? This paper is partially supported by the Prevention, Preparedness and Consequence Management of Terrorism and other Securityrelated Risks Programme European Commission - DirectorateGeneral Home Affairs under the FACIES project.

978-3-902823-61-8/2014 © IFAC

265

For example Alemu et al. (2011) used a linear programming model based on historical operations and rules in order to investigate the value of ensemble streamflow predictions and energy price forecasts as aid to decision makers in scheduling the quantity and timing of reservoir releases for daily, weekly, and seasonal operations under regulatory specific constraints. Their DSS generates a range of optimal reservoir releases using an ensemble streamflow forecast and identifies robust operational solutions. In another approach Koutsoyiannis et al. (2003) proposed a DSS reservoir management system that is based on two main modules. The first is a stochastic hydrological simulator and the second is a linear programming model. The stochastic simulator is responsible to generate simulations and forecasts of the hydrosystem inputs using historical data and, the linear programming model is responsible for the parameterization, simulation and optimization of the hydrosystem. Westphal et al. (2003) developed a real-time DSS for adaptive management of a reservoir system that is responsible to provide drinking water to the Boston metropolitan region. To achieve this, they used linear and nonlinear optimization algorithms for the watershed models, the reservoir hydraulic models, and the reservoir water quality model. In another work by Yingchun et al. (2013) the target was to provide solution for the water allocation in agricultural regions for irrigation purposes. In order to achieve this, authors used models for calculating quantifiably key parameters, like the crop water consumption, the agricultural water demand, and the water use efficiency. Stedinger et al. (1984) have developed a stochastic dynamic programming model which employs the best forecast of the current period’s inflow to define a reservoir release policy and to calculate the expected benefits from future operations. Finally Homayounfar et al.

10.3182/20140514-3-FR-4046.00088

WODES 2014 Cachan, France. May 14-16, 2014

(2011) address the problem using a stochastic dynamic game approach. Despite the numerous solutions proposed, the need to further develop decision-support tools especially with the existence of different water sources like water dams and desalination plants is widely recognized. The major challenge for such tools is to manage multiple interdependent units such as environmental, hydrological, social and economical (Bouwer, 2000; Albert et al., 2001), to a unified integrated system able to provide effective decisions. Furthermore, the output of such systems are quite sensitive to the accuracy of the input parameters while obtaining these parameters is not a trivial task. In this paper we adopt a completely different point of view. We model the problem as a composition of open stochastic hybrid automata. Hybrid automata serve as models of hybrid systems and can describe the continuous and discrete dynamics of a hybrid system in a single model formulation (Lygeros et al., 1999). Hybrid automata are divided into two categories, the autonomous hybrid automata and the open hybrid automata (Lynch et al., 2003). The key difference of the two is that in contrast with the autonomous, the open hybrid automata enable inputs and outputs to the model that can change/control the continuous and discrete evolution. Open hybrid automata can also model parts of the system that can be composed together to implement the complete model of the system. The decision whether to produce water from the expensive sources is based on a parametrized policy, while the parameters are optimized on line using stochastic approximation together with infinitesimal perturbation analysis (IPA) of hybrid systems (see Cassandras et al. (2010) and references their in). The benefit of the approach is that it does not make any assumptions on the distributions of the processes that drive the system dynamics. Rather, it utilizes observations from the system state as it operates and periodically makes the parameter updates. 2. SYSTEM MODEL For the purposes of this paper we adopt a single water tank (reservoir) that is supplied by two sources as explained in the following subsections. Note however, that extensions to multiple sources and/or multiple interconnected tanks are possible. 2.1 Water Tank System For the purpose of demonstrating the concept of the proposed method we use the water tank system example shown in Fig. 1. The tank is continuously supplied with water from an uncontrolled source (e.g. river) and also by a controlled source when needed (e.g. desalination station). The input water flow rates from the uncontrolled and the controlled sources are denoted as vT1 (t) and vT2 (t) respectively, while the output water demand flow rate is denoted as vT 3 (t). The use of controlled water source is decided comparing the tank water volume xT1 (t) with a time-varying threshold value calculated by a function g(·). The policy that is used is the following: as long as the tank water volume is greater than the threshold (i.e., xT1 (t) ≥ g(·)) the controlled source, which produces the 266

Controlled Source

Uncontrolled Source T 1 (t )

T 2 (t )

X xT 1 (t )

Water Tank

T 3 (t )

Fig. 1. Water Tank System Example. more expensive water, remains off. If at some point the tank water volume becomes less than the time varying threshold (i.e., xT1 (t) < g(·)) then the controlled water source is turned on. The controlled source remains on until the water volume of the tank reaches the threshold again (i.e., xT1 (t) ≥ g(·)). At that point, a timer is activated while the source will still remain on. Once the timer expires, then the source is turned back off. The purpose of the timer is to avoid having the controlled source continuously chattering between on and off states which is not practical. Note also that the controlled source, in case the water tank gets completely full (i.e., xT1 (t) = X), will be immediately turned off to prevent the loss of the more expensive water. The time varying threshold function g(·) takes into consideration the current season time and also constant parameters values that may be given by the operator. This allows the change of the threshold value with respect to the season period. For example during winter when it is expected that more water will be available at a low cost from the uncontrolled source (e.g., rain), the threshold will have small values to keep the use of the (expensive) controlled source as little as possible but without any unwanted water shortages. Having the optimal threshold value at the right time is very important since the total cost (i.e., both operational and water shortage penalties) will be minimized. For the purposes of this paper, a specific threshold function has been selected, however, we point out that the approach is valid for any other function. The goal of this case study example is to implement a model for the water tank system of Fig. 1. The model will be used to determine the optimal parameter values for a certain time varying threshold to minimize the total cost of the system. The total cost is considered to be the sum of the cost from using the controlled source and also the penalty cost when the tank is unable to serve the water demand. Furthermore, we assume that all input stochastic processes presented in the sequel are defined in a common probability space (Ω, F, P ). 2.2 Hybrid System Modeling of the Water Tank System The water tank system of Fig. 1 can be considered as a hybrid system since there is continuous evolution of the water flow and also discrete events, such as turning on

WODES 2014 Cachan, France. May 14-16, 2014

qT 1 : Healthy xT 1 T 1 T 2 T 3 xT 2 0 xT 1 0

qT 2 : Drained xT 1 T 1 T 2 T 3 xT 2 1 yT xT 1 xT 1 0

T 1 T 2 T 3 0

T 1 T2 T 3

qU 1 : Season A xU 1

yT xT 1 0 xT 1 X xT 1 0

xT 1 X

xU : 0

qU 2 : Season B xU 1

yU 1 U 1

yU 1 U 2

yU 2 xU ; yU 3 0

yU 2 xU ; yU 3 1

0 xU TA qT 3 : Overflow xT 1 0

xU TA

xU : 0

xU TB

0 xU TB

U 1 U 2

yU 1 yU 2 yU 3

yT

xT 2 0

Fig. 3. Uncontrolled water source open hybrid automaton graphical representation

yT xT 1 xT 1 X

Fig. 2. Water tank open hybrid automaton graphical representation and off the controlled source. A system with the particular behavior can be modeled using hybrid automata. For the modeling of the water tank system in Fig. 1 we use open hybrid automata models. Specifically we divide the system into three parts: (a) the water tank, (b) the uncontrolled water source and (c) the controlled water source. Each one of the parts is modeled with an open hybrid automaton and the complete system model is the result of the composition of the three compatible open hybrid automata. The open hybrid automaton formulation that we use for the models is the same as in Liu et al. (1999) where they do not restrict the input variables from the output function. The composition formulation for open hybrid automata can be also found in Lygeros et al. (2008) with some minor adjustments to cover the use of input variables to the output function. Next we describe the open hybrid automata models for each one of the three parts. Water Tank Open Hybrid Automaton The graphical representation of the water tank open hybrid automaton is shown in Fig. 2. The water tank model has three discrete states {qT1 , qT2 , qT3 }. These discrete states are determined by the tank’s water volume xT1 (t) which is the continuous state of the model. If the water volume is larger than zero and less than the tank’s maximum capacity X (i.e., 0 < xT1 (t) < X) then the tank is in the Normal state (qT1 ). Similarly for the two other discrete states, if xT1 (t) ≤ 0 the tank is in the Drained state (qT2 ), and if xT1 (t) ≥ X the tank is in the Overflow state (qT3 ). The transitions from one discrete state to the other are enabled by the guard functions as can be seen in Fig. 2. The tank’s water volume xT1 (t) is calculated using the three inputs {vT1 (t), vT2 (t), vT3 (t)} of the model. Specifically vT1 (t) is the water flow rate from the uncontrolled source, and vT2 (t) is the water flow rate from the controlled source. Finally vT3 (t) is the output water demand flow rate. Using the above input variables the tank’s water volume dynamics are given by: x˙ T1 (t) = vT1 (t) + vT2 (t) − vT3 (t) (1) Note that both vT1 (t) and vT2 (t) are inputs coming from the outputs of the other two parts of the system (i.e., uncontrollable and controllable sources), while vT3 (t) is a general random process that represents the water demand flow rate of the system. 267

Since we are interested for the penalty cost due to unserved water demand by the water tank we introduce another continuous state to the model, xT2 (t). This xT2 (t) state is used as a timer and logs the length of time the tank spends in Drained state (see Fig. 2). In the Drained state since the tank’s water volume is xT1 (t) ≤ 0 there is no water available for supply and the water demand remains unserved. Even though the demand cannot be served at the specific time, we assume that it is backlogged and will be served in future time. Thus negative xT1 (t) is possible and represents backlogged demand. Even though for the purposes of this paper we assume that tank dynamics do not change during the drained state, in general, various demand patters could be utilized. However, accumulating backlog demand is not desirable since it affects the quality of service offered to the customers thus a penalty cost is introduced as soon as xT1 (t) ≤ 0. Finally the only output of the water tank model yT (t), is the tank’s water volume xT1 (t). This output variable (sensor) will be an input to the controlled water source model. Uncontrolled Water Source Open Hybrid Automaton The uncontrolled water source is modeled by the open hybrid automaton model shown in Fig. 3. The uncontrolled water source model has two discrete states {qU1 , qU2 } that are associated with two season periods (e.g., winter/autumn and summer/spring). For these two season periods the water supply rate is different due to the different weather conditions. To determine the time into each season period we use the continuous state xU (t) as a timer. The season periods have specific durations and are denoted with TA and TB for Season A and Season B respectively. As can be seen in Fig. 3 the model stays in Season A as long as 0 ≤ xU (t) < TA and in Season B as long as 0 ≤ xU (t) < TB . As soon as xU (t) ≥ TA if in Season A or xU (t) ≥ TB if in Season B, the specific guard for each case is enabled, xU (t) is reset to zero and a discrete transition takes place changing the water supply rate. The water supply rate for each season period is determined by the two inputs {vU1 (t), vU2 (t)} of the model. Both inputs are general stochastic processes representing the water flow rate for each season, vU1 (t) for Season A and vU2 (t) for Season B. We consider Season A as the period where the water is available in larger quantities and due to that the expected mean values for the two inputs are E [VU1 (t)] > E [VU2 (t)] where VUj , j = 1, 2 is the cumulative water supply over the entire season. The output water flow rate by the uncontrolled source is given in the output variable yU1 (t) of the model depending

WODES 2014 Cachan, France. May 14-16, 2014

g(θ, vC3 (t), vC4 (t)) = s (t) v C3 min c + θ1 , X if vC4 (t) = 0 TA s (t) v C3 if vC4 (t) = 1 max 0, c + θ1 − θ2 TB

C 2 g ( , C 3 , C 4 )

qC 2 : On1 xC1 C1 ; xC 2 0

qC1 : Off xC1 0; xC 2 0

yC C1 C 2 g ( ,C3 ,C4 )

yC 0

C 2 g ( ,C3 ,C4 )

C1 C2 C 3 C 4

(2)

C 2 g ( ,C 3 , C 4 )

xC 2 Ton C 2 X

qC 3 : On2 xC1 C1 ; xC 2 1

where: xC 2 : 0

yC

yC C1 xC 2 Ton C 2 X

Fig. 4. Controlled water source open hybrid automaton graphical representation on the season period. This output will be an input to the water tank model (i.e., vT1 (t) = yU1 (t)). The other two outputs of the model {yU2 (t), yU3 (t)} provide information about the time in each season. Specifically the continuous output yU2 (t) gives the time xU (t) that the model is in each season while the discrete output yU3 (t) gives a discrete value that represents each season (e.g., yU3 (t) = 0 for Season A and yU3 (t) = 1 for Season B ). Both outputs will be used as inputs to the controlled water source model. Controlled Water Source Open Hybrid Automaton As a controlled water source we consider a desalination plant. The water supply by such a source has a constant value with some variations, due to either technical problems or maintenance of the equipment, or when higher priority demand is present. Such a source is used when necessary with usually high cost. We model a controlled water source using the open hybrid automaton shown in Fig. 4. The model has three discrete states {qC1 , qC2 , qC3 } that determine when the controlled source is On or Off. When the controlled source is in the Off state the output continuous variable of the model yC (t), representing the water flow rate by the controlled source, is zero (yC (t) = 0). In the case the controlled source is in the On1,2 states yC (t) is equal with the input vC1 (t) of the model (i.e., yC (t) = uC1 (t)). The input vC1 (t) is a general stochastic process that models the water flow rate of the controlled source when is in operating condition. Note that since the output yC (t) is the water flow rate by the controlled source then, is also an input to the water tank model shown in Fig. 2 with vT2 (t) = yC (t). The controlled source remains in the Off state as long as the water volume of the tank is larger than a threshold value. The water volume measurement is obtained from the water tank model of Fig. 2 through the continuous input variable vC2 (t) (i.e., vC2 (t) = yT (t)). The threshold value on the other hand is calculated by a time varying function g(θ, vC3 (t), vC4 (t)). An example of such a function is given in (2). In the case that the water volume becomes less than the calculated threshold value (i.e., vC2 (t) < g(θ, vC3 (t), vC4 (t))) the specific guard is enabled and a transition takes place turning the controlled source on.

268

• θ = {θ1 , θ2 } are the parameters to be optimized, with {θ1 ∈ R | 0 ≤ θ1 , ≤ X − c} and θ2 ∈ R, where X is the tank’s capacity and c a constant assigned by the utility operator; • vC3 (t) is continuous input variable to the model providing the current time in each season from the uncontrollable water source model of Fig. 3 (i.e., vC3 (t) = yU2 (t)); • vC4 (t) is a discrete input variable to the model and is used as a flag to determine the current season. This is also provided from the uncontrollable water source model of Fig. 3 (i.e., vC4 (t) = yU3 (t)) with vC4 (t) = 0 for Season A and vC4 (t) = 1 for Season B ; • TA and TB are constants and denote the time duration for Season A and Season B respectively. The controlled source remains in the On1 state until the water volume of the tank reaches the threshold g(·). Then it moves to state On2 for a period Ton . In both states On1,2 the controlled source supplies the tank with water. The continuous state variable xC2 (t) is used as timer and resets to zero just before a transition takes place from On1 to On2 . As long as xC2 (t) is less than the specified duration Ton (i.e., xC2 (t) < Ton ) and the tank is not full (i.e., vC2 (t) < X) the controlled source will remain to the On2 . If either of the two conditions fails the specific guard will enable the transition to the Off state stopping also the water supply to the tank. To explain the logic behind the time varying threshold function in (2) we use the plot illustration in Fig. 5. As noted earlier the uncontrolled water source can supply more water when in Season A rather than in Season B. Due to that the function provides small threshold values at the beginning of the Season A, since more rainfall is expected which can be used to satisfy the demand, and increases the threshold as time passes, achieving less use of the controlled source that comes with a cost. In Season B the opposite is happening, the threshold function has large values at the beginning and decreases them as the time passes, with the purpose to minimize any water shortages due to limited water reserve in the tank. As can be observed from (2) and Fig. 5 the constant c determines the threshold function staring point while the parameters θ1 and θ2 determine the “slope” of the function for each season. Both θ1 and θ2 should be optimized in such a way so that the cost of using the controlled source and the cost due to unserved water demand is minimized. Finally since we are interested in the cost when the controlled source supplies with water the tank we introduce another continuous state variable, xC1 (t) to the model. This xC1 (t) state logs all the water volume that is supplied by the controlled source to the tank.

WODES 2014 Cachan, France. May 14-16, 2014

g ( ,C3 ,C4 ) Season A, C4 0 X 1 c

L(θ) =

Season B, C4 1

NT X

τk

Z

yC (t)dt

c1 τk−1

k=1

Z

!

τk

1 [xT1 (t, θ) ≤ 0] dt

+ c2

(6)

τk−1

Let, c

0

Z

τk

L1,k (θ) = C3 0

C3 TA C3 0

C3 TB

1 [xT1 (t, θ) ≤ 0] dt + , θ) ≤ 0 (τk − τk−1 ) = 1 xT1 (τk−1

L2,k (θ) =

τk−1

2.3 Objective Function The objective of this setup is to minimize the expected cost of the controlled water supply and the penalty cost of any water shortages. where L(θ) is the sample function Z T Z T 1 [xT1 (t, θ) ≤ 0] dt yC (t)dt + c2 L(θ) = c1 0

(3)

(4)

0

where 1 [·] is the indicator function that takes the value 1 if the condition is satisfied and 0 otherwise. The parameters c1 and c2 are cost weights with the appropriate units so that the addition of the two terms is meaningful while T is the observation period. At this point it is worth pointing out that the water volume of the tank depends on the parameters θ, thus this dependence is explicitly denoted since it will be useful for the next section. In the more general case, the objective function should also include the cost for the uncontrolled source, however, to simplify the derivation, for the purposes of this paper, we assume that the capacity of the water tank X is infinite. Thus the cost of the water from the uncontrolled source will be constant over the entire interval [0, T ] irrespective of the value of θ, thus it can stay out of the optimization function. We are interested in determining the optimal parameters θ which will be obtained using a stochastic approximation scheme of the form θn+1 = θn + αn Hn (θ)

(7)

Z

Fig. 5. Time varying threshold function plot

J(θ) = E [L(θ)]

yC (t)dt τk−1 τk

t

(5)

where αn is the step size and Hn (θ) is an estimate of the gradient of the objective function (3) with respect to the parameter vector θ. To obtain the gradient estimate, we adopt the Infinitesimal Perturbation Analysis (IPA) framework of Cassandras et al. (2010).

(8)

Then, differentiating with respect θj , j ∈ {1, 2} we obtain: NT ∂L(θ) X ∂L1,k (θ) ∂L2,k (θ) c1 = + c2 ∂θj ∂θj ∂θj k=1

Next we compute each term of the summation: ∂L1,k (θ) + 0 = yC (τk− )τk0 − yC (τk−1 )τk−1 ∂θj ∂L2,k (θ) + 0 = 1 xT1 (τk−1 , θ) ≤ 0 (τk0 − τk−1 ) ∂θj

(9) (10)

0 k where τk0 = ∂τ ∂θj (in general, with · we denote the partial derivative with respect to θj ). Note that yC (t) is the output of the controlled source, thus it can be observed. + , θ) ≤ 0 can be observed with In addition, 1 xT1 (τk−1 the water volume sensor. Next we present an iterative algorithm for computing τk0 but first we compute the perturbation in the state trajectory.

From the water tank dynamics (1) we can compute: Z τk xT1 (τk , θ) = xT1 (τk−1 , θ) + (yU1 (t) + yC (t) − vT3 (t))dt τk−1

Letting f (t) = yU1 (t) + yC (t) − vT3 (t) and differentiating with respect to θj we get: + + 0 x0T1 (τk− , θ) = x0T1 (τk−1 , θ) + f (τk− )τk0 − f (τk−1 )τk−1 (11)

Note that the state trajectory xT1 (·) is a continuous function of t and θ, however, the derivative with respect to θ is not necessarily continuous. On the other hand, τk0 is a continuous function of θ. Eq. (11) propagates a state perturbation from τk−1 to the next time instant τk . To propagate the perturbation at a point where the dynamics exhibit a discontinuity, one can integrate the dynamics to obtain: Z τ+ k xT1 (τk+ , θ) = xT1 (τk− , θ) + f (t)dt. τk−

Differentiating with respect to θj we get:

3. INFINITESIMAL PERTURBATION ANALYSIS Let us define the time instances τk , k = 0, · · · , NT with τ0 = 0 to be the time instances when the stochastic hybrid automaton switches state, and NT is the total number of state transitions within the interval [0, T ]. In general τk (θ) is a function of the parameter vector θ, however, the argument θ will be omitted to simplify the notation. Using τk , the sample function (4) can be written as: 269

x0T1 (τk+ , θ) = x0T1 (τk− , θ) + (f (τk+ ) − f (τk− ))τk0 (12) Thus, if the dynamics do not exhibit a discontinuity, i.e., f (τk+ ) = f (τk− ), then x0T1 (τk+ , θ) = x0T1 (τk− , θ). Next, we compute the time derivatives τk0 for all k and we identify the following cases. Case 1: Instant τk corresponds to an exogenous event. These are the state transitions of the uncontrolled

WODES 2014 Cachan, France. May 14-16, 2014

source model (see Fig. 3), then τk is independent of θ thus: τk0 = 0 (13) Case 2: Instant τk corresponds to an endogenous event, then we have two sub-cases. If the endogenous event is the crossing of a guard function in the tank then xT1 (t, θ)−sm = 0 where sm = 0 or X. Differentiating with respect to θ we get: x˙ T1 (τk− )τk0 + x0T1 (τk− , θ) = 0 Therefore, using (1) and substituting the inputs to the tank with the outputs of the controlled and uncontrolled sources we get: x0T1 (τk− , θ) τk0 = − (14) yU1 (τk− ) + yC (τk− ) − vT3 (τk ) If sm = X, then x0T1 (τk− , θ) = 0 while, if sm = 0, + x0T1 (τk− , θ) = x0T1 (τk−1 , θ) since the dynamics of the tank do not change. If the endogenous event is the crossing of a guard function of the controlled source, then: xT1 (t, θ) − g(θ, vC3 (t), vC4 (t)) = 0 Note that vC4 (t) ∈ {0, 1}, is independent of θ and is used only to select the appropriate function as shown in (2). Also, vC3 (t) is simply a timer measuring the time t in each season. Differentiating with respect to θj we get: x0T1 (τk− , θ) + x˙ T1 (τk− )τk0 − g 0 (θ, τk− , vC4 ) − g(θ, ˙ τk− , vC4 )τk0 0 Therefore, solving for τk we get: −x0T1 (τk− , θ) + g 0 (θ, τk− , vC4 ) τk0 = x˙ T1 (τk− ) − g(θ, ˙ τk− , vC4 )

=0

(15)

Note that x˙ T1 can be obtained from (1) while g(·) ˙ by differentiating (5) with respect to t = vC3 (t). Case 3: Instant τk corresponds to an induced event. This corresponds to the transition of the controlled source from the On2 state to the Off state. This will happen Ton time units after the water tank volume has increased to the guard level. In this case, one can easily show that: 0 τk0 = τm (16) where m < k is the index of the event that transitioned the controlled source in the On2 state. Computing (13)-(16) and substituting them in (7)-(8), an estimate of the gradient of the sample function (4) is computed which under mild assumptions (see Cassandras et al. (2010)) it is also an unbiased estimate of the gradient of the objective functions (3) and thus it can be used in the stochastic approximation algorithm (5). 4. CONCLUSIONS AND FUTURE WORK In this paper we have adopted the stochastic hybrid automaton modeling framework to model the water supply of a water utility company. A threshold policy was introduced to decide when to switch on the higher cost source. Infinitesimal perturbation analysis together with stochastic approximation were used to dynamically optimize the parameters of the threshold policy. In the future, we plan to investigate threshold policies with different functions as well as different cost functions, e.g., with multiple sources. 270

REFERENCES Albert, X., Mark, O., Babel, M., Gupta, A., and Fugl, J. (2001). Integrating Resource Management in South East Asia. Water21, 15(5), 25–30. Alemu, E., Palmer, R., Polebitski, A., and Meaker, B. (2011). Decision Support System for Optimizing Reservoir Operations Using Ensemble Streamflow Predictions. J. Water Resour. Plann. Manage., 137(1), 72–82. Bouwer, H. (2000). Integrated Water Management: Emerging Issues and Challenges. Agricultural Water Management, 45, 217–228. Cassandras, C.G., Wardi, Y., Panayiotou, C.G., and Yao, C. (2010). Perturbation analysis and optimization of stochastic hybrid systems. European J. of Control, 16(6), 642 – 661. Homayounfar, M., Ganji, A., and Martinez, C. (2011). A Novel Solution for Stochastic Dynamic Game of Water Allocation from a Reservoir Using Collocation Method. Water Resources Management, 25(13), 3427–3444. Kenneth, C.G. and Richard, M.S. (1982). TVA Hydro Scheduling Model: Theoretical Aspects. J. Water Resour. Plann. Manage., 108(1), 21–36. Kim, Y., Eum, H., Lee, E., and Ko, I. (2007). Optimizing Operational Policies of a Korean Multireservoir System Using Sampling Stochastic Dynamic Programming with Ensemble Streamflow Prediction. J. Water Resour. Plann. Manage., 133(1), 4–14. Koutsoyiannis, D., Karavokiros, G., Efstratiadis, A., Mamassis, N., Koukouvinos, A., and Christofides, A. (2003). A decision support system for the management of the water resource system of Athens. Physics and Chemistry of the Earth, 28(14-15), 599–609. Liu, J., Liu, X., J.Koo, T.K., Sinopoli, B., Sastry, S., and Lee, E. (1999). A hierarchical hybrid system model and its simulation. In In proc. of IEEE conf. on Decision and Control, volume 4, 3508–3513 vol.4. Lygeros, J., Pappas, G., and Sastry, S. (1999). An introduction to hybrid system modeling, analysis, and control. Preprints of 1st Nonlinear Control Network Pedagogical School. Lygeros, J., Tomlin, C., and Sastry, S. (2008). Hybrid Systems: Modeling, Analysis and Control. Lynch, N., Segala, R., and Vaandrager, F. (2003). Hybrid I/O automata . Information and Computation, 185(1), 105 – 157. Rani, D. and Moreira, M. (2010). Simulation Optimization Modeling: A Survey and Potential Application in Reservoir Systems Operation. Water Resources Management, 24(6), 1107–1138. Stedinger, J.R., Sule, B.F., and Loucks, D.P. (1984). Stochastic Dynamic Programming Models for Reservoir Operation Optimization. Water Resources Research, 20(11), 1499–1505. Westphal, K., Vogel, R., Kirshen, P., and Chapra, S. (2003). Decision Support System for Adaptive Water Supply Management. J. Water Resour. Plann. Manage., 129(3), 165–177. Yingchun, G., Xin, L., Chunlin, H., and Zhuotong, N. (2013). A Decision Support System for irrigation water allocation along the middle reaches of the Heihe River Basin, Northwest China. Environmental Modelling and Software, 47(0), 182 – 192.