Controlled Finite Markov Chains in Process Control (FICO)

Finite state probabilistic dynamic models in process control design

Tiivistelmä: Tutkimus pohtii ohjattujen äärellisten Markovin ketjujen, ns. (ohjattujen) Markovin tilansiirtomallien käyttöä epävarmojen dynaamisten epälineaaristen teollisuusprosessien mallintamisessa, sekä po. mallien käyttämistä havainnoijien ja säätimien suunnittelussa ja analyysissä.

Abstract: The research examines controlled finite Markov chains (CFMC), alias (controlled) Markov transition models (MTM), as a means for modelling uncertain dynamic nonlinear industrial processes, and the design and analysis of process observers and controllers based on such models.

Field in brief

The field of CFMC and the related stage-wise optimization techniques is commonly referred to as Markov decision processes (MDP). The MDP approach to process control can be seen as an extension of model predictive control (MPC), or model-based (optimal) process control, which has become increasingly popular in industrial applications of process control. MDP is often seen as a part of reinforcement learning (RL); the field of operations research (OR) is also closely related. Important research groups in this area include those of Lee and Lee (Georgia Tech, U Alberta), and the works by Bertsekas and Kaelbling (MIT), just to mention a few. This field is related to our work with genealogical decision trees (particle filtering, see here) and stochastic learning automata (finite state machines, see here).

Current state of research

With the advent of increased and reasonably priced computing power and memory resources, the applications of these techniques has become feasible. Compared to current approaches for modelling, monitoring and control design, we expect the main strengths of such an approaches to be found in:

• improved modelling of propagation of process uncertainties
• availability of means for designing optimal controllers for non-linear uncertain dynamic systems
• simplified learning from recorded process data
• improved methods for exploitation of physical process models and simulation models in process control

In our work in this field, we have examined the feasibility of process control design for various processes:

• van der Vusse CSTR (a non-minimum phase nonlinear system)
• two-tank level-temperature control (with both continuous and discrete variables)
• fluidized bed combustion secondary air feeding system (4x4 MIMO control problem)
• student study path optimization (modeling based on real data records)
• reservoir pumping schedule optimization (a large scale problem)

We have also examined adaptive control based on local CFMC models:

• multi-model gain scheduling based on Kullback-Leibler distance, with
• applications to the control of a non-ideal CSTR

Future directions

We now have built an understanding of the capabilities and limits of the MTM approach in several standard process control tasks, as well as an efficient software package for building and analysing models and controllers. The future work will be targeted towards realization of the potential benefits in handling of uncertainties. Whereas the problems with non-linearities and learning from data are largely handled by modern monitoring and control techniques, dealing with uncertainties is a field where significant difficulties remain.

• Further exploitation of Kullback-Leibler distance -based ideas on multiple model control is a potential direction, with an enhanced focus on state estimation problems.
• State estimation is a field of interest in itself, with all its applications in process monitoring. The results extend easily to process control and automated tasks in higher level process optimization.
• The overlap between mixed integer programming and CFMC is to be properly examined.

In addition, a number of engineering problems need to be solved, while aiming towards better usability of these techniques in the industrial practice:

• Even if we do not see the problem of dimesionality explosion as a significant limiting factor (suitably decentralized control is in general more robust, reliable and safe in industrial applications), the examination of approximate dynamic programming techniques (neuro-dynamic programming, ADP) would allow to improve the accuracy and applicability of MDP-based process control.
• Hybrids of MDP and conventional local controllers (PID, IMC, RST, etc.) is a field of significant practical interest.

Publications on MTM

Journal papers, chapters, theses

• Ikonen, E and K. Leppäkoski (2010). Model-based multivariable control of a secondary air system using Markov chains. In P. Szentannai (Ed.) Power Plant Applications of Advanced Control Techniques (invited), ProcessEng, Vienna. ISBN978-3-902655-11-0. (to appear)

• Ikonen, E. and K. Najim (2009). Multiple model based control using finite controlled Markov chains. Cognitive Computation, 1 (3), pp. 234–243.

• Ikonen, E., E. Gómez-Ramírez and K. Najim (2009). Process regulation via genealogical decision trees. Optimal Control Applications and Methods, 30 (2), pp. 121 – 133. (published on-line 2008, DOI 10.1002/oca.848)

Conference papers and other

• Ikonen, E., I. Selek and M. Tervaskanto (2010) Short-term pump schedule optimization using MDP and neutral GA. IFAC Conference on Control Methodologies anbd Technology for Energy Efficiency (CMTEE 2010), 29-31 March 2010, Vilamoura, Portugal.

• Ikonen, E. (2009). Gain scheduling control with Markov transition models. The 7^th IEEE Intenational Conference on Control and Automation (ICCA’09), 9-11 Dec 2009, Christchurch, New Zealand.

•  Ikonen, E., K. Najim and M. Tervaskanto (2009). Analysis of students’ study paths using finite Markov chains. The 7^th IEEE Intenational Conference on Control and Automation (ICCA’09) , 9-11 Dec 2009, Christchurch, New Zealand.

•  Ikonen, E, K. Leppäkoski and U. Kortela (2009). Model-based multivariable secondary air control using controlled finite Markov chains. IFAC Power Plants and Power Systems Control (PP&PSC’09), Tampere, 5-8 July 2009.

•  Ikonen, E and U. Kortela (2008). Adaptive process control using controlled finite Markov chains based on multiple models. 17^th IFAC World Congress, 6-11 July 2008, Seoul, Korea, pp. 7919–7924.

•  Ikonen, E, K. Najim and U. Kortela (2007). State estimation and predictive control using finite Markov chains. IFAC Workshop on Advanced Fuzzy and Neural Control (AFNC'07), 29-30 Oct 2007, Valenciennes, France.

•  Ikonen, E. (2007). Model-based process control via finite Markov chains. 9’th IFAC Workshop on Adaptation and Learning in Control and Signal Processing (ALCOSP’07), 29-31 Aug 2007, St-Petersburg, Russia. (available at open library http://lib.physcon.ru)

•  Ikonen, E. (2007). Process control using controlled finite Markov chains with an application to a multivariable hybrid plant. 4 ^th International Conference on Informatics in Control, Automation and Robotics (ICINCO'07), 9-12 May 2007, Angers, France.

Selected papers on learning automata

• Ikonen, E. and K. Najim (2008). On-line optimization of replacement policies using learning automata. International Journal of Systems Science, 39 (3), pp. 237–249.

• Najim, K., A. Poznyak and E. Ikonen (2004). Optimization based on a team of automata with binary outputs. Automatica, 40(8), pp. 1349-1359.

• Najim, K., P. Del Moral and E. Ikonen (2004) An improved version of the McMurtryFu reinforcement learning scheme. International Journal of Systems Science, 34, no. 1, pp. 37–47.

• Ikonen, E. and K. Najim. (1997). Use of learning automata in distributed fuzzy logic processor training. IEE Proceedings - Control Theory and Applications, 144, no. 3, pp. 255–262.

• Poznyak, A., K. Najim and E. Ikonen. (1996). Adaptive selection of the optimal model order of linear regression models using learning automata. International Journal of Systems Science, 27, no 2, pp. 151–159.

Enso 12/2009