Runoff Prediction Based on Deep Belief Networks
Runoff prediction has recently become an essential task with respect to assessing the impact of climate change to people’s livelihoods and production. However, the runoff time series always exhibits nonlinear and non-stationary features, which makes it very difficult to be accurately predicted. Machine learning have been recently proved to be a powerful tool in helping society adapt to a changing climate and its subfield, deep learning, showed the power in approximate nonlinear functions. In this study, we propose a method based
on Deep Belief Networks (DBN) for runoff prediction. In order to evaluate the proposed method, we collected runoff datasets from Srepok and Dak Nong rivers located in mountain regions of the Central Highland of Vietnam in the periods of 2001-2007 at Dak Nong hydrology station and 1990-2011 at Buon Don hydrology station, respectively. Experimental results show that DBN outperforms, respectively, LSTM, BiLSTM, Multi-Layer Perceptron (MLP) trained by Particle Swarm Optimization (PSO) and MLP trained by stochastic gradient descent (SGD) in which gradients are computed using the backpropagation (BP) procedure. The results also confirm that DBN is suitable to employ for the task of runoff prediction.
H. Delafrouz, A. Ghaheri, and M. A. Ghorbani, “A novel hybrid neural network based on phase space reconstruction technique for daily river flow prediction,” Soft Computing,
vol. 22, no. 7, pp. 2205–2215, 2018.
E. Gusev, G. Ayzel, and O. Nasonova, “Runoff evaluation for ungauged watersheds by swap model. 1. application of artificial neural networks,” Water Resources, vol. 44, no. 2, pp. 169–179, 2017.
H. R. Maier, A. Jainb, G. C. Dandya, and K. Sudheer, “Methods used for the development of neural networks for the prediction of water resource variables in river systems: Current status and future directions,” Journal of Hydrology, vol. 25, pp. 891–909, 2010.
D. An, N. H. Kim, and J.-H. Choi, “Practical options for selecting data-driven or physics-based prognostics algorithms with reviews,” Reliability Engineering & System Safety, vol. 133, pp. 223–236, 2015.
Z. He, X. Wen, H. Liu, and J. Du, “A comparative study of artificial neural network, adaptive neuro fuzzy inference system and support vector machine for forecasting river flow in the semiarid mountain region,” Journal of Hydrology, vol. 509, pp. 379–386, 2014.
Z. Liang, Y. Li, Y. Hu, B. Li, and J. Wang, “A data-driven svr model for long-term runoff prediction and uncertainty analysis based on the bayesian framework,” Theoretical and applied climatology, vol. 133, no. 1-2, pp. 137–149, 2018.
E. Meng, S. Huang, Q. Huang, W. Fang, L. Wu, and L. Wang, “A robust method for non-stationary streamflow
prediction based on improved emd-svm model,” Journal of hydrology, vol. 568, pp. 462–478, 2019.
A. P. Piotrowski and J. J. Napiorkowski, “Monthly river flow forecasting using artificial neural network and support vector regression models coupled with wavelet transform,” Computers & Geosciences, vol. 54, pp. 1–8, 2013.
H. I. Erdal and O. Karakurt, “Advancing monthly streamflow prediction accuracy of cart models using ensemble learning paradigms,” Journal of Hydrology, vol. 477, pp. 119–128, 2013.
X. Wen, Q. Feng, R. C. Deo, M. Wu, Z. Yin, L. Yang, and V. P. Singh, “Two-phase extreme learning machines
integrated with the complete ensemble empirical mode decomposition with adaptive noise algorithm for multi-scale runoff prediction problems,” Journal of hydrology, vol. 570, pp. 167–184, 2019.
X. Zhang, Q. Zhang, G. Zhang, Z. Nie, and Z. Gui, “A hybrid model for annual runoff time series forecasting using elman neural network with ensemble empirical mode decomposition,” Water, vol. 10, no. 4, p. 416, 2018.
P. K. d. M. M. Freire, C. A. G. Santos, and G. B. L. da Silva, “Analysis of the use of discrete wavelet transforms coupled with ann for short-term streamflow forecasting,” Applied Soft Computing, vol. 80, pp. 494–505, 2019.
M. Khashei and M. Bijari, “A novel hybridization of artificial neural networks and arima models for time series forecasting,” Applied Soft Computing, vol. 11, no. 2, pp. 2664–2675, 2011.
F.-F. Li, Z.-Y. Wang, and J. Qiu, “Long-term streamflow forecasting using artificial neural network based on preprocessing technique,” Journal of Forecasting, vol. 38, no. 3, pp. 192–206, 2019.
M. Shoaib, A. Y. Shamseldin, S. Khan, M. M. Khan, Z. M. Khan, T. Sultan, and B. W. Melville, “A comparative study of various hybrid wavelet feedforward neural network models for runoff forecasting,” Water resources management, vol. 32, no. 1, pp. 83–103, 2018.
M. Zounemat-Kermani, O. Kisi, and T. Rajaee, “Performance of radial basis and lm-feed forward artificial neural networks for predicting daily watershed runoff,” Applied Soft Computing, vol. 13, no. 12, pp. 4633–4644, 2013.
S. d. O. Domingos, J. F. de Oliveira, and P. S. de Mattos Neto, “An intelligent hybridization of arima with machine learning models for time series forecasting,” KnowledgeBased Systems, vol. 175, pp. 72–86, 2019.
S. M. Hosseini and N. Mahjouri, “Integrating support vector regression and a geomorphologic artificial neural network for daily rainfall-runoff modeling,” Applied Soft Computing, vol. 38, pp. 329–345, 2016.
V. Nourani, A. H. Baghanam, J. Adamowski, and O. Kisi, “Applications of hybrid wavelet–artificial intelligence models in hydrology: a review,” Journal of Hydrology, vol. 514, pp. 358–377, 2014.
M. Shoaib, A. Y. Shamseldin, B. W. Melville, and M. M. Khan, “A comparison between wavelet based static and dynamic neural network approaches for runoff prediction,” Journal of Hydrology, vol. 535, pp. 211–225, 2016.
D. Rolnick, P. L. Donti, L. H. Kaack, K. Kochanski, A. Lacoste, K. Sankaran, A. S. Ross, N. MilojevicDupont, N. Jaques, A. Waldman-Brown et al., “Tackling climate change with machine learning,” arXiv preprint arXiv:1906.05433, 2019.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
S. Pouyanfar, S. Sadiq, Y. Yan, H. Tian, Y. Tao, M. P. Reyes, M.-L. Shyu, S.-C. Chen, and S. Iyengar, “A survey on deep learning: Algorithms, techniques, and applications,” ACM Computing Surveys (CSUR), vol. 51, no. 5, p. 92, 2018.
Y. Chen, Y. Kang, Y. Chen, and Z. Wang, “Probabilistic forecasting with temporal convolutional neural network,” In KDD 2019, Workshop on Mining and Learning from Time Series, 2019.
H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.- A. Muller, “Deep learning for time series classification: a review,” Data Mining and Knowledge Discovery, pp. 1–47, 2019.
T. Kuremoto, S. Kimura, K. Kobayashi, and M. Obayashi, “Time series forecasting using a deep belief network with restricted boltzmann machines,” Neurocomputing, vol. 137, pp. 47–56, 2014.
M. Langkvist, L. Karlsson, and A. Loutfi, “A review of ¨ unsupervised feature learning and deep learning for timeseries modeling,” Pattern Recognition Letters, vol. 42, pp. 11–24, 2014.
S.-Y. Shih, F.-K. Sun, and H.-y. Lee, “Temporal pattern attention for multivariate time series forecasting,” Machine Learning, 2019. [Online]. Available:https://doi.org/10.1007/s10994-019-05815-0
C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” IEEE Communications Surveys and Tutorials, 2019.
G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural computation, vol. 18, no. 7, pp. 1527–1554, 2006.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
Y. Bengio et al., “Learning deep architectures for ai,” Foundations and trends® in Machine Learning, vol. 2, no. 1, pp. 1–127, 2009.
J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proc. of the IEEE International Conference on Neural Networks. Perth, Australia: IEEE, 1995, pp. 1942–1945.
F. Kratzert, D. Klotz, C. Brenner, K. Schulz, and M. Herrnegger, “Rainfall–runoff modelling using long short-term memory (lstm) networks,” Hydrol. Earth Syst. Sci, vol. 22, no. 11, pp. 6005–6022, 2018.
T. T. Tran, N. N. Giang, H. N. Duong, H. T. Nguyen, T. Van Hoai, and V. Van Nghi, “A comprehensive study on predicting river runoff,” in 2017 9th International Conference on Knowledge and Systems Engineering (KSE). IEEE, 2017, pp. 251–256.
X. He, J. Luo, G. Zuo, and J. Xie, “Daily runoff forecasting using a hybrid model based on variational mode
decomposition and deep neural networks,” Water Resources Management, vol. 33, no. 4, pp. 1571–1590, 2019.
T. Xie, G. Zhang, J. Hou, J. Xie, M. Lv, and F. Liu, “Hybrid forecasting model for non-stationary daily runoff
series: A case study in the han river basin, china,” Journal of Hydrology, p. 123915, 2019.
R. Salakhutdinov and G. Hinton, “Deep boltzmann machines,” in Artificial intelligence and statistics, 2009, pp.
R. Cheng and Y. Jin, “A social learning particle swarm optimization algorithm for scalable optimization,” Information Sciences, vol. 291, pp. 43–60, 2015.
A. Ahani, M. Shourian, and P. R. Rad, “Performance assessment of the linear, nonlinear and nonparametric data driven models in river flow forecasting,” Water resources management, vol. 32, no. 2, pp. 383–399, 2018.
Z. Alizadeh, J. Yazdi, J. Kim, and A. Al-Shamiri, “Assessment of machine learning techniques for monthly flow
prediction,” Water, vol. 10, no. 11, p. 1676, 2018.
A. K. Poul, M. Shourian, and H. Ebrahimi, “A comparative study of mlr, knn, ann and anfis models with wavelet transform in monthly stream flow prediction,” Water Resources Management, pp. 1–17, 2019.
Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Muller, ¨ “Efficient backprop,” in Neural networks: Tricks of the trade. Springer, 2012, pp. 9–48.
Y. Liu, L. Ye, H. Qin, S. Ouyang, Z. Zhang, and J. Zhou,“Middle and long-term runoff probabilistic forecasting based on gaussian mixture regression,” Water Resources Management, vol. 33, no. 5, pp. 1785–1799, 2019.
O. Terzi, “A genetic programming approach to river flow ¨ modeling,” Journal of Intelligent & Fuzzy Systems, vol. 27, no. 5, pp. 2211–2219, 2014.
M. A. Ghorbani, R. Khatibi, A. D. Mehr, and H. Asadi, “Chaos-based multigene genetic programming: A new hybrid strategy for river flow forecasting,” Journal of hydrology, vol. 562, pp. 455–467, 2018.
X. Yuan, C. Chen, X. Lei, Y. Yuan, and R. M. Adnan, “Monthly runoff forecasting based on lstm–alo model,”
Stochastic environmental research and risk assessment, vol. 32, no. 8, pp. 2199–2212, 2018.
S. Mouatadid, J. F. Adamowski, M. K. Tiwari, and J. M. Quilty, “Coupling the maximum overlap discrete wavelet transform and long short-term memory networks for irrigation flow forecasting,” Agricultural Water Management, vol. 219, pp. 72–85, 2019.
R. Feng, G. Fan, J. Lin, B. Yao, and Q. Guo, “Enhanced long short-term memory model for runoff prediction,” Journal of Hydrologic Engineering, vol. 26, no. 2, p. 04020063, 2021.
M. Gauch, F. Kratzert, D. Klotz, G. Nearing, J. Lin, and S. Hochreiter, “Rainfall–runoff prediction at multiple
timescales with a single long short-term memory network,” Hydrology and Earth System Sciences, vol. 25, no. 4, pp. 2045–2062, 2021.
Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu, “Advances in optimizing recurrent networks,” in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013, pp. 8624–8628.
Y. Shi and R. Eberhart, “A modified particle swarm optimizer,” in 1998 IEEE international conference on evolutionary computation proceedings. IEEE world congress on computational intelligence (Cat. No. 98TH8360). IEEE, 1998, pp. 69–73.
J.-R. Zhang, J. Zhang, T.-M. Lok, and M. R. Lyu, “A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training,” Applied mathematics and computation, vol. 185, no. 2, pp. 1026–1037, 2007.
G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science, vol. 313, no. 5786, pp. 504–507, 2006.
G. E. Hinton, “A practical guide to training restricted boltzmann machines,” in Neural Networks: Tricks of the Trade: Second Edition, G. Montavon, G. B. Orr, and K.-R. Muller, ¨ Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 599–619.
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” in Advances in neural information processing systems, 2007, pp. 153– 160.
A. Fischer and C. Igel, “Training restricted boltzmann machines: An introduction,” Pattern Recognition, vol. 47, no. 1, pp. 25–39, 2014.
R. McGill, J. W. Tukey, and W. A. Larsen, “Variations of box plots,” The American Statistician, vol. 32, no. 1, pp. 12–16, 1978.
J.-Y. Franceschi, A. Dieuleveut, and M. Jaggi, “Unsupervised scalable representation learning for multivariate time series,” in Advances in Neural Information Processing Systems, 2019, pp. 4652–4663.
X. Lyu, M. Hueser, S. L. Hyland, G. Zerveas, and G. Ratsch, ¨ “Improving clinical predictions through unsupervised time series representation learning,” Machine Learning for Health (ML4H) Workshop at NeurIPS 2018, 2018.
N. Wu, B. Green, X. Ben, and S. O’Banion, “Deep transformer models for time series forecasting: The influenza prevalence case,” arXiv preprint arXiv:2001.08317, 2020.