Software tools and code
This web page forms Deliverable 6.1 from the QUICS project: "Database of openly accessible software tools".
During their research, most fellows wrote code in languages such as Python, Matlab and R. Where the codes were likely to be useful to others, and in a format capable of being used, they have been shared through various different sites.
The table below provides a list of these tools, the fellows involved and a link to the tool. Click on the tool name for further details, including links to associated reports and papers.
EmiStatR and stUPscales (formerly AggProp)
EmiStatR: This R-package provides a fast and parallelised calculator to estimate wastewater emissions from combined sewers. It also supports the planning and design of urban drainage systems, without the requirement of extensive simulation tools.
EmiStatR is available on CRAN and also further described in this MDPI Water paper.
stUPscales (Spatio-temporal Uncertainty Propagation across scales) - formerly known as AggProp: Provides several R methods and functions for temporal aggregation of environmental variables as precipitation and pollutants of Urban Drainage Models (UDMs). Also, it provides methods and functions for uncertainty propagation for lumped UDMs via Monte Carlo simulation.
stUPscales is available on Github and should soon be available on CRAN. It is further described in this MDPI Water paper.
These tools are described in deliverable report 2.3, although have continued to be developed so also so the journal papers mentioned for each tool.
spup: Spatial Uncertainty Propagation Analysis
The R package ‘spup’ implements uncertainty propagation analysis in spatial environmental modelling following the methodology described in Heuvelink et al. (2007) and Brown and Heuvelink (2007). The package provides functions for examining uncertainty propagation starting from input data and model parameters, via the environmental model onto model outputs. The functions include uncertainty model specification, stochastic simulation and propagation of uncertainty using Monte Carlo (MC) techniques. Uncertain variables are described by probability distributions. Both numerical and categorical data types are handled. Spatial auto-correlation within an attribute and cross-correlation between attributes is accommodated for. The MC realizations may be used as input to the environmental models called from R, or externally.
The package is available on CRAN.
The main author of this tool is Tobias Houska. Within the QUICS project, Carla has used and added to this tool.
A Statistical Parameter Optimization Tool, SPOTPY is a Python tool that enables the use of Computational optimization techniques for calibration, uncertainty and sensitivity analysis techniques of almost every (environmental) model. The package is puplished in the open source journal PLoS One: Houska, T, Kraft, P, Chamorro-Chavez, A and Breuer, L; SPOTting Model Parameters Using a Ready-Made Python Package; PLoS ONE; 2015 The simplicity and flexibility enables the use and test of different algorithms without the need of complex codes, doi: 10.1371/journal.pone.0145180.
SPOTPY is available on PyPI. There is also further information in Deliverable 1.4 and a case study utilising SPOTPY is described in Deliverable 4.1.
Various tools related to rainfall ensembles, mergingin radar and rain gauge data and kriging
Ensemble Generator.R: Geostatistical Radar Ensemble Generator (GREG)
KEDUD.R: Kriging with External Drift for Uncertain Data (KEDUD)
KED_with_optimal_BoxCox.R: KED radar-gauge merging with optimised Box-Cox lambda
Negentropy.R: KED radar-gauge merging with optimised Box-Cox lambda
OKUD.R: Ordinary Kriging for Uncertain Data (OKUD)
QQplots_function.R: KED radar-gauge merging with optimised Box-Cox lambda
RG_validation_function.R: KED radar-gauge merging with optimised Box-Cox lambda
Residual_gaussianity_function.R: KED radar-gauge merging with optimised Box-Cox lambda
These tools are available on Github and are also closely linked to the work in Deliverable 3.2.
Python scripts for water quality simulators WEST (MIKE DHI). Sensitivity analysis, inference and automated runs.
The scripts are available on Github.
The novelty of this work is the consideration of (short) time series for the simulation inputs and outputs. Such simulation setup is interesting in applications such as Model Predictive Control (MPC) in which numerous, fast and frequent simulation results are required. Here, an emulator is developed to predict a storage tank’s volume in a small case study in Luxembourg. Three main inputs are considered as the GPE’s parameters: initial volume in the tank, the level in which the outlet pump of the tank must start to work, and the time series of expected rainfall in the upcoming 2 hours. The output of interest is the total volume of the storage tank for the next 24 hours.
The code (in R) and the simulation and validation data-sets will be available after publication of this research.
Temporal Uncertainty Propogation Asross Scales, stUPscales, provides several R functions for temporal aggregation of enviornmental variables used in Urban Drainage Models, such as precipitation and pollutants. It also provides methods and functions for ucertainty propgation via Monte Carlo simulation; and provides specific analysis functions for urban drainage system simulation to evaluate water quantity and quality in combined sewer overflows (CSOs).
The package is available on Github.
One challenge in uncertainty estimation of environmental models is the high number of parameters, as well as model structure errors. The first make traditional optimisation routines difficult due to co-linearities of model parameters. The second often require additional error terms which capture these structural deficits with stochastic processes.
Inference with the latter requires Bayesian inference methods for model calibration, which typically rely on sampling from the posterior. Obtaining a reliable estimate of the posterior usually requires many thousand iterative model runs and there is the danger that sampling a very complex posterior distribution is inadequate due to too few iterations.
Efficient samplers have been proposed to relax this problem by increasing the acceptance rate due to sophisticated proposal algorithms, such as adaptive sampling or parallelizing the inference with ensembles.
Three open source computational libraries of efficient algorithms for adaptive (1) and ensemble-based MCMC (Monte Carlo Markov Chain) sampling (2). In addition, we explore and compare the effectiveness of each sampler in standardized inference problems are presented. For full details, see Deliverable 1.2.
The MCMC sampler is available on github for both R and Julia. The Generic adaptive MCMC sampler proposed by Vihola (2011) is available on github.
An improved sediment wash-off model for urban impervious surfaces
Deliverable 2.4 describes the development of an improved sediment wash-off model for urban impervious surfaces. The revised model has been implemented in R and is available as a web app.
Sampling design optimisation of a non-stationary variance model
These codes are closely linked to Deliverable 3.1 and the journal papers "Sampling design optimisation for rainfall prediction using a non-stationary geostatistical model" which is Open Access published in Advances in Water Resources and "Accounting for non-stationary variance in geostatistical mapping of soil properties" which is under review.
The codes are available on Github.
Tool to advise on using appropriate river pollutant transport model
This software tool has been developed to enable evaluation of structural and parameter uncertainty in models to describe pollutants in rivers. This tool can therefore assist in selecting appropriate river pollutant transport models.
The tool is written in Python and described in Deliverable 4.7 and the code is available as a zip file from the QUICS site.