Uncertainty Analysis Using GEM-SA - Tony O'Hagan

Uncertainty Analysis Using GEM-SA - Tony O'Hagan

Uncertainty Analysis Using GEM-SA Outline Setting up the project Running a simple analysis Exercise More complex analyses GEM-SA course - session 4 2 Setting up the project Create a new project

Select Project -> New, or click toolbar icon Project dialog appears Well specify the data files first GEM-SA course - session 4 4 Files Using Browse buttons, select input and output files The Inputs file contains one column for each

parameter and one row for each model training run (the design) The Outputs file contains the outputs from those runs (one column, in this example) GEM-SA course - session 4 5 Our example Well use the example model1 in the GEM-SA DEMO DATA directory This example is based on a vegetation model with 7 inputs RESAEREO, DEFLECT, FACTOR, MO, COVER, TREEHT, LAI

The model has 16 outputs, but for the present we will consider output 4 June monthly GPP GEM-SA course - session 4 6 Number of inputs Click on Options tab Select number of inputs using Or click From Inputs File GEM-SA course - session 4

7 Define input names Click on Names The Input parameter names dialog opens Enter parameter names Click OK GEM-SA course - session 4 8 Complete the project

We will leave all other settings at their default values for now Click OK The Input Parameter Ranges window appears GEM-SA course - session 4 9 Close and save project Click Defaults from input ranges button Click OK Select Project -> Save

Or click toolbar icon Choose a name and click Save GEM-SA course - session 4 10 Running a simple analysis Build the emulator Click to build the emulator

A lot of things now start to happen! The log window at the bottom starts to record various bits of information A little window appears showing progress of minimisation of the roughness parameter estimation criterion The Main Effects tab is selected, in which several graphs are drawn Progress bar at the bottom GEM-SA course - session 4 12 Focus on the log window The Main Effects and Sensitivity Analysis

tabs are concerned with SA, and will be considered in the next session We are interested just now simply in Uncertainty Analysis (UA) The Output Summary tab contains all we need and more But the key things can be seen more simply in the log window at the bottom Diagnostics of the emulator build The basic uncertainty analysis results GEM-SA course - session 4 13 Emulation diagnostics

Note where the log window reports Estimating emulator parameters by maximising probability distribution... maximised posterior for emulator parameters: precision = sigmasquared = 0.342826, roughness = 0.217456 0.0699709 0.191557 16.9933 0.599439 0.459675 1.01559 The first line says roughness parameters have been estimated by the simplest method The values of these indicate how non-linear the effect of each input parameter is Note the high value for input 4 (MO) GEM-SA course - session 4 14 Uncertainty analysis mean Below this, the log reports

Estimate of mean output is 24.145, with variance 0.00388252 So the best estimate of the output (June GPP) is 24.1 (mol C/m2) This is averaged over the uncertainty in the 7 inputs Better than just fixing inputs at best estimates There is an emulation standard error of 0.062 in this figure GEM-SA course - session 4 15 Uncertainty analysis variance The final line of the log is

Estimate of total output variance = 73.9033 This shows the uncertainty in the model output that is induced by input uncertainties The variance is 73.9 Equal to a standard deviation of 8.6 So although the best estimate of the output is 24.1, the uncertainty in inputs means it could easily be as low as 16 or as high as 33 GEM-SA course - session 4 16 Exercise

A small change Run the same model with Output 11 instead of Output 4 Calculate the coefficient of variation (CV) for this output NB: the CV is defined as the standard deviation divided by the mean GEM-SA course - session 4 18 More complex analyses Input distributions Default is to assume the uncertainty in each

input is represented by a uniform distribution Range determined by the range of values found in the input file or separately input A normal (gaussian) distribution is generally a more realistic representation of uncertainty Range unbounded More probability in the middle GEM-SA course - session 4 20 Changing input distributions Reopen Project

dialog by Project > Edit or clicking on Select Options tab Click All unknown, product normal Then OK A new dialog opens to specify means and variances GEM-SA course - session 4 21 Model 1 example Uniform

distributions from input ranges Normal distributions to match Range about 4 std deviations Except for MO Narrower distribution Uniform Normal

Parameter Lower Upper Mean Variance RESAEREO 80 200

140 900 DEFLECT 0.6 1 0.8 0.01 FACTOR

0.1 0.5 0.3 0.01 MO 30 100 60

100 COVER 0.6 0.99 0.8 0.01 TREEHT 10

40 25 100 3.75 9 6.5 1 LAI

GEM-SA course - session 4 22 Effect on UA After running the revised model, we see: It runs faster, with no need to rebuild the emulator The emulator fit is unchanged The mean is changed a little and variance is halved Estimate of mean output is 26.2698, with variance 0.00784475 Estimate of total output variance = 38.1319 GEM-SA course - session 4 23

Reducing MO uncertainty further If we reduce the variance of MO even more, to 49: UA mean changes a little more and variance reduces again Estimate of mean output is 26.3899, with variance 0.0108792 Estimate of total output variance = 27.1335 Notice also how the emulation uncertainty has increased (0.004 for uniform) This is because the design points cover the new ranges less thoroughly GEM-SA course - session 4 24

A homework exercise What happens if we reduce the uncertainty in MO to zero? Two ways to do this Literally set variance to zero Select Some known, rest product normal on Project dialog, check the tick box for MO in the mean and variance dialog What changes do you see in the UA? GEM-SA course - session 4 25

Cross-validation Reopen the Project dialog and select the Options tab Look at the bottom menu box, labelled Crossvalidation There are 3 options None Leave-one-out Leave final 20% out CV is a way of checking the emulator fit Default is None because CV takes time GEM-SA course - session 4 26 Leave-one-out CV

After estimating roughness and other parameters, GEM predicts each training run point using only the remaining n-1 points Close to 1 Results appear in log window Cross Validation Root Mean-Squared Error = 0.907869 Cross Validation Root Mean-Squared Relative Error = 4.34773 percent Cross Validation Root Mean-Squared Standardised Error = 1.15273 Largest standardised error is 4.32425 for data point 61 Cross Validation variances range from 0.18814 to 3.92191 Written cross-validation means to file cvpredmeans.txt Written cross-validation variances to file cvpredvars.txt (Model 1, output 4, uniform inputs) GEM-SA course - session 4 27

Leave final 20% out CV This is an even better check, because it tests the emulator on data that have not been used in any way to build it Emulator is built on first 80% of data and used to predict last 20% Cross Validation Root Mean-Squared Error = 1.46954 Cross Validation Root Mean-Squared Relative Error = 7.4922 percent Cross Validation Root Mean-Squared Standardised Error = 1.73675 Largest standardised error is 5.05527 for data point 22 Cross Validation variances range from 0.277304 to 4.886 Standardised error a bit bigger But not bad for just 24 runs predicted GEM-SA course - session 4

28 Output Summary tab The Output Summary tab presents all of the key results in a single list Tidier than searching for the details in the log window Although the log window actually has more information Can print using GEM-SA course - session 4 29

Other options There are various other options associated with the emulator building that we have not dealt with See built in help facility for explanations Also slides at the end of session 3 But weve done the main things that should be considered in practice And its enough to be going on with! GEM-SA course - session 4 30 When it all goes wrong How do we know when the emulator is not working?

Large roughness parameters Especially ones hitting the limit of 99 Large emulation variance on UA mean Poor CV standardised prediction error Especially when some are extremely large In such cases, see if a larger training set helps Other ideas like transforming output scale A suite of diagnostics is being developed in MUCM See Bastos and OHagan on my website http://tonyohagan.co.uk/academic/pub.html Not implemented in GEM-SA yet GEM-SA course - session 4

31

Recently Viewed Presentations

  • ASQ Kaizen Presentation Mar 24 2005

    ASQ Kaizen Presentation Mar 24 2005

    Introduction Kaizen Philosophy and Approach Kaizen Toolbox Results from Some Kaizen Events Lessons Learned ("The Bigger Picture") "KAI" - Take apart and make better "ZEN" - Think. Make good the actions of others. Do good deeds.
  • Congressional District Projects New CD-4 North I-25 Record

    Congressional District Projects New CD-4 North I-25 Record

    Project increases the capacity of storm drains that cross US 50 west of La Junta from every 1-2 years to every 25 years. We are adding pipes that cross the highway and adding a new drainage crossing under the BNSF...
  • College Academy Update Rebecca Hoyt Dean of College

    College Academy Update Rebecca Hoyt Dean of College

    A Brief Update on the 1st Cohort of the College Academy. The 1st cohort has now earned 24 hours of college credit! Participated in a Balloon Satellite research project with UAH Space Hardware Club, digital animation project during their sophomore...
  • Supporting Families to build Mathematical skills in their child

    Supporting Families to build Mathematical skills in their child

    Ways you can help Preschool - Year 2 Ways you can help Years 3-6 Examples of Websites, Apps and online Maths Games Examples of Websites, Apps and online Maths Games Resources Domino games Dice games card games Newman's Prompts for...
  • Apostrophes English Bellwork English Bellwork- Monday, March 5

    Apostrophes English Bellwork English Bellwork- Monday, March 5

    When it comes to the use of apostrophe-s after a proper noun ending in s, what do is the right thing to do? A. use the -s ending only for names of people. B. only use the -s ending sometimes...
  • The Reagan and Bush Years Chapter 25 1980

    The Reagan and Bush Years Chapter 25 1980

    A.) America a nation ready for change in 1980 Low Spirits People lacked confidence in government. The turbulent 1960s, Watergate, the Soviet invasion of Afghanistan, the Iranian hostage crisis, and long gasoline lines put Americans in an uneasy mood. Critics...
  • stench - rcboe.org

    stench - rcboe.org

    uneasy. Synonym-uncomfortable, Antonym-calm, comfortable, unpleasant, nervous, worried. happy
  • "Skye Waulking Song" - Capercaillie

    "Skye Waulking Song" - Capercaillie

    "Skye Waulking Song" - Capercaillie Area of Study 4 Learning Objectives By the end of the lesson you will be able to: Gain a broader understanding relating to folk music and the set work Explain the key features in the...