go to SAMSI home page
19 T.W. Alexander Drive
P.O.Box 14006
Research Triangle Park, NC 27709-4006
Tel: 919.685.9350
Fax: 919.685.9360

A Working Group in Development, Assessment and Utilization of Complex Computer Models


Group Leaders: Susie Bayarri ( Univ. of Valencia) and Robert Wolpert (Duke)

Password Protected Site (future)

Suggested Readings
Meeting Activities
Group Members



Meeting Time: Monday 11:00 - 1:00
Location: NISS building, Room 203

Remote access for the meeting is available.  WebEx and teleconference dial-in instructions here. Via teleconference at 919-685-9338.

Note: Webcam access is not available this year.
           Please send the needed slides / documents to fei@stat.duke.edu before the working group meeting. 

Suggested Readings

  1. Tony O’Hagan reading list. Suggested by Tony O’Hagan for MUCM.

  2. A Linear Bayes Methodology reading list, suggested by Michael Goldstein

  3. SAVE methodology from NISS project (NISS/SAMSI/Duke group).

    1. A preliminary 2002 Technical Report evolved into two shorter 2005 technical reports:

    2.   A Framework for evaluation of computer models, which specifies the methodology and gives a couple of examples (one of them an easy “pedagogical” or “toy” example, absent in the original report), and

    3.  Bayesian Validation of a Computer Model for Vehicle Crashworthiness, for smooth functional outputs and hierarchical Bayes analysis for outputs from related models or untried situations.

    4. A third Technical Report Computer Model Validation with functional Output deals with ‘rough’ functional outputs, uncertainty in the ‘controlable’ inputs, and with issues of extrapolation.

    5. Our analysis of the SANDIA challengein which each field data has different values for the calibration parameter, is available here.

  4. Statistical Sciences Group at Los Alamos National Laboratory

    1. D. Higdon, J. Gattiker and B. Williams (2005). Computer Model Calibration using High Dimensional Output. Explains a basis approach for dealing with high dimensional output. A slide on this approach is available here.

    2. D. Higdon, C. Nakhleh, J. Gattiker and B. Williams (2007). A Bayesian Calibration Approach to the Thermal Problem. LANL's solution to the V and V Challenge problem.

    3. K. Heitmann, D. Higdon, C. Nakhleh and S. Habib (2006). Cosmic Calibration. An application in cosmology.

    4. B. Williams, D. Higdon, J. Gattiker, L. Moore, M. D. McKay and S. Keller-McNulty (2006). Combining Experimental Data and Computer Simulations, with an Application to Flyer Plate Experiments. This uses a kronecker approach to deal with the time series output.

    5. M. A. Christie, J. Glimm, J. W. Grove, D. Higdon, D. H. Sharp, M. M. Wood-Schultz (2005). Error Analysis and Simulation of Complex Phenomena, in Los Alamos Science. From a different contingent at LANL; gives a slightly different perspective.

    6. D. Higdon, M. Kennedy, J. Cavendish, John Cafeo and R. D. Ryne (2003). Combining Field Data and Computer Simulations for Calibration and Prediction. A basic 1-d formulation which is very similar to the first SAMSI papers. Looks at the spotwelding application as well as one in modeling a linear accelerator.
  5. Herbie Lee’s Treed Gaussian Processes to deal with non stationary simulators

    1. Robert B. Gramacy and Herbert K. H. Lee (2006).  Bayesian treed Gaussian process models ---This is the main reference for treed Gaussian Processes.

    2. Robert B. Gramacy and Herbert K. H. Lee (2006). Adaptive design of supercomputer experiments. --- This is the main reference for adaptive sampling (using the results of a treed GP emulator to help choose new run locations)

Treed GPs and some of the adaptive design computations are in the R package tgp, which can be accessed from within R using install.packages("tgp") or from the CRAN website.

    6.  A different and promising way to deal with simulator inadequacies (bias), in which the bias is transmitted by the simulator is used in Tomassini et al. (2006). A smoothing algorithm for estimating stochastic, continuous time model parameters and an application to a simple climate model.
    7. Screening. This is an area that has not appeared much in our discussions. We discussed it some in the October joint Methodology and Engineering Workshop. The paper : Variable Selection for Gaussian Process Models in Computer Experiments (by Crystal Linkletter, Derek Bingham, Nicholas Hengartner, David Higdon, and Kenny Q. Ye) describes the Bayesian approach that Derek Bingham was referring to.

   8. Terminology. There seems to be terms like "validation" that are used with different meanings by different groups. Tony O'Hagan gave a summary of his proposed terminology in the final part of his 11-7-2006 talk. Tom Loredo told us that actually there is a specific DOE whitepaper defining how DOE expects its contractors to use the terms verification, validation, and uncertainty quantification. It is available directly from the DOE at this URL. This document was prepared for the DOE's Predictive Science Academic Alliance Program (PSAAP), a major initiative "to establish validated, large-scale, multidisciplinary, simulation-based 'Predictive Science' as a major academic and applied research program." You can learn more about PSAAP from its URL.

  9. Quantile Regression. A couple summary papers on quantile regreesions are those of Koenker and Hallock (2001) and Cade and Noon (2003); an implementation in R appears in Koenker (2006) .

 10. Non-Gaussian Spatial Processes.  Diggle et al. (1998) dicusses Generalized linear mixed model for Geostatistics.

 11. Tapering. Kaufman (2007) is considering using Covariance Tapering method to build efficient emulators for computer experiments. Two revelant papers are: Wendland (1995) introduces Piecewise polynomial, positive definite and compactly supported radial functions of minimal degree. Wendland (1998) discusses error estimates for interpolation by compactly supported radial basis functions of minimal degree.

 12. The link to the Gaussian Process website is here.

 13.- Use of derivatives (list provided by Tom Loredo).

        1 -  O'Hagan, A. (1992). Some Bayesian numerical analysis (with discussion). In Bayesian Statistics 4, (J. M. Bernardo et al Eds.), 345--363. Oxford University Press.
        2 - M. D. Morris, T. J. Mitchell, D. Ylvisaker (1993).  Bayesian design and analysis of computer experiments: Use of derivatives in surface prediction. Technometrics 35, 243-255.
        3 - K. V. Mardia, J. T. Kent, C. R. Goodall, and J. A. Little (1996).  Kriging and splines with derivative information. Biometrika, 83, 207-221.
        4 - D. J. Leith, W. E. Leithead, E. Solak, R. Murray-Smith (2002).  Divide and conquer: identification using Gaussian process priors. Proceedings of the 41st IEEE Conference on Decision and Control, 624-629.
       5- R. Murray-Smith and B. A. Pearlmutter (2003).  Transformations of Gaussian Process Priors. Tech report
       6 - E. Solak et al. (2003).  Derivative observations in Gaussian Process Models of Dynamic Systems. NIPS
       7 - J. Kocijan, A. Girard, and D. J. Leith (2003).  Incorporating Linear Local Models in Gaussian Process Model. Tech report.
       8 - K. Azman and J. Kocijan (2005).  Comprising Prior Knowledge in Dynamic Gaussian Process Models. CompSysTech 2005 Proceedings.
       9 - P. Hall and A. Yatchew (2007).  Nonparametric estimation when data on derivatives are available; to appear in Annals of Statistics, volume 35, 1. This paper deals with frequentist nonparametric estimation with derivative data. They show that knowing just first derivatives dramatically improves the convergence rate of nonparametric estimators such as KDEs, local polynomials, and splines (GP are not considered). If you know just some first derivatives, the improvement still holds, though only in the directions for which you have the derivative info.

Meeting Activities


September 2006
October 2006
November 2006
December 2006
January 2007 Febuary 2007 March 2007 April 2007

Date Topics & Readings Note
We will be having the joint SAMSI/MUCM workshop, and hence there will be no meeting of the working group. Please be reminded that there will be webex access (different from the usual one) for those of you who have e-mail Margaret about it.
We'll have an interesting brainstorming session with Michael Goldstein about the informal theme "What I'm finding currently interesting and challenging about computer model  problems". This promises to be a very stimulating discussions, and I hope you can make it. There will probably be no slides, but we'll activate the Smart Board in case we do some writing.

We will have another discussion session with Michael Golstein. This time he will be addressing the issue of relating coarse computer models with fine ones, and relating a simulator to reality. The ideas develop from the work in the  Linear Bayes Methodology reading list. Several papers there discuss issues of linking up simulators at different resolutions. Perhaps the last paper on the list "Constructing partial prior specifications for models of complex physical
." might be helpful, because it doesn't get involved in trying to solve problems with the combined specification. On the other hand "Pressure matching for hydocarbon reservoirs: a case study in the use of Bayes linear strategies for large computer experiments" does describe (more or less) how we actually went about linking two specific models and our current paper "Reified Bayesian Modelling and Inference for Physical Systems" pushes these ideas a lot further. These are some slides on `reification'.  

We will meet at 10:00-11:30. Professor Jim Zidek is going to give a talk as the following. Here are the slides.

Title:  Bayesian melding in the hunt for the elusive PRB level
Speaker: Jim Zidek, U British Columbia

Abstract: A PRB (Policy related background) level of a criterion pollutant such as ozone addressed in US regularity policy, is its (hypothetical) level if there were no anthropogenic sources in North America. It is a foundation above which to set regulatory strandards.

However it cannot be measured since pristine areas no longer exist anywhere over the continental US. Hence it is has been inferred (predicted) by deterministic CTMs (chemical transport models).

That strategy generates a lot of concerns, in particular about the accuracy and bias in the predictions. To address
those concerns, (seemingly favorable) comparisons have made of those predictions and measurements with the anthro-sources turned ON, to enhance the credibility of the predictions when those sources are turned OFF.

The Bayesian melding approach of Fuentes and Raftery offers a refined approach for comparing CTM outputs and that approach will be the main topic of this presentation. I will describe its potential for use in the hunt for the PRB level and our experiences with the method, in work that stemmed from my 6 month visit to SAMSI in 2003.  In particular it
was stimulated by conversations with Duke's  Prasad Kasibhatla, who supplied for the MAQSIP CTM outputs as well as AIRS hourly ozone concentrations we used. I will in particular describe some of the joy and DESPAIR of using MCMC for parameter high dimensional parameter vector.

No meeting.

We join the Engineering Methodology working group for a detailed exposition of a novel method to emulate dynamic models. Peter Reichert and Gentry White will be presenting.
Please  send email to the group leaders if you have new results and/or questions which would like to discuss with the group by Thursday 12:00 noon. Otherwise, we will only meet on Tuesday (12:30 - 14:00).  Email notification will be sent  if meeting will be held on Monday.
We meet again jointly with the engineering methodology working group (on their usual time slot on Tuesday 12:30 - 14:00) to continue discussion of the dynamic emulator. This might be the last of our regular scheduled meetings, but if someone has a different possibility in mind, or interesting ideas for future activities, or whatever, please take them to the meeting on Tuesday.

Group Members

Name Affiliation Email Address
Dianne Bautista Ohio State bautista (at) stat.ohio-state.edu
Susie Bayarri University of Valencia/SAMSI susie.bayarri (at) uv.es
Sunyoung Bu UNC-CH agatha (at) email.unc.edu
Tseui-long Chen N.C. State tchen3 (at) ncsu.edu
Ariel Cintron-Arias SAMSI ariel (at) samsi.info
Jim Crooks Duke/SAMSI bigjim (at) samsi.info
Tiangang Cui Univ of Auckland tcui001 (at) math.auckland.ac.nz
Michael Goldstein Durham University michael.goldstein (at) durham.ac.uk
Genetha Gray Sandia gagray (at) sandia.gov
Eitan Greenshtein N.C. State/SAMSI eitan.greenshtein (at) gmail.com
Serge Guillas Georgia Tech/SAMSI guillas (at) math.gatech.edu
Gang Han Ohio State han.191 (at) osu.edu
Leanna House Durham University house (at) stat.duke.edu
Mark Huber Duke mhuber (at) math.duke.edu
Ying Hung Georgia Tech yhung (at) isye.gatech.edu
Cari Kaufman NCAR/SAMSI cgk (at) samsi.info
Herbie Lee UC-Santa Cruz herbie (at) ams.ucsc.edu
Fei Liu Duke fei (at) stat.duke.edu
Tom Loredo Cornell loredo (at) astro.cornell.edu
Simon Lunagomez Duke/SAMSI simon.lgz (at) duke.edu
Chunsheng Ma Wichita State/SAMSI cma (at) stat.ncsu.edu
Monica Martinez-Canales Sandia National Laboratories mmarti7 (at) sandia.gov
Max Morris Iowa State mmorris (at) iastate.edu
Nancy Nichols University of Reading n.k.nichols (at) rdg.ac.uk
Tony O'Hagan Sheffield a.ohagan (at) shef.ac.uk
Abani Patra University of Buffalo abani (at) eng.buffalo.edu
Rui Paulo
Universidade Tecnica de Lisboa
rui (at) iseg.utl.pt
Luis Pericchi University of Puerto Rico luarpr (at) gmail.com
E. Bruce Pitman
University at Buffalo
Zhiguang Qian
University of Wisconsin - Madison
Peter Riechert
Swiss Federal Institute of Aquatic Science and Technology
Jonty Rougier Univ. of Bristol J.C.Rougier(at)bristol.ac.uk
Jerry Sacks
Tom Santner Ohio State santner.1 (at) osu.edu
Christine Shoemaker Cornell cas12 (at) cornell.edu
Lenny Smith Oxford University lenny (at) maths.ox.ac.uk
Elaine Spiller SAMSI spiller (at) math.duke.edu
David Steinburg
Tel Aviv University
dms (at) post.tau.ac.il
Curtis Storlie N.C. State storlie (at) stat.ncsu.edu
Matt Taddy UC-Santa Cruz taddy (at) ams.ucsc.edu
Gentry White N.C. State white (at) stat.ncsu.edu
Darren Wilkinson Newcastle University d.j.wilkinson (at) ncl.ac.uk
Robert Wolpert Duke wolpert (at) stat.duke.edu
Henry Wynn LSE h.wynn (at) lse.ac.uk
Zhengyuan Zhu



Send your comments about this site to: Susie Bayarri

Last updated on Mon, 29 Jan 2007 13:30:13 GMT.