Symbol § indicates a completely new section, chapter or appendix.

PREFACE

PREFACE TO V&V1

ACKNOWLEDGMENTS

PART I OVERVIEW

CHAPTER 1 INTRODUCTION

1.1 SKETCH OF HISTORICAL DEVELOPMENT OF COMPUTATIONAL SCIENCE AND ENGINEERING

1.2 THE NEW IMPETUS TOWARD HIGHER QUALITY SOLUTIONS.

1.2.1 ASME JOURNAL OF FLUIDS ENGINEERING 1986 - "EDITORIAL POLICY STATEMENT ON THE CONTROL OF NUMERICAL ACCURACY"

1.2.2 Δ LATER POLICY STATEMENTS AND OTHER INITIATIVES

1.2.3 Δ COMMERCIAL CODES AND USERS

1.3 PERSONAL ANECDOTE ILLUSTRATING THE IMPORTANCE OF SYSTEMATIC CODE VERIFICATION

1.4 § V&V CREDIBILITY CHECKLIST FOR NON-SPECIALISTS

CHAPTER 2 SEMANTICS: TERMINOLOGY, TAXONOMIES, DEFINITIONS

2.1 INTRODUCTION

2.2 SEMANTICS

2.3 Δ VERIFICATION AND VALIDATION: NUMERICAL VS. CONCEPTUAL MODELING

2.3.1 § DEFINITIONS AND INTERPRETATIONS OF VALIDATION

2.3.2 § DEFINITIONS AND INTERPRETATIONS OF ERROR AND UNCERTAINTY

2.3.2.1 § Numerical Errors

2.3.2.2 § Uncertainty

2.3.2.3 § Coding Errors

2.3.3 DEFINITIONS AND INTERPRETATIONS OF CODE VERIFICATION

2.3.3.2 Δ Solution Instability and Over-Stability

2.3.3.3 § No Physical Experiments in Verifications

2.4 CODE CONFIRMATION

2.5 BENCHMARKS AND INTER-CODE COMPARISONS

2.6 CODE CERTIFICATION, QUALITY ASSURANCE, AND ACCREDITATION

2.7 VERIFICATION OF CALCULATIONS (SOLUTIONS)

2.8 QUANTIFICATION OF UNCERTAINTY

2.9 GRID CONVERGENCE VS. ITERATION CONVERGENCE

2.10 ERROR TAXONOMIES

2.10.1 INADEQUATE ERROR TAXONOMIES

2.10.2 A MEANINGFUL ERROR TAXONOMY

2.10.3 GRAY AREAS: "JUSTIFICATION"

2.10.4 AN EXPANDED ERROR TAXONOMY

2.11 TRUNCATION ERROR VS. DISCRETIZATION ERROR

2.12 CALIBRATION AND TUNING

2.13 QUALITY ASSURANCE (QA) VS QUALITY WORK

2.14 CUSTOMER ILLUSIONS VS. CUSTOMER CARE

2.15 OTHER DISTINCTIONS: AUTHORS, USERS, MODELERS, CODES, AND SOFTWARE

2.16 Δ SENSITIVITY, UNCERTAINTY, AND RISK

2.17 ETYMOLOGY AND NEAR-SYNONYMS

2.18 ACCURACY VS. RELIABILITY

2.19 ADDITIONAL REMARKS ON VERIFICATION

2.20 § DOES MODEL INCLUDE THE GRID?

2.21 CONCLUSION: LIMITATIONS OF SEMANTIC DISTINCTIONS

PART II VERIFICATIONS

CHAPTER 3 METHOD OF MANUFACTURED SOLUTIONS: A METHODOLOGY FOR VERIFICATION OF CODES

3.1 INTRODUCTION

3.2 WARNINGS: THE DIVISION OF LABOR IN CODE DEVELOPMENT AND USE

3.3 ORDER OF CONVERGENCE

3.4 Δ METHOD OF MANUFACTURED SOLUTIONS

3.5 EXAMPLE: 3-D POISSON EQUATION AND NONORTHOGONAL 3-D GRID GENERATION

3.5.1 VERIFICATION OF CODE GENERATED BY SYMBOLIC MANIPULATION

3.5.2 HOSTED EQUATION CONVERGENCE TESTING METHOD

3.5.3 HOSTED EQUATION CONVERGENCE RESULTS IN 3-D

3.5.4 HOSTED EQUATION CONVERGENCE RESULTS FOR STRONG STRETCHING .

3.5.5 GRID GENERATION RESULTS IN 3-D

3.5.6 DISCUSSION OF THE CODE VERIFICATION PROCEDURE USING MANUFACTURED SOLUTIONS.

3.5.7 DEBUGGING WITH MANUFACTURED SOLUTIONS

3.6 ANOTHER PATH TO MANUFACTURED SOLUTIONS

3.7 CODE VERIFICATION INCLUDING SHOCK WAVES

3.8 NEED FOR A THEOREM

3.9 SPECIFIC ANALYTICAL SOLUTIONS

3.10 Δ MANUFACTURED SOLUTIONS VS. OTHER NUMERICAL BENCHMARKS

3.10.1 INFINITE SERIES SOLUTIONS

3.10.2 § ODE AND PDE SOLUTIONS

3.11 SENSITIVITY OF GRID CONVERGENCE TESTING

3.12 EXAMPLES OF UNANTICIPATED CONVERGENCE RATES DETERMINED BY SYSTEMATIC GRID CONVERGENCE TESTS

3.12.1 REDUCTION TO PERIODICITY METHOD: UNEQUAL ORDERS OF ACCURACY FOR DERIVATIVES.

3.12.2 COMPLETED RICHARDSON EXTRAPOLATION: HIGHER ORDER TRUNCATION ERROR INTERACTION

3.12.3 Δ THREE MORE EXAMPLES

3.13 Δ MULTIPLE SCALES, MULTIPHYSICS, AND TURBULENCE MODELING

3.14 Δ WARNINGS: WHAT THE METHOD DOES NOT "VERIFY"

3.15 ROBUSTNESS AND CONFIDENCE

3.16 § ULTIMATE RESPONSIBILITY FOR CODE VERIFICATION

3.17 § CODE VERIFICATIONS AT COMPONENT AND SYSTEM LEVELS

3.18 § FURTHER APPLICATIONS OF MMS

3.18.1 § FURTHER APPLICATIONS OF MMS OUTSIDE OF CODE VERIFICATION

3.18.2 § FURTHER APPLICATIONS OF MMS IN CODE VERIFICATION

CHAPTER 4 ERROR ESTIMATION FOR QUANTIFICATION OF UNCERTAINTY: VERIFICATION OF CALCULATIONS

4.1 INTRODUCTION

4.2 ERROR ESTIMATION FOR GRID ADAPTATION VS. QUANTIFICATION OF UNCERTAINTY

4.3 TAXONOMY FOR ADDITIONAL INFORMATION FOR ERROR ESTIMATES

4.4 GRID REFINING AND COARSENING

4.5 LEVELS OF SIMULATION USE

4.6 VERIFICATION OF COMPUTER ROUND-OFF ERRORS

4.7 Δ EFFECT OF DIFFERING FORMULATIONS

CHAPTER 5 SYSTEMATIC GRID CONVERGENCE STUDIES AND THE GRID CONVERGENCE INDEX (GCI)

5.1 Δ INTRODUCTION

5.2 BACKGROUND ON GRID CONVERGENCE REPORTING

5.3 RICHARDSON EXTRAPOLATION

5.4 Δ GENERALIZATION OF RICHARDSON EXTRAPOLATION

5.5 RICHARDSON'S EXTRAPOLATION FOR Δ

5.6 GRID CONVERGENCE INDEX FOR THE FINE GRID SOLUTION

5.6.1 Δ GRID CONVERGENCE INDEX FOR THE EXTRAPOLATED SOLUTION

5.7 GRID CONVERGENCE INDEX FOR THE COARSE GRID SOLUTION

5.8 EXAMPLE GCI CALCULATION

5.9 SHOULD THE COEFFICIENT BE "1" OR "3" OR "1.25"?

5.9.1 DETERMINING THE FACTOR OF SAFETY

5.9.2 Δ SUMMARY RECOMMENDATIONS FOR THE FACTOR OF SAFETY

5.9.3 § ERROR ESTIMATE VS UNCERTAINTY ESTIMATE OR ERROR BAR

5.9.4 § MIXED ORDER METHODS

5.10. ADDITIONAL FEATURES OF GRID CONVERGENCE STUDIES FOR VERIFICATION OF CODES AND CALCULATIONS

5.10.1 NON-INTEGER GRID REFINEMENT

5.10.2 Δ INDEPENDENT COORDINATE REFINEMENT AND MIXED ORDER METHODS

5.10.3 Δ NON-CARTESIAN GRIDS, BOUNDARY FITTED GRIDS, ADAPTIVE GRIDS, UNSTRUCTURED GRIDS

5.10.3.1 Non-Cartesian Grids and Boundary Fitted Grid

5.10.3.2 Δ Adaptive Grids

5.10.3.3 Δ Unstructured Grid

5.10.4 Δ SHOCKS, DISCONTINUITIES, SINGULARITIES

5.10.4.1 § Detection and Treatment of Singularities

5.10.5 ACHIEVING THE ASYMPTOTIC RANGE

5.10.6 Δ EXTRACTION OF THE OBSERVED ORDER OF CONVERGENCE FROM GRID CONVERGENCE TESTS

5.10.6.1 § Asymmetrical Grid Refinement

5.10.6.2 § Consistent Quadrature

5.10.6.3 § Misleading Convergence Rate

5.10.7 METHOD OF CHARACTERISTICS AND SPECTRAL METHODS

5.10.8 NON-SMOOTH PROPERTY VARIATION AND THE GCI

5.10.9 NON-SMOOTH PROPERTY VARIATION AND GEOSTATISTICAL REALIZATIONS

5.10.10 Δ ITERATION CONVERGENCE

5.10.10.1 Δ Stopping Criteria for Iteration Convergence

5.10.10.2 § Interaction of Iteration Convergence with Discretization Error

5.10.10.3 § Estimation of Incomplete Iteration Error and Uncertainty by Least-Squares

5.10.10.4 § Alternative Estimation of Incomplete Iteration Error

5.10.11 § DISCRETIZATION ERROR ESTIMATION FROM A GRID TRIPLET WITHOUT EXPLICIT EVALUATION OF P

5.11 § LEAST SQUARES GCI

5.11.1 § CHARACTERIZATION OF APPARENT GRID CONVERGENCE BEHAVIOR

5.11.2 § NOISY AND DEGRADED CONVERGENCE RATES

5.11.3 § EVALUATION OF OBSERVED P BY LEAST SQUARES

5.11.4 § REPLACEMENT OF E BY DATA RANGE

5.11.5 § CHOICE OF FS FOR THE LEAST SQUARES GCI

5.11.6 § FURTHER REFINEMENTS AND SUMMARY FOR THE LEAST SQUARES GCI

5.12 § PRACTICAL ALTERNATIVE APPROACH TO CALCULATION UNCERTAINTY

5.13 § INCORRECT ALTERNATIVE APPROACH TO UNCERTAINTY AND VALIDATION

5.14 § BAYESIAN VS STRICT FREQUENTIST INTERPRETATIONS FOR GCI

5.14.1 § REPRESENTATIVE POPULATIONS FOR COMPUTATIONAL UNCERTAINTY

5.14.2 § IMPLIED UNCERTAINTY OF AN ERROR ESTIMATE

5.14.3 § BAYESIAN VS. STRICT FREQUENTIST STATISTICS

5.14.4 § HIGH CONSEQUENCE APPLICATIONS

5.15 § EVALUATION OF UNCERTAINTY ESTIMATORS FROM SMALL SAMPLE STUDIES

5.16 § ON NOT DISCARDING OUTLIERS

5.17 § INCREMENTAL COSTS OF GRID CONVERGENCE STUDIES:

THE BLESSING OF DIMENSIONALITY

5.18 CONCLUSION

CHAPTER 6 APPLICATIONS OF SYSTEMATIC GRID CONVERGENCE STUDIES AND THE GRID CONVERGENCE INDEX (GCI)

6.1 INTRODUCTION

6.2 TWO FURTHER EXAMPLES OF (PARTIAL) CODE VERIFICATION IN GROUNDWATER FLOW

6.2.1 DARCY FLOW IN STRETCHED ORTHOGONAL COORDINATES

6.2.2 DARCY FLOW WITH TENSOR CONDUCTIVITY IN NON-ORTHOGONAL COORDINATES

6.3 ISSUES IN CALCULATION VERIFICATION

6.3.1 FORMAL, ACTUAL AND OBSERVED CONVERGENCE RATES

6.4 TWO EXAMPLES OF THE EFFECTIVE GRID REFINEMENT RATIO

6.4.1 A POSTERIORI APPLICATION OF GCI SCALING

6.4.2 § JUSTIFICATION OF EFFECTIVE GRID REFINEMENT RATIO FOR HEAT CONDUCTION.

6.5 BENCHMARK PROBLEMS FOR DRIVEN CAVITY FLOW

6.6 BENCHMARK PROBLEM FOR FREE CONVECTION

6.7 LAMINAR PLANE JET IMPINGING ON A HEATED FLAT PLATE

6.8 K- ε MODEL OF A FREE SHEAR LAYER

6.9 TRANSONIC AIRFOIL CALCULATIONS

6.10 Δ FAR FIELD BOUNDARY ERRORS

6.10.1 ORDERED ESTIMATION OF FAR-FIELD BOUNDARY ERRORS

6.10.2 § IMPORTANCE OF FAR-FIELD BOUNDARY ERRORS

6.10.3 § MAPPING FOR FAR-FIELD BOUNDARY ERRORS

6.11 ARTIFICIAL DISSIPATION EFFECTS

6.12 SINGLE AND DUAL POROSITY CONTAMINANT TRANSPORT: SOURCE LOCATION

6.12.1 TRANSPORT CODE

6.12.2 PROBLEM DEFINITION FOR SOURCE LOCATION

6.12.3 RESULTS ON SOURCE LOCATION

6.12.4 SUMMARY ON SOURCE LOCATION: 1ST-ORDER PERFORMANCE WITH A 2ND-ORDER CODE

6.13 Δ CONVERGENCE BEHAVIORS FOR MIXED-ORDER METHODS

6.14 GRID CONVERGENCE OF ZERO DRAG COEFFICIENT

6.15 ANOMALOUS RESULT POSSIBLY DUE TO GRID STRETCHING

6.16 NON-SMOOTH PROPERTY VARIATION: GLOBAL ERROR NORMS

6.17 Δ DIFFICULTIES OF SPECIAL METHODS

6.17.1 DISCRETE VORTEX METHODS

6.17.2 § LES AND DNS METHODS

6.18 OBSERVED CONVERGENCE RATES FOR EULER EQUATIONS WITH SHOCKS

6.19 COMPLETED RICHARDSON EXTRAPOLATION

6.20 TRUNCATION ERROR IN ELLIPTIC GRID GENERATION

6.21 ONE DIMENSIONAL MOVING ADAPTIVE GRID PROBLEMS

6.22 GCI APPLICATION IN SOLUTION-ADAPTIVE NON-INTEGER GRID REFINEMENT

6.23 HIGH QUALITY GRID STUDIES LEADING TO A SAFETY FACTOR OF 1.25

6.23.1 ORIGINAL STUDIES

6.23.2 § TERRASSA GROUP RESULTS

6.23.3 § CONFINED DETONATION PROBLEM BY SWRI

6.23.4 § OTHER GCI APPLICATIONS AT SWRI

6.23.5 § IIHR COMPILATION AND LLNL STUDY

6.23.6 § LISBON V&V WORKSHOPS

6.23.6 § COMMON SENSE AND THE GCI FACTOR OF SAFETY

6.24 TRANSPORT CODE VERIFICATIONS USING THE GCI: PARTITIONING THE OPTION MATRIX

6.25 Δ OTHER UNCERTAINTY ESTIMATORS BASED ON RICHARDSON EXTRAPOLATION

6.25.1 Δ CELIK AND KARATEKIN METHOD FOR TURBULENT SEPARATED FLOW

6.25.2 § ITTC CORRECTION FACTOR METHOD

6.26 LEVEL OF ACCURACY ESTIMATES FROM GRID CONVERGENCE STUDIES

6.27 OTHER EXAMPLES OF CAREFUL USE OF RICHARDSON EXTRAPOLATION

6.27.1 FLUID DYNAMICS EXAMPLES

6.27.2 § QUANTUM CHROMODYNAMICS CALCULATION IN 4-D LATTICE

6.28 PARAMETER CONVERGENCES OF A COMPRESSIBLE FLOW CODE NEAR THE INCOMPRESSIBLE LIMIT

6.29 JUSTIFICATION OF THE DUPUIT APPROXIMATION.

6.30 PARAMETER UNCERTAINTY AND NUMERICAL UNCERTAINTY

6.31 § PARAMETER UNCERTAINTY AND MODEL FORM UNCERTAINTY

6.32 § PARAMETER UNCERTAINTY IN VALIDATION VS PREDICTIVE ANALYSIS

6.33 § PARAMETER UNCERTAINTY AND THE ECONOMICS OF GRID CONVERGENCE STUDIES

CHAPTER 7 SINGLE GRID ERROR ESTIMATORS

7.1 ERROR ESTIMATION FROM HIGHER OR LOWER ORDER ACCURACY SOLUTIONS ON THE SAME GRID (CATEGORY B)

7.1.1 HIGHER ORDER ACCURACY SOLUTIONS (CATEGORY B.1)

7.1.2 LOWER ORDER ACCURACY SOLUTIONS (CATEGORY B.2)

7.2 Δ AUXILIARY PDE SOLUTIONS ON THE SAME GRID (CATEGORY C)

7.2.1 Δ ERROR TRANSPORT EQUATIONS

7.2.2 § ADJOINT EQUATIONS

7.3 Δ AUXILIARY ALGEBRAIC EVALUATIONS ON THE SAME GRID: SURROGATE ESTIMATORS (CATEGORY D)

7.3.1 NON-CONSERVATION OF CONSERVATION VARIABLES (CATEGORY D.1)

7.3.2 NON-CONSERVATION OF HIGHER MOMENTS (CATEGORY D.2)

7.3.3 Δ ZHU-ZIENKIEWICZ AND WIBERG TYPE ESTIMATORS (CATEGORY D.3)

7.3.4 CONVERGENCE OF HIGHER ORDER QUADRATURES (CATEGORY D.4)

7.4 TIME ACCURACY ESTIMATION

7.5 § UNCERTAINTY ESTIMATES FROM SINGLE GRID ERROR ESTIMATORS

7.6 § COMPARISON OF GCI AND SINGLE-GRID UNCERTAINTY ESTIMATORS

7.7 § VERIFICATION WITHIN SOLUTION ADAPTATION

7.8 § CONCLUDING REMARKS ON SINGLE-GRID ERROR ESTIMATORS

CHAPTER 8 HARD STORIES

8.1 FACTORS INFLUENCING CONVERGENCE RATES

8.1.1 HIGHER ORDER TRUNCATION TERM COMPETITION

8.1.2 THE EFFECT OF SPACE-TIME TRUNCATION TERM CANCELLATION AND SUPERCONVERGENCE

8.1.3 EFFECT OF PHYSICAL PARAMETER RESOLUTION ON GRID CONVERGENCE

8.1.4 SUMMARY OF FORMAL VS. ACTUAL ASYMPTOTIC VS. OBSERVED CONVERGENCE RATES

8.1.5 OTHER CONSIDERATIONS IN DEFINING CONVERGENCE RATES

8.1.6 DEFINING WAVENUMBER DEPENDENT CONVERGENCE

8.1.7 ARTIFICIAL FLOW FEATURES

8.1.8 L2 AND L∞ NORMS AS ERROR INDICATORS; CAFE CURVES

8.1.9 SUMMARY OF OTHER CONSIDERATIONS IN DEFINING CONVERGENCE RATES

8.2 BEHAVIOR OF QUASI-HIGHER-ORDER METHODS

8.3 Δ SOME GOOD NEWS FOR TURBULENCE MODELING

8.4 MYTH OF THE "CONVERGED SOLUTION"

8.5 ESOTERIC CODING MISTAKES

8.6 FALSE VERIFICATION TEST OF A PARTICLE TRACKER

8.6.1 SPATIAL CONVERGENCE OF TRACKER CODES

8.6.2 TEMPORAL CONVERGENCE OF TRACKER CODES: A FALSE NEGATIVE TEST

8.7 INADEQUACY OF SINGLE GRID CALCULATIONS FOR PARAMETER TRENDS

8.8 HARD-WIRED DATA VS. USER INPUT DATA

8.9 DEGRADED RATE OF CONVERGENCE DUE TO USER MODELING ERRORS

8.10 Δ LESSONS FROM NONLINEAR DYNAMICS

8.11 ADAPTIVE AND LOCAL TIME STEPPING, AND STEADY STATE

8.12 OTHER QUESTIONS RELATED TO THE STEADY STATE

8.13 § LAGRANGIAN CALCULATIONS

8.14 § LEAST SQUARES GCI FOR NOISY CONVERGENCE (RANS)

8.14.1 § OVERCOMING FALSE INDICATION OF CONVERGENCE DUE TO SAMPLING

8.14.2 § DON'T SHOOT THE MESSENGER!

PART III VALIDATION

CHAPTER 9 DIFFICULTIES WITH EXPERIMENTS AND VALIDATION.

9.1 CREDULOUSNESS

9.2 Δ VALIDATION IN SCIENCE THEORY AND COMPUTATION

9.2.1 HISTORICAL METHODS OF VALIDATING SCIENTIFIC THEORIES

9.2.2 § OBJECTIONS TO VALIDATION BASED ON THE PHILOSOPHY OF KARL POPPER

9.2.3 § VALIDATION IN ECOLOGICAL MODELING

9.2.4 § VALIDATING TEMPORAL "PREDICTIONS" VS OUTCOMES

9.3 THEORY-LADEN EXPERIMENT

9.4 RANDOM AND SYSTEMATIC ERRORS IN EXPERIMENTS

9.5 EXPERIMENTAL ERRORS IN PHYSICAL PROPERTIES

9.6 BOUNDARY CONDITIONS, CONTINUUM AND NUMERICAL

9.7 TRENDS, COMPUTATIONAL AND EXPERIMENTAL

9.8 FALSE NEGATIVES AND FALSE POSITIVES

9.9 "NEARBY" PROBLEMS

9.10 DIFFICULTY OF THE OPTION TREE

9.11 DATA SPARSITY AND LACK OF SYNCHRONICITY: GROUNDWATER, OCEAN/LAKE, AND METEOROLOGY MODELING

9.12 EFFECT OF PARAMETER RESOLUTION ON GRID CONVERGENCE

9.13 SCALE OF UNSTEADINESS

9.14 SPATIAL SCALES, SCALING UP, AND DIMENSIONALITY

9.15 ASSUMPTIONS OF PERIODICITY

9.16 OTHER DIFFICULTIES OF VALIDATION IN AEROSPACE

9.17 UNIVERSAL TURBULENCE MODELS VS. ZONAL MODELING

9.18 Δ SPECIFIC AND GENERAL SENSES OF MODEL AND MODEL VALIDATION

9.19 MYTH OF THE "TOTALLY VALIDATED CODE"

9.20 § FRAUDULENCE IN FINANCIAL RISK MODELING

9.21 § NEED FOR CONTROLLED AND MEASURED EXPERIMENTS

CHAPTER 10 VALIDATIONS BY ERROR BARS

10.1 SOURCES OF PHYSICAL MODELING ERRORS IN AERODYNAMICS CFD

10.2 ACCURACY LEVEL FOR VALIDATION /CERTIFICATION

10.3 GENERIC MODELS VS. REALISTIC MODELS FOR VALIDATION/CERTIFICATION AND CALIBRATION: PHASES OF VALIDATION/CERTIFICATION

10.4 CFD AND EXPERIMENTAL FACILITY CORRECTIONS

10.5 VERIFICATION MUST BE INDEPENDENT OF VALIDATION: AIRFOIL CALCULATIONS

10.6 SYNERGISM BETWEEN COMPUTATION AND VALIDATION EXPERIMENTS

10.6.1 ARTIFICIAL HEART VALVES

10.6.2 TRANSONIC FLOW

10.6.3 SIMPLE-TO-COMPLEX GEOMETRY FLOWS

10.6.4 OPERATION OF ADAPTIVE WIND TUNNEL WALLS

10.6.5 BOUNDARY LAYER TRANSITION

10.7 DIFFICULTY OF DEFINING A "NEARBY" PROBLEM

10.8 MISSING EXPERIMENTAL INFORMATION

10.9 ONSET OF 3-DIMENSIONALITY IN BACKSTEP FLOW

10.10 GRAY AREA: "VALIDATION" FROM A CALCULATED BENCHMARK

10.11 GRAY AREA: "VALIDATION" OF AN EXPERIMENTAL TECHNIQUE BY A COMPUTATION

10.12 THE MADE-2 EXPERIENCE: CAN GROUNDWATER FLOW MODELS BE VALIDATED?

10.13 DYNAMIC STALL WIND TUNNEL DATA: WHO DOES THE TWEAKING?

10.14 CONSORTIUM EFFORT AT CFD CODE CERTIFICATION

10.14.1 ISOLATED WING C

10.14.2 NOZZLE/BOATTAIL FLOWS

10.14.3 WING-BODY COMBINATION

10.14.4 TURBINE/COMPRESSOR

10.14.5 SELECTED OBSERVATIONS OF MELNIK ET AL

10.15 SIMULATION TEAM RESPONSIBILITIES IN VALIDATION/CERTIFICATION

10.16 SHIFTING RESPONSIBILITIES AND GRAY AREAS

10.17 WUA BENCHMARKS IN 1994 AND 1996

10.18 CFD TRIATHLONS

10.19 CANADIAN CFD SOCIETY TEST CASE

10.20 WORKSHOPS

10.20.1 OLDER WORKSHOPS

10.20.2 § LISBON III V&V WORKSHOP

10.20.3 § INL NUCLEAR SYSTEMS V&V WORKSHOP

10.20.4 § AIAA DRAG PREDICTION WORKSHOPS

10.21 AGARD 1988 VALIDATION OF COMPUTATIONAL FLUID DYNAMICS

10.22 CASE STUDY FOR CFD CODE VALIDATION METHODOLOGY

10.23 Δ DYNAMIC DATABASES FOR VALIDATION

10.24 § STATISTICALLY ASSESSING STATE OF THE ART

10.25 Δ JOINT CONSIDERATION OF EXPERIMENTAL AND SIMULATION UNCERTAINTIES

CHAPTER 11 § VALIDATION UNCERTAINTY: ASME ANSI STANDARD V&V 20

11.2 § V&V20 BACKGROUND AND MOTIVATION

11.3 § ERRORS AND UNCERTAINTIES

11.4 § DEFINING VALIDATION UNCERTAINTY

11.5 § ESTIMATING VALIDATION STANDARD UNCERTAINTY

11.6 § PARAMETER ERRORS AND MODEL FORM ERRORS

11.7 § ESTIMATING PARAMETRIC UNCERTAINTIES

11.8 § INTERPRETATION OF VALIDATION RESULTS USING STANDARD UNCERTAINTIES

11.8.1 § INTERPRETATION OF VALIDATION RESULTS USING E AND UVAL WITHOUT ASSUMPTIONS MADE ABOUT ERROR DISTRIBUTIONS

11.8.2 § INTERPRETATION OF VALIDATION RESULTS USING E AND UVAL WITH ASSUMPTIONS MADE ABOUT ERROR DISTRIBUTIONS

11.9 § ESTIMATING VALIDATION PROBABILISTIC UNCERTAINTY

11.9.1 § STANDARD UNCERTAINTY VS. "EXPANDED" OR PROBABILISTIC UNCERTAINTY

11.9.2 § COMBINING STANDARD UNCERTAINTY AND PROBABILISTIC UNCERTAINTY

11.9.3 § SUMMARY PROCEDURE FOR PROBABILISTIC VALIDATION UNCERTAINTY

11.10 § INTERPRETATION OF VALIDATION RESULTS USING PROBABILISTIC UNCERTAINTIES

11.11 § MODEL QUALITY VS. VALIDATION QUALITY

11.12 § EXTENDING THE DOMAIN OF VALIDATION

11.12.1 § INTERPOLATION METHODS

11.12.2 § ESTIMATING δMODEL BY INTERPOLATION

11.12.3 § ESTIMATING UδMODEL BY INTERPOLATION

11.12.4 § REPORTING NEW MODELING RESULTS

11.12.5 § DIVISION OF RESPONSIBILITIES

PART IV BROADER ISSUES

CHAPTER 12 QUALITY ASSURANCE ISSUES

12.1 INTRODUCTION

12.2 QUALITY ASSURANCE (QA) VS QUALITY WORK

12.3 QA VS CREATIVITY

12.4 QA AND TEMPERAMENT TYPES

12.5 PREVALENCE OF ERRORS IN SCIENTIFIC SOFTWARE: USE OF STATIC ANALYZERS

12.5.1 THE "T" EXPERIMENTS OF HATTON

12.5.2 USE OF STATIC ANALYZERS

12.6 CODE DOCUMENTATION

12.7 COMMERCIAL CODES AND THEIR USERS

12.8 CODE TO CODE COMPARISONS

12.9 QA FOR LARGE PUBLIC POLICY PROJECTS

12.10 QA OF ANALYSES

12.11 QA / CERTIFICATION OF USERS AND REGULATORS

12.12 ASSESSMENT OF CODES? OR USERS?

12.13 § VALIDATION CLAIMS WITH USER-SPECIFIED INPUT PARAMETERS

12.14 OTHER QA ASPECTS

12.15 CONCLUDING REMARKS ON QA

CHAPTER 13 CONCLUSIONS

13.1 THE OVERALL PROCESS FOR QUANTIFICATION OF UNCERTAINTY

13.2 FULFILLING THE PROMISE OF COMPUTATIONAL POWER

REFERENCES AND BIBLIOGRAPHY

APPENDIX A

NEED FOR CONTROL OF NUMERICAL ACCURACY

ABSTRACT

I. INTRODUCTION

II. RESISTANCE AND OBJECTIONS

III. DIFFICULTIES IN APPLICATIONS

IV. EXAMPLES OF WHAT CAN BE DONE

V. CONCLUSIONS AND RECOMMENDATIONS

APPENDIX: EDITORIAL POLICY STATEMENT ON THE CONTROL OF NUMERICAL ACCURACY

ACKNOWLEDGMENTS

REFERENCES FOR APPENDIX A

APPENDIX B. § VALIDATION - WHAT DOES IT MEAN ?

ABSTRACT

INTRODUCTION

HISTORY OF THE DEFINITION

ISSUE #1. ACCEPTABILITY CRITERIA (PASS/FAIL)

ISSUE #2. NECESSITY FOR EXPERIMENTAL DATA

ISSUE #3. INTENDED USE

RECOMMENDED INTERPRETATION AND ALTERNATIVE DESCRIPTION

RECOMMENDATION ON ISSUE #1

RECOMMENDATION ON ISSUE #2

RECOMMENDATION ON ISSUE #3

ALTERNATIVE DESCRIPTION

CALIBRATION IS NOT VALIDATION

IMPLICATIONS FOR CONTRACTUAL AND REGULATORY REQUIREMENTS

ACKNOWLEDGEMENTS

REFERENCES FOR APPENDIX B

APPENDIX C. § TUTORIAL ON CODE VERIFICATION BY THE METHOD OF MANUFACTURED SOLUTIONS

ABSTRACT

INTRODUCTION

THE METHOD OF MANUFACTURED SOLUTIONS

THREE EXAMPLE PROBLEMS IN MMS

APPLICATION TO VERIFICATION OF CODES

RECENT WORK AND FURTHER DISCUSSION

BLIND STUDY

TWO MULTIDIMENSIONAL FEATURES MIXED ORDER METHODS

RADIATION TRANSPORT CODE INCLUDING EIGENVALUE PROBLEMS

NONHOMOGENEOUS BOUNDARY CONDITIONS

NONLINEAR BOUNDARY CONDITIONS

SHOCKS

REQUIREMENT FOR SOURCE TERMS

SOLUTION REALISM

CODE VERIFICATION WITH A CLEARLY DEFINED COMPLETION POINT

PROOF?

AN ALTERNATIVE VIEW ON CODE VERIFICATION WITH A CLEARLY DEFINED COMPLETION POINT

CONCLUDING REMARKS

ACKNOWLEDGMENTS

REFERENCES FOR APPENDIX C

APPENDIX D. § BASIC FORMULAS FOR V&V

D.1 CODE VERIFICATION

D.2 SOLUTION VERIFICATION

D.3 VALIDATION

APPENDIX E. Δ BIOGRAPHICAL SKETCH OF LEWIS FRY RICHARDSON

SUBJECT INDEX