1 Analysis of Contract-type and Acquisition Performance Have fixed-price contracts resulted in superior acquisition outcomes? 2 Testing conventional wisdom Some believe that more use of fixed-price contracting will reduce cost and schedule growth rates on MDAP contracts If this is true, we would expect fixed-price contracts in the past to have experienced systematically lower cost growth and schedule growth rates, other things being equal (ceteris paribus)

The hypothesized predictor is: an indicator variable (i.e. dummy variable) for fixed-price contracts But, we must control for effects of other significant predictors as well 3 The data for development contracts There were 433 contracts in the sample Break-down by contract-type: 74 CPIF 108 CPAF 100 CPFF 78 fixed-price 73 hybrid (mixture of CLIN contract types) DoD performance data for large MDAP contracts from January 1970 through December 2011

All data not adjusted for inflation (then-year) 4 The data for early production contracts There were 440 contracts in the sample Break-down by contract-type: 23 CPIF 16 CPAF 48 CPFF 293 fixed-price 60 hybrid (mixture of CLIN contract types) DoD performance data for large MDAP contracts from January 1970 through December 2011 All data not adjusted for inflation (then-year) 5

Data methodology Collected performance data by contract as recorded in DAMIR segregated by Development (433) Early production (440) Converted data into comparable metrics Percentage cost growth as difference between final reported PMs EAC and initial baseline (initial CBB) Percentage schedule growth as difference between final schedule and initial baseline schedule Percentage scope growth as difference between final reported CBB and initial baseline (initial CBB) Appropriate indicator (dummy) variables 6

Sample development contracts characteristics Sample cost growth characteristics Average cost growth: 75% Median cost growth: 33% Minimum cost growth: -61% Maximum cost growth: 1221% Only 6.7% of contracts had negative cost growth Sample schedule characteristics Average schedule growth: 32% Median schedule growth: 14% Minimum schedule growth: -49% Maximum schedule growth: 582% Only 5.8% of contracts had negative schedule growth Duration of contract Average: 6.7 years

Median: 6.2 years 7 Sample early production contracts characteristics Sample cost growth characteristics Average cost growth: 46% Median cost growth: 11% Minimum cost growth: -46% Maximum cost growth: 677% Only 21% of contracts had negative cost growth Sample schedule characteristics Average schedule growth: 30% Median schedule growth: 11% Minimum schedule growth: -34% Maximum schedule growth: 691%

Only 7% of contracts had negative schedule growth Duration of contract Average: 5.2 years Median: 4.6 years 8 Specific analysis methodology Regressed outputs on inputs or likely predictors of selected outputs Did regression diagnostics Did sample analysis using standard robust non-parametric sample tests Ran bootstrap simulations (1000) on all regressions to obtain Unbiased coefficient estimates Correct standard errors

Correct confidence intervals 9 Development regression Modeled total cost growth as a function of Scope growth Aircraft indicator UCA indicator Indicator for fixed-price contract-type IF more fixed-price contracting would have improved performance, expect the coefficient estimate to be negative and significant A constant 10

Early production regression Modeled total cost growth as a function of Scope growth Schedule growth Indicator for an Army contract Indicator for fixed-price contract-type IF more fixed-price contracting would have improved performance, expect the coefficient estimate to be negative and significant A constant 11 Results of analysis of cost growth in development phase Variable Percentage growth in work

content of contract Indicator for aircraft contract Indicator for UCA Indicator for fixed price contract-type Constant Coefficient (Standard error) Significant at 5% Level of Signicance? 1.025 (0.0200) 0.230 (0.0612)

0.0791 (0.0348) 0.0698 (0.0447) 0.0999 (0.0245) Yes Yes Yes No Yes Model p-value 0.0000*** Coefficient of determination

0.939 Adjusted coefficient of determination 0.939 12 Results of analysis of cost growth in early production phase Variable Percentage growth in work content of contract Schedule growth rate Indicator for Army Indicator for fixed price

contract-type Constant Coefficient (Standard error) Significant at 5% Level of Signicance? 1.07 (0.0304) 0.0949 (0.0327) 0.124 (0.0458) 0.0340 (0.0313)

0.0270 (0.0256) Yes Yes Yes No No Model p-value 0.0000*** Coefficient of determination 0.922 Adjusted coefficient of

determination 0.922 13 Results of non-parametric tests of development contract sample No significant difference between fixedprice and cost-reimbursable contracts with respect to total cost growth No significant difference between fixedprice and cost-reimbursable contracts with respect to schedule growth Level of significance used in analysis: 5% 14 Results of non-parametric tests of early production contract sample

No significant difference between fixedprice and cost-reimbursable contracts with respect to total cost growth No significant difference between fixedprice and cost-reimbursable contracts with respect to schedule growth Level of significance used in analysis: 5% 15 Results of analysis of development regression Regression explains 94% of variation in the data Other things being equal A 1 percentage point increase in scope growth predicts a 1.025 percentage point increase in total cost growth Aircraft contracts generally experience 23% higher cost growth than other commodity contracts Contracts with a UCA generally experience 8% higher cost growth than contracts not experiencing a UCA

There was no significant difference noted between the cost performance of fixed-price contracts and costreimbursable contracts All hypotheses were tested at the 5% level of significance 16 Results of analysis of early production regression Regression explains 92% of variation in the data Other things being equal A 1 percentage point increase in scope growth predicts a 1.07 percentage point increase in total cost growth A 1 percentage point increase in schedule growth predicts a 0.095 percentage point increase in total cost growth Army contracts generally experience 12% higher cost growth than the other two services There was no significant difference noted between the cost performance of fixed-price contracts and cost-reimbursable

contracts All hypotheses were tested at the 5% level of significance 17 Conclusions In general, managers have correctly selected contract-type for large development contracts for MDAP programs In general, managers have correctly selected contract-type for large early production contracts for MDAP programs 18 Backups

19 Results of diagnostics run on OLS regression of development contracts sample Test of normality of residuals Smirnov- Kolmogorov test: Residuals are nonnormal Shapiro- Wilk test: Residuals are non-normal Cook- Weisberg test for heteroskedasticity: Heteroskedasticity is a problem Variance inflation factor (VIF) test: Multicolinearity is not a problem Ramsey RESET omitted variable test: Model has no omitted variables Linktest: model is correctly specified 20

Results of diagnostics run on OLS regression of early production contracts sample Test of normality of residuals Smirnov- Kolmogorov test: Residuals are nonnormal Shapiro- Wilk test: Residuals are non-normal Cook- Weisberg test for heteroskedasticity: Heteroskedasticity is a problem Variance inflation factor (VIF) test: Multicolinearity is not a problem Ramsey RESET omitted variable test: Model has no omitted variables Linktest: model is correctly specified 21

Non-parametric tests used on samples Wilcoxon rank-sum test Kolmogorov- Smirnov test Median test