Production and Operations Management 2014
Managing Risks in Federal Government Information Technology Projects: Does Process Maturity Matter?
Abstract
As the Obama administration steps up oversight of high-risk IT projects, contracting organizations must take greater responsibility to provide a level of confidence in the services they offer. That is where one of the latest offerings from the Software Engineering Institute at Carnegie Mellon University can help. (Sacks 2010). A number of US federal government information technology (IT) initiatives (e.g., implementation of the Health Insurance Marketplace or Obamacare, development of navigation systems in missiles and unmanned vehicles) are frequently organized in the form of IT projects (Kundra 2010). The Office of Management and Budget (OMB), which tracks the progress of all federal IT projects indicates that such projects face significant schedule and cost overruns. Nearly 25% of federal IT projects with a cumulative budget exceeding $10 billion and spread over 28 government agencies are facing moderate to severe problems in meeting their schedule and budgetary targets (Source: www.itdashboard.gov). Additionally, the US Government Accountability Office (GAO) reports that nearly 72% of federal IT projects, with a total budget of $27 billion, are poorly planned and face significant schedule and cost overruns (US GAO Report 2010). Despite the evidence of schedule and budget overruns in federal IT projects, the challenges associated with the management of such projects, and more generally, of IT projects in the public sector, have received limited attention in both practice and research. Public sector projects differ from private sector projects in a number of ways (Boyne 2002). First, federal IT projects are primarily funded with taxpayer's money and are aimed at maximizing public utility, instead of maximizing profits, as in private sector IT projects. Hence they face greater scrutiny from the media, the US Congress, and any number of watchdog organizations. Second, federal IT projects face challenges and risks due to increased complexity, technological uncertainty, significant resource requirements, governmental rules and regulations, and the frequent involvement of multiple stakeholders with disparate and sometimes conflicting goals. To date, much of our understanding of IT project management has been drawn from studies that have focused on the private sector (McKinsey 2012). Given the notable differences between federal and private sector IT projects and growing calls in political and media circles for the efficient utilization of tax payer contributions (Fortune 2011, P. 56), an empirical investigation of challenges in federal IT projects presents a fruitful area of research with significant implications for practitioners. Our study has two major objectives. First, we identify and conceptualize a set of salient risks in federal IT projects using a lifecycle framework. As Figure 2 indicates, we focus on the planning and execution processes within an IT project and identify three distinct types of risk—namely, complexity risk and contracting risk that arise in the planning process, and execution risk that arises in the execution process. We define each of the three risks below. As per industry standards and federal legislation (i.e., the Clinger-Cohen Act of 1996), the performance of federal IT projects is reported to the OMB using earned value management (EVM) metrics for schedule and cost performance. EVM is a project planning and control approach which compares actual accomplishment of scheduled work and associated cost against an integrated schedule and budget plan on a periodic basis. In this study, we examine the performance impact of each of three risks using a composite earned value metric, that is, schedule-cost performance index (SCPI). Second, prior research has emphasized the need for mature processes to manage IT projects; which leads to improvements in the control and predictability of project outcomes. In the context of federal IT projects, the vendor's capability to reliably deliver mission-critical IT solutions (i.e., the vendor's use of mature processes within a project) is assessed using the Capability Maturity Model Integration (CMMI) framework developed by the Software Engineering Institute (SEI). The framework consists of five levels (levels 1 – 5) which assess the evolution of a firm's processes from immature and informal to mature and formal, and define the related infrastructure necessary to support these processes at an organizational level (CMMI for Development 2010). As a signal of process excellence, CMMI level 3 represents a significant step toward process maturity, with vendor certification at this level being frequently used as a key qualifying criterion by the federal government for awarding project contracts. Given the considerable commitment of time and organizational resources required to obtain CMMI certification and the performance challenges associated with federal IT projects, we examine whether higher levels of process maturity—that is, level 3 and higher—play a significant role in mitigating the effects of risk on performance in federal IT projects. The projects for this study are drawn from a proprietary database of technology projects from Lockheed Martin, a Fortune 100 global technology firm that specializes in the development of large aerospace, defense, and security systems for the federal government (i.e., the client organization). Time-series panel data are collected across 519 quarterly time periods from 82 federal IT projects that were completed during the period 2002-2012. The firm uses a rigorous two-step procedure for collecting data on federal IT projects. In the first step, tactical and project specific details are collected on a monthly basis as part of a monthly review process. The review process is typically conducted by a panel consisting of project managers, deputy project managers, and vice presidents in the IT domain within the firm. In the second step, the monthly data are aggregated to form quarterly status reports that are used for strategic review and evaluation of project performance. In addition, these reports are used to track data for internal auditing purposes and to create lessons learned. To ensure accuracy in data collection, the data are triangulated through multiple sources (e.g., interviews with project managers, project documents, etc.). Given the time-series nature of the data, we use the generalized least squares (GLS) regression method that corrects for both panel-specific autocorrelation and heteroskedasticity in the analysis. In addition, we control for a number of factors pertaining to project characteristics (e.g., project budget, project size, project priority) in our analysis. The results provide empirical support for our arguments that each of the three types of risks—complexity, contracting, and execution risks—reduce a project's ability to meet its cost and schedule targets. More importantly, our results highlight the effect of higher CMMI levels in attenuating the negative effects of project risks on performance in federal IT projects. In addition, the attenuating effect of CMMI on the risk-performance relationship is stronger at high risk levels; at low risk levels, projects with higher maturity levels (e.g., levels 4 and 5) exhibit inferior performance on schedule and cost metrics compared to projects with lower (e.g., level 3) maturity levels. To demonstrate the economic impact of increasing process maturity levels in federal IT projects, we conduct post-hoc analysis to examine the magnitude of savings (and overruns) in project costs across different levels of CMMI and project risks. This analysis is conducted in three steps. In the first step, we estimate the dependent variable (SCPI) at low (-2 SD), average, and high levels (+2 SD) of project risks across different process maturity levels, holding all control variable values at their means. Next, based on a median project budget of $35 million in the sample, we determine the estimated cost of completion (EAC) using the formula: EAC = (ProjectBudget/SCPI) × 100. Finally, we examine the differences in EAC values across different maturity and risk levels, to determine potential savings in project budgets. The results, shown in Table 1 below, highlight the potential cost savings that may result for executing projects at higher maturity levels when project risk levels are high. Specifically, given a project budget of $35 million (based on the median project budget value in our sample), executing the project at CMMI 4 or CMMI 5 when project risks are high is associated with potential savings of $11.87 million and $8.56 million, respectively, compared to executing the project at CMMI 3. In contrast, when project risk levels are low, executing the project at CMMI 4 or CMMI 5 is associated with potential cost overruns amounting to $8.27 million and $7.26 million, respectively, compared to executing the project at CMMI 3. Findings from our study make the following important contributions to the extant literature. First, our study focuses on an important and largely understudied area of research in the OM literature—the management of public sector operations (Verma et al. 2005), and particularly, the context of federal IT projects. The second contribution of our study arises from identifying and positioning IT project risks in the context of a lifecycle framework. The importance of identifying and planning for risks has been widely discussed in the extant project management literature. Our study represents a concerted attempt to conceptualize key risks in an IT project by using the project lifecycle framework, which allows us to identify and map risks by processes associated with specific project phases. The third contribution lies in developing a nuanced understanding of the mode by which process maturity influences project performance. This is important to both theory and practice given the significant investment of resources and time that is required to acquire CMMI certification. Toward this end, our results provide the following insights to managers of federal IT projects - while the implementation of CMMI 4 relative to CMMI 3 attenuates the negative performance effects of risks in the planning process only; the implementation of CMMI 5 relative to CMMI 3 level attenuates the negative performance effects of risks in both planning and execution processes. The fourth contribution of our study is based on our results that the intrinsic benefits of CMMI implementation in federal IT projects become particularly salient at high levels of project risk; at low risk levels, the benefits of higher maturity levels (i.e., CMMI 4 and CMMI 5) on project schedule and cost metrics are inferior compared to projects with CMMI 3 maturity level. We surmise that at low levels of project risk, the improvements in project performance accruing from increased levels of process maturity, may not fully compensate the costs of implementing higher CMMI levels, thereby diminishing overall project performance. The study's final contribution arises from our use of an integrated measure of schedule-cost performance (SCPI) in the context of earned value management (EVM). While vendors working on federal IT projects are mandated to use EVM for tracking and reporting project progress as per industry standards and federal legislations, the use of EVM has also grown significantly in the private sector. Though widely used in project management, there is a dearth of studies that have used EVM for evaluating project performance. Therefore, our study provides a welcome addition to the project management literature with respect to earned value management.
- DOI
- 10.1111/poms.12258_4
- Sources
- openalex