Anti-PLA Study Bought by ABC Debunked by Reputable Economist
July 22, 2011 - ABC is hoping to generate anti-worker press coverage this week by releasing a study it bought and paid for from a right-wing think tank, that not surprisingly, purports to conclude that PLAs increase the cost of school construction. They have distributed the report to the news media and have asked that it be released on Friday,
Before its release, however, respected labor economist Dr. Dale Belman of Michigan State University has already published a response to the authors, finding that their conclusion is not even supported by their own research. In fact, Dr. Belman states, the report’s research actually supports the opposite conclusion: “When appropriate controls are included for differences in the characteristics of schools built including school type and location, building pecifications, materials used etc., there is no statistical evidence that PLA schools are more costly compared to non PLA schools.”
Below is a link to the report, and following that is Dr. Belman’s response in its entirety. http://www.nusinstitute.org/assets/resources/pageResources/Measuring-the-Cost-of-Project-Labor-Agreements-on-School-Construction-in-California.pdf
July 18, 2011
Mr. Vince Vasquez, Mr. W. Erik Bruvold and Dr. Dale Glaser
National University System, Institute for Policy Studies
11355 North Torrey Pines Road
La Jolla, California 92037
Dear Mr. Vasquez, Mr. Bruvold and Dr. Glaser:
I have read your study, Measuring the Cost of Project Labor Agreements on School Construction in California, with great interest. As you know, I have researched and published studies on project labor agreements (PLAs) and school construction costs including Project Labor Agreements’ Effect on School Construction Costs in Massachusetts, Industrial Relations 49, no. 1 (2010) and remain interested in all new research in this area.
I have reviewed your work closely to assure that I understood your data, model and methods. I find that your study’s conclusion is not supported by your research; that you have overlooked important factors that affect costs, and that you have misinterpreted and drawn erroneous conclusions from my work; mistakes that I hope you will want to correct. Correctly interpreted, your results are basically consistent with those presented in my article on PLAs and Massachusetts school construction costs. The take-away from your results can be summarized as follows: When appropriate controls are included for differences in the characteristics of schools built including school type and location, building specifications, materials used etc., there is no statistical evidence that PLA schools are more costly compared to non PLA schools.
Since you may not realize that this is what your research results mean, let me provide some detail:
•As I have shown in the past, it is challenging to separate out the effect of PLAs on school construction costs from the effect of the specific characteristics of particular projects. Because different schools can have very different construction specifications and can differ in other ways, it is critical to separate out any PLA effect from the effect of building characteristics. If the effect of school characteristics is not distinguished from any possible PLA effect, there will be omitted variable bias which, potentially, completely invalidate the research results.
• Because of the set of factors used to explain school construction costs is so limited (you have only six explanatory factors other than PLA) there is considerable risk of omitted variable bias. For example, although all workers on school construction projects are paid prevailing wage, the prevailing wage varies systematically by region. If high prevailing wage regions, such as the Los Angeles area, are more likely to use PLAs, your PLA variable will be biased upward by the omission of a control for differences in prevailing wages between regions. A control, such as the prevailing wage for a benchmark trade such as carpenter, at the time the school was constructed, would likely have been sufficient to remove the bias. Similarly, if PLAs are used for schools built to higher seismic standards, and controls for differences in construction are not incorporated into the model, the measure of the cost effect of PLAs will be biased upward. 1
• There is considerable evidence in your study of omitted variable bias in the estimates provided in chart 6. The estimates with a sample of PLA and non-PLA schools which are matched on their characteristics better control for the characteristics of schools and of the construction environment than other estimates in the report. On page 15, you write:
“In our second phase, we analyzed the matched set of 130 projects (incorporating a propensity weight covariate) using the ordinary least squares method. We found that PLAs were not statistically significant. Similar results were found when the propensity score was omitted from the model.”
The implications from this are clear, but downplayed in the report: when the model better controls for differences in characteristics between PLA and non-PLA schools, PLAs do not affect school construction costs.
• This section also indicates that, parallel to my work, there are statistically meaningful differences between PLA and non PLA schools and that the majority of schools built without PLAs are unlike schools built with PLAs. These differences suggest that PLAs are, as they should be, used on challenging projects rather than “plain vanilla” schools.
• The estimates in Chart 7 also indicate that your study suffers from omitted variable bias. Similar to my work, you find that, when controls for construction in a large urban district are included in the model, the PLA variable is no longer statistically significant. The district in question, LAUSD, builds to higher seismic standards than other school districts and is more likely to build multi-story steel structures which differ considerably from typical schools. When a control for construction by the LAUSD school district is included in your model (Chart 7), the PLA variable becomes small in magnitude and is far from statistical significance. Again, this is consistent with omitted variable bias.
• I am concerned that your results do not provide apples-to-apples comparisons. For example, when you estimate a model which excludes LAUSD schools, you change the explanatory variables in the specification without explanation. As a result, the reader doesn’t know whether the seeming positive effect of the PLA variable in this sample is indeed a positive effect, or the result of changing the specification when you switch your data sample. To avoid concern about manipulating your results, you need to use the same model when testing for PLA cost effects on data using LAUSD data and when excluding that data.
• Another apples-to-oranges comparison in your research of the mixing of rehab, renovation and remodeling data in with new construction. This is a bad idea simply because the specific needs of individual renovation projects can vary so widely. One school might just need a roof while another might require a rebuild to meet earthquake seismic standards. The two schools could have exactly the same square foot size and greatly different square foot renovation costs. These kinds of projects should not be lumped together, much less thrown in with new construction. A cleaner data set would have used new construction only to avoid apples-to-oranges comparison mistakes. But at the very least, you should have had a control variable in your model indicating whether the project was a renovation project and what type of renovation was done.
•Some of the results reported in Chart 6 are not sensible. An implication of your model is that a very large school will cost nothing. While this might be viewed as good news for financially stressed school boards, it is obviously the result of a misspecified variable. Similarly, your model indicates that school costs rise without limit by $7.50 per square foot each year. It is not credible that school construction costs will rise by $75 per square foot between now and 2021, or by $300 per square foot by 2051 simply because of the passage of time. This is obviously wrong and again, suggests serious misspecification.
• Another apples-to-oranges mistake in your work is failing to use clustered errors to allow for common factors affecting school costs among schools in a single school system. School district construction policies can be very different across school districts. As noted previously, the LAUSD builds to different seismic standards than many other school districts and this certainly affects both costs and the error term of schools built in the LAUSD. This is a technical point but an important one: errors in your estimates will not be independent across observations and your estimates of statistical significance will be wrong. The large effect of controlling for the LAUSD on the estimates strongly suggests that observations are not independent.
There are also some errors with respect to my work. The 2005 paper is an early version of the article which, having gone through peer review, appeared in the January 2010 issue of Industrial Relations. The 2010 work builds on the prior working paper and extends that work, it would be most appropriate to use only the 2010 version. Also, your chart 4 took the estimate of the PLA effect from a model which I was using to demonstrate the effects of under specification on estimates of PLA effects. My final conclusion, based on the whole of my work, was that there was considerable evidence that PLAs did not affect school costs, but that it was difficult to separate out the effects of PLAs from the effects of characteristics which cause PLAs to be used in school construction.
All said, I was pleased to review your report and find that, similar to my work, it supports the view that PLAs do not affect the cost of construction of schools. I doubt that is the conclusion which you intended, but it is clearly there in your results. To the degree you disagree with this, the appropriate forum for deciding the merit of your work would be a peer reviewed journal. This is the accepted avenue for the evaluation of research as it provides review by disinterested experts on the subject and methodology. I would suggest you try Industrial Relations, The Industrial and Labor Relations Review or Economic Inquiry as these are journals which are well respected and are likely to be interested in this issue.
I am interested in taking a closer look at the data and would be most grateful if you would share your data with me; I would be happy to provide you with the data my colleagues and I collected from Massachusetts.
School of Human Resources and Labor Relations
Michigan State University
1The problem of confusing the effect of PLAs on costs with the higher costs of project on which PLAs are used can be illustrated with a hypothetical example. A mile of above-ground light-rail track in Los Angeles costs about $80 million. A mile of below ground track, with tunneling and other challenges, costs about $400 million. The end result of both projects is a mile of track. If, because of the challenges of the below ground project, a PLA is used on the below ground segments, and if we don’t allow for the differences between above and below ground construction, we will erroneously conclude that PLAs raise the cost of construction by several orders of magnitude. Our PLA measure is capturing not only any effects PLAs may have on construction costs, but also the higher costs of building below ground. If we include measures of factors related to the added costs of below ground construction to our model, it may be possible to get an accurate measure of the effect of PLAs on cost. Given that architects and engineers use more than 100 characteristics in developing cost estimates for schools, it is doubtful that the six which are used in the Chart 6 and 7 models are sufficient to guard against omitted variable bias.
Print this Page