Print
E-MailRegarding the recent posting by Mr. Agrawal and Professor Kaplan,
Ashwini Agrawal, a graduate student at the University of Chicago, posted a paper on this blog that used a statistical model whose key variables were custom built by him to assert that the AFL-CIO votes its public company proxies based not on proxy voting guidelines, but on the union affiliation of public company employees. Through a series of e-mails (he has refused to meet in person or communicate over the phone) we told him he was completely and utterly wrong and asked him to release his data set. Mr. Agrawal accused the AFL-CIO of not responding to his questions after refusing to meet or release his data. University of Chicago Professor Steven Kaplan, who is advising Mr. Agrawal on this project, wrote a lengthy post defending these opaque methods.
Mr. Agrawal’s claim that he contacted the AFL-CIO and was denied information is false. Mr. Agrawal has never contacted a member of the AFL-CIO program staff to discuss his paper or ask for any data, and has refused every opportunity to meet and ask us questions.
Both posts also contain a series of important contradictions. Professor Kaplan and Mr. Agrawal repeatedly assert that the study can be easily replicated using publicly available sources of data. Kaplan emphasizes that this is “an important point. It does not rely on data that can be shaded by an interested party.”
Despite this, Kaplan later asserts that “in putting together a data set, a researcher spends a great deal of time and effort.” Which is it then? Is it a lengthy endeavor worthy of “great time and effort,” or something that’s “easily replicated?”
We continue to demand access to Agrawal’s data because it cannot be replicated. His data collection efforts were more subjective than mechanical. For example, when data on company unionization was incomplete Mr. Agrawal relied on information “from the Investor Relations departments of firms themselves.” [Appendix A, pg. 29]
The difficulty of replicating this skewed effort at data collection is obvious. How would the AFL-CIO go about determining which companies he contacted directly? Should we selectively call random Investor Relations departments and ask for the individual who spoke with Mr. Agrawal two years ago? What if the person he spoke with no longer works at the company? How do we know what source the Investor Relations Department used, and was it the same across all companies? Was a record of his phone conversations kept to back up his methodology?
Mr. Agrawal and Professor Kaplan assert that his paper has not been published, and that because it is not published they should be able to keep their data secret. It’s true that it hasn’t appeared in any peer reviewed setting–but it has been twice cited on the editorial page of the Wall Street Journal as evidence for repeated false accusations against the AFL-CIO, as well as being posted on this blog and widely circulated in academic and business circles.
Professor Kaplan’s defense that they won’t release data to a competing researcher is misplaced. We are the subject of a widely published study which makes false accusations based on unreproduceable statistical models. We are not seeking to complete a research project for a rival journal, but instead correct the record.
We would be happy to receive Mr. Agrawal’s data on the strict condition that we won’t turn it over to competing researchers or publish it in a competing paper. As outlined above, we need to review the accuracy of Mr. Agrawal’s data and statistical model, and when given the opportunity to talk to him, inform him of the serious flaws in his research.
A copy of the AFL-CIO’s recent report, Facts About the AFL-CIO’s Proxy Votes, is available here. We repeat our request that Mr. Agrawal release his data set or withdraw his paper.
7 Comments
Methinks Mr. Pedroty protesteth too much. It’s rather unattractive for the might AFL-CIO to be bullying a graduate student, and his thesis committee, just because his research comes to conclusions they don’t like.
It’s not a requirement to duplicate Mr. Agrawal’s exact data set. If the effect he noticed truly exists, anyone who reproduces the effort to gather information on the subject from publicly available information and applies his analysis methods will be able to duplicate his findings.
Mr. Pedroty’s position is obvious, it is much easier for him to copy Mr. Agrawal’s data set and try to poke holes in it, than it is for him to gather up a representative data set from scratch and apply the analysis himself. And if Mr. Pedroty picks and chooses his data to provide the result he desires, it will become very obvious when the data sets are eventually compared. He’s in a no-win situation, and nitpicking about about “contradictions” such as “easy” vs. “requires effort” is very weak.
As a former graduate student, “great time and effort” and “easy to replicate” are not contradictions. Replication of a statistical procedure is DAMN easy–sometimes just a few lines of code. This is “easy” because it’s not mentally taxing to to run a model.
Data, on the other hand, takes a long time to clean and assemble, great care needs to be taken to make sure the data represents the correct information you want to analyze. It’s tedious and requires thought, but once someone does it the results are easy enough to replicate. So it makes sense that Mr. Agrawal doesn’t want to hand over hundreds of hours of work–but a good paper always explains how to do assemble the data from the same sources in the same way (takes the thinking part out, so it’s EASY). AFL-CIO can pay for their own data monkeys to follow the instructions laid out in the paper and replicate. Hence, EASY.
“… we need to review the accuracy of Mr. Agrawal’s data and statistical model, and when given the opportunity to talk to him, inform him of the serious flaws in his research.”
Apparently, Mr. Pedrotty has become certain there are “serious flaws” even before he has had a chance to “review the accuracy” of the model.
I’m impressed by Mr. Pedrotty’s confidence, even before he has seen the data and statistical model, that the research contains “serious flaws”!
“Which is it then? Is it a lengthy endeavor worthy of ‘great time and effort,’ or something that’s ‘easily replicated?’”
I guess it’s both. Digging a ditch by hand is easy, but I’d still rather hire a backhoe. If he doesn’t want to replicate the data himself, hire a graduate student… at union wages. Either way, he’ll understand why Mr. Agrawal doesn’t just give the data away.
“The difficulty of replicating this skewed effort at data collection is obvious.”
Ummm, this is Mr. Kaplan’s whole point. If you think the data is skewed or misinterpreted, dig your own ditch – I mean, assemble your own data. Do your own analysis. Then show what the difference is.
Once you have two contradictory sets of data, then is the time to compare votes, companies, and union affiliation data and see who missed or erred in what. There are far too many studies in the Soft Sciences worlds that simply consist of churning the same chamber pot collections over and over. A new study should be built of new logs.
Instead of all the ideological posturing, please do not post here unless you have read the study and considered whether or not it is methodologically sound. I suspect that the paper might get published because most economists, including the author of the paper, don’t know very much about union structure and politics, only outcomes. But there are serious flaws in the paper arising from its failure to recognize institutional and timing issues that undermine its methodology.
1. The study compared apples and oranges: a small staff pension fund for employees of the national labor federation (AFLCIO) and a huge Taft-Hartley multiemployer fund for construction workers (members of the Carpenters). The consequences of this problem are not discussed in the study.
2. The paper has a problematic timing problem.
The data are supposed to capture the breaking away of several unions from the AFLCIO into a new federation called Change To Win (CTW). The time period of the study is 2003-2006. But the Carpenters disaffiliated from the AFL-CIO in 2001. So one would expect, if the paper’s thesis is correct (that union pension funds vote in the interests of their members rather than their beneficiaries) that the Carpenters would already, prior to the sampling period, have changed their voting. This too is not mentioned in the paper.
3. One might argue that the relevant change point occurs not when the Carpenters left the AFLCIO but occurs at the time that they affiliated with Change to Win. Problem here is that the affiliation with CTW did not occur until June 2005 and CTW itself was not formally organized until October 2005. Pretty slim temporal pickings, i.e., 14-18 months.
4. Taft-Hartley pension plans are joint union-management funds. They are not controlled by the union. Employers get an equal vote. Why would employers support a union-initiated shift in proxy voting? In fact they did not and do not.
5. Much of the data in the study did not come from the unions but from an asset manager called Marco Associates. Marco votes its clients’ proxies. It provides justification for each of its votes. It is impartial. See this page if you don’t understand this part of the pension industry:
http://www.marcoconsulting.com/2.3.html
Bottom line: The study compares apples and oranges; does not have a sampling framework that is defensible; and fails to recognize that the pension plans it considers to do not vote their own proxies.
Why does the study obtain findings that seem to support the thesis that union proxies support non-fiduciary objectives? In my opinion, one possibility is that the results are a fluke. The other possibility is that there are econometric flaws in the study that were overlooked in the effort to generate an interesting result. I leave it to the econometricians here to look CAREFULLY at the paper and ask themselves if there are flaws. I won’t go into that here. But I contend that, in addition to methodological problems, the econometric framework is problematic.