Science is, at its best, a self-correcting process – a continuous search for truth through rigorous evidence, transparent methods, and open debate. But what happens when mistakes are introduced, maintained, and defended? The story of the so-called “Frankenstein data set” in hurricane damage research, brought to light by Roger Pielke Jr., shows how flawed data practices can distort both public perception and policy.
The Frankenstein dataset, which resulted from undocumented changes to a rigorous, peer-reviewed dataset, highlights a crisis in scientific integrity. His story is not just a cautionary tale about bad data, but a case study in how the scientific process can fail when there is a lack of institutional accountability. Let's find out how this data set came about, why it's important, and what it reveals about the state of climate science.
The origins of the original data set
The original data set, developed over decades of research by Pielke and his colleagues, aimed to normalize hurricane damage by adjusting for inflation, population growth and other economic factors. This normalization process allowed researchers to compare historical hurricane damage individually and isolate trends in economic losses from changes in wealth or development.
The data set used in studies such as Weinkle et al. is documented in detail. (2018) and Pielke et al. (2008) served as a reliable tool for understanding the impacts of hurricanes. It was based on NOAA's “Best Track” data covering hurricanes in the U.S. and followed consistent methodologies.
However, the story took a dark turn when this data set fell into the hands of ICAT, an insurance company.
How the Frankenstein record was created
After Pielke et al. (2008), Pielke's team worked with ICAT to develop the ICAT Damage Estimator, an online tool for visualizing hurricane damage using the peer-reviewed dataset. Initially, the collaboration worked as intended: the tool improved access to high-quality research for industry stakeholders.
But in 2010 ICAT was taken over by another company and Pielke ended his involvement. In subsequent years, ICAT staff, who lacked disaster normalization expertise, made undocumented changes to the data set. These changes included replacing post-1980 entries with data from NOAA's Billion-Dollar Disasters (BDD) database, which used a completely different methodology.
Important changes
- Replacement with NOAA BDD data: ICAT replaced post-1980 entries with BDD data that included inland flood damage (from the National Flood Insurance Program, NFIP) and broader economic impacts such as natural resource losses and disaster relief disbursements. These additional factors increased damage estimates after 1980 and created an artificial upward trend.
- Additional events: ICAT17, the modified data set, resulted in 61 additional storm damage events, none of which were procured or documented. Most of these undocumented events occurred after 1980, further distorting the data set.
- Methodological discontinuity: NOAA's BDD methodology, adopted in 2016, was incompatible with the original data set. For example, there were no NFIP payouts before 1968, so comparisons between claims before and after 1968 are fundamentally flawed.
- Unattended changes: Beyond the replacement of BDD data, ICAT17 contained other undocumented changes to the original data set. These changes introduced upward biases even before the normalization adjustments were applied.
Steve McIntyre commented on Pielke Jr.'s post.
By the time ICAT published this Frankenstein data set online, it had already diverged so far from the original peer-reviewed data that it no longer bore any resemblance to a rigorous research product.
How the Frankenstein Dataset was misused
The ICAT17 data set, which was later expanded and renamed “XCAT/ICAT 23” in Willoughby et al. 2024” was adopted by researchers who believed it was a professionally maintained and credible resource. Above all:
- Grinsted et al. (2019) and Willoughby et al. (2024) used XCAT to claim an upward trend in normalized US hurricane damage and attributed this trend to climate change.
- These studies were published in prestigious journals such as PNAS and JAMC and subsequently cited in influential reports, including the IPCC's AR6.
However, Pielke's analysis shows that these trends disappear when the original dataset (Weinkle et al. 2018) is used instead of XCAT/ICAT23. In other words, the upward trends claimed in these studies are entirely a product of flawed data practices.
The implications for climate science and policy
The consequences of these errors are far-reaching:
- Distorted public perception: The flawed studies, amplified by major journals and the IPCC, reinforce the narrative that climate change is leading to increased hurricane damage. While this narrative makes political sense, it is not supported by NOAA's direct measurements, which do not show long-term trends in hurricanes making landfall in the U.S. or their intensity.
- Undermined scientific integrity: The willingness of peer-reviewed journals to publish studies based on undocumented, methodologically inconsistent data, AND REFUSION OF WITHDRAWAL IF SUCH DEFECTS ARE CLEARLY DISCOVEREDindicates a breakdown in the scientific process. This failure undermines public trust.
- Misguided political decisions: Policies based on flawed data risk diverting resources from effective disaster response strategies. By overstating the role of climate change in hurricane damage, these studies obscure the true causes of vulnerability, such as poor land use planning and inadequate building codes.
A call for course correction
As Pielke states: “Mistakes happen in science.” What matters is how the scientific community reacts when these errors are discovered. The Frankenstein records saga presents an opportunity for course correction:
- Journals such as PNAS and JAMC should retract the flawed studies to prevent further misuse of the ICAT17/XCAT datasets.
- The climate science community needs to adopt stricter standards for data transparency and provenance to avoid similar mistakes in the future.
- Policymakers should demand higher quality evidence before implementing costly climate policies based on unverified claims.
This case isn't just about bad data – it's about the integrity of the scientific process. If climate science is to be credible and fulfill its proclaimed role in informing policy, it must adhere to the highest standards of accuracy, transparency and accountability.
Final thoughts
The Frankenstein data set is a stark reminder of the dangers of uncritical acceptance in science. While the temptation is to fit data into a convenient narrative, real scientific progress requires resisting this impulse. As Pielke's criticism shows, science can only fulfill its promise as a self-correcting endeavor by confronting and correcting errors. This should be a wake-up call for climate science: integrity must come before ideology.
Sources:
Don’t Use the ICAT Hurricane Loss “Dataset”: An Opportunity to Course-Correct Climate Science
A Frankenstein dataset results from splicing together two time series found online
Below is an example for US hurricane damage 1900-2017
Data for 1980-2017 was replaced with a different time series in the green box
Upwards trend results (red —)Claim: Due to climate change! pic.twitter.com/lRxcTst0EO
— The Honest Broker (@RogerPielkeJr) December 21, 2024
The new hurricane damage time series trick
Step 1: create Frankenstein dataset w/ an increasing trend where there was not an increasing trend before
Step 2: Attribute the increasing trend to climate change
Step 3: Use Frankenstein dataset to impeach other research w/ no trend pic.twitter.com/mKMWFXvkj6
— The Honest Broker (@RogerPielkeJr) December 22, 2024
Like this:
Load…
Related
Explore more from Watts Up With That?
Subscribe to receive the latest posts by email.
Comments are closed.