The Declassification Engine: What History Reveals About America's Top Secrets

by Matthew Connelly

A captivating study of US state secrecy that examines how officials use it to hoard power and prevent meaningful public oversight
The United States was founded on the promise of a transparent government, but time and again we have abandoned the ideals of our open republic. In recent history, we have permitted ourselves to engage in costly wars, opened ourselves to preventable attacks, and ceded unaccountable power to officials both elected and unelected. Secrecy may now be an integral policy to preserving the American way of life, but its true costs have gone unacknowledged for too long.
Using the latest techniques in data science, historian Matthew Connelly analyzes the millions of state documents both accessible to the public and still under review to unearth not only what the government does not want us to know, but what it says about the very authority we bequeath to our leaders. By culling this research and carefully studying a series of pivotal moments in recent history from Pearl Harbor to drone strikes, Connelly sheds light on the drivers of state secrecy—especially consolidating power or hiding incompetence—and how the classification of documents has become untenable.
What results is an astonishing study of power: of the greed that develops out of its possession, of the negligence that it protects, and of what we lose as citizens when it remains unchecked. A crucial examination of the self-defeating nature of secrecy and the dire state of our nation’s archives, The Declassification Engine is a powerful reminder of the importance of preserving the past so that we may secure our future.

Genres: History, Nonfiction, Politics, American History, Military History, Science, Technology

540 pages, Hardcover

First published February 13, 2023

Reviews

[J Earl · Rating 5 out of 5]

The Declassification Engine by Matthew Connelly is an engaging and surprisingly (for me, since it is about data) very interesting read. Both as a history and a statement about our current state of government, this book delivers.

I think what made this a particularly appealing book is the way data, and the analysis of data, is presented in both an understandable and relatable manner. From discussing the history surrounding various moments in the past to the author's own journey trying to get the project off the ground, it was easy to see why this is an important and valuable research project. Not to mention anyone who has ever had to deal with a bureaucracy that has multiple personalities will relate to his dealings with foundations and the government.

I read this while also reading a book about the history of the deep state and what became apparent is the various ways one defines the term. Here, when Connelly discusses the deep state, he is specifically talking about the use of state secrets to mask accountability, as well as the associated disorganization of any supposed guidelines, whether for classifying or declassifying. The other book is more about basic dishonesty and greed and how keeping things hidden helps that. There is overlap but the other book is using a broad and loose definition of deep state.

Even knowing beforehand that the United States used to be more transparent, I was surprised at some of the ways. That said, we have never been as transparent as I think we should be to minimize abuse of the powers we give our government. And since WWII we have become dysfunctional as people and organizations have used the guise of national security to hide far too much that the citizens should have access to.

Highly recommended for anyone who wants a glimpse at what and how things are classified, as well as steps that might help increase accountability, both in the moment and in the future through historical research.

Reviewed from a copy made available by the publisher via NetGalley.

[Mitch]

Interested in this, but the intro is so quintessentially liberal. He argues it's not that he wants to expose secrets, but to make the state more efficient and to protect the CIA 🤮🔫

[Jamele (BookswithJams) · Rating 4 out of 5]

This was an interesting book that got a little tech-y at times, but I listened to this via audio and it was easier that way for me. I was able to stay engaged given how much is about data but it is such a fascinating topic and it was very well done. I liked how it went through history and how we have gone from transparency to not so much anymore and the author gives nuggets throughout that are kind of shocking. Overall I am glad I read this one and do recommend it if you like these sort of topics.

Thank you to Pantheon for the finished copy to review.

[Ale Caceres · Rating 3 out of 5]

Entonces la culpa de todo la tiene el hombre hetero blanco, homofobico y racista, shoker.

[Shanereads · Rating 5 out of 5]

This was a wonderful non biased view on how America classifies documents as top secret, and the process to declassify top secret documents. This book was fascinating and a must read for American history buffs and nonfiction lovers!

This finished copy was provided by the publisher in exchange for a fair and honest review. Huge thanks to Vintage for my review copy!

[Dan Burt · Rating 5 out of 5]

4.7 stars.

[Kanako Okiron · Rating 4 out of 5]

Truthers and the “Adams” of the world won’t like this book—too hooky, too informative for the sake of being informative, too intense in a motion to sound convincing and anti-propagandist—and, need I say more, the book’s praise has been echoed by Jon Stewart. However, this book does exceptionally well at not being a near-miss. It is careful not to over-dramatise, acknowledges the contradictory senses of its sources, and neither praises nor criticises presidential administrations, or the work of the CIA (I kid you not).

[John · Rating 3 out of 5]

Historical snafus and research methods are intriguing but ironically feels like a data dump. Repetitive in part bc the book is organized thematically instead of chronologically. The main solution that we should just fund the National Archive more seems both unrealistic and insufficient.

[Dave Reads · Rating 5 out of 5]

Recent headlines about Presidents, former Presidents, and other officials who have kept government 'top secret' documents in their private homes make the book "The Declassification Engine: What History Reveals About America's Top Secrets" even more relevant today. Matthew Connelly, professor of international and global history, writes about the secrets the government keeps and the enormous amounts of money spent to store the documents. He also writes about the challenges of declassifying papers and the types of information hidden from the world.

We see a dramatic increase in government secrets after World War II. Before that, the government boasted a spirit of transparency. But during the war, it was more common to justify information as secret. Following the war, officials didn't want to give up that power.

Connelly shares information about a project devised to interpret declassified documents to determine what types of data are classified. They got lots of pushback from the government even for a project like this. He and his colleagues have learned that many of the 'top-secret documents are marked for legitimate and curious reasons, and some documents were classified only to cover up mistakes.
Among the most interesting takeaways from the book:

The government doesn't have a policy that specifies what information requires safekeeping. It can't estimate how many secrets are created each year, and even during the Obama presidency, during which he pledged increased transparency, the number of top secret documents increased dramatically. No president is immune. We know that the National Archives can't accept any more paper documents because it has no place to store them. It is estimated that the US spent $18.4 billion in 2017 trying to keep its secrets, double what was spent five years previously.

Some, including former President Trump, have argued that government secrets are the work of the 'deep state.' But some 1.3 million Americans now have top-secret security clearances. Trump complained about the deep state and its secrets but regularly tore up presidential papers and fired the record manager who tried to reassemble them.

There are so many top-secret documents that they can't all be processed and shared with the appropriate people. An example in the book is the Pearl Harbor bombings. Intercepted Japanese communications were kept so secret that even the President didn't have immediate access.

There have been other instances when information isn't processed, and in some cases, the US military was prepared to launch nuclear weapons based on false warnings.

Because the laws of secrecy are so vague, if you start scribbling plans for a nuclear weapon on a paper napkin and share it with anyone, you could be sent to prison for the rest of your life even if you and your friend were not aware that the drawings you created in your brain happened to be classified.

The book also contains many stories of conflict between military leaders and civilian elected officials over secret information. Before he was assassinated, President Kennedy said his advice to the next President would be to 'watch the generals.'

"The Declassification Engine: What History Reveals About America's Top Secrets" is a well-written book that will be enjoyed by anyone interested in history and the inner working of American politics and government.

[Greg Brown · Rating 2 out of 5]

Very smooth prose with plenty of colorful details, but the structural problems made this a frustrating read.

You’d think this would be a fairly straightforward account of how structured secrecy metastasized from the success of the Manhattan Project into the larger military-industrial complex during the Cold War, piggybacking on the accelerating expansion of the executive branch (and what political scientists would call “presidentialism”) during the 20th century.

But Connelly instead decides to just write about whatever seems interesting that was secret at the time, which leads to an easy-reading book that kinda spins its wheels. And while he’s pulling in this extra material, he also pulls his punches on what stopping the problem would entail, instead offering extra funding and “data science” to declassify more. It feels studiously inoffensive, which widens the appeal of the book but also makes it a door stop.

The other big problem is his angle in, which is the work they did with big batches of data that’s been (partially) declassified over the years. You can feel him stretching to keep it relevant to each chapter, and the applications he talks about range from useful (finding declassified versions of redacted documents elsewhere) to teaching a dog how to count (computers helpfully pointing out that nuclear weapons documents tend to be secret). And of course, the whole thing predates the recent LLM craze so it feels a bit quaint.

Really, a more straightforward book on the history of classification and the larger growth of the security state—thinking something like Garry Wills’ Bomb Power—would have been a much better approach here.

[Caden Thomas McQueen · Rating 1 out of 5]

A masterclass in the limited hangout, Matthew Connelly's "The Declassification Engine" leverages popular histories of U.S. subterfuge to make the argument that the ineffaceable pattern of duplicity that mars our national past is not evidence of anything except that bad, incompetent people who somehow came to work in the government designed bad, incompetent systems for managing information. In this entirely facile view, all that is needed to set the sorry state of affairs in order is to bring in some good, competent people (like himself) to design some brand new, good, competent systems for managing information. And then, thank God, American democracy and national security will be ensured for all time.

After 221 pages of attempting to parse out what kernels of value I could, I lost the will to carry on. Connelly and his team have undeniably made some valuable additions to the historical record in the course of completing this project - a 1954 memorandum written by Henry Cabot Lodge, Jr. recounting a rowdy White House dinner where Winston Churchill spoke extensively of U.S. foreknowledge of the Pearl Harbor attack is of particular note - but the book that came out of it, this New and Improved stenography of the American Empire, largely belongs in the bin.

[Daniel · Rating 5 out of 5]

There are too many secrets
1. We need to learn from history. Less and less is revealed after longer and longer
2. Each agency keeping their own secrets will cause intelligence failure like 911
3. Secrecy is used to cover up misdeeds and mistakes
4. No one is interested in declassifying information
5. Too many secrets are now generated so it will take centuries to go thru them.
6. AI can help but need to be checked for its accuracy

[Jordan · Rating 4 out of 5]

An interesting history book, that digs deep into America's secrets, what the government hides and the history around it all.

I'm not the biggest history buff, that aside, I found this book interesting. I'd caution going into this if you have strong political stances one way or another. To really enjoy this and get the most out of it, put those to the side and be educated.

Thank you Penguin Random House and Vintage Books for a copy of this book!

[Kathryn · Rating 4 out of 5]

I can neither confirm nor deny my level of enjoyment of this book. There's a lot to process here, as this book simultaneously fascinated and stressed me.

As someone who worked for the National Archives at a presidential library, I really appreciated how much attention the author paid to archivists and the important (and overwhelming) work they do.

[Ben · Rating 4 out of 5]

[Review Redacted]

review redacted

3.8/5

[Dennis]

Just started this book. Heard interviews with author and decided to give it a try.

[Faith Hurst-Bilinski · Rating 4 out of 5]

The Declassification Engine was surprisingly engrossing for a book about data. No matter how much we think we know about the dark parts of our government, there is always more there. There are a lot of sayings about history and the present-history repeats itself or those who don’t know their history are doomed to repeat it. Everything here is not only a history of the loss of transperancy in our government, but a grim look at the present and the future. But, we do need to learn from our history. I’d recommend this book to anyone who is interested in how the goverment takes care of its secrets, history, or data (there are some of us out here) I’d recommend a slow paced read of this one.

[Graham · Rating 3 out of 5]

A lot of potential that really bogs down in the middle section in favor of rehashing a litany of foreign policy grievances from the middle few decades of the twentieth century. Not that they're wrong, per se, but whole chapters go by without making clear the connection between poor decisions and overclassification.

The book picks up again towards the end, though, when Connelly asks important questions about the need to keep history and the dire situation facing the National Archives when it comes to declassification and document preservation. This is a serious problem and as he make clear, all too few people care about improving it.

Read in conversation with Wellerstein's Restricted Data for a better view of the origins of our current classification policy. But also keep in mind just how relatively little from the 1980s onwards has been declassified versus the massive bulk of materials still waiting, out there, somewhere, for some enterprising historian to make sense of them.

[Clay Wescott · Rating 4 out of 5]

The US government has a long history of secrecy, mainly dating back to the Manhattan Project. Before that, transparency was the norm, and the government only maintained secrecy during wartime. The Cold War exacerbated secrecy, as the government sought to protect its intelligence gathering and military capabilities. This secrecy has had several negative consequences, including:

o It has created a culture of impunity within the government, where officials are less likely to be held accountable for their mistakes.
o Documents and emails of a classified nature were discovered in the residences of Biden, Clinton, Pence, and Trump, breaking the usage rules intentionally or by mistake.
o It costs over $16 billion a year to maintain the system, including the security checks for hundreds of thousands of government staff and contractors.
o It has led to the suppression of crucial historical information. How have past presidents made alliances with tyrants? How did they decide to assassinate America’s enemies as a matter of foreign policy? Who has the government assassinated, and how many civilians died as collateral damage? Why did the Pentagon lose its 20-year war in Afghanistan? What was behind Trump’s bromance with Vladimir Putin? What did they say to each other behind closed doors? Maybe our grandkids will find out. Perhaps not: It’s classified.

The government’s secrecy system is vast and complex. It comprises a variety of laws, regulations, and policies. The system safeguards national security data, yet it has loopholes to keep information secret for other motives. The government’s secrecy system is often abused. Officials often use secrecy to hide their mistakes, protect their reputations, or cover up wrongdoing. It covers conflicts of interest, self-serving deals, bribery, and other forms of corruption. The government’s secrecy system is a threat to democracy. It makes it difficult for the public to hold the government accountable for its actions. It also undermines public trust in government.

The only redress against secrecy is reporters and activists that get secret information and share it with the public, for example:
• the Pentagon Papers (lies and corruption behind Vietnam War);
• Files released by WikiLeaks (torture of prisoners; lies and corruption behind Iraq and Afghanistan wars);
• Edward Snowden (National Security Agency’s domestic spying), and
• Investigative reporting revealed illegal surveillance and weapon sales.

These channels are problematic for many reasons. Investigators may reveal information marked “HUMINT,” which means it has been gathered by sources that could be in grave peril if revealed. Journalists and activists are at risk of the same severe criminal sanctions facing former President Trump today if they refuse to return classified information or obstruct justice; and when they leave it vulnerable or share it with others.

Perhaps the book’s most surprising conclusion is that most of the closest held secrets aren’t secret at all. Connelly described a C.I.A. project that tasked Yale historians to locate and assess critical weapons systems for defending the country. If any information should be secret, it would be that. Yet the scholars compiled a report of great accuracy; the principal source was public relations materials distributed by the military! The C.I.A. quickly confiscated all notes and reports produced by the project as a threat to national security.

There has been a growing movement to reform the government’s secrecy system in recent years. This movement has successfully declassified some critical documents, but much work still needs to be done—every president since Truman has worsened the classification system, despite promises to reform it. Connelly proposes several reforms to the current system of government secrecy. First, he argues that the government should be required to declassify all information over 25 years old unless there is a compelling reason to keep it secret. Second, he argues that the government should be more transparent about its classification decisions. Third, he argues that the government should create a new agency to oversee the declassification process. Even if the government adopted all these reforms, the challenges would remain daunting. The US government has a backlog of over 600 million classified documents that need to be reviewed before release. It would take the National Archives 250 years to process all Freedom of Information Act requests at the George W. Bush Presidential Library.

To address this problem of scale, Connelly is building an archive of declassified government records and creating a search engine using machine-learning algorithms to predict the contents of redacted text and examine documents’ metadata for patterns. The goal is to reveal what the government doesn’t want us to know and why. So far, the system predicts topics such as the Middle East are more likely to be reported secretly than others such as Latin America. A good start, but there’s a long way to go.

Future research on the topic should include a more international comparative perspective. OECD countries have implemented initiatives to promote freedom of information, fiscal transparency, and other measures to support democracy without sacrificing security. It would be helpful to understand better where the US government stands regarding its peers in this area. Also, there may be lessons in other countries’ reform efforts that could inform reforms in the US and vice versa.

In summary, the book provides a comprehensive history of the government’s secrecy system and makes a strong case for reform. Connelly’s book is a must-read for anyone who cares about democracy, transparency, and accountability in the USA.

[Ben Rothke · Rating 5 out of 5]

While data classification is an information security concept, it's an intuitive idea to everyone. Certain information is sensitive, and access must be limited. That relates to another intuitive concept, the need to know. Unless there's a compelling reason for a person to know about something, access to that information should be restricted.

The US government and military have long considered data classification as part of their core security programs. They have formalized the program and embedded it into all aspects of government and military life. But over the last half-century, that best practice of data classification has become a monstrosity.

In The Declassification Engine: What History Reveals About America's Top Secrets (Pantheon Books), author Matthew Connelly, professor of international and global history at Columbia University, has written a fascinating account of the topic. He details how data classification has betrayed its intent, and can in fact, be a danger to democracy.

Connelly makes a compelling argument that the US government's current policies and practices regarding data classification are outdated and ineffective. China, for example, has exfiltrated tens of millions of records revealing personal information about US government employees, and Russia has infiltrated hundreds of government and corporate networks. And all of that data was classified.

For decades, government committees and commissions have identified the same fundamental problem: officials have not clearly and consistently specified what information actually requires safekeeping, making it impossible to prioritize what is truly sensitive. And for decades, there has been nothing significant done to fix that, as the book states.

He writes that those in the military will painstakingly study forty-year-old military records page by page because of the infinitesimal risk that a nuclear bomb design might slip by. Meanwhile, sniper manuals and recipe books for high explosive documents that could easily kill people are accidentally released and left open on shelves of the National Archives.

Unless someone can systematically analyze what kinds of information are classified and why, we can't begin to develop practical techniques for a more rational, risk-management approach to releasing non-sensitive records.

Connelly and his team at History Lab developed a sophisticated software tool to do that systematic analysis. After presenting their findings and approach to government and military officials, Connelly was surprised to learn that they were not interested in moving forward..

They developed this declassification engine, a platform that combined big data, high-performance computing, and sophisticated algorithms to reveal what the government did not want us to know and why they did not want us to know it.

A large part of the government and military's reticence to change a clearly broken system is that data is a valuable and powerful asset for them. Reclassifying data, even if it is in the best interests of everyone else, would diminish their power.

Connelly does a fantastic job of detailing the history of data classification over the last 100 years. He writes that as far back as 1940, Franklin Delano Roosevelt became the first president to issue an executive order that began to define a hierarchy of classified information for the federal government.

While the idea of creating a system of data classification had legitimate merits, it soon collapsed under its own weight. The problem was so bad that a 1956 Defense Department study found that data overclassification had reached serious proportions.

Numerous reviews about data classification all came to the same conclusion: that all of the data classification systems made it harder to prioritize the protection of truly sensitive information.

In addition, a bigger problem with data overclassification is that it prevents Americans from knowing what their government and military are doing. Connelly writes that they have a legitimate right to know this information, but it's often unlawfully withheld from them.

The culture of secrecy within the military and government that data classification endears not only makes it harder for people to access information they have a legitimate right to, but it also weakens democracy as a whole.

A key point from the book is that the situation is only getting worse as the digital footprint of US top-secret data grows at a massive scale—so much so that the National Archives is clueless about its actual size. The government as a whole lacks the capabilities to track all of the classified data.

At a little over 400 pages, this is an extremely dense and broad book, and an important information security read. Connelly takes the reader through the history of classification, various wars, surveillance and espionage, military conflicts, and more. And shows how data classification affected these activities and history.

[Dan · Rating 4 out of 5]

My thanks to both NetGalley and the publisher Knopf, Pantheon, Vintage and Anchor for and advanced copy of this new history book that looks at the untold history of the United States as shown through declassified records and documents.

Secrets are both a big business and a sign of power. Governments spend huge amounts of money trying to get secrets, hide their own secrets, and even more destroying the secrets they are supposed to share. People in power use their levels of clearance like keys to the executive washroom, lording it over fellow workers, and underlings, I would tell you but you aren't cleared for this. This access does them well in life, allowing people to enter the world of boardrooms and consulting with almost a preset salary. Governments keep secrets for many reasons, to protect the nation, would be cited a lot, but to protect it from what is the real question. Matthew Connelly, professor of international and global history writes about the business to classification, and even more about difficulty in declassifying in the book The Declassification Engine: What History Reveals About America's Top Secrets a reveling look at America its secrets, the whys and what are really being hidden.

The book begins with a project that seems to have a lot of interest, using data science to interpret declassified documents, and using those documents to get a better grasp of what is being classified, and why. There is much push back, threats, NDA's until finally funding is secured for History Lab project, which is better explained by the book than myself. Readers are then introduced to an America that before World War II prided itself on transparency. Yes during wartime, secrecy would be instilled, sure even the Constitution might be bent a tad, but at the end, things, and America would go back to normal. Except after World War II. Was it the situation, the global powers being up for grab, the money that was being tossed around, the fact that people in power liked controlling the power. Or was it the hubris that generals and governments liked more to be feared, than to be known as making mistakes. Even worse putting American lives in danger for what seems like now totally stupid ideas.

The book is extremely well written and sourced, with information on every page that makes the reader go what the heck. Stories of Churchill telling a dinner party about what he saw the Americans doing when Pearl Harbor was announced. The numerous trials of the effects of radiation on American citizens without their consent. My favorite was surgically altering a cat to be a transmitter to hear conversations in a park, but the cat didn't have any interest in going where it was told, and eventually was run over by a cab. Stories of generals going from the Pentagon to the boardroom, the Joint Chiefs not saving any of their notes for the national archive. The numerous atomic incidents and accidents that could have caused a world war. This stories are found in declassified documents, which also points out the problems and th industry of keeping secrets, different departments decided what should be classified, and what shouldn't be.

An extremely interesting book, especially for lovers of history and people who like to know what is really going on. The problem with something like this is that it almost makes conspiracy theory people seem right. However it seems the biggest secret in government is how many mistakes they make, and would rather be thought of as doing bad things for America and competent rather than doing stupid things constantly without anyone calling them on it. A book that covers many subjects, and does a great job in keeping the reader interested, and informed.

[David Partikian · Rating 4 out of 5]

Columbia University history professor and data cruncher, Matthew Connelly, attempts to analyze the United States of America from the perspective of what we can infer about the USA from what the “dark state”—for lack of a better phrase—has kept hidden and for how long. According to Connelly, whose thesis is very persuasive, the US dark state (or deep state) and the necessity to keep secrets largely began the Japanese attack on Pearl Harbor. Connelly synopsizes the diplomacy leading up to the attack, basically comparing it to Iraq attacking Kuwait after the US Secretary of State voiced no objections. In other words, Roosevelt was creating a condition which cornered Japan and forced an attack; deny a country fuel imports and that country is apt to act aggressively. Call it a Golden Rule of the modern world. Although Connelly doesn’t uncover anything earth shattering, the primary axiom he stresses concerning Pearl Harbor and secret information is as relevant as ever: ‘Secrets are often kept to hide incompetence, and this short-sighted mistake has been repeated so many times it should no longer seem original. But after more than eighty years, it can still take people by surprise.” (pg. 55).


Connelly then addresses major eras and presidential administrations as well as the over-deployment of designating documents “Top Secret” (among other classifications). Although cursory to any expert on these eras, the chapters on the Manhattan Project, the leaking of secrets to the Russians after the two atomic bombs were deployed in Japan, and the Cold War arms race are worthy of review. That first axiom that Connelly uncovers about incompetence keeps rearing its head, as do others about overclassification and compartmentalization, which both lead to further incompetence. For anyone but an outright expert in the field, these chapters lead to gems of arcane knowledge buried in the past, e.g. that General MacArthur was guilty of trying to ransom Philippine heads of state and their families after the Japanese invaded. Or that, when he wanted to deploy nuclear weapons in North Korea although he had no idea of what the then current nuclear capabilities of the USA were, due to compartmentalization of secret documents; he was not in the loop.


The book gets mired down in later chapters on the details of security clearances and the—at this point—rather well-known fact that most classified documents are rather anodyne. The USA is virtually buried in so much classified material that it would take a high-powered computer an eternity to declassify it all with any precision and uniformity. Thus, secrets remain buried, again, largely due to incompetence and structural issues. The closing scene from the original Raiders of the Lost Ark is correctly used as a metaphor for how the Deep State has morphed into an out-of-control archive, or rather a document mortuary where retrieval is nearly impossible, creating even more secrecy: Secrecy cloaked under mounds of paper, or now, electronic secrecy.


The Declassification Engine is an engrossing read that let’s us think about the deep state, but without the complication of current politics, though Hillary Clinton’s emails are used as an example. The chapters are somewhat uneven, some dealing with historical examples and others dealing with social science experiments on unclassified and still classified documents. Nevertheless, the book is extremely revealing, which is a strong endorsement for a book on secrecy.

[Richard Thompson · Rating 4 out of 5]

This is a good update to Daniel Patrick Moynihan's 1998 book, "Secrecy." Sadly, the problem of excessive government secrecy combined with poor security for protecting secrets that matter has only gotten worse in the last 25 years. The volume of classified materials continues to expand geometrically, and the budget and processes for handing the masses of secret data are unable to keep up. Freedom of Information Act requests are so backed up that they will take a hundred years or more to process. The spirit of FOIA has been entirely undermined. The same issues arise and continue to get worse in both Democrat and Republican administrations. The big change from Moynihan's time is the development of technologies of big data and AI, which allow data scientists to analyze and manage massive amounts of data. I was interested to learn about techniques of "distant reading" that give scholars a view into gigantic stores of documents that could not possibly be manually studied in a meaningful way even with techniques of statistical sampling.

Mr. Connelly proposes creation of a declassification engine - software designed to comb through older documents and determine by algorithm what should be declassified and made available to the public. He makes a strong case that this is more accurate than human curation and that the staggering amount of classified material makes any meaningful human curation of the declassification process impossible. I suppose that it is not surprising that government officials not only resisted his proposal, but even threatened him with prosecution because the nature of the engine would potentially give a view into materials that they would not want to declassify. The sad joke is that much of the information that they want to protect can already be found in public records or pulled out of redacted documents by statistical analysis. If scholars can think about doing such things, we can be sure that the Russians and Chinese, among others, are already using computers to comb through the data to uncover the secrets.

It's a sad mess. What we really need is a new much stricter standard for allowing information to be classified and a better more open judicial system for ruling on whether the standard is being properly applied. The risk is that a few things that might better be kept secret will get out, but most of those things will get out anyway. I believe that much greater openness will have an incalculably greater benefit in helping democracy to function better and in enabling scholars and both for profit and not for profit organizations to study our society, government and history in ways that are presently impeded by the culture of secrecy.

[Blair · Rating 3 out of 5]

“The Declassification Engine” is a book about Matthew Connelly’s attempts to sell in an AI tool for removing special classification of documents – mostly those from America’s government organisations relating to security (e.g., CIA, NSA) – so that the world can learn from the lessons of past decisions.

The book is largely a story about how there are growing trends to classify more and more documents within the various branches of the US Government and how it is taking much longer periods of time (i.e., an increasing number of decades) before such documents can be declassified.

Data and information collection is increasing exponentially in this digital age and the ability to manage this is decreasing, because declassification is not seen to be a priority and in response, budgets are not created to deal with it.

Within this overarching story, the author talks about his AI engine which can help declassify documents and why there is resistance within government to embrace these initiatives. Some of this comes down to coverups while others have to do with turf-wars where bureaucrats will not cede ground to elected officials.

I liked some of the findings of declassified information in the book. Some were a surprise to me including the story about why President Truman fired General Douglas MacArthur for insubordination. I had read more about the General from books like William Manchester's “American Caesar” and didn’t know that he tried to instigate a war with China. He both sabotaged an effect for the White House to make an armistice with China and ordered a gunboat into the Chinese territorial waters off the city of Shantou and sent in waves of jets to mock attack the junks that intercepted this gunboat. (Page 136.) No wonder Truman fired him.

There were also humorous anecdotes where it took until 2011 to release government documents that contained recipes for making invisible ink, long after the First World War where this was classified. At that point few secrets would not be done without digital technology. (Page 210.) Too funny!

From a negative perspective I felt the book meandered a lot and covered too much why the author’s technology was not adopted by various government bodies. Was this book just ‘Sour grapes”? Perhaps the author could have avoided this argument by discussing other attempts to declassify documents from the American government and/or other countries where declassification was more successful.

It was an okay read overall. I buy the author’s desire to declassify documents for the purpose of gaining a historical perspective and in not being used to cover up bad decisions. But I felt this was more of his story than an overall story about declassification.

[Book Club of One · Rating 4 out of 5]

Matthew Connelly's, professor and lead investigator for Columbia's History Lab, The Declassification Engine details and explains the work to create the largest database of declassified government documents and then the use of machine learning tools to analyze these documents to decipher what the government wants to keep us from learning and why. It is a troubling history with a clear need for rethinking our practices and demanding greater transparency from politicians in the digital age.

Across 10 chapters, Connelly explores different facets of the data, looking both historically and at specific issues. There are chapters focused on Pearl Harbor, the bomb, codes, the military industrial complex, surveillance and quasi-science. In each of these Connelly details important events, the timeline of information distribution and the almost continual legal re-interpretation or outright dismissal of laws intended to make information freely available. It is frequently noted that the more often secrecy is invoked, the more information is hidden. An example used several times in the book is that more information about UFOs has been declassified than details about nuclear bombs.

When America entered a war, traditional freedoms were restricted for the course of the war, Connely states this as the case, until World War II, when new technologies (the bomb) leading into the Cold War caused the US Government to continue a heightened sense of information security. To the point that the President wasn't always informed of the status of the nuclear arsenal. At least when those granted the permission to know the information did not leak it for their own purposes. That is one of the striking facts, that in many cases those reading classified information typically found details of the reports in the news media, or were the ones leaking it. One of Connelly's points is that too much is classified that does not need to be.

Most impactful to our future is chapter 10, Deleting the Archive. You can keep something secret if you destroy all evidence. As the National Archive is no longer accepting paper records, this means all materials they collect are now going to be digital with all the access issues and long term storage concerns. As in many of the less well-funded portions of the government, those managing the records have frequently and continue to need to do more with less.

A frightful look at the ways the dark state operates, but one that asks us to question some of the traditional narratives and emphasis that history is an additive process. The more information available, the more one is able to create a better, well rounded interpretation of the past.

I received a free digital version of this book via NetGalley thanks to the publisher.

[Chris Jaffe · Rating 4 out of 5]

This book uses a database and algorithms to try to determine what sorts of documents are declassified - and what sort of documents are not.

One of the main takeaways is how little "there" there is there. The intelligence community has a culture of secrecy and conformity, where your default response is to tie things down. Also, there's tons of compartmentalization so that people don't know how their intel fits into any larger picture - and it often doesn't fit into any larger picture; it's just random stuff. And that random stuff is often outdated. The book quotes one president (Truman, I think) who said that often CIA reports are just things said three days earlier in the New York Times. Now add to the explosion of data and info of modern times and the community can't even track what it has in its archives. Our national archives can't do it, either, as they're increasingly underfunded and understaffed as the nation's data grows exponentially.

Beyond that, info is often declassified as spin, in order to make the organization look better. There are ties to business, though the book says you really can't find too clear links to helping business, in part due to the agency's frequent dysfunction. The end of the book notes how often women and racial minorities who work there receieve hostile treatment, but rarely complain, as they try to get along with a job they see as important. Much secrecy is just to hide mistakes and embarassments, which has been the case since Pearl Harbor. One of the biggest things kept under wraps is codes and codebreaking, with invisible ink staying silent until it was outdated. Oh, and much archives are now electronic only, and if they're deleted, that's it - they're deleted.

It makes some good points, but Connelly does get carried away at times. When talking about how we don't even know what the records are, he ponders if it's the end of history. I mean, there's plenty of history that isn't dependent on government archives. Also, while his skepticism towards government secrecy is frequently warranted, it's a bit knee-jerk at times.

Decent book, as it upholds the idea that never ascribe to maleviolence what can be understood by stupidity.

[Bookoholiccafe · Rating 4 out of 5]

"The Declassification Engine" by Matthew Connelly is a compelling exploration of the use of state secrecy in the United States. Despite the country's founding principles of transparency and open government, secrecy has become a pervasive policy that often hinders meaningful public oversight. Connelly uses advanced data science techniques to analyze millions of state documents to reveal what the government wants to keep hidden and what it says about the power dynamics between leaders and citizens. Through careful study of pivotal moments in history, such as Pearl Harbor and drone strikes, Connelly exposes the drivers of state secrecy and its negative impacts on society. This book is a sobering reminder of the self-defeating nature of secrecy and the importance of preserving the past to secure a better future.

This book is strongly recommended for those interested in understanding the classification process, including ways to improve accountability through historical research in both the present and future. It provides valuable insights into the what and how of classified information, and offers potential solutions to enhance transparency and oversight.



«موتور حذف طبقه‌بندی» نوشته متیو کانلی یک کاوش متقاعدکننده در مورد استفاده از رازداری دولتی در ایالات متحده است. علیرغم اصول بنیادین کشور مبنی بر شفافیت و دولت باز، رازداری به یک سیاست فراگیر تبدیل شده است که اغلب مانع نظارت معنی دار عمومی می شود. کانلی از تکنیک‌های پیشرفته علم داده برای تجزیه و تحلیل میلیون‌ها سند دولتی استفاده می‌کند تا نشان دهد دولت می‌خواهد چه چیزی را پنهان نگه دارد و در مورد پویایی قدرت بین رهبران و شهروندان چه می‌گوید. کانلی از طریق مطالعه دقیق لحظات مهم تاریخ، مانند حملات پرل هاربر و هواپیماهای بدون سرنشین، عوامل مخفی کاری دولتی و تأثیرات منفی آن بر جامعه را افشا می کند. این کتاب یادآوری هشیارکننده از ماهیت خود باختن رازداری و اهمیت حفظ گذشته برای تضمین آینده ای بهتر است.

[Maria · Rating 3 out of 5]

Connelly introduces the reader to the complexity and scale of classified documents in American history since the start of WWII. He's worked and built computer software to help government officials declassify old documents so that they can be preserved and accessed by the public.

Why I started this book: Classified history and secrets? Sign me up.

Why I finished it: Infuriating. This book started with a sledge hammer and an agenda about Pearl Harbor. Since I just finished reading Our Man in Tokyo: An American Ambassador and the Countdown to Pearl Harbor I was more familiar than most about what Washington did and did not know about Japanese plans at the start of the war. And Connelly spent the first part of the chapter implying conspiracy theories... only to debunk them slightly later. I was wound up and off on the wrong foot. I felt like most of this book was trying to prove malicious intent on the part of the government, failing to hit that standard in most instances and then saying that incompetency and apathy were just as bad. A stronger arguement would be look at all that we are losing to imcompetency and apathy and these failings can mask the truly malicious actors. Is this an arguement of semantics or a way of framing the argument to more fully prove the point?

[Grandt White · Rating 4 out of 5]

I'm not sure this book will successfully bridge the gap between general lovers of non-fiction, and those like me, primarily interested in history books. And indeed, Connelly's ability to bridge gaps will be crucial in the status of this particular book. We all, no matter our political affiliations wonder about the classification/declassification of documents and how that's decided, and perhaps more important, the necessity of government secrets in national security. But we don't talk enough about it. I think Connelly understands all this very well, which could explain his several references to Hillary Clinton's infamous emails; he knows that, as cliche as this phrase has become, this should be a bipartisan issue. History is important and documentation is hardly a matter of opinion. What I mean by that is a piece of paper relaying a conversation is not a matter of debate-what to do about it should the conversation hold problematic implications is, but the documentation of the thing is merely an objective account. Connelly does a good job keeping his analysis fair and data-based but also not at all lacking in personality, in fact one could argue Connelly's tone is at times unserious. Regardless of this, he fully recognizes the importance of this issue. If all aisles of the political, umm, universe, I guess, of the US, heed Connelly's warnings then this book could propel itself to a legendary status. Connelly is trying desperately to save the past, and his attempts do not lack long-term relevance whatsoever.

[Chad Manske · Rating 4 out of 5]

Interesting and timely new book about how the classification of our country’s intelligence and secrets work, and the outcomes it’s produced. Dating back to the birth of our nation, secrets have been kept and hoarded by those in power such that their classification would only be available to those properly vetted and trained. And their declassification is mired in overly bureaucratic processes unnecessarily, and often to the detriment of those who rely need to know. Failures of such processes led to Pearl Harbor and 9/11, by which the intelligence existed to uncover the potential of these events, yet because of the system the information is collected and segmented, no one can put it altogether with their limited credentials. 9/11 as a case in point led to the creation of the Office of the Director of National Intelligence (ODNI) as a way to harness the disparate intelligence agencies (18 of them) in the US to help stitch the info of future catastrophes together before they happen. Author Michael Connelly and his team used algorithms to comb through voluminous and recently classified and redacted documents made publicly available to study code words and terms and determine their value as to whether they should have been classified or not—objectively subjective. Power, greed, and negligence are often drivers of this study of power, and the importance of a book like this helps us to see the blind spots in our ways to help preserve our very nation.