“Through failure to contemplate these Infinites, men have rashly rushed into the examination of nature, as though they bore some proportion to her. It is strange that they have wished to understand the beginnings of things, and thence to arrive at the knowledge of the whole, with a presumption as infinite as their object. For surely this design cannot be formed without presumption or without a capacity infinite like nature.”
“If Tycho had had instruments 10 times as precise, we would never have had a Kepler, or a Newton, or astronomy.”
The fact that we had entered an era of epistemological crisis became fully evident during the COVID pandemic even if its roots lie earlier. In an epistemological crisis all the old traditions and institutions that normally constitute a shared social reality begin to fray and unravel. In these crises once widely shared distinctions between truth and falsehood, fact and opinion, become politicized and subject to bitter disputes between partisans. Elites lose authority not just over political institutions but over understanding itself. Unfortunately, this crisis that began before the pandemic did not end with it, and now, atop has been thrown a revolutionary change in the way knowledge itself is produced in the form of mechanized intelligence.
All of us have an existential interest in how this crisis eventually gets resolved, for without its resolution it’s impossible to see how we can live in a technological society at all. But perhaps this assumes too much of a shared perspective from the outset. In an epistemological crisis even the fact that one is in one is disputed. It’s probably best then to start off with laying out why such a crisis is indeed the thing we are suffering through.
Take the case of what is thought to be the pinnacle of our knowledge- science: On the surface at least, science has never been better. There are far more scientists now than there have ever been in human history, and a larger number of scientific publications.
All scientific fields have benefited from the unfolding of Moore’s Law. The ability to gather and process data has increased to a degree almost unimaginable even a few decades ago. The internet now facilitates a degree of scientific communication and collaboration that would make the journal publishers and letter writers at the birth of science blush.
Many of these new capacities were on full display with the development of novel vaccines during the Covid-19 pandemic- the first truly global crisis of the internet age. In addition to the creation- almost overnight- and deployment of a novel form of mRNA vaccination for a never-before-seen pathogen, scientists were able to quickly and globally share research in real time using preprint servers and engage in debates over merits using social media. Yet as the public and institutional response to that crisis made painfully clear, there are a myriad of problems afoot.
As in the early modern period, there has been a notable decline in all forms of authority, including scientific. Vaccines may have been created and deployed in record time for a novel virus, yet a significant portion of the population didn’t trust the medical establishment enough to take them. To a degree probably unprecedented, science has become politicized with positions on one side of an issue or another driven less by evidence than by political allegiance and ideology, as seen in the “March for Science” protests during Trump’s first stint in office and the administration’s assault on fundamental science upon his return.
The same communication capacities granted scientists by the internet enabled a whole host of pseudo-scientific personalities to emerge as guides for those of the public deeply suspicious, and not without reason, of both the government and pharmaceutical companies. The attitude of public authorities in such a charged atmosphere was often not to communicate rational decision making under what are necessarily conditions of uncertainty, but to police dissent.
Perhaps nowhere was this clearer than in the question of the origin of the pandemic itself. Rather than admit that the question demanded multiple epistemic and especially forensic models of evidence, the community of virology researchers doubled down on their insistence on the zoonotic origin of the disease and in their defense of clearly dangerous (whatever the ultimate origin of COVID) gain-of-function research that was largely in their own institutional interest.
Their defense of this position allowed the public for the first time to really see how scientific facts were made, a process that included collusion with leading journals and deliberate efforts to discredit reputable researchers- most notably Jesse Bloom- who dared to question the leading orthodoxy.
While debates in the West tended to focus on issues such as the actual virulence of the disease and its origin, the safety of what was at least seen by the public of a novel form of vaccination, the efficacy of lockdowns and public safety measures such as masks, and especially the costs vs benefits of moving a whole generation of school children online to be educated solely via Zoom, by far the largest moral failure lie in an area the Western press barely attended to.
The failure to give developing countries access to Western mRNA technology largely out of proprietary concerns resulted in untold numbers of unnecessary deaths. That this happened should not come as a great surprise given that societies outside the sphere of the major scientific powers continue to be exploitatively mined for knowledge and labor. This exploitation ranges from bioprospecting, to red markets, to training machine learning systems and providing the raw resources necessary for large scale giga-scale computation.
Still, while the problems of contemporary science can most clearly be seen in life sciences and were especially on display during the extreme societal pressures experienced during the height of the pandemic, they do not end there.
For quite some time now the so-called “soft sciences” have found themselves in a replication crisis. Findings once thought gospel have proven to an unnerving degree to not be replicable upon concerted attempts to do so. The replication crisis is but one aspect of a broader problem with the practice of scientific publication itself. The internet age has seen the rise of predatory journals ravenous for studies regardless of the quality of their findings or the qualifications of their authors. “Publish or perish!” has become an all-too-common mantra in academia, where researchers are also burdened under the weight of seemingly ever-increasing administrative tasks.
Given the sheer volume of information and the need for a laser focused specialization bordering on myopia, peer review faces the danger of being far too superficial. Nor do enough researchers make thorough review by outside experts easy by, for example, including a detailed methods section that includes access to the code used by those who performed the study- an omission perhaps largely driven by the pressures to frequently publish or by proprietary concerns.
Funding, or more importantly the lack of it, decides which of sciences’ many interpretative paths are explored and which are relegated to the status of permanent terra incognita. Theories outside of the established orthodoxy are often not investigated to the degree they could be because funding and education drives forward in only one direction. This lack of sufficient funding is partially driven by the increasing cost of science itself.
As we peer deeper and deeper under the veil of Nature’s secrets we need ever more powerful instruments at ever greater costs. Thankfully, fundamental questions are still deemed important enough for society to willingly bear these costs, but one wonders how sustainable this will be if science continues to be politicized, and especially if solutions to major puzzles continue to elude us even if the politics normalizes and the anti-elite demagogues exit the scene.
Nowhere is this clearer perhaps than in physics where existing theories are so successful they seem irreplaceable and yet leave us with massive unanswered questions such as the nature of Dark Matter and Dark energy-phenomena that just so happen to comprise the bulk of the universe. Some have gone so far as to blame this impasse on dogmatism among physicists, a kind of echo of the caricature of medieval scholastics with mathematics playing the role of God.
Even in an era where there are more scientists than ever and with an enormous number of resources directed towards science, the rate at which paradigm shifts occur may be be slowing down rather than speeding up.
The same tendencies that have helped give rise to a growing public distrust of science- namely the enormous influence that corporate and state interests have over what passes for scientific truth- have eroded the status of pure science itself.
Many of these problems might be traced back to the surrender of nearly every element of contemporary life to market forces- neoliberalism- towards the end of the 20th century. And such problems are not merely a matter of the distortion of research, the perverse incentives of scientific publications, or the corruption of academia by the drive for profits, but have affected the ways in which science communicates its findings to the public, that is, science journalism itself. Much of this journalism appears to be in the service of advertisers, rather than public education, its headlines are driven by the need for “click-bait”, ripe with hyperbole and prone to whatever reflects and feeds the current hype-cycle.
The dream of many of the figures who birthed modern science at the dawn of the modern age was that they were on the verge of creating a natural philosophy that not only provided a complete and whole picture of Nature, but that this picture would be comprehensible to the individual mind. Nothing could be further from the truth today. Specialization both between and even within fields has given us a fractured, kaleidoscopic image of nature. Knowledge is so prolific it has become the enemy of understanding, connections and interrelationships are missed or ignored not just between widely different fields, but within the very same subject. Yet this cannot be how the world really is. Our systems of knowledge are tools and abstractions, indeed fictions, of our own creation with clear lines of division, but the world itself really is one world.
The War between the Guilds and the Clankers
One should perhaps place most of the blame for the fracturing of knowledge where it properly lies, that is, not with us, but with Nature. With the exception of Pascal, none of science’s early pioneers seemed to have hit upon the idea that Nature would not prove ultimately simple, but inexhaustibly complex, that even our invented abstractions had no natural limit and leaned out towards infinity.
It was inevitable, therefore, that our systems of knowledge would ultimately end up looking something like the guilds of the Middle Ages. Complex systems of knowledge would require generation -spanning communities of practice, with a socially mediated understanding for what constituted excellence- or in the case of knowledge guilds standards of evidence and pedagogy. The Guilds, like other institutions, inevitably inherit human flaws due to the fact that they are composed of human beings, but at the same time they are indispensable, being the social means by which limited creatures construct and pass on knowledge with the depth and complexity that matches the world we confront.
The danger in a world dependent on such Guilds is that their sustaining of human interest and knowledge, over generations, on whatever aspect of reality a Guild separates from the joints of nature and defines as its domain, gives way to group-think and the politics found everywhere amongst courtiers. Guilds in such circumstances inevitably confuse their own interests with the common good and lose sight of their social role of discovering what for the historical moment can be considered the best of human understanding on some subject of inquiry. This danger was all too apparent over the course of the COVID pandemic. And now the major model builders- those who own and run the Clankers- can be said to be enacting a coup against the Guilds during a period of weakness brought about by the latter’s systematic failures.
Yet whatever our frustration with the Guilds this should not lead us to throw our lot in with the Clankers. For if the latter were a force for public good they would have focused their efforts on providing solutions for the obvious problems of science as practiced by the Guilds during a period of stress. Instead, everything they have done so far suggests that mechanized intelligence will be deliberately used to exacerbate those problems to the breaking point so that the Guilds can be absorbed. This is clear from the way the model builders have set their sights upon their first target- the Guild of mathematics.
Pure mathematics possesses several characteristics that made it a prime target for the model builders. It is a high prestige field with a vast open-source literature, which in part reflects the fact that its explorations almost never have the kind of economic value that would encourage hoarding and concealment. Above all, it is deductively verifiable, and mechanistically so using proof assistants such as Lean.
It seems to be the belief among the builders that not only is mathematics in the position where Clankers are most likely to see rapid progress, but that because mathematics serves as the foundation for all of science, surpassing the Guilds in pure math will somehow make their conquest of other fields, the lucrative ones they are truly after, where verification is difficult to mechanize, easier.
There is no need to go into the narrative surrounding the solution to the Navier-Stokes problem by an internal model of OpenAI, which has been written about ad nauseum, but I do want to lay out what I believe have been the most thought provoking responses to it, all written in anticipation of the Mathpocalypse of 10/6. People outside of the Guild of mathematics might easily have been surprised at many of the world’s most renowned mathematicians’ negative reaction to the solution of Navier-Stokes. Their dismay, however, was not as some had claimed out of fear that mechanized intelligence was coming for their jobs- Terence Tao has no worries about his employment prospects- whether mathematics is “solved” or not. Rather, the way in which it was solved poses grave dangers to the survival of mathematics as a human practiced and understandable endeavor itself and serves as a prelude to the future of every science dependent upon it- meaning all of them.
Iconoclasts had long dreamed of a revolution that would overthrow the power of the Guilds and at last give room for “heroic” science. I should be clear that this is not what happened or what is bound to happen in any coming age of science done by machines. What individual models do is allow any interested individual to conceal their pet theory, scoffed at by experts, in a scientific looking wrapper that gives them a chance of the limelight of publication. In other words, it increases exponentially the volume of slop, which was a problem already before LLMs and has gotten so bad now that the arxiv has needed to impose submission limits.
This is a problem that will only grow as open models become more effective and as paid ones are applied to the task by lone cranks, but the pseudo-science and slop it threatens is, for the moment, the weaker half of the vice now squeezing the Guilds. The stronger half and more existential problem for the Guilds is the amassing of clones of undisclosed prototype models, something only possible for a limited number of players with access to enormous amounts of computation and training data- precisely what was seen with the solution of Navier-Stokes. In that case a breakthrough was brought by the industrial scale application of Clankers to the problem- a swarm of 10,000 of them in all. What this means for the future is that frontier knowledge, in fields that once required very little capital, is now the sole prerogative of who owns or can rent the greatest or most advanced amount of compute. It is open science that is especially vulnerable here given the easy access to training data for the labs, and the consequence of their setting loose the Clankers on some open domain, or some research problem within it, is to effectively nuke it for academic researchers.
One might resign oneself to this were it the case that the solutions themselves were human readable and not just certifiably true, but alas, the solutions can be too alien for even the collective work of the Guilds to actually parse. There is indeed an argument to be made that mathematicians should simply embrace this new state of affairs. Perhaps we’ve reached the stage where not just Nature but our abstractions themselves have given rise to questions that require industrial scale efforts to resolve. The Riemann Hypothesis, or questions of even greater complexity, might demand LHC style efforts of “Big Science”. The physicist Martin Bauer has made just such a point, but the obvious question then is who exactly will pay for it given that abstract mathematics, despite the common refrain, rarely has any material payoffs. And whoever does pay for it- those willing to devote the scale of compute necessary to breakthrough these problems- will for that reason also own it.
Other questions that arise once mathematics, or most of open science for that matter, is replaced by solution mining at industrial scale by whoever possesses the capital to set the Clankers loose upon them is where exactly this process actually stops, whether any hard stopping point leaves us at a place which human minds can after long effort assimilate, and what will be left of the Guilds after this process is complete?
The fact that our understanding of Nature is “island like” with every discovery only expanding the horizon of new questions gives us good evidence that the setting loose of the Clankers, should, exhaust itself at some point, even ignoring their growing energy and material costs, and there should be no question at all in the case of mathematics which is known to be infinite in its depth. This is the “optimistic” case made by Kevin Buzzard– that the Clankers will run out of juice at a level we will eventually be able to understand:
“I thus believe that in the future we will reach a new “natural boundary” in mathematics, beyond (and perhaps way beyond) where we are now, but where machines are going to get stuck and where it is not viable to expend any more resources to make the next big leap. I am aware that I might be wrong. I am an optimist, so I am expecting future machines to be awesome; but I simply cannot see how they can get to infinity with finite resources, so they must stop somewhere. I believe that the optimal thing to do (at least from my personal perspective) is to let the machines loose, see what happens, and then begin the journey to where they have stopped. Things are currently moving fast. They cannot move fast forever. But if we get on board now then they will take us to extraordinary new places. And after we have arrived, the new adventure will begin.”
The arrival of mechanized intelligence would be akin here to discovering the archive of a lost civilization (ironically the way many of the founders of science thought of their project), an encounter with an alien civilization or the European conquest of the New World. In such a scenario there would still be a place for human mathematics. In the words of the anonymous mathematician who goes by the pseudonym doomslide:
“Human mathematics could exist in such a world as a perfectly healthy hermeneutic discourse on top of an empirical background consisting of a growing formal math library. The library is then to human mathematics roughly what experimental data is to physics.”
It’s an interesting perspective, but Buzzard especially uses the physical fact that there is no possibility for an infinite mind in a finite universe, to make an argument of faith- that the mechanization of intelligence will come to a stop at a point biological intelligence and its historical institutions can actually absorb and that this will happen before such institutions are completely wrecked.
All evidence so far is that the Clankers are already acting as wreckers. This isn’t so much a matter of amateur barbarians armed with LRMs barging through the gates of academic mathematics and proving their long-standing conjectures, as it is of the corporations in charge of the most advanced models sweeping through mathematical problems as a PR stunt. But what if the problems themselves are a resource? If lessons learned in trying to solve them rather than their solutions is the source of conjectures’ value, then setting the Clankers loose upon them doesn’t add to mathematics but consumes it. Tao has compared what is happening to strip mining:
“…the indiscriminate automated strip-mining of open problems for solutions may destroy the ecosystem from which the next generation of mathematical techniques, problems, and practitioners would have developed, similarly to how using excavators to dig out treasures from an archeological site destroys the rich historical context and data that gives such treasures so much of their meaning and value.”
Because the way new problems are found is by searching the literature along with open communication between mathematicians and the way the closed models are trained is in part a matter of the questions asked of them, this industrial scale scooping up of problems threatens the foundation of open science itself.
Erik Hoel has analogized the condition scientists now find themselves in to the the Dark forest of Liu Cixin’s sequel to his bestseller The Three Body Problem, where galactic civilizations hide lest they be poached:
“And yes, sure, books will still be published, papers still written, new programs and applications launched. But the future of intellectual work is to be crouched behind a tree, frantically trying to complete your contribution as silently as possible. You’ll feel the pressure of being hunted, since all it takes is a prompt to produce an almost-as-good version of what you’re attempting to make great, and so to mine away the inspiration and insight and reward.”
The above-mentioned doomslide agrees with him:
“Mathematics with no public exchange of ideas is definitionally impossible. As such, under the postulate, the continued operation of AI companies is fundamentally incompatible with any objectively meritocratic conceptualization of academic mathematics.
The labs are therefore simultaneously competing for mathematical results, selling mathematical assistance, and controlling most of the information required to evaluate either. This adversarial situation, where information about proofs and methods by which proofs were obtained are withheld whether deliberately or out of neglect, is one the mathematical community has never had to address.”
By far the most erudite piece I have come across on these questions has been Alexander Gamburd in his essay The Siren Call of the Silicon Leviathan. There Gamburd makes the case that mathematics constitutes the beating heart of the Enlightenment itself, which asserts that the world is comprehendible to individual human reason. He writes:
“There were no mysteries in Euclid beyond the reach of human reason; that was the point the Enlightenment made of Euclid. Mathematics was, strictly speaking, not the Enlightenment’s proof of concept but its existence proof: it did not show that autonomous reason might work; it exhibited an instance. That is why one might say that the Enlightenment—in effect—bound itself to the mast of its firstborn, and why the ship of mathematics entering thoroughly turbulent waters threatens the Enlightenment project, potentially in its totality.”
The arrival of mechanical provers such as Lean, delivered a part of Hilbert’s dream- the mechanization of proof- what in the process it also engendered is the construction of proofs of such monstrous complexity that no human mind will ever understand them. What this means is that we can know for certain that something is true while having no idea as to why.
He concludes regarding the September solution of Navier-Stokes:
“Hobbes was not in love with a certificate; he was in love with having understood. Nothing that happened in September makes understanding unavailable. It makes it, for the first time, optional: a theorem can now be certified without anyone’s having understood it. The question before the mathematical community is whether it can frame a covenant under which what has become optional is still required—under which nothing is counted as a piece of mathematics until a human being has understood it, and can show the next person why.”
The difficulty I see with Gamburd’s view is that it’s not even clear that such human understanding has been available even in mathematics at least since the days of Poincaré. Indeed, given the inevitable splintering of the Guilds, really the only means we have at our disposal of recovering this past ideal in light of the flood of knowledge and the complexity of the world that confronts us today is to turn to the machines in the hopes of finding connections between and translations across what Quine called our “web of knowledge”.
We could use machine intelligence to uncover unknown connections across the web between disparate fields, locate and clear science from the accumulated detritus of the replication crisis, to distinguish the signal in a world ever more drowned in noise, and to realize something like the Enlightenment ideal of making the whole of human knowledge broadly understandable to the individual. But even if we did everything right here, the quest for total knowledge that has so animated us as a civilization may ultimately be vain and futile, as Pascal pointed out. And even well used tools may historically short circuit abstractions both more revealing of Nature and better fit for human understanding, as only a genius who turned his mind to the process of science, like Poincaré could intuit. The real problem is that the machines are not being used for ends which though they might not be final causes or without epistemic risks, are at least human and historical, and instead are being driven by their creators in ways that potentially return us to a darker age of knowledge (for however much we resent the Guilds our light depends upon them) where the dictates of Nature, without parsable reason, are handed down by oracles possessing tickets of certification from blasphemously presented or mistaken gods.
* Title from: Glanvill, Joseph. The Vanity of Dogmatizing, or, Confidence in Opinions Manifested in a Discourse of the Shortness and Uncertainty of Our Knowledge, and Its Causes: With Some Reflexions on Peripateticism, and an Apology for Philosophy. Printed by E.C. for Henry Eversden, 1661p. 229
