The case for professional critics in science

In many areas of science, there is an increasingly urgent unmet need, a role that could be simultaneously fascinating, rewarding, and potential remunerative. It is a role that already exists in various forms, but which could be made into something much more potent, especially if forces converged to make it more prominent. I am talking, of course, about the professional science critic.

In the popular imagination, science operates something like a priesthood: scientists enter elite institutions as novices and emerge years later as full-fledged representatives of The Truth. Along the way, they are trained to be experts and professionals in their subject, to excel in action as well as in thought, and to sacrifice their worldly interests for the sake of their calling. The scholarly journals and peer review process are imagined to operate like an ecumenical council, guiding and filtering the thoughts of the broader community of the devout, and only allowing to pass what is deemed to be true and useful and good.

The reality is that science, at least as it is practiced today, is actually much more like the film or music industry. Although the analogy is far from perfect, individual scientists in academia bear a strong resemblance to film directors. Devoted to their vision, they marshal their forces, bringing together diverse elements into a controlled chaos, and produce scholarly outputs (i.e. academic papers) which bear their name, but aren’t singularly theirs. Like most directors, however, academics are beholden to the broader industry, and all but an elite few must perpetually seek out more funding, in many cases taking work that would not be their first choice in order to be able to fund a different project that is closer to their true artistic / scientific vision.

Extending the analogy, we might think of graduate students as a type of apprentice, starting out like production assistants, helping and learning where they can, and gradually climbing the ranks as they gain skills and experience. While contributing to collaborative work in many capacities along the way, many may harbour their own directorial ambitions, and eventually seek to start their own production companies / labs, from which they can produce work that is all the more their own. While many choose to specialize further into more technical roles (often outside academia), or may simply be unable to get their projects greenlit, those who are fortunate and determined enough to get hired in a tenure-track job are given the chance to make their vision a reality, while simultaneously contributing to the broader industry by bringing in funding, providing opportunities for others, and helping to teach the next generation.

As in the arts, the difficulty arises of how to make this whole enterprise financially viable. There are many academics who are essentially auteurs at heart, who want to create work that is personal and outside the norm, and the feedback they value the most will be that of their peers. But science is expensive, and even a modest undertaking may require resources beyond what can realistically be raised in advance of a speculative endeavour. In the film industry, studios serve this crucial organizing function, connecting production and creation to distribution and sales. Audiences will ultimately fund the ongoing production of work, but it is the studios that take the risks of betting on which projects they think will turn a profit.

In academia, the structure is different, although not entirely so. The funding for science largely comes from grant agencies rather than ticket sales, and funders such as the NSF naturally have their own tastes. However, they, too, recognize that the opinion of the audience is ultimately what matters, with the audience in this case being a heady mixture of the scientific community itself, politicians, the media, and the general public. Work that receives the most attention / citations is generally accepted as the most successful, though the rough cut demo tape that eventually inspires a thousand papers is also highly valued. As in the arts, proxies such as awards provide signals about what work is considered great, though everyone of course knows many underappreciated creators who are just waiting for their big break.

Scientific journals hold an ambiguous position in this ecosystem. For a long time they played the essential role of providing distribution. That is, before information could so easily be sent over the internet, the journals, through the process of editorial and peer review, chose what content to publish, printed out those papers in bound form, and then mailed them to libraries, where they were made available to other scientists. But journals have also traditionally been the most directly business-focused aspect of this ecosystem. Especially in recent years, as in film, there has been massive consolidation of the publishing industry, such that a small cabal of companies essentially control the entire space of professional academic journals, and earn record profits through their near oligopoly.

All of that has been destabilized, however, with the invention of the internet. Despite the common perception, peer review and professional publication has never served as any kind of guarantee of quality, though it does sometimes help to weed out low quality work. Rather, it is best thought of as a kind of preview screening, testing the waters for how the scholarly community will respond to an article, and providing feedback on which aspects the authors should tweak if they want to increase their impact. In the worst cases, peer review can act like a kind of layperson audience, pushing all products towards a generic pablum that satisfies a conventionally accepted formulaic structure, but doesn’t push any creative boundaries or provide any novel insight.

The truth is, however, that peer review is no longer strictly necessary for the ecosystem to operate, especially now that the cost of publishing and distribution has fallen virtually to zero. Just as nearly anyone can now shoot a film on their phone, edit it on a laptop, and share it with the world via YouTube, anyone can equally do their own research, write a paper, and share it with whoever is interested online. Many of these burgeoning creators might benefit greatly from stronger technical skills, greater familiarity with the history of the field, and better production design, but that is no reason that they should not make and share the work. Indeed, in certain communities — beginning with physics but increasingly expanding to other disciplines — this has become the norm. So-called pre-print servers, such as arXiv.org, allow anyone in the world to upload a manuscript and make it available to everyone, in perpetuity. This is truly a wondrous innovation, an underrated success story of the internet, for which its creator, Paul Ginsparg, rightly gets a great deal of credit.

Not surprisingly however, such pre-print servers do create a number of new challenges. First, as is the case with amateur film on YouTube, the amount of content being created vastly exceeds anyone’s time to consume it. There are far more papers uploaded to arXiv than anyone could possibly have time to read, even within narrowly scoped subfields, such as deep reinforcement learning. Second, because there is no barrier to entry, the quality of work is highly variable. Most of the important papers in a field like machine learning will be posted to arXiv, but so will a bunch of unfinished drafts that essentially contribute nothing of scholarly importance. Reputation is the name of the game in science, and doing bad work will very quickly earn you a bad reputation. But for someone starting out, or someone who doesn’t understand the industry, there may be little incentive to make sure work is fully polished before publishing it, or even to put in the work to understand what qualities might make a paper valuable to others.

As one would expect, various partial solutions to these problems have emerged organically. The opinions of high profile people continue to matter a lot, especially in the form of letters of recommendation. In addition, many scientists have become their own one-person marketing departments, promoting their work via press releases and social media. In some fields, various technological solutions are being tried, from specialized search engines, to recommendation systems, which try to help people find what they might want to read.

Potentially more important than any of these, however, is all of the writing about science that is done by scientists themselves. While blogs have seemingly declined in popularity, newsletters have started to fill the void, especially in fields such as machine learning and natural language processing. These often appear following major conferences, pointing out and commenting on the most interesting papers that appeared at those venues, or offering monthly summaries of ongoing developments. These can be extremely valuable for those looking to stay on top of a fast-moving field, and are often written by researchers themselves. Although this means that the authors have strong technical familiarity with the work being discussed, it also means that they are mostly aimed at insiders. Moreover, any researcher’s time will inevitably be limited, and their insider status may make it harder to develop a critical distance that is sometimes required to most usefully comment on and critique ongoing developments.

In these newsletters (as well as podcasts, zines, letters, editorials, and so on), we can see a kind of nascent critical community, one that could be so much more. Indeed, if we look again to artistic industries, there is arguably great value in the role played by the professional critic. Although somewhat of a rarefied role for both music and film (at least in terms of those who are able to earn a living from it) critics perform an important function in sorting through the work that is produced, providing insight, and helping to inform audiences about what they are likely to enjoy. Far more important than that, however, critics collectively work to establish and extend a critical discourse about the thing that they spend their time studying, and can exert an important influence on pushing a medium in a useful and interesting direction.

In practice of course, some critics are little more than another part of the promotional arm of the distributors, providing another means by which to advertise. Not only do critics promote certain works over others, they contribute to the prominence of an industry as a whole, encouraging people to play more video games, see more theater, or consume whatever their chosen medium might be. (A nearest neighbour equivalent for science might the type of science journalism which does little more than promote the latest nutrition or health study, without applying any kind of critical lens). In exceptional cases, however, with people such as Richard Brody, or Susan Sontag, or Manny Farber, critics can be true savants who bring a unique perspective to a medium, who can take a longer-term, more sweeping view, and who can help an audience to appreciate new work on a deeper level than it otherwise would. Moreover, they arguably play an important role in shaping the evolution of a medium by promoting work that breaks new ground, and encouraging creators to take it farther.

In the best cases, critics seek out and promote fascinating, challenging, and marginalized work that might otherwise never get seen. The provide a public service in helping people to understand and appreciate new work, and to see why what is happening now is new and exciting. The considered opinions of critics help to raise the level of public conversation, and to encourage everyone to see new work in the context of history. Critics help to establish a vocabulary and shared idea space in which people can discuss ongoing work, and identify trends and patterns that may not be visible to the casual observer. They provide recognition of work that has strong qualities, even if not conventionally successful, and can provide essential encouragement to emerging creators, at a much more grass-roots level than established awards. Finally, they can productively point out when a particular line of work has grown stale, or has sacrificed its integrity, or may be tending in a dangerous direction.

What would a professional science critic look like in practice? In an idealized form, one could imagine either a media company or a non-profit employing someone to write a column covering the papers that are regularly being written in a specific field. It is unlikely that they would be able to read everything, but like the film critic, there would be some expectation that they would at least look at a broad swath of output, both the slick commercial work, as well as the hidden-gem indy paper. By devoting all of their time to reading the work in an area, and developing their thoughts by ceaselessly writing about it, they might develop a deeper understanding than even experts in an area, for whom many hours are taken up with writing grants, teaching, service, and so on. Moreover, just as in other industries, they would likely command vast power, as a either a favourable or unfavourable review could likely serve to direct a great deal of attention to particular work.

This, of course, could be a problem. In many ways academia is already a system in which the rich get richer, and those with privilege (being at a top institution, or working with a famous advisor, for example) enjoy enormous advantages which can more easily be sustained than obtained. However, critics can also push back against this anti-meritocracy. While it is not hard to find some critics who will provide glowing reviews of just about everything directed by, for example, a Martin Scorsese, the most interesting critics typically push against the mainstream. Most great critics are nerds at heart, and they are in the game largely to seek out the rare and challenging work; they have had more than enough of repetitive state-of-the-art superhero films, and review such products only because it is required of them (in many cases leading to some of their best and most enjoyably scathing critical writing).

There is an obvious objection to this, which is the question of who will pay for it. Criticism in artistic industries such as movies, music, books, or video games, is viable precisely because these are consumer markets, meaning that potential customers want to get a considered opinion of something before they pay for it. Whether something similar could work for science is an open question, but I personally want to believe that there would be an audience for this. Based on book sales, there is clearly an enduring fascination with certain academic areas, such as psychology. A critic who adopted the role of educating the public over a longer period of time, of helping the reader to understand how to engage with the primary literature, rather than simply reporting on the latest press release, would provide a valuable service both to the scientific community itself and to the broader public. (In fact, an interesting one-off example of this was recently provided by the New York Times).

Indeed, the current moment is perhaps the greatest opportunity that has ever arisen for creating such a role. Interest in the science of pandemics and epidemiology is surely at an all time high, and the pace of publishing in those areas is currently unprecedented. Many people, such as Helen Branswell, Nicholas Cristakis, and Tyler Cowen are already devoting much of their time to sorting through ongoing developments in the area and summarizing them for their audiences. Andrew Gelman has been doing long been doing incredible critical work for statistics and social science more broadly, and could be an ideal role model. It seems plausible that if a major newspaper hired a professional science critic to cover a topic such as epidemics, or even just covid-19, and gave them a column, they would be richly rewarded with clicks. Whether such a position could be maintained in normal times or not is debatable, but it does not seem unimaginable that it could, especially for fields badly in need of this sort of criticism, such as epidemiology, economics, nutrition, and machine learning.

Public trust in science has reportedly surged in recent months, though there is still much misunderstanding about what it is that scientists do, and how science actually works. While not everyone is likely to develop an interest in any one niche, I am convinced there is a hunger out there among people for better coverage of what is happening in science, more insightful analysis, and better critical discourse, both among the general public, and within the scientific community itself. I am grateful to those who are already doing this work, and hope that many more will follow in their footsteps!