Recently a webpage asked me to confirm my humanity by identifying cars in a mosaic of nine photos. A challenge-response test designed to thwart bots, reCAPTCHA (“Completely Automated Public Turing test to tell Computers and Humans Apart”) also obliges internet users to help train image recognition algorithms for free. Since 1958, when Frank Rosenblatt first presented his Perceptron to the US Office of Naval Research, demonstrating how a program could detect the location of a square on flashcards, practitioners in the field of Artificial Intelligence have aspired, in the words of Kate Crawford, a scholar of the social and political implications of AI, “to capture the planet in a computationally legible form.”
Squinting at my screen, I zeroed in on the photos containing sedan-like vehicles, like the one next to the instruction “select all images with cars.” But other photos made me pause: one featured a bridge that cars might be traversing, while another contained a rank of buses. Abandoning nuance, I selected the sedans and passed: I am not a robot. It’s an excellent lesson in how artificial intelligence demands that humans curtail our tolerance for ambiguity in order to accommodate the objectives of the program.
Listen beautiful relax classics on our Youtube channel.
Such encounters with computer vision formed the mundane background to my reading of Computational Formalism: Art History and Machine Learning, Amanda Wasielewski’s timely endeavor to examine how the most rapidly advancing technology affecting society touches on the domains of art history, collecting, and the art market. The book opens by asserting the existence of a conflict between art historians and computer scientists grounded in mutual misunderstandings about each other’s fields, which Wasielewski aims to reconcile through an explanation of the strengths and weaknesses of artificial intelligence as an aid in the analysis of art.
At the center of the book are two chapters: the first provides an overview and critique of computer scientists’ use of digital reproductions of artworks in their research on how to improve the capacity for computer vision to identify and designate pictorial style; the second examines how museums and collectors are using machine learning to attribute and authenticate works of art. Wasielewski then closes by offering suggestions for how to bridge disciplinary divides between art historians and computer scientists. Given the increasing integration of artificial intelligence into many areas of academia, a study that investigates this terrain is sorely needed. But there are significant discrepancies between what the book does and how it frames its subject.
Wasielewski specifies that she is addressing neither the digital humanities more broadly nor artificial intelligence at large, but looking narrowly at machine learning and computer vision. Of course, machines don’t learn. The term “machine learning” was coined by IBM in 1959 for marketing purposes, and all it describes is a mathematical formula (an algorithm) applied to a collection of inputs (a dataset), which produces an output. That same formula, after training, should then produce the same outputs when applied to a different dataset.
The “supervised” version of machine learning involves training an algorithm on thousands of labeled digital images. For instance, an algorithm whose training set includes images of apples labeled “apple” can be expected to predict accurately which images contain an apple when applied to a different assortment of images. The problem is that if the training dataset contains only red A new book offers an unconvincing look at the art historical implications of machine learning. by Sonja Drimmer Art History Is Not a Robot Computational Formalism: Art History and Machine Learning by Amanda Wasielewski, Cambridge, MA, MIT Press, 2023; 200 pages, 4 color and 6 black-and-white illustrations. BOOK REVI EW 5 2 A r t i n A m e ri ca / Summer 2023 Photo Amanda Wasielewski apples, the algorithm will fail to identify an image of a green one.
In the “unsupervised” version, algorithms are applied to training sets without labels, and the program determines for itself which features define a target output. If the majority of the set’s images of apples include bowls, it may conclude that this is a defining characteristic of apples.
In both cases, supervised and unsupervised, we confront what scholar Brian Christian has termed “the alignment problem,” the divergence between human norms and values, and computational models’ abstraction of those norms to categories. Whatever happens between input and output happens in a black box and, at present, there is little humans can do to make its internal working scrutable.
Scrutable or not, the process creates three conditions that constrain the promise of computer vision for art historical research. First, any application of computer vision to works of art must be carried out on digital reproductions of them as opposed to the works themselves, a necessity for making any object in the world assimilable to a computer. The second stumbling block is the reliance of algorithms in supervised learning on tagging systems—meaning the metadata attached to a digital reproduction, with all the biases of the humans who created that metadata. Finally, the utility of an algorithm depends on the nature of the dataset on which it is trained, and since, as Wasielewski points out, canonical works of Western art grossly overpopulate the landscape of digitized art (for the same reasons of imperialism and inequity that made the Western canon canonical in the first place), then computer vision will continue to wrongly categorize or discount anything outside of that.
With these constraints in mind, the first chapter of Computational Formalism offers a valuable survey of research carried out by computer scientists since 2005 with aims to automate the process of sorting works of art into stylistic categories. Wasielewski issues a trenchant critique of those applications in their return to the unreformed formalism of academic art history’s past, an era when Heinrich Wölfflin’s Principles of Art History (1915) took a commanding position in the field and categories like “Archaic” and “Hellenistic,” “Gothic” and “Romanesque” were construed as if they assumed an autonomous status apart from the human minds that conceived them.
Frankly, I am in awe of Wasielewski for her patient wading through what is, from the perspective of an art historian, an abundance of stunningly inane, and in some parts morally repugnant, writing that presumes “that the core aim in the study of art is to find groups of images that look similar” or that conflates style with beauty that can be quantitatively and objectively measured. The problem, as she shows, is that this research confuses human categories with truth inhering in form, makes no distinction between the digitized image and the object it digitizes, and fails to recognize partiality in the available datasets.
After handily dispatching the failures of this research, Wasielewski addresses another, more profound, issue plaguing it: ignorance that all categories are constructed, contingent, and relational. One example is the problematic label “Primitive,” which has been applied to works of diverse appearance and which makes meaning differently depending on context, whether as a pejorative for non-Western or pre-modern works, a condescending reference to a self-taught artist, or a celebration of one who subverted the conventional academic style in which they were trained. Clear to the art historian, but obscure to the computer scientists responsible for this research, is that assigning a category is an exercise of power.
From the domain of general style Wasielewski turns to individual style, and specifically, the high-stakes world of connoisseurship and attribution. While Wölfflin served as the avatar for the formalist revival examined in the preceding chapter, the 19th-century art historian Giovanni Morelli takes the stage as the mascot for the formalist renaissance pursued here. Giving new life to Morelli’s methods—attempting to “scientifically” identify the oeuvres of painters based on symptomatic features like earlobes—tech entrepreneurs use computational connoisseurship to atomize digital reproductions of authenticated works into patterns, which they promise their algorithms can detect in digital reproductions of unattributed works by the same artist.
Where Wasielewski is unequivocal in stating that “using image-based computational techniques to answer humanistic questions in art history is … a flawed endeavor,” she is less pessimistic when it comes to matters of attribution. The distinction, she argues, is between the fallacy that a general style resides in form and the certitude that an individual’s does. But given the historical peculiarity of this notion of authorship—whether applicable to select modern artists, or to a 19th-century fantasy of old masters’ individualistic practice—its remit is narrow.
Attuned to this shortcoming and its service to the Western canon, Wasielewski shrugs, citing computational connoisseurship as just one in a box of variously unreliable tools that collectors and museums will or will not use in their pursuit of wealth. After a brief section floating the potential value of machine learning to automate the identification of iconography, Wasielewski moves on to more interesting issues relating to the epistemology of authenticity itself, which, with the advent of generative artificial intelligence and its capacity to confect and even forge images, has emerged as a pressing concern among artists and society at large.
Wasielewski sets up her study as an effort to quell a battle between art historians and researchers in computation, arguing that we are witnessing a resurgence of the “science wars” of the 1990s, when scientists and humanists brawled over the concept of objectivity. But in order to position the current situation as such, Wasielewski must paint today’s antagonists as equally invested in and inhibited by ignorance about one another’s areas of expertise: on the one side are computer scientists who fail to understand basic art historical premises (like the conditional nature of stylistic categories); on the other side are art historians who produce faulty research by failing to understand the limitations of computation.
Yet the book’s only examples of humanists using machine learning in the interpretation of images are not the work of art historians: they are all studies undertaken by a single new media scholar, Lev Manovich. For Wasielewski, “computational formalism” describes “a revival of formalist methods in art history facilitated by digital computing,” and she frames the book’s central question as an examination of this shift’s implications for the discipline. If such a shift has happened in art history as the result of image processing by means of machine learning, Wasielewski provides no evidence for it.
In the final section of her book, Wasielewski suggests that humanists’ apprehensions about collaborating with computer scientists are about policing disciplinary purity. But whether or not that is the case, one needs to qualify that allegation with the recognition that this is also about disciplinary empathy. There’s a reason why we have seen the growth of “humanities labs” and not “sciences salons.”
None of this impugns Wasielewski’s incisive commentary on machine learning— the book musters the most skillfully limned assessments of its functionalities and limitations when applied to works of art to date. But I don’t share her optimism that a rapprochement can be achieved, since the “war” isn’t between humanists and scientists but rather between humanists and the administrators and legislators defunding and legislating us into extinction. If there is any hope of nurturing critical machine learning to induce a jolt to art history, then it will need to do more than disclose the violence of its brute reliance on inherited categories, which, using traditional forms of humanistic research, Wasielewski portrays so well.
Wasielewski concludes by endorsing metaphorical language as a channel for easing communication between scientists and humanists. But it seems to me that a metaphor is the very source of our alienation: namely, the description of computer vision as vision. If, in order to “see,” a computer demands that anything in the world be homogenized and processed as data, apart from all context and embodiment, then it is not vision; it is information management. And at present such information management can only induce us to see the world as information to be managed, plunging us further into a new era of categories, with all their errors.
Listen beautiful relax classics on our Youtube channel.
Source: artnews.com