In our epilogue episode, we look at how an engineering professor, Naomi Leonard, has collaborated with artists to show how birds fly in a flock without bumping into each other; how robots can reflect our humanity back at us; and how other peoples’ rhythmic movements affect our nervous systems.
Engineering faculty at Princeton are increasingly working with artists to create an array of projects, and in the process, shining light on how people use, and perceive, the marvels that engineers create. We wrap up this oral history podcast series with a story about how a computer musician got a late night call from Stevie Wonder to talk shop, and in the process may have changed the way the music legend thought about digital voice synthesis.
List of music played in this episode:
“Red,” from “They Might Be Others,” music by Dan Trueman.
“The Story of Our Lives,” by Charles Dodge
“Race Babbling,” by Stevie Wonder
Sources:
“Flock Logic” video
Transcript:
Aaron Nathans:
In 2009, Naomi Leonard, a Princeton professor of mechanical and aerospace engineering, gave a talk on how flocks of birds and schools of fish move in patterns that seem highly choreographed. They stay together but somehow keep from colliding with each other. In her lecture, she talked about how these principles could be applied to designing robotic devices that work together efficiently.
Aaron Nathans:
In the audience was Susan Marshall, a professor and Princeton’s director of dance. She posed an interesting question: Would it be possible to explore these flocking ideas with dancers?
Aaron Nathans:
What followed was an elegant collaboration, which, perhaps surprisingly for a project that started out about birds, fish, and robots, shone light on how human beings respond to others when they’re in a group. You can still see a video from the December 2010 Flock Logic performances that followed. These performances included students, professional dancers, and volunteers from across campus who were given some training beforehand. Their dance appeared choreographed as they floated across the floor like fish swimming in tandem.
Aaron Nathans:
But they were simply following their training, which described the rules of flocking as found in nature. The presentations and the applause were over in a flash, but for Naomi Leonard, her work combining engineering with dance and other art forms was just getting started. <<THEME MUSIC>>
Aaron Nathans:
From the School of Engineering and Applied Science at Princeton University, this is Composers and Computers, a podcast about the amazing things that can happen when artists and engineers collaborate.
Aaron Nathans:
I’m Aaron Nathans.
Aaron Nathans:
Part Six, Dance Me to the End.
Aaron Nathans:
I say artists and engineers, not just musicians and engineers, because this spirit of collaboration at Princeton goes well beyond coaxing sounds out of a digital device.
Aaron Nathans:
Just a few weeks ago, in a walk-up Philadelphia art gallery, several unassuming sculptures sat on the ground of a small exhibit room. The gallery is a few blocks away from the city’s convention center, where the Institute for Electrical and Electronics Engineers was putting on its robotics convention. And at lunchtime, conventiongoers with name tags and lanyards around their necks began wandering through.
Aaron Nathans:
As they entered the room, the previously still sculptures perked up as if at attention. You could see as people walked into the gallery how their physical expression changed when the robots appeared to take notice of their presence, and how both the people and the robots appeared a little bit more at ease as they got used to each other.
Naomi Leonard:
Ah, I think it just reacted to you. Or maybe to Aaron moving out. So it has a little bit of visual perception.
Aaron Nathans:
The motion of the robots gradually became almost meditative and certainly calming. When the visitors walked out, they were inevitably a little bit more serene.
Naomi Leonard:
We might have just done it again.
Visitor:
So when you move in and out here, that’s the…
Naomi Leonard:
Right, we wanted it to first and foremost be responsive, just to acknowledge that before you came in it was all by itself, and now it’s in the presence of human visitors. And so it pauses a little bit. And then it does that funny little, it’s like a little stutter-stutter. And because we have springs you get a little bounciness.
Aaron Nathans:
Here’s some of my interview with Naomi Leonard.
Naomi Leonard:
What I really want to do is explore how something mechanical… This is really, I think, important in this space of human robotic interaction, which is a really important and growing topic. How do we engage with these kinds of machines? And here in this project I think we can explore, in this case we’re thinking about rhythm and how a mechanical system that engages in a rhythmic motion can connect to members of an audience. How does it make you feel?
Naomi Leonard:
Can we explore how communication through bodily movement can inform say how robots should respond to an audience member? And moreover, how we can make this sculpture make people feel smoothed or calmed and create an active and meditative public space of connection?
Naomi Leonard:
So I do think that these opportunities to explore the human side of technology is absolutely one of the important new ways that art and engineering can push us.
Aaron Nathans:
It’s hard not to feel inspired talking to Naomi Leonard, who graduated from Princeton in 1985. She recalls taking dance classes when she was a young girl, dancing into her 20s and coming to appreciate the many forms of the arts.
Aaron Nathans:
Her parents took her to art museums and music, and she watched a lot of dance growing up. At the same time she also loved mathematics. She went to Princeton as an undergraduate, arriving as an undecided liberal arts major.
Naomi Leonard:
But then I discovered engineering as a place where I could be both creative but with the tools that I love, like mathematics and physics.
Naomi Leonard:
And I actually took the course that would change my life. This field of, it’s called control theory, but it’s all about feedback and dynamics. It’s all about how a system, whether it’s a mechanical or whether it’s human or organic in some way, is constantly taking measure of what its state is, how it’s behaving and responding. And changing how it behaves constantly to regulate, to respond to change.
Naomi Leonard:
And this was a course where I discovered it was all about… There was nothing that was turnkey about it. It was all about making your own choices based on your insights. And it felt super creative to me, and this is what I’ve been doing ever since.
Naomi Leonard:
And I don’t think back then I necessarily connected it to my interest in dance and art more generally, but I think that came together over time. It’s while I’ve been at Princeton and working with lots of different kinds of collaborators, including people in ecology and evolutionary biology and people in neuroscience, thinking about groups and collective dynamics and collective behavior, a natural connection to dance and choreography. And even to thinking about music and other forms of art.
Aaron Nathans:
The Rhythm Bots Project is only the latest collaboration between Leonard, who is the Edwin S. Wilsey Professor of Mechanical and Aerospace Engineering, and Susan Marshall and her colleagues at the Lewis Center for the Arts.
Aaron Nathans:
In a different project in 2018, Leonard performed a mathematical analysis of dancers’ decisions and opportunities for collaborative composition in a rule-based improvisational work entitled “There Might Be Others.” It was choreographed by Rebecca Lazier, a senior lecturer in dance at Princeton’s Lewis Center for the Arts. The music was composed by Princeton Professor of Music, Dan Trueman. <<MUSIC>>
Aaron Nathans:
More recently she worked with Marshall to create a performance installation called Rhythm Bath, which explores how observing other people’s rhythmic movements affects our nervous systems. That inspired the questions about the rhythmic connection between humans and robots, which led to the Rhythm Bots Project.
Aaron Nathans:
In the spring of 2018, Leonard and several other engineering faculty members began offering a course co-taught by several arts and music faculty. It’s been offered ever since each spring. The class is called Transformations in Engineering and the Arts. It’s a partnership between the Council on Science and Technology, which Leonard directs, the School of Engineering and Applied Science, the Department of Music, and the Lewis Center for the Arts.
Aaron Nathans:
Leonard led the development of the course, it was broken into four parts. Adam Finkelstein, a professor of computer science led off teaching about visuals. Jeff Snyder from the music department – he’s the fellow who runs the Princeton Laptop Orchestra – taught the unit on sound.
Aaron Nathans:
Civil and environmental engineering faculty members, Maria Garlock and Sigrid Adriaenssens, led the unit on structure, and Leonard led the unit on movement. Meanwhile other faculty members from engineering and the arts participated to provide their own expertise.
Aaron Nathans:
Students created a wide array of projects. For instance, one computer science major designed a system in which a wheeled robot tracks a human performer’s movements through a motion capture system and translates that data onto paper.
Aaron Nathans:
Three other students, including a comparative literature major, built “Mother Womb,” a portable tent that envelopes the user with calming visual, auditory, and tactile sensations.
Aaron Nathans:
Here’s Ryan Osminkowski, who was a student at the time, when I interviewed him about the class in 2017. His project with his two partners transported their audiences into the jungles of South America, using motion capture, sound design, lighting, set building, projection, and a whole lot of coding.
Ryan Ozminkowski:
You’re being taught by mechanical engineers and computer scientists, but you’re getting lecturers from the Lewis Center.
Ryan Ozminkowski:
Tony-nominated lighting designers and sculptors and choreographers coming in weekly to teach you what to do. For me, I know I want to go in a show business when I’m older, whether that be Hollywood or Broadway or resorts or theme parks. And for me this was an amazing class where it’s teaching you the theatrics of it, but at the same time on a tech side. I think today in this day and age, technology is leading the way in everything. And to be good at entertainment you need to know technology.
Aaron Nathans:
The physical space that the class has been taught in is called the StudioL ab. Naomi Leonard.
Naomi Leonard:
So we created a space to provide the infrastructure for teaching this course, and I got input from all kinds of people across campus.
Naomi Leonard:
So we have theater lights because Jane Cox from the theater department helped us understand what that meant in this space. We have a floor that’s sprung for dance. We have motion capture so that we can use signals, not just to control robotics but also to create art around movement.
Naomi Leonard:
I think it just allows students to feel liberated from their labels, their disciplinary labels, to learn about different kinds of perspectives that they don’t get to think about all the time. Some of the students clearly are both engineers and artists. Some of them are majoring in English or neuroscience, doing all different kinds of things.
Naomi Leonard:
And yet we forget who’s from which departments by the end. And the students have really enjoyed, especially this opportunity to ask their own questions, not feel like they have to be driven by a prompt or a challenge. And really make something new. And often it’s most exciting just to hear them talk about having done something that they couldn’t imagine that they could have done before they’ve taken the class.
Aaron Nathans:
This class has led to a wider effort for engineers and artists to collaborate at Princeton. CreativeX is a new group of faculty who are interested in the intersection between engineering and the arts. It includes engineering faculty as well as those from music, architecture, and the Lewis Center for the Arts. Including dance, theater, visual arts, and creative writing.
Naomi Leonard:
We come together to ask questions, share our questions, share our perspective, learn about what each of us are doing, bring ourselves into collaborations or not.
Naomi Leonard:
Sometimes somebody is both engineering artist and sometimes it’s very much the meeting of the minds. I think it’s exciting to talk about things where there are parallels in ways. We think and do and make. It’s also really interesting to think about the tensions, the different kinds of tensions that we might experience together or that might be created between engineering and the arts, and how all of this can lead to new kinds of works.
Aaron Nathans:
Here’s Princeton Engineering Dean, Andrea Goldsmith, talking about how engineers thrive on intersections and collaborations.
Andrea Goldsmith:
I think one of my favorite examples is we have an initiative on engineering in the arts, which looks to the arts for inspiration, design, new ideas, new ways of doing things in engineering. And an example of that is Naomi Leonard who actually uses dance choreography to understand better how teams of robots can collaborate with each other towards a particular task.
Andrea Goldsmith:
So I think that that’s a wonderful example. Another example is looking at the intersection of architecture and engineering, where now we have robots that can actually build buildings.
Andrea Goldsmith:
And so architecture is very much of an intersection of science and engineering and design, and building robots that can create architectural structures that would not be possible for humans to create is another really exciting area of the intersection of engineering and disciplines outside of engineering.
Aaron Nathans:
Leonard’s colleague in the School of Engineering and Applied Science, Sigrid Adriaenssens, Associate Professor of Civil and Environmental Engineering, is working with Rebecca Lazier, the Associate Director of the Princeton Dance Program on an interesting new project. It uses the movements of dancers moving around and within nets as a strategy to explore how to build more resilient nets.
Aaron Nathans:
This project aims to create choreographic works that generate a new understanding of how nets turn stiff when loaded in this case with dancers, and soften when unloaded.
Aaron Nathans:
At the same time, dancers have been exploring the interaction between net and human. Lazier is choreographing in collaboration with visual artist Janet Echelman, and a team of dancers and riggers.
Aaron Nathans:
The project got its start at the University of Washington and the Princeton Atelier Program, and now includes a team of engineers from Princeton and the architecture and engineering firm, SOM.
Aaron Nathans:
They’re working to collect data to learn more about the net structure and properties. Rebecca Lazier.
Rebecca Lazier:
The apparatus of a net is capable of such drastic transformation, it becomes this world into itself. And so artistically I’m really compelled by something that has that much malleability. And unlike a simple fabric, the malleability relates to structure in terms of how it can be a surface we can stand on, how it can be a wave form, how it can be something we hang under, how it can be something we navigate in space.
Rebecca Lazier:
And the volume that’s created by draped nets also is this huge potential for sculptural transformation. And so I’m sitting in this space between the visual creative world of working with the visual artist, Janet Echelman, and her points of view on the design space of nets. And then we are taking this very embodied experimental approach that we’ve done long, long improvisations on nets, where we find what’s possible.
Rebecca Lazier:
And that time that we’ve put into with the intelligence of these incredible dance and circus artists, where they learn physically, if you push all of your force while you’re standing away from each other, you can stand forever on a net.
Rebecca Lazier:
And that only came from trial and error and trial and error, and lots of pain. (Laughs)
Aaron Nathans:
Sigrid Adriaenssens.
Sigrid Adriaenssens:
I’m very interested in how slender structures behave. So whether they are shells or grids or flexible nets. And they behave in very unexpected ways, especially net structures, there is actually not so much known about how they behave when they are loaded. And I really like working with Rebecca, because Rebecca and her dancers spend a lot of time in the nets because they want to achieve certain things within the nets.
Sigrid Adriaenssens:
And they see and observe phenomena that I cannot really think of myself. So basically they’re a little bit like guinea pigs trying out so many things with the nets, which I could not even imagine doing. Because nets are not really used that much in engineering applications, there’s very little known about them. And definitely people do not spend time rolling around in them or jumping in them and seeing how they are behaving.
Sigrid Adriaenssens:
And that’s what Rebecca and the dancers are doing, and they’re really challenging the nets to what I would call very extreme behaviors. And in doing that some very interesting engineering phenomena come out. And I’m interested in this phenomena, but I would never discover these things just by looking at a net hanging in my lab.
Sigrid Adriaenssens:
It takes people to really challenge those nets and do very extreme things with them, and that’s what they’re doing.
Aaron Nathans:
Lazier said it’s interesting that movement in one part of the net can affect what happens in another part of the net. Today, Lazier is a professor of practice and dance at the Lewis Center for the Arts. And Adriaenssens will be promoted to full professor in July.
Aaron Nathans:
Engineering in the arts is just one example of how engineers have been increasingly looking to collaborate outside the discipline for ways to help humanity.
Aaron Nathans:
Andrea Goldsmith.
Andrea Goldsmith:
I think that interdisciplinarity is a relatively new phenomenon in university. So if you look at how universities grew up, they grew up around disciplines and they’re structured around disciplines. You have departments that follow particular disciplines, then you have schools that are a larger umbrella around particular disciplines, like the school of engineering.
Andrea Goldsmith:
And it was, I would say in the last few decades, where the problems that engineering could solve became much more interdisciplinary, bigger problems, like climate change, like food shortages, like smart cities, like mitigating flooding. There’s so many things, even in biomedical research where the need for bringing larger groups of people together to solve these problems, larger groups of people that were not just engineers, became apparent.
Andrea Goldsmith:
And I think that one of the advantages that Princeton has, I would say unfair advantage, is that we are engineering within a liberal arts university, which has a long and storied tradition of the liberal arts thriving here.
Andrea Goldsmith:
And so our engineers are educated within that culture. They operate and work within that culture, and I think that that gives them a much broader sense of engineering design beyond what they’re learning in their classes or in their research labs. And that is where engineering, the profession, needs to go with to thrive. Is really reaching out and creating connections to other disciplines, creating centers and institutes and initiatives that are interdisciplinary. Because some of the most important problems that engineering and technology can solve no longer lie within the disciplines but across disciplines.
Aaron Nathans:
When we come back from the break, I’ll give some parting thoughts.
Aaron Nathans:
If you’re enjoying this podcast, you might want to check out our other podcast which also deals with technology. “Cookies: Tech Security and Privacy,” deals with the many ways technology finds its way into our lives. In ways we notice and in ways we might not. If you’re looking to shore up the security of your personal data and communication, you’ll find some great tips from some of the best informed people in the business.
Aaron Nathans:
You can find “Cookies” in your favorite podcast app or on our website, engineering.princeton.edu, that’s engineering.princeton.edu.
Aaron Nathans:
Let’s get back to our epilogue episode of “Composers & Computers.
Aaron Nathans:
This is part six of this podcast. If you missed the prior episodes, please check out the rest. It’s a fascinating story about how a batch of young composers at Princeton in the 1960s found their way into the computer center in the new Engineering Quadrangle, hoping to coax sound out of a room-sized mainframe computer. The work they did changed the sound of music.
Aaron Nathans:
Of course, this story was told with Princeton at its center. Other institutions have their own stories to tell in this area, and they have told those stories often.
Aaron Nathans:
Go to the Columbia Computer Music Center’s website and you’ll see their great story meticulously told, they even have their own archives. Or go visit the website of CCRMA, the Center for Computer Research and Music and Acoustics at Stanford. There’s plenty to see there too.
Aaron Nathans:
There are centers for computer music at Indiana, the University of Virginia, Brooklyn College, the University of North Texas, I could go on. Each has their own computer music origin story, testifying to why their work has been consequential.
Aaron Nathans:
At Princeton, our part of this story fell between the cracks over the last few decades. Numerous people that I interviewed mentioned how lucky we were that we didn’t create a formal center around computer music, which might have pigeonholed our work in this area and constrained the wild creativity of the faculty and students that made it possible.
Aaron Nathans:
But since the work relied solely on the interest and availability of the people who were here at the time, when they left, well, the stories went with them.
Aaron Nathans:
My goal through this podcast has been to try to piece this story back together while most of its participants are still alive and able to share their memories. And it is a consequential story.
Aaron Nathans:
In the words of Seth Cluett, who received his doctorate in music from Princeton and who is assistant director of the Computer Music Center at Columbia, quote, “I look at it like this. If you think about how many people listen to music and then realize that 100% of music is consumed via technology that had at its origins the innovations of the Princeton, Columbia, Bell Labs, and RCA research on audio technology and computing, it paints a picture of radical curiosity, driven creativity, and divergent, artistic thinking, pushing linear thinking engineers to innovate decades earlier than they would’ve done on their own.”
Aaron Nathans:
So let’s not forget the work of the composers and engineers in the computer center, work that started 60 years ago. They sparked that spirit of curiosity, creativity, and artistic and collaborative thinking in the digital space.
Aaron Nathans:
As you can see from the work of Naomi Leonard and her colleagues, it’s a spirit that shines brighter than ever today here at Princeton Engineering.
Aaron Nathans:
I want to finish with a story that really didn’t fit into any of the other episodes, but it’s a great story so I’ll tell it here.
Aaron Nathans:
You might remember from Episodes Two and Three, Charles Dodge, the Columbia grad student turned faculty member who used to hang out at the Princeton computer center a lot in the 1960s and ‘70s. Because at the time we had better equipment.
Aaron Nathans:
He was briefly a computer music instructor at Princeton. He learned linear predictive coding, a form of digital synthesis of the human voice from Princeton music professor Godfrey Winham just before Winham passed away in 1975.
Aaron Nathans:
It was also in 1975 that the phone rang in the middle of the night at Dodge’s home, 3:00 AM to be exact, but it was midnight Los Angeles time. Dodge struggled to find the telephone and picked it up. Charles Dodge.
Charles Dodge:
And the voice on the other end said, “Hi, it’s Stevie.”
Aaron Nathans:
Yes, that Stevie.
Charles Dodge:
We had a mutual acquaintance, and our mutual acquaintance had played for him or had sent him, I guess, maybe not in person, but sent him a tape of a piece of mine. And it was called “(The) Story of Our Lives.” <<MUSIC>>
Charles Dodge:
And Stevie Wonder really liked it and was really interested in it, and wanted to talk about it. And we talked for probably an hour about computers and how you could use them in music.
Charles Dodge:
Well, “The Story of Our Lives,” this is about a couple who were reading a book. And at a certain point the book speaks, and the voice of the book was a, what do you call them? Feeding a complex electronic sound through the changing resonances of a speaking voice, removing essentially the simulation of a vocal cord and instead implanting a complex electronic sound. So there’s this complex electronic sound that’s actually articulating language as if it were reading from this book. <<MUSIC>>
Charles Dodge:
Stevie said, “That was really scary.” And it is, it’s chilling, I guess you could say.
Aaron Nathans:
The method that Dodge used to create that chilling sound was called cross-synthesis. He did it by feeding a composed piece of electronic music through the linear predictive coding analysis of his voice, reading the text of the poem upon which the piece is based. It results in ghostly, barely intelligible electronic speech.
Aaron Nathans:
So, did Stevie Wonder use the method that Dodge explained to him in his music that followed to produce his own ghostly sound? Hard to say for sure, but take a listen to the eerie background vocal sounds in his song “Race Babbling,” from his 1979 album, “Journey Through the Secret Life of Plants.”
Aaron Nathans:
Dodge says it sounds like he was taking the idea of cross-synthesis and using it with vocoder technology instead of linear predictive coding. And that, just maybe, is how you get from Milton Babbitt to Stevie Wonder.
Aaron Nathans:
This has been “Composers and Computers,” the production of the Princeton University’s School of Engineering and Applied Science. I’m Aaron Nathans, your host and producer of this podcast.
Aaron Nathans:
I conducted all the interviews. Our podcast assistant is Mirabelle Weinbach.
Aaron Nathans:
Thanks to Dan Kearns for helping us out with audio engineering. Thanks to Dan Gallagher and the folks at Mendel Music Library for collecting music for this podcast. Graphics are by Ashley Butera. Steve Schultz is the director of communications at Princeton Engineering.
Aaron Nathans:
Thanks to Scott Lyon and thank you to the 25 or so people who agreed to be interviewed for this oral history podcast project. This would not have been possible without your enthusiastic cooperation.
Aaron Nathans:
This podcast is available on iTunes, Spotify, Google Podcasts, Stitcher, and other platforms. Show notes, including a listing of music heard on this episode, sources, and an audio recording of this podcast are available at our website, engineering.princeton.edu.
Aaron Nathans:
If you get a chance please leave a review, it helps.
Aaron Nathans:
The views expressed on this podcast do not necessarily reflect those of Princeton University. Thanks for listening everybody. Peace.
