Protein Counterpoint Sonification

We are researching the use of algorithmic composition to generate counterpoint through fracture. Driven by the universal physical forces near a singularity, captured in the so-called Cauchy stress tensor (read more about it here: en.wikipedia.org/wiki/Cauchy_stress_tensor), we are creating patterns of interdependent polyphony through independent patterning in time and pitch.

FRACTURES ARE UNIVERSAL

A fracture is a crack in a material (often seen in glass, but also represented in earthquakes when tectonic plates fracture, or in the unfolding of a protein). Fracturing is the process of destruction of structure, and hence is the division between function and failure in material and has correspondences in philosophy. The tipping point of the crack, the locality of greatest action, and where bond breaking occurs, is the singular crack tip, the sharp edge.
All fractures feature a universal distribution of stresses – the forces that act inside the material and which stretch/compress chemical bonds. No matter what material, as you get closer to the singularity – the tip of the crack – the distribution of forces are identical.

In the three pieces in this set, we explore these universal features to generate algorithmically composed music. We try to explore whether the universal features of fractures have the capacity to generate counterpoint – universal features of music, to better understand the analogy across manifestations.
Read more about the physics of fracture in [1].


Fracture-generated counterpoint hence may reflect the universal features of fractures and provides a model thereof. Owing to the universal significance of fracturing as a process in material and amaterial manifestations, the musical effects generated hereby can be an interesting subject of further analysis and inspiration.

[1] M. Buehler, Atomistic Modeling of Materials Failure, 2008 www.springer.com/gp/book/9780387764252

Counterpoint Through Fracture

Based on work by:

Isabelle Su, Ian Hattwick, Christine Southworth, Evan Ziporyn, Eran Egozy, Tomas Saraceno, and Markus Buehler. 

In this experiment we use the sound extracted from moving through a spider web to determine equal temperament tuning pitches, and then arpeggiate through them. As we move around the spider web, the detected notes change over time, leading to an evolution of the sounds.  

The melodies that emerge through this process show how musical information can be extracted from the vibrational patterns, and their changes due to movements, from a web.

References:

Imaging and analysis of a three-dimensional spider web architecture

Isabelle Su, Zhao Qin, Tomás Saraceno, Adrian Krell, Roland Mühlethaler, Ally Bisshop and Markus J. Buehler

https://doi.org/10.1098/rsif.2018.0193

Spiders are abundantly found in nature and most ecosystems, making up more than 47 000 species. This ecological success is in part due to the exceptional mechanics of the spider web, with its strength, toughness, elasticity and robustness, which originate from its hierarchical structures all the way from sequence design to web architecture. It is a unique example in nature of high-performance material design. In particular, to survive in different environments, spiders have optimized and adapted their web architecture by providing housing, protection, and an efficient tool for catching prey. The most studied web in literature is the two-dimensional (2D) orb web, which is composed of radial and spiral threads. However, only 10% of spider species are orb-web weavers, and three-dimensional (3D) webs, such as funnel, sheet or cobwebs, are much more abundant in nature. The complex spatial network and microscale size of silk fibres are significant challenges towards determining the topology of 3D webs, and only a limited number of previous studies have attempted to quantify their structure and properties. Here, we focus on developing an innovative experimental method to directly capture the complete digital 3D spider web architecture with micron scale resolution. We built an automatic segmentation and scanning platform to obtain high-resolution 2D images of individual cross-sections of the web that were illuminated by a sheet laser. We then developed image processing algorithms to reconstruct the digital 3D fibrous network by analysing the 2D images. This digital network provides a model that contains all of the structural and topological features of the porous regions of a 3D web with high fidelity, and when combined with a mechanical model of silk materials, will allow us to directly simulate and predict the mechanical response of a realistic 3D web under mechanical loads. Our work provides a practical tool to capture the architecture of sophisticated 3D webs, and could lead to studies of the relation between architecture, material and biological functions for numerous 3D spider web applications.

---

The raw audio can be listened to here: https://soundcloud.com/user-275864738/spider-web-sonification-porous-web-melody-x

Arpeggiated Spider Web

Inspiration from Nature knows no boundary and can take on unusual forms.

Fire has fascinated humankind since the prehistoric era. Rooted in the interactions between sound and flames, we report a method to use fire for a variety of purposes, including sonification, art, and the design and manufacturing nature-inspired materials.

We present a method to sonify fire, thereby offering a translation from the silent nature of flames, to represent audible information, and to generate de novo flame images.

To realize material specimen derived from fire, we use the autoencoder to generate image stacks to yield continuous three dimensional geometries that are manufactured using 3D printing.

This represents the first generation of nature-inspired audible signals from fire, and can be a platform to be used for other natural phenomena in the quest for de novo architectures, geometries, and design ideas, thus creating additional directions in artistic and scientific research through the creative manipulation of data with structural similarities across fields.

Original paper: https://www.cell.com/iscience/fulltext/S2589-0042(21)00841-5

Fire sonification: Making silent flames audible

Musical reflection of the molecular Intersection of nucleocapsid antibody and SARS-CoV-2, for piano, violin, and cello. In collaboration with Yun Yoo-jung, Alex Wooju Lim and Hyung Joon Won

Molecular Intersection of Nucleocapsid Antibody and SARS-CoV-2 (for Piano, Violin, and Cello)

Molecular structures are translated into a multi-instrument score, focused on the nuclocapsid spike in SARS-CoV-2 interacting with an antibody molecule. The complex piece, resembling the entanglement of three protein chains, is analyzed and a series of musical fragments are extracted (parts I, II, III, IV and V). The parts build on each other and are organized in increasing complexity, I being the simplest and V being the most complex version.

N Spike Antibody Fragment V

Fire is one of the key elements in Nature, used from the prehistoric era to heat living environments, to cook food for survival and enjoyment. 

Challenging the silent nature of fires, we utilize an assembly of deep neural networks to find a way to better understand the underlying mechanics of fire, and render complex but silent flames audible by associating them with a unique sonic spectrum.

The total image of the fireplace is segmented into small partials, each of which is analyzed for its sonic equivalence in terms of the local flame features, then rendered into audible signal using wavetable synthesis.

This offers a thorough exploration of the deep, complex structural content of the flames in this fireplace.

Fireplace Sonification

The construction of biological matter, and life itself, features striking similarities with musical creativity. In my research, I have explored these relationships with a combination of biology, chemistry, as well as mathematical and music theoretical tools. 

Listen here how dinoflagellate (a single-celled organism with two flagella, occurring in large numbers in marine plankton and also found in fresh water)  induce hierarchical vibrations translated to musical form.

Superfluous Dreams (Violin Adagio) - Part V

Swamp Variations: Emergence, Evolution, Division, Spring, Retreat 

Part of a series of 5 parts: This is Evolution.

Ambient version (reduced orchestration, focus on piano and flute, string instruments)

Swamp Variations - Part II - Evolution (((AMBIENT SPACE VERSION )))

This piece is constructed entirely from the sounds generated from amino acids, the building blocks of proteins. Encoded by the DNA sequence, amino acids form chains that assembly into hierarchical materials that serve diverse functions in biology (as enzymes, structural materials, signal transmitters, and many others). 

All sounds used to compose this piece are generated from the sounds of amino acids, computed from their quantum mechanical vibrations, reflecting their elementary molecular features. For further details on the scientific foundation, see [1]. 

The main melody is derived from the amino acid sequence of three proteins:

1) 6cjc (adrenodoxin reductase, a flavoenzyme that is involved in the biosynthesis of steroid hormones; https://www.rcsb.org/structure/1CJC)
2) A de novo protein synthesized using a machine learning algorithm as reported in [1]
3) 194l (lysozyme enzyme; found in tears of humans and animals and egg white; https://www.rcsb.org/structure/194L)

The melodic backbone is composed of a progression from proteins 1) to 2) to 3). Motifs derived from these proteins are used throughout to generate repetitive elements and to conjoin melodic concepts (e.g., segments extracted from 6cjc are used as transitional elements between 1cjc and the de novo protein and again during the transition to 194l. 

All rhythmic elements (bass drum, snare-like, hi-hat, and others) are generated from soundings of amino acids. Both melodic and rhythmic patterns used are derived from the three proteins cited above. 

No other synthetic or sampled sounds have been used in the composition. What you hear is a composition in the natural 20-tone amino acid scale. 

BACKGROUND:

Materials and music have been intimately connected throughout centuries of human evolution and civilization. Indeed, materials such as wood, animal skin or metals are the basis for most musical instruments used throughout history. Today, we are able to use advanced computing algorithms to blur the boundary between material and sound and use hierarchical representations of materials in distinct spaces such as sound or language to advance design objectives. The approach used in this work is that the translation of protein materials representations into music not only allows us to create musical instruments, but also enables us to exploit deep neural network models to represent and manipulate protein designs in the audio space. Thereby we take advantage of longer-range structure that is important in music, and which is equivalently important in protein design (in connecting amino acid sequence to secondary structure and folding). This paradigm goes beyond proteins but rather enables us to connect nanostructures and music in a reversible way, providing an approach to design nanomaterials, DNA, proteins, or other molecular architectures from the nanoscale upwards. 

Reference:

[1] C.H. Yu, Z. Qin, F. Martinez, M.J. Buehler, A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design using Artificial Intelligence, ACS Nano, in press, 2019

Orchestra Of Amino Acids

Sonification of a silk protein, rich in beta-sheet secondary structure.

Reference:

[1] C.H. Yu, Z. Qin, F. Martinez, M.J. Buehler, A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design using Artificial Intelligence, ACS Nano, in press, 2019

Last Move 84: Silk Protein Sonification

This piece reflects the sonification of an amyloid protein. Amyloids are a type of folding of a protein into highly stable, sticky fibrils, which are associated with around 50 diseases. 

Learn more about amyloids: https://en.wikipedia.org/wiki/Amyloid

Reference:

[1] C.H. Yu, Z. Qin, F. Martinez, M.J. Buehler, A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design using Artificial Intelligence, ACS Nano, in press, 2019

Amyloid protein sonification

This piece is constructed entirely from the sounds generated from amino acids, the building blocks of proteins. Encoded by the DNA sequence, amino acids form chains that assembly into hierarchical materials that serve diverse functions in biology (as enzymes, structural materials, signal transmitters, and many others).

All sounds used to compose this piece are generated from the sounds of amino acids, computed from their quantum mechanical vibrations, reflecting their elementary molecular features. For further details on the scientific foundation, see [1].

The main melody is derived from the amino acid sequence of a the lysozyme enzyme with Protein Data Bank identifier 194l.  Motifs derived from this  proteins are used throughout to generate repetitive elements and to conjoin melodic concepts.

Learn more about lysozyme here: https://en.wikipedia.org/wiki/Lysozyme

All rhythmic elements (bass drum, snare-like, hi-hat, and others) are generated from soundings of amino acids. Both melodic and rhythmic patterns used are derived from the protein cited above. No other synthetic or sampled sounds have been used in the composition. What you hear is a composition in the natural 20-tone amino acid scale.

BACKGROUND:

Materials and music have been intimately connected throughout centuries of human evolution and civilization. Indeed, materials such as wood, animal skin or metals are the basis for most musical instruments used throughout history. Today, we are able to use advanced computing algorithms to blur the boundary between material and sound and use hierarchical representations of materials in distinct spaces such as sound or language to advance design objectives. The approach used in this work is that the translation of protein materials representations into music not only allows us to create musical instruments, but also enables us to exploit deep neural network models to represent and manipulate protein designs in the audio space. Thereby we take advantage of longer-range structure that is important in music, and which is equivalently important in protein design (in connecting amino acid sequence to secondary structure and folding). This paradigm goes beyond proteins but rather enables us to connect nanostructures and music in a reversible way, providing an approach to design nanomaterials, DNA, proteins, or other molecular architectures from the nanoscale upwards.

Reference:

[1] C.H. Yu, Z. Qin, F. Martinez, M.J. Buehler, A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design using Artificial Intelligence, ACS Nano, in press, 2019

Concert of Enzyme Lysozyme

Going deeper inside a spider web. Multiple spider web variations interact across a spectrum of sonic textures. The sounds are based on a sonification method of a spider web as described in the references below [1-2], part of a larger collaboration with Tomás Saraceno, Saraceno Studios, Ally Bisshop, Thomas Muehletahler, and Evan Ziporyn, and the MIT Center for Art, Science and Technology (CAST).

Several individual webs constructed by Cyrtophora citricola are played in this piece, creating an overlaying soundscape of different shades. The piece includes a rhythm track for added structure, but all other sounds you hear are generated from the sonification of the spider web. 

The image [3] shows a 3D spider web, lit up using a green laser during scanning. The spider is visible in the top middle as she constructs the web. 

References:

[1] Su, I.; Qin, Z.; Saraceno, T.; Krell, A.; Mühlethaler, R.; Bisshop, A.; Buehler, M. J. Imaging and Analysis of a Three-Dimensional Spider Web Architecture. J. R. Soc. Interface 2018, 15 (146), 20180193. doi.org/10.1098/rsif.2018.0193.

[2] Su, I.; Qin, Z.; Bisshop, A.; Muehlethaler, R.; Ziporyn, E.; Buehler, M. J. Sonification of a 3D Spider Web and Reconstitution into Musical Composition Using Granular Synthesis. in submission

[3] Image by I. Su, MIT and M. Buehler, of a tidarren sisyphoides spider building a web, in collaboration with Tomas Saraceno Studio (Berlin)

Spider Variations

A musical experiment with proteins, coyote, bells, guitar, and a rooster. This is an audible journey through a variety of natural sources of sound, invoking various forms of life, materials, as well as electronic synthesis methods.

Typically, sound is made from vibrating a material, like a guitar string, and music is made by arranging sounds in hierarchical patterns. Here we explore a variety of modes of sound generation - from string vibrations (guitar), to protein vibrations (molecular music), to animals (coyote and rooster), to bells, and a few other unusual sources. In one sweep, our ears pick up all of the features a complex hierarchical system: pitch, timbre, volume, melody, rhythm, and chords. 

Music, as a universal language, connects people across the world. More so, it allows us to explore the physical features of matter - both living systems and inanimate - in their ability to generate sound through energetic excitation. Hence, it is a model of our shared humanity. And we can cross scales and species, as done in this piece, where we visualize distinct organisms like animals, materials, and humans interacting in a sort of virtual jam, in musical space.

Energy Equilibrium: Proteins, Coyote, Bells, Guitar, and a Rooster (Expy Madsy)

What you hear is the sonified expression of the 3D structure of the protein with PDB ID 1akg (ALPHA-CONOTOXIN PNIB FROM CONUS PENNACEUS, see: http://dx.doi.org/10.2210/pdb1AKG/pdb), translated into an orchestral setting (koto [a Japanese string instrument], cello, double bass, viola, flute, and fluegelhorn). The koto is used to express the primary, secondary and tertiary structure of the protein.  

The piece reflects a classical period model of the hierarchical mapping to describe the 3D folded structure of a protein in musical space. The musical mapping begins from the primary amino acid sequence (GCCSLPPCALSNPDYCX) and encodes multiple hierarchical structural levels, and adds musical experimentation. 

We use the concept proposed by Yu, Buehler et al. (ACS Nano) to map amino acid sequences to music, and augment the mapping by adding more data. The 3D folded structure of a protein is coded by inserting strummed chords and interwoven melodies that code for the geometric arrangement of amino acids, the protein's tertiary structure. Rhythm and note volume code for secondary structure.

[1] The protein used for this example is 6cjc (CRYSTAL STRUCTURE OF A FC FRAGMENT LALA MUTANT (L234A, L235A) OF HUMAN IGG1 (CRYSTAL FORM 3))

[2] https://doi.org/10.1021/acsnano.9b02180

Koto Ensemble of Protein 1akg (Venom of Predatory Marine Snails)

Sounds of the escherichia virus T4,  a species of bacteriophages that infect Escherichia coli bacteria. What you hear is the sonified expression of the 3D structure of the protein needle at the core of this virus, classified in the protein data bank with ID 4osd (triple-stranded beta-helix comprising the central spike of bacteriophage T4. 

See http://dx.doi.org/10.2210/pdb4OSD/pdb for more information.  

The music represents the spike-shaped protein that functions to disrupt the membrane of the target cell during phage infection. The protein structure has first been reported in [1].

The protein structure is translated into an orchestral setting (koto [a Japanese string instrument], cello, double bass, viola, flute, and a range of other orchestral instruments). The various instruments is used to express the primary, secondary and tertiary structure of the protein. The piece is composed of a A-B-A' structure, where A and A' reflect the beginning and end coding of the protein, the B part represents the coding of the primary backbone sequence and structure. 

The track image shows a picture of this protein, reflecting a highly organized, repetitive structure with continuous slight variations, which is also reflected musically. The extensive length of 40+ minutes is due to the long protein chains included in this protein assembly. 

The piece reflects a classical period model of the hierarchical mapping to describe the 3D folded structure of a protein assembly (consisting of 9 protein chains) in musical space. The musical mapping begins from the primary amino acid sequence and encodes multiple hierarchical structural levels, and is complemented with musical experimentation. 

We use the concept proposed by Yu, Buehler et al. (ACS Nano) [2] to map amino acid sequences to music, and augment the mapping by adding more data. The 3D folded structure of a protein is coded by inserting strummed chords and interwoven, cascaded melodies and rhythms that code for the geometric arrangement of amino acids, the protein's tertiary structure. Rhythm and note volume code for secondary structure.

References:

[1] Original research paper describing the gp5β-BC2 fragment: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4576200/

[2] https://doi.org/10.1021/acsnano.9b02180

[3] Track image shows the phage, a virus that infects and replicates within bacteria and archaea (source: Wikipedia, by: Adenosine, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=8643737, depicting the structure of a typical myovirus bacteriophage)

Viral koto ensemble of protein 4osd with cello, violin, flute and bass

In this piece we use algorithmic composition to generate counterpoint  through fracture. We are creating interdependent polyphony through independent patterning in time and pitch, driven by the universal physical forces near a singularity. 

FRACTURES ARE UNIVERSAL 

A fracture is a crack in a material (often seen in glass, but also represented in earthquakes when tectonic plates fracture, or in the unfolding of a protein). Fracturing is the process of destruction of structure, and hence is the division between function and failure in material and has correspondences in philosophy. The tipping point of the crack, the locality of greatest action, and where bond breaking occurs, is the singular crack tip, the sharp edge. 

All fractures feature a universal distribution of stresses – the forces that act inside the material and which stretch/compress chemical bonds.  No matter what material, as you get closer to the singularity – the tip of the crack – the distribution of forces are identical. 

Here we explore these universal features to generate algorithmically composed music. We try to explore whether the universal features of fractures have the capacity to generate counterpoint – universal features of music, to better understand the analogy across manifestations. 

Read more about the physics of fracture in [1].

HOW THE MUSIC IS CREATED

A set of Euclidean rhythmic motifs (with sparsity: 1/3, 5/12, and 7/12 to reflect different atomic organizations in crystalline forms) are acted upon based on the principal stresses near a crack under tensile loading.  We use the singular principal stress field to drive melodic content formation, whereas several independent moving observers assess the local principal stresses. The observes move in circular fashion, towards and away from the singularity in a spiral, oscillatory motion (see track image).  

Each individual note played is generated based on physical model of a string exposed to different stress levels. Hence, large stresses pull notes into high pitch, lower stresses compress it to lower pitches. 

The violent nature of forces experienced by the observer circling around the crack tip generate complex but highly organized melodic forms. As the stress field features universal, respective patterning, the various melodic lines generated have relationships rooted in the nanoscale physics surrounding the singularity. 

Fracture-generated counterpoint hence may reflect the universal features of fractures and provides a model thereof. Owing to the universal significance of fracturing as a process in material and amaterial manifestations, the musical effects generated hereby can be an interesting subject of further analysis and inspiration. 

[1] M. Buehler, Atomistic Modeling of Materials Failure, 2008 https://www.springer.com/gp/book/9780387764252

Counterpoint Through Fracture: Part I – Spiral Movers

For piano. We are researching the use of algorithmic composition to generate counterpoint through fracture. Driven by the universal physical forces near a singularity, captured in the so-called Cauchy stress tensor (read more about it here: https://en.wikipedia.org/wiki/Cauchy_stress_tensor), we are creating interdependent polyphony through independent patterning in time and pitch. 

FRACTURES ARE UNIVERSAL

A fracture is a crack in a material (often seen in glass, but also represented in earthquakes when tectonic plates fracture, or in the unfolding of a protein). Fracturing is the process of destruction of structure, and hence is the division between function and failure in material and has correspondences in philosophy. The tipping point of the crack, the locality of greatest action, and where bond breaking occurs, is the singular crack tip, the sharp edge.

All fractures feature a universal distribution of stresses – the forces that act inside the material and which stretch/compress chemical bonds. No matter what material, as you get closer to the singularity – the tip of the crack – the distribution of forces are identical.

In this study we explore these universal features to generate algorithmically composed music. We try to explore whether the universal features of fractures have the capacity to generate counterpoint – universal features of music, to better understand the analogy across manifestations.

Read more about the physics of fracture in [1].

HOW THE MUSIC IS CREATED

The piece is constructed from multiple movers in the singular stress field, and using the full stress tensor to derive musical structure. The observes move in circular fashion around the singularity in an oscillatory motion.  Each individual note played is generated based on the stress tensor, a physical quantity that describes the forces inside a material. Large stresses pull notes into high pitch, lower stresses compress it to lower pitches.

The violent nature of forces experienced by the observer circling around the crack tip generate complex but highly organized melodic forms. As the stress field features universal, respective patterning, the various melodic lines generated have relationships rooted in the nanoscale physics surrounding the singularity.

Fracture-generated counterpoint hence may reflect the universal features of fractures and provides a model thereof. Owing to the universal significance of fracturing as a process in material and amaterial manifestations, the musical effects generated hereby can be an interesting subject of further analysis and inspiration.

[1] M. Buehler, Atomistic Modeling of Materials Failure, 2008 www.springer.com/gp/book/9780387764252

Counterpoint Through Fracture: Part II - Stress Tensor Variations

This piece is a reprise of the motifs presented in Part I, now set into a more complex musical arrangement, and with more rhythmic guidance. 

As the other compositions in this series, this part is created based on the universal stress field of a crack, exploiting the unique features of the singular stress tensor to derive melodic and counterpunctual content through overlapping stress fields. 

GENERAL FEATURES: FRACTURES ARE UNIVERSAL 

A fracture is a crack in a material (often seen in glass, but also represented in earthquakes when tectonic plates fracture, or in the unfolding of a protein). Fracturing is the process of destruction of structure, and hence is the division between function and failure in material and has correspondences in philosophy. The tipping point of the crack, the locality of greatest action, and where bond breaking occurs, is the singular crack tip, the sharp edge. 

All fractures feature a universal distribution of stresses – the forces that act inside the material and which stretch/compress chemical bonds.  No matter what material, as you get closer to the singularity – the tip of the crack – the distribution of forces are identical. 

Here we explore these universal features to generate algorithmically composed music. We try to explore whether the universal features of fractures have the capacity to generate counterpoint – universal features of music, to better understand the analogy across manifestations. 

Read more about the physics of fracture in [1].

HOW THE MUSIC IS CREATED

A set of Euclidean rhythmic motifs (with sparsity: 1/3, 5/12, and 7/12 to reflect different atomic organizations in crystalline forms) are acted upon based on the principal stresses near a crack under tensile loading.  We use the singular principal stress field to drive melodic content formation, whereas several independent moving observers assess the local principal stresses. The observes move in circular fashion, towards and away from the singularity in a spiral, oscillatory motion (see track image).  Large stresses pull notes into high pitch, lower stresses compress it to lower pitches.  These motions provide the basis for the music you hear. 

The violent nature of forces experienced by the observer circling around the crack tip generate complex but highly organized melodic forms. As the stress field features universal, respective patterning, the various melodic lines generated have relationships rooted in the nanoscale physics surrounding the singularity. 

Fracture-generated counterpoint serves as a means to explore the universal features of fractures, and provides a model thereof. Owing to the universal significance of fracturing as a process in material and amaterial manifestations, the musical effects generated hereby can be an interesting subject of further analysis and inspiration. 

[1] M. Buehler, Atomistic Modeling of Materials Failure, 2008 https://www.springer.com/gp/book/9780387764252

Counterpoint Through Fracture: Part III - Spiral Roosters

This piece is an illustration how fractures can be used to design sound, here realized in a fracture drone. The piece is constructed from a single sine wave, by using several independent movers in the singular fracture stress field (see more details on the method in the description of Parts I and II). 

All 9 movers live within the fracture stress field and move stochastically (in angle, distance to singularity, stress intensity factor).  Additional harmonic content is derived from the melodies constructed in Part II. 

FRACTURES ARE UNIVERSAL

A fracture is a crack in a material (often seen in glass, but also represented in earthquakes when tectonic plates fracture, or in the unfolding of a protein). Fracturing is the process of destruction of structure, and hence is the division between function and failure in material and has correspondences in philosophy. The tipping point of the crack, the locality of greatest action, and where bond breaking occurs, is the singular crack tip, the sharp edge.

All fractures feature a universal distribution of stresses – the forces that act inside the material and which stretch/compress chemical bonds. No matter what material, as you get closer to the singularity – the tip of the crack – the distribution of forces are identical.

In this composition we explore these universal features to generate algorithmically composed music. We try to explore whether the universal features of fractures have the capacity to generate counterpoint – universal features of music, to better understand the analogy across manifestations.

Read more about the physics of fracture in [1].

[1] M. Buehler, Atomistic Modeling of Materials Failure, 2008 www.springer.com/gp/book/9780387764252

Counterpoint Through Fracture: Part IV - Fracture Drone

Protein resonances inspire a minimalist evolution of ideas across the resonances of piano and solo strings. The structure evolves slowly as the motion of jellyfish, a natural hydrogel, in water. Ethereal resonance.

Piano And Cello (VIII)

Ethereal Resonance: Protein Inspiration - VIII (PIano and Cello)

Protein resonances inspire a minimalist evolution of ideas across the resonances of piano and solo strings. The structure evolves slowly as the motion of jellyfish, a natural hydrogel, in water. Ethereal resonance.

The series of short pieces is based upon this musical reflection of infection, inverting the original molecular attachment: https://soundcloud.com/user-275864738/coronavirus-spike-protein-bound-to-ace2

Piano And Cello (VI)

Ethereal Resonance: Protein Inspiration - VI (PIano and Cello)

This piece is a sonification of the spike protein of the pathogen of COVID-19, 2019-nCoV (protein data bank identifier 6VSB [1]). What you hear is a multi-layered sonification approach featuring both the vibrational spectrum of the entire protein (expressed in sound and rhythmic elements), as well as the sequence and folding of amino acids that compose the virus spike structure [2]. 

References
[1] Wrapp et al., "Cryo-EM structure of the 2019-nCoV spike in the prefusion conformation," Science, 2020, DOI: 10.1126/science.abb2507
[2] Buehler et al., "A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design Using Artificial Intelligence," ACS Nano, 2019, DOI: https://pubs.acs.org/doi/10.1021/acsnano.9b02180

Sonification of a virus spike protein

Sonification of the Coronavirus Spike Protein (Amino Acid Scale)

While we cannot see small nanoscopic objects like proteins or other molecules that make up virtually all living matter including our cells, tissues, as well as pathogens such as viruses, our computational algorithm allows us to make its material manifestation audible.  This piece is a musical representation of the amino acid sequence and structure of the spike protein of the pathogen of COVID-19, 2019-nCoV (protein data bank identifier 6VSB [1]). 

MUSICAL DECEIT

A virus’ genome hijacks the host cell’s protein manufacturing machinery, and forces it to replicate the viral genome and produce viral proteins to make new viruses from it.  This musical art teaches us something about the fine line between beauty of life and death as an opposite pole.  As you listen to the protein you will find that the intricate design results in incredibly interesting and actually pleasing, relaxing sounds. This doesn’t really convey the deadly impacts this particularly protein is having on the world. This aspect of the music shows the deceiving nature of the virus, how it hijacks our body to replicate, and hurt us along the way. So, the music is a metaphor for this nature of the virus to deceive the host and exploit it for its own multiplication. 

HOW IT IS DONE: 

What you hear is a multi-layered algorithmic composition featuring both the vibrational spectrum of the entire protein (expressed in sound and rhythmic elements), the sequence and folding of amino acids that compose the virus spike structure, as well as interwoven melodies - forming counterpoint music - reflecting the complex hierarchical intersecting geometry of the protein [2].

Scientific references:

[1] Wrapp et al., "Cryo-EM structure of the 2019-nCoV spike in the prefusion conformation," Science, 2020, DOI: 10.1126/science.abb2507

[2] Buehler et al., "A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design Using Artificial Intelligence," ACS Nano, 2019, DOI: pubs.acs.org/doi/10.1021/acsnano.9b02180

Viral counterpoint of the coronavirus spike protein (2019-nCoV)

Viral Counterpoint of the Coronavirus Spike Protein (2019-nCoV)

Musical representation of de novo protein designed using artificial intelligence, and folded and equilibrated using molecular dynamics, as reported in [1]. The musical composition renders the 3D structure of this alpha-helix rich protein in an audible space, presenting complex melodic and rhythmic patterns that reflect the hierarchical design of this protein. This is a de novo protein, in other words, a protein that does not exist in nature, but which was designed through our AI algorithm that generated new music, then translated back into a protein amino acid sequence. 

[1] C.H. Yu, M.J. Buehler, "Sonification based de novo protein design using artificial intelligence, structure prediction and analysis using molecular modeling," APL Bioengineering, DOI: 10.1063/1.5133026, 2020.

Concerto of a De Novo Protein Designed using Artificial Intelligence

Collagen constitutes one-third of the human proteome, providing mechanical stability, elasticity, and strength to organisms and is the prime construction material in biology. Collagen is also the dominating material in the extracellular matrix and its stiffness controls cell differentiation, growth, and pathology. 

As a molecule, it forms a rope-like structure with three interwoven amino acid chains, resembling a powerful string, or triple helix. As one of the most abundant proteins, it plays a foundational role in all of biology. Read more about collagen in the references below [1-3].  

Creating its unique triple helical shape is a particular sequence pattern of amino acids, which is reflected in the regular, periodic realization of the musical score presented here. The protein sonified in this composition is a short collagen segment with Protein Data Bank ID 1cag, with identical chains, each of which has this sequence:

PPGPPGPPGP PGPPAPPGPP GPPGPPGPPG 

[1] https://en.wikipedia.org/wiki/Collagen 

[2] A. Gautieri, M.J. Buehler, et al., Nano Letters, 2012, https://pubs.acs.org/doi/10.1021/nl103943u

[3] M.J. Buehler, PNAS, 2006,https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1567872/

Collagen Etude

This is a sonification of a protein extracted from honeybees, and folded using a deep learning method described in [1].

BACKGROUND:

The development of rational techniques to discover new mechanically relevant proteins for use in variety of applications ranging from mechanics, agriculture to biotechnology remains an outstanding nanomechanical design problem. The key barrier is to design a sequence to fold into a predictable structure to achieve a certain material function. Focused on alpha-helical proteins (as found in skin, hair, and many other mechanically relevant protein materials), we report a Multi-scale Neighborhood-based Neural Network (MNNN) model to learn how a specific amino acid sequence folds into a protein structure. 

The algorithm predicts the protein structure without using a template or co-evolutional information at a maximum error of 2.1 Å. We find that the prediction accuracy is higher than other models and the prediction consumes less than six orders of magnitude time than ab initio folding methods. We demonstrate that MNNN can predict the structure of an unknown protein that agrees with experiments, and our model hence shows a great advantage in the rational design of de novo proteins.

Reference: Z. Qin, B. Marelli, M.J. Buehler, et al., "Artificial intelligence method to design and fold alpha-helical structural proteins from the primary amino acid sequence," Extreme Mechanics Letters, Vol. 36, p. 100652, 2020 https://doi.org/10.1016/j.eml.2020.100652

Honeybee Silk

This piece explores melodic and rhythmic content from a variety of protein structures, including the binding event of COVID-19 to human cells. The progression models pre-pandemic humanity and the virus alone ("growth"), then the virus interacting with human cell - a struggle modeling repeated infection of new cells ("infection") - and finally back to virus alone overlaid with the regular rhythm of human life ("release").  As expressed musically, life after the virus is different than before , since we all changed due to the pandemic. How, we do not know yet. 

LYRICS:

Growth, Infection and Release.
Growth represents prepandemic humanity and the virus alone.
Infection represents the period during with the virus is interacting with human cells.
There is a struggle between infection of new cells and human defenses.
Finally, we return to the regular rhythm of human life, the release.
Life after the virus is different than before. We all changed due to the pandemic.
How we have changed, we do not know yet.

MORE DETAILS:

The rhythm is derived directly from the vibrational spectrum of the virus spike protein attached to the human ACE2 receptor, scaled to the frequencies that human ears perceive as rhythm. 

The construction of this piece involves the vibrational frequencies of virus proteins alone, melding into overlays of the virus with human proteins attached, peaking and then decaying as the virus releases its grip on humanity.  The piece ends with the steady motions of human existence, as we enter release.

The entire composition is written in the amino acid scale, making molecular vibrations audible in a range of experiences from rhythms, to harmony, to melodies. For sources of sonic content, we explore a variety of protein structures - virus spike protein alone, spike protein attached to human cell, human cell alone, and various states of the human cell alone. These provide the melodic foundation on which the progression is based. 

The track image is taken from [1], a recent study to make protein vibrations visible via water waves.

[1] M. Buehler, Liquified protein vibrations, classification and cross-paradigm de novo image generation using deep neural networks, https://arxiv.org/abs/2004.07603, 2020

Growth, Infection and Release

Matter is sound and sound is matter. Since the mapping of amino acids to sound is unique, it can be reversed – out of sound, comes a unique sequence of amino acids. Here we experiment with an exercise to make sound into material, whereby we compose new music derived from protein sonifications of different virus species. This offers a new avenue for interspecies communication, using sound as a sort of universal language of life. 

The result sounds otherworldly, as we travel into the strange sounds, rhythms, and melodies of the inner workings of proteins - not just one, but a musical integration, a jamming, of different ones. The music is performed using an ensemble of violin, flute, koto and other harp instruments, brass, timpani drums, and string sections. 

To do this we generated a large amount of "songs" from spike proteins found in various coronavirus species.  We then teach an artificial neural network - AI - how these different proteins sound like. And then, we use AI to generate new music, and translate the music back to proteins. This allowed us to come up with new protein designs that nature has not yet invented. 

What you hear in this piece is the musical realization of this new protein design, a result of proteins from different viruses jamming together through machine learning, and self-assembling into new forms. It is a journey into a new world where proteins across species interact and recombine, each contributing patterns, ideas, from their respective language of life. 

This work was first presented at TEDx Basel on May 1, 2020. 

BROADER IMPACTS

Through music, we can learn to speak the language of life. This may revolutionize our ability to understand life, treat diseases, and develop new medical technologies and biomaterials, or even drugs. 

Music is everywhere. Matter is sound and sound is matter.

TRACK IMAGE

The track image was generated using the method described in [1].

[1] M. Buehler, "Liquified protein vibrations, classification and cross-paradigm de novo image generation using deep neural networks," https://arxiv.org/abs/2004.07603

Viral Protein Jam

Piano And Cello (VIIII)

Ethereal Resonance: Protein Inspiration - VIIII (PIano and Cello)

A musical experiment with proteins, coyote, bells, guitar, and a rooster. This is an audible journey through a variety of natural sources of sound, invoking various forms of life, materials, as well as electronic synthesis methods.

Typically, sound is made from vibrating a material, like a guitar string, and music is made by arranging sounds in hierarchical patterns. Here we explore a variety of modes of sound generation - from string vibrations (guitar), to protein vibrations (molecular music), to animals (coyote and rooster), to bells, and a few other unusual sources. In one sweep, our ears pick up all of the features a complex hierarchical system: pitch, timbre, volume, melody, rhythm, and chords.

Music, as a universal language, connects people across the world. More so, it allows us to explore the physical features of matter - both living systems and inanimate - in their ability to generate sound through energetic excitation. Hence, it is a model of our shared humanity. And we can cross scales and species, as done in this piece, where we visualize distinct organisms like animals, materials, and humans interacting in a sort of virtual jam, in musical space.

Energy Equilibrium: Proteins, Coyote, Bells, Guitar, and a Rooster (((Ambient Version)))

Ethereal Resonance: Protein Inspiration - VII(PIano and Cello)

The pathogen of COVID-19 continues to impact the world, as the pandemic wreaks havoc and spreads arounds the world. As time goes on, mutations of the virus emerge, which can affect how infectious and how lethal the virus is.

And, there are also different forms of coronavirus diseases – like SARS, MERS, and now COVID-19 – all caused by viruses in the same “family”.

What is the reason why MERS is so much deadlier, but less infectious, than COVID-19, which transmits easily but has a lower lethality rate than MERS or SARS? Can we better understand how the molecular details of the viruses work, and if they may provide clues for treatments or cures? 

It's in the music! In a paper published in Cell Matter, we have identified a strong direct relationship between the nature, rate and intensity of the virus spike protein vibrations and how readily the virus could penetrate the human cell, as well as an opposite relationship with the fatality rate of a given coronavirus. 

The key result of our work is that both, the overall flexibility of the upward receptor-binding domain and the mobility ratio of the receptor-binding domain in different conformations, represent two significant factors that show a positive scaling with virus lethality.  

The series of tones is: MERS, SARS, COVID-19, COVID-19 G614, COVID-19 G614, COVID-19, SARS, MERS, COVID-19 G614, COVID-19, COVID-19 G614, COVID-19 G614, COVID-19 G614.

Reference: Y. Hu, M.J. Buehler, Matter, 2020, 10.1016/j.matt.2020.10.032 

MIT News: https://news.mit.edu/2020/vibrations-coronavirus-proteins-1119

Sonic Landscape of MERS, SARS, COVID-19 and COVID-19 G614 Virus Spike Proteins

As mutations of the original COVID-19 pathogen emerge, we explore a musical assessment of different forms of coronavirus structures. In this example we focus on the G614 mutation, which has been dominating the spread of the virus in late 2020. 

The motions of molecules can not only be captured as “tones” or “timbres” in musical form, but can slowed down such that they are audible to the human auditory system as rhythm. 

Using this method, the vibrations of proteins in two mutants of the COVID-19 pathogen are translated into rhythms, and played simultaneously on the left and right audio channel, respectively. You can hear the slight differences between the mutations as a type of "stereo effect". 

Just like drum patterns follow a regular schedule of execution, the rhythms are played simultaneously forward, and backward, so that the resulting music features a symmetrical shape. 

A break in the symmetry is achieved through ever so slight alteration of the timbre of the beat, realized through the use of vocoders.

Emergence of Mutations: Differential Rhythms SARS-CoV-2 versus SARS-CoV-2 D614G

Original piece used in "The oracle of the virus, the spider". This is part of a series of musical explorations of SARS-CoV-2 virus-antibody interactions.

Winter Reprise

The oracle of the virus, the spider

Musical model of the COVID-19 antibody interacting with the tip of the spike protein. Composed by MIT professor Markus Buehler, arranged by In-Kyu Kim and Jae-Yoon Lim.  In collaboration with Hyung Joon Won and the Lindenbaum Festival Orchestra who premiered this piece on March 7, 2021.

Protein Antibody in E minor

Venom Progression

Markus J. Buehler is the McAfee Professor of Engineering at MIT, and a composer of experimental, classical and electronic music, with an interest in sonification. Using an approach termed "materiomusic", his artistic work explores the creation of new forms of musical expression - such as those derived from biological materials and living systems - as a means to better understand the underlying science and mathematics. One of his goals is to use musical and sound design as a novel and abstract way to model, optimize and create new forms of matter from the bottom up, and to assess cross-system design relationships. He is also interested in research to explore relationships between classical music, mathematics, and the physical and biological sciences, an in the mapping of models of consciousness across systems. In recent work he has developed a new framework to compose music based on proteins – the basic molecules of all life, as well as other physical phenomena such as fracturing, to explore similarities and differences across species, scales and between philosophical and physical models. 

Contact: mbuehler@MIT.EDU

Markus J. Buehler

Pinned to spotlight

Popular tracks