FM synthesis – How one of Stanford University’s most profitable licensing deals defined the sound of the 1980s

Recommended listening – a-ha, Take On Me

Everything covered so far in this music technology series has been fully analog technology (i.e. using physical circuit components to directly manipulate voltages). However, the development of transistor computers in the late 1950s brought with it the possibility of generation and manipulation of sound in the digital realm. 

One area where analog electronics excelled was the generation of harmonically rich sound – it was relatively easy to connect circuit elements together in a way that generated sharp waveforms such as sawtooth, square or triangle waves. The Eccles-Jordan circuit (featured in the first article of this series) was able to produce a square wave as early as 1920, and the theremin (featured in the second article of this series) used distortion as a convenient way to add harmonics. Synthesisers such as the Moog (featured in the third article of this series) can be classified as “subtractive” synthesisers in that they start with a sound with lots of high harmonics, which are selectively removed or shaped with filters. Therefore, if a digital synthesiser was to have any chance of competing with existing analog options, it would need to be capable of generating lots of high harmonics.

However, this was an area where early digital synthesisers struggled. Initially, the only computationally feasible approach was known as “additive synthesis”, where each sine wave harmonic needed to be added individually with separate digital oscillators. Unfortunately, this required hundreds of individual oscillators before any kind of complex or realistic sound could be produced.

In view of this, engineers in the 1960s were wondering if there was a more efficient way to generate harmonics digitally. The breakthrough was made in the late 1960s by a Stanford University researcher called John Chowning. His proposed approach was seemingly simple – you modulate the frequency of one sound using another sound, similar to an incredibly fast vibrato. This technique (called “frequency modulation” or “FM” synthesis) could produce hundreds of natural-sounding harmonics using only a small number of digital oscillators. This is demonstrated in the video below, where it takes only a few seconds and three digital oscillators to create a funky metallic bass sound.

Given its clear potential for musical synthesis, a patent was filed in 1974 on behalf of Stanford University, and this was granted in 1977 (US4018121A). Chowning initially thought that the technology would be of most interest to electric organ manufacturers such as Lowrey and Hammond who were still the dominant players in the market for electrical musical instruments. However, although Hammond expressed interest in the FM concept, Stanford reported that its engineers did not have sufficient experience with digital electronics to implement it.

Hammond’s struggles were understandable – in the end it was Yamaha that obtained an exclusive license for the FM synthesis patent; the exclusivity of the license was in part justified by the fact that Yamaha had to put in significant research of their own to turn the initial concept into a viable product. For example, frequency modulation in its literal sense was very difficult to run in real-time on existing digital processors; to enable real-time sound production, Yamaha discovered that modulation of the phase could produce an equivalent sound to modulation of the frequency, but in a manner that was much easier to implement (see US4643066A, filed in 1975). In fact, many software-based “FM synthesisers” in use nowadays actually use phase modulation behind-the-scenes.

The many years of development eventually led to Yamaha’s DX7 synthesiser, released in 1983. This synthesiser was a massive success and arguably defined the sound of music in the 1980s. For example, the inspiration for my video demonstration above was the DX7’s “BASS 1” preset, which was used in songs such as a-ha’s “Take On Me”, and “Danger Zone” by Kenny Loggins. The “E PIANO 1” preset was perhaps even more ubiquitous; it was featured on 40% of all the US Billboard Hot 100 number ones in 1986 and can be heard on songs like George Michael’s “Careless Whisper”.

By the time Stanford’s original FM patent expired in 1994, it had generated 20 million dollars in licensing revenue, which made it “the second most lucrative licensing agreement in Stanford’s history”. 

The monumental success of FM synthesis clearly demonstrated the benefits of being an inventor and early adopter for new groundbreaking music technology, and both Stanford and Yamaha were on the lookout for other technologies that could be used to replicate this success. The approach they identified as having the most potential was “physical modelling synthesis”. In essence, physical modelling seeks to simulate the physics of musical systems thereby enabling more realistic reproductions of real instruments. 

The key invention that kick-started this field was made by Stanford researchers Kevin Karplus and Alexander Strong. They realised that many real-world instruments create sound from some kind of excitation that repeatedly reflects and interacts with itself within a resonant system. This applies to many real instruments, such as the impact of a drumstick resonating inside a drum shell, the noise of a violin bow resonating along a string, or the buzz of a trumpet mouthpiece resonating within a long tube (i.e., a trumpet).

With this insight, Karplus and Strong created an algorithm (the “Karplus-Strong algorithm”) that uses a filtered feedback-loop to transform noise into a musical and realistic sound. A patent application (US4649783) for this algorithm was filed in 1983 and granted in 1987. 

The video below demonstrates how this algorithm can be used to make a reasonably realistic snare drum sound. The sound starts with a noisy impulse (representing the impact of the drumstick). When this is fed into a delayed feedback loop (representing the reflections within the drum shell) a tone is generated with its pitch dependent on the delay time. Putting a filter inside this feedback loop (representing how frequencies are absorbed within the material of the drum) quickly provides a resonant ringy drum sound. Finally, a white noise burst (representing the rattle of the snare drum wires) is blended into the output.

Refinements were made to the algorithm by another Stanford researcher called Julius Smith, with patent applications filed from 1986 (e.g., EP0583043A2). The refined technology (known as “digital waveguide synthesis”) was licensed to Yamaha and was deployed in the groundbreaking VL1 and VP1 synthesisers in 1993 and 1994. The realism and expressiveness that these products were able to achieve was remarkable, as is clear from these demonstrations of the VL1 and VP1.  The downside was their cost – at the time the VL1 and VP1 would set you back £4,000 and £10,000 respectively. 

Nevertheless, physical modelling modules were licensed and incorporated into several popular synthesisers over the next few years. For example, the Korg Prophecy synthesiser (used prominently by groups such as The Prodigy and Radiohead) included a plucked string model alongside more traditional FM and analog-style oscillators. Howard Jones confirmed that he used the Yamaha VL1 for many instrumental parts on his 2000 album Perform.00 (e.g., see the “trombone” solo two minutes into the second track). Thanks to the speed of modern digital processors, physical modelling synths have become a staple part of digital audio workstations (DAWs), such as Collision or Corpus (included in Ableton Live), or Sculpture (included in Logic Pro). As I demonstrated with the snare drum video above, you can even manually set up a Karplus-Strong feedback loop on your computer (and I recommend this video by Zion Jaymes who uses a more complicated version of the algorithm to create realistic cymbal sounds).

In summary, the collaborations between Stanford and Yamaha in the 1970s and 1980s have left a lasting impact on the world of synthesisers and were enormously profitable for the parties involved. Moreover, the strategy used for filing the patents discussed above demonstrates a key advantage of the patent system in that it allows for later filings to build upon earlier filings as the technology develops. For example, even though the initial implementations (i.e. direct frequency modulation and the Karplus-Strong algorithm) were not the ones that were deployed in Yamaha’s products, these early filings nevertheless cleared space in the industry so that Stanford and Yamaha could refine their approach first (i.e., phase modulation and digital waveguide synthesis) before filing further patent applications and fully commercialising the technology. 

Although the above approaches were often motivated by a desire to produce more realistic recreations of physical instruments, this does prompt the reasonable question of: “would it not be easier to simply record a real instrument and play the recording back later?” The answer to this question (and many more) will be answered in the next instalment of this music technology series.

If you would like to discuss anything in this article further, or you have an invention that you would like to protect, then please contact the author, or get in touch with our patents team at gje@gje.com.