Just over a year ago I was scheduled to demonstrate my research at NIME 2019, which took place at the Federal University of Rio Grande do Sul (UFRGS) in Porto Alegre, Brazil. My journey there comprised three flights (Manchester to Paris, Paris to Rio de Janeiro, and, finally, Rio de Janeiro to Porto Alegre), with a total time of approximately 18 hours. Unfortunately, immediately prior to boarding my flight to Paris, I received a text from the airline telling me that my flight from Paris to Rio de Janeiro had been cancelled. Assuming the provision of a suitable alternative means of continuing my journey, I decided to fly to Paris anyway. Upon arrival at Charles de Gaulle airport, however, I was informed that, in order to complete my journey, I would be required to fly to Lima in Peru the next morning before catching a connecting flight to Porto Alegre; a change that added an addition 12 hours to my total journey time and would see me miss the first day of the conference. Accordingly, I instead decided to return to Manchester the next day, a decision which meant I would miss the conference.
Thankfully, I was able to showcase my research later that summer at Audio Mostly at the University of Nottingham and was again invited to demonstrate my research at NIME 2020, which was scheduled to take place at the Royal Birmingham Conservatoire in the UK. When the COVID-19 pandemic struck, however, it looked like my attempts to attend NIME would again be thwarted. The response from the conference organisers, however, was amazing and, in the face of adversity, they ran the conference remotely with great success. I was required to provide a real time demonstration of my research via Zoom, but also produce a demonstration video, which can be seen below. Despite less than ideal circumstances, my experience at NIME was great and I hope to attend again in the future.
Since completing my PhD in April 2017 I have started a new job working as a full time Lecturer in Music Production at Edge Hill University. My research time in this role has been dedicated to the continued development of ScreenPlay, building on its existing strengths to make it a more stable, user-friendly, and practicable system.
Whilst some time has been spent on tweaking the underlying Markovian generative and topic-theory-inspired transformative algorithms, the majority of my attention has been focussed on a complete redesign of the GUI, as well as allowing for the Max for Live devices to be used completely independently of the GUI.
For the GUI redesign I decided to move over to Liine’s Lemur, having previously used Hexler’s TouchOSC, in order to harness its massively increased power and flexibility when compared with TouchOSC. The newly redesigned GUI is contained within a single page; a significant reduction from the three pages over which the TouchOSC GUI was previously distributed. Key/scale selection is now accessible via simple drop-down menus, as opposed to the selection matrix implemented in the previous iteration of the GUI. Additionally, changes to the key/scale applied to the grid-based playing surface are now handled entirely within Lemur, with the MIDI note numbers assigned to individual pads changing according to the key/scale selection. Previously, this had been handled within the ScreenPlay-CTRL Max for Live MIDI device, with the pads comprising the playing surface consistently outputting the same MIDI pitch information and this being transposed accordingly upon being received by ScreenPlay-CTRL. Likewise, velocity control of notes as defined by the velocity slider on the GUI is also accounted for inside Lemur, rather than in Max for Live as had previously been the case. The summation of these changes amount to an increased level of stability when using the system as well as reduced latency and CPU load.
In order to constrain the entire GUI to a single page – one of the goals of the redesign – it was necessary to devise a way of accommodating both the button matrix gird and the generative/transformative algorithm controls within the confines of a relatively small space. This has been achieved by utilising the same space on the GUI for both elements and enabling users to switch between to two. The surrounding controls remain unchanged regardless of what is displayed in the centre of the GUI, affording users access to clip, key/scale, tempo, velocity, and quantisation settings at all times.
Changes have also been made to the meta-level controls afforded by the GUI, with parameter controls for existing elements being improved and entirely new meta control functionality being added. Specifically, parameters such as record/global quantisation and clip/loop length are now accessible via drop-down menus (much in the same way as key/scale selection), and clip management has been improved through the removal of dedicated buttons for deletion, which have been replaced by hold-to-delete functionality. In conjunction with the changes to the organisation/distribution of the playing surface and algorithmic controls, all of this amounts to a streamlined interface with reduced demands on screen real estate. Newly introduced meta-level control functionality includes the addition of a play/stop button as well as a MIDI mappable drop-down menu for part selection when the system is being used in single-mode. Both elements are important additions with respect to improving the integration of ScreenPlay into the existing studio/live setups of practising electronic musicians.
Continuing in this vein, controls for the Markovian generative and topic-theory-inspired transformative algorithms have been added directly to the Max for Live MIDI Device GUIs, allowing for them to be used entirely independently of the Lemur GUI. As before, two-way communication between interfaces is exhibited when ScreenPlay is running in multi-mode, in that changes made to global parameters by one user are reflected in the interfaces of the others. Similarly, when running in single-mode, the GUI updates to reflect the status of the currently selected part. This functionality has now been extended so that two-way communication exists between the Max for Live MIDI Device GUIs and the Lemur GUI, which is particularly useful when using ScreenPlay in single-mode as part of an existing compositional/performative setup.
To read more about the conceptual framework underpinning ScreenPlay please refer here.
Between 2013 and 2016 I was working primarily towards my PhD thesis at the University of Salford; the focus of which was human-computer interaction (HCI) in music. Since then, however, I have struggled to find the time to post about the results of my research over the course of those 3 years amidst finding and starting a new job, and ongoing teaching responsibilities.
The focal point of my research has been the design and development of a unique and innovative interactive computer music system (ICMS), ScreenPlay. There are 3 main approaches to the design of ICMSs: sequenced, generative, and transformative. Sequenced systems are usually tailored towards a lone user and afford the opportunity to orchestrate/arrange a predetermined composition from constituent parts/loops. These systems are excellent for novice users due to the coherency of their musical output and often simple and engaging user interfaces (UIs). However, they are often devoid of any meaningful influence over the generated musical output by the computer. Incredibox is an excellent example of a sequenced ICMS.
Generative systems rely on an underlying algorithmic framework to generate musical responses to the input of the user. Better suited to supporting multiple users simultaneously, the musical output of generative systems is often stylistically ambient, and there can be little discernible connection between the control actions/gestures of the user(s) and the resulting musical output – thus limiting the scope for user(s) to exert a tangible influence over the music. As a result, generative systems can struggle to engage users with a higher level of musical proficiency. Examples of generative systems includeNodeBeat and Brian Eno’s Bloom and Reflection apps.
In most instances transformative systems are designed to respond to the incoming audio signal from a live instrument, and transform the sound of the instrument through various means of manipulation. Many early ICMSs were transformative in nature, with the process of design and implementation often being explicitly aligned with the composition and performance of a specific musical work. As a result, such systems are known as “score followers”; Pluton by Philippe Manoury being a prime example. The reliance of transformative systems on a level of instrumental proficiency means that contemporary examples are scarce, in particular in the context of electronic music.
A common trait among each of the three separate approaches to ICMS design outlined above is that the resulting systems often prioritise the affordance of influential control to the user(s) over one or two distinct musical parameters/characteristics, while at the same time ignoring the immense creative possibilities offered up by the multitude of other musical parameters/characteristics available. The systems mentioned above are just a few examples of ICMSs that each exhibit the characteristics of only one of the three overarching approaches to ICMS design: sequenced, generative and transformative. A common hindrance to the vast majority of ICMSs, ScreenPlay seeks to combat this exclusivity of focus through the encapsulation and evolution of the fundamental principles behind the three system design models in what is a novel approach to ICMS design, along with the introduction of new and unique concepts to HCI in music in the form of a bespoke topic-theory-inspired transformative algorithm and its application alongside Markovian generative algorithms in breaking routine in collaborative improvisatory performance and generating new musical ideas in composition, as well as providing new and additional dimensions of expressivity in both composition and performance. The multifunctionality of the system, which allows it to exist as both a multi-user-and-computer interactive performance system and single-user-and-computer studio compositional tool, including the affordance of dedicated GUIs to each individual involved in collaborative, improvisatory performance when in multi mode and the technicality of hosting up to sixteen separate users through a single instance of Ableton Live in order to achieve this, is another of ScreenPlay’s unique design features. The primary goal throughout the ScreenPlay‘s development cycle has been that the convergence of all these different facets of its design should culminate in the establishment of an ICMS that excels in providing users/performers of all levels of musical proficiency and experience with ICMSs an intuitive, engaging and complete interactive musical experience.
As previously mentioned, one of the most unprecedented inclusions in the design of ScreenPlay within the context of HCI in music is the topic-theory-inspired transformative algorithm. Topic theory, which was particularly prevalent during the Classical and Romantic periods, is a compositional tool whereby the composer employs specific musical identifiers – known as “topics” – in order to evoke certain emotional responses and cultural/contextual associations from the audience. In ScreenPlay the concept of topic theory is implemented in reverse, with textual descriptors presented to the user(s) via the GUI being used to describe the transformative audible effects had upon the musical output of the system by a variety of “topical oppositions”.
In total ScreenPlay affords the user(s) a choice of four “topical oppositions”, each of which is presented on the GUI as two opposing effectors at opposite ends of a horizontal slider; the position of which between the two poles dictates the transformative effect on the musical output of the system had by each of the oppositions. The four oppositions are “joy-lament”, “light-dark”, “open-close”, and “stability-destruction”, the first of which acts by altering the melodic and rhythmic contour of a musical phrase/loop to imbue the musical output of the system with a more joyous or melancholic sentiment respectively. The three remaining oppositions serve to transform the textural/timbral characteristics of the music in various ways. In order to achieve the desired effect as indicated by the position of the corresponding slider at the moment the transformation is triggered by the user, the algorithm which underpins the “joy-lament” “topical opposition” performs a number of probability-based calculations, the weightings of which change in accordance with the position of the slider, and, as a result, alter the relative transformative effect. These calculations include the application of specific intervals between successive pitches in a melodic line – with certain intervals being favoured more heavily depending on the position of the slider; the increase/decrease in the number of notes within a melody, note duration and speed of movement between notes; and the overall directional shape of the melody – whether favouring upward or downward movement. The respective positions of the sliders for the three textural/timbral “topical opposition” transformations – “light-dark”, “open-close” and “stability-destruction” – work by informing the parameter settings of a number of macro controls on Ableton Effect Racks, each of which is mapped to a multiple of parameters across numerous effects.
ScreenPlay‘s GUI is currently designed as a TouchOSC template, with the playing surface mimicking that of an Ableton Push. When in multi mode, each user (up to sixteen in total) is able to interact with the system through a single Ableton Live set via individual GUIs, which grant them influential control over a specific part within the arrangement of the musical output of the system. In single mode, the user can control up to sixteen distinct parts from a single GUI, with the interface updating in real time to display the status of the currently selected part. In order to achieve this, large swathes of the three Max for Live devices that constitute the system as a whole are dedicated to facilitating the two-way transmission of OSC messages between Ableton Live and TouchOSC. The assignment of pitches to the “pads” on the playing surface of the GUI, depending on the user-defined key signature/scale selection, is also currently undertaken by one of the three Max for Live devices. As such, I plan to develop a dedicated GUI in Lemur, which can process most of these tasks internally, thus reducing the demand on CPU of the central computer system, and will also facilitate the implementation of a wired connection between the GUI and the central system – if so desired by the user(s)/performer(s) – thus negating the impact of weak/fluctuating WiFi on the reliability and fluidity of the interactive experience. While it is already possible to bypass the playing surface displayed on the GUI and use any MIDI controller with which to play and record notes into the system, it is still necessary use the TouchOSC GUI to control the generative and transformative algorithms. Providing the option to bypass the GUI entirely by affording control over these aspects of the system directly through the Max for Live devices themselves is also an intended development of ScreenPlay.
ScreenPlay will undoubtedly be made publicly available at some point or another, although I am not yet sure when or in what form. As already pointed out, there are definite improvements to the design and functionality of the system that can be made – some more pressing than others – and I would like to take the time to refine some of these aspects of the system before releasing it.
Metrosync 