VISTA MUSIC

This is software I developed standing for visual and tactile music. This is the name I gave to software I developed myself for people who were going extremely deaf. It contained all the features described in “helping with hard of hearing” in a particularly convenient form including means to control relative volumes.

It also had especially good facilities for creating expressive performances because deaf people do not like unexpressive music anymore than anyone else (and for this reason as a spin off I produced a version for physically disabled people to create their own expressive performances).

The visual display which was a colour graphic representation of each musical phrase at a time on the screen was greatly liked by hearing people and mildly hard of hearing but to my great disappointment did not help in cases of very great deafness.

PITCH PERCEPTION

Vibro acoustic sensation from the vibro acoustic chair gives a good general impression of the music especially of the rhythm but the problem to be solved with people with very little residual hearing is pitch perception. I believe that only coarse pitch differences can be perceived by vibration alone perhaps a minor third, e.g. C2E Flat or A2C. Some people think I am wrong and that some people can detect tiny fractions of semitones by vibration alone but I think they are mistaken and some other mechanism is involved. I think the most likely explanation for this difference of opinion is hearing but not consciously. Hearing below the conscious level is a scientifically recognised phenomenon. If this is not the explanation, then it is one of the great mysteries that I have come across when working with deaf people.

So in an attempt to get precise pitch perception for a very deaf person, the obvious move is to use the place of vibration on the body. In people with normal hearing, the place on the cochlea, where the resonance of the hair cells occurs gives much of the pitch information so it would seem logical to make an array of small vibration transducers on the body with different ones being activated for different pitches. For the time being, limiting ourselves to just two part music, we placed an array of small vibration transducers on the right forearm for the upper part and the left forearm for the lower part. This works for MIDI music only (which could be either computer generated or played live from a MIDI-enabled instrument). Getting this to operate was a considerable technological achievement because the electronics had to look ahead and see which note was coming next and route it to the appropriate transducer. I did this in conjunction with a final year electronics student as his degree project and was aided by some brilliant technicians in the electronics workshop of the department. The result from having the transducers in a row on the forearm was not good for the following reason. Imagine notes in a rising scale, C,D,E, F, G, AB, ‘C but the C would jump to a transducer at the bottom end of the array and despite vibrating twice as fast it would be sensed as a stronger version of the C at the bottom of the array. It would sound like a stronger version of C rather than the C above B. To get round this, I thought of using a transducer on each finger and thumb. The initial thought was to use pentatonic music (5 notes in a scale) but I thought this too limiting and realised I could use the heel of the hand on the left and right of the hand giving a location for seven transducers. This gave a much better result and one could imagine a rising scale of three octaves as a rising helix. This arrangement was done in both hands and the two part music could be perceived in positions of a right and a left helix. Despite all of this, it ran into an unexpected problem. The strong vibration on the back from the vibro acoustic chair tended to swamp the more delicate vibration on the fingers and if one turned down the volume of the sensation of the back so we could concentrate on the vibration of the fingers. It took away the main sensation of the music, the brain did not seem to be able to cope with both the sensation on the back of the vibro acoustic chair and the vibration in the fingers simultaneously. This was a most unwelcome and unexpected problem. But I think even if it could be overcome with training, there was a deeper flaw to the idea. That is, that it was giving information about the music rather than the music itself. I can explain this with the following analogy. If you write down CC,GG,AA,G-, that is the notes of the start of twinkle, twinkle little star but it is not twinkle , twinkle little star. So reluctantly, I had to admit defeat on this idea of vibro acoustic arrays.

VISUAL DISPLAY OF VISTA MUSIC

The other part of vista music that could be developed for very deaf people is the visual display. I did not put much effort into this because it seemed unlikely. Very deaf people like the vibration but did not pay much attention to the visual display. I had developed a much better way to allocate colours to notes as shown in figure 1, 2 and 3 which made musical sense rather than just going round sequentially through the colour circle and people with some hearing agreed it was much better than the previous methods which had all involved going round the colour circle sequentially. But it was not good enough; I think the reason for this is as follows. Colours were allocated to the pitch of the notes but that’s not enough because notes require another dimension. Take for example the note B; this can function for example as the leading note in the key of C or the tonic in the key of B or the mediant in the key of G. And that is equally important as is pitch but was not shown in the main vista music display. A small effort was made in an experimental computer programme to show how the note was functioning as well its pitch.

If we take the analogy with Braille, Braille could do nothing for you to convey a beautiful illuminated manuscript where you already knew the content. And similarly, the colour display can never convey the beauty of the sound. This could be further explored. At least as a start, it seems sensible to try it with only two kinds of music where the interest is not primarily in the beauty of the sound. For example A: Music in two parts such as Bach’s Two Part Inventions where the interest in the music is in the interplay of the two parts and would retain their interest even when played on an instrument which did not produce a beautiful sound and even if a little out of tune. And B: Block Corded Music such as Hymn tunes where the interest in the music is in the sequence of harmonies and these would retain their interest even if played on an cheap church harmonium which could not possibly be described as a beautiful sound. The argument being if we can’t do it for these two types of music; we can’t do it at all.