Activity 12 : Playing Notes by Image Processing

(Welcome, Interwebs traveler. A note before you go: please click any image to view in enlarged/theater mode. Now explore on!)

Today we'll be doing something that I find weird... we'll program our computer to play some sheet music via Scilab/MATLAB. It's weird because I used to play piano (a looooong time ago), and for me I'd rather just relearn how to play than prep my computer to play because:

•  if I play the piano, it's just a matter of relearning how to read sheet music and hitting the right keys; whereas,
•  if the computer plays, I have to crop the sheet music, apply morphological operations, detect each note, etc...

In any case, if I accomplish this activity, maybe I can put on my resume: plays the piano via MATLAB. Haha.

The sheet music that I chose is the piano piece "Pop Goes the Weasel", from our source [1]. (I tried to look for a piece with only one kind of note so I could start from there, but unfortunately even "Twinkle Twinkle" has more than one kind of note.)

Let's go over sheet-reading quickly. (Thank you, source [2], for the refresher course. Haha.) We only focus on the top lines, which means we'll only be reading the notes that are played with the right hand. There are three notes present for the right hand: the quarter note, the half note, and the dotted half note. We also have one kind of rest, the quarter rest. Our goal is to use image processing to isolate each note/rest, identify each note/rest, and translate the note/rest to a particular sound.

From our sheet music, I crop the desired lines and combine them to make one long line of notes:

Figure 1. One long line of notes for the right hand in "Pop Goes the Weasel."

To remove the lines, I applied an opening operation with a structuring element of a diagonal line of length 3 and degree 45. I isolated the quarter notes (sample in Figure 2(a)) from the cleaned image using correlation:

Figure 2(a). Sample quarter notes.
Image taken from [2].

Figure 2(b). Isolated quarter notes.

...and I isolated the half notes (sample in Figure 3(a)):

Figure 3(a). Sample quarter notes.
Image taken from [2].

Figure 3(b). Isolated half notes.

...as well as the dotted half notes:

Figure 4. Isolated dotted half notes.

...and finally the quarter rests:

Figure 5. Isolated quarter rests.

Initially, I got really frustrated with the half notes and the dotted half notes, because the quarter notes kept getting identified with the half and dotted half notes. In the interest of simplicity, I manually blocked out all the components which were not half or dotted half notes and took the correlation of that with the half/dotted half notes. :S

After isolating each type of note/rest, I took the centroid of each note to represent them as a single point (idea from blog [3]; I initially had no intention of using the centroid, but my idea of scanning through the whole image was too complicated!). Recalling Activity 5, the centroid [4] is simply the sum of all pixel coordinates on one axis divided by the number of pixels. I then took the convolution of a certain marker with the centroids, so that I have now clearly defined each type of note. To recombine the isolated notes, I simply added the images produced by convolution with the markers.

Figure 6 shows the markers, denoted in grayscale to better separate each marker:

Figure 6. Notes and rests converted to user-defined markers.

Since I now know the what each marker stands for, I can set the amount of time a certain note will play or a certain rest will umm, rest. I first used this source [5] to set the time, but later on I'll change the duration based on my preference.

What's left is to detect the row position of each marker on the image. With a corresponding note/rest per row value, MATLAB begins to sing!

And here is MATLAB's rendition of "Pop Goes the Weasel":
https://dl.dropboxusercontent.com/u/14710699/pop.wav

Grading

I got really frustrated with this activity, particularly while I was trying to differentiate the quarter notes, the half notes and the dotted notes. I am not satisfied with my performance here, so I give myself an 8/10.