Charles Petzold

Ever since sound and music were integrated into Windows I've wanted to write a program that plays some music and animates a printed score in synchronization. I recently had the opportunity to try it.

The music I chose for this experiment was Franz Schubert's dramatically riveting “Gretchen am Spinnrade” (“Gretchen at the Spinning Wheel”), one of the 700-odd songs written by Schubert before his death at the age of 31.

From a link at the bottom of the “Gretchen am Spinnrade” Wikipedia page I got access to PDFs of public domain editions of the score that I was able to turn into a giant bitmap. (Details below.)

ArkivMusic.com lists 94 recordings of “Gretchen am Spinnrade”, and I originally was planning to get permission to use a commercial recording. Then I discovered a video of the song performed in recital by a very talented teenager named Emily, who kindly gave me permission to use her video in this program.

Uniting the PNG file of the score and the WMV file of Emily's performance is a 160-line XAML file.

The text of “Gretchen am Spinnrade” is from a scene in Goethe's Faust in which Gretchen sits at the spinning wheel in a state of romantic despair. Schubert wrote a brilliant piano accompaniment that obviously mimics the turning of the spinning wheel, but with a claustophobic repetition that intensifies the feeling of Gretchen’s sexual obsession. We hear her working the spinning wheel with nervous energy, and the harmonies of the song follow the ups and downs of her desires and pain:

GretchenAmSpinnrade.xaml

A few caveats: About 10 megabytes of downloading is required before the song begins playing. The program requires a browser client window size of at least 1024 × 768. (If your screen resolution is set higher than 96 DPI, you'll need a larger window.) If your screen size is exactly 1024 × 768 you'll want to put Internet Explorer into full-screen mode by pressing F11. The animation might be a little jumpy on your machine (it is on my notebook) but it's much smoother on faster machines with multiple processors.

To play it again from the beginning, use the browser Refresh button.

It is known that Schubert composed “Gretchen am Spinnrade” in Vienna on October 19, 1814. It is known that the day before he wrote this song Schubert attended a massive celebration to mark the first anniversary of the German/Austrian defeat of Napoleon at the battle of Leibzig. This celebration might have inspired Schubert to perform his own little German/Austrian collaboration by writing a song based on a text by Germany’s greatest living writer.

What remains a mystery, however, is how Schubert managed to achieve such penetrating psychological and emotional insights inherent in this song just one month short of his 18th birthday.

The GretchenAmSpinnrade.xaml file is fairly simple in structure: It references the video as a WMV file in a MediaElement, and displays the PNG file of the score as an Image element in a Canvas. When the MediaElement begins playing the video, it fires a MediaOpened event, which is used to trigger the key-frame animation that pulls the Image element across the Canvas.

Although no actual code is apparent in this program, I wrote two ad hoc C# programs behind the scenes to help prepare the bitmap and the XAML animations.

There are probably better ways to turn a PDF file of scanned pages back into bitmaps, but I did it by displaying the six pages of the score on the screen a half page at a time, and then hitting PrintScreen, pasting each half-page into a bitmap-editing program, and cutting it into pieces. I ended up with 29 bitmaps, all but two of which consisted of four bars each.

I then wrote a program using the WPF bitmap classes to load these 29 bitmaps and assemble them into one very wide bitmap. This program got more and more complicated as I worked on this problem. Of course the 29 bitmaps didn't match up exactly, so I added some code to search for the first black pixels from the top and align them all. I then discovered that the original scans were tilted somewhat so I enhanced the program even further to slightly rotate each image. (For the bitmap rotation logic I used code I derived from my June 2008 article in MSDN Magazine.)

Even after aligning the top of the staff in each bitmap, often the staves of the piano accompaniment didn't join precisely. I contemplated more work but I had to let it go. Thanfully it isn't very obvious as the bitmap moves across the screen.

I enhanced this bitmap-creation program even further to add 60 pixels of blank vertical space under the German words of the song. I then pulled the resultant bitmap into Windows Paint and added the English translation, consulting translations included with the (I don't want to mention how many) CDs of “Gretchen am Spinnrade” that I own, combined with a very literal translation from the REC music web site.

The final bitmap of the score is nearly 32K pixels in width, which is about as wide as you can go with bitmaps before the WPF classes start acting a little funny.

I originally had a very ambitious scheme to synchronize the bitmap and the music. I thought that I'd have a program begin by doing a Fourier analysis of the music and find where all the notes are, and then I could peg certain notes of the music to certain offsets notes in the printed music. I eventually decided this would be overkill, and something considerably simpler might work just as well.

I wrote another WPF program (much smaller than the first) that contained a MediaElement to play Emily's video and a button. Every time the music got to the note at the beginning of one of the 29 four-bar staves, I clicked the button. The program accessed the Position property of the MediaElement and logged the elapsed time in a file. Each key frame of the animation of the bitmap score was based on this elapsed time and the accumulated widths of these 29 bitmaps. The only part that required a little fiddling is when the music slows down and stops at the word "Kuss."

I was pleased I was able to do the presentation entirely in a XAML file, and it worked fine on my machine.

Then I copied the three files to my web site, and the whole thing fell apart.

The problem was simple: The XAML file accesses both GretchanAmSpinnrade.wmv (over 7 megs) and GretchenAmSpinnrade.png (over 3 megs). However, the MediaElement can begin playing the WMV file before it's fully downloaded, and that's what it did: The video began playing almost immediately before the PNG file was fully downloaded. About 15 seconds into the video, the score would show up on the screen, fortunately synchronized but still a little late.

I needed to delay playing the video until the bitmap was fully downloaded, but there's no good way to do this in a XAML file. The BitmapImage class (which the Image element implicitly uses, and which can be specified directly) has an IsDownloaded property, but it's not backed by a dependency property and has no other change notification. There's a DownloadCompleted event (inherited from BitmapSource) but it's not a routed event so it can't be used in an EventTrigger. Of course the MediaElement has Play, Pause, and Stop commands but these are all methods and not accessible in a XAML file.

My solution, I'm afraid, was a bit of a kludge: I noticed that the Image element had an ActualWidth of 392 before the bitmap was fully loaded (the same value as the Height I gave it for the height of the bitmap). I then wrote a Style for the MediaElement that set the Source to null if the ActualHeight of the Image element is 392.

I then used that same logic to display text indicating the bitmap is downloading, and to display a Wait cursor during this time, and to delay the display of text under the video.

Someday every event will be a routed event, and every property will be a dependency property, and solving problems like the one I encountered will be so much easier.