Piano Recognition For Smartphone - What Do You Think?

Hello fellow pianists :-) I would like to hear your opinion. I am the university student and this year I am working on my bachelor's thesis. I would like to make a smartphone application that would help the piano students or composers to easily transfer their ideas to piano score. The application would be capable of recording someone playing the piano then it would recognize the single notes or polyphony, the rythm and finally it would generete the MIDI output. The MIDI output could be then imported to some 3rd party dekstop software (like Finale, etc.) to obtain the classical piano score.

Before I start I need to know whether you, pianists, would even like to use such an application. So would you use an application that can recognize and process the piano sound and generate the MIDI output? And if so what mobile platform would you prefer? Android / iOS / Windows Phone / Symbian / BlackBerry / anything else? What do you think?

I really need your feedback and I would appreciate every answer! :-) Thanks in advance.

it would be really tough to make. Even so hard to the extend, I think it is impossible.

Is the result a single MIDI channel with both left and right hand? then the result would be pretty ugly, and quite useless...

Is the resulting MIDI capable of splitting the audio input (2 midi channels, left and right hand separated ), then you shouldnt bother creating an app for it, just start a new company, and make money. Only a insanely ingenious algoritm (if at all) could split this. This splitting is a prerequisite for pianistic scores, but I think impossible to make.

AVID

noble effort

professional quality

i'm afraid you have been sadly misled mr carter

(ps <333 the sqs and dialogues xoxo)

Thanks for your answers! :-)

it would be really tough to make. Even so hard to the extend, I think it is impossible.

Is the result a single MIDI channel with both left and right hand? then the result would be pretty ugly, and quite useless...

Is the resulting MIDI capable of splitting the audio input (2 midi channels, left and right hand separated ), then you shouldnt bother creating an app for it, just start a new company, and make money. Only a insanely ingenious algoritm (if at all) could split this. This splitting is a prerequisite for pianistic scores, but I think impossible to make.

Well I don't expect it would be easy at all but I'll do my best. I don't think the application would be capable of splitting the two hands as it just can't know, which hand played whoch notes. Maybe if both hands played in different octaves. Then they dould be split according to a pitch. Anyway, iy you gave such app a shot which mobile paltform would you prefer?

If you can create a MIDI track you can split it evenly at a certain pitch, that is easy. The hard part is getting it to actually hear everything properly. AVID has made a noble effort at a professional quality program aimed to do just this. AND, even with a $300 price tag and the backing of a major corporation this program is basically broken. Reading any real review of audioscore (i.e. one not from AVID or someone paid by them) will reveal just how useless the program is. Further, I can testify from personal experience that it is absolutely useless.

I don't believe you would be able to do any better on your own, especially in something as scaled down and simple as a phone app.

I tried AudioScore too and I have to admit this software sucks. But it tries to be complex, it tries to recognize multiple instruments. What I want to do is making simple application that recognizes just a piano from just a pinao sound. It wouldn't be capable of processing let's say an MP3 where the whole music band is playing. So I hope it wouldn't be impossible to create such an app:-) Anyway, what mobile platform would you prefer?

And what do other musicians thnk of this project? :-)

Maybe if both hands played in different octaves. Then they could be split according to a pitch. Anyway, if you gave such app a shot which mobile platform would you prefer?

this is my whole point. Yes, software can split a piece around middle C, but if the purpose is to be the middle-man in a process "to obtain the classical piano score" I have little to no hopes for such a project. Human readable scores do not have that kind of artificial split point. Ever.

I think musicians should rely on their own aural skills. An app to train those I might use (although I rather train my aural skills in a closed room with proper equipment, and not while I am on the move with a mobile device). An app to replace those I won't use. I just don't see the purpose.

If I wanted to have a score from some music I heared, I'll train my solvege skills, look for a score online, or buy one.

If your question is to see what kind of devices musicians use I would do a more general survey. I use iDevices, so that's my vote. But if you intent to serve a big as possible audience, why don't you develop a general (html5 or at least flash-free) website, that can be accessed from all types of devices (iphone, android, windows) as well as PC's (that could be there in the closed room that I rather use for my aural skills)

this is my whole point. Yes, software can split a piece around middle C, but if the purpose is to be the middle-man in a process "to obtain the classical piano score" I have little to no hopes for such a project. Human readable scores do not have that kind of artificial split point. Ever.

I think musicians should rely on their own aural skills. An app to train those I might use (although I rather train my aural skills in a closed room with proper equipment, and not while I am on the move with a mobile device). An app to replace those I won't use. I just don't see the purpose.

If I wanted to have a score from some music I heared, I'll train my solvege skills, look for a score online, or buy one.

If your question is to see what kind of devices musicians use I would do a more general survey. I use iDevices, so that's my vote. But if you intent to serve a big as possible audience, why don't you develop a general (html5 or at least flash-free) website, that can be accessed from all types of devices (iphone, android, windows) as well as PC's (that could be there in the closed room that I rather use for my aural skills)

Well you're absolutely right that the piano score should divide left and right hand. But the application is not almighty and it is not the replacement for musicians' aural skills. the main intention is to ease the process of transfering the idea into piano score. So as I see it, if you wanted to write your own composition into piano score, you could for instance play at first one hand alone and generate the MIDI output, then the other hand alone and generate thi MIDI output, then you cna mix it in the appropriate software. In this way you would be able to create a piano score much faster in comparison to manal writing in any software.

What do you think about this point of view?

Well you're absolutely right that the piano score should divide left and right hand. But the application is not almighty and it is not the replacement for musicians' aural skills. the main intention is to ease the process of transfering the idea into piano score. So as I see it, if you wanted to write your own composition into piano score, you could for instance play at first one hand alone and generate the MIDI output, then the other hand alone and generate thi MIDI output, then you cna mix it in the appropriate software. In this way you would be able to create a piano score much faster in comparison to manal writing in any software.

What do you think about this point of view?

If one needs this approach to compose, he'd rather use a MIDI keayboard...

If one needs this approach to compose, he'd rather use a MIDI keayboard...

That's another option :-) Not everyone has the MIDI keyboard though. It's the kind of thing the copmposers who consider themself professionals possess. On the other hand, almost everyone has a smartphone today. That's why I think the ordinary pianists might find this app handy :-)

It's the kind of thing the copmposers who consider themself professionals possess. On the other hand, almost everyone has a smartphone today.

I think anyone who can afford a smartphone should be able to afford a MIDI keyboard as an iPhone 5 costs many times that of a cheap MIDI keyboard.

However, as a professional software developer, I disagree with the others who say your project is practically impossible, because you've got one thing going for you: You can add a calibration feature where the users play each note on their piano several times at different velocities and you can use that information to find out what notes they're playing later. Unfortunately, because of resonance, 1 + 2 does not equal 3 when it comes to keys being played on a piano, however I'm willing to bet your app could still pick out the notes being played, because it already knows exactly what each note sounds like, which is a luxury you don't have when instead trying to process a recording made on a piano you don't have access to.

As for hand separation, that's a challenge, but people already came up with some pretty good solutions: https://docs.google....wr48bQkQmCuGI_w

Basically, it's possible to simulate hands and fingers and come up with optimal fingering and that of course also gives you hand split. If you did the same, I don't see why your project wouldn't work.

(Personally I'd just use two MIDI keyboards instead, one elevated just above the other and play the left hand part on one of them, so I can separate the hands in the output.)

I think anyone who can afford a smartphone should be able to afford a MIDI keyboard as an iPhone 5 costs many times that of a cheap MIDI keyboard.

However, as a professional software developer, I disagree with the others who say your project is practically impossible, because you've got one thing going for you: You can add a calibration feature where the users play each note on their piano several times at different velocities and you can use that information to find out what notes they're playing later. Unfortunately, because of resonance, 1 + 2 does not equal 3 when it comes to keys being played on a piano, however I'm willing to bet your app could still pick out the notes being played, because it already knows exactly what each note sounds like, which is a luxury you don't have when instead trying to process a recording made on a piano you don't have access to.

As for hand separation, that's a challenge, but people already came up with some pretty good solutions: https://docs.google....wr48bQkQmCuGI_w

Basically, it's possible to simulate hands and fingers and come up with optimal fingering and that of course also gives you hand split. If you did the same, I don't see why your project wouldn't work.

(Personally I'd just use two MIDI keyboards instead, one elevated just above the other and play the left hand part on one of them, so I can separate the hands in the output.)

Thanks for your reply it is very helpful. You're right about the calibration it is the essential thing for my application. As for the hands splitting I didn't know there were papers issued about this problem, I'll look into the method (using hidden markov models) you suggested and I might eventually use it :-)