Gyrophone: Recognizing Speech from Gyroscope Signals

My advisor Dan Boneh, colleague Gabi Nakibly and I have recently published a paper “Gyrophone: Recognizing Speech from Gyroscope Signals”.
It was presented at the 23rd USENIX Security conference in San Diego, and at
BlackHat Europe 2014 in Amsterdam.

To get a quick idea of what this research is about the following video should do:

We show that the MEMS gyroscopes found on modern smart phones are sufficiently sensitive to measure acoustic signals in the vicinity of the phone. The resulting signals contain only very low-frequency information (< 200 Hz). Nevertheless we show, using signal processing and machine learning, that this information is sufficient to identify speaker information and even parse speech. Since iOS and Android require no special permissions to access the gyro, our results show that apps and active web content that cannot access the microphone can nevertheless eavesdrop on speech in the vicinity of the phone.
This research attracted quite a bit of media attention and the first one to be published was an article in Wired.comThe Gyroscopes in Your Phone Could Let Apps Eavesdrop on Conversations. They interviewed us directly, and that article is probably the most technically accurate (Engadget and many more others followed and cited this original article).
We’ve been also addressed with some questions by a French journalist, which I answered quite in detail in an email. So to clarify certain points regarding this work I’m pasting the Q&A here:

Q: What is the best results that you get in term of recovering sound? What are the limit of your work until now? What sort of sounds couldn’t you recover? Could you recover a complete human conversation, for example ?
A: To be precise we currently do not recover the original sound in a way that it will be understandable to a human. We rather try to tell what was the original word (in our case digits) based on the gyroscope measurements. The fact that the recording is not comprehensible to a human ear doesn’t mean a machine cannot understand it, and that’s exactly what we do.
We managed to reach a recognition accuracy of 65% for a specific speaker, for a set of 11 different words, with a single mobile device, and 77% combining measurements from two devices. While that is far from full speech recognition the important point is that we can still identify potentially sensitive information in a conversation this way.
We also outline a direction for potential reconstruction of the original sound using multiple phones, but that requires further research, and we don’t claim yet whether it is possible or not with smartphone devices. Another, not less important result is the ability to identify the gender of the speaker, or to identify a particular speaker among a group of possible users of the mobile device.

Q: Why has nobody worked on and proposed this approach before? Is it because the technical tools (algorithm) weren’t available? I mean, what is really the most performance? What was the most difficult? The algorithm? What are the advantages in comparison to traditional microphones?
A: I’m not completely sure nobody hasn’t but definitely no prior work demonstrated the capabilities to this extent. The fact itself, that gyroscopes are susceptible to acoustic noise, was known. Manufacturers were aware of it but they didn’t look at it from a security point of view, but rather as an effect that might just add noise to the gyro measurements. We think there hasn’t been enough awareness regarding the possibility of sensing speech frequencies, and the security implications of it. In particular in smartphones, access to the gyro doesn’t require any permission from the user, which makes it a good candidate for leaking information, and a such, an interesting problem to look into. That is also the advantage compare to the regular microphone.
The hardest part apart from the idea itself was to adapt speech processing algorithms to work with the gyroscope signals and obtain results despite of the low sampling frequency and noise.

Q: What are the applications for Gyrophone that you imagine for the future? Spying?
A: The application can definitely be eavesdropping on specific words in a conversation, or knowing who is near the mobile device at a certain moment.

Q: What are the next steps in your work? I mean what is your next work now to progress in this direction? Do you plan to publish soon with new results?
A: The next steps in this direction would be to study better what are the limits of this attack: What physical range is possible? Can the recognition accuracy be improved?
Is there a way to synchronize two or more mobile devices to potentially recover sound?Currently we don’t plan to publish new results for this attack but rather exploring more ways to leak sensitive information from mobile phones by unexpected means.

Q: Do you imagine that in the future such system could be used by everybody? To do what?
A: It is not that easy to make such a system work for practical attacks on a large scale, although more research effort in this direction might yield more surprising results.

Q: How could we avoid this spying risk?
A: The attack is not so hard to prevent either by limiting the sampling rate available to applications, requiring specific permissions, or filtering the high frequencies.
Our hope is that the general issue of side-channel attacks will be addressed by mobile device manufacturers in way that will make it impossible.

The project page http://crypto.stanford.edu/gyrophone provides access to our code and dataset, as well as a link to the published paper.

Linux Audio Conference 2014

I’ve recently attended LAC’14, the 12th Linux Audio Conference, held this time at the ZKM (Zentrum fur Kunst und Media) in Karlsruhe, Germany. This conference is free and serves as a gathering opportunity for developers of Linux audio tools, experimental electronic music composers and open-source contributors.
I was presenting a contribution to the Faust musical signal processing language compiler. The main maintainers of the Faust project are currently Yann Orlarey and Stéphane Letz.
The joint work with Prof. Julius O. Smith (CCRMA at Stanford University) and Andrew Best (Blamsoft Inc.) added support for various useful features to the Faust VST architecture.
The title was 
Extending the Faust VST Architecture with Polyphony, Portamento and Pitch Bend.
I was initially introduced to Faust during a short workshop Yann gave in CCRMA in 2013.
Later on, while taking the Software for Sounds Synthesis class at CCRMA, I was trying to use Faust in combination with the MuLab DAW that can run on Windows and Mac OS X and supports VSTs as instrument plug-ins. Noticing the lack of some common features I’ve decided to turn it into my class project. The motivation was to enable using Faust to create plug-ins for as many free and commercial DAWs and production tools as possible and making the produced plug-ins functional enough to be useful for actual music production.
It was also exciting to meet open-source contributors for projects such as Ardour, QTractor and other great tools. Currently I’m continuing to work on improving Faust’s VST architecture, debugging and making it compatible with more production tools (like Ardour) out there. Next goal is solving some issues when trying to use Faust VSTs with Ableton Live.

Facilitating energy efficiency on mobile devices

I’ve recently read the Bartendr paper (386f5adade0bd7d0970b5ff555d643c6). It explores an algorithm for increased energy efficiency on mobile devices, based on prediction of good conditions for transmitting data to the cellular network.

Bartendr
The Bartendr  approach to communication scheduling is based on the fact that mobile devices consume more power when the signal is weak to compensate for the low SNR of the transmission channel. Applications should preferably communicate when the signal is strong which is indicative of a good transmission channel.
By anticipating good or bad channel conditions it is possible to determine the periods of time that are favorable for energy-saving communication. Since the signal strength is highly correlated with the user’s location relative to the cellular antennas, moving along a certain route the user experiences changes in signal strength. By learning the signal strength profile as a function of location along a certain route, it is possible to anticipate the time windows that are favorable for energy-saving communication.

While the algorithm was evaluated in simulation as well as on an actual device by incorporating the scheduling decisions into the test application, currently there is no practical framework for an actual implementation of Bartendr, or a similar scheme since it requires changes in the related software architecture. This post discusses several architectural changes that are needed in order to accommodate such schemes.

While it is possible for mobile applications to use timer mechanisms to run callbacks asynchronously at specific times, it requires the application to be aware of the right times to schedule the execution.
We can imagine how it would simplify things for the developer to simply push a synchronization request into a queue from which tasks are popped according to a power efficient schedule, maintained by the operating system. This could be a nice feature for a mobile operating system.
It would also be great to have something like a Strategy design pattern implemented for scheduling – allowing plugging-in a different schedule algorithm according to the user’s choice.

Bibliography

Dropbox Recovery Tools: When Dropbox client goes crazy…

It all started a couple of days before a paper submission deadline – when I really needed my data accessible, and my code available. I store all my academic stuff in my Dropbox folder, so it is backed up to the cloud. Lo and behold, Murphy entered the game and caused my Mac to halt, and when I forced a reboot it apparently resulted in some inconsistent state of the Dropbox client. This drove Dropbox  into deleting tens of thousands of files from my account, causing files to be deleted remotely and locally, under the tips of my typing fingers. The only thing I had is previous remote revisions of the files.
I don’t have Dropbox Pro and therefore don’t have PackRat, which anyway I’m not sure would help in this case. Besides, I didn’t actually care about indefinitely long history, but about quick and convenient recovery. I’ve decided to take a look at the Dropbox API. I found it quite easy, using the supplied Python Dropbox SDK, to write several tools to help me search and recover my deleted files.
The Dropbox Python SDK can be either downloaded and installed using setup.py or using

# pip install dropbox

In order to access your Dropbox you need to create a Dropbox application that’s linked to your account. It can be done through the Dropbox application console (https://www.dropbox.com/developers/apps) and gets you an application key and application secret which are later used by the Python scripts to obtain a token that enables accessing your Dropbox storage.
I store the application key and secret in a JSON configuration file, and written a tiny helper script to generate this JSON (make_config.sh):

echo "{
  "app_key" : "$1",
  "app_secret" : "$2"
}"

The code to obtain an access token for all further actions looks like that

import dropbox
def get_token():
  config = open(CONFIG_FILE, 'r')
  config_data = json.load(config)
  app_key = config_data['app_key']
  app_secret = config_data['app_secret']

  flow = dropbox.client.DropboxOAuth2FlowNoRedirect(app_key, app_secret)
  authorize_url = flow.start()

  # Have the user sign in and authorize this token
  authorize_url = flow.start()
  print '1. Go to: ' + authorize_url
  print '2. Click "Allow" (you might have to log in first)'
  print '3. Copy the authorization code.'
  code = raw_input("Enter the authorization code here: ").strip()

  # This will fail if the user enters an invalid authorization code
  access_token, user_id = flow.finish(code)
  return access_token

The access token is stored in a cookie file for later use.

def create_client():
  try:
    cookie_content = open(COOKIE_FILE, 'r').read()
    access_token = json.loads(cookie_content)
    client = dropbox.client.DropboxClient(access_token)
  except Exception:
    access_token = get_token()
    client = dropbox.client.DropboxClient(access_token)
    cookie_content = json.dumps(access_token)
    open(COOKIE_FILE, 'w').write(cookie_content)

Once we have initialized a client we have access to all kind of methods such as metadata, revisions,  restore, file_delete, etc. There are several utilities provided:
find_deleted – Finds deleted file under a given path (optionally by date)
ls – Lists files (optionally by date)
restore_file – Restores files. Paths are provided through standard input.
delete_files – Deletes files. Paths are provided through standard input.

Using these utilities I was able to restore all my accidentally deleted files in a matter of hours. One small technical detail I had to apply is to handle rate limiting imposed by Dropbox. Once in a while an operation would result in an exception if too many actions were attempted in a short amount of time. I had to catch this, and retry the operation.

The code is available under https://github.com/ymcrcat/DropboxRecoveryTools.

Registering a specific file-type handler on Android

(My example is in C# since I was working with Mono for Android but the same is applicable for Java applications as well via the usual manifest XML)

While many answers already exist on forums and several tutorials are available, I’ve still encountered a certain problem when I tried to register a handler for my own file type (let’s call it .xyz). I’ve added an IntentFilter attribute, specifying that I’m handling the View and Edit actions and choosing the Default category.
Since there is no existing MIME type for my extension I’ve used DataPathPattern for filtering and a wildcard for DataHost.
Overall the code looked like that

[IntentFilter(new[]{Intent.ActionView, Intent.ActionEdit},
Categories=new[]{Intent.CategoryDefault},
DataPathPattern=".*\.xyz",
DataHost="*")]
public class MyActivity : Activity { ...

The problem is that my application was now taking over all possible intents such as calling and messaging, and when I was browsing my contacts my application icon showed there as a possible candidate for handling the contact entry (i.e. making the call). What finally solved the problem was specifying the DataMimeType as “application/octet-stream”, like that

[IntentFilter(new[]{Intent.ActionView, Intent.ActionEdit},
Categories=new[]{Intent.CategoryDefault},
DataPathPattern=".*\.xyz",
DataHost="*", DataMimeType="application/octet-stream")]
public class MyActivity : Activity { ...

supost.com Chrome Extension

I’ve recently posted a Chrome extension for the benefit of the Stanford community. The SUPost Checker extension notifies the user about new items posted on supost.com. In addition it enables filtering according to user-specified keywords so in case the used is looking for specific items she can list all them in the extension settings and receive notifications about relevant posts. More info about the extension and a link to installation are on http://supost.voicebrowsing.net.

Towards (even more) practical Faust

I’ve recently completed my final project for the MUSIC 420B class in Stanford University. Prior to taking this class I happened to attend a workshop by Yann Orlarey, the author of the Faust musical signal processing language, that was held in CCRMA. I was fascinated by the ease and speed with which one could create effects and sound synthesizers using Faust. Also, having missed electronic music composition by that time I took the class that offered the possibility to play more with Faust and music production software. My initial goal was to demonstrate practical usage of Faust for music production in combination with software like Logic Pro or Ableton Live and write a whole piece using Faust generated sounds. VST in my opinion was the way to go because the tools that I wanted to use on Mac supported it. Although it was possible to create VSTi plug-ins with Faust, they were lacking some features that are expected from most synthesizers, with polyphony being the most noticeable of them. I’ve decided to take on this and fill this gap. The DSSI plug-in architecture supported by Faust already had support for polyphony so I could get an idea how to implement it for VSTi-s.

The following diagram describes in general the VSTi architecture design. The VST host interacts with the plug-in through the AudioEffectEx interface. Faust class implements this interface and using multiple instances of the Voice class supports polyphony. Each Voice class contains an instance of mydsp class which is produced by the Faust compiler and implements the signal processing/synthesis part.

Faust VSTi architecture design

Faust VSTi architecture design

Support for portamento slide was added by storing the last played voice in a dedicated member of the Faust class. In addition the architecture recognizes the “pitchbend” control as one that has to be updated according to MIDI pitch-bend event. The following short loop demonstrates how a whole piece can be produced using Faust instruments. All the instruments are Faust VSTi-s except for percussion that wasn’t fully implemented yet:

That’s how the bastard-synth VST used in this loop looks in MuTools’ MULab. Here we can see the controls recognized by the Faust architecture: freq, gain, gate, pitchbend and prevfreq.

pasted1

A more detailed project summary is in the following paper http://stanford.edu/~yanm2/files/mus420b.pdf. The project code is part of Faust source and can be check out from http://git.code.sf.net/p/faudiostream/code.

LED T-Shirt

During the last couple of months I’ve been working on a fun side-project with my friend Shlomoh Oseary. For a long time I wanted to make a T-shirt with an equalizer display on it that will light up in correspondence with surrounding sounds and music, and once I had a buddy excited about this idea too we started working.

We decided to use E-textile dedicated components. Arduino Lilypad with its 8 MHz Atmega processor seemed suitable for the task. Now we had to understand how will will drive the LEDs. The naive approach of connecting each LED to ground and to one of the Lilypad’s outputs would limit the number of LEDs we can drive this way. After searching a bit we found that what we want is to build a LED matrix. The principle in a LED matrix is that all the LEDs in the same row or column are connected. In our case all the minus legs of the LEDs in the same column are shorted and all the plus legs of the LEDs in the same row are shorted. To light up a LED we need to feed positive voltage to the corresponding row and short to ground the corresponding column. To light up multiple LEDs our LED matrix driver code  loops over all the rows and columns and constantly lights up each LED that is required to be turned on for a fraction of a second thus achieving the effect of those LED being constantly turned on.

Testing the microphone and the FFT calculation

Each column of the LED-matrix represents a frequency range with lower frequencies on the right. The more energy is sensed in a certain bin – the more LEDs in this column will be turned on. To find the energy for each frequency range we used FFT over a window of  128 samples. The sampling frequency was chosen to be 4000 Hz providing according to Nyquist theorem coverage for tones up to 2000 Hz. A predefined threshold (which we need to calibrate) is subtracted from the calculated energy to filter out small fluctuations and the outcome is mapped to the number of rows of the LED matrix to represent an energy level.
We used an existing FFT implementation for Arduino from http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1286718155.
There is still a final touch missing to the algorithm which is applying a low-pass filter to clean frequencies higher then 2000 Hz from the recorded signal prior to FFT calculation.

Connecting the electret microphone and the power supply to the Lilypad.

LED T-shirt @ work

When beauty and electronics meet… (Julia Shteingart modeling)

Code

Project’s code (except for FFT implementation which can be downloaded using the link above and the TimerOne library which can be downloaded from Arduino site) is available through SVN under

https://bitbucket.org/ymcrcat/led-t-shirt/

Credits

To Shlomoh’s mom for sewing.