sciencetech-blog

comments

Next time you hold a secret conversation in a secure room, you might want to make sure there aren't any crisp packets or plants in the line of sight of a window.

Scientists have found that using a high speed camera, the vibrations of seemingly inanimate objects like these can actually 'record' sounds in a room.

The incredible technique can be used to identify music, speech and more just by filming objects and monitoring their tiny vibrations.

Scroll down for video 

Research led by the Massachusetts Institute of Technology has shown that sounds can be identified by monitoring tiny vibrations of objects with a high speed camera. In one experiment the researchers identify words being spoken by filming a crisp packet through soundproof glass (shown)

The research was carried out at the Massachusetts Institute of Technology (MIT) in partnership with Microsoft and Adobe.

The team developed an algorithm that can reconstruct an audio signal by analysing tiny vibrations of objects.

For example, in one experiment the scientists were able to recover intelligible speech from the vibrations of a crisp packet filmed from 15 feet (4.6 metres) away through soundproof glass.

In another experiment, the researchers show how filming the vibrations encountered by a plant's leaves enabled them to identify that a song playing nearby was 'Mary had a little lamb.'

'When sound hits an object, it causes the object to vibrate,' says Abe Davis, a graduate student in electrical engineering and computer science at MIT and first author on the new paper.

'The motion of this vibration creates a very subtle visual signal that's usually invisible to the naked eye.

'People didn't realise that this information was there.'

The algorithm developed by the researchers is able to analyse the motions of an object as a whole when it's struck by sound waves.

As different edges move in different directions, the algorithm aligns all the measurements so that they don't cancel each other out.

The result is that it can then identify signals of different frequencies and play them back.

This means speech and music can be picked out and identified from the vibrations alone. 

The Visual Microphone: Passive Recovery of Sound from Video

The team used a high speed camera filming at thousands of frames per second and found nanoscale vibrations corresponded to sounds that had been played. This meant audio such as music and speech could be identified from video. In the above experiment a song was recorded in the vibrations of plant leaves

'We're scientists, and sometimes we watch these movies, like James Bond, and we think, "This is Hollywood theatrics. It's not possible to do that. This is ridiculous,"' says Alexei Efros, an associate professor of electrical engineering and computer science at the University of California at Berkeley.

'And suddenly, there you have it. This is totally out of some Hollywood thriller.

'You know that the killer has admitted his guilt because there's surveillance footage of his potato chip bag vibrating.'

While the researchers' technique has obvious applications in law enforcement and forensics, Davis is also enthusiastic about the possibility of what he describes as a 'new kind of imaging'.

In ongoing work the researchers are trying to determine the shape and structure of objects from just their visible responses to short bursts of sound.

'We're recovering sounds from objects,' he says.

'That gives us a lot of information about the sound that's going on around the object, but it also gives us a lot of information about the object itself, because different objects are going to respond to sound in different ways.'

To clearly reconstruct audio from a video, the frequency of the video samples - that is the number of frames captured in the video per second - needs to be higher than the frequency of the audio signal. The song 'Mary had a little lamb' was used in a number of experiments by the researchers