How much will it cost?

SoundScape will only cost $60. It is printed with inexpensive but flexible TPU filament which can be scaled to an injection mold for mass-production. Other parts, like the Raspberry Pi and Teensy 4.0, total to under $45.

What sounds can it classify and how many?

SoundScape can classify 50 sounds as well as recognize speech. These include emergency sounds, such as a baby crying or a car's engine, to common sounds like rain or footsteps. The user can choose specific sounds to receive a notification for.

Do you plan to use a server going forward?

By using bluetooth streaming to transmit audio from the microphone array to the phone, we can run computational tasks on a phone. We can compile our computer vision models on CoreML to utilize an iPhone's neural engine.

How does background noise filtering work?

Noise reduction is done by first recording a 0.5 second audio clip from an initial interval of 10 seconds. Then we use fourier transforms to extract the frequencies from the background noise to produce a mask. Then we use the mask to filter those frequencies from the signal audio.

How have you accounted for different environments?

We do this in two ways: background noise reduction and data augmentation. Dynamically reducing background noise allows us to adapt to a variety of environments. Data augmentation allows our classifier to account for outdoors environments.

Does your system work in real-time?

Yes - the entire system has a latency under 0.2 seconds. We do this with GPU acceleration using CuPy and classifying audio cumulatively, similar to how voice assistants like Siri work.

Why do you do sound classification?

People who are deaf and hard of hearing rely on their eyes to identify what made a sound. However, sound sources are often out of sight, such as a baby crying or a car incoming behind you. In such cases, our users need to differentiate between "emergency" sounds - cars, babies - and less urgent ones, like wind or rain.

Where and how have you tested this?

We've tested indoors, outdoors, and with varying degrees of background noise. We found that indoors, with constant background noise at 20 dB, we could localize sounds up to 35 ft away.

In your motor pairs, how did you decide when to turn on the red or green LED?

In each pair the red LED is for low frequencies and the green for high. We determined the threshold frequency by calculating the average median frequency for each audio sample in the ESC-50 dataset.

What made you choose the four base models?

We chose the models based on three criteria: accuracy, run-time, and diversity in terms of classification methods. Our goal was to incorporate each model into the meta-classifier, so having variation in terms of how they classify audio data increased the accuracy of the meta-classifier. The 4 models we chose, Densenet, Resnet18, Cnn10, and Talnet, had some of the highest accuracies on other sound classification datasets such as ESCS10 and the Urban8k dataset.

How do you localize multiple sound sources?

Each microphone has a set time difference of arrival which offsets to the generated audio wave. So for every audio signal, the microphone array offsets the sound wave such that the sound waves generated by every mic are in sync for the same sound source. This applies when multiple audio signals coming from different directions are added. So if the same audio signal from one mic is detected on another, the offset sound waves should almost entirely be in sync. This allows the SRP model to filter out repeating sound sources.

How do you track sound sources?

Using localization data from the SRP-PHAT-HSDA model, we know the direction of arrival for each sound source. We then determine if the sound source is one of four states: static, constant velocity, accelerating, or a new source. Then we use a 3D Kalman filter, called M3K, to accurately track the sound source over time.