Are there plans to make WLED audio-reactive on itself without using external hardware/software like LedFx? A modern ESP32 should have enough horsepower and time left to do the audio processing in the times when it’s waiting for the next update cycle.
2 Likes
Hi @apleschu,
yes, I have just got some microphone modules for ESP32 / 8266 from AliExpress and will test to modify effects with audio input.
But I do not have a timeline planned for this.
5 Likes
it would be great to have such function included.
Would it be wrong of me to assume the microphone input would be a simple analog input on an 8266 (or esp32), polled every update for a “VU” meter type effect (for a 1-D effect anyway).
The strip illuminates primary color proportional to “100%” mark around 3/4 of total strip length, calibrated to some analog value (or hard-coded). The strip’s top 1/4 would be reserved for over-driving of the sound above acceptable levels, and perhaps display RED (the traditional over-driving VU meter color.
Am I thinking correctly?
If there are multiple strips working together, perhaps a fancy 2-D visualization could be done like they do at some concerts or some software music players.
However, that’s likely raspberry pi and up to do that because of the processing power required to perform the FFTs for frequency analysis (or a bunch of filters and analog inputs).
1 Like
Hi @huggy-d1,
yes - kind of “yes”, because the microphone needs to be a special one to get good results. A MEMS does work well here. Doing a FFT ananlysis will be tough,because of the time it consumes.
It could result in bad Wifi or the whach dog timer could be pulled (that would result in a reset).
But instead of doing a frequency analysis doing just a linear smoothing could be enough - this can be integrated in the loop with only small computing powers.
1 Like
The way I understand it now is: One core is working the network, while the other core is working the “foreground”, handling the less.
So, I wonder: It should be possible to put the FFT and the network onto the same core. The FFT will come up and have air every 50ms or so and then could handle network traffic. While there is a sound reactive effect being displayed I would argue that we can work with slightly delayed network reaction. Especially since once we have sound reactive effects the NEED to be Jonny on the spot for network kind of diminishes. We should not let the perfect be the enemy of the good.
Wouldn’t offloading the processing to a phone or computer and sending it over bluetooth or wifi be easier on the eps8266?
@apleschu, if you use an ESP32 this is true - the one core does WiFi, etc. while the other can do tasks in the loop. But with an ESP8266 this is not true, because it only has one core in total.
@nagmier, WLED does support this (offline processing) right now: you only need a software that creates E1.31 streams - normaly these programs run on a PC …
I would be OK to put another ESP32 into the mix and transfer the audio/FFT data from the audio processor to the display processor via the network. What I abhor is a solution with a PC in the middle. most/all of the solutions I am aware of are either horribly unstable or like LedFX have taken a wrong turn somewhere in the development and are using interpretative languages wasting lots of processor time in the interpreter.
So with that in mind and I understand the wish to be inclusive of the 8266 - I believe a second processor, transferring data to the display processor would scratch that itch
1 Like
For that you would definitely need at least an ESP32 so that FFT does work in parallel to WiFi 
Actually, I really meant a second chip, communicating to the display processor via the network. As I said, I would not have a problem dedicating a second ESP32 or 8266 to do the FFT and then transmit the data over to the display processor
Hi @apleschu, yes, me too - buit this second chip must be an ESP32, too (I think so, because FFT are somehow computing intensive so that WiFi could drop and then the results could not been sent to the other chip that displays this).
LOL, no problem from me again. There is a requirement to do real time sensitive work. And for me as an old industrial programmer the old saying still holds: If you need REAL real time, you just have to commit to spend one core per real time problem. Its really quite easy.
I do not believe in cobbled together solutions which only work when they feel like. Which is incidentally the reason why I don’t like the existing python solutions.
2 Likes
Hi,
I’ve got sound reactive code running on ESP8266/ESP32 and am working with apleschu who’s going to be adding ESP32 based FFT functionality. An early video of that running is at:
I tried in vain to get arduinoFFT() running on an ESP8266, however the WLED overhead made that a non-starter.
I merged my code into Johnny’s repository and the first FFT audio effect (limited to ESP32) is implemented and works beautifully.
The way Johnny and I have this implemented is that the FFT code runs on core0, while all the rest of WLED code runs on core1. I cannot see any problems with network response times or any other problems. What we had to do in order to avoid triggering the core watchdog is to insert a delay(1) into the FFT code in order to give other tasks enough time to do housekeeping.
At the same time we are currently fighting a slight problem which has little to do with our code but is more a restriction in general of WLED, because WLED has only two sliders to change parameters(Intensity and Speed) This is an acceptable solution for effects that are not audio driven, but with audio/FFT driven effects in order to make this really work for all available LED strip lengths we would need at least one, but better two additional sliders.
I think I speak for both of us that we would love to see our code being merged back into the main branch of WLED, now that it is in a workable, compilable and decent shape. If that happens I would ask at the same time for the two mentioned extra sliders. I can make that work for my projects, because I know how long my strips are, but then those effects are not as useful to everybody else.
For the audio input I added a microphone breakout connected to GPIO 36 and defined an additional thread to be executed on core 0 (you can see that code in userSetup() in wled06_usermod.ino and the FFT code for that towards the bottom of this file.
The first FFT audio effect is in mode_asound10() towards the bottom of FX.cpp
Please do NOT delete the delay(1) statements they really have a function in this context.
The repository to clone is at : https://github.com/atuline/WLED
4 Likes
It’s nice to be working with someone who’s got some excellent DSP chops and can get the FFT functionality running on core 0 of the ESP32, while still being able to run WLED on the other. Coming from the Nano and more recently the ESP8266 world, this is still PFM (pure fucking magic) to me.
1 Like
Well, now we just need to be able to rope someone in to change the UI code for us. AS we have both found out the whole HTML/JSON/Javascipt is PFM to me as well.
I am certain we could make 5 extra sliders and find a way to make it known to the users of the software which function uses which sliders and in which capacity.
Is there anybody out there volunteering to change the UI to include 5 extra sliders so that we can accommodate all future functions? Bonus points if those sliders can be hidden and made visible from the audio effects routines. But those are really only bonus points.
Thank you everyone for creating such an amazing product in the first place that gives me/us a terrific jump off point to implement things that would otherwise much more complicated.
2 Likes
Well, I figured out and documented how to add another slider and passed that along to apleschu.
Thanks guys for adding this. I’m using Johnny’s repository on 8266 with line out from a stereo into A0 and it works perfectly. I’ve been running this solid for several days now with no hiccups.
Wow, it’s merged. I know it was a lot of work but thanks all involved to merge the .10 commits over. I was able to successfully get the sound reactive branch uploaded to ESP32 dev with Platformio. I haven’t tested the FFT but the regular sound reactive works perfectly with line in from a headphone jack.
Thanks again for all the work to make this happen.