WiFiUDP Packet Loss
Hello, I am having issues with WiFiUDP silently abandoning my packets.
Summary of the system
- Board: ESP32s3 (Fake freenove style one with camera support)
- IDE: Arduino IDE
My program has three responsibilities:
- (1.) Send a camera feed to the server. (640x320 rgb, jpeg encoded, aim for 10-15fps)
- (2.) Send control signals to three SG90 servos based on incoming instructions.
- (3.) Listen for instructions from the server (Doesn't really matter how, currently using websockets)
Each responsibility has its own task.
From the above: (1.) is on core 1; (2.) is also core 1; and (3.) is on core 0.
WiFi and system events are also on core 0.
Order of priority for timing accuracy is (1.) > (2.) > (3.).
The Problem
1. When calling udp.endPacket(), it frequently gives a return value of 0.
2. This happens when WiFiUDP has silently failed to transmit the packet.
3. UDP is connectionless, so (AFAIK) this means the packet isn't getting lost in the network - it's being dropped before it ever makes it out of the memory buffer.
4. Verbose logging doesn't tell me much.
*
[ 42802][E][NetworkUdp.cpp:255] endPacket(): could not send data: 12
5. The behaviour is quite random.
* Sometimes, the program works great and packet are rarely dropped, if ever.
* Other times, the program drops virtually every packet from the outset after boot.
* The longer the program runs, the more frequently packets are dropped.
6. Dropping occasional packets is fine, but the issue is when the program is dropping almost every packet.
The code
I've temporarily disabled the websockets (incoming instructions) task just to test the camera feed.25 Replies
Verbose logging doesn't tell me much. [ 42802][E][NetworkUdp.cpp:255] endPacket(): could not send data: 12That is produced at this site https://github.com/espressif/arduino-esp32/blob/0a45a0614244002f82fe7f006effeda7c8075469/libraries/Network/src/NetworkUdp.cpp#L253-L256 And
errno = 12 means "Cannot allocate memory"
PSRAM is enabled?You should have 8MB PSRAM on board=Here's some example logs.
FPS = Frames acquired, not necessary sent.
TT = Total time per frame.
ND = Number of Dropped (or rather, how many frames were abandoned due to packet loss).
NR = Number of Retries - Ignore this, I stopped using it.
When its working good:
Note how ND is almost always 0
When its working bad:
Note how ND is almost as high as the acquired FPS, meaning almost all of the frames failed to get sent.
But what could cause a "cannot allocate memory" state based on my code?
I believe so - I'm using it for the OV2640.
On boot I make these logs:
The first heap log is the very start of setup()
The last heap log is after I initialised WiFi and all the tasks.
Heap usage is very very low.
So I'm confused why "cannot allocate memory" is happening?
LWIP simbly uses malloc(): https://github.com/espressif/esp-idf/blob/master/components/lwip/port/include/lwipopts.h#L93-L98. By default
CONFIG_SPIRAM_USE_MALLOC should be on for Arduino-ESP32 so it should allocate from there.
346k is very low, that can't be PSRAM. Is it enabled in the Arduino IDE -> tools setting or implicitly via the selected Board?Sorry, why do we care about PSRAM?
Under Tools > PSRAM, I have selected
OPI PSRAM
I'm using the PSRAM for my camera frame buffer.
But by default, wifi uses DRAM - of which my tasks only use 20% of so no issues with overuse or fragmentation?
https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/lwip.html#limitations
that's exactly what you're using, a failure in
sendto() with ENOMEMOh cool yeah that looks relevant
But I already do exactly what it suggests - I detect dropped packets and implement a short delay.
This does not sufficiently kick the program out of being stuck in a constant cycle of dropped packets?
Then the delay is not short enough to give the WiFi / driver stack enough time to flush its buffers?
Eh, I really don't think time matters
In fact, if I raise the wait times, performance actually seems to get worse
The doc you linked also says:
Increasing the number of TX buffers in the Wi-Fi or Ethernet project configuration as applicable may also help.Is this possible in the arduino implementation? I have so much DRAM spare
https://github.com/espressif/esp32-arduino-lib-builder/blob/53a318f5b07d52156e38729719fed9b4ef2a1ad1/configs/defconfig.common#L26-L27
Arduino-ESP32 sets it to a static configuration of 8
If PSRAM is enabled, "Static" should be selected to guarantee enough WiFi TX buffers. If PSRAM is disabled, "Dynamic" should be selected to improve the utilization of RAM.https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/kconfig.html#config-esp-wifi-tx-buffer
Well I have psram enabled but I don't think I want wifi using it?
Or are you saying I should?
Well if it's failing due to out of memory and it only has 8 buffers then yes it should use more RAM
From debugging, I know that one of my frames gets split into around 21 packets, each with approx 1kb written in.
So Does that mean I want at least 21 buffers?
But is there a reason it should be PSRAM instead of DRAM?
I mean, DRAM is faster than the PSRAM which might be good for the WiFi stack
I would try to jack up the number of static buffers or switch it to dynamic, see how it behaves
you don't recompile these statically set ESP-IDF parameters in the Arduino IDE though. You'd have to use ESP-IDF itself with "Arduino as an ESP-IDF component" or do it via e.g. PlatformIO (https://github.com/pioarduino/platform-espressif32/tree/main/examples/espidf-arduino-blink>)
I tried using ESP-IDF last week for the first time.
Took ages to install all the dependencies, and then it failed anyway. So I gave up on it lol
I don't really care what IDE I use though, so long as it works
Yes ESP-IDF is a big and slow to compile mess. But that's what Arduino-ESP32 is sadly based on
What is platformIO?
A build system written in Python based on SCons. It's also exposed as a handy VSCode extension.
In essence, you can install the PlatformIO extension in your VSCode, install the "pioarduino" platform from above, then import the espidf-arduino-blink example that has that special "Arduinp as ESP-IDF component" setup
from there you can then modify the ESP-IDF setting, aka KConfig, for
CONFIG_ESP_WIFI_STATIC_TX_BUFFER or CONFIG_ESP_WIFI_DYNAMIC_TX_BUFFER and CONFIG_ESP_WIFI_STATIC_TX_BUFFER_NUM=8 respectivelyHmm ok its worth a try at least.
But I'm guessing my code will need a huge refactoring to get rid of the arduino style cpp?
No, it can be pasted 1:1
all you're doing is building ESP-IDF from source with a different config and putting Arduino-ESP32 on top
your source files remain the same
Oh that's convenient
Is platformio going to be easier to setup than esp-idf was? lol
That was a real nightmare...
PlatformIO is in between the Arduino IDE and native ESP-IDF in terms of complexity.
It downloads all toolchains and framework code for you, so you don't have to do a lot
just once installing the "pioarduino" platform so that you can use their latest ESP-IDF + Arduino examples. But that's documented in their repo.
I see
I've gotta head out now so I'll try setting that up tomorrow.
Thank you very much for taking the time to help and explain 🙂
I'm joining this conversation cus I got a r4 yesterday and played around with WebsocketsClient trying to send string data to a server, but getting the same behaviour as you, packet loss and it also crashes the server.
Did you find a solution to this?