Arduino Serial buffer overflowing/overwriten
Hi all, I'm new to arduino's technical stuff. I want to make a c++ program(app) that can take in files, then convert it into smaller chunks os bytes and send it to an arduino nano over the serial com ports and stuff over a cable. The arduino nano will then send it over nrf2l01 to another arduino that will be connected to another computer running another c++ program that will reconstruct all the byte chunks into the file.
currently, my question is, how do I make sure that my arduino nano's serial buffer will not overflow and lose data? Also how do I make a windows explorer "select file thing" simply (for the c++ app)?
Thank you for your time!
13 Replies
You could read the incoming data as it is received and store it in a byte array.
The Arduino Nano has only 2kb of SRAM, so the amount of data you can hold in volatile memory is limited.
You might need to pass the data along as it is received rather than storing it in memory if the file size is too large.
So your saying that I can increase the buffer from 64bytes(from google) into a max of 2k by using a byte array intead of the default buffer? Also. What do you mean passing the data along? You mean I can not use a buffer? Is there a way for the computer to know if the buffer is full?
In a way, yes. The buffer remains 64 bytes, you're just reading from it and storing what is read elsewhere.
By passing the data along I mean reading the data and immediately sending the data to the nrf2l01. So, there would only be a minimal buffer used to store the data currently being passed through.
I see..., yes, I do plan on sending the data as quickly as I can to the nrf2l01. How ever what I am afraid might happen is that
1. the computer sends data faster than the arduino can send to the nrf24l01 or
2. If some delay happen or if a package was failed to be sent, it will delay everything and could make the buffer fill up quickly. Is there a way to mitigate this? Maybe... is there a way for the arduino to tell the computer when it needs the next package data or when it needs to stop and wait for a moment?
@Mr Hypnotizd you use a ack package. The pc sends a package.
The arduino receives it and sends back the total byte count it received to the pc. If its corect the pc sends the next packet. If wrong the pc sends an error packet and resend the packet.
And yes this slows things down.
And yes this is how its done.
And yes there are many other ways of doing this. Im just giving one example of hunderds.
I assume I am to implement this ack package system manualy right? And also in this ack system, I can include some way to make it so that the arduino tells the computer if it's buffer is full right?
Oh yea one more thing, I read that the serial com could be delayed due to buffers and stuff(message efficiency thingy?), how to mitigate this problem? Since I don't want it to become a delay.
Anyways, thanks you replying @Maderdash
In case if you want faster speeds, you use an ESP32, which has around 500KB of SRAM and some boards have extendable/built in 8MB of PSRAM. This means a larger packet size and if the module is fast enough, can reduce the entire process time by a lot
Yes, right now speed itsnt really a big concern of mine since this is mostly a prove of concept, but I do want to make sure that if there is a delay due to interference or something like that, no data will be lost
Sorry, just saw the post. Yes, you will implement the acknowledgment (ack). If done correctly, the buffer will always remain empty, so there's no risk of overfilling.
Think of it like this: if you have a bucket that holds 1 gallon of water, and you only ever pour 0.8 gallons into it, the bucket will never overflow. The acknowledgment will always empty the bucket for you, so if you're receiving it, the bucket is already empty.
The acknowledgment won't allow anything to go into the bucket until it's empty. For instance, if data gets corrupted and you send 10 bytes, but the Arduino acknowledges receiving only 9, the computer sends a FAULT command (which you would create). This tells the Arduino there was an error and prepares it to receive the packet again, but to dump the current data. Let's say the PC tries this process 5 times (we'll set 5 tries as the "timeout"). If after 5 attempts the packet still can't sync properly, the PC sends an all-clear, dumps the packet on its side, refreshes its data, and creates a new packet to start the process over again.
[Note: This is what is referred to as a "packet drop" in gaming, streaming, or similar activities, often caused by corrupted packets or even Wi-Fi interference.]
Note all of this would be arojnd 30ms from first failed bpacket to last assuming 10 bytes.
Hello Mr Hypnotizd,
nice to meet you
I am software developer,
I can help you in some ways
I see, thanks for the explanation. The part when the PC dumps the oacket on its side and refreshes data, what does this mean? So in case the problem was on the PC's side, it recompute the packages and retry again?
Hello James Hui, thank you for wanting to help
correct. and all PC's do this.
Cellphones Litteraly everything that is "digital" operates under some kind of principle of this.
Thank you Maderdash and everyone for helping me with this. I will try this out and I will let you know what happen.😁
Oh yeah, I still want to know abt the serial com buffer, and message not being sent stuff problem tho.