Flushing read buffer on Open

[mrspirytus]

Hello,

I am using version 1.0 and noticed something I was hoping there is a built int solution? or if you can suggest a solution.

I have a GPS device sending data every 1Hz to ttyUSB0. I can process data just fine.

Problem:
When I start my test application I get sometimes a lot (what is queued up messages) after Open(). I do start reading data after open without any delay.
When I quit and restart my application right away I do not see any data queued up.
When I quit my test app and wait a minute and start I right away see a bunch of messages and then I get updates 1Hz expected.

Is there a way to "eat" all messages right after Open() so I do not read stale data and start processing the latest data? I did try to use _tty.FlushIOBuffers() but that did not help

  _tty.Open("/dev/ttyUSB1", std::ios_base::in);
  _tty.SetBaudRate(baud);
  _tty.SetCharacterSize(charSize);
  _tty.SetFlowControl(flow);
  _tty.SetParity(parity);
  _tty.SetStopBits(stopBits);
  
  cout << _serial.GetNumberOfBytesAvailable() << std::Endl;
  _serial.FlushIOBuffers();
  cout << _serial.GetNumberOfBytesAvailable() << std::Endl;

  std::string nmeaString;

  while(true) {
        try {
            // Read a single byte of data from the serial ports.
            // 25ms timout
            nmeaString.clear();
            _tty.ReadLine(nmeaString, '\r', 25);
            std::cout << nmeaString << std::endl;

        }
        catch(const ReadTimeout&) {
            std::cout << "Timeout" << std::endl;
        }
  }

I get the value of 0 for the above prints, but when I got to reading I get a bunch of then and then to 1Hz update. Is FlushIOBuffers() the right call to use? If not what is the best way to handle it and what is the purpose of FlushIOBuffers() since we are on this subject as well :)

[mrspirytus]

So I modified the above code by adding a function to "eat" any messages at the start.
If I do not run this program for 3-5 min and start, I got
"Read Stale Data: 456" on the first run. As the program tries to read 20 messages per run()
each time the next run() is executed, I get "Read Stale Data: 1"

There has to be a better way then random 500 counter in my example.

void readStaleData(SerialPort& port)
{
    int         count(1);
    std::string nmea;

    while(count < 500) {
        try {
            port.ReadLine(nmea, '\r', 25);
        }
        catch(const ReadTimeout&) {
            std::cout << "Read Stale Data: " << std::to_string(count) << std::endl;
            break;
        }

        ++count;
    }
}

void run() {
  _tty.Open("/dev/ttyUSB1", std::ios_base::in);
  _tty.SetBaudRate(baud);
  _tty.SetCharacterSize(charSize);
  _tty.SetFlowControl(flow);
  _tty.SetParity(parity);
  _tty.SetStopBits(stopBits);
  readStaleData(_tty);

  std::string nmeaString;
 
  int idx(20);

  while(--idx) {
        try {
            // Read a single byte of data from the serial ports.
            // 25ms timout
            nmeaString.clear();
            _tty.ReadLine(nmeaString, '\r', 25);
            std::cout << nmeaString << std::endl;

        }
        catch(const ReadTimeout&) {
            std::cout << "Timeout" << std::endl;
        }
    }
   _tty.Close();
}

int main(argc i, char* argv[]) 
{
  int idx(100);

  while(--idx)
      run();

  return 0;
}

[mcsauder]

Hi @mrspirytus , there are indeed a few methods in addition to what you have identified to flush the hardware buffers, although I'm not quite sure why calling FlushIOBuffers() isn't working straight away for you. I wonder if you might need add a sleep() for a few milliseconds or tens of milliseconds to allow that command to physically complete. To start, perhaps try calling FlushInputBuffer() in your program and immediately follow with a sleep() at an appropriate execution point to empty data that is already filling the hardware input buffer. Let us know what effect this has and I'll try to look at it more over the weekend as well.

        /**
         * @brief Flushes the serial port input buffer.
         */
        void FlushInputBuffer() ;

        /**
         * @brief Flushes the serial port output buffer.
         */
        void FlushOutputBuffer() ;

        /**
         * @brief Flushes the serial port input and output buffers.
         */
        void FlushIOBuffers() ;

[mcsauder]

Helllo again @mrspirytus , after you call Open() the buffers are already getting flushed. I'll see if I can try to catch this in the unit tests to replicate what you are seeing. What are you setting the flow control value to?

[mrspirytus]

I am going to try your suggestion in a min and get to you in a few. But so you know about your second comment. My GPS data is coming at 1Hz. When I open same port with GTKSerial and moserial app I get the same behavior. as your library. And, based on GPS data being publish 1Hz if I wait 10sec and start program I get 10 messages right away and so on. I am not sure what the upper limit is, but based on my earlier test trying to eat them on start I did see as much as ~470

[mrspirytus]

Ok, so I added this line to my last example

    _serial.SetStopBits(stopBits);
    
    _serial.FlushIOBuffers();
    std::this_thread::sleep_for(std::chrono::milliseconds(1000));

    readStaleData();

So I still see readStaleData() getting about 77 messages before it times out

EDIT: Btw, I am testing on Ubuntu 20.04 (Linux spirytus 5.4.0-52-generic 57-Ubuntu)

Here are parameters I am using

        void    connect(const std::string tty = "/dev/ttyUSB0",
                        BaudRate baud = BaudRate::BAUD_115200,
                        CharacterSize charSize = CharacterSize::CHAR_SIZE_8,
                        FlowControl flow =FlowControl::FLOW_CONTROL_NONE,
                        Parity parity = Parity::PARITY_NONE,
                        StopBits stopBits = StopBits::STOP_BITS_1);

[mrspirytus]

Hello @mcsauder

I also found another strange issue. Not sure if you want me to open a new ticket. Let me know.
With the above sample, if I increase the timeout value from 25 (also you are using this in your samples) to 1000, I get more errors. Meaning the NMEA strings are "copped" in between.
So in run() I change
_tty.ReadLine(nmeaString, '\r', 25);
to
_tty.ReadLine(nmeaString, '\r', 1000);

nmeaString are more often than with 25 I get partial strings which I have to drop. This is pretty consistent between these two values.

Here is pgk version I am using

libserial-dev/focal,now 1.0.0-4 amd64 [installed]
  Serial port programming in C++ -- development files

[mcsauder]

EDIT: @mrspirytus , I misread what your wrote above. Can you check that '\r' and not '\n' or '\r\n' is the line ending you are actually receiving?

https://en.wikipedia.org/wiki/Newline

[mrspirytus]

@mcsauder I am little puzzled by your comment. I am using '\r'.
Are you saying I should use '\n'? I can not use '\r\n' since that is a two characters :)
So I understand what you referring to since I posted few issues/concerns here.

Should this make it better with my solution to "eating" stale data on start?
Should it make a difference related to my observation to large timeout value?
I will try right now '\n' with FlushIOBuffers() followed with sleep() and another run with my readStaleData(). Will let you know in a few min

[mrspirytus]

@mcsauder Ok so I did the first run. I replaced \r with \n, commented out readStaleData(), and only left Flush and sleep(500) per your earlier suggestion. Still, if I start up I get stale data. I know it is stale because if I look at each message, the timestamp on it changes. Just saying since you might think this is some other weird issue and I am reading the "same" message. I will try not to use readStaleData() and wait for your further instructions. Can you reproduce this on your system?

[mcsauder]

Hi @mrspirytus , I'm still not sure what's going on with your setup. There is a unit test in place to test flushing the input buffer,

libserial/test/SerialPortUnitTests.cpp

Line 233 in 1d1e47a

SerialPortUnitTests::testSerialPortFlushInputBuffer()

, and I'm not getting it to fail on my Ubuntu 20.04 setup. Are you able to use an oscilloscope or signal analyzer to determine if the data is being transmitted by your device before or after your application starts?

Let us know if you have any other thoughts or details that might be useful.

[mrspirytus]

Yup, I have a scope with logic. Just so you know. The module I am using is a USB to the serial version. I did enable (programmed) the module to start sending GPS points when it is powered on.
Is this is what you are asking? If so then, yes it is automatically sending GPS without explicitly asking on connecting to it with libserial. Let me know if I answered your question or let me know what specifically you want me to do or test.

[mcsauder]

Cool. What I want to know is if it's been sending data the whole time uninterrupted, or if there is a burst of data when the port is opened. If there is no change in the data send rate when the port is opened, then we know the hardware buffer is not obeying the posix command to flush.

[mcsauder]

Hi @mrspirytus , were you able to determine when data is being transmitted per my comment above? Is this still an issue for you?

[mrspirytus]

Opps, sorry I dropped the ball. I got a work around to read bunch of messages and dropping them until first timeout. That seems to work well. Of course not optimal solution but I had to move on to other tasks. I should revisit this again. Do you have any suggestions what and where I should look at? Just wondering.

[mcsauder]

I seems like your hardware isn't honoring the posix command and is not actually flushing the buffer to me. Without knowing your hardware setup it is tough to know, however. If you have an oscilloscope, just watch the data lines to verify that your device is just sending data at its' given rate. If that is the case your hardware buffer isn't actually flushing when being commanded to do so by the OS.

[mrspirytus]

I am using SerialDevice. I have same behavior on rPI4 Ubuntu server 20.04 and NVIDA Nano 18.04. I believe I had same issue on my workstation also running 20.04.
Yes I have scope. So, are you suggesting to watch TX even if port in not open using SerialDevice?

[mcsauder]

There is no class named SerialDevice in libserial. Can you clarify?

[mrspirytus]

Correction: SerialPort

[mrspirytus]

Update. I will be working on a different project so I can revisit this again. In this case, it will be reading binary vs ASCII data. But I can look with scope to original issue

[mrspirytus]

@mcsauder I will start looking at this issue. Just letting you know I did not forget :)
But I am tracking some other issue that I am not sure if this is your library. Let me know if you want me to open a new ticket for this.

I am reading from GPS device (USB to serial). Each NMEA string is terminated with "\r\n". Once in a while, I am getting a garbage line. Looks like one-two incomplete NMEA strings on top of each other. The strange thing is that I used moserial app and I do see the same behavior once a while. The strange this is that with minicom I do not? Hmm. I am capturing all data using minicom -C to file and will look closer. The only change in the initial source example above is

_tty.ReadLine(nmeaString, '\n', 500);

To read on proper message boundaries.

Let me know if you ever experienced this and if you want me to open a new ticket. I will post later on my findings of minicom recordings.

[mrspirytus]

So I captured 56304 GPS points using minicom and no issue.

[mcsauder]

Hmmmm. It's possible that libserial is causing an issue, but it's also possible that minicom is not reporting one.... I'll see what I can replicate. Would you be willing to try again with picocom?

[mrspirytus]

I just installed it and running test. Will let you know soon
EDIT: Should I run with any special flags? -r -l?

picocom -b 115200 -r -l /dev/ttyUSB1

[mrspirytus]

So I collected 44132 NMEA strings

picocom -b 115200 -r -l /dev/ttyUSB1 | tee nmea.log
Next I checked if first lines start with '$'
cat nmea.log | cut -c-1 | uniq
This produced 23 other items not counting '$'

[mrspirytus]

I am still looking and testing...