[Bug]: Some log lost during the place_marker_to_syslog in loganalyzer

[nhe-NV]

Issue Description

In the place_marker_to_syslog, when the flush_rsyslogd() is called, sometimes it will have some log lost since "sudo kill -HUP {}".format(pid)"

This is a known issue This is a known issue: https://serverfault.com/questions/813871/proper-way-to-reload-rsyslog-with-a-config-file-for-a-custom-application

def place_marker_to_syslog(self, marker):
        syslogger = self.init_sys_logger()
        syslogger.info(marker)
        syslogger.info('\n')
        self.flush_rsyslogd()


def flush_rsyslogd(self):
        '''
        @summary: flush all remaining buffer in rsyslogd to disk

        rsyslog.service - System Logging Service
            Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
            Active: active (running) since Fri 2021-09-03 15:50:17 UTC; 14h ago
        TriggeredBy: syslog.socket
            Docs: man:rsyslogd(8)
             https://www.rsyslog.com/doc/
         Main PID: 17516 (rsyslogd)
         Tasks: 5 (limit: 3402)
         Memory: 3.3M
         CGroup: /system.slice/rsyslog.service
              17516 /usr/sbin/rsyslogd -n
        '''
        pid = None
        out = str(subprocess.check_output("systemctl status rsyslog", shell=True))
        for msgs in out.split('\n'):
            m = re.search(re_rsyslog_pid, msgs)
            if m:
                pid = int(m.group(1))
                break

        if pid:
            os.system("sudo kill -HUP {}".format(pid))

Results you see

Sometimes, there will be log lost when run loganalyzer

When run place_marker_to_syslog, sometimes there will be timeout exception:
[ INFO] (proxy_ssh_engine.py:622) Traceback (most recent call last):

File "/tmp/loganalyzer.py", line 870, in

main(sys.argv[1:])

File "/tmp/loganalyzer.py", line 851, in main

analyzer.place_marker(

File "/tmp/loganalyzer.py", line 262, in place_marker

raise RuntimeError(

RuntimeError: cannot find marker end-LogAnalyzer-test_check_errors_in_log_during_deploy_sonic_image in /var/log/syslog

The timeout is due to could not find the mark in the syslog.

Results you expected to see

place_marker_to_syslog could always pass and No log lost

Is it platform specific

generic

Relevant log output

No response

Output of show version

it could happen to any version

Attach files (if any)

No response

[yxieca]

@nhe-NV can you add more specific scenarios to the "Results you see" section? I suspect that 'failed to find the start marker' is one of them?

[nhe-NV]

@nhe-NV can you add more specific scenarios to the "Results you see" section? I suspect that 'failed to find the start marker' is one of them?

Updated