After Upgrade to 5.13 HTTP/HTTPS Farms not working anymore. #123

Trylok · 2023-02-28T10:52:43Z

After starting a HTTP farm the farms list ist empty.

l4xnat farms are working but not displayed anymore.
Only stopping the corresponding Virtual Interface and rebooting the system brings the List back.
For testing i did a fresh iso install, only creating a new HTTP farm. With the same result.

HTTP/HTTPS Farms do not work anymore, no matter on upgrade or fresh install.

Thanks, Michael

Rendone-zevenet · 2023-03-01T11:45:25Z

Hi,
Could you attach a supportsave file from the system?

The HTTP/HTTPS Farms work as expected.
Maybe something in the environment where the ISO is installed is causing that behavior.

Thanks in advance!
Regards.

Trylok · 2023-03-01T13:45:03Z

Its a Hyper-V environment.
The supportsave is from version 5.12. Because i did a rollback. Hope this helps.
Here is the file:
supportsave_ZEVENET_20230301_1409.tar.gz

Thanks, Michael

Sysadminfromhell · 2023-03-01T14:18:01Z

Same issue here, I upgraded and all my backends seem to run into a Timeout most of the times.
Sometimes it works but mostly it doesnt.

Mar 1 15:15:02 HMLRPX02 zproxy: timeout request to backend 172.16.24.112:8080
Mar 1 15:15:02 HMLRPX02 zproxy: timeout request for client x.x.x.x:49186

It feels super weird. Can I go back to 5.12?
I do not have a snapshot :(

Trylok · 2023-03-01T14:23:49Z

Just for fun i installed version 5.13 manually on an fresh debian 11 (bullseye).
Webgui does not work out of the box. But after manually installing zevenet-gui-ce the webgui works fine.
However, the problem remains the same.
On the dashboard i get a endless rotating circle after starting http farms.

Here the supportsave from this version.
supportsave_zevenet1_20230301_1509.tar.gz

Thanks, Michael

Sysadminfromhell · 2023-03-01T14:27:35Z

I cannot access the admin gui to create a supportfile. asap when im on the premises tomorrow I will create one.
But it looks like that the backend is the problem, the access from there to the backend system always runs into a timeout for some reason. No matter how much seconds I setup to wait for the timeout. I even put 60 seconds via cli into but still after 60 seconds I only got timeout to the backend.
Its not the backend system itself, theyre many different backend systems and all have the same issue.

Rendone-zevenet · 2023-03-01T14:37:30Z

Hi,

supportsave can be generated via the command line too.

/usr/local/zevenet/bin/supportsave

It will generate the file in /tmp directory.

Best regards.

Sysadminfromhell · 2023-03-01T18:31:29Z

Here u go :)
supportsave_HMLRPX02_20230301_1930.tar.gz

Sysadminfromhell · 2023-03-02T09:08:36Z

Some update?

Trylok · 2023-03-02T10:54:27Z

@Sysadminfromhell
stay calm ;-)
I'm sure it's not that easy.

Rendone-zevenet · 2023-03-02T13:30:29Z

Hi,
There are some "resurrected" backends, so it is caused by a reconnected communication between the system and the backend:
Mar 1 10:15:17 HOSTNAME zproxy: [bk:BACKEN_IP:BACKEND_PORT] The backend resurrected

If the the other services experience timeouts they are caused by the backend communication

Mar 1 15:11:26 HOSTNAME zproxy: timeout request for client YYYY:65486
Mar 1 15:12:13 HOSTNAME zproxy: timeout request to backend XXXXX:8006
Mar 1 15:12:13 HOSTNAME zproxy: timeout request for client YYYY:65514
Mar 1 15:12:13 HOSTNAME zproxy: timeout request to backend XXXXX:8006
Mar 1 15:12:13 HOSTNAME zproxy: timeout request for client YYYY:65513
Mar 1 15:14:35 HOSTNAME zproxy: error reading from client YYYY:25120 (Connection reset by peer)
Mar 1 15:14:35 HOSTNAME zproxy: [7f18dc2b5700] connection release for client YYYY:25120 (result=Connection reset by peer)
Mar 1 15:14:35 HOSTNAME zproxy: error reading from client YYYY:26623 (Connection reset by peer)
Mar 1 15:14:35 HOSTNAME zproxy: [7f18da2b1700] connection release for client YYYY:26623 (result=Connection reset by peer)
Mar 1 15:14:35 HOSTNAME zproxy: error reading from client YYYY:29355 (Connection reset by peer)
Mar 1 15:14:35 HOSTNAME zproxy: [7f18da2b1700] connection release for client YYYY:29355 (result=Connection reset by peer)

All the config files seems to be well-configured.
The routing tables are correct.
The Zevenet system and the backends are in the same network so there is no extra devices implied.
It seems the backend is not able to response due to the timeout reached.

Could you test the connection to the backends?
telnet BACKEND_IP BACKEND_PORT

Could you enable the farm logs to view more info in /var/log/syslog and test a connection to the service farm?
Change param LogLevel in /usr/local/zevenet/config/FARMNAME_proxy.cfg and restart the farm.
LogLevel 7

Could you paste the state of the farm after the test service farm?

curl --unix-socket /tmp/FARMNAME_proxy.socket http://localhost/listener/0

Thanks in advance.
Best regards.

Sysadminfromhell · 2023-03-02T13:51:57Z

Telnet works:

The curl doesnt work:, because my farm isnt existing.

And the log says:

Mar  2 14:46:10 HMLRPX02 zproxy: [7f5b8b16a700] accepting client connection from 192.168.178.12:63627
Mar  2 14:46:10 HMLRPX02 zproxy: Found cert for pve.safh.de
Mar  2 14:46:10 HMLRPX02 zproxy: [7f5b8a168700] accepting client connection from 192.168.178.12:63628
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:-][bk:-][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:702} GET / HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012Cache-Control: max-age=0#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012sec-ch-ua-platform: "Windows"#015#012Upgrade-Insecure-Requests: 1#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7#015#012Sec-Fetch-Site: none#015#012Sec-Fetch-Mode: navigate#015#012Sec-Fetch-User: ?1#015#012Sec-Fetch-Dest: document#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:-][bk:-][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 0 -> 1
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:-][bk:-][cl:192.168.178.12](debug)[caller/clearStats:121]The stream stats are not defined
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:735/735} GET / HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012Cache-Control: max-age=0#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012sec-ch-ua-platform: "Windows"#015#012Upgrade-Insecure-Requests: 1#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7#015#012Sec-Fetch-Site: none#015#012Sec-Fetch-Mode: navigate#015#012Sec-Fetch-User: ?1#015#012Sec-Fetch-Dest: document#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 2
Mar  2 14:46:10 HMLRPX02 zproxy: Found cert for pve.safh.de
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 2 -> 3
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:1101} HTTP/1.1 200 OK#015#012Cache-Control: max-age=0#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Pragma: no-cache#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 824#015#012Content-Type: text/html; charset=utf-8#015#012Expires: Thu, 02 Mar 2023 13:46:10 GMT#015#012#015#012#037#010
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](established) host:pve.safh.de - "GET / HTTP/1" "HTTP/1.1 200 OK" 824 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:1146/1146} HTTP/1.1 200 OK#015#012Cache-Control: max-age=0#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Pragma: no-cache#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 824#015#012Content-Type: text/html; charset=utf-8#015#012Expires: Thu, 02 Mar 2023 13:46:10 GMT#015#012Strict-Transport-Security: max-age=21600000#015#012#015#012#037#010
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:559} GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://pve.safh.de/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 3 -> 1
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:592/592} GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1.1#015#012Host: pve.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://pve.safh.de/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:16384} HTTP/1.1 200 OK#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 683505#015#012Content-Type: application/javascript#015#012Last-Modified: Tue, 14 Jan 2020 06:08:10 GMT#015#012#015#012#037#010
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](established) host:pve.safh.de - "GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1" "HTTP/1.1 200 OK" 683505 "https://pve.safh.de/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] Stream 7 changing status: resp_head_rcv -> resp_body_rcv
Mar  2 14:46:10 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:16429/683790} HTTP/1.1 200 OK#015#012Connection: Keep-Alive#015#012Date: Thu, 02 Mar 2023 13:46:10 GMT#015#012Server: pve-api-daemon/3.0#015#012Content-Encoding: gzip#015#012Content-Length: 683505#015#012Content-Type: application/javascript#015#012Last-Modified: Tue, 14 Jan 2020 06:08:10 GMT#015#012Strict-Transport-Security: max-age=21600000#015#012#015#012#037#010
Mar  2 14:46:40 HMLRPX02 zproxy: timeout request for client 192.168.178.12:63628
Mar  2 14:46:40 HMLRPX02 zproxy: [7f5b8a168700] connection release for client 192.168.178.12:63628 (result=ok)
Mar  2 14:46:40 HMLRPX02 zproxy: timeout request to backend 172.16.24.102:8006
Mar  2 14:46:40 HMLRPX02 zproxy: [f:][th:8b16a700] [st:7][svc:Proxmox][bk:172.16.24.102:8006][cl:192.168.178.12](error) e504 Gateway timeout "Host:pve.safh.de" "GET /pve2/ext6/ext-all.js?ver=7.0.0 HTTP/1.1"
Mar  2 14:46:40 HMLRPX02 zproxy: timeout request for client 192.168.178.12:63627
Mar  2 14:46:40 HMLRPX02 zproxy: [7f5b8b16a700] connection release for client 192.168.178.12:63627 (result=ok)

Different access, different backend:

Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:-][bk:-][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 0 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:-][bk:-][cl:192.168.178.12](debug)[caller/clearStats:121]The stream stats are not defined
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] Stream 13 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:687/687} GET /guacamole/angular.min.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"177368-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 2
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 2 -> 3
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:673} GET /guacamole/guacamole.22d162f5cef4ad7db069.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"368162-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 3 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] Stream 11 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:706/706} GET /guacamole/guacamole.22d162f5cef4ad7db069.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"368162-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:133} HTTP/1.1 304 #015#012ETag: W/"16386-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/1.guacamole.bc6250d982db6462ff59.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] Stream 10 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c16c700] [st:10][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:178/178} HTTP/1.1 304 #015#012ETag: W/"16386-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:622} GET /guacamole/app.js?b=20230227005223 HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-Modified-Since: Mon, 27 Feb 2023 08:54:22 GMT#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] Stream 8 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:655/655} GET /guacamole/app.js?b=20230227005223 HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-Modified-Since: Mon, 27 Feb 2023 08:54:22 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:134} HTTP/1.1 304 #015#012ETag: W/"177368-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/angular.min.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] Stream 13 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8b96b700] [st:13][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:179/179} HTTP/1.1 304 #015#012ETag: W/"177368-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:191} HTTP/1.1 304 #015#012Cache-Control: no-cache#015#012Pragma: no-cache#015#012Last-Modified: Mon, 27 Feb 2023 08:54:22 GMT#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/app.js?b=20230227005223 HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] Stream 8 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8a969700] [st:8][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:236/236} HTTP/1.1 304 #015#012Cache-Control: no-cache#015#012Pragma: no-cache#015#012Last-Modified: Mon, 27 Feb 2023 08:54:22 GMT#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:660} GET /guacamole/1.guacamole.8c18237a4a94ee9845d9.css HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: text/css,*/*;q=0.1#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: style#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012Range: bytes=65373-79743#015#012If-Range: W/"79744-1677459432000"#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 3 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] Stream 12 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:693/693} GET /guacamole/1.guacamole.8c18237a4a94ee9845d9.css HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: text/css,*/*;q=0.1#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: style#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012Range: bytes=65373-79743#015#012If-Range: W/"79744-1677459432000"#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:134} HTTP/1.1 304 #015#012ETag: W/"368162-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/guacamole.22d162f5cef4ad7db069.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] Stream 11 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:8c96d700] [st:11][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:179/179} HTTP/1.1 304 #015#012ETag: W/"368162-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:8192} HTTP/1.1 206 #015#012Accept-Ranges: bytes#015#012ETag: W/"79744-1677459432000"#015#012Last-Modified: Mon, 27 Feb 2023 00:57:12 GMT#015#012Content-Range: bytes 65373-79743/79744#015#012Content-Type: text/css#015#012Content-Length: 14371#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015#012{background:#822;border-color:#d44}.guac-keyboard .guac-keyboard-group{line-height:0}.guac-keyboard .guac-keyboard-group.guac-keyboard-alpha,.guac-keyboard .guac-keyboard-group.guac-keyboard-function,.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{display:inline-block;text-align:center;vertical-align:top}.guac-keyboard .guac-keyboard-group.guac-keyboard-main,.guac-keyboard .guac-keyboard-group.guac-keyboard-top{display:-ms-flexbox;-ms-flex-align:stretch;-ms-flex-direction:row;display:-moz-box;-moz-box-align:stretch;-moz-box-orient:horizontal;display:-webkit-box;-webkit-box-align:stretch;-webkit-box-orient:horizontal;display:-webkit-flex;-webkit-align-items:stretch;-webkit-flex-direction:row;display:flex;align-items:stretch;flex-direction:row}.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{-ms-flex:1 1 auto;-moz-box-flex:1;-webkit-box-flex:1;-webkit-flex:1 1 auto;flex:1 1 auto}.guac-keyboard .guac-keyboard-gap{display:inline-block}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key.guac-keyboard-uses-alt-gr .guac-keyboard-cap:not(.guac-keyboard-requires-alt-gr),.guac-keyboard.guac-keyboard-modifier-caps .guac-keyboard-key.guac-keyboard-uses-caps .guac-keyboard-cap:not(.guac-keyboard-requires-caps),.guac-keyboard.guac-keyboard-modifier-lat .guac-keyboard-key.guac-keyboard-uses-lat .guac-keyboard-cap:not(.guac-keyboard-requires-lat),.guac-keyboard.guac-keyboard-modifier-shift .guac-keyboard-key.guac-keyboard-uses-shift .guac-keyboard-cap:not(.guac-keyboard-requires-shift),.guac-keyboard:not(.guac-keyboard-modifier-alt-gr) .guac-keyboard-cap.guac-keyboard-requires-alt-gr,.guac-keyboard:not(.guac-keyboard-modifier-caps) .guac-keyboard-cap.guac-keyboard-requires-caps,.guac-keyboard:not(.guac-keyboard-modifier-lat) .guac-keyboard-cap.guac-keyboard-requires-lat,.guac-keyboard:not(.guac-keyboard-modifier-shift) .guac-keyboard-cap.guac-keyboard-requires-shift{display:none}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key:not(.guac-keyboard-uses-alt-gr):not(.guac-keyboard-key-alt-gr){opacity:.5}guac-player{display:inline-block;position:relative}guac-player .guac-player-display{position:absolute;top:0;left:0;width:100%;height:100%}guac-player .guac-player-controls{position:absolute;left:0;bottom:0;width:100%}.guac-player-controls .guac-player-seek{display:block;width:100%}.guac-player-controls .guac-player-pause,.guac-player-controls .guac-player-play{color:#fff;background:transparent;border:none;width:2em;height:2em;min-width:0;padding:0;margin:0}.guac-player-controls .guac-player-pause:hover,.guac-player-controls .guac-player-play:hover{background:hsla(0,0%,100%,.5)}.guac-player-controls .pause-icon,.guac-player-controls .play-icon{display:inline-block;width:2em;height:2em;background-size:contain;background-position:50%;background-repeat:no-repeat;vertical-align:middle}.guac-player-controls .play-icon{background-image:url(images/action-icons/guac-play.svg)}.guac-player-controls .pause-icon{background-image:url(images/action-icons/guac-pause.svg)}guac-player .guac-player-status{position:fixed;top:0;left:0;width:100%;height:100%;background:rgba(0,0,0,.5);z-index:1;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-moz-box-orient:vertical;-moz-box-direction:normal;-ms-flex-direction:column;flex-direction:column}.guac-player-display,guac-player .guac-player-status{display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}.guac-player-display{position:relative}.guac-player-display .guac-player-display-container{-webkit-box-flex:0;-webkit-flex:0 0 auto;-moz-box-flex:0;-ms-flex:0 0 auto;flex:0 0 auto}guac-player-progress-indicator{width:128px;height:128px;position:relative;display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}guac-player-progress-indicator .guac-player-progress-text{font-size:2em;font-weight:700}guac-player-progress-indicator .guac-player-progress-bar-container{position:absolute;right:0;top:0;width:50%;height:100%;overflow:hidden}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway{overflow:visible}guac-player-progress-indicator .guac-player-progress-bar,guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{position:absolute;left:-64px;top:0;width:128px;height:128px;-webkit-border-radius:128px;-moz-border-radius:128px;border-radius:128px;border-color:#5af transparent transparent #5af;border-style:solid;border-width:12px}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{content:" ";display:block;box-sizing:border-box;-webkit-transform:rotate(135deg);-moz-transform:rotate(135deg);-ms-transform:rotate(135deg);-o-transform:rotate(135deg);transform:rotate(135deg)}input[type=range]{background:transparent;width:100%;margin:0}input[type=range]:focus{outline:none}input[type=range]{-webkit-appearance:none}input[type=range]::-webkit-slider-runnable-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-webkit-slider-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;-webkit-appearance:none;cursor:pointer}input[type=range]:focus::-webkit-slider-runnable-track{background:#5af}input[type=range]::-moz-range-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-moz-range-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;cursor:pointer}input[type=range]::-ms-track{width:100%;background:transparent;color:transparent}input[type=range]::-ms-thumb,input[type=range]::-ms-track{height:.5em;margin:0;border:none;border-radius:0;cursor:pointer}input[type=range]::-ms-thumb{background:#fff;width:3px}input[type=range]::-ms-fill-lower,input[type=range]::-ms-fill-upper,input[type=range]:focus::-ms-fill-lower,input[type=range]:focus::-ms-fill-upper{border:none;border-radius:0;background:#5af}a.button.add-connection,a.button.add-connection-group,a.button.add-user,a.button.add-user-group{font-size:.8em;padding-left:1.8em;position:relative}a.button.add-connection-group:before,a.button.add-connection:before,a.button.add-user-group:before,a.button.add-user:before{content:" ";position:absolute;width:1.8em;top:0;bottom:0;left:0;background-repeat:no-repeat;background-size:1em;background-position:.5em .45em}a.button.add-user:before{background-image:url(images/action-icons/guac-user-add.svg)}a.button.add-user-group:before{background-image:url(images/action-icons/guac-user-group-add.svg)}a.button.add-connection:before{background-image:url(images/action-icons/guac-monitor-add.svg)}a.button.add-connection-group:before{background-image:url(images/action-icons/guac-group-add.svg)}.settings.connections .connection-list .new-connection,.settings.connections .connection-list .new-connection-group,.settings.connections .connection-list .new-sharing-profile{opacity:.5;font-style:italic}.settings.connect
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/1.guacamole.8c18237a4a94ee9845d9.css HTTP/1" "HTTP/1.1 206 " 14371 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] Stream 12 changing status: resp_head_rcv -> resp_body_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89166700] [st:12][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:8237/14704} HTTP/1.1 206 #015#012Accept-Ranges: bytes#015#012ETag: W/"79744-1677459432000"#015#012Last-Modified: Mon, 27 Feb 2023 00:57:12 GMT#015#012Content-Range: bytes 65373-79743/79744#015#012Content-Type: text/css#015#012Content-Length: 14371#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015#012{background:#822;border-color:#d44}.guac-keyboard .guac-keyboard-group{line-height:0}.guac-keyboard .guac-keyboard-group.guac-keyboard-alpha,.guac-keyboard .guac-keyboard-group.guac-keyboard-function,.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{display:inline-block;text-align:center;vertical-align:top}.guac-keyboard .guac-keyboard-group.guac-keyboard-main,.guac-keyboard .guac-keyboard-group.guac-keyboard-top{display:-ms-flexbox;-ms-flex-align:stretch;-ms-flex-direction:row;display:-moz-box;-moz-box-align:stretch;-moz-box-orient:horizontal;display:-webkit-box;-webkit-box-align:stretch;-webkit-box-orient:horizontal;display:-webkit-flex;-webkit-align-items:stretch;-webkit-flex-direction:row;display:flex;align-items:stretch;flex-direction:row}.guac-keyboard .guac-keyboard-group.guac-keyboard-movement,.guac-keyboard .guac-keyboard-group.guac-keyboard-virtual{-ms-flex:1 1 auto;-moz-box-flex:1;-webkit-box-flex:1;-webkit-flex:1 1 auto;flex:1 1 auto}.guac-keyboard .guac-keyboard-gap{display:inline-block}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key.guac-keyboard-uses-alt-gr .guac-keyboard-cap:not(.guac-keyboard-requires-alt-gr),.guac-keyboard.guac-keyboard-modifier-caps .guac-keyboard-key.guac-keyboard-uses-caps .guac-keyboard-cap:not(.guac-keyboard-requires-caps),.guac-keyboard.guac-keyboard-modifier-lat .guac-keyboard-key.guac-keyboard-uses-lat .guac-keyboard-cap:not(.guac-keyboard-requires-lat),.guac-keyboard.guac-keyboard-modifier-shift .guac-keyboard-key.guac-keyboard-uses-shift .guac-keyboard-cap:not(.guac-keyboard-requires-shift),.guac-keyboard:not(.guac-keyboard-modifier-alt-gr) .guac-keyboard-cap.guac-keyboard-requires-alt-gr,.guac-keyboard:not(.guac-keyboard-modifier-caps) .guac-keyboard-cap.guac-keyboard-requires-caps,.guac-keyboard:not(.guac-keyboard-modifier-lat) .guac-keyboard-cap.guac-keyboard-requires-lat,.guac-keyboard:not(.guac-keyboard-modifier-shift) .guac-keyboard-cap.guac-keyboard-requires-shift{display:none}.guac-keyboard.guac-keyboard-modifier-alt-gr .guac-keyboard-key:not(.guac-keyboard-uses-alt-gr):not(.guac-keyboard-key-alt-gr){opacity:.5}guac-player{display:inline-block;position:relative}guac-player .guac-player-display{position:absolute;top:0;left:0;width:100%;height:100%}guac-player .guac-player-controls{position:absolute;left:0;bottom:0;width:100%}.guac-player-controls .guac-player-seek{display:block;width:100%}.guac-player-controls .guac-player-pause,.guac-player-controls .guac-player-play{color:#fff;background:transparent;border:none;width:2em;height:2em;min-width:0;padding:0;margin:0}.guac-player-controls .guac-player-pause:hover,.guac-player-controls .guac-player-play:hover{background:hsla(0,0%,100%,.5)}.guac-player-controls .pause-icon,.guac-player-controls .play-icon{display:inline-block;width:2em;height:2em;background-size:contain;background-position:50%;background-repeat:no-repeat;vertical-align:middle}.guac-player-controls .play-icon{background-image:url(images/action-icons/guac-play.svg)}.guac-player-controls .pause-icon{background-image:url(images/action-icons/guac-pause.svg)}guac-player .guac-player-status{position:fixed;top:0;left:0;width:100%;height:100%;background:rgba(0,0,0,.5);z-index:1;-webkit-box-orient:vertical;-webkit-box-direction:normal;-webkit-flex-direction:column;-moz-box-orient:vertical;-moz-box-direction:normal;-ms-flex-direction:column;flex-direction:column}.guac-player-display,guac-player .guac-player-status{display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}.guac-player-display{position:relative}.guac-player-display .guac-player-display-container{-webkit-box-flex:0;-webkit-flex:0 0 auto;-moz-box-flex:0;-ms-flex:0 0 auto;flex:0 0 auto}guac-player-progress-indicator{width:128px;height:128px;position:relative;display:-webkit-box;display:-webkit-flex;display:-moz-box;display:-ms-flexbox;display:flex;-webkit-box-align:center;-webkit-align-items:center;-moz-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:center;-webkit-justify-content:center;-moz-box-pack:center;-ms-flex-pack:center;justify-content:center}guac-player-progress-indicator .guac-player-progress-text{font-size:2em;font-weight:700}guac-player-progress-indicator .guac-player-progress-bar-container{position:absolute;right:0;top:0;width:50%;height:100%;overflow:hidden}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway{overflow:visible}guac-player-progress-indicator .guac-player-progress-bar,guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{position:absolute;left:-64px;top:0;width:128px;height:128px;-webkit-border-radius:128px;-moz-border-radius:128px;border-radius:128px;border-color:#5af transparent transparent #5af;border-style:solid;border-width:12px}guac-player-progress-indicator .guac-player-progress-bar-container.past-halfway:before{content:" ";display:block;box-sizing:border-box;-webkit-transform:rotate(135deg);-moz-transform:rotate(135deg);-ms-transform:rotate(135deg);-o-transform:rotate(135deg);transform:rotate(135deg)}input[type=range]{background:transparent;width:100%;margin:0}input[type=range]:focus{outline:none}input[type=range]{-webkit-appearance:none}input[type=range]::-webkit-slider-runnable-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-webkit-slider-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;-webkit-appearance:none;cursor:pointer}input[type=range]:focus::-webkit-slider-runnable-track{background:#5af}input[type=range]::-moz-range-track{border:none;border-radius:0;background:#5af;width:100%;height:.5em;cursor:pointer}input[type=range]::-moz-range-thumb{border:none;border-radius:0;background:#fff;width:3px;height:.5em;cursor:pointer}input[type=range]::-ms-track{width:100%;background:transparent;color:transparent}input[type=range]::-ms-thumb,input[type=range]::-ms-track{height:.5em;margin:0;border:none;border-radius:0;cursor:pointer}input[type=range]::-ms-thumb{background:#fff;width:3px}input[type=range]::-ms-fill-lower,input[type=range]::-ms-fill-upper,input[type=range]:focus::-ms-fill-lower,input[type=range]:focus::-ms-fill-upper{border:none;border-radius:0;background:#5af}a.button.add-connection,a.button.add-connection-group,a.button.add-user,a.button.add-user-group{font-size:.8em;padding-left:1.8em;position:relative}a.button.add-connection-group:before,a.button.add-connection:before,a.button.add-user-group:before,a.button.add-user:before{content:" ";position:absolute;width:1.8em;top:0;bottom:0;left:0;background-repeat:no-repeat;background-size:1em;background-position:.5em .45em}a.button.add-user:before{background-image:url(images/action-icons/guac-user-add.svg)}a.button.add-user-group:before{background-image:url(images/action-icons/guac-user-group-add.svg)}a.button.add-connection:before{background-image:url(images/action-icons/guac-monitor-add.svg)}a.button.add-connection-group:before{background-image:url(images/action-icons/guac-group-add.svg)}.settings.connections .connection-list .new-connection,.settings.connections .connection-list .new-connection-group,.settings.connections .connection-list .new-shari
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:329]->-> {bytes:651} GET /guacamole/templates.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"97681-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/updateStats:166]Changing stats: 1 -> 1
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] Stream 9 changing status: req_header_rcv -> resp_head_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_request_parser:356]>>>> {bytes:684/684} GET /guacamole/templates.js HTTP/1.1#015#012Host: rmd.safh.de#015#012Connection: keep-alive#015#012sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"#015#012sec-ch-ua-mobile: ?0#015#012User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36#015#012sec-ch-ua-platform: "Windows"#015#012Accept: */*#015#012Sec-Fetch-Site: same-origin#015#012Sec-Fetch-Mode: no-cors#015#012Sec-Fetch-Dest: script#015#012Referer: https://rmd.safh.de/guacamole/#015#012Accept-Encoding: gzip, deflate, br#015#012Accept-Language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7#015#012If-None-Match: W/"97681-1677459432000"#015#012If-Modified-Since: Mon, 27 Feb 2023 00:57:12 GMT#015#012X-Forwarded-For: 192.168.178.12#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:368]<-<- {bytes:133} HTTP/1.1 304 #015#012ETag: W/"97681-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012#015
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](established) host:rmd.safh.de - "GET /guacamole/templates.js HTTP/1" "HTTP/1.1 304 " 0 "https://rmd.safh.de/guacamole/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:454]New request is expected
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] Stream 9 changing status: resp_head_rcv -> req_header_rcv
Mar  2 14:48:49 HMLRPX02 zproxy: [f:][th:89967700] [st:9][svc:Guacamole][bk:172.16.24.112:8080][cl:192.168.178.12](debug)[caller/zproxy_http_response_parser:464]<<<< {bytes:178/178} HTTP/1.1 304 #015#012ETag: W/"97681-1677459432000"#015#012Date: Thu, 02 Mar 2023 13:48:44 GMT#015#012Keep-Alive: timeout=20#015#012Connection: keep-alive#015#012Strict-Transport-Security: max-age=21600000#015#012#015

Rendone-zevenet · 2023-03-02T14:06:11Z

Hi,
the curl command did not work because you use "/tmp/FARMNAME_proxy.socket" as a socket file without changing it to the real farm name.
Anyway, we will analyze the debug log in order to find the issue. We guess there is enough information.

We will keep you updated.

Thanks!

nevola · 2023-03-03T13:47:51Z

Hi @Sysadminfromhell could you please try to increase the timeouts?
Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues.

https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/

In addition, which kind of application is running in the backend side?

Thanks!

nevola · 2023-03-03T13:54:32Z

Hi again @Sysadminfromhell
Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes:

Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

nevola · 2023-03-03T13:59:24Z

Hi again @Sysadminfromhell

Also, some "error reading from client" detected. Are those legitimated clients?

Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer)
Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer)
Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

Trylok · 2023-03-03T14:50:34Z

@nevola
Just to be clear. As soon as you start an http/https farm, the farmlist is empty and the dashboard chrashes, as I have already described above. Even if no backends are configured.
l4xnat farms to not show this beavior.
So, it has nothing to do with the backends, Changing the timing makes no difference.
Not even a simple url redirect works without crashing the farm.

Best regards.

emiliocampos-zevenet · 2023-03-03T14:55:49Z

We need to figure out what is happening with the web GUI,so please do the following: Press 12 the development console will be loaded Go to network tab where all the requests will be shown Reproduce the problem, a line in red colour will be shown or I guess. Finally export all the data in the dev console to a HAR, here explained how to export a HAR file. https://support.google.com/admanager/answer/10358597?hl=en Finally attach the file to this issue for our review Thanks El vie., 3 mar. 2023 15:50, Trylok ***@***.***> escribió:

…

@nevola <https://github.com/nevola> Just to be clear. As soon as you start an http/https the farmlist is empty and the dashboard chrashes, as I have already described above. Even if no backends are configured. l4xnat farms to not show this beavior. So, it has nothing to do with the backends, Changing the timing makes no difference. Not even a simple url redirect works without crashing the farm. Best regards. — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBQEPBQRE2JGM3MEQJLZG3W2IAMLANCNFSM6AAAAAAVKR3R5Y> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Trylok · 2023-03-03T15:13:13Z

Here you go. This is from the dashboard/home section, where i get the endless spinning cycle after starting the farm.
192.168.0.32.zip
This is my test installation with just one farm configured.

The farms section looks like this before i start the farm.

And this after starting it.

Sysadminfromhell · 2023-03-03T15:32:17Z

@nevola,

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues.

https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/

In addition, which kind of application is running in the backend side?

Thanks!

I tried, this fixed the "error" partial.
If I force refresh or access a different backend its not working.
supportsave_HMLRPX02_20230303_1630.tar.gz

Hi again @Sysadminfromhell

Also, some "error reading from client" detected. Are those legitimated clients?

Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer)
Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer)
Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

not legit, but nothing really to be worried about, Happen all the time.

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes:

Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

The guardian check on this backend is only "ping" this should work all the time.

Farm explenation:

pve => Proxmox VE Backend
Guacamole => Apache Guacamole backend
Grafana => Grafana Dashboard Backend
PBS => Proxmox Backup Server Backend

Update: The pve and pbs works sometimes, the grafana and Guacamole one is always just white and empty. Nothing comes up not even a error page.

emiliocampos-zevenet · 2023-03-03T15:38:02Z

It seems like the request: https://192.168.0.32:444/zapi/v4.0/zapi.cgi/farms is generating in the load balancer a 500 Error. Please send us again a fresh supportsave of the home server 192.168.0.32 for our review? Thanks! i/v4.0/zapi.cgi/farms El vie, 3 mar 2023 a las 16:13, Trylok ***@***.***>) escribió:

…

Here you go. This is from the dashboard/home section. 192.168.0.32.zip <https://github.com/zevenet/zlb/files/10883189/192.168.0.32.zip> This is my test installation with just one farm configured. The farms section looks like this before i start the farm. [image: image] <https://user-images.githubusercontent.com/50360497/222755287-780b4566-20d8-4c01-a6a0-ebc92c58d375.png> And this after starting it. [image: image] <https://user-images.githubusercontent.com/50360497/222754874-6a1b87a3-f6fe-44b1-8177-d4f5d03b57bf.png> — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBQEPH4NKHFZDF5MG7K22DW2IDBJANCNFSM6AAAAAAVKR3R5Y> . You are receiving this because you commented.Message ID: ***@***.***>

-- Emilio CamposZEVENET Teamwww.zevenet.com Review ZEVENET Product at Gartner Insights <https://gtnr.io/LfXtqnsSr> <https://www.linkedin.com/company/zevenet> <https://twitter.com/zevenet> <https://www.facebook.com/zevenet> <https://github.com/zevenet> [image: ZEVENET] <https://www.zevenet.com/signature/> DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee please notify the sender immediately by email if you have received it by mistake and delete it from your system, you should not disseminate, distribute or copy this email in whole or in part.

Trylok · 2023-03-03T15:40:53Z

Here you go.
supportsave_zevenet1_20230303_1639.tar.gz

Rendone-zevenet · 2023-03-03T16:32:12Z

Hi Trylok,
ZEVENET CE 5.13 is based on Debian Buster.
The har file and the supportsave says the system is using a kernel 5.10.
...
"compression": 67,
"text": "{\n "description" : "Get version",\n "params" : {\n "appliance_version" : "ZCE 5, hypervisor: HyperV",\n "hostname" : "zevenet1",\n "kernel_version" : "5.10.0-21-amd64",\n "system_date" : "Fri Mar 3 16:01:02 2023",\n "zevenet_version" : "5.13.0"\n }\n}\n"
....

Please, try to install ZEVENET CE 5.13 in a Debian Buster.
If the issue is experienced again, attach the supportsave and the har file again.

Thanks!

Trylok · 2023-03-04T00:25:55Z

@Rendone-zevenet
Fresh install from the 5.13 iso. With and without package updates.
The only diverence is, now the Dashboard does not hang anymore, and the farms remain listed.
But the same end result. They are still not working. I run into the same timeouts on http/https farms as Adminfromhell from above.
The l4xnat farm for MS-Exchange also no longer works correctly as I just found out. I get asked endlessly for credentials.
Same configuration on 5.12.2 works just perfect.
Here a supportsave from this try.
supportsave_zevenet1_20230304_0110.tar.gz

nevola · 2023-03-06T12:34:23Z

Hi @Sysadminfromhell, it seems that it is related to network issues, please try to leave a ping from ZEVENET command line to your backends and check the lost packets.

If you need us to review something else, please open a new issue.
Thank you.

@nevola,

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues.
https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/
In addition, which kind of application is running in the backend side?
Thanks!

I tried, this fixed the "error" partial. If I force refresh or access a different backend its not working. supportsave_HMLRPX02_20230303_1630.tar.gz

Hi again @Sysadminfromhell
Also, some "error reading from client" detected. Are those legitimated clients?
Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer)
Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer)
Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

not legit, but nothing really to be worried about, Happen all the time.

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes:
Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

The guardian check on this backend is only "ping" this should work all the time.

Farm explenation:

pve => Proxmox VE Backend Guacamole => Apache Guacamole backend Grafana => Grafana Dashboard Backend PBS => Proxmox Backup Server Backend

Update: The pve and pbs works sometimes, the grafana and Guacamole one is always just white and empty. Nothing comes up not even a error page.

Sysadminfromhell · 2023-03-06T13:45:20Z

Hi @Sysadminfromhell, it seems that it is related to network issues, please try to leave a ping from ZEVENET command line to your backends and check the lost packets.

If you need us to review something else, please open a new issue. Thank you.

@nevola,

Hi @Sysadminfromhell could you please try to increase the timeouts? Specially, the frequency to check resurrected backends (40 seconds as an example). Let's see if you experience less timeouts and discard some network issues.
https://www.zevenet.com/knowledge-base/community-edition/community-edition-v5-12-administration-guide/ce-v5-12-lslb-farms-update-http-profile/
In addition, which kind of application is running in the backend side?
Thanks!

I tried, this fixed the "error" partial. If I force refresh or access a different backend its not working. supportsave_HMLRPX02_20230303_1630.tar.gz

Hi again @Sysadminfromhell
Also, some "error reading from client" detected. Are those legitimated clients?
Feb 28 05:19:11 HMLRPX02 zproxy: error reading from client 134.209.254.251:25978 (Connection reset by peer)
Feb 28 05:37:44 HMLRPX02 zproxy: error reading from client 35.216.169.119:43618 (Connection reset by peer)
Feb 28 05:37:49 HMLRPX02 zproxy: error reading from client 35.216.169.119:43624 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33700 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33710 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33722 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33732 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33736 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33740 (Connection reset by peer)
Feb 28 05:37:52 HMLRPX02 zproxy: error reading from client 35.216.169.119:33756 (Connection reset by peer)

not legit, but nothing really to be worried about, Happen all the time.

Hi again @Sysadminfromhell Is the Farm guardian check correct? Cause the health checker is detecting down the backends every 5 minutes:
Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 03:42:01 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 03:42:17 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 03:42:17 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) lslb :: Deleting established sessions to a backend 0 from farm LoadBalancer in service Grafana-Dashboard
Feb 28 07:47:02 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 down in farm: 'LoadBalancer', service: 'Grafana-Dashboard'
Feb 28 07:47:18 HMLRPX02 zproxy: [bk:172.16.24.113:3000] The backend resurrected
Feb 28 07:47:18 HMLRPX02 farmguardian[744]: (INFO) BackEnd 172.16.24.113:3000 resurrect in farm: 'LoadBalancer', service: 'Grafana-Dashboard'

The guardian check on this backend is only "ping" this should work all the time.
Farm explenation:
pve => Proxmox VE Backend Guacamole => Apache Guacamole backend Grafana => Grafana Dashboard Backend PBS => Proxmox Backup Server Backend
Update: The pve and pbs works sometimes, the grafana and Guacamole one is always just white and empty. Nothing comes up not even a error page.

I cannot really see that this should be a problem.
The Backend of Guacamole is perfectly available and the monitoring system never has a ping problem.
A ping command also is perfectly in shape. Time between 0.050 - 0.300 ms. No lost packets:

72 packets transmitted, 72 received, 0% packet loss, time 751ms
rtt min/avg/max/mdev = 0.067/0.117/0.166/0.022 ms

Trylok · 2023-03-06T13:49:11Z

I can confirm that there are no losts packets to the backends. All of them perfectly available.

ct-utel · 2023-03-06T15:59:22Z

For what it's worth I can confirm that since the upgrade to 5.13 my HTTP/HTTPS Farms are no longer working properly either.
My dashboard is fine but even though backends show as ok pages are not loading, some of the sites work again if I disable the farm guardian for the site but are still experience lag before the upgrade they did not and worked fine.

nevola · 2023-03-06T16:10:32Z

For what it's worth I can confirm that since the upgrade to 5.13 my HTTP/HTTPS Farms are no longer working properly either. My dashboard is fine but even though backends show as ok pages are not loading, some of the sites work again if I disable the farm guardian for the site but are still experience lag before the upgrade they did not and worked fine.

Dear @ct-utel could you please enable the logs in the affected farm, then reproduce the problem and finally, generate a support save file to be shared with us? Also, a wireshark pcap file would help to analyze what is happening.

Thank you!

ct-utel · 2023-03-06T16:55:41Z

Hi nevola,

Please find a support save I generated earlier today.
The farm I've been trying to diagnose is Web-Redirects-HTTPS.

I'm about to head home for the day but I will try and generate the pcap file in the morning.

supportsave_svr-lb-01_20230306_1311.tar.gz

nevola · 2023-03-06T17:18:53Z

Hi @ct-utel

I'm seeing:

Mar 5 13:34:38 svr-lb-01 zproxy: [f:][th:61c5f700] Malformed chunk, size was not found
Mar 5 13:34:38 svr-lb-01 zproxy: [f:][th:61c5f700] Malformed chunk, size was not found
Mar 5 13:34:40 svr-lb-01 zproxy: unexpected state in zproxy_client_read: 3

If you could export an HAR file from your browser with the malformed request would help.

Please send the HAR file to [email protected]

Thank you!

Hi nevola,

Please find a support save I generated earlier today. The farm I've been trying to diagnose is Web-Redirects-HTTPS.

I'm about to head home for the day but I will try and generate the pcap file in the morning.

Trylok · 2023-03-13T20:21:04Z

Quick Update.
Just tried version 5.13.1
No difference. Still non functional farms with Microsoft Exchange backends.
Timeout on zproxy with https farms for OWA. Outlook endless asking for credentials on l4xnat farms.

naortega · 2023-03-14T07:36:37Z

Quick Update. Just tried version 5.13.1 No difference. Still non functional farms with Microsoft Exchange backends. Timeout on zproxy with https farms for OWA. Outlook endless asking for credentials on l4xnat farms.

Good day, @Trylok,

We're currently applying patches to fix this issue, and we seem to have narrowed down the issue and found a patch that appears to resolve it. To be sure we've narrowed it down correctly, can you tell me the properties of the request? Primarily I'd like to know if the Transfer-Encoding is chunked or not.

Thank you.

Trylok · 2023-03-14T08:50:22Z

@naortega
Microsoft Exchange uses chunked Transfer-Encoding.

naortega · 2023-03-14T09:01:55Z

@naortega Microsoft Exchange uses chunked Transfer-Encoding.

Okay. We have a patch ready for this issue which we hope to be releasing soon.

Thank you for your patience and cooperation.

naortega · 2023-03-17T09:29:36Z

I'm posting to notify that a new version of zproxy (0.9.2) has been release that should fix the issues with chunking and timeouts.

ct-utel · 2023-03-17T12:22:09Z

@naortega thanks for the update.

With the latest updates we are still having issues.

We have HTTPS redirect for a TeamCity server, the GUI doesn't load at all via the load balancer but going to IP:port directly its fine.

We have a Home Assistance instance that uses the same farm, the initial login page loads but times out trying to use the dashboard. Again going directly to it, loads fine.

There is also a Project Send instance on the same farm. It loads fine internally but externally it's very slow to respond.

I'd be happy to send another support save file it that would help? so you might be able to compare the before and after upgrade.

naortega · 2023-03-17T12:49:37Z

@naortega thanks for the update.

With the latest updates we are still having issues.

We have HTTPS redirect for a TeamCity server, the GUI doesn't load at all via the load balancer but going to IP:port directly its fine.

We have a Home Assistance instance that uses the same farm, the initial login page loads but times out trying to use the dashboard. Again going directly to it, loads fine.

There is also a Project Send instance on the same farm. It loads fine internally but externally it's very slow to respond.

I'd be happy to send another support save file it that would help? so you might be able to compare the before and after upgrade.

A support save would be nice, but it would also be helpful to know better what problems you are experiencing. Regarding the first issue you experience (HTTPS redirect for TeamCity server): what exactly doesn't load? What's the request, what's the expected result, and what's the erroneous result you're getting?

In regards to the second issue (Home Assistance): Are you sure this isn't due to a network issue or a malconfiguration of the Zevenet machine? Could you show me the logs pertaining to this problem?

Thank you for your cooperation.

Trylok · 2023-03-17T20:36:03Z

@naortega
I am sorry to inform you that nothing has changed after update zproxy to 0.9.2.
Same problems as before.
Here a new supportsave.
supportsave_zevenet1_20230317_2126.tar.gz

ct-utel · 2023-03-20T13:22:11Z

A support save would be nice, but it would also be helpful to know better what problems you are experiencing. Regarding the first issue you experience (HTTPS redirect for TeamCity server): what exactly doesn't load? What's the request, what's the expected result, and what's the erroneous result you're getting?

In regards to the second issue (Home Assistance): Are you sure this isn't due to a network issue or a malconfiguration of the Zevenet machine? Could you show me the logs pertaining to this problem?

Thank you for your cooperation.

@naortega I have just e-mailed a new support save to the existing e-mail chain, I believe you are in it.
The TeamCity server is getting "504 Gateway timeout - zproxy" just trying to get to it's web gui.

I'm pretty certain none of the issues are network related as the behavior started the moment we upgraded zevenet to 5.13, also getting to services directly without the load balance in the middle is fine.

emiliocampos-zevenet · 2023-03-20T13:29:29Z

one question, do you use websocket in this service? Thanks! El lun, 20 mar 2023 a las 14:22, ct-utel ***@***.***>) escribió:

…

A support save would be nice, but it would also be helpful to know better what problems you are experiencing. Regarding the first issue you experience (HTTPS redirect for TeamCity server): what exactly doesn't load? What's the request, what's the expected result, and what's the erroneous result you're getting? In regards to the second issue (Home Assistance): Are you sure this isn't due to a network issue or a malconfiguration of the Zevenet machine? Could you show me the logs pertaining to this problem? Thank you for your cooperation. @naortega <https://github.com/naortega> I have just e-mailed a new support save to the existing e-mail chain, I believe you are in it. The TeamCity server is getting "504 Gateway timeout - zproxy" just trying to get to it's web gui. [image: image] <https://user-images.githubusercontent.com/127128354/226350446-4db4d74b-0d64-41da-8aad-a2978f63f7ad.png> I'm pretty certain none of the issues are network related as the behavior started the moment we upgraded zevenet to 5.13, also getting to services directly without the load balance in the middle is fine. — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBQEPHHT5K2EGLQBY7Y6ADW5BKY3ANCNFSM6AAAAAAVKR3R5Y> . You are receiving this because you commented.Message ID: ***@***.***>

-- Emilio CamposZEVENET Teamwww.zevenet.com Review ZEVENET Product at Gartner Insights <https://gtnr.io/LfXtqnsSr> <https://www.linkedin.com/company/zevenet> <https://twitter.com/zevenet> <https://www.facebook.com/zevenet> <https://github.com/zevenet> [image: ZEVENET] <https://www.zevenet.com/signature/> DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee please notify the sender immediately by email if you have received it by mistake and delete it from your system, you should not disseminate, distribute or copy this email in whole or in part.

ct-utel · 2023-03-20T13:40:07Z

It looks like TeamCity might but should be able to fallback if unavailable and Home Assistant can for API calls but not the GUI itself.

emiliocampos-zevenet · 2023-03-20T17:27:29Z

Thanks for the confirmation, I guess we have to take this information into consideration in order to investigate further in this issue. Regards! El lun, 20 mar 2023 a las 14:40, ct-utel ***@***.***>) escribió:

…

It looks like TeamCity might but should be able to fallback if unavailable and Home Assistant can for API calls but not the GUI itself. [image: image] <https://user-images.githubusercontent.com/127128354/226357110-c6cad877-6df8-4425-8306-b7bb26bf44ef.png> — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBQEPFZPQX7U6XH3YUHUVLW5BM4FANCNFSM6AAAAAAVKR3R5Y> . You are receiving this because you commented.Message ID: ***@***.***>

-- Emilio CamposZEVENET Teamwww.zevenet.com Review ZEVENET Product at Gartner Insights <https://gtnr.io/LfXtqnsSr> <https://www.linkedin.com/company/zevenet> <https://twitter.com/zevenet> <https://www.facebook.com/zevenet> <https://github.com/zevenet> [image: ZEVENET] <https://www.zevenet.com/signature/> DISCLAIMER: This message contains confidential information and is intended only for the individual named. If you are not the named addressee please notify the sender immediately by email if you have received it by mistake and delete it from your system, you should not disseminate, distribute or copy this email in whole or in part.

Kroki0815 · 2023-03-21T15:20:18Z

Hi,
i also run in this situation ( timeout for request ...) and i figured out that only pages are not working which are using chunked transfers.
I have now removed the Request-Header 'Accept-Encoding' in the Farm-Configuration and all is working like before.

Kroki

nevola · 2023-03-22T15:53:04Z

Hi @Trylok could you please increase Client timeout from 30 to 60, and see if there is any improvement?

Thank you.

Trylok · 2023-03-22T17:07:36Z

@nevola
Makes no difference on the https farms.
On l4xnat farms there is no timeout option anyway. And they don't work properly either. Problems described above.
No problems what so ever on the previous versions (<=5.12.2) with the same settings on all farms.

nevola · 2023-03-29T07:44:30Z

Hi @Trylok , Good news.

We have a new binary with several fixes that might resolve your issues. Please download it from here:

https://drive.google.com/file/d/1s9iA28TsD07kcvr5OSIiHW5l8J562KsV/view?usp=share_link

You've to upload to the appliance and rename it in the path /usr/local/zevenet/app/zproxy/bin/zproxy
(Please rename the current one in case of rollback)

After that, please restart the farms and let us know how it goes,
Thank you!

Kroki0815 · 2023-03-29T08:52:07Z

Hi, i also run in this situation ( timeout for request ...) and i figured out that only pages are not working which are using chunked transfers. I have now removed the Request-Header 'Accept-Encoding' in the Farm-Configuration and all is working like before.

Kroki

With the new Binary i didn't see any issues with chunked transfers any more.

Kroki

nevola · 2023-03-29T08:55:54Z

@Kroki0815 thank you for the confirmation.

Trylok · 2023-03-29T20:45:45Z

@nevola
Made it worse. With the new zproxy http farms not starting anymore (error in configfile),
If you try to create a new one, error message says "cannot save farm"

Trylok · 2023-03-29T21:20:13Z

@nevola
Sorry, forgot to make the file executable.
So far it looks like its working now.
I have to test it further.

Trylok · 2023-03-30T21:05:30Z

@nevola
After further testing i had to rollback.
On https farm with ssl backends i have credentials errors (401) and pending connections.
Here a supportsave from this situation.
supportsave_ZEVENET_20230330_2303.tar.gz

naortega · 2023-03-31T06:28:18Z

On https farm with ssl backends i have credentials errors (401) and pending connections.

We'll have a look into the authorization errors (401 response code). The pending connections seems to be an issue with the counter, but it should not be an issue with actual connections pending. We'll try resolving these issues as soon as possible.

naortega · 2023-03-31T10:09:00Z

@Trylok Would it be possible for you to send a HAR that displays the 401 error you get? Is it when connecting via websocket? Is it a normal HTTP POST request?

Trylok · 2023-03-31T11:54:25Z

@naortega
I have to wait until evening to test it. But it is an HTTP POST request.

nevola · 2023-04-03T12:56:48Z

Hi @Trylok could you please enable farm logs, then reproduce the problem and then generate a new support save to [email protected] (please do not send it via github).
Thank you!

Trylok · 2023-04-03T21:26:53Z

File has been send.

naortega · 2023-05-10T08:38:09Z

@Trylok In the most recent release of Zproxy (0.9.4) we've resolved the issue of pending connections counting. Are you still having these issues? Also, in regards to the 401 responses, have these been resolved?

Trylok · 2023-05-10T21:01:56Z

@naortega
No improvement with 9.4. Same credentials errors (401) and pending connections on https farms. Rolled back again.
In the moment i have only a short time window for testing. Sorry i can not help you any further.

naortega · 2023-05-11T08:09:02Z

@naortega No improvement with 9.4. Same credentials errors (401) and pending connections on https farms. Rolled back again. In the moment i have only a short time window for testing. Sorry i can not help you any further.

Understood. Would you mind taking a look at the logs corresponding to these requests on the backends? Zproxy itself does not give 401 responses unless received from the backend. It may be possible that Zproxy is erroneously manipulating the requests such that your backend gives that response, in which case there should be some error log from your backend.

naortega mentioned this issue Mar 22, 2023

After Upgrade to v5.13.1 HTTPS Farms not working correctly #125

Open

After Upgrade to 5.13 HTTP/HTTPS Farms not working anymore. #123

After Upgrade to 5.13 HTTP/HTTPS Farms not working anymore. #123

Comments

Trylok commented Feb 28, 2023

Rendone-zevenet commented Mar 1, 2023

Trylok commented Mar 1, 2023

Sysadminfromhell commented Mar 1, 2023 • edited Loading

Trylok commented Mar 1, 2023 • edited Loading

Sysadminfromhell commented Mar 1, 2023

Rendone-zevenet commented Mar 1, 2023

/usr/local/zevenet/bin/supportsave

Sysadminfromhell commented Mar 1, 2023

Sysadminfromhell commented Mar 2, 2023

Trylok commented Mar 2, 2023

Rendone-zevenet commented Mar 2, 2023 • edited Loading

Sysadminfromhell commented Mar 2, 2023 • edited Loading

Rendone-zevenet commented Mar 2, 2023

nevola commented Mar 3, 2023

nevola commented Mar 3, 2023

nevola commented Mar 3, 2023

Trylok commented Mar 3, 2023 • edited Loading

emiliocampos-zevenet commented Mar 3, 2023 via email

Trylok commented Mar 3, 2023 • edited Loading

Sysadminfromhell commented Mar 3, 2023 • edited Loading

emiliocampos-zevenet commented Mar 3, 2023 via email

Trylok commented Mar 3, 2023

Rendone-zevenet commented Mar 3, 2023

Trylok commented Mar 4, 2023 • edited Loading

nevola commented Mar 6, 2023

Sysadminfromhell commented Mar 6, 2023 • edited Loading

Trylok commented Mar 6, 2023

ct-utel commented Mar 6, 2023

nevola commented Mar 6, 2023

ct-utel commented Mar 6, 2023

nevola commented Mar 6, 2023

Trylok commented Mar 13, 2023

naortega commented Mar 14, 2023

Trylok commented Mar 14, 2023

naortega commented Mar 14, 2023

naortega commented Mar 17, 2023

ct-utel commented Mar 17, 2023

naortega commented Mar 17, 2023

Trylok commented Mar 17, 2023 • edited Loading

ct-utel commented Mar 20, 2023

emiliocampos-zevenet commented Mar 20, 2023 via email

ct-utel commented Mar 20, 2023

emiliocampos-zevenet commented Mar 20, 2023 via email

Kroki0815 commented Mar 21, 2023

nevola commented Mar 22, 2023

Trylok commented Mar 22, 2023

nevola commented Mar 29, 2023

Kroki0815 commented Mar 29, 2023

nevola commented Mar 29, 2023

Trylok commented Mar 29, 2023

Trylok commented Mar 29, 2023

Trylok commented Mar 30, 2023

naortega commented Mar 31, 2023

naortega commented Mar 31, 2023

Trylok commented Mar 31, 2023

nevola commented Apr 3, 2023

Trylok commented Apr 3, 2023

naortega commented May 10, 2023

Trylok commented May 10, 2023

naortega commented May 11, 2023

Sysadminfromhell commented Mar 1, 2023 •

edited

Loading

Trylok commented Mar 1, 2023 •

edited

Loading

Rendone-zevenet commented Mar 2, 2023 •

edited

Loading

Sysadminfromhell commented Mar 2, 2023 •

edited

Loading

Trylok commented Mar 3, 2023 •

edited

Loading

Trylok commented Mar 3, 2023 •

edited

Loading

Sysadminfromhell commented Mar 3, 2023 •

edited

Loading

Trylok commented Mar 4, 2023 •

edited

Loading

Sysadminfromhell commented Mar 6, 2023 •

edited

Loading

Trylok commented Mar 17, 2023 •

edited

Loading