Filewatcher problem with rsync

[Iture]

Hi
I use rsync to deploy configuration to production, and have habapp running in a docker container. Recently I found a strange behavior of filewatcher. After copying a new file using rsync, it removes any references to it. Probably it happens because rsync creates temporary file and then moves it to final location. Habapp sees this move and then interprets it as file removal.
EDIT:
Probably the answer is here:

HABApp/src/HABApp/core/files/watcher/base_watcher.py

Line 55 in 5ccb2d6

if hasattr(event, 'dest_path'):

and here:

HABApp/src/HABApp/core/files/watcher/base_watcher.py

Line 61 in 5ccb2d6

def file_changed(self, dst: str):

You can see it in the logs (it's Loki, read from bottom):

  |   | 2024-03-13 12:02:39.232 | [2024-03-13 12:02:39,232] [             HABApp.files]    DEBUG \| Removed rules/presence/leave_arrival.py |  
-- | -- | -- | -- | --
  |   | 2024-03-13 12:02:39.231 | [2024-03-13 12:02:39,231] [             HABApp.files]    DEBUG \| rules/presence/leave_arrival.py changed to REMOVED |  
  |   | 2024-03-13 12:02:39.229 | [2024-03-13 12:02:39,228] [             HABApp.Rules]    DEBUG \| File rules/presence/leave_arrival.py successfully unloaded! |  
  |   | 2024-03-13 12:02:39.229 | [2024-03-13 12:02:39,228] [                   HABApp]    DEBUG \| Removed event listener for "presence_devices" (filter=ValueChangeEventFilter(value=OPEN)) |  
  |   | 2024-03-13 12:02:39.228 | [2024-03-13 12:02:39,228] [                   HABApp]    DEBUG \| Removed event listener for "presence_manual" (filter=ValueChangeEventFilter(value=OFF)) |  
  |   | 2024-03-13 12:02:39.228 | [2024-03-13 12:02:39,228] [                   HABApp]    DEBUG \| Removed event listener for "presence_devices" (filter=ValueChangeEventFilter(value=CLOSED)) |  
  |   | 2024-03-13 12:02:39.228 | [2024-03-13 12:02:39,227] [                   HABApp]    DEBUG \| Removed event listener for "presence_manual" (filter=ValueChangeEventFilter(value=ON)) |  
  |   | 2024-03-13 12:02:39.228 | [2024-03-13 12:02:39,227] [                   HABApp]    DEBUG \| Removed event listener for "presence_kids" (filter=ValueChangeEventFilter()) |  
  |   | 2024-03-13 12:02:39.228 | [2024-03-13 12:02:39,227] [                   HABApp]    DEBUG \| Removed event listener for "presence_devices" (filter=ValueChangeEventFilter(value=OPEN)) |  
  |   | 2024-03-13 12:02:39.223 | [2024-03-13 12:02:39,223] [             HABApp.Rules]    DEBUG \| Removing file: rules/presence/leave_arrival.py |  
  |   | 2024-03-13 12:02:38.618 | [2024-03-13 12:02:38,617] [       HABApp.file.events]    DEBUG \| DirModifiedEvent(src_path='/habapp/config/rules/presence', dest_path='', event_type='modified', is_directory=True, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.616 | [2024-03-13 12:02:38,616] [       HABApp.file.events]    DEBUG \| FileMovedEvent(src_path='/habapp/config/rules/presence/.leave_arrival.py.1upIJ8', dest_path='/habapp/config/rules/presence/leave_arrival.py', event_type='moved', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.616 | [2024-03-13 12:02:38,616] [       HABApp.file.events]    DEBUG \| FileModifiedEvent(src_path='/habapp/config/rules/presence/.leave_arrival.py.1upIJ8', dest_path='', event_type='modified', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.616 | [2024-03-13 12:02:38,615] [       HABApp.file.events]    DEBUG \| DirModifiedEvent(src_path='/habapp/config/rules/presence', dest_path='', event_type='modified', is_directory=True, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.615 | [2024-03-13 12:02:38,615] [       HABApp.file.events]    DEBUG \| FileClosedEvent(src_path='/habapp/config/rules/presence/.leave_arrival.py.1upIJ8', dest_path='', event_type='closed', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.615 | [2024-03-13 12:02:38,615] [       HABApp.file.events]    DEBUG \| FileModifiedEvent(src_path='/habapp/config/rules/presence/.leave_arrival.py.1upIJ8', dest_path='', event_type='modified', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.615 | [2024-03-13 12:02:38,614] [       HABApp.file.events]    DEBUG \| FileOpenedEvent(src_path='/habapp/config/rules/presence/.leave_arrival.py.1upIJ8', dest_path='', event_type='opened', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.614 | [2024-03-13 12:02:38,614] [       HABApp.file.events]    DEBUG \| DirModifiedEvent(src_path='/habapp/config/rules/presence', dest_path='', event_type='modified', is_directory=True, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.614 | [2024-03-13 12:02:38,614] [       HABApp.file.events]    DEBUG \| FileCreatedEvent(src_path='/habapp/config/rules/presence/.leave_arrival.py.1upIJ8', dest_path='', event_type='created', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.612 | [2024-03-13 12:02:38,612] [       HABApp.file.events]    DEBUG \| FileOpenedEvent(src_path='/habapp/config/rules/presence/leave_arrival.py', dest_path='', event_type='opened', is_directory=False, is_synthetic=False) |  
  |   | 2024-03-13 12:02:38.587 | [2024-03-13 12:02:38,585] [       HABApp.file.events]    DEBUG \| FileOpenedEvent(src_path='/habapp/config/rules/presence/leave_arrival.py', dest_path='', event_type='opened', is_directory=False, is_synthetic=False)

[spacemanspiff2007]

Does the file get loaded again after the unload?
If it's still there I would have expected that it gets unloaded and then immediately loaded again.

[Iture]

Nope, that's the full log. Habapp forgets about this file. What's interesting, when I copy files using scp, everythong works, rsync does it in some not compatible way. Another interesting thing: I tried to change behavior of rsync using --inplace flag, but it doesn't change anythong, habapp behaves in the same way.
Ah, systems: rsync runs on OSX, copies files using ssh to Linux@arm (raspberry)

[spacemanspiff2007]

I can't make sense of it. Could you additionally show me the corresponding file events from the HABApp events?
They are on the event bus under HABApp.Files

[Iture]

You have everything in the first post, or is there anything more?

[spacemanspiff2007]

No, the corresponding issued file events are missing. You typically find these in the HABApp_events.log under HABApp.Files

[Iture]

Ahh, I see.
Unfortunately I have logging level set to WARN for events, and all is sent to Loki - as I mentioned, whole log is in the first post

[spacemanspiff2007]

You can set the logging level for the event file to INFO, then do rsync and then set it back to warning again. It will be picked up on the fly.