Desynchronization issue between 3DBBs and sensors

[LevDukhovnyi]

Hello. So i'm trying to gather RGB+Lidar+Labels created using custom filtering each 100 frames. Originally i was ticking Lidar and RGB constantly (sensor_tick at 0.0) which led to slower performance but everything seemed to be in sync. Since it was too slow i've decided that i should try switching tick of the Lidar to 5.0 so i get the reading only when i need it. It boosted the speed of the script but created desync between RGB, Lidar and Labels of 3DBBs. I was able to sync RGB and Lidar after trying different settings but 3DBBs are seemingly coming from a later tick or something. They are ahead of the vehicles and walkers. If the actor is standing in place then it's fine. Ego vehicle turning also shifts BBs to the side.

Here we have a jam so no vehicle is on the move so all BBs are correct.

Hero ego is turning so all boxes are going to the side.

And here you can see that BB for the kid is moving faster than the actual actor

Here you see bicycle and van moving and they have the offset but bus and car are in place so their BBs are okay

My current setup is:
RGB sensor ticks constantly and it's image constantly is being pulled and used for the output window + further processing when the 100th frame comes.
Lidar only ticks with 5.0 sensor_tick.
RGB and Lidar get() happen one after the other after world.tick().
Almost instantly after that i get the actors i need with:
all_actors = world.get_actors()
actors = [actor for actor in all_actors if 'vehicle' in actor.type_id or 'walker' in actor.type_id]
And then i loop through them doing my filtering.

I know that i'm using references to live actors so maybe it makes sense that BBs move a bit further compared to sensors but i'm expecting a small difference not those huge offsets. It seems like i'm making some mistake that leads to the wrong order of execution or delayed execution but i've tried multiple things and wasn't really able to get better results.

Any advice would be greatly appreciated.