Update/validator

[vkresch]

Resolves #49
Resolves #46
Resolves #45
Resolves #44
Resolves #43
Resolves #42
Resolves #31

Change Notes:

• remove doxygen xml support
• remove sqlite log processing only use plaint text for validation log output (simplification)
• move default rules into rules folder from requirements-osi-3 folder
• replace progressbar with tqdm
• remove lzma decoding and encoding (simplification)
• bugfix: BaseMoving nested attribute validation
• bugfix: update yaml loading ruaml
• bugfix: setup.py fix the default rules folder copying into site-packages with different python versions
• remove protocpp from osi validator dependency
• Migrate pipeline from travisCI -> github workflow

How has it been tested?
I tested the rules with defining the x position range of all moving objects and it gave me validation errors outside this range for the small_test.txt. In the test trace there is only one moving object which moves in the positive x direction. To reproduce go into osi_common.yaml and change to this:

BaseMoving:
  dimension:
  position:
    x:
      - is_greater_than_or_equal_to: 10
      - is_less_than_or_equal_to: 20

Validation Result:

Ranges of timestamps    Message
----------------------  --------------------------------------------------------------------------------------------------------------------------------------------------
...
[0, 9]                  BaseMoving.position.x.is_greater_than_or_equal_to(10) does not comply in SensorView.global_ground_truth.moving_object.base.position.x
[0, 9], [21, 49]        BaseMoving.position.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.base.position
...
[21, 49]                BaseMoving.position.x.is_less_than_or_equal_to(20) does not comply in SensorView.global_ground_truth.moving_object.base.position.x

Mention a member

@jdsika please check the latest update
in terminal you should be able to install with:
$ pip install .

and to run with:
$ osivalidator --data data/small_test.txt --rules rules/

Check the checklist

• My code and comments follow the contributors guidelines of this project.
• I have performed a self-review of my own code.
• I have made corresponding changes to the documentation for osi-validation.
• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests / travis ci pass locally with my changes.

To Fix

• Decode example trace file and fix naming
• Fix install
• Delete KPIs
• Implement Build check in GitHub pipeline
• Fix reading binary trace files
• Enable reading human readable trace files
• Fix Readme
  • Add --data option
  • Fix documentation links
  • Fix CI pipeline badge
• Enable setting a rule to define, if a field is required. Giving a warning for every missing field is not really effective. -> Enable setting a rule to define, if a field is required #51
• Fix that when a rule is deleted, it is not checked anymore. -> A field removed from the rules is still tested #52
• Fix logging to correctly differentiate between error and warning analog to test results. -> Fix logging to correctly differentiate between error and warning analog to test results #53
• Fix documentation format

[jdsika]

@ClemensLinnhoff can you review please?

[ClemensLinnhoff]

@ClemensLinnhoff can you review please?

Yes, I will review this next week.

[ClemensLinnhoff]

I cannot extract the small_test sample trace file.
I tried the archive manager under Ubuntu 20.04 and also unlzma in the terminal. Both result in an error.

unlzma: Decoder error

[ClemensLinnhoff]

When running osi-validation with a different trace file, I get the following error:

... requirement version https://git-lfs.github.com/spec/v1 was not found: Invalid requirement, parse error at "'https://'"

... requirement oid sha256:c31b53fd99af8ac244854ea0d6659ed1d8e757e59b7dd32afbcc19a9f71a4876 was not found: Invalid requirement, parse error at "'sha256:c'"

... requirement size 168 was not found: Invalid requirement, parse error at "'168'"

Installing version https://git-lfs.github.com/spec/v1
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
ERROR: Invalid requirement: 'version https://git-lfs.github.com/spec/v1'
Hint: It looks like a path. File 'version https://git-lfs.github.com/spec/v1' does not exist.
Error: Could not install missing packages!

I am using pip 20.0.2 and python3.8 in venv.

[ClemensLinnhoff]

If I see this correctly, the KPIs folder is not changed by this PR. However, there are mistakes in it.
E.g. osi_SensorView.dox, ll. 167, 168:

source_horizontal_angle | \c Vector3d   |      | No       | Must be set and valid
source_vertical_angle   | \c Identifier |      | No       | Must be set and correspond to an \c Object

Both angles should be double values and the Requrements are also wrong. See official documentaion:

Furthermore, there seems to be a mix-up between RadarSensorView::Reflection and LidarSensorView::Reflection

Are the KPIs used for the validation? If yes, they should be corrected, if no, they should be deleted.

[ClemensLinnhoff]

Please implement a build check for the GitHub CI pipeline. The travis build check does not seem to work.

[ClemensLinnhoff]

The links in the Readme to the documentation do not work. This way, I don't know, which trace file formats are supported.
After working around the requirements issue before, I tried my own .osi trace file and also a converted .txth trace file. Both throw an error.

For the .osi trace file:

google.protobuf.message.DecodeError: Error parsing message

For the .txth trace file:

Exception: The defined file format my_trace_file.txth does not exist.

[ClemensLinnhoff]

You write in this PR, to run osi-validation with

osivalidator --data data/small_test.txt --rules rules/

However, the --data option is not listed in the README.

[vkresch]

Please implement a build check for the GitHub CI pipeline. The travis build check does not seem to work.

Good point! I will migrate the CI from travis to github workflow.

[vkresch]

If I see this correctly, the KPIs folder is not changed by this PR. However, there are mistakes in it. E.g. osi_SensorView.dox, ll. 167, 168:

source_horizontal_angle | \c Vector3d   |      | No       | Must be set and valid
source_vertical_angle   | \c Identifier |      | No       | Must be set and correspond to an \c Object

Both angles should be double values and the Requrements are also wrong. See official documentaion:

Furthermore, there seems to be a mix-up between RadarSensorView::Reflection and LidarSensorView::Reflection

Are the KPIs used for the validation? If yes, they should be corrected, if no, they should be deleted.

The KPI folder is not used. I will delete it.

[vkresch]

I cannot extract the small_test sample trace file. I tried the archive manager under Ubuntu 20.04 and also unlzma in the terminal. Both result in an error.

unlzma: Decoder error

Can you try?:

xz -dk small_test.txt.lzma

I will commit the decoded trace to the repo to avoid this issue.

[ClemensLinnhoff]

I cannot extract the small_test sample trace file. I tried the archive manager under Ubuntu 20.04 and also unlzma in the terminal. Both result in an error.

unlzma: Decoder error

Can you try?:

xz -dk small_test.txt.lzma

I will commit the decoded trace to the repo to avoid this issue.

With xz -dk small_test.txt.lzma I get the following error:

xz: small_test.txt.lzma: File format not recognized

[jdsika]

Ah, instructions for the protoc compiler are missing in the readme

[jdsika]

!!!! setup.py does have a different requirements list then the requirements.txt !!!!

https://stackoverflow.com/questions/14399534/reference-requirements-txt-for-the-install-requires-kwarg-in-setuptools-setup-py

[jdsika]

Q413005@LMUC1025428 MINGW64 /c/repositories/osi-validation (update/validator)
$ osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.osi --rules rules/
Instantiate logger ...
Reading data ...
100%|##########| 6080/6080 [00:00<00:00, 5931930.29it/s]
Retrieving messages in osi trace file until 6080 ...
50 messages has been discovered in 0.0020008087158203125 s
Collect validation rules ...

Caching ...
Caching done!
100%|##########| 50/50 [00:00<00:00, 393.41it/s]
Warnings (40)
Ranges of timestamps    Message
----------------------  --------------------------------------------------------------------------------------------------------------------------------------------------
[0, 49]                 SensorView.sensor_id.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.host_vehicle_id.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.version.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.timestamp.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.mounting_position.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.mounting_position_rmse.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.host_vehicle_data.is_set(None) does not comply in SensorView
[0, 49]                 GroundTruth.host_vehicle_id.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.country_code.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.stationary_object.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 MovingObject.vehicle_attributes.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.VehicleClassification.trailer_id.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.VehicleClassification.light_state.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.VehicleClassification.has_trailer.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.vehicle_classification.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 BaseMoving.velocity.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.acceleration.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.orientation_rate.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.orientation_acceleration.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.base_polygon.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 MovingObject.base.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 MovingObject.type.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.assigned_lane_id.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.model_reference.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 GroundTruth.moving_object.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 GroundTruth.traffic_sign.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.traffic_light.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.road_marking.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.lane_boundary.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.lane.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.occupant.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.environmental_conditions.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.proj_string.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.map_reference.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 SensorView.global_ground_truth.is_valid(None) does not comply in SensorView.global_ground_truth
[0, 49]                 SensorView.generic_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.radar_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.lidar_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.camera_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.ultrasonic_sensor_view.is_set(None) does not comply in SensorView
(.venv)

Q413005@LMUC1025428 MINGW64 /c/repositories/osi-validation (update/validator)
$ osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt --rules rules/
Instantiate logger ...
Reading data ...
 98%|#########7| 6056/6180 [00:00<00:00, 5386069.77it/s]
Retrieving message offsets in txt trace file until 6180 ...
50 messages has been discovered in 0.0011243820190429688 s
Collect validation rules ...

Caching ...
Processing index 49 with offset 6056: 100%|##########| 50/50 [00:00<00:00, 8418.58it/s]
Caching done!
100%|##########| 50/50 [00:00<00:00, 359.46it/s]
Warnings (40)
Ranges of timestamps    Message
----------------------  --------------------------------------------------------------------------------------------------------------------------------------------------
[0, 49]                 SensorView.sensor_id.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.host_vehicle_id.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.version.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.timestamp.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.mounting_position.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.mounting_position_rmse.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.host_vehicle_data.is_set(None) does not comply in SensorView
[0, 49]                 GroundTruth.host_vehicle_id.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.country_code.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.stationary_object.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 MovingObject.vehicle_attributes.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.VehicleClassification.trailer_id.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.VehicleClassification.light_state.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.VehicleClassification.has_trailer.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.vehicle_classification.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 BaseMoving.velocity.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.acceleration.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.orientation_rate.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.orientation_acceleration.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.base_polygon.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 MovingObject.base.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 MovingObject.type.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.assigned_lane_id.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.model_reference.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 GroundTruth.moving_object.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 GroundTruth.traffic_sign.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.traffic_light.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.road_marking.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.lane_boundary.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.lane.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.occupant.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.environmental_conditions.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.proj_string.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.map_reference.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 SensorView.global_ground_truth.is_valid(None) does not comply in SensorView.global_ground_truth
[0, 49]                 SensorView.generic_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.radar_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.lidar_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.camera_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.ultrasonic_sensor_view.is_set(None) does not comply in SensorView
(.venv)Q413005@LMUC1025428 MINGW64 /c/repositories/osi-validation (update/validator)
$ osivalidator --data data/20210818T150542Z_sv_312_50_one_moving_object.txt --rules rules/
Instantiate logger ...
Reading data ...
 98%|#########7| 6056/6180 [00:00<00:00, 5386069.77it/s]
Retrieving message offsets in txt trace file until 6180 ...
50 messages has been discovered in 0.0011243820190429688 s
Collect validation rules ...

Caching ...
Processing index 49 with offset 6056: 100%|##########| 50/50 [00:00<00:00, 8418.58it/s]
Caching done!
100%|##########| 50/50 [00:00<00:00, 359.46it/s]
Warnings (40)
Ranges of timestamps    Message
----------------------  --------------------------------------------------------------------------------------------------------------------------------------------------
[0, 49]                 SensorView.sensor_id.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.host_vehicle_id.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.version.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.timestamp.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.mounting_position.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.mounting_position_rmse.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.host_vehicle_data.is_set(None) does not comply in SensorView
[0, 49]                 GroundTruth.host_vehicle_id.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.country_code.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.stationary_object.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 MovingObject.vehicle_attributes.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.VehicleClassification.trailer_id.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.VehicleClassification.light_state.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.VehicleClassification.has_trailer.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 MovingObject.vehicle_classification.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.vehicle_classification
[0, 49]                 BaseMoving.velocity.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.acceleration.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.orientation_rate.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.orientation_acceleration.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 BaseMoving.base_polygon.is_set(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 MovingObject.base.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object.base
[0, 49]                 MovingObject.type.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.assigned_lane_id.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 MovingObject.model_reference.is_set(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 GroundTruth.moving_object.is_valid(None) does not comply in SensorView.global_ground_truth.moving_object
[0, 49]                 GroundTruth.traffic_sign.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.traffic_light.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.road_marking.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.lane_boundary.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.lane.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.occupant.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.environmental_conditions.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.proj_string.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 GroundTruth.map_reference.is_set(None) does not comply in SensorView.global_ground_truth
[0, 49]                 SensorView.global_ground_truth.is_valid(None) does not comply in SensorView.global_ground_truth
[0, 49]                 SensorView.generic_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.radar_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.lidar_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.camera_sensor_view.is_set(None) does not comply in SensorView
[0, 49]                 SensorView.ultrasonic_sensor_view.is_set(None) does not comply in SensorView
(.venv)

[jdsika]

@vkresch you have removed lzma encoding but it is still present in the OSI repository: https://github.com/OpenSimulationInterface/open-simulation-interface/blob/master/format/OSITrace.py

Update: Please create a ticket in OSI to remove lzma there in the scripts as well

[jdsika]

@vkresch my mission was successful. I killed the tests because of the minimum proto version from the OSI repository :))

[vkreschenski-plenty]

Nice :) this was my concern with the bumping.
Could you change the protobuf version to "protobuf>=3.15.0"?

[jdsika]

You downloaded a different (unknown) version of the protoc compiler in the ci workflow. I now updated it to be the same as in the osi repository.

[jdsika]

Twitter showing this after my review:
https://twitter.com/PR0GRAMMERHUM0R/status/1745221513320464514?t=RfVWw4DVkCv0SdQihltRzQ&s=19

[vkresch]

Thanks for the fix!

[vkresch]

@jdsika looks like I do not have permission to merge the PR. I noticed an issue with the parallel run option and fixed it. Please review it again. Thanks!