Set Statiques optional values to a default

[jmaupetit]

Purpose

The importation process may create many duplicates as the upsert technique we use does not compare rows with missing fields. Two tables are mostly concerned: Operateur and Amenageur. Hence we decided to fill optional (non filled) fields with default generic strings (e.g. "NA").

Proposal

• set defaults to problematic optional values in the Statique model
• test Statique model defaults
• bonus: improve json parsing performances using the pyarrow engine

[github-actions]

Current benchmark

Type	Name	Request Count	Failure Count	Median Response Time	Average Response Time	Min Response Time	Max Response Time	Average Content Size	Requests/s	Failures/s	50%	66%	75%	80%	90%	95%	98%	99%	99.9%	99.99%	100%	git	timestamp	version
GET	/auth/whoami	7	0	24.0	249.0865165714281	2.066695999985768	842.0040190000009	160.0	0.8168384616617316	0.0	24	32	830	830	840	840	840	840	840	840	840	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
POST	/dynamique/session/	2	0	19.0	168.0650499999956	18.55414100000985	317.57595899998137	45.0	0.2333824176176376	0.0	320	320	320	320	320	320	320	320	320	320	320	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
POST	/dynamique/session/bulk	3	0	34.0	240.9101370000049	23.896750000005795	664.473074	411.0	0.3500736264264564	0.0	34	34	660	660	660	660	660	660	660	660	660	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
GET	/dynamique/status/	7	0	80.0	164.8956128571431	24.497332999999344	452.3575810000011	285.0	0.8168384616617316	0.0	80	150	390	390	450	450	450	450	450	450	450	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
POST	/dynamique/status/	2	0	18.294405999995437	157.76369200000318	18.294405999995437	297.2329780000109	45.0	0.2333824176176376	0.0	300	300	300	300	300	300	300	300	300	300	300	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
GET	/dynamique/status/FRALLEGO002006P3	8	0	32.0	379.9029954999966	7.760507999989841	1127.970093000016	283.0	0.9335296704705504	0.0	390	470	970	970	1100	1100	1100	1100	1100	1100	1100	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
GET	/dynamique/status/FRALLEGO002006P3/history	4	0	560.0	529.9886504999947	24.604721999992307	969.0892109999822	16118.0	0.4667648352352752	0.0	570	570	970	970	970	970	970	970	970	970	970	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
POST	/dynamique/status/bulk	7	0	150.0	193.31018014284496	9.93689199998471	456.97257199998376	411.0	0.8168384616617316	0.0	150	300	410	410	460	460	460	460	460	460	460	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
POST	/statique/	5	2	430.0	457.4469628000032	8.871231000000535	972.62237999999	55.8	0.5834560440440939	0.2333824176176376	430	850	850	970	970	970	970	970	970	970	970	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
GET	/statique/?limit=10	4	0	43.0	400.93118099999714	23.444947000001548	828.4212089999983	13826.0	0.4667648352352752	0.0	710	710	830	830	830	830	830	830	830	830	830	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
GET	/statique/?limit=100	6	0	590.0	628.587342833337	144.40909500001453	1098.6413819999825	129006.0	0.7001472528529128	0.0	670	670	830	830	1100	1100	1100	1100	1100	1100	1100	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
PUT	/statique/FR073EP7QU5233	5	0	440.0	315.84723159999726	33.62888299997735	561.016010000003	1371.0	0.5834560440440939	0.0	440	480	480	560	560	560	560	560	560	560	560	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
GET	/statique/FRALLEGO002006P3	2	0	32.32216500001073	236.11397599999864	32.32216500001073	439.9057869999865	1202.0	0.2333824176176376	0.0	440	440	440	440	440	440	440	440	440	440	440	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
POST	/statique/bulk	8	0	660.0	724.8099392499939	397.51130199999807	1133.218836000026	1118.625	0.9335296704705504	0.0	670	720	1100	1100	1100	1100	1100	1100	1100	1100	1100	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0
	Aggregated	70	2	390.0	375.6719978142833	2.066695999985768	1133.218836000026	13170.971428571427	8.168384616617315	0.2333824176176376	390	530	660	710	970	1100	1100	1100	1100	1100	1100	62c5c15	2025-01-09 15:57:02.466723+00:00	0.16.0

Comparison with the latest previous benchmark

A lower (negative) value means the current version performs better than the previous one.

Type	Name	Request Count	Failure Count	Median Response Time	Average Response Time	Min Response Time	Max Response Time	Average Content Size	Requests/s
GET	/auth/whoami
POST	/dynamique/session/
POST	/dynamique/session/bulk
GET	/dynamique/status/
POST	/dynamique/status/
GET	/dynamique/status/FRALLEGO002006P3
GET	/dynamique/status/FRALLEGO002006P3/history
POST	/dynamique/status/bulk
POST	/statique/
GET	/statique/?limit=10
GET	/statique/?limit=100
PUT	/statique/FR073EP7QU5233
GET	/statique/FRALLEGO002006P3
POST	/statique/bulk
	Aggregated

[loco-philippe]

I agree to have the default value as proposed:

• DEFAULT_CHAR_VALUE: str = "NA"
• DEFAULT_EMAIL_ADDRESS: str = "[email protected]"
• DEFAULT_PHONE_NUMBER: FrenchPhoneNumber =
• FrenchPhoneNumber("+33.123456789")
• DEFAULT_SIREN_NUMBER: str = "123456789"

This solution is acceptable because these fields are optional but if these fields become mandatory, the default values ​​will have to be removed.

[jmaupetit]

This solution is acceptable because these fields are optional but if these fields become mandatory, the default values ​​will have to be removed.

Yes, we do agree on this!