Only update OpenCL environment when OpenCL changes are staged #3211

Job	Run time
Generate test matrix	7s
Test (ubuntu-20.04, 3.11)	2m 51s
Test (ubuntu-20.04, 3.10)	2m 41s
Test (ubuntu-22.04, 3.11)	2m 35s
Test (ubuntu-22.04, 3.10)	3m 3s
Test (windows-latest, 3.11)	8m 34s
Test (windows-latest, 3.10)	6m 35s
Test (macos-13, 3.11)	4m 3s
Test (macos-13, 3.10)	2m 51s
Test (macos-latest, 3.11)	2m 0s
Test (macos-latest, 3.10)	2m 15s
Test (ubuntu-latest, 3.11)	2m 40s
Test (ubuntu-latest, 3.10)	2m 38s
Installer (ubuntu-20.04, 3.11)	9m 0s
Installer (ubuntu-22.04, 3.11)	10m 1s
Installer (windows-latest, 3.11)	18m 46s
Installer (macos-13, 3.11)	27m 38s
Installer (macos-latest, 3.11)	16m 18s
Test installer (windows-latest, 3.11)	2m 4s
Test installer (ubuntu-20.04, 3.11)	1m 33s
	2h 8m 13s

Provide feedback