Test generator

[angelikatyborska]

This issue is to collect requirements and ideas for a test generator. We don't have a specific implementation plans yet.

A test generator would a script or project in this repository that:

1. Reads the tests.toml file from a practice exercise and chooses test ids that are not marked with include = false
2. Fetches the newest https://github.com/exercism/problem-specifications/blob/main/exercises/[exercise]/canonical-data.json and generates a test suite for the tests chosen from tests.toml

The process of updating tests.toml to include the newest tests cases is already automated with bin/configlet sync. The assumption is that we would run the test runner after bin/configlet sync.

It would be fine if the test generator didn't work for all exercises, but at least some of them. It would also be fine if it required a bit of manual work, or maybe per-exercise configuration stored in exercises/practice/[exercise]/.meta/.

Previous attempts:

• (WIP) start work on generator #521
• https://github.com/jiegillet/elixir/blob/92ba4784e6d0cd67da7e0bc730e04825a7b65d91/exercises/practice/complex-numbers/generate_tests.exs

[angelikatyborska]

An example of an exercise that might be problematic for the test generator: kindergarten-garden implements a few test cases from tests.toml in a single test block:

elixir/exercises/practice/kindergarten-garden/.meta/tests.toml

Lines 20 to 30 in 4ab39ca

	[149b4290-58e1-40f2-8ae4-8b87c46e765b]
	description = "first student's garden"
	[ba25dbbc-10bd-4a37-b18e-f89ecd098a5e]
	description = "second student's garden"
	[6bb66df7-f433-41ab-aec2-3ead6e99f65b]
	description = "second to last student's garden"
	[d7edec11-6488-418a-94e6-ed509e0fa7eb]
	description = "last student's garden"

elixir/exercises/practice/kindergarten-garden/test/garden_test.exs

Lines 42 to 57 in 4ab39ca

	@tag :pending
	test "gets the garden for all students" do
	garden_info = Garden.info("VRCGVVRVCGGCCGVRGCVCGCGV\nVRCCCGCRRGVCGCRVVCVGCGCV")
	assert garden_info.alice == {:violets, :radishes, :violets, :radishes}
	assert garden_info.bob == {:clover, :grass, :clover, :clover}
	assert garden_info.charlie == {:violets, :violets, :clover, :grass}
	assert garden_info.david == {:radishes, :violets, :clover, :radishes}
	assert garden_info.eve == {:clover, :grass, :radishes, :grass}
	assert garden_info.fred == {:grass, :clover, :violets, :clover}
	assert garden_info.ginny == {:clover, :grass, :grass, :clover}
	assert garden_info.harriet == {:violets, :radishes, :radishes, :violets}
	assert garden_info.ileana == {:grass, :clover, :violets, :clover}
	assert garden_info.joseph == {:violets, :clover, :violets, :grass}
	assert garden_info.kincaid == {:grass, :clover, :clover, :grass}
	assert garden_info.larry == {:grass, :violets, :clover, :violets}
	end

[jiegillet]

Is there any guarantee that all the tests in tests.toml have been implemented? Maybe another function of the tool could be checking that, probably from the test names. It would also be problematic in the case of kindergarten-garden though.

You're right to keep expectations realistic, there are going to be load of exceptions :)

[angelikatyborska]

Is there any guarantee that all the tests in tests.toml have been implemented?

There is a guarantee that a test in tests.toml will be defined in canonical-data.json, but not in our (Elixir) implementation of the exercise. The whole idea for adding new tests cases is that you first run configlet which just adds them to tests.toml, and then you can use the to-be-implemented generator to actually add them, somehow, to *_test.exs files.

About this, there are a lot of exercises where Elixir added some extra tests cases that don't exist in problem specifications. I think the generator shouldn't try to remove those, but I should try to remove tests listed in tests.toml with include = false 🤔. In a perfect world 😁.

[jiegillet]

Elixir 1.13 will have Code.string_to_quoted_with_comments/2 and Code.quoted_to_algebra/2, that might make it easier to insert new test cases.

[antoine-duchenet]

Is this topic still relevant ?

In the ongoing implementation of bottle-song, we did choose to use multi-line strings as output instead of the array / list of strings specified by the JSON specs.

Since JSON may be too poor to capture subtleties like this one, maybe some strongly structured comments in the TOML (the comments key of the specs, not regular ones indeed) could be used to pass options (something like gen::output::prefer-multiline-string) ? It should be considered as a "label" but I did not see such a key / feature in the current TOML specs structure.

Concerning the identification of extra cases vs specs cases (and include = false handling), is it conceivable to create a module attribute (similar to the @tag of ExUnit on some points) to keep track of the "link" ? Something like this :

@uuid "2c698d68-b39d-4615-817f-2cfacc9aa318"
@tag :pending
test "Lorem ipsum" do
...

A downside of this approach is that this @uuid thing is presented to the student. In fact they could delete it without harm but still, we are showing them something that they should not care about ! I do not feel like it is a big deal compared to the potential benefits though.

[angelikatyborska]

Sorry for the lack of response...

It's been over 2 years since I created this issue and I have to admit that my need for a test generator is very low. The work it takes to occasionally update an Exercise manually is not a lot, and there aren't many new exercises coming up that would need creating with a test generator.

@jiegillet If you don't mind, I would close this issue.

[jiegillet]

Yes, I think this can be closed.
The amount a work required for a generator that would cleanly do test updates is way more than updating by hand. And new exercises can use the current test generator. It's not perfect, but it works :)