diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json index 02c89c8..1d35e43 100644 --- a/dev/.documenter-siteinfo.json +++ b/dev/.documenter-siteinfo.json @@ -1 +1 @@ -{"documenter":{"julia_version":"1.10.3","generation_timestamp":"2024-05-22T15:49:27","documenter_version":"1.4.1"}} \ No newline at end of file +{"documenter":{"julia_version":"1.10.3","generation_timestamp":"2024-05-22T17:54:54","documenter_version":"1.4.1"}} \ No newline at end of file diff --git a/dev/contributing/index.html b/dev/contributing/index.html index ed4db3f..c5b919a 100644 --- a/dev/contributing/index.html +++ b/dev/contributing/index.html @@ -1,2 +1,2 @@ -Contributing · TestReports.jl
GitHub

Contributing

`TestReports.jl follows the ColPrac guide for collaborative practices. New contributors should make sure to read that guide

+Contributing · TestReports.jl
GitHub

Contributing

`TestReports.jl follows the ColPrac guide for collaborative practices. New contributors should make sure to read that guide

diff --git a/dev/index.html b/dev/index.html index 130ac7d..bdba28f 100644 --- a/dev/index.html +++ b/dev/index.html @@ -30,4 +30,4 @@ </testcase> <testcase name="pass (info lost)" id="_testcase_id_"/> </testsuite> -</testsuites> +</testsuites> diff --git a/dev/library/index.html b/dev/library/index.html index 6d1641c..a71589b 100644 --- a/dev/library/index.html +++ b/dev/library/index.html @@ -1,6 +1,6 @@ Library · TestReports.jl
GitHub

Library

Contents

Public

Documentation for TestReports.jl's public interface.

TestReports.test — Function
TestReports.test(; kwargs...)
-TestReports.test(pkg::Union{AbstractString, Vector{AbstractString}; kwargs...)

Keyword arguments:

  • coverage::Bool=false: enable or disable generation of coverage statistics.
  • julia_args::Union{Cmd, Vector{String}}: options to be passed to the test process.
  • test_args::Union{Cmd, Vector{String}}: test arguments (ARGS) available in the test process.
  • logfilepath::AbstractString=pwd(): file path where test reports are saved.
  • logfilename::Union{AbstractString, Vector{AbstractString}}: name(s) of test report file(s).

Generates a JUnit XML for the tests of package pkg, or for the current project (which thus needs to be a package) if no positional argument is given to TestReports.test. The test report is saved in the current working directory and called testlog.xml if both logfilepath and logfilename are not supplied. If pkg is of type Vector{String}, the report filenames are prepended with the package name, for example Example_testlog.xml.

If logfilename is supplied, it must match the type (and length, if a vector) of pkg.

The tests are run in the same way as Pkg.test.

source
TestReports.RUNNER_SCRIPT — Constant

The path to the test runner script provided by TestReports. Similar to running TestReports.test but allows the user to specify any test file to run and not just test/runtests.jl.

Example

run(`$(TestReports.RUNNER_SCRIPT) mytests.jl --output=junit-report.xml`)
source
TestReports.record_testset_property — Function
record_testset_property(name::AbstractString, value)

Associates a property with the current testset. The name and value will be turned into a <property> element within the corresponding <testsuite> element within the JUnit XML report.

Multiple properties can be added to one testset and child testsets inherit the properties defined by their parents. If a child testset records a property which is already set both will be present in the resulting report.

The suggested use of this function is to place it inside a testset with unspecified type (see Examples). This will ensure that Pkg.test is unnaffected, but that the properties are added to the report when TestReports.test is used. This is because properties are only added when the Testset type has a TestReports.testset_properties method defined, as does the ReportingTestSet used by TestReports. TestReports.testset_properties can be extended for custom TestSets.

If a child testset has this method defined but its parent doesn't, the property should be in the report when TestReport.test is used, assuming that the parent testset type doesn't do anything to affect the reporting behaviour. However this is not tested functionality.

The value argument must be serializable by EzXML, which gives quite a lot of freedom.

Examples

Using the default testset for compatiblity with Pkg.test and TestReports.test:

using TestReports
+TestReports.test(pkg::Union{AbstractString, Vector{AbstractString}; kwargs...)

Keyword arguments:

  • coverage::Bool=false: enable or disable generation of coverage statistics.
  • julia_args::Union{Cmd, Vector{String}}: options to be passed to the test process.
  • test_args::Union{Cmd, Vector{String}}: test arguments (ARGS) available in the test process.
  • logfilepath::AbstractString=pwd(): file path where test reports are saved.
  • logfilename::Union{AbstractString, Vector{AbstractString}}: name(s) of test report file(s).

Generates a JUnit XML for the tests of package pkg, or for the current project (which thus needs to be a package) if no positional argument is given to TestReports.test. The test report is saved in the current working directory and called testlog.xml if both logfilepath and logfilename are not supplied. If pkg is of type Vector{String}, the report filenames are prepended with the package name, for example Example_testlog.xml.

If logfilename is supplied, it must match the type (and length, if a vector) of pkg.

The tests are run in the same way as Pkg.test.

source
TestReports.RUNNER_SCRIPT — Constant

The path to the test runner script provided by TestReports. Similar to running TestReports.test but allows the user to specify any test file to run and not just test/runtests.jl.

Example

run(`$(TestReports.RUNNER_SCRIPT) mytests.jl --output=junit-report.xml`)
source
TestReports.record_testset_property — Function
record_testset_property(name::AbstractString, value)

Associates a property with the current testset. The name and value will be turned into a <property> element within the corresponding <testsuite> element within the JUnit XML report.

Multiple properties can be added to one testset and child testsets inherit the properties defined by their parents. If a child testset records a property which is already set both will be present in the resulting report.

The suggested use of this function is to place it inside a testset with unspecified type (see Examples). This will ensure that Pkg.test is unnaffected, but that the properties are added to the report when TestReports.test is used. This is because properties are only added when the Testset type has a TestReports.testset_properties method defined, as does the ReportingTestSet used by TestReports. TestReports.testset_properties can be extended for custom TestSets.

If a child testset has this method defined but its parent doesn't, the property should be in the report when TestReport.test is used, assuming that the parent testset type doesn't do anything to affect the reporting behaviour. However this is not tested functionality.

The value argument must be serializable by EzXML, which gives quite a lot of freedom.

Examples

Using the default testset for compatiblity with Pkg.test and TestReports.test:

using TestReports
 
 # Default testset used, record property calls are ignored by `Pkg.test` but will be used
 # when generating JUnit XML.
@@ -22,25 +22,25 @@
     @test 2 != 1  # `<testcase>` has the property "bar"
 end;
 
-prettyprint(report(ts))

See also: record_test_property and testset_properties.

source
TestReports.record_test_property — Function
record_test_property(name::AbstractString, value)

Associates a property with the tests contained with the testset. The name and value will be turned into a <property> element with the corresponding <testcase> element within the JUnit XML report.

Multiple test properties can be assigned within a testset and child testsets will inherit the test properties defined by their parents. If a child testset records a test property with an already used name both properties will be present in the resulting report.

For more details and examples see the documentation for record_testset_property.

See also: record_testset_property and test_properties.

source
TestReports.ReportingTestSet — Type
ReportingTestSet

Custom AbstractTestSet type designed to be used by TestReports.jl for creation of JUnit XMLs.

Does not throw an error when a test fails or has an error. Upon finishing, a ReportingTestSet will display the default test output, and then flatten to a structure that is suitable for report generation.

It is designed to be wrapped around a package's runtests.jl file and this is assumed when both the test results are displayed and when the TestSet is flatted upon finish. See bin/reporttests.jl for an example of this use. ReportingTestSets are not designed to be used directly in a package's tests, and this is not recommended or supported.

A ReportingTestSet has the description and results fields as per a DefaultTestSet, and has the following additional fields:

  • testset_properties: used to record testset properties to be inserted into the report.
  • test_properties: used to record test properties to be inserted into the report.
  • start_time::DateTime: the start date and time of the testing (local system time).
  • time_taken::Millisecond: the time taken in milliseconds to run the TestSet.
  • last_record_time::DateTime: the time when record was last called.
  • hostname::String: the name of host on which the testset was executed.

See also: flatten_results!, record_testset_property, record_testset_property, and report.

source
TestReports.any_problems — Function
any_problems(ts)

Checks a testset to see if there were any problems (Errors or Fails). Note that unlike the DefaultTestSet, the ReportingTestSet does not throw an exception on a failure. Thus to set the exit code from the runner code, we check it using exit(any_problems(top_level_testset)).

source
TestReports.report — Function
report(ts::AbstractTestSet) -> XMLDocument

Produce an JUnit XML report details about the contained TestSets and Results. As the JUnit XML schema does not allow nested testsuite elements the report will flatten the hierarchical TestSet structure. Each TestSet will become a testsuite element and each Result will become a testcase element.

A Result will only be reported once within its parent TestSet to avoid having duplicate entries within the report and avoid problems with total test counts not matching Julia output.

All AbstractTestSets contained within ts must have a description::AbstractString field and an iterable results field.

source

Private

Package internals documentation.

Report Generation

TestReports.checkexitcode! — Method
checkexitcode!(errs, proc, pkg, logfilename)

Checks proc.exitcode and acts as follows:

  • If 0, displays tests passed info message
  • If equal to TESTS_FAILED const, warning is displayed and pkg added to errs
  • If anything else, throws a PkgTestError
source
TestReports.gen_runner_code — Method
gen_runner_code(testfilename, logfilename, test_args)

Returns runner code that will run the tests and generate the report in a new Julia instance.

source
TestReports.get_deps — Method
get_deps(manifest, pkg) = get_deps!(String[], manifest, pkg)

Get list of dependencies for pkg found in manifest

source
TestReports.gettestfilepath — Method
gettestfilepath(ctx::Context, pkgspec::Pkg.Types.PackageSpec)

Gets the testfile path of the package. Code for each Julia version mirrors that found in Pkg/src/Operations.jl.

source
TestReports.isinstalled! — Method
isinstalled!(ctx::Context, pkgspec::Pkg.Types.PackageSpec)

Checks if the package is installed by using ensure_resolved from Pkg/src/Types.jl. This function fails if the package is not installed, but here we wrap it in a try-catch as we may want to test another package after the one that isn't installed.

For Julia versions V1.4 and later, the first arguments of the Pkg functions used is of type Pkg.Types.Context. For earlier versions, they are of type Pkg.Types.EnvCache.

source
TestReports.runtests! — Method
runtests!(errs::Vector, pkg, cmd, logfilename)

Runs cmd which will run the tests of pkg. The exit code of the process is then checked.

source
TestReports.record_test_property — Function
record_test_property(name::AbstractString, value)

Associates a property with the tests contained with the testset. The name and value will be turned into a <property> element with the corresponding <testcase> element within the JUnit XML report.

Multiple test properties can be assigned within a testset and child testsets will inherit the test properties defined by their parents. If a child testset records a test property with an already used name both properties will be present in the resulting report.

For more details and examples see the documentation for record_testset_property.

See also: record_testset_property and test_properties.

source
TestReports.ReportingTestSet — Type
ReportingTestSet

Custom AbstractTestSet type designed to be used by TestReports.jl for creation of JUnit XMLs.

Does not throw an error when a test fails or has an error. Upon finishing, a ReportingTestSet will display the default test output, and then flatten to a structure that is suitable for report generation.

It is designed to be wrapped around a package's runtests.jl file and this is assumed when both the test results are displayed and when the TestSet is flatted upon finish. See bin/reporttests.jl for an example of this use. ReportingTestSets are not designed to be used directly in a package's tests, and this is not recommended or supported.

A ReportingTestSet has the description and results fields as per a DefaultTestSet, and has the following additional fields:

  • testset_properties: used to record testset properties to be inserted into the report.
  • test_properties: used to record test properties to be inserted into the report.
  • start_time::DateTime: the start date and time of the testing (local system time).
  • time_taken::Millisecond: the time taken in milliseconds to run the TestSet.
  • last_record_time::DateTime: the time when record was last called.
  • hostname::String: the name of host on which the testset was executed.

See also: flatten_results!, record_testset_property, record_testset_property, and report.

source
TestReports.any_problems — Function
any_problems(ts)

Checks a testset to see if there were any problems (Errors or Fails). Note that unlike the DefaultTestSet, the ReportingTestSet does not throw an exception on a failure. Thus to set the exit code from the runner code, we check it using exit(any_problems(top_level_testset)).

source
TestReports.report — Function
report(ts::AbstractTestSet) -> XMLDocument

Produce an JUnit XML report details about the contained TestSets and Results. As the JUnit XML schema does not allow nested testsuite elements the report will flatten the hierarchical TestSet structure. Each TestSet will become a testsuite element and each Result will become a testcase element.

A Result will only be reported once within its parent TestSet to avoid having duplicate entries within the report and avoid problems with total test counts not matching Julia output.

All AbstractTestSets contained within ts must have a description::AbstractString field and an iterable results field.

source

Private

Package internals documentation.

Report Generation

TestReports.checkexitcode! — Method
checkexitcode!(errs, proc, pkg, logfilename)

Checks proc.exitcode and acts as follows:

  • If 0, displays tests passed info message
  • If equal to TESTS_FAILED const, warning is displayed and pkg added to errs
  • If anything else, throws a PkgTestError
source
TestReports.gen_runner_code — Method
gen_runner_code(testfilename, logfilename, test_args)

Returns runner code that will run the tests and generate the report in a new Julia instance.

source
TestReports.get_deps — Method
get_deps(manifest, pkg) = get_deps!(String[], manifest, pkg)

Get list of dependencies for pkg found in manifest

source
TestReports.gettestfilepath — Method
gettestfilepath(ctx::Context, pkgspec::Pkg.Types.PackageSpec)

Gets the testfile path of the package. Code for each Julia version mirrors that found in Pkg/src/Operations.jl.

source
TestReports.isinstalled! — Method
isinstalled!(ctx::Context, pkgspec::Pkg.Types.PackageSpec)

Checks if the package is installed by using ensure_resolved from Pkg/src/Types.jl. This function fails if the package is not installed, but here we wrap it in a try-catch as we may want to test another package after the one that isn't installed.

For Julia versions V1.4 and later, the first arguments of the Pkg functions used is of type Pkg.Types.Context. For earlier versions, they are of type Pkg.Types.EnvCache.

source
TestReports.runtests! — Method
runtests!(errs::Vector, pkg, cmd, logfilename)

Runs cmd which will run the tests of pkg. The exit code of the process is then checked.

source
TestReports.test! — Method
test!(pkg::AbstractString,
       errs::Vector{AbstractString},
       nopkgs::Vector{AbstractString},
       notests::Vector{AbstractString},
       logfilename::AbstractString;
       coverage::Bool=false,
       julia_args::Union{Cmd, AbstractVector{<:AbstractString}}=``,
-      test_args::Union{Cmd, AbstractVector{<:AbstractString}}=``)

Tests pkg and save report to logfilename. Tests are run in the same way as Pkg.test.

If tests error pkg is added to nopkgs. If pkg has no testfile it is added to notests. If pkg is not installed it is added to nopkgs.

source

TestSets

TestReports.ReportingResult — Type
ReportingResult{T}

Wraps a Result of type T so that additional metadata can be saved.

fields

  • result::T: Result that is being wrapped.
  • time_taken::Millisecond: the time taken (milliseconds) for this test to be run.
source
TestReports.add_to_ts_default! — Method
add_to_ts_default!(ts_default::DefaultTestSet, result::Result)
+      test_args::Union{Cmd, AbstractVector{<:AbstractString}}=``)

Tests pkg and save report to logfilename. Tests are run in the same way as Pkg.test.

If tests error pkg is added to nopkgs. If pkg has no testfile it is added to notests. If pkg is not installed it is added to nopkgs.

source

TestSets

TestReports.ReportingResult — Type
ReportingResult{T}

Wraps a Result of type T so that additional metadata can be saved.

fields

  • result::T: Result that is being wrapped.
  • time_taken::Millisecond: the time taken (milliseconds) for this test to be run.
source
TestReports.add_to_ts_default! — Method
add_to_ts_default!(ts_default::DefaultTestSet, result::Result)
 add_to_ts_default!(ts_default::DefaultTestSet, result::ReportingResult)
 add_to_ts_default!(ts_default::DefaultTestSet, ts::AbstractTestSet)
-add_to_ts_default!(ts_default::DefaultTestSet, ts::ReportingTestSet)

Populate ts_default with the supplied variable. If result is a Result or an AbstractTestSet (but not a ReportingTestSet) then it is recorded. If it is a ReportingTestSet then a new DefaultTestSet with matching description is created, populated by recursively calling this function and then added to the results of ts_default. If result is a ReportingResult, the Result contained by the ReportingResult is added to ts_default.

source
TestReports.flatten_results! — Method
flatten_results!(ts::AbstractTestSet)

Returns a flat vector of TestSets which only contain Results. This is necessary for writing a JUnit XML report the schema does not allow nested XML testsuite elements.

source
TestReports.hostname — Method
hostname(ts::ReportingTestSet)
-hostname(ts::AbstractTestSet)

Get the hostname of a ReportingTestSet, returns gethostname() for an AbstractTestSet. Can be extended for custom TestSets, must return a string.

source
TestReports.ispass — Method
ispass(result)

Return true if result is a Pass or a ReportingResult{Pass}, otherwise return false.

source
TestReports.record_test_property! — Method
record_test_property!(ts::AbstractTestSet, name, value)

Associate a property with the tests within the testset. Can be extended for custom testsets.

source
TestReports.set_start_time! — Method
set_start_time!(ts::ReportingTestSet, start_time)
-set_start_time!(ts::AbstractTestSet, start_time)

Sets the start time field of a ReportingTestSet. This is used when flattening ReportingTestSets for report generation and an be extended for custom TestSets.

source
TestReports.set_time_taken! — Method
set_time_taken!(ts::ReportingTestSet, time_taken)
-set_time_taken!(ts::AbstractTestSet, time_taken)

Sets the time taken field of a ReportingTestSet. This is used when flattening ReportingTestSets for report generation and an be extended for custom TestSets.

source
TestReports.start_time — Method
start_time(ts::ReportingTestSet)
-start_time(ts::AbstractTestSet)

Get the start time of a ReportingTestSet, returns Dates.now() for an AbstractTestSet. Can be extended for custom TestSets, must return a DateTime.

source
TestReports.test_properties — Method
test_properties(ts::AbstractTestSet) -> Union{Vector{Pair{String, Any}}, Nothing}

Get the properties associated with tests within a testset. Can be extended for custom testsets. Will return nothing for testsets which do not support test properties.

When generating a JUnit XML report these will be the properties associated with all testcase elements contained within a testsuite element.

See also: testset_properties.

source
TestReports.testset_properties — Method
testset_properties(ts::AbstractTestSet) -> Union{Vector{Pair{String, Any}}, Nothing}

Get the properties associated with a testset. Can be extended for custom testsets. Defaults to nothing for testsets which do not support testset properties.

When generating a JUnit XML report these will be the properties associated with a testsuite element.

See also: test_properties.

source
TestReports.time_taken — Method
time_taken(ts::ReportingTestSet)
-time_taken(ts::AbstractTestSet)

Get the time taken of a ReportingTestSet, returns Dates.Millisecond(0) for an AbstractTestSet. Can be extended for custom TestSets, must return a Dates.Millisecond.

source
TestReports.time_taken — Method
time_taken(result::ReportingResult)
-time_taken(result::Result)

For a ReportingResult, return the time taken for the test to run. For a Result, return Dates.Millisecond(0).

source
TestReports.update_properties! — Method
update_properties!(childts::AbstractTestSet, ts::AbstractTestSet)

Adds properties of ts to childts.

If the types of ts and/or childts do not a method defined for properties, this is handled as follows:

  • If method not defined for typeof(ts), it has no properties to add to childts and therefore nothing happens.
  • If method not defined for typeof(childts) and ts has properties, then a warning is shown.

See also: testset_properties and test_properties.

source

XML Writing

TestReports.add_properties! — Method
add_properties!(x_element, properties)

Add all key value pairs defined within the properties to the referenced XML element.

source
TestReports.check_ts — Method
check_ts(ts::AbstractTestSet)

Throws an exception if ts does not have the right structure for report or if the results of ts do not have both description or results fields.

See also: report

source
TestReports.error_xml — Method
failure_xml(message, test_type, content)

Create an error node (which will be the child of a testcase).

source
TestReports.format_period_for_xml — Method

formatperiodfor_xml(time::Millisecond)

Formats a millisecond value into a string in seconds with 3 decimal places.

@sprintf is used to ensure that value does not use scientific notification, which is not allowed in an XML decimal.

Methods for other Periods have not been written as are not currently required.

source
TestReports.get_error_info — Method
get_error_info(v::Error)

Return message and type of error for testcase attribute, and number of tests (either 1 or 0). Uses test_type field to determine what caused the original error.

source
TestReports.get_failure_message — Method

getfailuremessage(v::Fail)

Return message for failed test testcase attribute. Uses test_type field to determine what caused the original failure.

source
TestReports.set_attribute! — Method
set_attribute!(node, attr, val)
-set_attribute!(node, attr, val::Period)

Add the attritube with name attr and value val to node.

If val is of type Period, format with format_period_for_xml.

source
TestReports.testcase_xml — Method
testcase_xml(name, id, x_children)

Create a testcase element of a JUnit XML.

This is the generic form (with name, id and children) that is used by other methods.

source
TestReports.testcase_xml — Method
testcase_xml(v::Result, childs)

Create a testcase element of a JUnit XML for the result v.

The original expression of the test is used as the name, whilst the id is defaulted to testcaseid_.

source
TestReports.to_xml — Method
to_xml(ts::AbstractTestSet)

Create a testsuite node from a AbstractTestSet, by creating nodes for each result in ts.results. For creating a JUnit XML, all results must be AbstractResults, that is they cannot be AbstractTestSets, as the XML cannot have one testsuite nested inside another.

source
TestReports.to_xml — Method
to_xml(res::Pass)
+add_to_ts_default!(ts_default::DefaultTestSet, ts::ReportingTestSet)

Populate ts_default with the supplied variable. If result is a Result or an AbstractTestSet (but not a ReportingTestSet) then it is recorded. If it is a ReportingTestSet then a new DefaultTestSet with matching description is created, populated by recursively calling this function and then added to the results of ts_default. If result is a ReportingResult, the Result contained by the ReportingResult is added to ts_default.

source
TestReports.flatten_results! — Method
flatten_results!(ts::AbstractTestSet)

Returns a flat vector of TestSets which only contain Results. This is necessary for writing a JUnit XML report the schema does not allow nested XML testsuite elements.

source
TestReports.hostname — Method
hostname(ts::ReportingTestSet)
+hostname(ts::AbstractTestSet)

Get the hostname of a ReportingTestSet, returns gethostname() for an AbstractTestSet. Can be extended for custom TestSets, must return a string.

source
TestReports.ispass — Method
ispass(result)

Return true if result is a Pass or a ReportingResult{Pass}, otherwise return false.

source
TestReports.record_test_property! — Method
record_test_property!(ts::AbstractTestSet, name, value)

Associate a property with the tests within the testset. Can be extended for custom testsets.

source
TestReports.set_start_time! — Method
set_start_time!(ts::ReportingTestSet, start_time)
+set_start_time!(ts::AbstractTestSet, start_time)

Sets the start time field of a ReportingTestSet. This is used when flattening ReportingTestSets for report generation and an be extended for custom TestSets.

source
TestReports.set_time_taken! — Method
set_time_taken!(ts::ReportingTestSet, time_taken)
+set_time_taken!(ts::AbstractTestSet, time_taken)

Sets the time taken field of a ReportingTestSet. This is used when flattening ReportingTestSets for report generation and an be extended for custom TestSets.

source
TestReports.start_time — Method
start_time(ts::ReportingTestSet)
+start_time(ts::AbstractTestSet)

Get the start time of a ReportingTestSet, returns Dates.now() for an AbstractTestSet. Can be extended for custom TestSets, must return a DateTime.

source
TestReports.test_properties — Method
test_properties(ts::AbstractTestSet) -> Union{Vector{Pair{String, Any}}, Nothing}

Get the properties associated with tests within a testset. Can be extended for custom testsets. Will return nothing for testsets which do not support test properties.

When generating a JUnit XML report these will be the properties associated with all testcase elements contained within a testsuite element.

See also: testset_properties.

source
TestReports.testset_properties — Method
testset_properties(ts::AbstractTestSet) -> Union{Vector{Pair{String, Any}}, Nothing}

Get the properties associated with a testset. Can be extended for custom testsets. Defaults to nothing for testsets which do not support testset properties.

When generating a JUnit XML report these will be the properties associated with a testsuite element.

See also: test_properties.

source
TestReports.time_taken — Method
time_taken(ts::ReportingTestSet)
+time_taken(ts::AbstractTestSet)

Get the time taken of a ReportingTestSet, returns Dates.Millisecond(0) for an AbstractTestSet. Can be extended for custom TestSets, must return a Dates.Millisecond.

source
TestReports.time_taken — Method
time_taken(result::ReportingResult)
+time_taken(result::Result)

For a ReportingResult, return the time taken for the test to run. For a Result, return Dates.Millisecond(0).

source
TestReports.update_properties! — Method
update_properties!(childts::AbstractTestSet, ts::AbstractTestSet)

Adds properties of ts to childts.

If the types of ts and/or childts do not a method defined for properties, this is handled as follows:

  • If method not defined for typeof(ts), it has no properties to add to childts and therefore nothing happens.
  • If method not defined for typeof(childts) and ts has properties, then a warning is shown.

See also: testset_properties and test_properties.

source

XML Writing

TestReports.add_properties! — Method
add_properties!(x_element, properties)

Add all key value pairs defined within the properties to the referenced XML element.

source
TestReports.check_ts — Method
check_ts(ts::AbstractTestSet)

Throws an exception if ts does not have the right structure for report or if the results of ts do not have both description or results fields.

See also: report

source
TestReports.error_xml — Method
failure_xml(message, test_type, content)

Create an error node (which will be the child of a testcase).

source
TestReports.format_period_for_xml — Method

formatperiodfor_xml(time::Millisecond)

Formats a millisecond value into a string in seconds with 3 decimal places.

@sprintf is used to ensure that value does not use scientific notification, which is not allowed in an XML decimal.

Methods for other Periods have not been written as are not currently required.

source
TestReports.get_error_info — Method
get_error_info(v::Error)

Return message and type of error for testcase attribute, and number of tests (either 1 or 0). Uses test_type field to determine what caused the original error.

source
TestReports.get_failure_message — Method

getfailuremessage(v::Fail)

Return message for failed test testcase attribute. Uses test_type field to determine what caused the original failure.

source
TestReports.set_attribute! — Method
set_attribute!(node, attr, val)
+set_attribute!(node, attr, val::Period)

Add the attritube with name attr and value val to node.

If val is of type Period, format with format_period_for_xml.

source
TestReports.testcase_xml — Method
testcase_xml(name, id, x_children)

Create a testcase element of a JUnit XML.

This is the generic form (with name, id and children) that is used by other methods.

source
TestReports.testcase_xml — Method
testcase_xml(v::Result, childs)

Create a testcase element of a JUnit XML for the result v.

The original expression of the test is used as the name, whilst the id is defaulted to testcaseid_.

source
TestReports.to_xml — Method
to_xml(ts::AbstractTestSet)

Create a testsuite node from a AbstractTestSet, by creating nodes for each result in ts.results. For creating a JUnit XML, all results must be AbstractResults, that is they cannot be AbstractTestSets, as the XML cannot have one testsuite nested inside another.

source
TestReports.to_xml — Method
to_xml(res::Pass)
 to_xml(res::Fail)
 to_xml(res::Broken)
 to_xml(res::Error)
-to_xml(res::ReportingResult)

Create a testcase node from the result and return it along with information on number of tests.

source

Index

+to_xml(res::ReportingResult)

Create a testcase node from the result and return it along with information on number of tests.

source

Index

diff --git a/dev/manual/index.html b/dev/manual/index.html index fd600da..a19b190 100644 --- a/dev/manual/index.html +++ b/dev/manual/index.html @@ -61,4 +61,4 @@ <testsuite name="TopLevel" tests="1" failures="0" errors="0" time="0.156" timestamp="2022-03-30T07:55:12.173" hostname="hostname" id="2"> <testcase name="1 == 1" id="1" classname="TopLevel" time="0.000"/> </testsuite> -</testsuites>

Each test is recorded in a separate testsuite with the name showing the original nesting.

Custom TestSet Types

TestReports.jl has not been tested significantly with custom TestSet types, please raise an issue if you find any problems/have a request.

However at a minimum, for a custom TestSet type to work with TestReports it must:

The following information in a JUnit XML relies on the functionality of ReportingTestSets but can be added to your own custom TestSet as described in the table.

InformationDescription
testcase timeThis is extracted from a ReportingResult by the TestReports.time_taken function. For standard Results, rather than ReportingResults, this function returns Dates.Millisecond(0). This function can be extended for other custom Result types.
testsuite timeThis is extracted from a TestSet by the TestReports.time_taken function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to Dates.Millisecond(0).
testsuite timestampThis is extracted from a TestSet by the TestReports.start_time function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to Dates.now().
testsuite hostnameThis is extracted from a TestSet by the TestReports.hostname function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to gethostname().
testsuite propertiesThis is extracted from a TestSet by the TestReports.properties function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to nothing.

For further details on extending these fuctions, see the docstrings in TestSets.

The source code of TestReports can be used as a starting point for including this behaviour in your custom TestSets.

If no TestSet types are specified (as per the standard Test approach), TestSet functionality will ensure that all child TestSets inherit the ReportingTestSet type.

+</testsuites>

Each test is recorded in a separate testsuite with the name showing the original nesting.

Custom TestSet Types

TestReports.jl has not been tested significantly with custom TestSet types, please raise an issue if you find any problems/have a request.

However at a minimum, for a custom TestSet type to work with TestReports it must:

The following information in a JUnit XML relies on the functionality of ReportingTestSets but can be added to your own custom TestSet as described in the table.

InformationDescription
testcase timeThis is extracted from a ReportingResult by the TestReports.time_taken function. For standard Results, rather than ReportingResults, this function returns Dates.Millisecond(0). This function can be extended for other custom Result types.
testsuite timeThis is extracted from a TestSet by the TestReports.time_taken function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to Dates.Millisecond(0).
testsuite timestampThis is extracted from a TestSet by the TestReports.start_time function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to Dates.now().
testsuite hostnameThis is extracted from a TestSet by the TestReports.hostname function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to gethostname().
testsuite propertiesThis is extracted from a TestSet by the TestReports.properties function, which can be extended for custom TestSets. If not extended, the AbstractTestSet method will be used and the value defaults to nothing.

For further details on extending these fuctions, see the docstrings in TestSets.

The source code of TestReports can be used as a starting point for including this behaviour in your custom TestSets.

If no TestSet types are specified (as per the standard Test approach), TestSet functionality will ensure that all child TestSets inherit the ReportingTestSet type.