portman/newman not returning nonzero exit code when using "abortOnFailure"=false

[chyde80]

I am running in a Jenkins pipeline and am seeing "collection run completed" when the run actually has failed tests. Why is the exit code not coming back correctly w/ an error? I have "abortOnFailure" set to false so that I can run the entire collection w/ tests before failing the pipeline stage. I can see in the console that newman/portman is reporting "Collection run completed" instead of what I would expect from Newman's docs which should throw "collection run encountered an error", which I expect would throw a non-zero exit code.

[chyde80]

@thim81 @nicklloyd ?

[chyde80]

I had to write a custom workaround for this - I published results as junit, published them in jenkins using the junit plugin, and then parsed the # of failed tests. If >0, I set the build to failed. It would be cleaner if portman could correctly pass back a nonzero exit code when there are failed tests even when the run completes. Any updates on this?

[thim81]

@chyde80 Can you try this PR, that should provide a better exit code?
#530

[thim81]

@chyde80 did you had the chance to test the PR?

[chyde80]

I don't have a convenient way of testing until it's merged into the product - please do so if you're comfortable and I'll let you know. Or, publish it to an alternate version on NPM and I'll test on that version instead of latest.