Integration with Cake.Issues

[pascalberger]

First of all thank you for the great app.

I've written a bunch of addins for Cake called Cake.Issues which can be used to collect code analysis issues from different linters (like MsBuild warnings, R#, Markdownlint, ...). It contains also an API for writing them to pull requests. The only current implementation is for VSTS/TFS and I really would like to provide also an integration for GitHub.

The most basic implementation for a pull request system just posts an IEnumerable<IIssue> (where IIssue is the issue abstraction in Cake.Issues) to the pull request.

Do you think it would be possible to enhance and use your application so that it also can be used by the Cake.Issues addins and would you be interested in this?

[StanleyGoldman]

Hey @pascalberger.
Thanks for checking out our stuff. Cake.Issues is pretty impressive as well.

So I've only just started learning Cake, because I want to be able to tell people how to integrate our service when using something like Cake (https://github.com/cake-build/cake/pull/1792/files). From what I understand Cake gives you hooks into the entire build system as well as additional tasks and executables.

This application is based entirely on the msbuild binary log file. I'm using that to collect the msbuild warnings and errors, and hopefully more in the future.

Basically you have a much richer dataset about different build steps available to you, whereas I only have information that comes in from msbuild. I've only thought about it for a few minutes, but looking at your program gives me a few ideas of where we can at least learn from each other...

For MSBuildLogOctokitChecker:

1. Look into docFx. There is an msbuild task for it.

For Cake.Issues:

1. Fork/Port TBD logic that gets information from msbuild binlog files:
   You already have build warnings and errors from the console output in Cake, so you don't need to get that from the binlog files, but there are other things that we can figure out from binlog files. I'm trying to get a list of them here #87.
2. Consider forking Cake.Issues or the individual AddIns to be GitHub apps:
   The model of getting data into GitHub Apps would be a bit different from how your users of Cake.Issues would do it. In order to write this detailed data you are going to need to host a web service. We are basically taking captured output from the build step and sending it to a web service. Which is not too dissimilar from what your Cake addins do, (the capturing details from the build output part...). You could fork this project to take some other output from a build step and write it to the GitHub Checks API.
   • But there could also be an alternative approach, Using the markdownlint addin as an example. It may be easier to create a nodejs webservice that runs markdownlint directly, and checks all files that were changed in a pull request, instead of the passive, "send me log messages" approach that I'm taking.

My bad, I might've rambled..

[StanleyGoldman]

Let me know if my rambling makes sense.. I can try again...

[pascalberger]

@StanleyGoldman Thanks for getting back.

The idea behind Cake.Issues was to abstract issue handling in a build from concrete implementations making it easily enhancable for additional use cases and allows switching systems. Issues can come from different issue providers (MsBuild being one, there's also a PR for binlog support which currently waits for an update of the NuGet package: cake-contrib/Cake.Issues.MsBuild#38), pull requests can be reported to different pull request systems (with TFS being the only current implementation) or reports can be created (with a Razor template based approach being the only implementation. The implementation of a new issue provider will add instant support for the specific linter to create reports or write issues to pull requests.

What I see as a possibility between our projects is that Cake.Issue provides an abstraction layer over different linters while MsBuildLogOctokitChecker provides an integration to GitHub. On the other hand there's some duplicated effort (like bin log parsing).

Cake.Issues is currently bound to Cake and there is some Cake specific stuff which is used throughout the code (like e.g. Cakes logging) but most of the code is not really specific to Cake (like e.g. parsing for issues, which does not parse Cake output, but parses a file or string). So I can also imagine to split the generic code into some Cake independent NuGet packages, which then would provide the abstraction layer for issues (IIssue) and do the parsing. Cake.Issues would consume these addins for it's logic and provide Cake specific convenience aliases for calling the code. This approach for example would allow you to consume the same libraries for your project, which itself only would need to handle the GitHub integration but not the parsing stuff.

If you plan to integrate with Cake it would be great if we can do this through Cake.Issues as this would make your app also usable for other linters. If we use the same IIssue abstraction it would mean that if you can provide an endpoint accepting an IEnumerable that this can be called from Cake issues (or a new Cake.Issues.PullRequests.GitHub addin). And putting some logic into common NuGet packages would you allow to still provide endpoints which can be consumed from outside of Cake, while sharing effort on the parsing side.

Hope that makes sense :)

[StanleyGoldman]

Maybe we create a custom reporter (json?) for Cake.Issues to integrate with MSBuildLogOctokitChecker?
A user could send our web service the output from that reporter in Cake and we could figure out how to process it and create CheckRun data in GitHub with MSBuildLogOctokitChecker.

[StanleyGoldman]

Am I thinking about it the right way?

[pascalberger]

I see two ways of integrating:

1. Create a generic endpoint which accepts a json as you've suggested with an additional Cake.Issues.PullRequests.GitHub addin, which takes an IEnumerable, converts it to the json and reports it to MSBuildLogOctokitChecker. This should be quite straight forward and easy to implement on Cake.Issue side

2. Split issue provider abstraction logic into NuGet packages independent from Cake and provide different parsers which return an IEnumerable. Cake.Issue would consume them, as would MSBuildLogOctokitChecker or possible other tools. This would additionally to allow share parsing logic, but would be more effort to get there.

Of course it's also possible to start with 1. and work towards 2.

[StanleyGoldman]

We can definitely start with 1.

The output we are looking to create would be Check Runs and Annotations. Where the Check Run is the "linter" and the Annotations are the "individual reports" from the "linter".

MSBLOC's role in that case, would be providing users a secure way, (through means of our token), to send Check Runs and Annotations into GitHub.

[pascalberger]

Yes, the secure way to communicate to GitHub is the missing part in Cake.Issues at the moment. So sounds good for me 👍

[pascalberger]

And I'm also definitely interested in the work of https://github.com/justaprogrammer/MSBuildLogOctokitChecker/issues/87.

Does #50 mean to find a way to find out which issues are new for a specific build, compared to which ones already existed in previous builds? This is also a topic which would be interested to provide in Cake.Issues (you could do this currently with some custom code in your Cake build script, but it would be nice to provide default implementations for different build systems which can provide log files of previous builds)

[StanleyGoldman]

Re: #87 Sounds good. These should be things that you can definitely implement in Cake.Issues.AddIns

Re: #50 Was really about some devops work in this project. Seeing if we can get a nice way to get assembly version in our code here to be reflected in AppVeyor and deployments.

[pascalberger]

@StanleyGoldman FYI, I setup a project for GitHub integration in Cake.Issues.

https://github.com/cake-contrib/Cake.Issues.PullRequests.GitHub/blob/develop/src/Cake.Issues.PullRequests.GitHub/GitHubPullRequestSystem.cs#L31 would be the function which needs to be implemented with the call to the app.

[StanleyGoldman]

Hey @pascalberger so a lot has changed since this was called MSBuildLogOctokitChecker.
Take some time and read through the documentation.
If you follow how this all works you should be able to create Check Run data from your toolset.

Let me know how it all works for you, and then we can look into extending it to create issues or something else if you think it'd be useful and used.

[pascalberger]

@StanleyGoldman Thanks 👍! I'll give it a shot to integrate the submission tool into Cake.Issues.

If I understand it correctly you'll need to pass it a token. So there's no way to call the tool in a safe way for PRs in public repos without exposing the token (e.g. using AppVeyor)? Or do you have an approach how to handle this case?

[StanleyGoldman]

Yea, I have more work to do.

I need to identify the build server and job and look patch that into the web service for it to validate somehow. If people want to use this service, then I can figure it out as the demand grows.

[gep13]

@StanleyGoldman if would be great if you can make this happen!

[StanleyGoldman]

@gep13 @pascalberger I opened #147

I'm not 100% sure of the approach at this point. Let me know if the general outline makes sense to you guys and please weigh in if you think of something.