[CODE-874] New uploader is unable to find and upload coverage reports from Xcode

[chris-araman]

Describe the bug
The new uploader fails to find coverage reports from Xcode.

To Reproduce

1. git clone https://github.com/chris-araman/CombineCloudKit.git
2. cd CombineCloudKit
3. swift package generate-xcodeproj
4. xcodebuild test -enableCodeCoverage YES -scheme CombineCloudKit-Package
5. bin/codecov -n -Q github-action-2.0.1 -Z

Expected behavior
The coverage report is found and uploaded successfully.

Additional context
codecov-action@v2 uses the new uploader. codecov-action@v1 used the old bash uploader. The new uploader fails to find coverage reports from Xcode when the bash uploader found them successfully. I encountered this issue when attempting to migrate from codecov-action@v1 to @v2.

It seems this functionality was implemented in the bash uploader here:
https://github.com/codecov/codecov-bash/blob/ae76691c8554334bc1ddc355f6667fd36d6890f5/codecov#L1148

There are notes in #15 about wanting to keep this project a "pure uploader". I think separating the concerns of the uploader from the concerns of something processing reports is admirable. However, I think this sort of boilerplate logic is useful to users of the uploader. Perhaps it can be ported to a more coverage-friendly environment (Node.js, Typescript) alongside the uploader functionality, even if it's in a separate module?

This issue is preventing me from migrating to the new uploader via codecov-action@v2.

[drazisil-codecov]

I wonder if writing the llvm-cov commands as a Fastlane plugin would be an option.

I'm not an Apple programmer, so I don't know the % of XCode programers that use Fastlane as well.

[chris-araman]

It's a good idea. It might be useful to use a small part of Fastlane or another plug-in in this way. It's just odd that the Codecov action used to offer this functionality and no longer does.

Fastlane is great for projects that build iOS apps. I don't think it's commonly used for macOS or other apps. And in the case of CombineCloudKit, it's just a Swift package (something like a library or framework), not an app at all. I don't think Fastlane is commonly used for things that aren't iOS apps.

[ffried]

I wonder if writing the llvm-cov commands as a Fastlane plugin would be an option.

It would help some, but certainly not enough developers. We're not using fastlane for example and we have a ton of swift packages that are now failing to produce coverage reports.

To mitigate the issue temporarily, I've written an swift-coverage-action that performs the tasks previously done by the bash uploader. There are certain features missing (e.g. the -J parameter), but it should do the trick most of the time.

An example of how to use it can be found here.

I still hope to retire that action in the near future, though. 😉

[drazisil-codecov]

I still hope to retire that action in the near future, though.

This is great from a GutHub Actions standpoint. Is your desire to retire it due to being an extra action, or due to maintenance or some other cause?

[drazisil-codecov]

t's just odd that the Codecov action used to offer this functionality and no longer does.

I agree that we could do more in this regard. You saw my notes on wanting to seperate it, but I agree that we need to do better at a replacement.

Fastlane is great for projects that build iOS apps. I don't think it's commonly used for macOS or other apps.

That's a great point. As a non-Apple dev, I wasn't thinking that Fastlane was IOS only. Does XCode in general have something similar in it's ecosystem? IS Cocoapods also mainly IOS?

[ffried]

This is great from a GitHub Actions standpoint. Is your desire to retire it due to being an extra action, or due to maintenance or some other cause?

Currently, it's mostly because the actions is (in its current state) a rather quick re-write of the swiftcov function and its call-site in the bash uploader.

It's now a separate action with a rather limited use-case (I doubt it'll work with anything else other than Codecov tbh).
Of course I could extend it to cover more use-cases. Like also converting files on linux or converting the results of a swift test run inside a swift package. Or supporting more formats (e.g. lcov). I might even do so in the future (especially since we're having the swift test case in our actions).

It being a separate action means that those features will (sooner or later) be requested. That's something that is less likely to occur if the Codecov action "simply works" - since (until now) no one really cared about what more it could do - swift test being maybe the only exception.

[ffried]

That's a great point. As a non-Apple dev, I wasn't thinking that Fastlane was IOS only. Does XCode in general have something similar in it's ecosystem? IS Cocoapods also mainly IOS?

CocoaPods is "just" one of three major dependency management systems inside the Apple ecosystem: Swift Package Manager (being the youngest), Carthage and CocoaPods (being the oldest).
I think Swift Package Manager will eventually supersede the others, but that's just my two cents.

There are (binary) xcresult bundles since Xcode 11. I haven't spent too much time with them so far, but AFAIK they also contain coverage information. There's also a developer tool xcresulttool that can be used to interact with them.
The advantage of those bundles is, that you can open them in Xcode and see the results there as well.

[chris-araman]

To mitigate the issue temporarily, I've written an xcode-coverage-action that performs the tasks previously done by the bash uploader.

@ffried, would it be possible to create a PR on codecov/uploader with your code?

[ffried]

@ffried, would it be possible to create a PR on codecov/uploader with your code?

I just had a brief look at the code base of the uploader. One thing I noticed is that it's pure JavaScript, while the action is written in TypeScript. I could probably start with the compiled version of it and adjust it to look a bit more "natural"...
The other thing I'm not sure about is the extra arguments that come with it:

• derived data directory (previously -D)
• app name regex (previously -J)

[drazisil-codecov]

As an action, I'm thinking it may fit better at https://github.com/codecov/codecov-action, though I don't want to speak for @thomasrockhu

💡 goes off to create some feature dreams issues

[ncreated]

👋 Stepping across this issue, as I'm trying to integrate codecov into our iOS SDK repository 🙂. As I understand, the current situation is:

• "bash uploader" is deprecated (will stop functioning in Feb 2022);
• "new uploader" is the replacement.

I'm testing the new uploader 0.1.4(latest, macos) and it doesn't support Xcode coverage. What I read in this thread is that it will rather not support it - as described in #15 the Xcode functionality was removed.

• Is there any new / recommended approach for sending Xcode coverage file or do I need pre-process it by my own as other folks did in this thread?

Lack of Xcode coverage support seems to be a stalemate situation, as the official codecov-bitrise depends on the new uploader now (cc @thomasrockhu-codecov) and in this thread I read that it will not work.

Am I missing any puzzle 🧩🙂?