Investigate and fix projects with no platform

[lobsterkatie]

Somehow, projects are either getting created with no platform (null or empty string) or their platform is getting overwritten with null or the empty string. As of this issue, this applies to ~27K projects.

To do:

• Add some logs and do some debugging and figure out why this is happening.
• Fix it.
• Once we're sure it's fixed, go and fill in platforms for the broken projects based on event and/or group data. This won't be perfect - some people send more than one kind of event to the same project - but it will likely be better than what they've got now. (We could perhaps choose to sample some number of groups from each project, and if they all agree on platform, assign that, and if they don't, shove the project into other.)

Here is a query showing the last time a project was created with each platform (the platforms are normalized quite a bit). The projects we're interested in are the unknown ones. Currently, it's the 7th most popular platform... not great.

[priscilawebdev]

Thanks for reporting this, @lobsterkatie ! This is an intriguing issue, and it got me curious.

After investigating, I found that projects created or updated via the Sentry user interface always include a platform, with 'other' as the default. The issue very likely occurs when users create projects by sending a POST request to Sentry using a token (e.g user auth token). According to the documentation, the platform field can be null.

it was possible to replicate the issue and noticed that, in most cases, users rely on infrastructure management tools like Terraform to create projects, which confirms my assumption.

I agree that users won't get the most out of Sentry if the platform for a project is null. Should we update the endpoint to require the platform field and validate that it matches one of the supported platforms?

[priscilawebdev]

On the other hand, these users are likely experienced with Sentry and know what they're doing. While other and null are essentially the same, they would only be choosing other explicitly. My concern is that they might not realize that specifying a platform would improve their UI experience

[priscilawebdev]

Regarding the fill-in platforms for broken projects, it’s probably fine to leave them as they are. As I mentioned, who created those projects are probably experienced with Sentry and created them outside of our UI.

[lobsterkatie]

Ahhh... I hadn't thought about API-based project creation. That makes a lot of sense.

In a perfect world, yes, I'd say that field should be required, though since we don't version our API I'm not sure how we'd go about making that change without breaking people's terraform scripts. And I agree that just defaulting to other isn't a great idea, because if we do ever want to fix/automatically set the value, we don't want to overwrite ones that are other on purpose.

I disagree about leaving existing projects as they are, though, for a few reasons:

• IMHO the goal be that should be that every project has a platform, both because it makes for a better user experience and because we rely on project platform for lots of analytics/queries/etc and it'd be nice to have more accurate data there.
• Just because someone is experienced with Sentry doesn't mean they know the ins and outs of all the places we use the platform value, so I don't necessarily buy a "they knew what they were doing, leave it alone" argument. In at least some cases I'm sure it was just someone who read our API docs, got it to work once, had no idea anything was amiss, and set the script and never looked at it again. There's also a good chance that in a company big enough to need to automate project creation, the infra people writing terraform scripts aren't the same people as the devs who are the primary Sentry users, so then they extra wouldn't appreciate that something's missing.
• In most cases I think we'd probably get it right. An interesting query to run would be to see how many of the broken projects have groups from only one platform. My guess is it'd be most of them. This isn't a perfect proxy for what we'd do to fix them (we should look at sdk from a sample of events rather than platform from groups, but groups are what we can easily query in redash) but it'd give us a good sense of how many people are mixing platforms in a single project, which is the only case in which we might get the fix "wrong."
• Just in case we do pick wrong, we could pop up a banner in those projects being like, "Heads up, we automatically picked a platform for you, go here to change it if you think we picked wrong."

Fixing existing projects wouldn't solve the problem of new broken projects getting created, but maybe we could turn it into a cron job, or maybe once we've fixed existing ones, we could add it to the "sent first event" code, so that going forward, projects without platforms would just always take their platform from the first event. Platform auto-detection could even be a feature we advertise.

TL;DR - Good job figuring out how this is happening, but I'm of the strong opinion that we should in fact fix the data. This ticket came out of a conversation with Matt in our grouping sync, so lemme bring it up again on Monday and see what he wants to do.