Update ExtractDomain to extract apex domains. #520
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Resolves #519 - Add scala-uri as a dependency - Replace getHost method of extracting domains with apexDomain from scala-uri - Update tests as needed - Removed unused source parameter from ExtractDomain
|
Well, this blows up in Spark 3.0.3 and 3.1.2 with:
|
No clue. Went back in the blame and looks like it's been there forever (maybe from when Alice was working on this five or six years ago). If I had to guess it was some janky error handling. I think yanking it makes sense. |
😞 (that sucks! and odd behaviour... ) |
|
I built the branch on |
My memory is faint on this, but - per docs, "source an optional default url for urls with no valid domain host". I think we've run into cases before where the full URL that we're trying to extract from was mangled. Usually, we're doing extraction on an URL on some page - so we set "source" to that page. Rationale is that if the URL is mangled, we can back up to the page that URL was found on. That was the original rationale. As to whether it's a good one, that's a different matter... |
- Caused by: io.lemonlabs.uri.parsing.UriParsingException: Invalid URL could not be parsed. Error(4,NonEmptyList(EndOfString(4,146)))
Codecov Report
@@ Coverage Diff @@
## main #520 +/- ##
============================================
+ Coverage 88.83% 89.04% +0.20%
Complexity 57 57
============================================
Files 43 43
Lines 1012 1022 +10
Branches 85 83 -2
============================================
+ Hits 899 910 +11
Misses 74 74
+ Partials 39 38 -1 |
|
Tested locally on Spark 3.1.1: I'm doing the same test on tuna now on the full geocities dataset, and everything seems to be running smoothly. I'm not 100% happy with my nest try and catch to swallow @ianmilligan1 @lintool any objects to me merging once the tuna job finishes, and cutting a dot release? |
ianmilligan1
approved these changes
Oct 21, 2021
There was a problem hiding this comment.
Apologies for the delay (was just on a call).
Looks good! Builds locally and sample scripts running on my end. I am happy for this to be merged when you're good to go @ruebot.
|
Ran without issue on tuna! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
GitHub issue(s): #519
What does this Pull Request do?
How should this be tested?
Additional Notes:
I removed the
sourceparameter since I honestly had no idea what it was for, or what the use case was. Couldn't really see anything in the commit history. @ianmilligan1 @lintool do y'all recall what it was for?Once this is merged, I'll cut a new release since we'll need it for ARCH.