Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(commands): add resume on error for comments and emails #291

Merged
merged 1 commit into from
Jul 26, 2024

Conversation

joe-prosser
Copy link
Contributor

Adds a --resume-on-error flag for uploading comments and emails. When the api returns an error the CLI will split a particular batch into individual requests, attempt each one individually and skip any individual request that fail

@joe-prosser joe-prosser requested a review from dan-ri July 25, 2024 10:41
@joe-prosser joe-prosser self-assigned this Jul 25, 2024
api/src/lib.rs Outdated
Method::PUT,
self.endpoints.put_comments(source_name)?,
PutCommentsRequest {
comments: comments.to_vec(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because you're taking the input comments and just converting it to Vec, it would be better to declare these functions as taking a Vec instead of a slice.

api/src/lib.rs Outdated Show resolved Hide resolved
api/src/lib.rs Outdated
debug!("Attempting {} `{}`", method, url);
let result: Result<SuccessT> = self.request(
method.clone(),
url.clone(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of this cloning looks as if it should be unnecessary, e.g. IntoUrl is implemented for &str.

Comment on lines +1324 to +1331
match result {
Ok(response) => Ok(SplitableRequestResponse {
response,
num_failed: 0,
}),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice not to have to write the main body of code for this function nested inside a match branch and a conditional. Something like:

match result {
    Ok(response) => return Ok(SplitableRequestResponse {
                response,
                num_failed: 0,
            }),
    Err(error) if !Self::should_split(&error) => return Err(error),
    _ => {},
}
...

api/src/lib.rs Outdated
@@ -1250,6 +1401,7 @@ impl Client {
}
}

#[derive(Clone)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For small structs and enums (those with size <= 8 bytes, say), Copy should also be derived, as they're cheap to copy. Then clone()ing them is unnecessary.

Self {
new: a_new + b_new,
updated: a_updated + b_updated,
unchanged: a_unchanged + b_unchanged,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just write this as a.new + b.new etc. and omit let bindings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the let binding so that we would get a compiler error if new fields were added (to remind us to make any necessary changes here)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than do that, you can just put a single statement like this at the top:

// Ensure new fields are added here:
let Self { new: _, updated: _, unchanged: _ } = a;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - just pushed a slightly different implementation

Comment on lines 200 to 205
fn empty() -> Self {
Self {}
}
fn merge(_a: Self, _b: Self) -> Self {
Self {}
}
Copy link
Contributor

@dan-ri dan-ri Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can supply a default implementation for trait methods in the trait itself that will be used if there's no implementation for the type. This should allow you to avoid repeating these trivial implementations (you'll need a Default trait bound).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose not to use the Default trait initially as it doesn't make sense to have Default for these responses unless you're using the ReducableResponse trait. Don't have strong opinions here though

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case you can make your own version of Default called ReducibleDefault or whatever.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented with Default for the sake of simplicity :)

api/src/lib.rs Outdated
Comment on lines 1359 to 1356
fn should_split(error: &Error) -> bool {
if let Error::Api { status_code, .. } = error {
*status_code == reqwest::StatusCode::UNPROCESSABLE_ENTITY
|| *status_code == reqwest::StatusCode::BAD_REQUEST
} else {
false
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can define this within splittable_request, as it's a helper private to the method.

api/src/lib.rs Outdated
pub num_failed: usize,
}

pub trait ReducableResponse {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub trait ReducableResponse {
pub trait ReducibleResponse {

api/src/lib.rs Outdated
query.clone(),
retry.clone(),
) {
Ok(r) => response = SuccessT::merge(response.clone(), r),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this mutation and cloning doesn't look quite right. Conceptually we should be able to reduce the successful responses, although we need to filter out and handle the failures somehow.

@joe-prosser joe-prosser requested a review from dan-ri July 25, 2024 17:52
api/src/lib.rs Outdated

let response = SuccessT::reduce(
body.split()
.iter()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking much better, but we should be able to do better still as this is a standard fold pattern, if you switch back to the merge trait method:

let response = body
    .split()
    .iter()
    .filter_map(|request| {
        match self.request(&method, &url, &Some(request), &query, &retry) {
            Ok(response) => Some(response),
            Err(_) => {
                num_failed += 1;
                None
            }
        }
    })
    .fold(SuccessT::empty(), |merged, next: SuccessT| merged.merge(next));

api/src/lib.rs Outdated
@@ -137,6 +143,35 @@ pub use crate::{
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Token(pub String);

pub trait SplittableRequest {
fn split(self) -> Vec<Self>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, you should be able to define split as returning an impl Iterator instead of a Vec and remove the collect() at the end of the implementations.

@joe-prosser joe-prosser merged commit fba30bf into master Jul 26, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants