Transfer Trip Surveys to S3 #267

binh-dam-ibigroup · 2024-11-12T15:07:38Z

Checklist

Appropriate branch selected (all PRs must first be merged to dev before they can be merged to master)
Any modified or new methods or classes have helpful JavaDoc and code is thoroughly commented
[na] The description lists all applicable issues this PR seeks to resolve
The description lists any configuration setting(s) that differ from the default settings
All tests and CI builds passing

Description

This PR adds support for transferring responses and deleting from TypeForm, compiling them into one CSV file, and upload that as a ZIP file to S3.

New config params: TRIP_SURVEY_API_TOKEN, for downloading survey metadata and responses.

…uploads.

…b (no core logic yet).

…ads.

…ackage.

…, and S3 uploading.

src/main/java/org/opentripplanner/middleware/OtpMiddlewareMain.java

JymDyerIBI

👍 Looks good to me.

br648

Some minors and perhaps consider moving the type form stuff into it's own package.

src/main/java/org/opentripplanner/middleware/models/IntervalUpload.java

src/main/java/org/opentripplanner/middleware/triptracker/TripSurveyUploadJob.java

br648 · 2024-11-26T15:43:15Z

src/main/java/org/opentripplanner/middleware/triptracker/TripSurveyUploadJob.java

+        }
+    }
+
+    private static HttpResponseValues apiRequest(HttpMethod method, String subPath, String queryParams, String topic) {


Consider moving all typeform related classes and methods to its own package similar to bugsnag. To me this seems cleaner and separates typeform interactions from business logic. Lemme know what you think.

Good point, updated in c27a5de.

src/main/java/org/opentripplanner/middleware/triptracker/TripSurveyUploadJob.java

...est/java/org/opentripplanner/middleware/connecteddataplatform/ConnectedDataPlatformTest.java

src/test/java/org/opentripplanner/middleware/triptracker/TripSurveyUploadJobTest.java

…query params.

…ezone.

br648

Thanks for the changes. I'm approving, but see changes in order to get test to pass in a different time zone.

br648 · 2024-12-03T13:45:20Z

src/test/java/org/opentripplanner/middleware/typeform/TypeFormDispatcherTest.java

+
+    @Test
+    void canMakeResponsesParams() {
+        // October 25, 2024 00:00, US Pacific Daylight Time


In order to use the config param OTP_TIMEZONE: America/Los_Angeles and for this test to pass (outside of this time zone), the following lines have to be added to this test:

// Load default env.yml configuration.
ConfigUtils.loadConfig(new String[]{});

You are right, the class can actually extend OTPMiddlewareTestEnvironment that includes loading the config (18ecd9f).

…onment.

…0 responses.

binh-dam-ibigroup added 15 commits November 6, 2024 11:22

docs: Introduce new survey API config params.

defbc4e

refactor(IntervalUploadJob): Introduce abstract classes for periodic …

629de0c

…uploads.

refactor(TripHistoryUploadJob): Reuse IntervalUploadJob.

a5705e5

refactor(TripHistoryUpload): Extend from IntervalUpload.

6298eb1

refactor(TripSurveyUpload): Introduce class for trip survey upload jo…

505cfba

…b (no core logic yet).

refactor(Persistence): Add new persistence class for trip survey uplo…

9a9ffde

…ads.

refactor(TypeFormTripSurveyResponse): Add classes for survey responses.

67b30bd

refactor(TypeFormTripSurveyResponse): Add CSV output.

99ef868

refactor(TypeFormTripSurveyApiResponse): Add CSV output.

524b2f9

test(TripSurveyUploadJob): Add tests for TripSurveyUploadJob.

0d10cc9

refactor(TripSurveyUploadJob): Integrate CSV dumping into inner logic.

f21cf6a

refactor(TripSurveyUploadJob): Rework logic for download vs tests.

0fb1f2d

refactor(TripSurveyUploadJob): Reuse getFilePrefix function.

256a4ab

refactor(IntervalUploadJob): Reuse getFirst, getLast helpers.

6f11275

refactor(IntervalUploadJob): Orient code toward processing intervals.

d94a681

binh-dam-ibigroup changed the base branch from dev to post-deviated-trip-notification November 12, 2024 15:09

binh-dam-ibigroup added 3 commits November 12, 2024 10:13

fix(TripSurveyUploadJob): Generate correct TypeForm URL.

a2b9c97

refactor(typeform.Form): Introduce typeform package with Form response.

733edda

refactor(typeform.Response): Move TypeForm response classes to that p…

aa15498

…ackage.

Base automatically changed from post-deviated-trip-notification to dev November 13, 2024 16:25

binh-dam-ibigroup added 10 commits November 13, 2024 12:29

refactor(TripSurveyUploadJob): Support downloading survey headers.

355100b

refactor(TripSurveyUploadJob): Refactor id and token check.

a275729

fix(typeform.Response): Surround answers with quotes.

bbb5810

fix(typeform.Response): Support text answers.

dcddc13

feat(OtpMiddlewareMain): Schedule trip survey upload job.

0b56b4e

refactor(TripSurveyUploadJo): Extract api request code.

e1d73fd

refactor(ConnectedDataManager): Perform light refactoring.

c19bb81

fix(TripSurveyUploadJob): Pass midnight-23:59:59 local as timestamps.

9ca61b2

refactor(IntervalUploadFiles): Extract logic handling file names, zip…

5efcc81

…, and S3 uploading.

Merge branch 'dev' into transfer-trip-surveys

49dd51f

binh-dam-ibigroup added 3 commits November 14, 2024 11:41

refactor(IntervalUploadJob): Add method for marking uploads complete.

e2c2982

refactor(IntervalUpload): Rename method.

ed17dbb

refactor(TripHistoryUpload): Remove redundant code

d746c8d

binh-dam-ibigroup requested a review from JymDyerIBI November 14, 2024 17:02

binh-dam-ibigroup assigned JymDyerIBI Nov 14, 2024

binh-dam-ibigroup marked this pull request as ready for review November 14, 2024 17:02

JymDyerIBI reviewed Nov 19, 2024

View reviewed changes

src/main/java/org/opentripplanner/middleware/OtpMiddlewareMain.java Show resolved Hide resolved

JymDyerIBI approved these changes Nov 19, 2024

View reviewed changes

binh-dam-ibigroup requested a review from br648 November 19, 2024 22:28

binh-dam-ibigroup assigned br648 and unassigned JymDyerIBI Nov 19, 2024

br648 requested changes Nov 26, 2024

View reviewed changes

br648 assigned binh-dam-ibigroup and unassigned br648 Nov 26, 2024

binh-dam-ibigroup added 6 commits December 2, 2024 10:50

style(ConnectedDataPlatformTest): Fix indent.

8c5ecf0

refactor(ConnectedDataPlatformTest): Add comments regarding TypeForm …

6accf8a

…query params.

refactor(IntervalUpload): Fix comment typo.

0b10963

refactor: Move TypeForm types to higher-level package.

1323648

refactor: Move TypeForm-specific logic to TypeFormDispatcher.

c27a5de

refactor(TypeFormDispatcherTest): Clarify comments about expected tim…

79b4c7a

…ezone.

binh-dam-ibigroup requested a review from br648 December 2, 2024 23:42

binh-dam-ibigroup assigned br648 and unassigned binh-dam-ibigroup Dec 2, 2024

br648 approved these changes Dec 3, 2024

View reviewed changes

br648 assigned binh-dam-ibigroup and unassigned br648 Dec 3, 2024

binh-dam-ibigroup added 2 commits December 3, 2024 09:05

refactor(TypeFormDispatcherTest): Inherit from OTPMiddlewareTestEnvir…

18ecd9f

…onment.

refactor(TypeFormDispatcherTest): Add comment for supporting over 100…

65b64c6

…0 responses.

binh-dam-ibigroup merged commit c904f1a into dev Dec 3, 2024
2 checks passed

binh-dam-ibigroup deleted the transfer-trip-surveys branch December 3, 2024 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transfer Trip Surveys to S3 #267

Transfer Trip Surveys to S3 #267

binh-dam-ibigroup commented Nov 12, 2024 •

edited

Loading

JymDyerIBI left a comment

br648 left a comment

br648 Nov 26, 2024

binh-dam-ibigroup Dec 2, 2024

br648 left a comment

br648 Dec 3, 2024

binh-dam-ibigroup Dec 3, 2024

Transfer Trip Surveys to S3 #267

Transfer Trip Surveys to S3 #267

Conversation

binh-dam-ibigroup commented Nov 12, 2024 • edited Loading

Checklist

Description

JymDyerIBI left a comment

Choose a reason for hiding this comment

br648 left a comment

Choose a reason for hiding this comment

br648 Nov 26, 2024

Choose a reason for hiding this comment

binh-dam-ibigroup Dec 2, 2024

Choose a reason for hiding this comment

br648 left a comment

Choose a reason for hiding this comment

br648 Dec 3, 2024

Choose a reason for hiding this comment

binh-dam-ibigroup Dec 3, 2024

Choose a reason for hiding this comment

binh-dam-ibigroup commented Nov 12, 2024 •

edited

Loading