Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy OTP to AWS load balancer and manage OTP servers in separate collection (and misc other fixes) #225

Merged
merged 50 commits into from
Oct 9, 2019
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
ac0b00c
WIP spin up EC2 (no user data)
landonreed Aug 7, 2018
9c5e92c
Merge branch 'remove-r5' into deploy-to-ec2
landonreed Sep 18, 2018
d34b96a
Merge branch 'dev' into deploy-to-ec2
landonreed Nov 6, 2018
560c4c6
refactor(snapshot): remove legacy MapDB-based snapshot jobs
landonreed Nov 12, 2018
dbbd130
fix(delete): delete SQL namespace when feed version/snapshot deleted
landonreed Nov 12, 2018
c01eadc
feature(deploy-ec2): deployment enhancements for load balancers
landonreed Nov 12, 2018
f777b02
fix(user-mgmt): better error handling when Auth0 cannot update/create…
landonreed Nov 12, 2018
8898376
fix: move toGtfsDate from deleted class to FeedTx
landonreed Nov 12, 2018
fa5ce83
refactor: fix whitespace
landonreed Nov 12, 2018
e0a1eb6
refactor: add missing aws pom entry
landonreed Nov 12, 2018
61c04d2
build(pom): update gtfs-lib dependency
landonreed Nov 12, 2018
7231a95
feature(server-mgmt): manage deployment servers at the application level
landonreed Nov 15, 2018
b372303
refactor(server-job): attach just the project ID to the merge feeds job
landonreed Nov 29, 2018
7c92c1d
Merge branch 'dev' into deploy-to-ec2
landonreed Nov 30, 2018
aa0553b
refactor(deploy): shuffle deploy job code for clarity
landonreed Nov 30, 2018
c52d6f0
Merge pull request #133 from ibi-group/dev
landonreed Aug 7, 2019
56f7642
Merge branch 'dev' into deploy-to-ec2
landonreed Aug 7, 2019
6ae2055
refactor: fix issues resulting from merge
landonreed Aug 7, 2019
c33c290
refactor(deployment): tweak user script and update default config
landonreed Aug 8, 2019
05ec4df
refactor(deployment): improve validation of server fields
landonreed Aug 9, 2019
b6d7363
refactor(deployments): modify OtpServer fields and refactor server cr…
landonreed Aug 9, 2019
7a020c8
refactor: remove unused import
landonreed Aug 9, 2019
c753a31
refactor(ServerController): add comment about checking S3 permissions
landonreed Aug 13, 2019
b793727
refactor(ServerController): add missing exceptions to logMessageAndHalt
landonreed Aug 14, 2019
a3ed73c
refactor(deploy): fix check for s3 graph object
landonreed Aug 20, 2019
a07935a
refactor(deploy): revert to default instance type if none specified
landonreed Aug 20, 2019
a177e79
refactor(deploy): make instance profile arn optional
landonreed Aug 20, 2019
2507ab5
refactor(deploy): use set method rather than with for instance profile
landonreed Aug 22, 2019
e1fe1a3
Merge branch 'dev' into deploy-to-ec2
landonreed Sep 3, 2019
4a1ef29
refactor(deploy): move ec2 config into OtpServer
landonreed Sep 9, 2019
9b957dd
refactor(deploy): tweak deployJob for NPE fix and fix server delete
landonreed Sep 10, 2019
bf0f1bc
ci(config): update server.yml.tmp for e2e
landonreed Sep 10, 2019
ade0b40
refactor(EC2InstanceSummary): add empty constructor for serialization
landonreed Sep 10, 2019
95a2333
Merge branch 'dev' into deploy-to-ec2
landonreed Sep 12, 2019
c273350
test(.gitignore): don't ignore test config
landonreed Sep 12, 2019
3493570
test(mtc): fix broken MTC feed merge test with new test config
landonreed Sep 12, 2019
4992b32
refactor(ServerController): isolate jackson parse to utility method
landonreed Sep 12, 2019
3441fa2
refactor(ServerController): surround validation method calls in try/c…
landonreed Sep 12, 2019
a36a7d9
refactor(deploy-to-ec2): address PR comments
landonreed Sep 20, 2019
18bd9d3
refactor(deploy-to-ec2): add json property latest; add server ID to s…
landonreed Sep 20, 2019
c28c215
Merge branch 'dev' into deploy-to-ec2
landonreed Sep 20, 2019
41c21a8
refactor(deploy): fix check for S3 jar
landonreed Sep 20, 2019
7e1528b
refactor(deploy-to-ec2): address PR comments
landonreed Sep 24, 2019
f34affb
refactor(deploy-to-ec2): surround s3 checks in try/catch
landonreed Sep 24, 2019
4cc9b68
refactor(deploy-to-ec2): actually skip termination request
landonreed Sep 24, 2019
fb44a61
refactor(deploy): fix duration calc
landonreed Sep 30, 2019
101b7f9
refactor(deploy): use onboard nginx to signal ec2 deploy status
landonreed Oct 1, 2019
523801d
refactor(deploy): bump default otp version to 1.4
landonreed Oct 3, 2019
c399189
refactor(deploy): add terminate EC2 instance HTTP endpoint
landonreed Oct 8, 2019
240a6e0
refactor(deploy): refine terminate instances endpoint and check for g…
landonreed Oct 8, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions configurations/default/server.yml.tmp
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,15 @@ modules:
enabled: false
editor:
enabled: false
deployment:
enabled: false
ec2:
enabled: false
subnet: subnet-id
securityGroup: security-group-id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure that this is information we'd want to include as part of the appinfo response?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of this config info might actually make more sense to store in the OtpServer object... that might make more sense than splitting up the AWS info by putting the targetGroupArn in the OtpServer and this stuff in server.yml.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, there may be more than one EC2 environment we're deploying to, so that just strengthens the argument to place this info into OtpServer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I concur.

ami: ami-id
arn: arn
keyName: some-pem-file-without-suffix
user_admin:
enabled: true
# Enable GTFS+ module for testing purposes
Expand Down
11 changes: 11 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,17 @@
<artifactId>super-csv</artifactId>
<version>2.4.0</version>
</dependency>
<!-- AWS individual module imports (S3 imported by gtfs-lib) -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-ec2</artifactId>
<version>1.11.410</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-elasticloadbalancingv2</artifactId>
<version>1.11.410</version>
</dependency>
</dependencies>

</project>
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ public enum JobType {
EXPORT_SNAPSHOT_TO_GTFS,
CONVERT_EDITOR_MAPDB_TO_SQL,
VALIDATE_ALL_FEEDS,
MONITOR_SERVER_STATUS,
MERGE_FEED_VERSIONS
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ private static Snapshot deleteSnapshot(Request req, Response res) {
if (snapshot == null) logMessageAndHalt(req, 400, "Must provide valid snapshot ID.");
try {
// Remove the snapshot and then renumber the snapshots
Persistence.snapshots.removeById(snapshot.id);
snapshot.delete();
feedSource.renumberSnapshots();
// FIXME Are there references that need to be removed? E.g., what if the active buffer snapshot is deleted?
// FIXME delete tables from database?
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@
import java.util.concurrent.ConcurrentMap;
import java.util.stream.Collectors;

import static com.conveyal.datatools.editor.jobs.ProcessGtfsSnapshotExport.toGtfsDate;

/** a transaction in an agency database */
public class FeedTx extends DatabaseTx {
private static final Logger LOG = LoggerFactory.getLogger(FeedTx.class);
Expand Down Expand Up @@ -124,6 +122,10 @@ public FeedTx(DB tx, boolean buildSecondaryIndices) {
// editedSinceSnapshot = tx.getAtomicBoolean("editedSinceSnapshot") == null ? tx.createAtomicBoolean("editedSinceSnapshot", false) : editedSinceSnapshot;
}

private static int toGtfsDate (LocalDate date) {
return date.getYear() * 10000 + date.getMonthValue() * 100 + date.getDayOfMonth();
landonreed marked this conversation as resolved.
Show resolved Hide resolved
}

public void commit () {
try {
// editedSinceSnapshot.set(true);
Expand Down

This file was deleted.

Loading