Skip to content

Latest commit

 

History

History
278 lines (216 loc) · 21.8 KB

README.md

File metadata and controls

278 lines (216 loc) · 21.8 KB
Raw

Python App Engine app migration

To modern runtime, Cloud services, Python 3, and Cloud Run containers

Google App Engine (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs.

Each codelab begins with a "START" code base then walks developers through that migration step, resulting in a "FINISH" repo. If you make any mistakes along the way, you can always go back to START or compare your code with what's in the FINISH folder to see the differences. We also want to help you port to the Python 3 runtime, so some codelabs contain a bonus section for that purpose.

NOTE: These migrations are typically aimed at Python 2 users

  1. Python 3.x App Engine users: You're already on the next-gen platform, so only look for non-legacy service migrations
  2. Python 2.5 App Engine users: to revive apps from the original 2.5 runtime, deprecated in 2013 and shutdown in 2017, you must migrate from db to ndb and get those apps running on Python 2.7 before attempting these migrations.

Prerequisites

  • A Google account (Google Workspace/G Suite accounts may require administrator approval)
  • A Google Cloud (GCP) project with an active billing account
  • Familiarity with operating system terminal/shell commands
  • Familiarity with developing & deploying Python 2 apps to App Engine
  • General skills in Python 2 and 3

Cost

App Engine is not a free service. While you may not have needed to enable billing in App Engine's early days, all applications now require an active billing account backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an "Always Free" tier and as long as you stay within those limits, you won't incur any charges. Also check the App Engine pricing and quotas pages for more information.

Furthermore, deploying to GCP serverless platforms incur minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. For greater transparency, Cloud Build builds your application image which is than sent to the Cloud Container Registry; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via your Cloud Storage browser.)

Why

In App Engine's early days, users wanted Google to make the platform more flexible for developers and make their apps more portable. As a result, the team made significant changes to its 2nd-generation service which launched in 2018. As a result, there are no longer any built-in services, allowing users to select from standalone GCP products or best-of-breed 3rd-party tools used by the broader community. Summary:

  • Legacy platform: Python 2 only, legacy built-in services
  • Next generation: Python 3 only, use standalone services, flexible platform, some* built-in services available

* see Legacy services section below

While the 2nd-gen platform is more flexible, users of the legacy platform have two challenges:

  1. Migrate to unbundled/standalone services
  2. Porting to a modern language release

Neither upgrade may be particularly straightforward and can only be done serially. On top of this, direct replacements are not available for all formerly built-in services; alternatives come in 3 flavors:

  1. Direct replacement: Legacy services which matured into their own Cloud products (e.g., App Engine Datastore is now Cloud Datastore)
  2. Partial replacement: Some aspects of legacy services (e.g., Cloud Tasks supports App Engine push tasks; for pull tasks, Cloud Pub/Sub is recommended; use of Cloud MemoryStore with REDIS as an alternative for Memcache)
  3. No replacement: No direct replacement available, so third-party or other tools recommended (e.g., Search, Images, Users, Email)

These are the challenges developers are facing, so the purpose of this content is to reduce the friction in this process and make things more prescriptive. Review the runtimes chart to see the legacy services and current migration recommendation. The migration guide overview has more information.

NOTE: App Engine (Flexible) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine Standard vs. Flexible to find out more. Also, many of the Flexible use cases can now be handled by Cloud Run.

Progression (START and FINISH)

All codelabs begin with code in a START repo folder and end with code in a FINISH folder, implementing a single migration. Upon completion, users should confirm their code (for the most part) matches what's in the FINISH folder. The baseline migration sample app (Module 0; link below) is a barebones Python 2.7 App Engine app that uses the webapp2 web framework plus the ndb Datastore library.

  1. With Module 0 as the STARTing point, the Module 1 codelab migrates from the webapp2 web framework to Flask, FINISHing at code matching the Module 1 repo.
  2. Next, STARTing with the Module 1 application code (yours or ours), Module 2 migrates from ndb to Cloud NDB, ending with code matching the (Module 2) FINISH repo folder. There's also has a bonus migration to Python 3, resulting in another FINISH repo folder, this one deployed on the next-generation platform.
  3. Your Python 2 apps may be using other built-in services like Task Queues or Memcache, so additional migration modules follow, some more optional than others, and not all are available yet (keep checking back here for updates).

Beyond Module 2, with some exceptions, there is no specific order of what migrations modules to tackle next. It depends on your needs (and your applications').

Migration modules

The table below summarizes migration module resources currently available along with a more detailed table of contents below. Be sure to check back for updates as more resources are planned.

Summary table

Module Topic Video Codelab START here FINISH here
0 Baseline app link N/A (no tutorial; just review the code) N/A Module 0 code (2.x)
1 Migrate to Flask TBD link Module 0 code (2.x) Module 1 code (2.x) & code (3.x)
2 Migrate to Cloud NDB TBD link Module 1 code (2.x) Module 2 code (2.x) & code (3.x)
3 Migrate to Cloud Datastore TBD link Module 2 code (2.x) & code (3.x) Module 3 code (2.x) & code (3.x)
4 Migrate to Cloud Run with Docker TBD link Module 2 code (2.x) & Module 3 code (3.x) Module 4 code (2.x) & code (3.x)
5 Migrate to Cloud Run with Buildpacks TBD link Module 2 code (3.x) Module 5 code (3.x)
6 Migrate to Cloud Firestore (app) TBD link Module 3 code (3.x) Module 6 code (3.x)
7 Add App Engine push tasks TBD link Module 1 code (2.x) Module 7 code (2.x)
8 Migrate to Cloud Tasks TBD link Module 7 code (2.x) Module 8 code (2.x)
9 Migrate to Python 3, Cloud Firestore & Cloud Tasks v2 TBD TBD Module 8 code (2.x) Module 9 code (3.x)
10 Migrate to Cloud Firestore (data) TBD N/A N/A TBD
11 Migrate to Cloud Functions TBD TBD Module 2 code (3.x) Module 11 code (3.x)

Table of contents

If there is a logical codelab to do immediately after completing one, they will be designated as NEXT. Other recommended codelabs will be listed as RECOMMENDED, and the more optional ones will be labeled as OTHERS (and usually in some kind of priority order).

  • Module 1 codelab: Migrate from webapp2 to Flask

  • Module 2 codelab: Migrate from App Engine ndb to Cloud NDB

    • Required migration
      • Migration to Cloud NDB which is supported by Python 3 and the next-gen platform.
    • Python 2
    • Codelab bonus port to Python 3.x
    • RECOMMENDED:
      • Module 7 - add App Engine (push) tasks
    • OTHERS (somewhat priority order):
      • Module 11 - migrate to Cloud Functions
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks
      • Module 4 - migrate to Cloud Run container with Docker
      • Module 3 - migrate to Cloud Datastore

  • Module 7 codelab: Add App Engine (push) Task Queues to App Engine ndb Flask app

  • Module 8 codelab: Migrate from App Engine (push) Task Queues to Cloud Tasks v1

    • Required migration
      • Migration to Cloud Tasks which is supported by Python 3 and the next-gen platform.
      • Note this is only push tasks... pull tasks will be handled in a different codelab.
    • Python 2
    • NEXT: Module 9 - migrate to Python 3 and Cloud Datastore

  • Module 9 codelab (TBD): Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Firestore & Cloud Tasks (v2) app

    • Optional migrations
      • Migrating to Python 3 is not required but recommended as Python 2 has been sunset
      • Migrating to Cloud Firestore is very optional as Cloud NDB works on 3.x and most importantly, Cloud Firestore requires a completely new GCP project
    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 11 - migrate to Cloud Functions
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks
      • Module 4 - migrate to Cloud Run container with Docker

  • Module 4 codelab: Migrate from App Engine to Cloud Run with Docker

  • Module 5 codelab: Migrate from App Engine to Cloud Run with Cloud Buildpacks

    • Optional migration
      • "Containerize" your app (migrate your app to a container) with...
      • Cloud Buildpacks which lets you containerize your app without Dockerfiles
    • Python 3 only
    • RECOMMENDED:
      • Module 4 - migrate to Cloud Run container with Docker
    • OTHER OPTIONS (in somewhat priority order):
      • Module 7 - add App Engine (push) tasks
      • Module 11 - migrate to Cloud Functions

  • Module 3 codelab: Migrate from Cloud NDB to Cloud Datastore

    • Optional migration
      • Recommended only if using Cloud Datastore elsewhere (GAE or non-App Engine) apps
      • Helps w/code consistency & reusability, reduces maintenance costs
    • Python 2
    • Python 3
    • RECOMMENDED:
      • Module 7 - add App Engine (push) tasks
    • OTHER OPTIONS (in somewhat priority order):
      • Module 11 - migrate to Cloud Functions
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks
      • Module 4 - migrate to Cloud Run container with Docker
      • Module 6 - migrate to Cloud Firestore (app)

  • Module 6 codelab: Migrate from Cloud Datastore to Cloud Firestore (app)

    • Highly optional migration
      • Requires new project & Datastore has better write performance (currently)
      • If you must have Firestore's Firebase features
    • Python 3 only
    • NEXT:
      • Module 10 - migrate to Cloud Firestore (data)
    • RECOMMENDED:
      • Module 7 - add App Engine (push) tasks
    • OTHER OPTIONS (in somewhat priority order):
      • Module 11 - migrate to Cloud Functions
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks
      • Module 4 - migrate to Cloud Run container with Docker

  • Module 10 codelab (TBD): Migrate from Cloud Datastore to Cloud Firestore (data)

    • Highly optional migration
      • Requires new project & Datastore has better write performance (currently)
      • If you must have Firestore's Firebase features
    • Python 3 only
    • RECOMMENDED:
      • Module 7 - add App Engine (push) tasks
    • OTHER OPTIONS (in somewhat priority order):
      • Module 11 - migrate to Cloud Functions
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks
      • Module 4 - migrate to Cloud Run container with Docker

  • Module 11 codelab (TBD): Migrate from App Engine to Cloud Functions

    • Optional migration
      • Recommende for small apps or for breaking up large apps into multiple microservices
    • Python 3 only
    • RECOMMENDED:
      • Module 7 - add App Engine (push) tasks
    • OTHER OPTIONS (in somewhat priority order):
      • Module 5 - migrate to Cloud Run container with Cloud Buildpacks
      • Module 4 - migrate to Cloud Run container with Docker
      • Module 3 - migrate to Cloud Datastore

Considerations for mobile developers

If your original app users does not have a user interface, i.e., mobile backends, etc., but still uses webapp2 for routing, some migration must still be completed. Your options:

Canonical code samples

Accessing legacy services in second generation

Some legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available on a limited basis to second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) as part of an early-access program. There is no video or codelab available yet, however the Module 1 Flask migration Python 2 sample is available in Python 3 if you have access. (See its README to sign-up if interested.)

References