Public documentation and wiki for DoIT’s Digital team
$ yarn install
This repo is split into two broad categories: services and modules. Modules are JavaScript (and TypeScript) libraries that are shared across various apps. Services are individual apps.
Modules may depend upon each other (though not circularly). Services may only depend on modules. Nothing may depend on a service.
The majority of our services are written in JavaScript/TypeScript and are
therefore in the services-js directory. We have a few legacy Ruby apps in
services-ruby. These are stand-alone, without any dependencies on our own
libraries. Because we’re not doing new development in Ruby, we didn’t put the
effort into code-sharing infrastructure for it.
This uses “watch” features to rebuild all of our modules as their source changes.
$ cd modules-js/react-fleet
$ yarn watch
Note: This may not necessarily capture recompiling depending modules when their dependencies rebuild.
Note: You may need to make heavy use of restarting any TypeScript server in your editor if types in these modules change.
$ cd services-js/commissions-app
$ yarn dev
You also may want to run yarn watch-dependencies to recompile changes in any
modules that the service depends upon.
$ npx khaos create -d templates <template-name> <destination>
$ yarn install
- js-server-module: Module for libraries that are only going to be required by server-side Node apps. Uses TypeScript for all compilation.
- js-browser-module: Module that’s optimized for libraries that are included via Webpack. Uses TypeScript for type checking but compiles with the browser.js Babel configuration.
It’s best to copy from an existing one. See the New service setup documentation.
Please don’t. We have a few Ruby services that pre-date this repo, but want to focus future development entirely on JavaScript and Node.
Staging instances are created by adding them into the Terraform templates. Typically, each service will have a default staging deploy and may have other “variants” for special circumstances.
Staging branch names follow the pattern:
staging/<service-name>staging/<service-name>@<variant>
Where “<service-name>” is the service package’s name (minus any services-js.
prefix).
To deploy to staging, first force-push to the staging branch. A GitHub webhook
will fire and you’ll get a prompt in the #digital_builds Slack channel from
Shippy-Toe, asking you if you want to deploy. Press the “Deploy” button to
trigger the deployment, which is done via CloudBuild.
Tests are not run for the staging deploy so that you can put something on
staging even if the tests don’t pass. (Tip: use the --no-verify flag to git push to keep it from trying to run the tests locally either.)
If you need to roll back a staging release, force-push an earlier commit and re-deploy.
Deploying the Commissions App service to staging:
$ git co my-feature-branch
$ git push --force --no-verify origin HEAD:staging/commissions-app
Now press “Deploy” in #digital_builds.
Check on the status of the deploy by following the link in Shippy-Toe’s Slack message to the CodeBuild run.
When the deploy completes, the new app will be available at: https://commissions-app.digital-staging.boston.gov/commissions
We believe in continuous deployment, so anything that’s merged in to the
develop branch is eligible for immediate deployment.
At the end of the develop Travis run, the script “deploys” by running
deploy-tools’s report-updated-services tool. This uses Lerna to compare the
develop branch with all of the production/* branches for our services. If
develop contains any changes for a service—or one of its dependencies—that do
not exist in that service’s production/<service-name> branch, it notifies the
Shippy-Toe bot that a deploy is needed.
Just as with staging changes, Shippy-Toe will prompt in #digital_builds that
there are services to deploy. Press the “Deploy” button to release them.
Note: If the internal-slack-bot service comes up as deployable, wait to
deploy it last, since deploying it will wipe out the state of any running
deployments. They’ll still complete, but you won’t get notification of their
status.
Shippy-Toe will run the deployment by first pointing the
production/<service-name> branch to develop, then run the CodeBuild deploy
to release it.
If you need to roll back a production deploy, it’s cleanest to push a revert
commit through Travis and re-deploy. You can alternately force-push to the
appropriate production/<service-name> branch and run the CodeBuild deploy
manually. However, as long as the production/* branch lags behind develop,
Shippy-Toe will offer to deploy with every change that’s merged in.
Most of our modules and services have tests written in Jest. Services that use Storybook also use Storyshots to integrate stories with Jest’s snapshot testing.
Some of our services also have TestCafé tests to do frontend/backend integration tests. We use these typically for “critical path” tests (like going through the Access Boston registration flow), especially those that are a pain to manually test.
These tests get run on a headless browser via the test package.json scripts.
When developing, you can run yarn testcafe:dev to run TestCafé in your desktop
browser (or even a browser on another machine, like the BrowserStack cloud).
We want to write the bulk of our code in TypeScript because type checking is
good. Unfortunately the landscape of JavaScript modules, require vs. import,
and client-side bundlers (such as Webpack) tend to complicate things. The same
piece of code might be required server-side, used client-side by both Next’s
webpack config and Storybook’s, and run in a Jest test.
These tend to be the simplest to compile. We don’t need to worry about polyfills or bundle sizes.
This means that we can build directly with TypeScript, via tsc and
tsc-watch, and get both typechecking and compiling at the same time.
The build target for server-side code is Node 8. This means ES2017 syntax and
language features can be native (such as async/await), but the modules
must be CommonJS. This is all set in config-typescript’s
default.tsconfig.json config via "target": "es2017" and "modules": "commonjs".
Nevertheless, since we test with Jest and our Jest setup uses babel-loader,
these packages still need a .babelrc file. It should just use the
@cityofboston/config-babel/node and @cityofboston/config-babel/typescript
presets.
(Babel is fully capable of processing TypeScript files into JavaScript, but it doesn’t do any typechecking on them. It just strips the type annotations away and compiles the rest.)
These are libraries of frontend code, typically React components. We want to use
Babel to build these libraries to take advantage of the frontend-specific Babel
plugins (such as @babel/env, emotion, &c.) while still outputting
browser-compatible (ES5) code.
These packages should load the @cityofboston/config-babel/browser preset from
their .babelrc files, which uses @babel/env’s default to compile to ES5.
They will likely also use @cityofboston/config-babel/typescript.
Despite having ES5 code, we want to export these libraries using ES2015 modules so that Webpack can better tree-shake and keep unused code out of our application bundles so that they’re smaller to download. See: Webpack 4 tree shaking guide
We do this by using a special esm value for BABEL_NODE when doing builds,
which @cityofboston/config-babel/browser interprets to generate esm modules.
We point package.json’s module property at the entry point for these modules
(and set sideEffects to false).
However, we still need CommonJS files because these libraries won’t always be included via Webpack. This can be due to being included from server-side Node files or just used in a Jest test.
For Jest/Node to be able to handle imports of these modules from other packages,
we still need a main entry that points at a CommonJS build. We use Rollup to
convert the ESM build into a single .es5.js file that Jest can see.
Since we still want d.ts files so that there’s proper type checking when
importing these packages, we run tsc --emitDeclarationOnly during build. This
also has the side effect of doing the type checking that Babel does not.
These apps get tricky because they have server that runs in Node, client code that runs in the browser (and also on the server with SSR), and (optionally) shared library code that runs in both.
NextJS handles all of the web client code via Babel and Webpack. We use a
.babelrc file to include Next and TypeScript presets. So far we have not had
to make any of our own accommodations for the server-side rendering code.
Next and Webpack also handle hot-reloading of the client code when it changes.
For the server, we use tsc and tsc-watch but with a special
tsconfig.server.json TypeScript configuration that is limited to the
server-specific source directories. This keeps the server from restarting when
we make client-only changes.
The default tsconfig.json still includes all the code so that tooling will see
and typecheck everything.
In all cases the code is written in TypeScript. Do not use Babel plugins that
enable syntax that tsc can’t understand. Since the client-side code is not
typechecked when it’s compiled by Babel, we run tsc --noEmit on everything as
a pretest script to do type checking.
Note: We may soon switch to just using Babel for the server code as well and
avoiding tsc for these apps entirely (beyond typechecking). The latest version
of babel-watch finally supports Babel 7. Previously it did not, so we used
tsc-watch to automatically reload the TypeScript code.
As of this writing, Digital webapps support IE11 and the latest versions of the evergreen browsers. (See Browsers we support in the working agreement)
One exception to this is the public-notices app, which needs to run on the old version of Chrome that the digital display has installed.
Because we are always forgetting about polyfills, we’ve configured the
babel-env plugin to "usage" mode for useBuiltIns. This causes it to
automatically pull in core-js polyfills for functions as we use them. Note that
it does not polyfill functions and classes that our dependencies may need, nor
does it polyfill fetch.
We’ve added polyfills that our dependencies tend to need, as well as
isomorphic-fetch, to the polyfills.js file in next-client-common. The
withPolyfill mixin is used in next.config.js files to automatically include
these polyfills before any other code.
We build all of our interfaces to be responsive, down to 320px wide.
- 2020.12.10: React-Fleet - Node-Fetch 1.6.9 > 2.6.1
- Affected Apps
- modules-js
- react-fleat
| []
- react-fleat
- services-js
- 311
| [] - 311-indexer
| [] - access-boston
| [] - group-mgmt
| [] - internal-slack-bot
| [] - payment-webhooks
| [] - permit-finder
| [] - public-notices
| [] - registry-certs
| [x] | Docker > node:8.14-alpine
- 311
- modules-js
- Affected Apps
- 2020.11.10: Reworking how AWS gets around Docker Hub Rate limiting