The BI API powers the BI Web, as well as BrAPI powered applications.
The API is built using Java 13 and the Micronaut framework. The development guide for Micronaut can be found at: https://docs.micronaut.io/latest/guide/index.html
The API can run inside a Docker container using the Dockerfile for building the
API image and the docker-compose.yml
to run the container.
docker-compose up -d biapi-dev
The docker-compose.yml
should contain a service for each environment the API is
to be run in: e.g develop, test, staging, and production. Each service contains
under the environment key public values for environment variables used as params
for the API configuration.
Private values used in each environment are never placed in docker-compose.yml and never committed to the repo. Variables are passed into the docker-compose.yml from the .env file. You can either modify the .env file to hold your secrets, or run docker-compose with a custom .env file:
docker-compose --env-file .env.local up -d biapi-dev
If you have the docker socket (/var/run/docker.sock) and docker executable folder (/usr/bin/docker) mounted in your docker-compose.yml file, you will be able to run the biapi tests from within the docker container. The default docker-compose file has these mounted.
The biapi tests use the hosts docker instance to spin up test database containers for integration tests.
NOTE: Do not run a production container with the docker socket mounted. While useful for testing and development, mounting the docker socket into a container is a large security risk. The application will still work fine without the docker socket mounted, but tests will fail.
To run the tests, use the following command:
docker exec -it biapi mvn test -settings settings.xml
When evaluating a pull request for merge acceptance, verify that the following criteria are met:
- Any changes to API code have a corresponding change to the openAPI specification in the
docs
folder - Any changes to the API code have corresponding endpoint tests
The following sections provide instructions to get the application running in your local development environment. This section is provided as an alternative to running the application in Docker.
- Java 13 SDK installed
- Maven installed (or via IDE)
- Docker (and docker-compose) installed
Run this docker command in terminal to start up a postgres docker container with the empty bi_db database in it.
docker-compose up -d bidb
A fork of the brapi test server is used for local development,
https://github.com/Breeding-Insight/brapi-Java-TestServer
to run the brapi test server, update the proper variables in the .env file and then run the command:
docker-compose up -d brapi-server
Run this docker command in terminal to start up a redis docker container
docker-compose up -d redis
Then update the following environment variables:
REDIS_URL=redis://localhost:6379
Run this docker command in terminal to start up a Gigwa container
docker-compose up -d gigwa
Then update the following environment variable:
GIGWA_HOST=http://localhost:5080/
If you do not want to configure an S3 bucket and IAM roles in S3, you can use Localstack (https://localstack.cloud/) to emulate the S3 service on your machine
Run this docker command in terminal to start up a Localstack container
docker-compose up -d localstack
Then update the following environment variables:
AWS_ACCESS_KEY_ID=test
AWS_SECRET_KEY=test
AWS_S3_ENDPOINT=http://localhost:4566
Once you have the project pulled down and your database running, follow these steps to build the project.
You will need to specify variables specific to your environment in order to run the project.
Environment specific project variables are specified in the following files:
- src/build/build.properties
- src/main/resources/application-prod.yml
There are three options to specify these project variables: a. Specify environmental variables in your system or terminal session (this is what our docker build does) b. Specify environmental variables in your IDE run configuration (examples are given in IntelliJ) c. Create dev versions of build.properties and application-prod.yml and edit files directly.
See the .env file in the project's root directory for a list of environmental variables you will need to specify. NOTES:
- If using option b (IntelliJ) to specify environment variables, you will need to run all commands through a run configuration in IntelliJ. Running commands in terminal will not exposed the environment variables you set in your run configuration.
If you want to run your mvn or java commands through terminal, you will need to set your environmental variables in your terminal session or for the system. To set your environment variables for the terminal session you can use the following command at the beginning of your terminal session:
source .env && export $(grep --regexp ^[A-Z] .env | cut -d= -f1)
This will set your variables from your .env file while ignoring comments in the file.
You can set your environment variables in IntelliJ in two ways:
- Go to the run configuration of focus and navigate to the 'Runner' tab, enter environment variables in the 'Environment Variables' field.
- Use the EnvFile plugin, https://plugins.jetbrains.com/plugin/7861-envfile, and use your .env file.
NOTE: The EnvFile plugin does not work for maven run configurations.
To edit the config files directly, it is recommended to create dev versions of the config files and use then use dev run environments for maven and micronaut.
- Create a
build.dev.properties
file based on thebuild.properties
file. Enter your values in the${}
placeholders. - Create an
application-dev.yml
file base on theapplication-prod.yml
file. Enter your values in the${}
placeholders.
After setting the values for your environment, run:
mvn clean validate install -D maven.test.skip=true --settings settings.xml
NOTE: if using option c, run:
mvn validate clean install -D maven.test.skip=true --settings settings.dev.xml -P dev
This process will pull down all the application's dependencies to your local machine, create the application's database structure (with appropriate data), generate required source files (via JOOQ), and finally build the final JAR file.
The command above will not run unit tests. If you want to run unit tests during the project's dependency installation
remove the -D maven.test.skip=true
command. Alternatively, you can run the tests separately, see the testing
section below.
Once the project dependencies are installed successfully, you can run the project.
If running in an IDE (like IntelliJ):
- Create an
Application
run config with the main class beingorg.breedinginsight.api.Application
- Pass VM options:
--enable-preview --illegal-access=warn -Dmicronaut.environments=prod
If running as a packaged JAR:
java --enable-preview -Dmicronaut.environments=prod -jar bi-api*.jar
NOTE: if you used option c to configure the project's variables, replace -Dmicronaut.environments=prod
with -Dmicronaut.environments=dev
If you have run the project and the database once already, you will need to make sure your database is up to date by running:
mvn validate flyway:migrate -X --settings settings.xml
If you don't care about losing any of the data in the database, run:
mvn validate flyway:clean flyway:migrate -X --settings settings.xml
As structural database changes are made, you will need to re-generate Java classes via JOOQ (data model, base DAOs). To do so, run:
mvn clean generate-sources --settings settings.xml
NOTE: This step is not necessary if a mvn clean install
is run (see section Install Project Dependencies above).
The test database is started as part of the tests in a separate docker container. The JOOQ classes for the tests
are generated from the main database, so this will have to be up and running. The reference to the main database
can be found in src/build/build.properties
.
There are two ways to run the tests:
- JUnit
- Maven
For this option, you will need to run the tests through IntelliJ. You will need to create a file
src/test/resources/application-test.yml
based off the application-prod.yml
file. Edit the values
directly in the file, or use environment variables like described above.
This method uses the values set in the build.properties
or build.dev.properties
file to run your tests.
To run the tests, use the command:
mvn test --settings settings.xml
or
mvn test -P dev --settings settings.xml
Maven tests are also run as part of the install profile (if specified).
If not manually specifying values in application-test.yml, you will need to set all of the environment variables by following either option a or option b above.
If not manually specifying values in application-test.yml, you will need to set all of the environment variables found in the .env file for running the tests as well.
If you are having errors to the effect of invalid source release 12 with --enable-preview
and are using IntelliJ, change the jdk to 13 in the following places and it may help:
- File -> Project Structure -> Project Settings
- File -> Project Structure -> Modules -> Tab: Sources: Language Level
- File -> Project Structure -> Modules -> Tab: Dependencies: Module SDK
- Main -> Preferences -> Compiler -> Java Compiler -> Target bytecode version
- Run/Debug Configurations (Your run profiles) -> JRE select box
Connection issues during build:
- Check that you all of the environment variables in the src/build/build.properties file specified.
- Make sure you are able to connect to your database outside of the project.
JOOQ class errors during build:
- Make sure your database is up to date.
- See troubleshooting on connection issues.
Micronaut error, variable XXX is not specified:
- Make sure your application-dev.yml file exists.
Installation error for bi-jooq-codegen during build.
- Make sure your build command includes
--settings settings.xml
- Make sure you have values for
GITHUB_ACTOR
andGITHUB_TOKEN
. - Make sure your
GITHUB_TOKEN
is valid and that you have access to the bi-jooq-codegen repo. - See bi-jooq-codegen to make sure everything is setup correctly for pulling the repository, https://github.com/Breeding-Insight/bi-jooq-codegen.
Value ${SOME_VARIABLE} is not specified/found/valid.
- Check you have the mentioned variable specified in your IntelliJ run configuration or system variables.
- Make sure you are running your commands through IntelliJ run configuration if using that option.
Placeholder ${SOME_VALUE} is not specified/found/valid.
- Make sure your application-dev.yml and application-test.yml files are up to date with makeApplication-dev.yml.sh and makeApplication-test.yml.sh files.