Code

Refactor for Dataflow runner#3

Code

Merged

pmhalvor merged 31 commits intomainpmhalvor/whale-speech:mainfrom refactor-for-dataflow-runnerpmhalvor/whale-speech:refactor-for-dataflow-runnerCopy head branch name to clipboardOct 20, 2024

+825-105

Commits

Commits on Oct 17, 2024

• test packaging repo for deployment

  pmhalvor

  committed
  d410fe5

• dockerize main pipeline

  pmhalvor

  committed
  bf81301

• Merge branch 'main' into refactor-for-dataflow-runner

  pmhalvor

  committed
  6a61067

• dockerize and deploy model server

  pmhalvor

  committed
  609751f

• update pytest workflow to reflect refactored requirements

  pmhalvor

  committed
  d70cad4

• only test tests/ dir

  pmhalvor

  committed
  42494c4

• containerize pipeline

  pmhalvor

  committed
  6d2aaf1

Commits on Oct 18, 2024

• change is_local -> filesystem

  pmhalvor

  committed
  107219b

• add make target for python venv setup

  pmhalvor

  committed
  de9cf8e

• add more makefile targets

  pmhalvor

  committed
  97e5b00

• accept port from env var

  pmhalvor

  committed
  d291303

• remove expose from model server doccker file

  pmhalvor

  committed
  f779fc3

• hardcode default cloud run port

  pmhalvor

  committed
  0baaeeb

• specify platform on make build + default to port in config

  pmhalvor

  committed
  00a5fbd

• make pipeline worker image

  pmhalvor

  committed
  40f49a8

• fix Dockerfile for pipeline

  pmhalvor

  committed
  83b1665

• move stages to home in pipeline worker Dockerfile

  pmhalvor

  committed
  6ca6760

• update datetime storing

  pmhalvor

  committed
  e66ce24

Commits on Oct 19, 2024

• test limit on model server -> max batch duration = 60 sec

  pmhalvor

  committed
  258b596

• higher throughput resources for faster dataflow runs

  pmhalvor

  committed
  f999506

• add setup and teardown calls to init and deduplicating data

  pmhalvor

  committed
  a154ec2

• update docs

  pmhalvor

  committed
  d41657d

Commits on Oct 20, 2024

• test push model server bucket

  pmhalvor

  committed
  a45f8b6

• rm double run of push-deploy

  pmhalvor

  committed
  aa29bf6

• no need to install for docker builds

  pmhalvor

  committed
  e485e80

• bump actions/checkout@4

  pmhalvor

  committed
  b9263af

• add similar job for pipeline worker

  pmhalvor

  committed
  8cd65f6

• add publish make target

  pmhalvor

  committed
  6cc2c8c

• make dokcerhub username public, so the final image location is visible in the logs

  pmhalvor

  committed
  e708c30

• refactor publish job to separate actions workflow

  pmhalvor

  committed
  0d0b286

• update version to first offical release

  pmhalvor

  committed
  bb04df7