Updating from v106.4 breaks the immich-server and microservices #11403

MadFisa · 2024-07-27T18:13:20Z

MadFisa
Jul 27, 2024

The bug

Trying to update immich to any version beyond v1.106.4 results in immich-server and microservices getting stuck in a loop. I was trying to update to the latest releaser version (v1.110.0) from v1.106.4 . The server wouldn't start. Manually rolled back version by version to see if the error disappers, but seems to work only with v1.106.4.

The OS that Immich Server is running on

Debian Bookworm

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.109.0

Platform with the issue

Server
Web
Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=/data/data1/immich_photos/
DB_DATA_LOCATION=/srv/immich_db/
IMMICH_VERSION=release
DB_PASSWORD=jsbMF
DB_USERNAME=asifmp
DB_DATABASE_NAME=immich
IMMICH_VERSION=v1.106.4

Reproduction steps

1. The server was on version v1.106.4
2. Tried updating to latest release version
3. Server and microservice on bootloop
...

Relevant log output

Node.js v20.14.0
api worker exited with code 1
Detected CPU Cores: 4
Starting api worker
[Nest] 18  - 07/27/2024, 5:51:57 PM     LOG [Api:EventRepository] Initialized websocket server
Migration "AddFaceSearchRelation1718486162779" failed, error: missing chunk number 1 for toast value 58419 in pg_toast_17002
QueryFailedError: missing chunk number 1 for toast value 58419 in pg_toast_17002
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:18:9)
    at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
    at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
    at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9)
    at async /usr/src/app/dist/services/database.service.js:134:17
    at async /usr/src/app/dist/repositories/database.repository.js:177:23 {
  query: '\n' +
    '            INSERT INTO face_search("faceId", embedding)\n' +
    '            SELECT id, embedding\n' +
    '            FROM asset_faces faces',
  parameters: undefined,
  driverError: error: missing chunk number 1 for toast value 58419 in pg_toast_17002
      at /usr/src/app/node_modules/pg/lib/client.js:526:17
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
      at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:18:9)
      at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
      at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
      at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9)
      at async /usr/src/app/dist/services/database.service.js:134:17
      at async /usr/src/app/dist/repositories/database.repository.js:177:23 {
    length: 132,
    severity: 'ERROR',
    code: 'XX001',
    detail: undefined,
    hint: undefined,
    position: undefined,
    internalPosition: undefined,
    internalQuery: undefined,
    where: undefined,
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'heaptoast.c',
    line: '784',
    routine: 'heap_fetch_toast_slice'
  },
  length: 132,
  severity: 'ERROR',
  code: 'XX001',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'heaptoast.c',
  line: '784',
  routine: 'heap_fetch_toast_slice'
}
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^
QueryFailedError: missing chunk number 1 for toast value 58419 in pg_toast_17002
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:18:9)
    at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
    at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
    at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9)
    at async /usr/src/app/dist/services/database.service.js:134:17
    at async /usr/src/app/dist/repositories/database.repository.js:177:23 {
  query: '\n' +
    '            INSERT INTO face_search("faceId", embedding)\n' +
    '            SELECT id, embedding\n' +
    '            FROM asset_faces faces',
  parameters: undefined,
  driverError: error: missing chunk number 1 for toast value 58419 in pg_toast_17002
      at /usr/src/app/node_modules/pg/lib/client.js:526:17
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
      at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:18:9)
      at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
      at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
      at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9)
      at async /usr/src/app/dist/services/database.service.js:134:17
      at async /usr/src/app/dist/repositories/database.repository.js:177:23 {
    length: 132,
    severity: 'ERROR',
    code: 'XX001',
    detail: undefined,
    hint: undefined,
    position: undefined,
    internalPosition: undefined,
    internalQuery: undefined,
    where: undefined,
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'heaptoast.c',
    line: '784',
    routine: 'heap_fetch_toast_slice'
  },
  length: 132,
  severity: 'ERROR',
  code: 'XX001',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'heaptoast.c',
  line: '784',
  routine: 'heap_fetch_toast_slice'
}

Additional information

No response

mmomjian · 2024-07-27T19:58:05Z

mmomjian
Jul 27, 2024
Collaborator

Did you read the release notes? 1.106 no longer uses the microservices container: #10110

3 replies

MadFisa Jul 28, 2024
Author

I found that after I posted the issue. I modified the docker compose file accordingly, but nothing changed. Here is my new compose file. The same error pops up.
Here is my new docker compose file

name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}

    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
volumes:
  model-cache:

And Here is the error

Detected CPU Cores: 4
Starting api worker
Starting microservices worker
[Nest] 7  - 07/28/2024, 6:40:47 AM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 17  - 07/28/2024, 6:40:47 AM     LOG [Api:EventRepository] Initialized websocket server
Migration "AddFaceSearchRelation1718486162779" failed, error: missing chunk number 1 for toast value 58419 in pg_toast_17002
QueryFailedError: missing chunk number 1 for toast value 58419 in pg_toast_17002
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:31:13)
    at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
    at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
    at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9)
    at async /usr/src/app/dist/services/database.service.js:136:17
    at async /usr/src/app/dist/repositories/database.repository.js:177:23 {
  query: '\n' +
    '        INSERT INTO face_search("faceId", embedding)\n' +
    '        SELECT id, embedding\n' +
    '        FROM asset_faces faces',
  parameters: undefined,
  driverError: error: missing chunk number 1 for toast value 58419 in pg_toast_17002
      at /usr/src/app/node_modules/pg/lib/client.js:526:17
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
      at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:31:13)
      at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
      at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
      at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9)
      at async /usr/src/app/dist/services/database.service.js:136:17
      at async /usr/src/app/dist/repositories/database.repository.js:177:23 {
    length: 132,
    severity: 'ERROR',
    code: 'XX001',
    detail: undefined,
    hint: undefined,
    position: undefined,
    internalPosition: undefined,
    internalQuery: undefined,
    where: undefined,
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'heaptoast.c',
    line: '784',
    routine: 'heap_fetch_toast_slice'
  },
  length: 132,
  severity: 'ERROR',
  code: 'XX001',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'heaptoast.c',
  line: '784',
  routine: 'heap_fetch_toast_slice'
}
microservices worker exited with code 1

Just to riterate, everything works fine with v1.106.4

bo0tzz Jul 28, 2024
Maintainer

There's some level of database corruption going on in your instance. Can you try connecting to postgres and running REINDEX DATABASE;? Note that this could take a minute to complete.

MadFisa Jul 28, 2024
Author

There's some level of database corruption going on in your instance. Can you try connecting to postgres and running REINDEX DATABASE;? Note that this could take a minute to complete.

@bo0tzz I'm not familiar with databses. So, I'm not quite sure what you meant. But here is what I tried,

Got into immich_postgres container using docker exec -it immich_postgres bash
psql -U username -d immich to connect to pstgres database
REINDEX DATABASE inside the console to rindex

The re indexing completed almost instantaneously, unlike what @bo0tzz mentioned . I'm not sure if I am doing it correctly. Is there a way to check if it actually did something?

Also, this did not fix the issue. On updating to release version of immich, same error appears.

afrojewelz · 2024-11-26T08:47:06Z

afrojewelz
Nov 26, 2024

me update to v121.0 then try v120.2 had also the exact same issue even with new initialized empty db and totally new instance ,plus similar config. not only via docker,also tried hand written deployment.yaml in k3s ,same error message from main server in immich

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating from v106.4 breaks the immich-server and microservices #11403

{{title}}

Replies: 2 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Updating from v106.4 breaks the immich-server and microservices #11403

MadFisa Jul 27, 2024

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Relevant log output

Additional information

Replies: 2 comments · 3 replies

mmomjian Jul 27, 2024 Collaborator

MadFisa Jul 28, 2024 Author

bo0tzz Jul 28, 2024 Maintainer

MadFisa Jul 28, 2024 Author

afrojewelz Nov 26, 2024

MadFisa
Jul 27, 2024

Replies: 2 comments 3 replies

mmomjian
Jul 27, 2024
Collaborator

MadFisa Jul 28, 2024
Author

bo0tzz Jul 28, 2024
Maintainer

MadFisa Jul 28, 2024
Author

afrojewelz
Nov 26, 2024