TLS Client certificate Server is not reachable

[AdriDevelopsThings]

The bug

I'm trying to use the new client certificate feature on iOS 17.5.1 but I'm a Server is not reachable Toast.

The client certificate was set as you can see in the log section.

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.110.0

Version of Immich Mobile App

v1.110.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    labels:
      - traefik.enable=true
      - traefik.http.routers.immich.rule=Host(`**********`)
      - traefik.http.routers.immich.tls.options=client_certificate@file
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - default
      - traefik
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always
  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
volumes:
  model-cache:
networks:
  traefik:
    name: traefik
    external: true

Your .env content

UPLOAD_LOCATION=******
IMMICH_VERSION=release
DB_PASSWORD=******
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=./postgres
REDIS_HOSTNAME=immich_redis

Reproduction steps

1. configure your reverse proxy to require a client certificate
2. open the immich app on iOS and log out
3. import the client certificate in the api
4. try to connect to server

Relevant log output

2024-07-27 17:35:50.641116 | SEVERE   | ApiService           | Error while checking server availability | 
        TLSV1_ALERT_CERTIFICATE_REQUIRED(tls_record.cc:592) error 268436572 |
#0      IOClient.send (package:http/src/io_client.dart:96)
<asynchronous suspension>
#1      BaseClient._sendUnstreamed (package:http/src/base_client.dart:93)
<asynchronous suspension>
#2      Future.timeout.<anonymous closure> (dart:async/future_impl.dart:949)
<asynchronous suspension>
#3      ApiService._isEndpointAvailable (package:immich_mobile/services/api.service.dart:105)
<asynchronous suspension>
#4      ApiService._resolveEndpoint (package:immich_mobile/services/api.service.dart:85)
<asynchronous suspension>
#5      ApiService.resolveAndSetEndpoint (package:immich_mobile/services/api.service.dart:67)
<asynchronous suspension>
#6      LoginForm.build.getServerLoginCredential (package:immich_mobile/widgets/forms/login/login_form.dart:99)
<asynchronous suspension>
2024-07-27 17:33:34.408656 | INFO     | HttpSSLCertOverride  | Setting client certificate |

Additional information

No response

[alextran1502]

@yjiang-c Do you have any thoughts on this?

[yjiang-c]

What is the command to generate client certificate? As I know, you need -legacy option in OpenSSL command to generate client certificates for iOS

[AdriDevelopsThings]

What is the command to generate client certificate? As I know, you need -legacy option in OpenSSL command to generate client certificates for iOS

Yes, I know, I used the -legacy option. It's also possible for me to add the certificate on iOS natively.

[pedropombeiro]

I'm seeing the same issue, and I'm able to access the site through Safari on iOS after having imported the certificate in the system.

[yjiang-c]

Thanks for your reporting. I can confirm that the issue is reproduced on iOS but not on Android. Based on my debug log on server side, the client does not send certificate to server.

I will investigate the issue. BTW, is there anybody produce this issue on older iOS versions before 17.5.1?

[yjiang-c]

Based on log analysis, I cannot find any wrong with my code. In addition, the issue is not reproducible on Android client, I suspected it should be problem on dart library.

I raised an issue to dart http group for more investigation. dart-lang/http#1277

[yjiang-c]

After looking at source code of dart sdk source code, I suspected that SecurityContext::useCertificateChain has wrong or outdated documentation. If that is case, the fix should be very simple.

@alextran1502 Since I have no development/test environment for iOS app, is there any way for you run test internally to check whether the issue could be solved by a patch I am going to propose? Thanks!

[alextran1502]

@yjiang-c can you walk me to how to set it up for testing? I can help you test

[yjiang-c]

@alextran1502 If you like, you can follow this web page https://docs.nginx.com/nginx-management-suite/admin-guides/configuration/secure-traffic/ to setup nginx and generate client certificate, but it may need some learning curve if you have never done it before.

Another option is that I can email you my client certificate file and URL of my immich server. You just need to run iOS app to check whether you can got immich login page after logout.

Which one do you prefer?

[alextran1502]

@yjiang-c the second option sounds easier, let's do that. I won't be able to get around this until late tonight or early tomorrow

[yjiang-c]

@alextran1502 Thanks a lot for your quick reply. I just emailed you about my immich URL and corresponding client certificate. Please let me know if you did not receive it. Thanks!

[pedropombeiro]

If it is helpful, I can also help test the app, as I already have the proxy server and certificate configured, in case there's a beta version available for testing.

[yjiang-c]

@pedropombeiro thanks for your kind help offer. The currently we are still in investigation stage by doing some internal tests. We will let you know if we need help.

[yjiang-c]

We found the root cause and the issue will be fixed soon.

Thanks all involved for reporting issues and @alextran1502 for quick test.

[alextran1502]

Thanks @yjiang-c for implementing this mechanism ❤️