Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

THREESCALE-11020 Redis TLS certs and keys for porta and backend #1035

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

valerymo
Copy link
Contributor

@valerymo valerymo commented Nov 25, 2024

Jira: https://issues.redhat.com/browse/THREESCALE-11020

Add a way for the user to provide Redis TLS certs and keys for porta and backend

  • This PR enables Porta and Apisonator to load TLS configuration details for connecting to Redis. It introduces new environment variables that specify the locations of certificate files and indicate whether TLS mode is enabled.

  • The PR includes validation for Redis certificate-related fields and Redis URLs within the system-redis and backend-redis secrets, ensuring proper TLS communication.

  • Additionally, this PR validates the Sentinel Hosts to ensure they match the master Redis configuration defined in the APIManager CRs for system, backend, and queues.

  • Documentation has been updated to reflect these changes.

Validation

1. Install Redis Server for Test

- run make cluster/prepare/local DEV_SYSTEM_DB_POSTGRES=true - after command completion run: oc delete secret backend-redis system-redis, as we will create Secrets with Redis TLS fields later.
   export NAMESPACE=3scale-test
  • Copy the entire scripts block provided below, open your terminal and paste the script into the command line. This will create the Redis server, including following resources:
    • Secret: redis-tls-secret
    • ConfigMap: redis-config-redis
    • Deployment/pod: redis
    • Service: redis

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: redis-tls-secret
  namespace: 3scale-test
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFekNDQWZ1Z0F3SUJBZ0lVYS9FaGY1UDR3WWVnYVZhaGpqT0R6ME1QVTJnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUl0eld6OHcrTDBOYXAzN0FIcXR4Rjh6OEdXRFROUnoKOUxVa0JYcXU1SkZhc1lxejYrZ2k3S3dsWUlTeEFwVzd4NXprWDZ4bjBCeGxsMXVNdzExZHVyeDJNUjhlYXFxQQpxbElFVzZwTU1kTGNndkhJUit0TGhLRFVvaVpsajdKRkk1MGYvYjhUaXRKVTByY1dhYmd4SVA4QnlINUdxSkR2CmtFZGgvNC9SbTlYeWU4SmhuMEVqRzdPUWxTM1MvOGJHSGpoV2Zjdys2QnJidDI5TE1aTzdJRmFlRVh4azRaQjAKS1VNODJ0TFhxV1VwaUdxQUhuWncrZUtBSlZDOFBzUzB5NU9aTGt4TE9GUlJnajJRYkF3TlZ1emxBN0VMQ0U4dApyVkx3OWV3aEhsVkhUcVZ0UXYwNWZVb2ZqOURkb3Yxd3F2OG1uZXJ3OHBnRVR3Yi9RemY1VGRjQ0F3RUFBYU5UCk1GRXdIUVlEVlIwT0JCWUVGRGpocmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTApCUUFEZ2dFQkFDdCtPNTZnanZVTUo5MFdaekFFcEFLcTNFTDY3c1I4NXlYam5EeXZML1lRRjZZSzRhUmVaZ1JICms5WCs2cDNyd2tsQmF6MzFFTFNFSGlPUkVTWDJzNW9WSVlId01SNUpsOE5VZTB6NGVBZlNNcTNtQmFuYWxneWYKQktROHZRc1UxTGVJQkpGRVJ5dG5aVE03VThFbFVsUkZSZzAzVUcyVG5ibGNxalFRa1lTMmZYT3M1czBlMHI2eQpaektXVWZhemtzTlkzSHlUcUxRTDlEZUM5STd6R0s2c2E5dHhueTFOM0I5bWJHZ1l3dlFZNDlvTm1kem9oWW16CkxmVVZJeERSWUcwNnI1UldoTTVtZDhZNmJUSGo1TExHT1A2YlEvYWJteXZxbFpuVnZjWTZaandKUzhNWVJJSk8KMmxrczl3cXRjeENla3VJZFBrOUNsNlVBZ3E5TkJoVT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBakNDQWVxZ0F3SUJBZ0lVYUIyZGNLRXJuRHM0MEVqMitwT210bmpRQytVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09NVGN5TGpNd0xqSXdOeTR4T1RFd0hoY05NalF4TVRJME1EZ3lNakF6V2hjTgpNalV4TVRJME1EZ3lNakF6V2pBWk1SY3dGUVlEVlFRRERBNHhOekl1TXpBdU1qQTNMakU1TVRDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQVAxN2tGaHBRdDhpcUlRWEJOUEozbHBrOFFGUWJnUncKc2VlQkNoa1RTYTd5S0hvQS9VTHVHY3QzZHVubWZPeWdNeEhueW9LUmR0NDVnWWlDOGNnQjE4OWRDRHo2cy84ZAp2TUdoUFUrblhkOWxyVHNtYXpLNW9McnVETmI3TXhBVDFVdjNIMXkvbFErY2tPTkttLzVhWndoNnFxL25XVFZtCkdOZUVGcDJyNUczTkVXcTJXWm1KM0RCc3pMeUdFciszdUlJbFVsaUk2bXRZUGRRd1QwR3RqTDJpVldRVUd2Q2sKYWpHclNJQUVETVFCdFNYeE1yOTlzS2VhS21iVDJ1L09BOEJjYVlqbXdZVnN4VDIzczR2UFltaUhjTFVjVFE4MApTblBLSW03NDJrTzhLZE5Id1hjN1BpejhUTVB0WTU1QkU5ZlBIaGdueVNBMVZUeWNQR1cyZEMwQ0F3RUFBYU5DCk1FQXdIUVlEVlIwT0JCWUVGT3BwbncveVd2dFo2L0llVzhUUlhXeG1uMUtXTUI4R0ExVWRJd1FZTUJhQUZEamgKcmlHaDZxbUZVS2Q0OG14UERUWi9CYThBTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFCWnhOZGpZR1Y4M1RBUwp6amMyNjZnc3gwRTB0cTRVUnRGWHdFeXZTYkxvY3BmK1V2OCtmTFBYZGF0WG1lRm5MdlJZbXg0VGR1UlltMFBYCk1BRUpqcEE3UXdlM1NsSmd5R2loNk1PVVdIRFdNZnhQZWJQWUh5ZFQ3TXBKVVJNWkxjZ0tvWlVENWNKdDc1bVIKMk51Z3BvdDdxQnYybzZGdmI5cmpNYSt1WFdJUEdSekhidjJ6eVFSR1J3SUpZcitKaDNWbXcyRSt0Vit0VEUwQQovWlBxOTIxYnlWYkREZHI2aDMyUnRRN1NJUnAwLzBnWFZablY3NTB2Rm9Ub2xOK2wwLzJOeEVCWmZNSzVFcEdkCkViSFFvN2dETHg0REljZlhqZVlxUkFuYjNLbzhXK0ZrQS8wQUdQS2NSN2RIL2daVnFHMngrRk5QazQ4YzBDQkwKR0R2RjlRYncKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  redis-server.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRDllNUJZYVVMZklxaUUKRndUVHlkNWFaUEVCVUc0RWNMSG5nUW9aRTBtdThpaDZBUDFDN2huTGQzYnA1bnpzb0RNUjU4cUNrWGJlT1lHSQpndkhJQWRmUFhRZzgrclAvSGJ6Qm9UMVBwMTNmWmEwN0ptc3l1YUM2N2d6Vyt6TVFFOVZMOXg5Y3Y1VVBuSkRqClNwditXbWNJZXFxdjUxazFaaGpYaEJhZHErUnR6UkZxdGxtWmlkd3diTXk4aGhLL3Q3aUNKVkpZaU9wcldEM1UKTUU5QnJZeTlvbFZrRkJyd3BHb3hxMGlBQkF6RUFiVWw4VEsvZmJDbm1pcG0wOXJ2emdQQVhHbUk1c0dGYk1VOQp0N09MejJKb2gzQzFIRTBQTkVwenlpSnUrTnBEdkNuVFI4RjNPejRzL0V6RDdXT2VRUlBYeng0WUo4a2dOVlU4Cm5EeGx0blF0QWdNQkFBRUNnZ0VBRG5vaktWbUJyenJNZ3hiSmVNc2J2dS9xNzlkSElVdktiVjFxVlRwTHlBa2UKbExFL3hiWFJsVlJTWDFPQnFRWVJSS0dIYUdPa2RWYTFkalY4VjU3N1UyV04xZVcvcC85cnkyZEpHQ2FIN3YxZwpvbk0wUmlaaDdxc3Y0b3RnUkRmTnc5UHVYNTYxaGJtOGNLN1BML3k3eTdrdHpIUWJIVGlpakpTSHNpT2lIVDh1CitWc3oySzZmSnhHNUE2SE03bldvdlBuK1VydFd4OGd4OEJNQ1FOa3dUVnQvVmxzcE1wcUo1czdqeUNncGFQN2MKV0wrWnh2K2xUcFUwcnpJUTFZMWYvL0RSeUI3OVEycUl0dVhEdkJnNnI5ckExbytSV0JuNnA3WVA0T21SUE4vZQpiMVZZcG14c1BBL2ppQkhsRVFPZVJWbEVSYzhJSGVib2IxZFZ1Rm15b1FLQmdRRC9LeFlvVzNxaGNmUEYya2EvCmliZTkvKzlyQzNlTkdHUGwyaDVCeDJwK3k4dnlQb3NabmZ5STRKRzNtcXpVS1luSHFPVzhpTVpnR2ZOUlRNdVQKeVFSQitsVlFldHpOcDJUZCt4dWlCMUJmWHIyanowcVZvV3F4dFhrMklRWXFJN2NMSGZVZ2dGeUtsL2lRRGZlOQpwd0YwZ3c5Nm1vT012VWp1S1FmZjh2RTYxUUtCZ1FEK1R4SWZtc1VyY2VFdmpkeUdhNllweTBZcEx2OVhnSk0rCkxIc1ZmVTgzNXNWeVJaUDZrV1EwSGwyVCtHMDBqcStsU20vQ296cWNZeE5ObmtGMEhOdEw2SlJJWTBZZEExSFUKSmc1eTY3U2w0U0hHeWdYNTdrZEE4ajhyS1I4UXpmQ0RHT3Q5d2NwNjZ4MVlMSWEwRDYvV2h3elFOeXl3QzdnTwoxS3dvZlhyUCtRS0JnRzU4WUk2KzlYMWNVdnBUaGhpL2IvRDBGZDNhekR3cTJHNlpJRXJKSndLYUNjZnRidHQ3CnZmSWlrdFhXUW9sbkp3SnR6blB4SVR4UllEck9yc05oNGRjVHBzYy9POFpNZWU5b0lGSHJLdER3dTlwbkVsdHgKMWpuMll2S2VJQVkxQ3Jma2s5UXI0R1llWVlFMm14UGljVTNheGVRSGJYaU9LVHIrUnl1Z0RQVzFBb0dBZm5vdwoxMHNRR0tWUWkyZ1FiMElHcCs2Uy9GU0ZaYTFxalpkdHQ2aFV4OGFjR0ZNR1g2NERtZkFvTmpsdGhxQVlOeXFvCkhyTXpxU2VWS0JzM0RscHpybk1EbkdUVE1BYkFvYlF6cDNBV3JoRWp6VXdZWU03aTNTZ2R4b2R6RGRaK2NaVHAKT2VneG5hUmxPYjhiVjE0ZDQ2SFMrNU1WUkpEdmYyRENKbmtScFhFQ2dZRUFnVFB3MlVaSkJRVVdGc0NVRjczdgpGcWZSbzdZYmtweDAxcFlBcmV0ak41TTR6ZHZRUlZPQlFmVGRJTU9SbFJ4OVBnRitYL3UrT0szUzB1aS8xdWlECjhYU2QvMXFjS3ZOdUJ6ZnZNdTVGeVdKRHNCUGJaUHZuWnEzdTM5WktaYTNXNG92UE9VWlFERGlCTHNhM21CekwKcFdYNWFyVHNTazRSSDExQ28zRTN0akU9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
type: Opaque
EOF


cat << EOF | oc create -f -
apiVersion: v1
data:
  redis.conf: |+
    # redis.conf
    bind 0.0.0.0
    protected-mode no
    port 6379
    tls-port 6380
    tls-cert-file /etc/redis/certs/redis-server.crt
    tls-key-file /etc/redis/certs/redis-server.key
    tls-ca-cert-file /etc/redis/certs/ca.crt
    tls-auth-clients yes
    stop-writes-on-bgsave-error no
    save ""
kind: ConfigMap
metadata:
  name: redis-config-redis
EOF


cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: quay.io/fedora/redis-6
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/redis.conf
          subPath: redis.conf
        - name: redis-tls-volume
          mountPath: /etc/redis/certs
          readOnly: true
        command: ["/bin/sh", "-c", "redis-server /etc/redis/redis.conf"]
      volumes:
      - name: redis-config-volume
        configMap:
          name: redis-config-redis
      - name: redis-tls-volume
        secret:
          secretName: redis-tls-secret
EOF


cat << EOF | oc create -f -
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  ports:
    - port: 6379         # Non-TLS (unencrypted) port
      targetPort: 6379
      name: redis
    - port: 6380         # TLS port
      targetPort: 6380
      name: redis-tls
  selector:
    app: redis
  type: NodePort 
EOF

  • Expecting results example: redis server pod is running and service available
$ oc get svc |grep ^redis
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
redis   NodePort   172.30.56.166   <none>        6379:31290/TCP,6380:32389/TCP   10m

$ oc get pod |grep ^redis-
NAME                     READY   STATUS    RESTARTS   AGE
redis-5dc466fc8b-764hl   1/1     Running   0          10m

2. Certificates preparing

- Create CA, Client and Server Certificates, using Server IP as Common Name (CN): - Create directory `Certs`, `cd Certs`, and run following commands to create server and client certificates, that will be used for test.
openssl genpkey -algorithm RSA -out ca.key
openssl req -x509 -key ca.key -out ca.crt -days 365 -subj "/CN=172.30.47.231"

openssl genpkey -algorithm RSA -out redis-client.key
openssl req -new -key redis-client.key -out redis-client.csr -subj "/CN=redis-client.example.com"
openssl x509 -req -in redis-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-client.crt -days 365

openssl genpkey -algorithm RSA -out redis-server.key
openssl req -new -key redis-server.key -out redis-server.csr -subj "/CN=172.30.47.231"
openssl x509 -req -in redis-server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out redis-server.crt -days 365
  • Expected files to be created:
Certs $ ls
ca.crt			redis-client.crt	redis-server.crt
ca.key			redis-client.csr	redis-server.csr
ca.srl			redis-client.key	redis-server.key

3. Update Redis Server with new server certificate

  • Update redis-tls-secret secret, using new created:

    • ca.crt
    • redis-server.crt
    • redis-server.key
  • Restart redis pod

4. Install 3scale

#### 4.1. Create Redis secrets for 3scale
- Below is a script to create the system-redis and backend-redis secrets with dummy client certificates. In the next step, we will replace these dummy certificates with valid client certificates, using the UI for convenience. The valid certificates will be sourced from the files created in the previous section. Additionally, we will update the Redis server URL to reflect the service IP of our Redis server.

You may prepare the secrets in whichever way is most convenient for you.

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: system-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: system
    app: 3scale-api-management
    threescale_component: system
data:
  SENTINEL_HOSTS: ''
  SENTINEL_ROLE: ''
  REDIS_SSL_CA: ''
  REDIS_SSL_CERT: ''
  REDIS_SSL_KEY: ''
  URL: cmVkaXNzOi8vMTcyLjMwLjU2LjE2Njo2MzgwLzI=
type: Opaque
EOF

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata:
  name: backend-redis
  namespace: 3scale-test
  labels:
    apimanager.apps.3scale.net/watched-by: backend
    app: 3scale-api-management
    threescale_component: backend
data:
  REDIS_STORAGE_URL: cmVkaXNzOi8vMTcyLjMwLjU2LjE2Njo2MzgwLzA=
  REDIS_QUEUES_SENTINEL_HOSTS: ''
  REDIS_STORAGE_SENTINEL_ROLE: ''
  REDIS_SSL_CA: ''
  REDIS_SSL_CERT: ''
  REDIS_SSL_KEY: ''
  REDIS_SSL_QUEUES_CA: ''
  REDIS_SSL_QUEUES_CERT: ''
  REDIS_SSL_QUEUES_KEY: ''
  REDIS_QUEUES_URL: cmVkaXNzOi8vMTcyLjMwLjU2LjE2Njo2MzgwLzE=
  REDIS_QUEUES_SENTINEL_ROLE: ''
  REDIS_STORAGE_SENTINEL_HOSTS: ''
type: Opaque
EOF

  • Update Client Certificates in system-redis and backedn-redis secrets via UI. The following tables are for matching data field names with the certificate files created before:

  • Secret: system-redis

Data field Certificate file name
REDIS_SSL_CA ca.crt
REDIS_SSL_CERT redis-client.crt
REDIS_SSL_KEY redis-client.key
  • Secret: backend-redis
Data field Certificate file name
REDIS_SSL_CA ca.crt
REDIS_SSL_CERT redis-client.crt
REDIS_SSL_KEY redis-client.key
REDIS_SSL_QUEUES_CA ca.crt
REDIS_SSL_QUEUES_CERT redis-client.crt
REDIS_SSL_QUEUES_KEY redis-client.key

Please note:

  • We are using a common CA for both the Redis server and client certificates.
  • We are using the same client certificates for both the Redis system and the backend, including QUEUES.
  • Please don't forget to update the Redis IP in the URLs to match the service IP and CN.
  • Use secure Redis URLs. They should look like rediss://:6380/0, where rediss indicates a secure connection and port 6380 is used for secure Redis connections. For example:
    • REDIS_QUEUES_URL: rediss://172.30.56.166:6380/1
    • REDIS_STORAGE_URL: rediss://172.30.56.166:6380/0
    • URL: rediss://172.30.56.166:6380/2

4.2. Create s3-credentials secret

cat << EOF | oc create -f -
kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: QUtJQVY2SVpYMk9ZQ09OWERFSlkK
  AWS_SECRET_ACCESS_KEY: aU5VbWdZY3hjSDF3azBlUlB0SytmTERHVVMvU0hxM1pKNVBlQy9xYQo=
  AWS_BUCKET: dm1vY2NzZjZxOGxyZWRoYXRyaG9hbW9wZXJhdG9ydGhyZWUtYW9mZwo=
  AWS_REGION: ZXUtd2VzdC0xCg==
type: Opaque
EOF

4.3. Create APIManager CR and Run Operator

Please set wildcardDomain before creation APIManager:
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')

cat << EOF | oc create -f -
apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: 3scale
spec:
  system: 
    systemRedisTLSEnabled: true
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  backend:
    backendRedisTLSEnabled: true
    queuesRedisTLSEnabled: true    
  wildcardDomain: $DOMAIN
  externalComponents:
    backend:
      redis: true
    system:
      redis: true
      database: true
EOF
  • run Operator to install 3scale:
    make run
    

5 Check results

5.1 Check Environment Variables and Certificates in Pods

- Check that env vars are defined in pods, check cert files:

System pods: system-sidekiq and system-app

  • Login to the pod (oc rsh podname), and run the following commands to verify that the certificate environment variables are defined
    and the certificate files are populated for the System:
env |grep -E "REDIS_CLIENT_CERT|BACKEND_REDIS_CLIENT_CERT"
env |grep -E "REDIS_CA_FILE|BACKEND_REDIS_CA_FILE"
env |grep -E "REDIS_PRIVATE_KEY|BACKEND_REDIS_PRIVATE_KEY"
env |grep -E "REDIS_SSL|BACKEND_REDIS_SSL"
cat /tls/system-redis/system-redis-ca.crt
cat /tls/system-redis/system-redis-client.crt
cat /tls/system-redis/system-redis-private.key
cat /tls/backend-redis-ca.crt
cat /tls/backend-redis-client.crt
cat /tls/backend-redis-private.key
  • Backend pods:backend-cron, backend-listener, backend-worker
  • Login to the pod (oc rsh podname), and run the following commands to verify that the certificate environment variables are defined and the certificate files are populated for the Backend:
env |grep -E "CONFIG_REDIS_CA_FILE|CONFIG_QUEUES_CA_FILE"  
env |grep -E "CONFIG_REDIS_CERT|CONFIG_QUEUES_CERT"
env |grep -E "CONFIG_REDIS_PRIVATE_KEY|CONFIG_QUEUES_PRIVATE_KEY"
env |grep -E "CONFIG_REDIS_SSL|CONFIG_QUEUES_SSL"
cat /tls/queues/config-queues-ca.crt
cat /tls/queues/config-queues-client.crt
cat /tls/queues/config-queues-private.key
cat /tls/backend-redis-ca.crt
cat /tls/backend-redis-client.crt
cat /tls/backend-redis-private.key

6 TLS - Test Cases

In the previous sections, we provided details for testing of following scenario:
Test 1 - negative

  1. APIManagerCR: Redis TLS is Enabled for system, backend and queues.
  2. backend-redis and system-redis secrets are missing required TLS fields
  • Expected results:
    • installation is not progressing
    • backend-redis secret validation errors in operator log:
2025-02-10T07:54:15+02:00	ERROR	Reconciler error	{"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"3scale","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "3scale", "reconcileID": "27ade290-b64f-4b40-b99f-8fc4f90587b7", "error": "validation errors for Redis TLS configuration in 'backend-redis' secret: 'backendRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_CA' is required in secret 'backend-redis' Secret field 'REDIS_SSL_CERT' is required in secret 'backend-redis' Secret field 'REDIS_SSL_KEY' is required in secret 'backend-redis']\n'queuesRedisTLSEnabled: true' is set in apimanager. Secret validation errors: [Secret field 'REDIS_SSL_QUEUES_CA' is required in secret 'backend-redis' Secret field 'REDIS_SSL_QUEUES_CERT' is required in secret 'backend-redis' Secret field 'REDIS_SSL_QUEUES_KEY' is required in secret 'backend-redis']"}

Test 2 - negative

  1. APIManagerCR: Redis TLS is Enabled for system, backend and queues.
  2. backend-redis secret - TLS fields are empty
  3. system-redis secrets is missing required TLS fields
  • Expected results:
    • installation is not progressing
    • system-redis secret - validation errors in operator log - notification that fields are required
2025-02-10T08:43:20+02:00	ERROR	Reconciler error	{"controller": "apimanager", "controllerGroup": "apps.3scale.net", "controllerKind": "APIManager", "APIManager": {"name":"3scale","namespace":"3scale-test"}, "namespace": "3scale-test", "name": "3scale", "reconcileID": "b710a16d-e567-4e9a-b553-4347e83ea744", "error": "validation errors for Redis TLS configuration in 'system-redis' secret: Secret field 'REDIS_SSL_CA' is required in secret 'system-redis'\nSecret field 'REDIS_SSL_CERT' is required in secret 'system-redis'\nSecret field 'REDIS_SSL_KEY' is required in secret 'system-redis'"}

Test 3 - negative

  1. APIManagerCR: Redis TLS is Enabled for system, backend and queues.
  2. system-redis and backend-redis secrets - TLS fields are populated with valid certificates
  3. URL fields are correct and contains redis secure urls rediss
  • Expected results:
    • system and backend pods are not running, Errors like following:
oc logs backend-worker-65f4fcd89f-rb6qj  -c backend-redis-svc
Error connecting to Redis queue storage: Failed to load CA Certificate or CA Path
Error connecting to Redis queue storage: Failed to load CA Certificate or CA Path
....
  - No Errors in Operator log

Test4
1. APIManagerCR: Redis TLS is Enabled for system, backend and queues.
2. backend-redis secret - TLS fields are populated correctly
3. system-redis secrets - TLS fields are empty

  • Expected results:
    Installation completed successfully, All pods are running no errors.

@valerymo valerymo requested a review from a team as a code owner November 25, 2024 13:09
@valerymo valerymo changed the title [WIP] New - THREESCALE-11020 Redis TLS certs and keys for porta and backend [WIP] THREESCALE-11020 Redis TLS certs and keys for porta and backend Nov 25, 2024
@valerymo valerymo force-pushed the THREESCALE-11020-2 branch 4 times, most recently from 5414a05 to 2eea781 Compare December 2, 2024 13:34
@valerymo valerymo changed the title [WIP] THREESCALE-11020 Redis TLS certs and keys for porta and backend THREESCALE-11020 Redis TLS certs and keys for porta and backend Dec 4, 2024

SystemSecretSystemRedisSslCa = "SSL_CA"
SystemSecretSystemRedisSslCert = "SSL_CERT"
SystemSecretSystemRedisSslKey = "SSL_KEY"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valerymo think we need to agree on a naming scheme here as it may be possible that a customer has different certs for Redis and the System-db. Propose that we prefix with CACHE_ as may not be using redis going forward

Suggested change
SystemSecretSystemRedisSslKey = "SSL_KEY"
SystemSecretSystemRedisSslCa = "CACHE_SSL_CA"
SystemSecretSystemRedisSslCert = "CACHE_SSL_CERT"
SystemSecretSystemRedisSslKey = "CACHE_SSL_KEY"

and I use the prefix SYSTEMDB_

Suggested change
SystemSecretSystemRedisSslKey = "SSL_KEY"
SystemSecretSslCa = "SYSTEMDB_SSL_CA"
SystemSecretSslCert = "SYSTEMDB_SSL_CERT"
SystemSecretSslKey = "SYSTEMDB_SSL_KEY"

What do you think ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@austincunningham thanks for comment.
It's Done,

  • "REDIS_" prefix was added to "SSL" names that appear in backend-redis and system-redis secrtes
  • done in separate commit
  • Tested locally
  • Validation notes updated
    Thanks

@@ -1535,3 +1537,26 @@ type APIManagerList struct {
func init() {
SchemeBuilder.Register(&APIManager{}, &APIManagerList{})
}

func (apimanager *APIManager) IsSystemRedisTLSEnabled() bool {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the RedisTLSEnabled is a global flag, why do we make a differentiation between system and backend redis here?

Can you have system redis TLS enabled but not worker TLS enabled? Doesn't system app connect to system redis and backend redis - which would to me mean, you can't have TLS on system and not on worker? Not sure about the other way around

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for comment @MStokluska . We can't have TLS on System if no TLS on Backend. But it can be vice-a-versa - TLS on Backend but not on System. This is what I see from Jira - System is depended to Backend (system has env vars from backend-redis secret), but Backend is not related to system-redis secret. Hope it's ok. But please tell me if I need check it from Joan. Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So based on what you are saying:
Backend TLS - ON and System TLS - OFF is supported.
Backend TLS - OFF and System TLS - ON is not supported.

Based on the Jira description, it looks to me like they actually can be enabled independent of each other since backend is not reliant on system (meaning from backend pov, system SSL can be on or off). System is reliant on backend redis, but there seems to be individual flags for system/backend in system envs:
BACKEND_REDIS_SSL - REDIS_SSL.
Do you agree? Is this how it is currently working?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is how it's currently working.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok thanks, do we have any indication (apim error or so) to let users know that tls on backend must be enabled if system tls is enabled?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have it in documentation. I reviewed docs to make this is more clear, separate commit.
Thanks

@@ -398,6 +437,9 @@ func (backend *Backend) buildBackendWorkerEnv() []v1.EnvVar {
)
}

if backend.Options.RedisTLSEnabled {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have these in buildBackendCommonEnv - common in my understanding applies to both, worker and listener - is this really required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thank you

@@ -422,6 +464,9 @@ func (backend *Backend) buildBackendListenerEnv() []v1.EnvVar {
v1.EnvVar{Name: "CONFIG_LISTENER_PROMETHEUS_METRICS_ENABLED", Value: "true"},
)
}
if backend.Options.RedisTLSEnabled {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have these in buildBackendCommonEnv - common in my understanding applies to both, worker and listener - is this really required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Thank you

func (backend *Backend) backendVolumes() []v1.Volume {
res := []v1.Volume{}

if backend.Options.RedisTLSEnabled {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just side note - so are we saying here that Redis TLS can be enabled on worker or listener IF system fails to have TLS enabled?

Copy link
Contributor Author

@valerymo valerymo Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

System TLS certs are coming from both system-redis and backend-redis secrets. System TLS envs BACKEND_REDIS_CA_FILE, BACKEND_REDIS_PRIVATE_KEY - they are populated from Backend-redis secret. So System is depends on Backend TLS definitions.
But Backend TLS is not related to system - all certs are coming from backend-redis.
Hope I understand the quesion. Thank you Michal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I wrote before, as I understand - it can be TLS on backend, but not on system. Backend TLS is not dependent to system, but system TLS is dependent to backend. System is populating BACKEND_REDIS_CLIENT_CERT and BACKEND_REDIS_PRIVATE_KEY from backend-redis secret.

@@ -69,12 +70,14 @@ func (r *RedisOptionsProvider) GetRedisOptions() (*component.RedisOptions, error
r.setPersistentVolumeClaimOptions()
r.setPriorityClassNames()
r.setTopologySpreadConstraints()
//r.setTLSEnabled()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleaned. Thank you

| REDIS_SSL_KEY | The private key for the Redis client certificate | Required to set TLS Redis connection. Only for TLS |
| REDIS_SSL_QUEUES_CA | Redis Queues Certificate Authority (CA) certificate | Required to set TLS Redis connection. Only for TLS |
| REDIS_SSL_QUEUES_CERT | Redis Queues client certificate | Required to set TLS Redis connection. Only for TLS |
| REDIS_SSL_QUEUES_KEY | The private key for the Redis Queues client certificate | Required to set TLS Redis connection. Only for TLS |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you missing SSL_MODE ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @austincunningham ,
I'm not using SSL_MODE but using REDIS_SSL and BACKEND_REDIS_SSL (for system), CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL (for backend),
as it defined in Task.
These env vars are set in Pods, but not in secret.
They set to true if Any of related env vars defined in secret. This is how it's defined in Task/Requirements,
I rechecked it with Joan - thread https://app.slack.com/client/E030G10V24F/search).
These are also links to porta/backend:
System/porta:
- BACKEND_REDIS_SSL: https://github.com/3scale/porta/blob/master/config/examples/backend_redis.yml#L14
- REDIS_SS: https://github.com/3scale/porta/blob/master/config/examples/redis.yml#L13
Backend/apisonator:
- CONFIG_REDIS_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L44
- CONFIG_QUEUES_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L27

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure that is the same as ssl mode.

Although not sure if the same tls values are available for REDIS as are available for MySQL and Postgres

🤔

DATABASE_SSL_MODE. Values: Mysql ref., Postgres ref.

Don't see similar documented in redis docs so guess its ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@austincunningham , I'm not using SSL_MODE but using REDIS_SSL and BACKEND_REDIS_SSL (for system), CONFIG_REDIS_SSL and CONFIG_QUEUES_SSL (for backend),
as it defined in Task.
These env vars are set in pods, but not in secret. And these env vars are used in Porta and apisonator (links below)
These env vars are set to true if Any of related env vars defined in secret.
I rechecked it with Joan - thread https://app.slack.com/client/E030G10V24F/search).
These are also links to porta/backend:
System/porta:
- BACKEND_REDIS_SSL: https://github.com/3scale/porta/blob/master/config/examples/backend_redis.yml#L14
- REDIS_SS: https://github.com/3scale/porta/blob/master/config/examples/redis.yml#L13
Backend/apisonator:
- CONFIG_REDIS_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L44
- CONFIG_QUEUES_SSL: https://github.com/3scale/apisonator/blob/master/openshift/3scale_backend.conf#L27
Thank you

| AppLabel | `appLabel` | string | No | `3scale-api-management` | The value of the `app` label that will be applied to the API management solution
| TenantName | `tenantName` | string | No | `3scale` | Tenant name under the root that Admin UI will be available with -admin suffix.
| **Field** | **json/yaml field**| **Type** | **Required** | **Default value** | **Description** |
| --- | --- | --- | --- | --- |---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goland does this sort of auto edit, I usually edit md files in another ide to stop this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@austincunningham , Thanks a lot! I updated sections related to Redis TLS . I used now VSCodium.
Looks like there are more places where extra spaces added, but didn't touch them. Table of contents updated, as VSCodium required this, I checked, seems it's working fine.

- Client Private Key
- TLS certificate files are populated from the **backend-redis** and **system-redis** secrets.

- The tables below show the mapping between TLS certificate environment variables in the pods, their corresponding definitions in the related Redis backend and system secrets.
Copy link
Contributor

@austincunningham austincunningham Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think we should only be documenting the ENV VAR the user has to set and not the ones set my the operator to avoid confusion

Copy link
Contributor Author

@valerymo valerymo Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@austincunningham thank you for comment. It's User Guide. I thought it could help to User to understand how it's working, and investigate/debug if required. If you strongly disagree I will remove this, but seems to me it can be useful. Thanks

@valerymo valerymo force-pushed the THREESCALE-11020-2 branch 3 times, most recently from 31bfbdd to b695c71 Compare February 3, 2025 14:10
@@ -84,6 +84,13 @@ type APIManagerSpec struct {
PodDisruptionBudget *PodDisruptionBudgetSpec `json:"podDisruptionBudget,omitempty"`
// +optional
Monitoring *MonitoringSpec `json:"monitoring,omitempty"`

// +optional
SystemRedisTLSEnabled *bool `json:"systemRedisTLSEnabled,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please move this to system spec

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks

// +optional
SystemRedisTLSEnabled *bool `json:"systemRedisTLSEnabled,omitempty"`
// +optional
BackendRedisTLSEnabled *bool `json:"backendRedisTLSEnabled,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please move both backend redis tls flags to backend spec

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks

tlsEnabled := false
if apimanager.Spec.SystemRedisTLSEnabled != nil &&
*apimanager.Spec.SystemRedisTLSEnabled &&
apimanager.Spec.ExternalComponents != nil &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

external components check should not be here. All installations of 3scale on 2.16 + must use external.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks

tlsEnabled := false
if apimanager.Spec.BackendRedisTLSEnabled != nil &&
*apimanager.Spec.BackendRedisTLSEnabled &&
apimanager.Spec.ExternalComponents != nil &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

external components check should not be here. All installations of 3scale on 2.16 + must use external.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks

tlsEnabled := false
if apimanager.Spec.QueuesRedisTLSEnabled != nil &&
*apimanager.Spec.QueuesRedisTLSEnabled &&
apimanager.Spec.ExternalComponents != nil &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

external components check should not be here. All installations of 3scale on 2.16 + must use external.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks

}

func (a *APIManager) backendRedisTLSValidationBackendSecret() {

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty function

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed . Thanks

func (r *APIManagerReconciler) validateRedisTLSSystemRedisSecret(cr *appsv1alpha1.APIManager) field.ErrorList {
fieldErrors := field.ErrorList{}
fieldErrors = append(fieldErrors, r.validateRedisTLSRedisSecret(cr, "system-redis",
"REDIS_SSL_CA", "REDIS_SSL_CERT", "REDIS_SSL_KEY",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems to me we are doubling up the validation.
The validation could be synced with what we have so far when it comes to validation these secrets, which is here:
pkg/3scale/amp/operator/highavailability_options_provider.go

The validateCR function, which uses this function, is used for validation of the apim CR. Not secrets.
Please move the logic of validating the secrets to https://github.com/3scale/3scale-operator/blob/master/pkg/3scale/amp/operator/highavailability_options_provider.go

You can check on what @austincunningham did here: #1026

By doing this, we are validating our secrets early (the first thing that happens after preflights and before reconciling all 3scale components) and in the correct place.

@@ -1,43 +1,46 @@
# User Guide

<!--ts-->
* [User Guide](#user-guide)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know what triggered refactor of this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't know, but the doc is updated now, as TLS enabled flags moved to system and backend specs.
Thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be nice to figure out why this has happened?
Are you using the toc generator?

@@ -48,8 +48,6 @@ github.com/Azure/go-autorest/tracing v0.5.0/go.mod h1:r/s2XiOKccPW3HrqB+W0TQzfbt
github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/MStokluska/3scale-porta-go-client v0.0.0-20250124150520-23225e662421 h1:P/vWCWycdFARt5APkZ6yJs/1DWvkeESSRZZHMl1wkFo=
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! 🤦

}

// check sentinelHostsFieldName
data, exists := secret.Data[sentinelHostsFieldName]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sentinel field can exists as empty - does it mean we are expecting sentinel to be setup correctly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, enduser can deside not to use Sentinel, same as it was without TLS

if err != nil {
return fmt.Errorf("URL has an invalid structure: %s", host)
}
if parsedURL.Scheme == "rediss" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we know for sure that if using sentinels with tls, every host will have rediss?
are there any other scenarios where this can be set to a different value?
In general, I think I would just check if:

  1. TLS required KEYS are present in secret
  2. TLS required KEYS have non-empty VALUES

The rest of validation will be performed via preflight + components directly.

Copy link
Contributor Author

@valerymo valerymo Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

regarding "every host will have rediss" - Yes, as for example. We want use TLS for client communication with Redis Server. Redis Server - secured. We have 3 Sentinel hosts for Backend (for HA), first - using TLS (rediss), - other - not. If Sentinel is available - Client will not communicate with Redis Server directrly, it will use Sentinel. Secure Sentinel found, and communication established. Secure sentinel failed. Sentinel will do switch to "backup" Sentinel, but it's not secure. Communication - failed. We need prevent this case, and check that If Sentinel hosts defined and We want use TLS - then All Sentine hosts should be secure (rediss://)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validation moved to HA, and only basic validation done now. Thank you

@valerymo valerymo force-pushed the THREESCALE-11020-2 branch 3 times, most recently from 91ff191 to 00923cd Compare February 6, 2025 13:29
@valerymo valerymo force-pushed the THREESCALE-11020-2 branch 3 times, most recently from 76ac677 to f95667d Compare February 9, 2025 16:18
return false
}

func ValidateRedisURL(url string) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not used, I think we can drop this?

return nil
}

func ValidateRedisSentinelHostsForTLS(sentinelHosts string) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used, I think we can drop this?

@@ -927,9 +1059,10 @@ func (system *System) SidekiqDeployment(containerImage string) *k8sappsv1.Deploy
Command: []string{
"bash",
"-c",
"bundle exec sh -c \"until rake boot:redis && curl --output /dev/null --silent --fail --head http://system-master:3000/status; do sleep $SLEEP_SECONDS; done\"",
"bundle exec sh -c \"until rake boot:redis && curl --insecure --output /dev/null --silent --fail --head http://system-master:3000/status; do sleep $SLEEP_SECONDS; done\"",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we bypassing ssl cert validation here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants