Skip to content

Postgres Upgrade 13.3

Jump to bottom
lbakerchef edited this page Jul 29, 2021 · 33 revisions

Upgrades page: https://docs.chef.io/server/upgrades/

Standalone Internal Postgres

Upgrade steps

  1. It is strongly recommended to run Vacuum full for smooth upgrade $BINDIR/vacuumdb --all –full
  2. Stop the chef-server chef-server-ctl stop
  3. Download the upgrade build (14.6.33+20210719175317 used for testing)
  4. Install the package dpkg -i /path/to/chef-server-core-<version>.deb or rpm -Uvh --nopostun /path/to/chef-server-core-<version>.rpm
  5. Upgrade the chef-server CHEF_LICENSE='accept' chef-server-ctl upgrade
  6. Start and cleanse the chef-server 
    CHEF_LICENSE='accept' chef-server-ctl upgrade
chef-server-ctl start
chef-server-ctl cleanup
  7. Reindex the database $BINDIR/reindexdb --all

Testing data

  1. Install version -> 14.5.29
  2. Upgrade version -> 14.6.33+20210719175317
  3. PG data -> 158G /var/opt/opscode/postgresql/9.6/data
  4. Number of nodes -> 565596
  5. opscode_chef size -> 49 GB
  6. bifrost -> 68 GB
  7. oc_id -> 27 GB
  8. machine type -> t3.2xlarge

Vacuum testing data

  1. Total time take for vacuum full -> 183m49.091s
  2. PG data -> 21G /var/opt/opscode/postgresql/9.6/data (reduced from 158G)
  3. Number of nodes -> 565596
  4. opscode_chef size -> 21 GB (reduced from 49 GB)
  5. bifrost -> 7127 kB (reduced from 68 GB)
  6. oc_id -> 30 MB (reduced from 27 GB)

Upgrade testing data

  1. Total time take for upgrade process -> approx 15 min
  2. PG data -> 21G /var/opt/opscode/postgresql/13.3/data (same as before)
  3. Number of nodes -> 565596
  4. opscode_chef size -> 21 GB (same as before)
  5. bifrost -> 8949 kB (increased from 7127 kB)
  6. oc_id -> 32 MB (increased from 30 MB)

Reindex testing data

  1. Total time take for reindex process -> approx 5 min
  2. PG data -> 21G /var/opt/opscode/postgresql/13.3/data (same as before)
  3. Number of nodes -> 565596
  4. opscode_chef size -> 20 GB (reduced from 21 GB)
  5. bifrost -> 8773 kB (reduced from 8949 kB)
  6. oc_id -> 32 MB (same as before)

Recommendation and Results

  1. Running vacuum full before the upgrade is recommended as it will reduce the data size to optimal and all the further steps will take less time
  2. Need same amount of free space as the pg data consumes
  3. Total time take for vacuuming, upgrading and reindexing 158G data is approximately 3.5 hr

Tiered Internal Postgres

Upgrade steps

  1. Vacuum full (recommendation)
  2. Stop the front end chef server only
  3. Take a copy of the /opt/opscode/embedded/postgresql/* (need to remove this step)
  4. Download and dpkg install the upgrade build on front end(14.7.11+20210727061052 used for testing)
  5. Restore /opt/opscode/embedded/postgresql/* (need to remove this step)
  6. Stop the back end chef server
  7. Take a copy of the /opt/opscode/embedded/postgresql/* (need to remove this step)
  8. Download and dpkg install the upgrade build on back end(14.7.11+20210727061052 used for testing)
  9. Restore /opt/opscode/embedded/postgresql/* (need to remove this step)
  10. Upgrade the chef-server on the back end
  11. Zip and copy the /etc from back end to front end and unzip /etc on front end
  12. Reconfigure and start the chef-server on front end

Testing data (already vacuumed data)

  1. Install version -> 14.5.29
  2. Upgrade version -> 14.7.11+20210727061052
  3. PG data -> 23G /var/opt/opscode/postgresql/9.6/data/
  4. Number of nodes -> 565596
  5. opscode_chef size -> 22 GB
  6. bifrost -> 8311 kB
  7. oc_id -> 7327 kB
  8. machine type -> t3.medium (extended the storage to 150 GB)

Upgrade testing data

  1. Total time take for upgrade process -> approx 20 min
  2. PG data -> 22G /var/opt/opscode/postgresql/13.3/ (reduced from 23 GB)
  3. Number of nodes -> 565596
  4. opscode_chef size -> 22 GB (same as before)
  5. bifrost -> 8885 kB (increased from 8311 kB)
  6. oc_id -> 7909 kB (increased from 7327 kB)

Recommendation and Results

  1. Include vacuum full process
  2. Need same amount of free space as the pg data consumes
  3. Total time take for upgrading 23GB vacuumed data is approximately 20 min

External Postgres

Upgrade steps

chef-server box

  1. Download the upgrade build (typically using curl or mixlib).
  2. Install the package.
    sudo rpm -U -v --force <path-to-package>
    or
    sudo dpkg -iEG <path-to-package>
  3. Upgrade chef-server.
sudo CHEF_LICENSE='accept' chef-server-ctl upgrade
sudo chef-server-ctl start
sudo chef-server-ctl cleanup

postgresql box

  1. Update packages and install the new PostgreSQL 13.3.
  2. Check if there are any differences in the config files.Make sure to update them if required
diff /etc/postgresql/12/main/postgresql.conf /etc/postgresql/13/main/postgresql.conf
diff /etc/postgresql/12/main/pg_hba.conf /etc/postgresql/13/main/pg_hba.conf
  1. Stop the PostgreSQL service. sudo systemctl stop postgresql.service
  2. Log in as the postgres user. su postgres
  3. Check clusters (notice the --check argument, this will not change any data).
/usr/lib/postgresql/13/bin/pg_upgrade \
  --old-datadir=/var/lib/postgresql/9.6/main \
  --new-datadir=/var/lib/postgresql/13/main \
  --old-bindir=/usr/lib/postgresql/9.6/bin \
  --new-bindir=/usr/lib/postgresql/13/bin \
  --old-options '-c config_file=/etc/postgresql/9.6/main/postgresql.conf' \
  --new-options '-c config_file=/etc/postgresql/13/main/postgresql.conf' \
  --check
  1. Migrate the data (without the --check argument).
/usr/lib/postgresql/13/bin/pg_upgrade \
  --old-datadir=/var/lib/postgresql/9.6/main \
  --new-datadir=/var/lib/postgresql/13/main \
  --old-bindir=/usr/lib/postgresql/9.6/bin \
  --new-bindir=/usr/lib/postgresql/13/bin \
  --old-options '-c config_file=/etc/postgresql/9.6/main/postgresql.conf' \
  --new-options '-c config_file=/etc/postgresql/13/main/postgresql.conf'
  1. Go back to the regular user. exit
  2. Swap the ports for the old and new PostgreSQL versions.
sudo vim /etc/postgresql/13/main/postgresql.conf
# ...and change "port = 5433" to "port = 5432"

sudo vim /etc/postgresql/9.6/main/postgresql.conf
# ...and change "port = 5432" to "port = 5433"
  1. Start the PostgreSQL service. sudo systemctl start postgresql.service
  2. Log in as the postgres user again and Check the new PostgreSQL version.
sudo su postgres
psql -c "SELECT version();"
  1. log in to the chef-server box and see the status chef-server-ctl status

Draft Customer Instructions, Standalone Scenario

  1. INSERT VACUUM FULL STEPS HERE
    need more specific instructions here, eg cd <to where> (if applicable), the exact command to run, etc.

[insert instructions from Shahid HERE]

  1. If you are running a chef-server release prior to 12.17.15, you cannot upgrade directly to 14.8.X. You must perform a stepped upgrade first.

    If you are running chef-server 12.3.0, you must first upgrade 12.17.15.
    If you are running chef-server 11, you must first upgrade to 12.3.0, and then to 12.17.15.

    After performing the stepped upgrade, return here and continue these instructions at step 3. Please consult the docs linked below for additional information on these upgrades.

    https://docs.chef.io/server/upgrades/

  2. Consult the documentation on backup, and backup the database. MAKE A NOTE OF WHERE THE BACKUP WAS PLACED (normally under /var/opt/chef-backup). https://docs.chef.io/server/server_backup_restore/ Please note that chef-server will go offline to perform the backup. Example:

sudo chef-server-ctl backup

  1. Stop services.
# NOTE: umbrella scenarios stop rabbitmq, so I'm including that below.  Correct?  Incorrect?
`sudo chef-server-ctl stop || sudo chef-server-ctl kill rabbitmq
  1. *** WE MAY NOT USE THIS - THIS STEP MAY BE DELETED *** Move the postgresql directory to a safe location or it could be deleted and cause errors. Example:
sudo mkdir /my-postgresql-dir/opt-opscode-embedded-postgresql
sudo mv --verbose /opt/opscode/embedded/postgresql/* /my-postgresql-dir/opt-opscode-embedded-postgresql
  1. Download the chef-server version 14.8.X upgrade package.

https://downloads.chef.io/tools/infra-server

  1. Install the chef-server 14.8.X package.

dpkg: sudo dpkg -iEG /path/to/chef-server-core-<version>.deb

rpm: sudo rpm -U /path/to/chef-server-core-<version>.rpm

  1. *** WE MAY NOT USE THIS - THIS STEP MAY BE DELETED *** Restore the previously-moved postgresql directory. Example:

sudo cp --force --recursive --verbose /my-postgresql-dir/opt-opscode-embedded-postgresql/* /opt/opscode/embedded/postgresql

  1. Upgrade chef-server.

sudo CHEF_LICENSE='accept' chef-server-ctl upgrade

  1. If the upgrade was successful, start services and cleanup. If the upgrade failed, skip this step and continue at step 11 below.
sudo chef-server-ctl start
sudo chef-server-ctl cleanup
[Insert reindex database instructions HERE. Shahid? Vinay? Prajakta?]

You are now finished with the chef-server upgrade.
  1. If the upgrade failed, and you are left with a corrupted chef-server and/or corrupted database,

A. DO NOT UNDER ANY CIRCUMSTANCES RISK YOUR BACKUP OF THE DATABASE. It is recommended that you take any and all steps in order to preserve the backup, including copying it to another disk.

B. Reinstall the version of chef-server you were using before attempting the upgrade. Again, DO NOT RISK YOUR BACKUP OF THE DATABASE. Use a separate disk from your backup for the chef-server installation.

Download chef-server here: https://docs.chef.io/server/install_server/

Install instructions for chef-server: https://docs.chef.io/server/install_server/

B. Consult the documentation on restore, and restore the database from where it was saved previously. https://docs.chef.io/server/server_backup_restore/

Example: chef-server-ctl restore /var/opt/chef-backup/chef-backup-2021-06-11-18-59-19.tgz

  1. Consult the documentation on backup, and backup the database.
    MAKE A NOTE OF WHERE THE BACKUP WAS PLACED (normally under /var/opt/chef-backup).

    https://docs.chef.io/server/server_backup_restore/

    Please note that chef-server will go offline to perform the backup.
    Example:

sudo chef-server-ctl backup

  1. Stop services.
# NOTE: umbrella scenarios stop rabbitmq, so I'm including that below.  Correct?  Incorrect?
`sudo chef-server-ctl stop || sudo chef-server-ctl kill rabbitmq
  1. *** WE MAY NOT USE THIS - THIS STEP MAY BE DELETED ***

Move the postgresql directory to a safe location or it could be deleted and cause errors.
Example:

sudo mkdir /my-postgresql-dir/opt-opscode-embedded-postgresql
sudo mv --verbose /opt/opscode/embedded/postgresql/* /my-postgresql-dir/opt-opscode-embedded-postgresql
  1. Download the chef-server version 14.8.X upgrade package.

https://downloads.chef.io/tools/infra-server

  1. Install the chef-server 14.8.X package.

dpkg:
sudo dpkg -iEG /path/to/chef-server-core-<version>.deb

rpm:
sudo rpm -U /path/to/chef-server-core-<version>.rpm

  1. WE MAY NOT USE THIS - THIS STEP MAY BE DELETED

Restore the previously-moved postgresql directory.
Example:

sudo cp --force --recursive --verbose /my-postgresql-dir/opt-opscode-embedded-postgresql/* /opt/opscode/embedded/postgresql

  1. Upgrade chef-server.

sudo CHEF_LICENSE='accept' chef-server-ctl upgrade

  1. If the upgrade was successful, start services and cleanup.
    If the upgrade failed, skip this step and continue at step 11 below.
sudo chef-server-ctl start
sudo chef-server-ctl cleanup
[Insert reindex database instructions HERE. Shahid? Vinay? Prajakta?]

You are now finished with the chef-server upgrade.
  1. If the upgrade failed, and you are left with a corrupted chef-server and/or corrupted database,

A. DO NOT UNDER ANY CIRCUMSTANCES RISK YOUR BACKUP OF THE DATABASE. It is recommended that you take any and all steps in order to preserve the backup, including copying it to another disk.

B. Reinstall the version of chef-server you were using before attempting the upgrade. Again, DO NOT RISK YOUR BACKUP OF THE DATABASE. Use a separate disk from your backup for the chef-server installation.

Download chef-server here: https://docs.chef.io/server/install_server/

Install instructions for chef-server: https://docs.chef.io/server/install_server/

B. Consult the documentation on restore, and restore the database from where it was saved previously. https://docs.chef.io/server/server_backup_restore/

Example: chef-server-ctl restore /var/opt/chef-backup/chef-backup-2021-06-11-18-59-19.tgz

Clone this wiki locally