Upgrade from 8 to 9

Introduction

Proxmox Mail Gateway 9.x is based on the new major version of Debian (Trixie). Carefully plan the upgrade, make and verify backups before beginning, and test extensively. Depending on the existing configuration, several manual steps — including some downtime — may be required.

Note: A valid and tested backup is always required, before starting the upgrade process. Test the backup beforehand in a test lab setup.

In case the system is customized and/or uses additional packages or any other third party repositories/packages, ensure those packages are also upgraded to and compatible with Debian Trixie.

In general, there are two ways to upgrade a Proxmox Mail Gateway 8.x system to Proxmox Mail Gateway 9.0:

• A new installation (restoring the configuration and database from the backup)
• An in-place upgrade via apt (step-by-step)

In both cases, emptying the browser cache and reloading the GUI is required after the upgrade.

New Installation

• Install Proxmox Mail Gateway in one of the following three ways:
  • As a container on top of Debian Trixie
  • On top of Debian Trixie
  • By using the ISO image
• Restore the backup which you made before the upgrade.
• Change the IP address and hostname.
• For clustered setups:
  • On the master, remove all nodes from the cluster
  • Upgrade the master
  • Set the nodes up fresh, then join them to the upgraded master-node (recreate the cluster).

In-Place Upgrade

Preconditions

The following actions need to be carried out from the command line.

• Perform these actions via SSH, a physical console or a remote management console like iKVM or IPMI.
  • If you use SSH, you should use a terminal multiplexer (for example, tmux or screen) to ensure the upgrade can continue even if the SSH connection gets interrupted.
  • Important: Do not carry out the upgrade via the web UI console directly, as this will get interrupted during the upgrade.

• Upgrade to the latest version of Proxmox Mail Gateway 8.2, see the roadmap for potential important changes in the stable release.

  Use apt update and apt dist-upgrade (still with Debian Bookworm repos setup) to upgrade to latest 8.2

  • Verify version:

  You can check the web-interface (reload) at the top, or use pmgversion. Both must show a version with 8.2.5 (or newer), for example something like pmg-api/8.2.5/... for the CLI command.

  If you still see an older version, you should ensure that you have valid package repositories configured.

• Make a valid and tested backup of Proxmox Mail Gateway.

  You can either create and download one from the web-interface, store it on your Proxmox Backup Server or create it from the CLI with pmgbackup backup.

• Ensure that you have at least 10 GB free disk space on the root mount point:

df -h /

• Check known upgrade issues

In-place upgrades are carried out using APT. Familiarity with APT is required to proceed with this upgrade mechanism.

Actions step-by-step

First, ensure that your Mail Gateway 8 system is up-to-date and that a valid backup has been created before starting the upgrade process.

If you need to adapt the configuration, do this now. In case you have a cluster, wait for all config-changes to be synced to all nodes before continuing.

Continuously use the pmg8to9 checklist script

A small checklist program named pmg8to9 is included in the latest Proxmox Mail Gateway 8.2 packages. The program will provide hints and warnings about potential issues before, during and after the upgrade process. You can call it by executing:

 pmg8to9

This script only checks and reports things.

By default, no changes to the system are made and thus, none of the issues will be automatically fixed.

You should keep in mind that Proxmox Mail Gateway can be heavily customized, so the script may not recognize all the possible problems with a particular setup!

It is recommended to re-run the script after each attempt to fix an issue.

This ensures that the actions taken actually fixed the respective warning.

For clusters

• If you have a cluster, stop and mask all cluster-daemons on all nodes before you start the upgrade of the first node.

  systemctl stop pmgmirror pmgtunnel
  systemctl mask pmgmirror pmgtunnel

• Then proceed by upgrading all nodes sequentially.
• The Mail Gateway service will be provided by the other nodes, which aren't currently being upgraded.
• Certain operations (for example config changes) will only work once all nodes have been upgraded.

Update the configured APT repositories

First, make sure that the system is using the latest Proxmox Mail Gateway packages:

apt update
apt dist-upgrade
pmgversion -v

The last command should report a version of at least 8.2.5 or newer.

Ensure Repository Archive Keyring is Installed

To ensure your system trusts the new APT archive keyring for our Debian Trixie-based releases, install the proxmox-archive-keyring package before switching the repositories to Trixie.

apt install proxmox-archive-keyring

Update Debian Base Repositories to Trixie

Update all repository entries to Trixie:

sed -i 's/bookworm/trixie/g' /etc/apt/sources.list

Ensure that there are no remaining Debian Bookworm specific repositories left. Check all files in the /etc/apt/sources.list.d/ folder (like pmg-enterprise.list) and also the top-level /etc/apt/sources.list file. If you are already using sources in the new deb822 format, you will also need to check .sources files in the same location.

Note: Instead of removing older repositories, you can also disable them. In .list files simply comment them out by adding a # to the beginning of the line. In .sources files, you can add the line Enabled: false to any stanza you want to disable.

See the Package Repositories section in the reference docs for the correct Proxmox Mail Gateway / Debian Trixie repositories.

Add the Proxmox Mail Gateway 9 Package Repository

Update the enterprise repository to Trixie in the new deb822 format with the following command:

cat > /etc/apt/sources.list.d/pmg-enterprise.sources << EOF
Types: deb
URIs: https://enterprise.proxmox.com/debian/pmg
Suites: trixie
Components: pmg-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

After you added the new enterprise repository as above, check that apt picks it up correctly. You can do so by first running apt update followed by apt policy. Make sure that no errors are shown and that apt policy only outputs the desired repositories. Then you can remove the old /etc/apt/sources.list.d/pmg-enterprise.list file. Run apt update and apt policy again to be certain that the old repo has been removed.

If using the no-subscription repository, see Package Repositories. You should be able to add the Proxmox Mail Gateway 9 no-subscription repository with this command:

cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/pmg
Suites: trixie
Components: pmg-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

As with the enterprise repository, make sure that apt picks it up correctly with apt update followed by apt policy. Then remove the previous Proxmox Mail Gateway 8 no-subscription repository from either the /etc/apt/sources.list, /etc/apt/sources-list.d/pmg-install-repo.list or any other .list file you may have added it to. Run apt update and apt policy again to be certain that the old repo has been removed.

Make sure to check that all the .list files you added in /etc/apt/sources.list.d/ got switched over to Trixie correctly.

Stop and mask services before upgrade

This is necessary to prevent changes to the database before and during the upgrade.

• Stop postfix and all Proxmox Mail Gateway services (emails will be queued by the servers trying to contact the Proxmox Mail Gateway)

systemctl stop postfix pmg-smtp-filter pmgpolicy pmgdaemon pmgproxy pmgmirror pmgtunnel

• Mask postfix and all Proxmox Mailgateway services to prevent them from starting during the upgrade:

systemctl mask postfix pmg-smtp-filter pmgpolicy pmgdaemon pmgproxy pmgmirror pmgtunnel

Upgrade the system

Note that the time required for finishing this step heavily depends on the system's performance, especially the root filesystem's IOPS and bandwidth. A slow spinner can take up to 60 minutes or more, while for a high-performance server with SSD storage, the upgrade can be finished in less than 5 minutes.

Note: While the packages are being upgraded certain operations and requests to the API might fail (for example, logging in as a system user in the pam realm)

apt update
apt dist-upgrade

While running the apt dist-upgrade command, you may be asked to approve changes to configuration files and some service restarts among other prompts. This includes:

• The output of apt-listchanges: You can simply exit it by pressing q.
• Selecting your default keyboard settings: Simply use the arrow keys to navigate to the one applicable in your case and hit enter.
• Questions about service restarts (like Restart services during package upgrades without asking?): Use the default if unsure, as the reboot after the upgrade will restart all services cleanly anyway.
• Questions about (default) configuration changes: It's suggested to check the difference for each file in question and choose the answer accordingly to what's most appropriate for your setup. Common configuration files with changes, and the recommended choices are:

  /etc/issue

  Proxmox Mail Gateway will auto-generate this file on boot, and it has only cosmetic effects on the login console.

  Using the default "No" (keep your currently-installed version) is safe here.

  /etc/ssh/sshd_config

  If you have not changed this file manually, the only differences should be a replacement of ChallengeResponseAuthentication no with KbdInteractiveAuthentication no and some irrelevant changes in comments (lines starting with #).

  If this is the case, both options are safe, though we would recommend installing the package maintainer's version in order to move away from the deprecated ChallengeResponseAuthentication option. If there are other changes, we suggest to inspect them closely and decide accordingly.

  /etc/clamav/clamd.conf and /etc/clamav/freshclam.conf

  Those two configuration files are managed by Proxmox Mail Gateway directly, at will be re-generate on any relevant change and on boot.

  Using the default "No" (keep your currently-installed version) is safe here.

  /etc/default/grub

  Here you may want to take special care, as this is normally only asked for if you changed it manually; for example, if you added some kernel command line option.

  It's recommended to check the difference for any relevant change, note that changes in comments (lines starting with #) are not relevant.

  If unsure, we suggested to selected "No" (keep your currently-installed version)

  /etc/postfix/master.cf.proto, /etc/postfix/main.cf.proto

  These files are not used by Proxmox Mail Gateway - they are the templates for setting up multi-instance postfix instances, which was never used by Proxmox Mail Gateway.

  See the bugreport at bugs.debian.org for more context.

  We recommend to select "Yes" (install the new version), in order to not get asked again for a future upgrade.

  /etc/crontab and other cron related files on installations on containers on Proxmox VE

  The crontab gets randomized by Proxmox VE to prevent all jobs running at the same time in all containers.

  Using the default "No" (keep your currently-installed version) is preferred here.

  postgresql may print warnings regarding about collation version mismatch

  These are transitory and will disappear once the cluster has been upgraded to the new version.

Important: If configuration templates are used in /etc/pmg/templates, you will see a prompt about the changes in the new version that are not yet incorporated. Review the changes carefully and ensure that only the changes you want are shown in the diff.

It is not yet necessary to reboot your Proxmox Mail Gateway host at this point. Before doing so, first upgrade PostgreSQL database.

Upgrade the PostgreSQL database

• Upgrade the PostgreSQL main cluster from 15 to 17, using pg_upgradecluster
  • Ensure you run this step in a shell, which does not have non-standard locales set. easiest to achieve this is running a fresh su - session and checking that no locale related variables are set to a not installed locale:

su -
env |grep -E 'LC|LANG'

The output should be empty.

• • This step will need some time and enough free disk space as it will create another database containing your rules, statistics, and quarantine information.
  • If possible, use the default setting of dumping the old databases and restoring them, to avoid problems.

pg_upgradecluster -v 17 15 main

• Unmask postfix and all non-cluster Proxmox Mail Gateway services to enable them again.

systemctl unmask postfix pmg-smtp-filter pmgpolicy pmgdaemon pmgproxy

Reboot

Reboot the host with e command below. Then check the journal to ensure that everything is running correctly again.

reboot

Reconnect to the node after it successfully rebooted.

After the Proxmox Mail Gateway upgrade

Empty the browser cache and/or force-reload (CTRL + SHIFT + R, or for MacOS ⌘ + Alt + R) the Web UI.

Unmasking & Starting Cluster Services

After upgrading, unmask and start all cluster-daemons on all nodes. This applies to upgrades of a single node, as well as to upgrades of all nodes in a clustered setup:

systemctl unmask pmgmirror pmgtunnel
systemctl start pmgmirror pmgtunnel

Remove old PostreSQL Version

You can remove the old PostgreSQL version and its data now, if all is working as expected:

apt purge postgresql-15 postgresql-client-15

Optional: Modernize apt Repository Sources

You can migrate existing repository sources to the recommended deb822 style format, by running:

apt modernize-sources

By answering the following prompt with "n" you can check the changes the command would make before applying them. To apply them simply run the command again and respond to the prompt with "Y".

The command will also keep the old .list files around by appending .bak to them. So you will have the new .sources files and the old repository configurations in the .list.bak files. You can remove the leftover backup files once you verified that everything works smoothly with the new format.

Note: ensure that all external and third-party repositories (e.g. the one provided by avast have provided the keys in the correct places).

Potential Issues

General

As a Debian based Distribution, Proxmox Mail Gateway is affected by most issues and changes affecting Debian. Thus, ensure to read the upgrade specific issues for Trixie.

Please also check the known issue list for the Proxmox Mail Gateway 9.X minor releases as this gets updated with future minor releases:

• https://pmg.proxmox.com/wiki/Roadmap#9.0-known-issues

PostgreSQL

Setting Locale Failed During Postgres Cluster Upgrade

If you are performing the upgrade via SSH (as advised), running pg_upgradecluster -v 17 15 main may fail if your environment variables contain locales that do not exist on your PMG host:

# [...]
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
....
Error: The locale requested by the environment is invalid:
...
Error: Could not create target cluster

These environment variables may be set automatically depending on your shell's configuration or SSH settings. Most commonly, ssh can pass local environment variables along to your remote host when connecting. See man ssh_config and man sshd_config for more information.

To fix this follow the steps in Upgrade the PostgresSQL Cluster.

Breaking Changes

• Legacy timezones were split off.

  This should not be an issue as Proxmox Mail Gateway never offered the deprecated timezones for selection.

  However, if you've manually configured one such timezone and postgresql does not start, install the tzdata-legacy package.

• The external avast Virus Scanner with integration in Promxox Mail Gateway has not yet released a version for Debian Trixie.

  If you are using it consider delaying the upgrade until it becomes available

Upgrade wants to remove package 'proxmox-mail-gateway'

If you have installed Proxmox Mail Gateway on top of a plain Debian Trixie (without using the Proxmox Mail Gateway ISO), you may have installed the package linux-image-amd64, which conflicts with current 9.x setups.

To solve this, you have to remove this package with apt remove linux-image-amd64 before the dist-upgrade.

Network

Network Interface Name Change

The new kernel can recognize more hardware features such as virtual function of PCI(e) devices. Since network names are usually derived from PIC(e) addresses and features recognized by the kernel, the network configuration might need to be adapted to match the new interface names.

In such cases, the network connection to a Proxmox Datacenter Manager host might be lost during or after the upgrade process. Hence, it is generally recommended to have either physical access or an independent remote connection to the host (for example, via IPMI or iKVM).

The latest version of Proxmox Mail Gateway 8.2 and 9.0 provide a package called proxmox-network-interface-pinning that you can install. This package offers a CLI tool that helps you pin all network interfaces to NIC-based names and update the network configuration simultaneously.

Systemd-boot meta-package changes the bootloader configuration automatically and should be uninstalled

With Debian Trixie the systemd-boot package got split up a bit further into systemd-boot-efi (containing the EFI-binary used for booting), systemd-boot-tools (containing bootctl) and the systemd-boot meta-package (containing hooks which run upon upgrades of itself and other packages and install systemd-boot as bootloader).

As Proxmox Systems usually use systemd-boot for booting only in some configurations (ZFS on root and UEFI booted without secure boot), which are managed by proxmox-boot-tool, the meta-package systemd-boot should be removed.

The package was automatically shipped for systems installed from the PMG 8.0 to PMG 8.2 ISOs, as it contained bootctl in bookworm. If the pmg8to9 checklist script suggests it, the systemd-boot meta-package is safe to remove unless you manually installed it and are using systemd-boot as a bootloader. Should systemd-boot-efi and systemd-boot-tools be required, pmg8to9 will warn you accordingly.

The pmg8to9 checklist script will change its output depending on the state of the upgrade, and should be run continuously before and after the upgrade. It will print which packages should be removed or added at the appropriate time. The only situation where you should keep the meta-package systemd-boot installed is if you manually setup systemd-boot for your system.

See also the filed bug for systemd-boot.

External links

Release Notes for Debian 13.0 (trixie)