Upgrade from 7 to 8

Introduction

Proxmox Mail Gateway 8.x is based on the new major version of Debian (Bookworm). Carefully plan the upgrade, make and verify backups before beginning, and test extensively. Depending on the existing configuration, several manual steps — including some downtime — may be required.

Note: A valid and tested backup is always required, before starting the upgrade process. Test the backup beforehand in a test lab setup.

In case the system is customized and/or uses additional packages or any other third party repositories/packages, ensure those packages are also upgraded to and compatible with Debian Bookworm.

In general, there are two ways to upgrade a Proxmox Mail Gateway 7.x system to Proxmox Mail Gateway 8.0:

• A new installation (restoring the configuration and database from the backup)
• An in-place upgrade via apt (step-by-step)

In both cases, emptying the browser cache and reloading the GUI is required after the upgrade.

New Installation

• Install Proxmox Mail Gateway in one of the following three ways:
  • As a container on top of Debian Bookworm
  • On top of Debian Bookworm
  • By using the ISO image
• Restore the backup which you made before the upgrade.
• Change the IP address and hostname.
• For clustered setups:
  • On the master, remove all nodes from the cluster
  • Upgrade the master
  • Set the nodes up fresh, then join them to the upgraded master-node (recreate the cluster).

In-Place Upgrade

Preconditions

The following actions need to be carried out from the command line.

Perform the actions via console or SSH. If you use SSH you should use a terminal multiplexer (for example, tmux or screen) to ensure the upgrade can continue even if the SSH connection gets interrupted.

• Perform these actions via SSH, a physical console or a remote management console like iKVM or IPMI.
  • If you use SSH, you should use a terminal multiplexer (for example, tmux or screen) to ensure the upgrade can continue even if the SSH connection gets interrupted.
  • Important: Do not carry out the upgrade via the web UI console directly, as this will get interrupted during the upgrade.

• Upgraded to the latest version of Proxmox Mail Gateway 7., see the roadmap for potential important changes in the stable release.

  Use apt update and apt dist-upgrade (still with Debian Bullseye repos setup) to upgrade to latest 7.3

  • Verify version:

  You can check the web-interface (reload) at the top, or use pmgversion. Both must show a version with 7.3-6 (or newer), for example something like pmg-api/7.3-6/... for the CLI command.

  If you still see an older version, you should ensure that you have valid package repositories configured.

• Make a valid and tested backup of Proxmox Mail Gateway.

  You can either create and download one from the web-interface, store it on your Proxmox Backup Server or create it from the CLI with pmgbackup backup.

• At least 5 GB free disk space on root mount point.
• Check known upgrade issues

In-place upgrades are carried out using APT. Familiarity with APT is required to proceed with this upgrade mechanism.

Actions step-by-step

Please first ensure that your Mail Gateway 7 system is up-to-date and that a valid backup has been created before starting the upgrade process. If you need to adapt the configuration, do this now. In case you have a cluster, wait for all config-changes to be synced to all nodes before continuing.

Continuously use the pmg7to8 checklist script

A small checklist program named pmg7to8 is included in the latest Proxmox Mail Gateway 7.3 packages. The program will provide hints and warnings about potential issues before, during and after the upgrade process. You can call it by executing:

 pmg7to8

This script only checks and reports things. By default, no changes to the system are made and thus, none of the issues will be automatically fixed. You should keep in mind that Proxmox Mail Gateway can be heavily customized, so the script may not recognize all the possible problems with a particular setup!

It is recommended to re-run the script after each attempt to fix an issue. This ensures that the actions taken actually fixed the respective warning.

For clusters

• If you have a cluster, stop and mask all cluster-daemons on all nodes before you start the upgrade of the first node.

  systemctl stop pmgmirror pmgtunnel
  systemctl mask pmgmirror pmgtunnel

• Then proceed by upgrading all nodes sequentially.
• The Mail Gateway service will be provided by the other nodes, which aren't currently being upgraded.
• Certain operations (for example config changes) will only work once all nodes have been upgraded.

Update the configured APT repositories

Change the apt sources to Bookworm - see Package Repositories Update all Debian repository entries to Bookworm.

sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list

Update the enterprise repository to Bookworm:

echo "deb https://enterprise.proxmox.com/debian/pmg bookworm pmg-enterprise" > /etc/apt/sources.list.d/pmg-enterprise.list

Make sure to also update any extra files that you added to /etc/apt/sources.list.d/ to Bookworm accordingly.

Stop and mask services before upgrade

This is necessary to prevent changes to the database before and during the upgrade.

• Stop postfix and all Proxmox Mail Gateway services (emails will be queued by the servers trying to contact the Proxmox Mail Gateway)

systemctl stop postfix pmg-smtp-filter pmgpolicy pmgdaemon pmgproxy pmgmirror pmgtunnel

• Mask postfix and all Proxmox Mailgateway services to prevent them from starting during the upgrade:

systemctl mask postfix pmg-smtp-filter pmgpolicy pmgdaemon pmgproxy pmgmirror pmgtunnel

Upgrade the system

apt update
apt dist-upgrade

During the above step, you will be asked to approve changes to configuration files, where the default config has been updated by their respective package.

It's suggested to check the difference for each file in question and choose the answer accordingly to what's most appropriate for your setup.

Common configuration files with changes, and the recommended choices are:

• /etc/issue -> Proxmox Mail Gateway will auto-generate this file on boot, and it has only cosmetic effects on the login console.

  Using the default "No" (keep your currently-installed version) is safe here.

• /etc/clamav/clamd.conf and /etc/clamav/freshclam.conf -> Those two configuration files are managed by Proxmox Mail Gateway directly, at will be re-generate on any relevant change and on boot.

  Using the default "No" (keep your currently-installed version) is safe here.

• /etc/ssh/sshd_config -> If you have not changed this file manually, the only differences should be a replacement of ChallengeResponseAuthentication no with KbdInteractiveAuthentication no and some irrelevant changes in comments (lines starting with #).

  If this is the case, both options are safe, though we would recommend installing the package maintainer's version in order to move away from the deprecated ChallengeResponseAuthentication option. If there are other changes, we suggest to inspect them closely and decide accordingly.

• /etc/default/grub -> Here you may want to take special care, as this is normally only asked for if you changed it manually, e.g., for adding some kernel command line option.

  It's recommended to check the difference for any relevant change, note that changes in comments (lines starting with #) are not relevant.

  If unsure, we suggested to selected "No" (keep your currently-installed version)

It is not necessary to reboot the Proxmox Mail Gateway host yet after the dist-upgrade finished.

Adapt modified configuration templates to new shipped versions

If configuration templates are used in /etc/pmg/templates, you will see a prompt about the changes in the new version that are not yet incorporated. Review the changes carefully and ensure that only the changes you want are shown in the diff.

Disable ClamAV On-Access Scanner Service

The new ClamAV on-access scanning service is not useful for Proxmox Mail Gateway setups and is disabled for new installations as it not only slows down the entire system, but also affects the spam and virus detection mechanisms managed by Proxmox Mail Gateway.

During upgrades, the service may get enabled and will then be marked as failed. It is recommended to disable this service:

systemctl disable clamav-clamonacc.service

Upgrade the PostgreSQL database

• Before upgrading the PostgreSQL main cluster, you need to remove the automatically created cluster in the new version.

pg_dropcluster --stop 15 main

• Upgrade the PostgreSQL main cluster from 13 to 15, using pg_upgradecluster
  • This step will need some time and enough free disk space as it will create another database containing your rules, statistics, and quarantine information.
  • If possible, use the default setting of dumping the old databases and restoring them, to avoid problems.

pg_upgradecluster -v 15 13 main

• Unmask postfix and all non-cluster Proxmox Mail Gateway services to enable them again.

systemctl unmask postfix pmg-smtp-filter pmgpolicy pmgdaemon pmgproxy

• Reboot and then check the journal to ensure that everything is running correctly again.

reboot

• Reconnect to the node after it successfully rebooted

• You can remove the old PostgreSQL version and its data now, if all is working as expected:

apt purge postgresql-13 postgresql-client-13

After the Proxmox Mail Gateway upgrade

Empty the browser cache and/or force-reload (CTRL + SHIFT + R, or for MacOS ⌘ + Alt + R) the Web UI.

Unmasking & Starting Cluster Services

After upgrading, unmask and start all cluster-daemons on all nodes. This applies to upgrades of a single node, as well as to upgrades of all nodes in a clustered setup:

systemctl unmask pmgmirror pmgtunnel
systemctl start pmgmirror pmgtunnel

Potential Issues

General

As a Debian based Distribution, Proxmox Mail Gateway is affected by most issues and changes affecting Debian. Thus, ensure to read the upgrade specific issues for Bookworm.

Please also check the known issue list from the Proxmox Mail Gateway 8.0 changelog: https://pmg.proxmox.com/wiki/Roadmap#8.0-known-issues

Breaking Changes

• Changed defaults for Bayes and AWL
  • Since the two options cause worse results when enabled in most average installations their defaults changed in 8.0, and they are now disabled

  If your installation explicitly disabled, or enabled the feature nothing will change

  In case your installation never set the setting the old default will now be written to /etc/pmg/pmg.conf to keep your system consistent.

• The ClamAV antivirus daemon clamav-daemon now uses socket-activation

  To disable the service you need to disable clamav-daemon.service and clamav-daemon.socket

• Postgresql config change

  the stats_temp_directory server variable is no longer supported

  the postgresql.conf template shipped with Proxmox Mail Gateway accounts for the change

  if you have modified the template (or copied it without modification) in /etc/pmg/templates/ - make sure to remove the line and/or remove the complete template override (if you don't have any modifications)

Upgrade wants to remove package 'proxmox-mail-gateway'

If you have installed Proxmox Mail Gateway on top of a plain Debian Bookworm (without using the Proxmox Mail Gateway ISO), you may have installed the package 'linux-image-amd64', which conflicts with current 8.x setups. To solve this, you have to remove this package with

apt remove linux-image-amd64

before the dist-upgrade.

Network

Network Interface Name Change

Due to the new kernel recognizing more features of some hardware, like for example virtual functions, and since interface naming often derives from the PCI(e) address, some NICs may change their name, in which case the network configuration needs to be adapted.

This can also happen in virtualized environments (and has been reported with Xen for this upgrade)

In general, it's recommended to either have an independent remote connection to the Proxmox Mail Gateways's host console, for example, through the hypervisor in case of a VM or container setup, IPMI or iKVM, or physical access for managing the server even when its own network doesn't come up after a major upgrade or network change.

Network Fails on Boot Due to NTPsec Hook

Some users reported that after the upgrade their network failed to come up cleanly on boot, but worked if triggered manually (e.g., using ifreload -a), when ntpsec was installed.

We're still investigating for a definitive root cause, but it seems that an udev hook which the /etc/network/if-up.d/ntpsec-ntpdate might hang on some hardware, albeit due to changes not directly related to ntpsec.

The simplest solution might be switching to that via apt install chrony.

Systemd-boot (for ZFS on root and UEFI systems only)

Systems booting via UEFI from a ZFS on root setup should install the systemd-boot package after the upgrade. You will get a Warning from the pve7to8 script after the upgrade if your system is affected - in all other cases you can safely ignore this point.

The systemd-boot was split out from the systemd package for Debian Bookworm based releases. It won't get installed automatically upon upgrade from Proxmox Mail Gateway 7.3 as it can cause trouble on systems not booting from UEFI with ZFS on root setup by the Proxmox Mail Gateway installer.

Systems which have ZFS on root and boot in UEFI mode will need to manually install it if they need to initialize a new ESP (see the output of proxmox-boot-tool status).

Note that the system remains bootable even without the package installed.

It is not recommended installing systemd-boot on systems which don't need it, as it would replace grub as bootloader in its postinst script.

External links

Release Notes for Debian 12.0 (bookworm), 64-bit PC