文書番号: 000222081
NetWorker: Email Notifications Stop Working on Red Hat 8 Server After Upgrade to 19.10
概要: Protection policy email notifications stop working after upgrading a Red Hat 8 NetWorker server to NetWorker 19.10.0.1.
文書の内容
現象
- NetWorker server is installed on Red Hat 8:
root@nwserver:~# cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 (Ootpa)
- NetWorker 19.10 was installed on the Red Hat 8.x system. This can be a fresh install or an upgrade from a previous NetWorker release.
- NetWorker Protection Policy resources were configured using the Linux mail or mailx command:
- Prior to upgrading NetWorker, email notifications were received without issue.
- Sending email messages from the NetWorker server command line are also received:
root@nwserver:~# mailx -s "test email" backupadmin@domain.com < /dev/null Null message body; hope that's ok root@nwserver:~# tail -n 7 /var/log/maillog Feb 14 16:13:49 nwserver sendmail[24024]: 41ELDnaE024024: from=root, size=229, class=0, nrcpts=1, msgid=<202402142113.41ELDnaE024024@nwserver.amer.lan>, relay=root@localhost Feb 14 16:13:49 nwserver sendmail[24024]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256 Feb 14 16:13:49 nwserver sendmail[24025]: STARTTLS=server, relay=localhost [127.0.0.1], version=TLSv1.3, verify=NOT, cipher=TLS_AES_256_GCM_SHA384, bits=256/256 Feb 14 16:13:49 nwserver sendmail[24025]: 41ELDn4l024025: from=<root@nwserver.amer.lan>, size=490, class=0, nrcpts=1, msgid=<202402142113.41ELDnaE024024@nwserver.amer.lan>, proto=ESMTPS, daemon=MTA, relay=localhost [127.0.0.1] Feb 14 16:13:49 nwserver sendmail[24024]: 41ELDnaE024024: to=backupadmin@domain.com, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30229, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (41ELDn4l024025 Message accepted for delivery) Feb 14 16:13:50 nwserver sendmail[24027]: STARTTLS=client, relay=mailhub.domain.com., version=TLSv1.2, verify=OK, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256/256 Feb 14 16:13:50 nwserver sendmail[24027]: 41ELDn4l024025: to=<backupadmin@domain.com>, ctladdr=<root@nwserver.amer.lan> (0/0), delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=120490, relay=mailhub.domain.com. [10.10.10.10], dsn=2.0.0, stat=Sent (ok: Message 225328373 accepted)
- The workflow logs under /nsr/logs/policy/POLICY_NAME/ report the following error:
root@nwserver:~# nsr_render_log /nsr/logs/policy/Server\ Protection/workflow_Server\ backup_064001.raw
133550 02/14/2024 11:01:35 AM 1 0 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow NSR notice Starting Protection Policy 'Server Protection' workflow 'Server backup'.
199800 02/14/2024 11:01:35 AM 1 5 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow NSR notice Consider starting action 'Server db backup', enabled 1, schedule action '1'
204318 02/14/2024 11:01:35 AM 1 1 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow SYSTEM notice Request to run the enabled action 'Server db backup' that has level configured as '1'.
201496 02/14/2024 11:01:35 AM 1 1 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow SYSTEM notice Starting the scheduled action 'Server db backup'.
123316 02/14/2024 11:01:35 AM 1 0 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow NSR notice Starting action 'Server Protection/Server backup/Server db backup' with command: 'nsrdbsave -l 1'.
123321 02/14/2024 11:01:35 AM 1 0 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow NSR notice Action 'Server Protection/Server backup/Server db backup's log will be in '/nsr/logs/policy/Server Protection/Server backup/Server db backup_064002.raw'.
123325 02/14/2024 11:02:25 AM 1 0 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow NSR notice Action 'Server Protection/Server backup/Server db backup' succeeded.
199800 02/14/2024 11:02:25 AM 1 5 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow NSR notice Consider starting action 'Expiration', enabled 1, schedule action 'exec'
204318 02/14/2024 11:02:25 AM 1 1 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow SYSTEM notice Request to run the enabled action 'Expiration' that has level configured as 'exec'.
201496 02/14/2024 11:02:25 AM 1 1 0 3269973824 446260 0 nwserver.amer.lan nsrworkflow SYSTEM notice Starting the scheduled action 'Expiration'.
5 02/14/2024 11:02:25 AM 1 1 0 0 unknown unknown LOG unrendered /bin/mailx: symbol lookup error: /bin/mailx: undefined symbol: SSLv3_client_method, version OPENSSL_1_1_0
NetWorker: How to use nsr_render_log.
原因
One of the improvements that went into NetWorker 19.10 was regarding SSL. NetWorker provides its own SSL library which is exported in the networkerrc file:
root@nwserver:~# cat /opt/nsr/admin/networkerrc | grep "NSR_LIBS\|LD_LIBRARY_PATH"
NSR_LIBS=/usr/lib/nsr
NSR_LIBS=/usr/lib/nsr/lib64
# It must be the first entry of LD_LIBRARY_PATH otherwise it fails to load libraries.
NSR_LIBS=/usr/lib/nsr/lib64/cst:/usr/lib/nsr/lib64
LD_LIBRARY_PATH=${NSR_LIBS}:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
The version of mail on Red Hat 8.x includes an SSL library which includes SSv3:
root@nwserver:~# ldd /bin/mail | grep ssl
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f116357b000)
root@lnx-client02:~# nm -D /lib64/libssl.so.1.1 | egrep -i "SSLv3_"
0000000000022780 T SSLv3_client_method
0000000000022760 T SSLv3_method
0000000000022770 T SSLv3_server_method
root@nwserver:~#
NetWorker 19.10 removed SSLv3 from its library as it is a potential security vulnerability and is exploitable:
root@nwserver:~# nm -D /usr/lib/nsr/lib64/libssl.so.1.1 | egrep -i "SSLv" root@nwserver:~#
The mail package is using an old libssl library. The issue is with the mail package; thus outside of NetWorker.
解決方法
Note: If email messages sent from the command line are not received, this must be resolved before proceeding with the next steps. This requires investigation from the system and mail or exchange administrator. See article NetWorker: How to troubleshoot failing email notifications.
Option 1:
The exchange or mail admin can review the mail servers configuration and check if options are available to disable SSLv3 in favor of TLS 1.2.
Option 2:
A fix for this must be released in the Linux mail/mailx package. Mail must be updated to include a libssl which does not use SSLv3. At the time of writing this KB, the only mail package listed for Red Hat 8.x is mailx-12.5-29.el8.x86_64 which was released in 2019. This issue impacts this mail package.
Confirm the mail package that is installed:
rpm -qa | grep mail
Example:
root@nwserver:~# rpm -qa | grep mail
sendmail-8.15.2-34.el8.x86_64
mailcap-2.1.48-3.el8.noarch
mailx-12.5-29.el8.x86_64
procmail-3.22-47.el8.x86_64
Get the mail package details by running:
yum provides mail
Example:
root@nwserver:~# yum provides mail Updating Subscription Management repositories. Last metadata expiration check: 1:55:22 ago on Wed 21 Feb 2024 09:35:43 AM EST. mailx-12.5-29.el8.x86_64 : Enhanced implementation of the mailx command Repo : @System Matched from: Filename : /bin/mail mailx-12.5-29.el8.x86_64 : Enhanced implementation of the mailx command Repo : rhel-8-for-x86_64-baseos-rpms Matched from: Filename : /bin/mail
This package should be updated automatically during operating system updating if a newer release becomes available. If the command
yum provides shows a newer release, you can also try:
yum upgrade mailx
Workaround:
1. Update the
Example:
/opt/nsr/admin/networkerrc to contain OS_LIBS=/lib64
Example:
linux86w)
# cst path is required to locate dynamic libraries of cst (also loads internal bsafe crypto libraries).
# It must be the first entry of LD_LIBRARY_PATH otherwise it fails to load libraries.
# cst comes with own bsafe library. cst bsafe library version may or may not be the same
# version of bsafe library used by other components in the product.
NSR_LIBS=/usr/lib/nsr/lib64/cst:/usr/lib/nsr/lib64
OS_LIBS=/lib64
ERLCOOKIE_HOME=/nsr/rabbitmq
:::::::
esac
LD_LIBRARY_PATH=${OS_LIBS}:${NSR_LIBS}:$LD_LIBRARY_PATH
2. Restart NetWorker:
systemctl restart networker
その他の情報
On Red Hat 9.x, a different package provides mail/mailx which uses a newer
libssl library. This newer library does not use SSLv3. The issue is not reproducible and no workaround was needed with NetWorker 19.10 on Red Hat 9.x.
root@lnx-nwserv:~# ldd /usr/bin/mail
linux-vdso.so.1 (0x00007fffb211a000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007fad91b73000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fad91600000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fad91b1c000)
libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fad91afb000)
libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fad91acb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fad91200000)
libz.so.1 => /lib64/libz.so.1 (0x00007fad91aaf000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fad91525000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fad91a96000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fad91a8f000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fad91a7e000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fad91a77000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fad91a61000)
libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fad9107b000)
/lib64/ld-linux-x86-64.so.2 (0x00007fad91d2e000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fad91a34000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fad91489000)
root@lnx-nwserv:~# nm -D /lib64/libssl.so.3 | egrep -i "SSLv3_"
root@lnx-nwserv:~# NetWorker: Nsrworkflow NSR severe Unable to stat file /usr/bin/mail文書のプロパティ
影響を受ける製品
NetWorker
製品
NetWorker Family, NetWorker Series
最後に公開された日付
23 7月 2024
バージョン
9
文書の種類
Solution