Data Domain: How to troubleshoot Boostfs Installation, configuration and mount issues

Summary: This article helps the user identify and resolve some typical BOOSTFS installation and configuration issues.

This article applies to This article does not apply to This article is not tied to any specific product. Not all product versions are identified in this article.
Check out other resources

Instructions

BoostFS is a software component that may be installed in some operating systems. It allows the user to run DD Boost backups to a Data Domain without using any particular backup application, while also leveraging some of the features that come with using a Boost-enabled backup application such as NetBackup, NetWorker, or Avamar.
 
For more "to the point" installation and utilization details, check Data Domain: Expedited configuration steps and details for Data Domain Boost File System.
 
This KB, however, contains some of the typical issues that may be found when configuring and using BoostFS. It serves as a quick reference for those typical errors, error messages, and how to solve them. Typical such errors would be "Unable to install Boostfs onto Linux server," "Boostfs mount fails with error," or "Backup or restore fails with error," to name a few.
 

Common BoostFS issues

  1. Required dependency missing when installing the BoostFS software
If there is a failure to install the provided BoostFS software RPM in the client Linux operating system, first determine which are the required package dependencies for the BoostFS software:
# rpm -qpR DDBoostFS-1.0.0.1-539441.rhel.x86_64.rpm
fuse >= 2.8
fuse-libs >= 2.8
/bin/sh
/bin/sh
/bin/sh
/bin/sh
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
 
In the example above, the package is asking for the FUSE ("Filesystem in USEr space") binaries and libraries to be installed (BoostFS is based on Linux's FUSE), as well as any package providing a shell (sh) and the default RPM libraries. Hence, this BoostFS package should install normally in the Linux system the "rpm" has been run on.
 
  1. Cannot mount /mnt/mountpoint/: unexpected error
The above error may occur when trying to mount the remote Data Domain storage unit locally in the Linux client through BoostFS. Review the BoostFS log file on the Linux client. For example, in the first example below, the lockbox file does not contain an entry for the host data.domain.com: 
# less /opt/emc/boostfs/log/ddboostfs_0_0.log
Jul 29 03:45:25 22795 3267069888 [E] bfs_lockbox_get_user_info: Failed to find key entry data.domain.com:LSU2 in config file /opt/emc/boostfs/lockbox/boostfs.lockbox. The requested Lockbox entry could not be found.
Jul 29 03:45:25 22795 3267069888 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections)
Jul 29 03:45:25 22795 3267069888 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo
Jul 29 03:45:25 22795 3267069888 [E] bfs_initialize_mntopts: initialization failed
 
The solution in this case is to add a lockbox entry for the destination Data Domain host and storage unit to be mounted locally. Syntax would be similar to the example below:
# /opt/emc/boostfs/bin/boostfs lockbox set -u sysadmin -d data.domain.com -s LSU2
Enter storage unit user password:
Enter storage unit user password again to confirm:
Lockbox entry set
 
Another possible error that can be seen in the local BoostFS client logs is the Data Domain hostname not resolving from the client. For example:
# less /opt/emc/boostfs/log/ddboostfs_0_0.log
Jul 29 04:05:50 22882 3322156992 [E] bfs_conn_open: connect failed (0 connections): 5037
Jul 29 04:05:50 22882 3322156992 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections)
Jul 29 04:05:50 22882 3322156992 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo
Jul 29 04:05:50 22882 3322156992 [E] bfs_initialize_mntopts: initialization failed
 
The solution in this case is to either add a static hostname to IP mapping to the Linux client's /etc/host file, or to configure the mapping in the DNS server being used.
 
  1. The mount point /mnt/mountpoint is nonempty.
BoostFS cannot be mounted on a nonempty mount point. Please try mounting on an empty mount point.
 
The above message means that the mount point specified (/mnt/mountpoint) cannot be used to show the remote Data Domain storage unit, as the mount point cannot have another file system mounted on it previously, or have any contents. /mnt/mountpoint/ must be an empty and unused directory in the Linux client. If the path already has a mount, specify another mount point. For example: 
# mount /dev/mapper/vg00-lv_root on / type ext4 (rw) /dev/sda1 on /boot type ext4 (rw) boostfs on /mnt/mountpoint type fuse.boostfs (rw,nosuid,nodev)
 
In this case, the remote Data Domain storage unit is already mounted under /mnt/mountpoint/ and hence cannot be mounted again at the same directory.
 
  1. BoostFS mount stops responding (may take even more than 10 minutes to return)
This could occur when a firewall in between the Linux client and the target Data Domain device is dropping traffic to TCP ports 111 and 2049. Verify that the Linux client can reach the Data Domain on TCP ports 111 and 2049 and then try again. For example:
# /opt/emc/boostfs//bin/boostfs mount -d data.domain.com -s LSU2 /mnt/mountpoint/
 
This is an example of what would appear in the Linux client BoostFS logs in such a case:
# less /opt/emc/boostfs/log/ddboostfs_0_0.log
Jul 27 06:34:53 32762 3724339136 [E] bfs_conn_open: connect failed (0 connections): 5037
Jul 27 06:34:53 32762 3724339136 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections)
Jul 27 06:34:53 32762 3724339136 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo
Jul 27 06:34:53 32762 3724339136 [E] bfs_initialize_mntopts: initialization failed
 
  1. "Operation not permitted" when trying to access or list contents under the mount point
# ls -l /mnt/mountpoint/
ls: reading directory .: Operation not permitted
total 0
 
The problem could be that there is no TCP connectivity to the remote Data Domain ports 111 and 2049, or that the lockbox authentication has expired. Verify that the Linux client can still reach the Data Domain on the specified ports and try setting up the lockbox for the remote Data Domain, storage unit, and DD Boost user again.
 
  1. "Insufficient access to or storage-unit does not exist" when using Kerberos authentication
# boostfs mount -s LSU3 -d data.domain.com /mnt/mountpoint/
Insufficient access to or storage-unit LSU3 does not exist
 
When using Kerberos authentication for the lockbox, the most likely reason for the error is that the Kerberos authentication ticket has expired. Check the mentioned ticket from Linux as below:
# /opt/emc/boostfs/bin/boostfs kerberos query -s LSU3 -u username
Client Principal: username@DOMAIN.COM
Valid Starting: Tue Aug 23 11:02:49 2016
Expires: Tue Aug 23 21:01:07 2016
Renew Until: Tue Aug 30 11:02:49 2016
Service Principal: krbtgt/DOMAIN.COM@DOMAIN.COM
 
On the Data Domain, verify that the DD Boost user assigned to the storage-unit matches the user on the Linux client when setting up the lockbox and mounting BoostFS. The AD username and storage-unit username must be identical:
# ddboost storage-unit show
Name   Pre-Comp (GiB)   Status   User       Report Physical   Tenant-Unit
                                               Size (MiB)
----   --------------   ------   --------   ---------------   -----------
LSU3              0.0   RW       username                 -   -
----   --------------   ------   --------   ---------------   -----------
 
Also confirm that the dates and times on the Data Domain, the Linux client, and the Kerberos server (AD) are not more than 5 minutes apart. Issues with tickets expired due to inconsistent time may occur if they are too far apart.
 
Below is an example of what would appear in the Data Domain logs if time was more than 5 minutes different:
# log watch debug/ddfs.info
08/23 18:52:09.654 (tid 0x7f738141fb00): nfs3 accepted 3000004b6 552 from 10.10.10.10:55042
08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_print_error:117 - event gss_error, gss_accept_sec_context failed: maj=0xd0000, min=0x96c73a25
08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_print_error:126 - event gss_error, Unspecified GSS failure. Minor code may provide more information
08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_print_error:141 - event gss_error, Clock skew too great
08/23 18:52:09.660 (tid 0x7f738140c890): [dd_rpc2_nfs x3000004b6] dd_rpc_gss_accept:217 - event gss_accept_failed, maj=851968, min2529638949
08/23 18:52:09.660 (tid 0x7f73814233f0): nfs3 destroyed tcp 3000004b6
 
And these are for the BoostFS Linux host logs:
# less /opt/emc/boostfs/log/ddboostfs_0_0.log
Aug 23 18:04:39 1044 2019465280 [E] bfs_conn_open: connect failed (0 connections): 5075
Aug 23 18:04:39 1044 2019465280 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections)
Aug 23 18:04:39 1044 2019465280 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo
Aug 23 18:04:39 1044 2019465280 [E] bfs_initialize_mntopts: initialization failed
 
  1. BoostFS mount with Kerberos authentication fails with the error "Not able to access lockbox or lockbox entry cannot be found"
# /opt/emc/boostfs/bin/boostfs mount -s LSU3 -d data.domain.com /mnt/mountpoint/
Not able to access lockbox or lockbox entry cannot be found
 
If you intend to use BoostFS configuration options in "/opt/emc/boostfs/etc/boostfs.conf," verify the "[global]" setting in the file, and confirm that it is not commented out. The keyword "[global]" is commented by default, and must be uncommented in order for any of the global options to work.
 
Example contents of the BoostFS "/opt/emc/boostfs/etc/boostfs.conf" configuration file: 
# Comments are not allowed after the option value pair.
#
#############################################################################
[global]
# Data Domain Hostname or IP address
# data-domain-system=data.domain.com
 
Example output from the Linux client BoostFS logs:
# less /opt/emc/boostfs/log/ddboostfs_0_0.log
Aug 24 08:23:35 27227 3565582272 [E] bfs_lockbox_get_user_info: Failed to find key entry datadomain.techsupp.local:LSU3 in config file /opt/emc/boostfs/lockbox/boostfs.lockbox. The requested Lockbox entry could not be found.
Aug 24 08:23:35 27227 3565582272 [E] bfs_conn_lookup: connection lookup failed for node 0 file /.boostfs_sysinfo. (0 connections)
Aug 24 08:23:35 27227 3565582272 [I] bfs_node_lookup: failed to obtain a connection for file /.boostfs_sysinfo
Aug 24 08:23:35 27227 3565582272 [E] bfs_initialize_mntopts: initialization failed
 
The error may also be the result of using a different hostname, storage unit name, or DD Boost username in the Linux client "mount" command, compared to the one used for setting up the lockbox, or the ones that correspond to the target Data Domain. Ensure all names match across the Data Domain, the BoostFS lockbox, and the "mount" command.
 
  1. Attempts to set "boostfs" user credentials fail with the following error:
bfs_krb5_err_handler: Kerberos error: -1765328360 [ERROR_MESSAGE=Failed getting initial credentials.]
 
Cannot set Kerberos credentials.
  1. Verify that Active Directory user credentials are correct by logging into the DC with Active Directory user credentials.
  2. Verify that the times on the Linux client and Kerberos server are no more than 5 minutes apart. Kerberos authentication requires that the clock times are no more than 5 minutes apart.
 
  1. Mounting the BoostFS in the Linux client fails with the error "fusermount: mount failed: Operation not permitted."
This can be due to any of several reasons. "fusermount" is the underlying Linux command that ultimately uses FUSE to mount the DD Boost storage unit locally in the Linux client namespace. Doing so requires special privileges. That is why "fusermount" is a SUID binary in the Linux client: 
# ls -l /usr/bin/fusermount
-rwsr-xr-x. 1 root root 38680 May 11 2019 /usr/bin/fusermount
 
Being a SUID binary means that regardless of the user running the attempt to mount the storage unit, the "fusermount" runs with "root" privileges. If "fusermount" is not SUID to root, only the Linux client root user can mount the remote BOOST storage unit. This may not be a problem if setting up the mount under /etc/fstab, but would be when a non-root user must perform the mount.
 
  1. BoostFS mounts OK, but attempts to access the storage unit contents at the mount point fail with permission errors
When mounting the storage unit manually using either the Data Domain "boostfs" command or /etc/fstab (or even "mount.fuse" from the CLI), it is the default's underlying FUSE implementation to only allow access to files to the Linux client userid that mounted the BoostFS. So if the mount was made as root, only root may access the files. If the mount was made as user "postgres" (for backing up a PostgreSQL database), only this user would have the permission to access the files.
 
If other users must be able to access the files in the mounted storage unit from the Linux cloud, the /etc/fuse.conf may require tuning. See more details here: Ubuntu Manpage: fuseThis hyperlink is taking you to a website outside of Dell Technologies.
 
Set the following option in /etc/fuse.conf: 
user_allow_other
       Allow non-root users to specify the allow_other or allow_root mount options (see below).
And then use the following one when mounting the BOOST storage unit, from the command line:
allow_other
       This option overrides the security measure restricting file access to the user mounting the filesystem.
       So all users (including root) can access the files. This option is by default only allowed to root, but
       this restriction can be removed with the configuration option described above (user_allow_other).
 
  1. BoostFS does not support one-way or two-way authentication using certificates. Use two-way-password instead.
Aug  1 15:46:39.436 4632 1188 [I] [ddp log] [1218:4A4] number of sslquery 1
Aug  1 15:46:39.436 4632 1188 [E] [ddp log] [1218:4A4] number of ssl_query_failed2 = 1
 
  1. If errors with SSL occur when mounting the storage unit, verify that the network is not reducing the MTU between the client and the Data Domain.
  2. Ensure that the Data Domain is not configured to require one-way or two-way authentication using certificates. 
    # ddboost option show
Option                           Value
------------------------------   -------
distributed-segment-processing   enabled
virtual-synthetics               enabled
global-authentication-mode       two-way
global-encryption-strength       medium
------------------------------   -------

# ddboost clients show config
Client   Encryption Strength   Authentication Mode
------   -------------------   -------------------
*        medium**                two-way
------   -------------------   -------------------
(**) The global security settings take precedence over these client(s) specific settings.
  3. Ask the customer if they can change the configuration to two-way-password or none. 
    ddboost option set global-authentication-mode none global-encryption-strength {none | medium | high}

or

ddboost option set global-authentication-mode two-way-password global-encryption-strength {medium | high}
  4. If the clients are also required to be one-way or two-way, this must also be changed to allow BoostFS to mount to the Data Domain. Request permission on which of the following changes would be their choice. If the current settings on the Data Domain are both "none," then no changes need to be made. Proceed to test a BoostFS mount. 
    # ddboost client show config
Client   Encryption Strength   Authentication Mode
------   -------------------   -------------------
*        none**                none
------   -------------------   -------------------
    1. Option 1: Modify their existing configuration to use two-way-password and existing encryption strength. 
      ddboost clients modify * authentication-mode two-way-password encryption-strength { medium | high}
    2. Option 2: Add a new client for this current BoostFS mounting client. 
      ddboost clients modify <client hostname> authentication-mode two-way-password encryption-strength medium

Additional Information

BoostFS Configuration Related Questions:

  1. What is the maximum number of mount point sections that can be defined in the BoostFS configuration file?
There is no limit to the number of mounts in the mount point section of the boostfs.conf file. 
# Mount point sections are separated by [mountpoint] tags
#

# [/path/to/mount]
# [/mnt/bofs]
# Data Domain Hostname or IP address
# data-domain-system=data.domain.com

# Storage Unit
# storage-unit=su-name

# Storage Unit Username
# storage-unit-username=sysadmin

# Subdirectory within the storage-unit to mount to
# directory-name=path/to/subdir
 
  1. Does NetWorker support BoostFS?
NetWorker supports BoostFS by default, and can handle on-the-fly mounts for performing certain tasks and backups.
 
Even though NetWorker is a DD Boost-enabled backup application, it also supports BoostFS on Linux client systems where it is installed. One example NetWorker backup taken from the Linux client command line, once the BoostFS lockbox has been set up, would be the following (PostgreSQL backup):
# nsroappbackup -z /nsr/apps/config/backup_postgresql.cfg
174908:(pid 16487):Saving the backup data in the pool 'DB'.
175019:(pid 16487):Received the media management binding information on the host 'dd.example.com'.
174910:(pid 16487):Connected to the nsrmmd process on the host 'dd.example.com'. + /usr/pgsql-11/bin/pg_dump --file=/nsr/apps/tmp/e3106c82_294324_16487/dump.sql --format=plain
Continued processing with the returned value 0. + /bin/cp /data/postgresql.conf /nsr/apps/tmp/e3106c82_294324_16487/
Continued processing with the returned value 0.
The files in the save set 'PostgreSQL_postgres_backuppostgre_full' at time '01/04/20 15:40:36' are:
               Size:   Name:
			    2645   dump.sql
			   24000   postgresql.conf
			    2 File(s)       26645 bytes
The backup command '/nsr/apps/config/scripts/backup-postgre-dbon1-full.sh' completed successfully.
The backup completed successfully.
 
Once the backup job is configured, it can also be kicked off from the NetWorker UI and progress monitored there. On the Data Domain, /ddr/var/log/debug/messages.engineering would show log entries like the one below when the mount issued by NetWorker is done: 
Apr 1 15:20:49 dd.example.com ddfs[17040]: NOTICE: ddboost-<client.example.com-49808>: Boostfs:
Apr 1 15:22:52.395 16275 704915520 [I] DDBoost Plugin Version is: [7.0.0.0.633508]
Apr 1 15:20:49 dd.example.com ddfs[17040]: NOTICE: ddboost-<client.example.com-49808>: Boostfs:
Apr 1 15:22:52.395 16275 704915520 [I] BoostFS Version info: [BOOSTFS:7.0.0.0-633922 FUSE:2.9.7]
Apr 1 15:20:49 dd.example.com ddfs[17040]: NOTICE: ddboost-<client.example.com-49808>: Boostfs:
Apr 1 15:22:52.395 16275 704915520 [I] bfs_lib_init: Mounting dd.example.com:LSU_NAME on /mnt/mountpoint

Affected Products

Data Domain

Products

Data Domain
Article Properties
Article Number: 000064347
Article Type: How To
Last Modified: 11 May 2026
Version:  4
Find answers to your questions from other Dell users
Support Services
Check if your device is covered by Support Services.