NetWorker depends on name resolution. If name resolution is not correct and entirely consistent, problems may arise in many of NetWorker's operations. Since NetWorker manages potentially sensitive data, it must ensure the identities of the hosts with whom it interacts by various means.
Any number of symptoms in NetWorker can be the result of name resolution imperfections in NetWorker:
- Error messages indicating forward or reverse name lookup problems.
- Inability to probe clients during backup.
- Inability of clients to manually save to the server or recover.
- Problems cloning or accessing Storage Node devices.
- Browsing or media database record issues.
- Server or Storage Node stops responding at startup or during the course of regular operation.
- Misnamed or nested index directories.
- Misconfigured client errors.
Name resolution happens by various mechanisms and at various levels:
- NetWorker name cache
- Local host resolver cache
- Local hosts file entries
- DNS server lookups
Name Resolution Methodology
1. NetWorker Caching: NetWorker maintains an internal name cache and may sometimes not reflect environmental changes made to correct name resolution issues without restarting services. NetWorker also has a configurable field which is global for all client instances called 'Aliases', which must reflect all names resolvable for that client instance to avoid problems of multiple types. If a required name is present in NetWorker's internal cache, it is used from there and the host Resolver cache is not consulted. The NetWorker server daemon (nsrd) has the ability to dump its resolved name cache into the daemon.raw, flush the cache, or flush and re-resolve names:
- dbgcommand -n nsrd PrintDnsCache=1 (Dump to daemon.raw)
- dbgcommand -n nsrd FlushDnsCache=1 (Flush)
- dbgcommand -n nsrd FlushDnsCache=9 (Flush and resolve)
- For the above commands either "-n process name" or "-p PID" can be used. To use the PID you will need to run other commands first to get the PID; for example "ps -ef | grep nsr" (Linux) or "tasklist | findstr nsr" (windows)
2. Resolver Cache: All hosts and Operating Systems have a local resolver cache to aid host resolution and speed without relying on either hosts files or DNS servers. This cache is checked next per normal operating system name resolution rules and if the host record exists (even if old or incorrect), it is used before querying any other data source. Resolver cache entries are entered into the cache at the first 'successful' resolution attempt and remain for a predetermined amount of time. Some operating systems can display their resolver cache contents (for example, Windows hosts can run ipconfig /displaydns), and all have a mechanism to forcibly clear or flush it:
- Flushing DNS on Linux can vary depending on Linux distribution and packages installed, refer to vendor documentation.
- Windows: ipconfig /flushdns
3. Hosts files: The legacy mechanism for host resolution is to explicitly enter the IP address, followed by any names that are wanted to resolve to that address, on separate lines, delimited by white space. This is checked before DNS by default on Windows, In Linux resolution source order can usually be configured in /etc/nsswitch.conf or /etc/netsvc.conf. Hosts files are only parsed for the first matching entry that is being queried - thus any second instance of an IP address or hostname, short or long, will never be found when attempting to resolve names. Each name or IP should only appear once as all names should be entered on the same line of the corresponding IP address.
- Unix/Linux: /etc/hosts
- Windows: %systemroot%\System32\drivers\etc\hosts
NOTE: Hosts files can become corrupt, like any other file - when in doubt, rename the existing hosts file, create the new one, clear the resolver cache and retry.
4. Forward Resolution: To communicate with a foreign host when the name is provided, a host must find that foreign hosts' IP address for TCP/IP communications to ensue; when using DNS, the query must be performed for that hostname in the Forward Lookup Zone to retrieve the IP address. Multiple DNS hosts may be configured for a client to use; on Windows systems, use ipconfig /all to get DNS names; on Linux/UNIX operating systems, /etc/resolv.conf generally carries the DNS server order. nslookup is the most common tool for querying DNS and exists on all platforms, but is frequently misused; to query the forward zone:
- Run nslookup with no arguments to enter the interactive prompt.
- Enter the name iteration to lookup and press enter to retrieve forward record from the DNS server you have connected to.
- Enter the same name twice more to see if the name record is round-robining silently between different hosts, or returns the same data.
- Repeat the same process for any instance of any name that the host may be called by other hosts or regard itself as for the same IP address.
- Repeat the same process for any other DNS server that the host is configured to potentially use by entering server next_dns_server.
NOTE: All records that are returned must be consistent among themselves, and consistent with what is understood to be correct by the admin; and all known names must be accounted for and checked.
5. Reverse Resolution: To communicate with a foreign host when the IP address is provided, the host may need to find the host's name; when using DNS, the query is performed against the Reverse Lookup Zone to find the hostname of the IP address in question. Again, this is frequently misused, as entering nslookup IP_Address or even entering the IP address in nslookup will not query the Reverse Lookup Zone:
- Run nslookup with no arguments to enter the interactive prompt.
- Enter set q=ptr to change the query type to the Reverse Zone.
- Enter the IP address to reverse resolve, and press enter.
- Ensure that the name that is returned in the reverse record matches the forward record name/IP.
[root@linux_a~]# nslookup linux_a
Server: 1.2.3.4
Address: 1.2.3.4#53
Name: linux_a.domain.com
Address: 5.6.7.8
[root@linux_a~]# nslookup 5.6.7.8
Server: 1.2.3.4
Address: 1.2.3.4#53
Name: linux_a.domain.com
Address: 5.6.7.8
[root@linux_a~]# nslookup
> set q=ptr
> 5.6.7.8
Server: 1.2.3.4
Address: 1.2.3.4#53
Non-authoritative answer:
8.7.6.5.in-addr.arpa name = linux_a.domain.com.
In the above example, it is clear that running nslookup noninteractively never queries the reverse lookup zone.
NOTE: NetWorker is heavily dependent on perfect and consistent reverse resolution as well as standard forward resolution - this is part of its authorization process and is architected this way deliberately to prevent IP spoofing or other types of attacks that are intended to steal backup data.
Resolution
All NetWorker hosts must be guaranteed to have consistent name resolution, both forward and reverse, for any host with whom they communicate (which varies depending on Data zone role). It is critical for NetWorker administrators to ensure that any host resolution problems are addressed immediately and completely.
When troubleshooting name resolution problems, or to rule them out in your NetWorker Data zone:
1. Find all hosts involved in the failing operation - Server, Client(s), and possibly Storage Node(s), etc.
2. For each determine the IP addresses configured locally and all expected resolvable names for those IPs.
3. Configure all hosts to use the hosts file before DNS for host resolution.
4. At the beginning of one hosts' hosts file, configure a single entry for each IP, with every name corresponding it on the same line.
5. Copy those lines exactly from the first host to the hosts files of the other involved hosts.
6. Edit the NetWorker client objects to have Aliases correctly corresponding to the wanted IPs.
7. Shut down NetWorker on all involved hosts
8. Clear the resolver cache on each host using the appropriate operating system mechanism
9. Restart NetWorker and attempt the problematic operation again
To prove the name being resolved by a given host, use this test:
1. From the first NetWorker host (for example, the Client), connect to the second (for example, the Server) using
nsradmin -s remote_host -p nsrexec - leave the session open
2. On the same host, determine the process for nsradmin (for example, Windows,
tasklist | findstr nsradmin)
3. Run netstat to show the socket associated with that process (for example, Windows,
netstat -ao | findstr process_id)
4. Determine the connecting socket from that host (the leftmost IP:port pairing in the output)
5. On the remote host - run netstat -a and findstr/grep for :calling_port_from_first_host (it will now appear on the right side)
6. The hostname which appears on the left side of its colon is the name of the first host that the second host resolves the inbound connection as
7. Run again with the -n switch added to the netstat command to verify the IP of the same socket, to check if the IP/route is expected
8. Reverse the same test to ensure that the second host is resolving the first host within expected parameters