Unsolved
1 Message
0
997
Avamar rollback error
Hi
I have a problem with Avamar. After a power outage, the gsan cannot be started. When I try to rollback, I get the following messages
admin@avamar:~/>: tail -f /tmp/dpnctl-gsan-restart-output-8148
(0.0) scp -q -o GSSAPIAuthentication=no /tmp/.mapall8983 admin@10.10.15.243:.
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'tar xzf .mapall8983; rm -f .mapall8983'
Unpacking checks for node group "#.s"...
mapall --givestatus --noerror --nodes=#.s 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'
Running checks for node group "#.s"...
mapall --givestatus --noerror --nodes=#.s 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'
[avamar: check.dpn (Mrun-all) v1.73]
[ checklib::clear_results_file(): "/usr/local/avamar/var/Mrun-all.9074.out" doesn't exist. ]
cd /usr/local/avamar/var/check.dpn.8885-61bc5721
check.dpn.d/prestart/../init.d/checkstoragestatus --run --error --ping_only --noerror --prestart --error --run --verbose --results=Mrun-all.9074.out --single
checkstoragestatus: INFO: [avamar] storage controller "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "2" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "1" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "5" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "1" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "2" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "3" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "4" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage controller "0" shows patrol read state "Stopped"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "0" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "1" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "2" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] drive "5" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "0" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "1" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "2" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "3" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "4" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus passed
Mrun-all: Fri Dec 17 10:24:00 2021, 1 check run
=== PASS === AVAMAR PASSED
rm -f /usr/local/avamar/var/Mrun-all.9074.out
Removing files previously copied to nodes...
mapall --givestatus --noerror --nodes=#.s 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'
End of checks for node group "#.s".
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Using /usr/local/avamar/var/probe.xml
/bin/tar czhf /tmp/.mapall9154 ./avhostname
(0.0) scp -q -o GSSAPIAuthentication=no /tmp/.mapall9154 root@10.10.15.243:.
(0.0) ssh -q -x -o GSSAPIAuthentication=no root@10.10.15.243 'tar xzf .mapall9154; rm -f .mapall9154'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no root@10.10.15.243 '[ -d /usr/local/avamar/bin ] && mv ./avhostname /usr/local/avamar/bin/'
Using /usr/local/avamar/var/probe.xml
Starting checks for node group "#.#"...
[ writing master run file "/usr/local/avamar/var/check.dpn.8885/check.dpn.d/var/Mrun-all" ]
Copying files to nodes...
/bin/tar Cczf /usr/local/avamar/var/check.dpn.8885 /usr/local/avamar/var/check.dpn.8885/check.dpn-8885.tgz check.dpn check.dpn.d dpn.pm node_util.pm node_db.pm
cd /usr/local/avamar/var/check.dpn.8885
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
Using /usr/local/avamar/var/probe.xml
/bin/tar czhf /tmp/.mapall9221 check.dpn-8885.tgz
(0.0) scp -q -o GSSAPIAuthentication=no /tmp/.mapall9221 admin@10.10.15.243:.
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'tar xzf .mapall9221; rm -f .mapall9221'
Unpacking checks for node group "#.#"...
mapall --givestatus --noerror --nodes=#.# 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'
Running checks for node group "#.#"...
mapall --givestatus --noerror --nodes=#.# 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'
[avamar: check.dpn (Mrun-all) v1.73]
[ checklib::clear_results_file(): "/usr/local/avamar/var/Mrun-all.9305.out" doesn't exist. ]
cd /usr/local/avamar/var/check.dpn.8885-61bc5721
check.dpn.d/prestart/../init.d/checkstoragestatus --run --error --ping_only --noerror --prestart --error --run --verbose --results=Mrun-all.9305.out --single
checkstoragestatus: INFO: [avamar] storage controller "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "2" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "1" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "5" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "1" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "2" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "3" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "4" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage controller "0" shows patrol read state "Stopped"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "0" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "1" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "2" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] drive "5" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "0" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "1" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "2" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "3" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "4" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus passed
Mrun-all: Fri Dec 17 10:24:16 2021, 1 check run
=== PASS === AVAMAR PASSED
rm -f /usr/local/avamar/var/Mrun-all.9305.out
Removing files previously copied to nodes...
mapall --givestatus --noerror --nodes=#.# 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'
End of checks for node group "#.#".
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Using /usr/local/avamar/var/probe.xml
/bin/tar czhf /tmp/.mapall9384 ./avhostname
(0.0) scp -q -o GSSAPIAuthentication=no /tmp/.mapall9384 root@10.10.15.243:.
(0.0) ssh -q -x -o GSSAPIAuthentication=no root@10.10.15.243 'tar xzf .mapall9384; rm -f .mapall9384'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no root@10.10.15.243 '[ -d /usr/local/avamar/bin ] && mv ./avhostname /usr/local/avamar/bin/'
Using /usr/local/avamar/var/probe.xml
Starting checks for node group "0.m"...
[ writing master run file "/usr/local/avamar/var/check.dpn.8885/check.dpn.d/var/Mrun-all" ]
Copying files to nodes...
/bin/tar Cczf /usr/local/avamar/var/check.dpn.8885 /usr/local/avamar/var/check.dpn.8885/check.dpn-8885.tgz check.dpn check.dpn.d dpn.pm node_util.pm node_db.pm
cd /usr/local/avamar/var/check.dpn.8885
Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)
Using /usr/local/avamar/var/probe.xml
/bin/tar czhf /tmp/.mapall9451 check.dpn-8885.tgz
(0.0) scp -q -o GSSAPIAuthentication=no /tmp/.mapall9451 admin@10.10.15.243:.
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'tar xzf .mapall9451; rm -f .mapall9451'
Unpacking checks for node group "0.m"...
mapall --givestatus --noerror --nodes=0.m 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'
Running checks for node group "0.m"...
mapall --givestatus --noerror --nodes=0.m 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'
[avamar: check.dpn (Mrun-all) v1.73]
[ checklib::clear_results_file(): "/usr/local/avamar/var/Mrun-all.9536.out" doesn't exist. ]
cd /usr/local/avamar/var/check.dpn.8885-61bc5721
check.dpn.d/prestart/../init.d/checkstoragestatus --run --error --ping_only --noerror --prestart --error --run --verbose --results=Mrun-all.9536.out --single
checkstoragestatus: INFO: [avamar] storage controller "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "2" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "1" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage vdisk "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "5" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "0" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "1" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "2" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "3" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage pdisk "4" shows status "Ok"
checkstoragestatus: INFO: [avamar] storage controller "0" shows patrol read state "Stopped"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "0" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "1" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "2" on storage controller "0" is set correctly as "Disabled"
checkstoragestatus: INFO: [avamar] drive "5" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "0" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "1" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "2" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "3" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus: INFO: [avamar] drive "4" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list
checkstoragestatus passed
Mrun-all: Fri Dec 17 10:24:32 2021, 1 check run
=== PASS === AVAMAR PASSED
rm -f /usr/local/avamar/var/Mrun-all.9536.out
Removing files previously copied to nodes...
mapall --givestatus --noerror --nodes=0.m 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'
Using /usr/local/avamar/var/probe.xml
(0.0) ssh -q -x -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'
End of checks for node group "0.m".
rm -f /usr/local/avamar/var/check.dpn.8885/check.dpn.d/var/Mrun-all
rm -f /usr/local/avamar/var/check.dpn.8885/check.dpn-8885.tgz
rm -rf /usr/local/avamar/var/check.dpn.8885
=== PASS === CHECK.DPN PASSED OVERALL (prestart)
[ checklib::clear_results_file(): "/usr/local/avamar/var/check.dpn.out" doesn't exist. ]
keeping last three most recent getlogs tarballs
removing /usr/local/avamar/var/log/rollback/logs.211217.080425.tar
running getlogs for 2 day(s)
[ ! -d /usr/local/avamar/var/log/rollback ] && mkdir -p /usr/local/avamar/var/log/rollback; cd /usr/local/avamar/var/log/rollback; /usr/local/avamar/bin/getlogs --server=2
Identity added: /root/.ssh/rootid (/root/.ssh/rootid)
Copying getnodelogs script to nodes
Running getnodelogs script on nodes
ERROR: command "ssh -q -x -o GSSAPIAuthentication=no root@10.10.15.243 './getnodelogs --server=2 > getnodelogs.out'" failed, return: 256, exitcode: 1, signal: 0, dumped core: 0
Getting nodelogs.tgz from nodes
Creating tar file: logs.211217.102434.tar
Removing nodelogs.tgz from nodes
Removing nodelogs.tgz from the work directory on the utility node
rm: nie można usunąć `0.?/getnodelogs.out': Nie ma takiego pliku ani katalogu
getlogs: ERROR: command "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; rm 0.?/getnodelogs.out" exited with error status = 1 - quitting
getlogs: ERROR: command "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; ssh-agent bash -c "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; ssh-add /root/.ssh/rootid && /usr/local/avamar/bin/getlogs --_rerun_ssh_agent --_rerun_sudo --server=2"" exited with error status = 1 - quitting
getlogs: ERROR: command "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; sudo -H -n /usr/local/avamar/bin/getlogs --_rerun_sudo --server=2" exited with error status = 1 - quitting
ERROR: command "[ ! -d /usr/local/avamar/var/log/rollback ] && mkdir -p /usr/local/avamar/var/log/rollback; cd /usr/local/avamar/var/log/rollback; /usr/local/avamar/bin/getlogs --server=2" failed, return: 256, exitcode: 1, signal: 0, dumped core: 0