Start a Conversation

Unsolved

B

1 Message

997

December 17th, 2021 03:00

Avamar rollback error

Hi

I have a problem with Avamar. After a power outage, the gsan cannot be started. When I try to rollback, I get the following messages

admin@avamar:~/>: tail -f /tmp/dpnctl-gsan-restart-output-8148

(0.0) scp -q  -o GSSAPIAuthentication=no /tmp/.mapall8983 admin@10.10.15.243:.

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'tar xzf .mapall8983; rm -f .mapall8983'

Unpacking checks for node group "#.s"...

mapall  --givestatus --noerror  --nodes=#.s 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'

Running checks for node group "#.s"...

mapall  --givestatus --noerror  --nodes=#.s 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'

[avamar: check.dpn (Mrun-all) v1.73]

[ checklib::clear_results_file(): "/usr/local/avamar/var/Mrun-all.9074.out" doesn't exist. ]

cd /usr/local/avamar/var/check.dpn.8885-61bc5721

check.dpn.d/prestart/../init.d/checkstoragestatus --run --error --ping_only --noerror --prestart --error --run --verbose --results=Mrun-all.9074.out --single

checkstoragestatus: INFO: [avamar] storage controller "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "2" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "1" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "5" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "1" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "2" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "3" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "4" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage controller "0" shows patrol read state "Stopped"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "0" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "1" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "2" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] drive "5" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "0" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "1" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "2" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "3" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "4" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus               passed

Mrun-all: Fri Dec 17 10:24:00 2021, 1 check run

=== PASS === AVAMAR PASSED

rm -f /usr/local/avamar/var/Mrun-all.9074.out

Removing files previously copied to nodes...

mapall  --givestatus --noerror  --nodes=#.s 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'

End of checks for node group "#.s".

Identity added: /root/.ssh/rootid (/root/.ssh/rootid)

Using /usr/local/avamar/var/probe.xml

/bin/tar czhf /tmp/.mapall9154 ./avhostname

(0.0) scp -q  -o GSSAPIAuthentication=no /tmp/.mapall9154 root@10.10.15.243:.

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no root@10.10.15.243 'tar xzf .mapall9154; rm -f .mapall9154'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no root@10.10.15.243 '[ -d /usr/local/avamar/bin ] && mv ./avhostname /usr/local/avamar/bin/'

Using /usr/local/avamar/var/probe.xml

Starting checks for node group "#.#"...

[ writing master run file "/usr/local/avamar/var/check.dpn.8885/check.dpn.d/var/Mrun-all" ]

Copying files to nodes...

/bin/tar Cczf /usr/local/avamar/var/check.dpn.8885 /usr/local/avamar/var/check.dpn.8885/check.dpn-8885.tgz check.dpn check.dpn.d dpn.pm node_util.pm node_db.pm

cd /usr/local/avamar/var/check.dpn.8885

Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)

Using /usr/local/avamar/var/probe.xml

/bin/tar czhf /tmp/.mapall9221 check.dpn-8885.tgz

(0.0) scp -q  -o GSSAPIAuthentication=no /tmp/.mapall9221 admin@10.10.15.243:.

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'tar xzf .mapall9221; rm -f .mapall9221'

Unpacking checks for node group "#.#"...

mapall  --givestatus --noerror  --nodes=#.# 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'

Running checks for node group "#.#"...

mapall  --givestatus --noerror  --nodes=#.# 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'

[avamar: check.dpn (Mrun-all) v1.73]

[ checklib::clear_results_file(): "/usr/local/avamar/var/Mrun-all.9305.out" doesn't exist. ]

cd /usr/local/avamar/var/check.dpn.8885-61bc5721

check.dpn.d/prestart/../init.d/checkstoragestatus --run --error --ping_only --noerror --prestart --error --run --verbose --results=Mrun-all.9305.out --single

checkstoragestatus: INFO: [avamar] storage controller "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "2" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "1" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "5" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "1" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "2" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "3" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "4" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage controller "0" shows patrol read state "Stopped"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "0" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "1" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "2" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] drive "5" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "0" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "1" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "2" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "3" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "4" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus               passed

Mrun-all: Fri Dec 17 10:24:16 2021, 1 check run

=== PASS === AVAMAR PASSED

rm -f /usr/local/avamar/var/Mrun-all.9305.out

Removing files previously copied to nodes...

mapall  --givestatus --noerror  --nodes=#.# 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'

End of checks for node group "#.#".

Identity added: /root/.ssh/rootid (/root/.ssh/rootid)

Using /usr/local/avamar/var/probe.xml

/bin/tar czhf /tmp/.mapall9384 ./avhostname

(0.0) scp -q  -o GSSAPIAuthentication=no /tmp/.mapall9384 root@10.10.15.243:.

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no root@10.10.15.243 'tar xzf .mapall9384; rm -f .mapall9384'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no root@10.10.15.243 '[ -d /usr/local/avamar/bin ] && mv ./avhostname /usr/local/avamar/bin/'

Using /usr/local/avamar/var/probe.xml

Starting checks for node group "0.m"...

[ writing master run file "/usr/local/avamar/var/check.dpn.8885/check.dpn.d/var/Mrun-all" ]

Copying files to nodes...

/bin/tar Cczf /usr/local/avamar/var/check.dpn.8885 /usr/local/avamar/var/check.dpn.8885/check.dpn-8885.tgz check.dpn check.dpn.d dpn.pm node_util.pm node_db.pm

cd /usr/local/avamar/var/check.dpn.8885

Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)

Using /usr/local/avamar/var/probe.xml

/bin/tar czhf /tmp/.mapall9451 check.dpn-8885.tgz

(0.0) scp -q  -o GSSAPIAuthentication=no /tmp/.mapall9451 admin@10.10.15.243:.

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'tar xzf .mapall9451; rm -f .mapall9451'

Unpacking checks for node group "0.m"...

mapall  --givestatus --noerror  --nodes=0.m 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'washere=`/bin/pwd` ; export TERM=${TERM:-dumb} ; mkdir -p /usr/local/avamar/var/check.dpn.8885-61bc5721 && tar Cxzpf /usr/local/avamar/var/check.dpn.8885-61bc5721 $washere/check.dpn-8885.tgz 2>/dev/null'

Running checks for node group "0.m"...

mapall  --givestatus --noerror  --nodes=0.m 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; /usr/local/avamar/var/check.dpn.8885-61bc5721/check.dpn.d/var/Mrun-all --run --verbose --single'

[avamar: check.dpn (Mrun-all) v1.73]

[ checklib::clear_results_file(): "/usr/local/avamar/var/Mrun-all.9536.out" doesn't exist. ]

cd /usr/local/avamar/var/check.dpn.8885-61bc5721

check.dpn.d/prestart/../init.d/checkstoragestatus --run --error --ping_only --noerror --prestart --error --run --verbose --results=Mrun-all.9536.out --single

checkstoragestatus: INFO: [avamar] storage controller "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "2" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "1" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage vdisk "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "5" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "0" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "1" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "2" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "3" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage pdisk "4" shows status "Ok"

checkstoragestatus: INFO: [avamar] storage controller "0" shows patrol read state "Stopped"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "0" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "1" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] disk cache policy setting for vdisk "2" on storage controller "0" is set correctly as "Disabled"

checkstoragestatus: INFO: [avamar] drive "5" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "0" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "1" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "2" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "3" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus: INFO: [avamar] drive "4" with product id "ST1000NM0033-9ZM173" and revision "GT06" on storage controller "0" is not on the watch list

checkstoragestatus               passed

Mrun-all: Fri Dec 17 10:24:32 2021, 1 check run

=== PASS === AVAMAR PASSED

rm -f /usr/local/avamar/var/Mrun-all.9536.out

Removing files previously copied to nodes...

mapall  --givestatus --noerror  --nodes=0.m 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'

Using /usr/local/avamar/var/probe.xml

(0.0) ssh -q  -x  -o GSSAPIAuthentication=no admin@10.10.15.243 'export TERM=${TERM:-dumb} ; rm -rf check.dpn-8885.tgz /usr/local/avamar/var/check.dpn.8885-61bc5721'

End of checks for node group "0.m".

rm -f /usr/local/avamar/var/check.dpn.8885/check.dpn.d/var/Mrun-all

rm -f /usr/local/avamar/var/check.dpn.8885/check.dpn-8885.tgz

rm -rf /usr/local/avamar/var/check.dpn.8885

=== PASS === CHECK.DPN PASSED OVERALL (prestart)

[ checklib::clear_results_file(): "/usr/local/avamar/var/check.dpn.out" doesn't exist. ]

keeping last three most recent getlogs tarballs

removing /usr/local/avamar/var/log/rollback/logs.211217.080425.tar

running getlogs for 2 day(s)

[ ! -d /usr/local/avamar/var/log/rollback ] && mkdir -p /usr/local/avamar/var/log/rollback; cd /usr/local/avamar/var/log/rollback; /usr/local/avamar/bin/getlogs --server=2

Identity added: /root/.ssh/rootid (/root/.ssh/rootid)

 

Copying getnodelogs script to nodes

 

Running getnodelogs script on nodes

ERROR: command "ssh -q  -x  -o GSSAPIAuthentication=no root@10.10.15.243 './getnodelogs --server=2 > getnodelogs.out'" failed, return: 256, exitcode: 1, signal: 0, dumped core: 0

 

Getting nodelogs.tgz from nodes

 

Creating tar file: logs.211217.102434.tar

 

Removing nodelogs.tgz from nodes

 

Removing nodelogs.tgz from the work directory on the utility node

rm: nie można usunąć `0.?/getnodelogs.out': Nie ma takiego pliku ani katalogu

getlogs: ERROR: command "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; rm 0.?/getnodelogs.out" exited with error status = 1 - quitting

getlogs: ERROR: command "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; ssh-agent bash -c "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; ssh-add /root/.ssh/rootid && /usr/local/avamar/bin/getlogs --_rerun_ssh_agent --_rerun_sudo --server=2"" exited with error status = 1 - quitting

getlogs: ERROR: command "[ -r /etc/profile ] && . /etc/profile >/dev/null 2>&1 ; sudo -H -n /usr/local/avamar/bin/getlogs --_rerun_sudo --server=2" exited with error status = 1 - quitting

ERROR: command "[ ! -d /usr/local/avamar/var/log/rollback ] && mkdir -p /usr/local/avamar/var/log/rollback; cd /usr/local/avamar/var/log/rollback; /usr/local/avamar/bin/getlogs --server=2" failed, return: 256, exitcode: 1, signal: 0, dumped core: 0

 

No Responses!
No Events found!

Top