Unsolved
This post is more than 5 years old
22 Posts
0
3781
Scaleio RF cache
Hello, we have test scaleio cluster of latest Version: 2.0.12000
Cluster have nodes with RFCache enabled.
And dmesg -T show such errors:
[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10013) failed with error 1.
[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdg failed with error 1.
[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10012) failed with error 1.
[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdl failed with error 1.
[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10012) failed with error 1.
[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdl failed with error 1.
[Wed Feb 22 15:41:14 2017] Read from the source /dev/sdc failed with error 1.
[Wed Feb 22 15:41:14 2017] SFC-cache Error:Read from the source (Hnd 10008) failed with error 1.
If I disable RFcache dmesg output is like this:
[Wed Feb 22 15:42:14 2017] sdq:
[Wed Feb 22 15:42:14 2017] sdq:
[Wed Feb 22 15:42:14 2017] sdq:
[Wed Feb 22 15:42:15 2017] sdq:
[Wed Feb 22 15:42:15 2017] sdq:
[Wed Feb 22 15:42:15 2017] sdq:
[Wed Feb 22 15:42:16 2017] sdq:
[Wed Feb 22 15:42:16 2017] sdq:
[Wed Feb 22 15:42:16 2017] sdq:
[Wed Feb 22 15:42:16 2017] sdq:
[Wed Feb 22 15:42:16 2017] sdq:
[Wed Feb 22 15:42:16 2017] sdq:
Is it normal or some kind of bug ?
Thanks,
Matas
pawelw1
306 Posts
0
February 23rd, 2017 02:00
Hi Matas,
I haven't seen this error. Could you please open an SR so we can look deeper into it?
Thanks,
Pawel
carterbury
12 Posts
0
May 22nd, 2017 08:00
What was found with this error? I am having the same error messages. ScaleIO 2.0.12000.122 Ubuntu Linux.
Thanks,
Chad
Matas1
22 Posts
0
May 22nd, 2017 12:00
Hello, Chad, I wasn't able to solve this problem yet. If you will be lucky, please let us know.
Thanks,
Matas
pawelw1
306 Posts
0
May 24th, 2017 02:00
Hi Chad,
No, I could not find such an error in any of the tickets we worked so far, that's why I suggested to open an SR.
Thank you,
Pawel
carterbury
12 Posts
0
May 24th, 2017 08:00
We are evaluating Scaleio so I don't have support yet. I would have opened a ticket with this one already otherwise. We have a lot of other gear with you guys but haven't convinced anyone yet to replace one of our aging San's with this yet.
I configured one SSD to cache on each of the SDSs. Running Ubuntu 16.04 on all nodes with the driver_sync correctly syncing from the EMC repository for Xcache. The rest of the drives in the server are sata HDD. I tail the syslog or kern.log and within 10 seconds of enabling Flash cache at the SP level I get flooded with these errors for each of the HDD's. There is no xcache log directory so I assumed all of it is logged at the SDS Log location. Looking at trc.0 there is nothing significant after enabling Flash cache. It seems to add all the devices and I get no error in SDS logs.
Kern and syslog errors per device show...
May 23 09:30:28 hostname kernel: [69151.019224] SFC-cache Error:Read from the source (Hnd 10155) failed with error 1.
May 23 09:30:29 hostname kernel: [69151.021394] Read from the source /dev/sdc failed with error 1.
I had a bunch of extra RAM in these servers so I have switched to using 15GB of RAM cache which gave me a significant boost to read performance in my benchmark tests. Still curious about using RFcache and its performance compared to RAM cache as Ram I would assume to be faster but the SSD can hold a lot more cache-able data.
pawelw1
306 Posts
0
May 31st, 2017 01:00
Hi Matas,
From what I remember, you do have a support contract, right? If yes, please open an SR and we'll look into it deeper.
Thx,
Pawel
RHasleton1
73 Posts
0
June 8th, 2017 13:00
Carterbury,
There is an issue with RFCache we are chasing down and hope to have fixed soon. It only affects Linux kernels greater than 4.30, which I assume with Ubuntu you are running. Stay tuned in the next couple of weeks for an update to the 2.0.1.3 code which will include this fix.
Thanks!
Rick
Matas1
22 Posts
0
June 8th, 2017 13:00
Hi, our kernel 3.10 also have this issue. We run CentOS flavor.
RHasleton1
73 Posts
0
June 9th, 2017 07:00
Hi Matas,
With the older kernel, it will definitely be a different issue than the one being looked at currently. As there were some significant changes made to the Linux kernel at 3.13 and at 4.30. If you can, open a support case and we'll take a look at the logs to see what is going on.
Thanks,
Rick
carterbury
12 Posts
0
June 12th, 2017 16:00
I am using this repository so it appears to be related for me. It looks like I would need to update the OS though in order to apply the fix?
/Ubuntu/2.0.12000.122/4.4.0-62-generic/
carterbury
12 Posts
0
August 3rd, 2017 13:00
RickH,
I updated to the latest Ubuntu and Scaleio but seeing the same behavior. I assume the fix wasn't included in 2.0.1.3?
root@kal-l-sio-fil03:~# dpkg --list |egrep emc
ii emc-scaleio-gateway 2.0-13000.211 amd64 rpm
ii emc-scaleio-lia 2.0-13000.211.Ubuntu.16.04 amd64 scaleio lia package
ii emc-scaleio-mdm 2.0-13000.211.Ubuntu.16.04 amd64 scaleio meta-data manager package
ii emc-scaleio-sds 2.0-13000.211.Ubuntu.16.04 amd64 scaleio sds package
ii emc-scaleio-xcache 2.0-13000.211.Ubuntu.16.04 amd64 scaleio xcache package
RHasleton1
73 Posts
0
August 4th, 2017 07:00
Correct. The fix will be in the 2.0.1.4 code, which is supposed to be due out in early to mid-September. What you have above is the shipping 2.0.1.3 code. Hope that helps!
carterbury
12 Posts
0
August 7th, 2017 07:00
Thanks Rick,
I will wait until then. You had originally thought it would make 2.0.1.3 is why I was hopeful after installing.
SysEng777
25 Posts
0
November 30th, 2017 11:00
Downloading 2.0.1.4 now and will let you know how it goes once its installed and I try RF Cache again.