Isilon Vs ECS

Thanks Sluetze for the explanation.

This one is slightly out of topic, However still to understand better about object storage ,ECS -one of the use case is data analytics , how do they achieve the performance in Big data or other analytics solution with just having the SATA disk bundle ?

since I have no skills in data anlytics i can't help.

Maybe someone else can jump in?

In general "big data"/distributed/object protocols like ceph/hadoop/s3 utilize SATA by allowing the reads and writes to scale horizontally with near linear gains.  One SATA disk isn't that impressive but once you get to dozens, hundreds, or thousands you could easily surpass the performance of a high-end storage platform.  The presumption is that the applications have sufficient parallelization as well since a single stream will usually perform no better than what SATA can provide.

Object stores also give up many things POSIX filesystems might take for granted like close-to-open consistency opting for eventual consistency and trade it for strong guarantees that if a file is visible then it's intact.

That said I have not been terribly impressed with the ECS units we have.  They are okay, but not really great.  Frankly I was expecting more given the hardware but it seems like the software platform is just too heavy.  64GB of ram and I often find them swapping due to the nature of their java applications.  Throughput is sufficient for out needs but slow metadata performance ( enumeration ), slow space reclamation after deletion, and we still have issues with larger objects behaving badly.

Think of ECS in the same way as you may have thought of Centera in the past, a really big archive.  It can do more as sluetze  said with regards to geo-load balancing and so-forth, and supports the Centera CAS API, and S3.  File protocols were added later to ECS initially NFSv3 only, and I think recently SMB as well.  But on the big data side it doesn't speak HDFS like Isilon does, and it's not meant to be best of breed performance for really large HDFS datasets.  As I understand it the HDFS access is actually over NFS using a java library.  So maybe a good spot for cold buckets of data, not hot.

It's really about a different dataset type.  ECS is an object store.  So think of data that has lots of associated metadata like PACS images from radiology where there are tons of identifying DICOM tags, age, date of birth, gender, etc.

Isilon while it can do that do is more about being a traditional file services NAS.  It does HDFS very well because of it's architecture, and is far faster at data loads and much less overhead than DAS attached to data nodes.  Isilon is also really great at big file, giant dataset things like video rendering farms, video surveillance, genome sequencing in life sciences, giant flat file database backups, and then even with some of the more advanced file services features, like snapshots, quotas with notifications, it organically became a good place for large enterprise stores for traditional NAS data, like home directories, group/departmental shares, etc.

Just my 2 cents,

~Chris Klosterman

Principal SE, Datadobi

chris.klosterman@datadobi.com

Got it .. Thanks for your reply. So,object storage can't be used for an application with demanding performance.

Usually no, especially object storage with built-in geo-protection.  One reason why is that most systems like this have data protection policies that require remote protection of the data in some form (either sync or a-sync) before an ack can be sent back of a successful write.  That adds write latency, which could be substantial depending upon the link speed, file size, and latency of the link.

For a read, sure object storage could be quite fast as long as the system you're talking to has a copy of the object in question, and doesn't have to pull it from somewhere else.