How Does Oracle Behave on Thin Provisioned Devices?

Nick,

Do you have more information about the DBverify issues on thin Dev? I researched on Metelink without any related issues or bug fix.

To your other 2 questions,

1. Oracle does not take any specific optimization on thin Dev, what it does is as same as it does on other device.
2. 3.  To resolve Oracle limitation on Thin Dev, EMC has recommendation of planning, implementation consideration.

You could look @ White Paper: Implementing Virtual Provisioning on EMC Symmetrix VMAX with Oracle Database 10g and 11g — Applied Technology https://community.emc.com/docs/DOC-14378

Thanks,

Eddy

Nick,

NickPOW wrote: 1.     Check how Oracle behaves on EMC Thin provisioned devices 2.     Explain how the Checksum/DB-Verify process works with thin provisioned devices 3.     Explain how EMC has circumvented this process

For Q1: Oracle is not the storage aware application. It doesn't know if it's using thin or thick device. Its behavior on Thin remains same as on thick device. When Oracle apply for new storage space to use, VMAXe will allocate corresponding tracks from pool and return the meta information(address) back to the oracle.

For Q2: oracle has special data structure. If you run a CRC test on any given Oracle block, the value should be equal to 0, even if there is actual data residing. There is a built in checksum mechanism  to ensure data integrity. Same as the first question, I don't see any difference between thin and thick.

I have found some posts discussing about checksum generated by Oracle DB from a mail list. They are worth reading for  DBAs. Especially this one http://bartsjerps.wordpress.com/2011/12/14/oracle-data-integrity/ , it also discussed EMC storage background scrub behavior to ensure data integrity. The other 2 are very good documents as well.

http://docs.oracle.com/cd/B28359_01/server.111/b28319/dbverify.htm

http://www.pythian.com/news/252/oracle-block-corruptions-dbv-vs-rman/

I really found http://bartsjerps.wordpress.com/2011/12/14/oracle-data-integrity/ a good comprehensive article explaining block checksum (a.k.a CRC) related to HARD initiative(Hardware Assisted Resilient Data). EMC Double Checksum, as first Application Delivered for New Oracle HARD Initiative, play a integrity and reliable role in protecting database data.

As article mentioned storing checksum information on every written track on disk by some vendor(including EMC, but EMC go beyond just saving checksums, more details see the link). I was doubt if Checksum/DB-Verify process will works specially with thin provisioned devices. Cerntainly we can @Bart Sjerps for more accurate answers to the question 2. He is also a big contributor to community and at the same time, the author of the article.

Oracle works very well with VMAXe thin provisioned devices. The Symmetrix white paper referenced above goes into more details on this. In short though, for customers who fully allocate their thin devices in the pool it doesn't matter how friendly Oracle is to thin devices, since the space is fully accounted for and just make use of it, while also benefiting from Virtual Provisioning on VMAXe (wide striping, ease of management, etc.). For customers that do care about the thin aspect and don't want to fully allocate the thin devices from start, we should remember that Oracle data files allocate space (Oracle doesn't write zeros, it writes metadata with each block, even if the datafiles are still empty). For that reason deploy a methodology of 'grow-as-you-go'. Start with a reasonable db size and grow it over time (auto-extend, manual extend, or just add data files). Like I said the white paper goes into more details. Bottom line though that Oracle and VMAXe work well together, and the choice of using the 'thin' aspect of virtual provisioning is given to the customer.

ASM is also thin friendly and EMC has a write_same based integration with ASM Reclamation Utility (ASRU) to reclaim empty space in ASM disk groups in the thin pools.

Finally, regarding data integrity, HARD, DBV, etc. DBV comes to verify the Oracle data integrity. If Oracle wrote that data it is allocated in the thin pool so when DBV is run (a read only process), there is no conflict with thin devices or Virtual Provisioning as a whole. If DBV tries to read from a location that Oracle didn't already write to (allocated) the Symmetrix will return zeros (not that DBV has any business verifying unallocated space). No issue here.

Oracle checks their own checksums during reads natively (release dependent) and EMC Double Checksum helped with verifying the writes in 'real-time'. EMC is also looking into the T10 PI model (Protection Information, aka DIF) and you can find more details in the end of the Oracle World 2011 session we lead https://community.emc.com/docs/DOC-13213 (we had two sessions where one was joint with Oracle). DIF protects both reads and writes all the way from Oracle (aka DIX) to the storage (aka DIF) and back.

Hope this helps.

I think I did respond to this or a very similar question on the Oracle Specialists DL, but what Yaron points out is correct, albiet more nicely stated than I would have put it.

My statement was, simply, storage-based thin provisioning in an Oracle context is meaningless nonsense. This is because, as Yaron points out, Oracle does not zero the file. Therefore zero reclaim does not work. Instead, Oracle writes a block header to every block, thus defeating zero reclaim. The net-net is that Oracle datafiles cannot be thinly provisioned.

Of course, the advantages of storage virtualization with pools and such still apply as usual.

In order to get true thin provisioning of Oracle database files, you need a true database virtualzation product. The only one of these that I am currently aware of is Delphix. Delphix gives you the ability to provision a multi-TB database in seconds (as well as clone this database in a similar period). This is because the database does not actually exist. Rather, the database has been virtualized by Delphix's software.

Thanks for Jeff's clarification. Well said! As oracle will write block head to every block, thin provisioning will become the thick provisioning. The good thing for VMAXe is pool thick LUN has comparable performance as traditional LUN, we need  not to worry about performance lose as VNX pooled LUN.

why do you say you lose performance with VNX pooled LUNs ?

Thanks for mentioning my blog post!

I believe question 2 is answered by now. Oracle block checking is not doing any special I/O, it just saves checksum data in each Oracle database block. The CRC checksums in EMC storage are hidden from the application. Both work nicely together.

In the same context you might be interested in my last blogpost where I discuss the false idea that Oracle Data Guard would protect better against corruptions than EMC tooling (i.e. SRDF, Recoverpoint etc).

http://bartsjerps.wordpress.com/2012/04/16/data-guard-emc-block-corruptions/

I also got a question from one of the readers asking how to prevent corruptions from happening outside the storage. (i.e. EMC disk block checks only work after a write was accepted in a front-end port. In between the database writer process and the front-end port a lot of things can still go wrong). We used to offe EMC Double Checksum to deal with this but this method is now outdated (and a feature called EMC Generic Safewrite which is still valid AFAIK).I plan to write a new post on this sometime. Suggestions on such prevention is welcome.

Bart

Below is a slightly ammended copy a post  I sent out to the Oracle-L back in December on the subject of double check sum and T10 PI.
Hope it helps

Allan

The Oracle double checksum method had  "smart" storage systems aware of  different data blocks stored inside the storage system as holding Oracle  database pages.

The idea here is that when a host write arrives at the storage destined for a  region known to hold Oracle data, some logic would be exercised to execute  a  special form of "checksum re-verification".  This is about identifying that an  incoming 8KB block write is an Oracle DB page, and that there is an Oracle checksum that is located specifically at bytes 24 through 32 within that 8KB that should be used. This is over and above the regular SCSI data block CRC transmission check summing, etc.

If a corruption is detected,  on that 8KB "Oracle DB page" that has been  received, the storage system is supposed to immediately flag a SCSI IO error going back, as opposed to corrupting the previously stored data.

Now, however,  Oracle has been working with a number of partners, e.g. EMC and  Emulex,  in driving a new end-to-end data integrity standard into the T10 standard body.

With this new standard, each component that is T10 PI  compliant, formerly  called T10 DIF, observes an expanded standard in the SCSI IO block structure which would ensure that they not only check the received data for correctness,
but also passes along the standard data integrity checking information to the next physical device inside the SCSI request packet.  Each component, starting from the host's HBA port, through the SAN switches, the array front side, the
array backend ports, the physical drives, etc should all enforce the check along the way.

Instead of having to worry about which block that is written down is "relevant Oracle data", we are, inside the storage, contending with a standard SCSI write request that direct us specifically to take additional checking action on the
data block received per the T10 standard definition, reporting back any potential errors back in the manner specified by the standard.  The devices just need to be T10 DIF compliant.  They do not have to worry about distinguishing between Oracle data and something else.

EMC's  Yaron Dar, who wrote the techbook you quoted - "Oracle Databases on EMC Symmetrix Storage Systems", presented at Openworld this year covering T10 PI. A copy of the OOW slides can be found here with the info on T10 PI at  slide 33 onwards

https://oracleus.wingateweb.com/published/oracleus2011/sessions/33580/S33580_1542630.pdf (url currently unavailable)

EMC and Oracle also  co-presented at OOW 2011 on An Integrated End-to-End Data Integrity Solution for Oracle Products
A copy can be found on the Oracle site at  http://oss.oracle.com/~mkp/docs/OOW2011-DI.pdf

Allan

Hi Allan,

Good info, added the link to my blogpost....

AFAIK this T10 PI is a proposed standard and has not been implemented yet... correct?

For anyone that missed it there is now an ELAB paper, on EMC Symmetrix VMAX and co-branded  with Oracle and Emulex, describing the implementation of T10 PI with DIX (Data Integrity Extensions) to ensure data integrity from Application To Disk.

You can also see some information From the OS side here

Allan