Exporting InsightIQ data

If you do upgrade, make sure you export your data FIRST. For some reason, EMC/Isilon doesn't support just dropping the data into a new instance of IIQ post upgrade, so if something goes wrong, you'll be up a creek.

Also, 3.2.1 doesn't work quite right, so make sure you're on 3.2.2.

Thanks Katie, would you happen to know if those reports can be sent via email automatically? and or is it possible to automatically pull those reports for processing through a script?  doing it manually is a horrible.

I had to go to File System Reporting -> File System Analytics -> Select a report date -> Click on View Report -> then click on the Directory -> then Download CSV.

Imagine if you have 3 months worth of data.. and moving forward I dont want to go to insight everyday for download the CSV file.

i guess i will be staying on 3.1.1 unless 3.2.2 will allow me to get the data i wanted automatically..

Hi brutuz,

I'm a Technical Marketing Engineer, working on the Isilon product line.  I'm covering InsightIQ technicals in this new role.

I'm in the process of pulling together a list of blogs to answer some of these common questions.  Yours is a very common ask.  While I'd like to offer a more thoroughly documented process, I'll provide a slimmed down version here to show what this product can do today.

Since you intend on pulling the filesystem knowledge fairly frequently, tinkering with the GUI can be painful, as you stated.  I would try using the CLI.

The following has been tested on InsightIQ versions 3.2.0 and 3.2.1.

On the node that is running your InsightIQ instance:

$ iiq_data_export fsa list --reports

  Available Reports for: Time Zone: PDT

    ID  FSA Job Start          FSA Job End            Size

--------------------------------------------------------------------------------

    765  Sep 30 2015, 10:00 PM  Oct 01 2015, 11:28 PM  3.500G

--------------------------------------------------------------------------------

$ iiq_data_export fsa export --data-module directories --cluster --report 765 --filter-rule directory:TEST/MY/products/xxxx/vxx

    Successfully exported data to: directories_ _765_1443800888.csv

$ cat directories_ _765_1443800888.csv

path[directory:/ifs/TEST/MY/products/xxxx/vxx/],dir_cnt (count),file_cnt (count),ads_cnt,other_cnt (count),log_size_sum (bytes),phys_size_sum (bytes),log_size_sum_overflow,report_date: 1443675617

TEST/MY/products/xxxx/vxx/test,1,0,0,0,0,2048,0

Note: The directory specified in the filter rule "TEST/MY/products/xxxx/vxx" -- this directory exists on the cluster as "/ifs/TEST/MY/products/xxxx/vxx".

Hope this helps.

Hi Rob,

     This is a great help! After playing around with it.. I was able to get all the information I needed..

DATEREF=$(date -d yesterday +"%b %d %Y")

REPORT_ID=$(iiq_data_export fsa list --report MYISILON | grep "${DATEREF}" | awk '{print $1}')

iiq_data_export fsa export --data-module directories --cluster isi-01 --report ${REPORT_ID} --filter-rule ${DIRECTORY}

RETVAL=$?

# Process

${WORKDIR}/process-iiq-report.pl

i noticed that iiq_data_export is a python code.. you can actually extend it.. though i need to learn python..

and also if there is documentation on the schema on the files stored in the /datastore files that would be great.. IIQ is using sqlite3 as db to store all information..

Thank you,

Good to hear!

In terms of the historical performance metrics repository that is InsightIQ's "/datastore" -- many customers have asked us about a published schema.  Unfortunately these do not exist.  Going forward, we urge customers to use the Isilon API (old name "Platform API") to query for performance metrics from the cluster itself, instead of trying to mine the datastore in InsightIQ.  Everything performance related that InsightIQ gets from the cluster CAN be obtained through Isilon API.

Also as of IIQ version 3.2 (code name Lumen), IIQ's datastore (which stores only historical performance metrics) has been changed to PostgreSQL -- making it leaner and faster.  So trying to mine IIQ data is only going to be short-lived.

Thanks again for your feedback!

I noticed some minor error:

path[directory:/ifs/],dir_cnt (count),file_cnt (count),ads_cnt,other_cnt (count),log_size_sum (bytes),phys_size_sum (bytes),log_size_sum_overflow,report_date: 1444168813

nfs-shares,272737,7593935,0,54654,44072371405990,55915296110080,0

the logical size here is bigger than physical size but when it gets presented in IIQ web interface the physical because the bigger value..

i guess the the header in the result for the iiq_export is wrong...

Rob,

I "re-sell" storage to my internal customers who would like to be able to look at trending information in FSA to help with storage budgets.  Specifically tending information on quota utilization. My customers would like to login to IIQ, and look at their directory quota and see how storage utilization in that quota increased over time. Is quota related information exported from IIQ as well ?  I would be ok with throwing more storage at the IIQ appliance and have this information delivered by IIQ instead of my exporting it and rolling it into my own plotting tool.  Thoughts ?

hi dynamox,

  i started using IIQ version 3.0.1 and after i upgraded to 3.1.1.0022 to be exact.  They have included "Quota Reporting" under "File System Reporting" which as far as I remember it wasn't there.

  and the tool the rob provided can be helpful as well if you want to export tons of data with ease.

hth,

Hi Brutuz,

Thanks for the feedback.  I checked the iiq_data_export output with my own IIQ instance and it looked normal.

I took the output that you provided and parsed it out in Excel:

It looks like the logical size is smaller than physical size.  Physical size is 11,842,924,704,090 bytes bigger than logical due to data protection overhead.

Could you check again?

Thanks!

Hi dynamox & brutuz,

Thank you for your comments.

What dynamo is referring to is a “charge-back” feature that many customers have been asking for in the recent months.   While Isilon product management and engineering teams are developing roadmap and solutions to a full-fledged charge-back feature, in the interim, OneFS and InsightIQ can get you what you need today in terms of quota reporting & quota usage trending.

Let me attach two screenshots from InsightIQ version 3.2.0.  (Note that the newest release is version 3.2.2 with minor bug fixes.) The two screenshots are coming from the same Quota Reporting page.

Under File System Reporting, the Quota Reporting tab allows you to display quota reports.  As indicated in the screenshot below, you get to choose which quota report to view.  And yes you are seeing this correctly -- “live data” is available!

In the screenshot below, from the same quota reporting page, I’m selecting a particular report, and then highlighting a row that indicates a “user quota”.  This “user quota” was set at the level of /ifs/home, which includes all subdirectories under that path.  So anything owned by user “rchang” under /ifs/home and its subdirectories get counted towards rchang’s quota.  In use, you might setup one of these user quotas per project team, for example.

With the user quota row selected, the “View Latest 10 Reports” button allows you to plot quota trends for this user with 10 data points.

These quotas can be setup on the cluster itself through the WebUI.

Would this work for your needs?

Hi RobChang-Isilon, I do recognise this is a very old thread, but the topic of "charge-back" is interesting! Is there any plan to introduce RBAC (role based access) for the IIQ portal so we can give specific users access only to specific type of reports, or clusters? Ideally I would not want the customer to have full performance reporting access, but happy to share File System Analytics.

Thanks!

Hi jmhere,

What you are referring to -- the RBAC of viewability of specific reports and directory/contents -- has been on the radar of product management for a while.  The specifics are being defined/refined right now.  If you wouldn't mind, please work with your Isilon SE to file a feature request.  We try to capture each customer's proposed use case and engineer a solution that fits most.

Thanks!