DataIQ 2.2 Admin Guide

From the DataIQ browser: Select Settings > Data management configuration.

In the Plugins section, select the ⋮ (vertical ellipses) icon next to Previewer, and select Edit configuration.

Configuration file conventions:

• Use spaces for indentations, not tabs.
• Ensure indentation matches exactly.
• When using Boolean values and numbers, do not use quote marks. For all other values, use quote marks.

watermark (required)

The watermark value is a path to an image that is superimposed on graphics in the DataIQ previewer.

To add a custom watermark:

• Create a transparent .png image. The recommended size is 160x100 pixels.
• Add the image to the DataIQ host in the /opt/dataiq/maunakea/data/plugins/previewer/ directory.
• Refer to the plugin in the configuration file as /hoststorage/<filename>, for example:

  "watermark":
    value: "/hoststorage/MyCustomWatermark.png"
    default: "watermark.png"

exclude_s3 (required)

The exclude_s3 value determines whether to refuse to preview image objects on S3 volumes.

True: Image objects on s3 volumes are not shown in the preview window.

False: Image objects on s3 volumes are shown in the preview window.

 "exclude_s3":
   value: False

s3_max_bytes (required)

Use exclude_s3 to prevent situations from attempting to preview very large files. When a large file is selected for preview, DataIQ immediately displays a message that a preview is not available.

The value is the maximum size in bytes of S3 image objects to try to preview. The default value is 1073741824 (= 1 GiB).

 "s3_max_bytes":
   value: 1073741824

S3 buckets (optional)

Format:

"S3 buckets:"
  <On the DataIQ host, add "value:". Do not add "value:" on worker nodes>
    "<bucket_name>": # Repeat this section for each bucket
      "dataiq_volume": "< DataIQ Volume Name >"
      "access_key": "< bucket access key >"
      "secret_key": "< bucket secret key >"
      "endpoint": "< endpoint url >"

< bucket_name > (required)

The name of the S3 bucket. The name must match the bucket name that is used on the S3 service. Buckets with the same name are not supported.

dataiq_volume (optional)

The name of the volume within DataIQ. Only use this setting when the S3 bucket is already represented as a volume within DataIQ. The name must match the DataIQ volume name, and it must be prepended with a forward slash (/). If this value is not present, the bucket may have files that are transferred to it, but no transfer will be allowed from the bucket to the file system.

access_key (required)

The access key for the S3 bucket.

secret_key (required)

The secret key for the S3 bucket.

endpoint (required)

The endpoint URL of the S3 bucket.

Example for Data Mover Host:

"S3 buckets:"
  value:
    "amazon-s3-bucket": 
      "dataiq_volume": "/amazon_s3"
      "access_key": "123234345456567678789890"
      "secret_key": "ABCDE12345"
      "endpoint": "s3.ap-northeast-1.amazonaws.com"
    "ecs-s3-bucket": 
      "dataiq_volume": "/ecs"
      "access_key": "dataiquser"
      "secret_key": "FGHIJ67890"
      "endpoint": "http://10.123.45.67:9090"
    "google-s3-bucket-3": 
      "dataiq_volume": "/gcp_on_us_west_4"
      "access_key": "12345abcde7890a1b2c3d4e5"
      "secret_key": "PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc"
      "endpoint": "https://us-west4.storage.googleapis.com"
    "google-s3-bucket-on-asia-east2": 
      "dataiq_volume": "/gcp_on_asia_east_2"
      "access_key": "12345abcde7890a1b2c3d4e5"
      "secret_key": "PqRsT67890uVwXyZ/A1fB2gC3hD4iE5j\abc"
      "endpoint": "https://asia-east2.storage.googleapis.com"

"S3 buckets:"
    "amazon-s3-bucket": 
      "dataiq_volume": "/amazon_s3"
      "access_key": "123234345456567678789890"
      "secret_key": "ABCDE12345"
      "endpoint": "s3.ap-northeast-1.amazonaws.com"

    (...etc...)