PowerScale OneFS HDFS Configuration Guide

PDF

HDFS service settings overview

HDFS service settings affect the performance of HDFS workflows.

You can configure the following HDFS service settings:

Table 1. Setting SpecificationsThe following table describes each setting.
Setting	Description
Block size	The HDFS block size setting on the PowerScale cluster determines how the HDFS service returns data on read requests from Hadoop compute client. You can modify the HDFS block size on the cluster to increase the block size from 4 KB up to 1 G. The default block size is 128 MB. Increasing the block size enables the PowerScale cluster nodes to read and write HDFS data in larger blocks and optimize performance for most use cases. The Hadoop cluster maintains a different block size that determines how a Hadoop compute client writes a block of file data to the PowerScale cluster. The optimal block size depends on your data, how you process your data, and other factors. You can configure the block size on the Hadoop cluster in the hdfs-site.xml configuration file in the dfs.block.size property.
Checksum type	The HDFS service sends the checksum type to Hadoop compute clients, but it does not send any checksum data, regardless of the checksum type. The default checksum type is set to None. If your Hadoop distribution requires sending a checksum type other than None to the client, you can set the checksum type to CRC32 or CRC32C.