[gpfsug-discuss] Initial file placement

[Jonathan Buzzard]

On Fri, 2016-06-17 at 15:18 +0000, Buterbaugh, Kevin L wrote:
> Hi yet again all, 
> 
> 
> Well, this has turned out to be an enlightening and surprising morning
> in GPFS land… 
> 
> 
> What prompted my question below is this … I am looking to use the new
> QoS features in GPFS 4.2.  I have QoS enabled and am trying to get a
> baseline of IOPs so that I can determine how much I want to assign to
> the maintenance class (currently both maintenance and other are set to
> unlimited).  To do this, I fired off a bonnie++ test from each of my
> NSD servers.
> 
> 
> The filesystem in question has two storage pools, the system pool and
> the capacity pool.  The system pool is comprised of a couple of
> metadata only disks (SSD-based RAID 1 mirrors) and several data only
> disks (spinning HD-based RAID 6), while the capacity pool is comprised
> exclusively of data only disks (RAID 6).
> 

My preference in these scenarios is to have the system pool for metadata
only. Then have a "fast" pool and a "slow" pool. Pool names more than 7
characters is a recipe for backup headaches with TSM at least, so "fast"
and "slow" are deliberately chosen as names :)

That way you can tweak block sizes more optimally.

> 
> When the bonnie++’s were creating, reading, and rewriting the big file
> they create I was quite surprised to see mmlsqos show higher IOP’s on
> the capacity pool than the system pool by a factor of 10!  As I was
> expecting those files to be being written to the system pool, this was
> quite surprising to me.  Once I found the mmlsattr command, I ran it
> on one of the files being created and saw that it was indeed assigned
> to the capacity pool.  The bonnie++’s finished before I could check
> the other files.
> 
> 
> I don’t have any file placement policies in effect for this
> filesystem, only file migration policies (each weekend any files in
> the system pool with an atime > 60 days get moved to the capacity pool
> and any files in the capacity pool with an atime < 60 days get moved
> to the system pool).

Note due to the way applications handle saving files by creating a new
file, moving files from slow to fast disk is mostly a waste of effort in
my experience. By the time you do that, 99% of the time the user has
finished with the file anyway. When they haven't generally the slow disk
has gazillions of free IOPs anyway.

>
> In the GPFS 4.2 Advanced Administration Guide, it states, “If a GPFS
> file system does not have a placement policy installed, all the data
> is stored in the first data storage pool.”
> 

My understanding is that no placement policy and it can go anywhere,
where anywhere is the next available NSD with free space. I found that
you needed to do something like this at the end of your policy file for
the default placement.

/* move files over 1GB onto nearline storage and keep them there */
RULE 'big' MIGRATE FROM POOL 'system' TO POOL 'nearline' WHERE FILE_SIZE
> 1073741824 

/* migrate old files to the RAID6 disks to keep the RAID1 disks free */
RULE 'ilm' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(weighting)
TO POOL 'slow'

/* by default new files to the fast RAID1 disk unless full, then to
RAID6 */
RULE 'new' SET POOL 'fast' LIMIT(95)
RULE 'spillover' SET POOL 'slow'

You have to stop allocation to the fast disk from ever approaching 100%.
Otherwise a you get disk full errors on new file writes. Imagine you
create a new file when there is 1KB of space free in the fast pool, then
try and write 10KB of data; it will fail with an error. So first rule
forces large files to the slow pool anyway (you can tweak this size to
suite your environment) but it must be significantly less than the the
space free when you hit 95% full on the fast disk.

You then need to have a rule to kick the tiering migration in lower than
when you start allocating to the slow pool.

Finally you allocation new files to the fast disk till it gets quite
full, before spilling over to the slow pool (which one presumes has
loads of free capacity) if the fast pool fills up.

Not shown you ideally need to kick in tiering migration every night so
that it does not run during the day when users might notice the
performance degradation. Ideally you want to have enough free space come
the morning in your fast disk pool that all new files can be created
there during the day. 

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.