[gpfsug-discuss] GPFS 4.2.3.4 question

Wed Aug 30 15:28:07 BST 2017

Hi Bryan,

NO - it has the fix for the mmrestripefs data loss bug, but you need the efix on top of 4.2.3-4 for the mmadddisk / mmdeldisk issue.

Let me take this opportunity to also explain a workaround that has worked for us so far for that issue … the basic problem is two-fold (on our cluster, at least).  First, the /var/mmfs/gen/mmsdrfs file isn’t making it out to all nodes all the time.  That is simple enough to fix (mmrefresh -fa) and verify that it’s fixed (md5sum /var/mmfs/gen/mmsdrfs).

Second, however - and this is the real problem … some nodes are never actually rereading that file and therefore have incorrect information *in memory*.  This has been especially problematic for us as we are replacing a batch of 80 8 TB drives with bad firmware.  I am therefore deleting and subsequently recreating NSDs *with the same name*.  If a client node still has the “old” information in memory then it unmounts the filesystem when I try to mmadddisk the new NSD.

The workaround is to identify those nodes (mmfsadm dump nsd and grep for the identifier of the NSD(s) in question) and force them to reread the info (tsctl rereadnsd).

HTH…

Kevin

On Aug 30, 2017, at 9:21 AM, Bryan Banister <bbanister at jumptrading.com<mailto:bbanister at jumptrading.com>> wrote:

Ok, I’m completely confused… You’re saying 4.2.3-4 *has* the fix for adding/deleting NSDs?
-Bryan

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: Wednesday, August 30, 2017 9:13 AM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] GPFS 4.2.3.4 question

Note: External Email
________________________________
Aha, I’ve just realised what you actually said, having seen Simon’s response and twigged. The defect 1020461 matches what IBM has told me in my PMR about adding/deleting NSDs. I’m not sure why the description mentions networking though!

Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Sobey, Richard A
Sent: 30 August 2017 14:56
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] GPFS 4.2.3.4 question

No worries, I’ve got it sorted and hopefully about to grab the 4.2.3-4 efix2.

Cheers for your help!
Richard

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L
Sent: 30 August 2017 14:55
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] GPFS 4.2.3.4 question

Hi Richard,

Well, I’m not sure, which is why it’s taken me a while to respond.  In the README that comes with the efix it lists:

Defect APAR    Description

1032655 None AFM: Fix Truncate filtering Write incorrectly
1020461 None FS can't be mounted after weird networking error

That 1st one is obviously not it and that 2nd one doesn’t reference mmadddisk / mmdeldisk.  Plus neither show an APAR number.

Sorry I can’t be of more help…

Kevin

On Aug 29, 2017, at 12:52 PM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

Thanks Kevin, that's good to know. Is there an apar I need to quote in my pmr?
Get Outlook for Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=op1veAuespuhwL9zsBMZe%2FwVAn6NC%2FkL0U4IKtAmVT4%3D&reserved=0>

________________________________
From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> <gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>> on behalf of Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
Sent: Tuesday, August 29, 2017 4:53:51 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] GPFS 4.2.3.4 question

Hi Richard,

Since I upgraded my cluster to GPFS 4.2.3.4 over the weekend IBM created an efix for it for the NSD deletion / creation fix.  I’m sure they’ll give it to you, too…  ;-)

Kevin

On Aug 29, 2017, at 9:30 AM, Sobey, Richard A <r.sobey at imperial.ac.uk<mailto:r.sobey at imperial.ac.uk>> wrote:

So I can upgrade to 4.2.3-4 to get the mmrestripe fix, or 4.2.3-3 efix3 to get the NSD deletion and creation fix? Not great when on Monday I’m doing a load of all this. What’s the recommendation? Is there a one size fits all patch?

From: gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Frederick Stock
Sent: 27 August 2017 01:35
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Subject: Re: [gpfsug-discuss] GPFS 4.2.3.4 question

The only change missing is the change delivered  in 4.2.3 PTF3 efix3 which was provided on August 22.  The problem had to do with NSD deletion and creation.

Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
stockf at us.ibm.com<mailto:stockf at us.ibm.com>

From:        "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu<mailto:Kevin.Buterbaugh at Vanderbilt.Edu>>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date:        08/26/2017 03:40 PM
Subject:        [gpfsug-discuss] GPFS 4.2.3.4 question
Sent by:        gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
________________________________

Hi All,

Does anybody know if GPFS 4.2.3.4, which came out today, contains all the patches that are in GPFS 4.2.3.3 efix3?

If anybody does, and can respond, I’d greatly appreciate it.  Our cluster is in a very, very bad state right now and we may need to just take it down and bring it back up.  I was already planning on rolling out GPFS 4.2.3.3 efix 3 over the next few weeks anyway, so if I can just go to 4.2.3.4 that would be great…

Thanks!

—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=7aIpvwHm9jiSsj0kOAXLwEO1EvXb%2FH6ntKysDCh0WuY%3D&reserved=0>
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=p_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw&m=7r9GsD1C2HiY4j21vPYIoQPHXePHxeMhzQeaw_ne4lM&s=-SFnqoJw--FN3wqClEEBGa9-XSLljgSseIU_SxGoWy0&e=<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttp-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss%26d%3DDwICAg%26c%3Djf_iaSHvJObTbx-siA1ZOg%26r%3Dp_1XEUyoJ7-VJxF_w8h9gJh8_Wj0Pey73LCLLoxodpw%26m%3D7r9GsD1C2HiY4j21vPYIoQPHXePHxeMhzQeaw_ne4lM%26s%3D-SFnqoJw--FN3wqClEEBGa9-XSLljgSseIU_SxGoWy0%26e%3D&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=fBBdaghWvQ9%2By%2B8eIM1%2FRJ9PlxJ63MjNwr7UJ50AeNM%3D&reserved=0>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org%2F&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=7aIpvwHm9jiSsj0kOAXLwEO1EvXb%2FH6ntKysDCh0WuY%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=qYxCMMg9O31LzFg%2FQkCdQg8vV%2FgL2AuRk%2B6V2j76c7Y%3D&reserved=0>

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fspectrumscale.org&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=%2F2nCbkq2CJh3FX4UzyT3rWiImQE2Q%2BphLeFqaD9fhMg%3D&reserved=0>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=qYxCMMg9O31LzFg%2FQkCdQg8vV%2FgL2AuRk%2B6V2j76c7Y%3D&reserved=0>

—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633

________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40vanderbilt.edu%7C493f1f9e41e343324f1508d4efb25f4f%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636396996783027614&sdata=qYxCMMg9O31LzFg%2FQkCdQg8vV%2FgL2AuRk%2B6V2j76c7Y%3D&reserved=0

—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170830/daf503fb/attachment-0002.htm>