From patchwork Thu May 27 18:32:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Scott Mayhew X-Patchwork-Id: 12285089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28E6AC4708A for ; Thu, 27 May 2021 18:32:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 09308613D4 for ; Thu, 27 May 2021 18:32:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234155AbhE0SeR (ORCPT ); Thu, 27 May 2021 14:34:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:58598 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235787AbhE0SeQ (ORCPT ); Thu, 27 May 2021 14:34:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622140362; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=de/Y/qlIvfzMz0jKsKeUXwFze4xGleIcl6uqoXpcELc=; b=OOBsgkd5B6bBBa8b6QwqfRY8aObnk566RZzerdIix84EeDT11t49JQ3TiT3M8PJMpCWuX2 KFZJLb4L+QzVRqOylFh2wYZ+dw67044lUdJ09eKEiYXZucYcAAip7geBdYX38w3jTiL2j5 21HlqvQFuCnOS/bcYV9pzHXxJ8l2+yk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-374-vstTTJXNPbqrN2GYu1i66g-1; Thu, 27 May 2021 14:32:40 -0400 X-MC-Unique: vstTTJXNPbqrN2GYu1i66g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C90091903100 for ; Thu, 27 May 2021 18:32:39 +0000 (UTC) Received: from aion.usersys.redhat.com (ovpn-114-18.rdu2.redhat.com [10.10.114.18]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AF37F6061F for ; Thu, 27 May 2021 18:32:39 +0000 (UTC) Received: by aion.usersys.redhat.com (Postfix, from userid 1000) id F2C491A003D; Thu, 27 May 2021 14:32:38 -0400 (EDT) From: Scott Mayhew To: linux-nfs@vger.kernel.org Subject: [nfs-utils PATCH v2 0/2] Two rpc.gssd improvements Date: Thu, 27 May 2021 14:32:36 -0400 Message-Id: <20210527183238.584349-1-smayhew@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Changes since v1: - Replaced the upcall_thread_info.cancelled field with a flags field, to facilitate having the watchdog thread print an error message only once for each timed-out upcall thread. - Removed the "created thread id" log message. - Added missing break when parsing the "-C" option. - Added some comments. These patches provide the following improvements for rpc.gssd: 1) deal with failed thread creation 2) add a timeout for upcall threads Both of these issues can leave kernel mount processes hanging indefinitely. A timeout was originally proposed in the kernel (https://lore.kernel.org/linux-nfs/20180618172542.45519-1-steved@redhat.com/) but this approach was rejected by Trond: I'm saying that we can do this entirely in userland without any kernel changes. As long as that hasn't been attempted and proven to be flawed, then there is no reason to accept any kernel patches. So this is my attempt at doing the timeout in userland. The first patch was tested using a program that intercepts clone() and changes the return code to -EAGAIN. For the second patch, I have two different tests I've been running: 1) In an IPA domain in our lab, I have a server running 100 kerberized nfsd containers. The client has mountpoints to all 100 of those servers defined in its /etc/fstab. I run 'systemctl start remote-fs.target' to kick off all those mounts in parallel, while running the following systemtap script to periodically mess with the mount processes: ---8<--- global i probe begin { i=0 } probe process("/lib64/libgssapi_krb5.so.2").function("gss_acquire_cred") { if (++i % 100 == 0) { printf("delay (i=%d)\n", i) mdelay(30000) } } ---8<--- I actually run the test in a loop... the driver script looks like this: ---8<--- #!/bin/bash let i=1 while :; do echo "Round $i" echo "Mounting" systemctl start remote-fs.target echo -n "Waiting on mount.nfs processes to complete " while pgrep mount.nfs >/dev/null; do echo -n "." sleep 1 done echo -e "\nNumber of nfs4 mounts: $(grep -c nfs4 /proc/mounts)" echo -e "Unmounting" umount -a -t nfs4 if ! pgrep gssd >/dev/null; then echo "gssd is not running - check for crash" break fi echo "Sleeping 5 seconds" sleep 5 let i=$i+1 done ---8<--- 2) In an AD environment in our lab, I added 1000 test users. On a client machine I have all those users run a script that writes to files on a NetApp SVM and while that script is running I trigger a LIF migration on the filer. That forces all those users to establish new creds with the SVM. That test looks basically like this # for i in `seq 1 1000`; do su - testuser$i -c "echo 'PASSWORD'|kinit"; done # for i in `seq 1 1000`; do su - testuser$i -c "date >/mnt/t/tmp/testuser$i-testfile" & done # for i in `seq 1 1000`; do su - testuser$i -c test.sh & done where test.sh is a simple script that writes the date to a file in a loop: ---8<--- #!/bin/bash filename=/mnt/t/tmp/$(whoami)-testfile for i in $(seq 1 300) do date >$filename sleep 1 done ---8<--- While the test users are running the script I run one of the following commands on the NetApp filer: network interface migrate -vserver VSERVER -lif LIF -destination-node NODE network interface revert -vserver VSERVER -lif LIF -Scott Scott Mayhew (2): gssd: deal with failed thread creation gssd: add timeout for upcall threads nfs.conf | 2 + utils/gssd/gssd.c | 256 +++++++++++++++++++++++----------- utils/gssd/gssd.h | 29 +++- utils/gssd/gssd.man | 31 ++++- utils/gssd/gssd_proc.c | 306 ++++++++++++++++++++++++++++++++++------- 5 files changed, 491 insertions(+), 133 deletions(-)