From patchwork Tue Aug 9 23:39:52 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew R. Ochs" X-Patchwork-Id: 9272421 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 827F460231 for ; Tue, 9 Aug 2016 23:40:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 593A727D8D for ; Tue, 9 Aug 2016 23:40:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C62227F85; Tue, 9 Aug 2016 23:40:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CADE27D8D for ; Tue, 9 Aug 2016 23:40:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932648AbcHIXkJ (ORCPT ); Tue, 9 Aug 2016 19:40:09 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:30914 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932614AbcHIXkI (ORCPT ); Tue, 9 Aug 2016 19:40:08 -0400 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.11/8.16.0.11) with SMTP id u79Ncv1B069441 for ; Tue, 9 Aug 2016 19:40:08 -0400 Received: from e32.co.us.ibm.com (e32.co.us.ibm.com [32.97.110.150]) by mx0a-001b2d01.pphosted.com with ESMTP id 24qm9sa9b7-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 09 Aug 2016 19:40:07 -0400 Received: from localhost by e32.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 9 Aug 2016 17:40:06 -0600 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e32.co.us.ibm.com (192.168.1.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 9 Aug 2016 17:40:03 -0600 X-IBM-Helo: d03dlp03.boulder.ibm.com X-IBM-MailFrom: mrochs@linux.vnet.ibm.com Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id CAF4B19D8040; Tue, 9 Aug 2016 17:39:35 -0600 (MDT) Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u79Ne0dQ12517670; Tue, 9 Aug 2016 16:40:00 -0700 Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 71934BE042; Tue, 9 Aug 2016 17:40:00 -0600 (MDT) Received: from p8tul1-build.aus.stglabs.ibm.com (unknown [9.3.141.206]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP id 97E54BE039; Tue, 9 Aug 2016 17:39:59 -0600 (MDT) From: "Matthew R. Ochs" To: linux-scsi@vger.kernel.org, James Bottomley , "Martin K. Petersen" , Uma Krishnan , "Manoj N. Kumar" , Al Viro Cc: Brian King , linuxppc-dev@lists.ozlabs.org, Ian Munsie , Andrew Donnellan , Frederic Barrat , Christophe Lombard , "Matthew R. Ochs" Subject: [PATCH 4/6] cxlflash: Transition to application close model Date: Tue, 9 Aug 2016 18:39:52 -0500 X-Mailer: git-send-email 2.1.0 In-Reply-To: <1470785888-9112-1-git-send-email-mrochs@linux.vnet.ibm.com> References: <1470785888-9112-1-git-send-email-mrochs@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16080923-0004-0000-0000-0000101A14F0 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00005571; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000178; SDB=6.00742429; UDB=6.00349471; IPR=6.00514986; BA=6.00004652; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00012282; XFM=3.00000011; UTC=2016-08-09 23:40:05 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16080923-0005-0000-0000-000077D72912 Message-Id: <1470785992-9274-1-git-send-email-mrochs@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-08-09_10:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1604210000 definitions=main-1608090248 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Caching the adapter file descriptor and performing a close on behalf of an application is a poor design. This is due to the fact that once a file descriptor in installed, it is free to be altered without the knowledge of the cxlflash driver. This can lead to inconsistencies between the application and kernel. Furthermore, the nature of the former design is more exploitable and thus should be abandoned. To support applications performing a close on the adapter file that is associated with a context, a new flag is introduced to the user API to indicate to applications that they are responsible for the close following the cleanup (detach) of a context. The documentation is also updated to reflect this change in behavior. Inspired-by: Al Viro Signed-off-by: Matthew R. Ochs Acked-by: Manoj N. Kumar --- Documentation/powerpc/cxlflash.txt | 42 +++++++++++++++++++--- drivers/scsi/cxlflash/superpipe.c | 71 ++++++++++---------------------------- drivers/scsi/cxlflash/vlun.c | 13 ++----- include/uapi/scsi/cxlflash_ioctl.h | 19 +++++++--- 4 files changed, 73 insertions(+), 72 deletions(-) diff --git a/Documentation/powerpc/cxlflash.txt b/Documentation/powerpc/cxlflash.txt index 4202d1b..f4c1190 100644 --- a/Documentation/powerpc/cxlflash.txt +++ b/Documentation/powerpc/cxlflash.txt @@ -171,11 +171,30 @@ DK_CXLFLASH_ATTACH destroyed, the tokens are to be considered stale and subsequent usage will result in errors. + - A valid adapter file descriptor (fd2 >= 0) is only returned on + the initial attach for a context. Subsequent attaches to an + existing context (DK_CXLFLASH_ATTACH_REUSE_CONTEXT flag present) + do not provide the adapter file descriptor as it was previously + made known to the application. + - When a context is no longer needed, the user shall detach from - the context via the DK_CXLFLASH_DETACH ioctl. + the context via the DK_CXLFLASH_DETACH ioctl. When this ioctl + returns with a valid adapter file descriptor and the return flag + DK_CXLFLASH_APP_CLOSE_ADAP_FD is present, the application _must_ + close the adapter file descriptor following a successful detach. + + - When this ioctl returns with a valid fd2 and the return flag + DK_CXLFLASH_APP_CLOSE_ADAP_FD is present, the application _must_ + close fd2 in the following circumstances: + + + Following a successful detach of the last user of the context + + Following a successful recovery on the context's original fd2 + + In the child process of a fork(), following a clone ioctl, + on the fd2 associated with the source context - - A close on fd2 will invalidate the tokens. This operation is not - required by the user. + - At any time, a close on fd2 will invalidate the tokens. Applications + should exercise caution to only close fd2 when appropriate (outlined + in the previous bullet) to avoid premature loss of I/O. DK_CXLFLASH_USER_DIRECT ----------------------- @@ -254,6 +273,10 @@ DK_CXLFLASH_DETACH success, all "tokens" which had been provided to the user from the DK_CXLFLASH_ATTACH onward are no longer valid. + When the DK_CXLFLASH_APP_CLOSE_ADAP_FD flag was returned on a successful + attach, the application _must_ close the fd2 associated with the context + following the detach of the final user of the context. + DK_CXLFLASH_VLUN_CLONE ---------------------- This ioctl is responsible for cloning a previously created @@ -261,7 +284,7 @@ DK_CXLFLASH_VLUN_CLONE support maintaining user space access to storage after a process forks. Upon success, the child process (which invoked the ioctl) will have access to the same LUNs via the same resource handle(s) - and fd2 as the parent, but under a different context. + as the parent, but under a different context. Context sharing across processes is not supported with CXL and therefore each fork must be met with establishing a new context @@ -275,6 +298,12 @@ DK_CXLFLASH_VLUN_CLONE translation tables are copied from the parent context to the child's and then synced with the AFU. + When the DK_CXLFLASH_APP_CLOSE_ADAP_FD flag was returned on a successful + attach, the application _must_ close the fd2 associated with the source + context (still resident/accessible in the parent process) following the + clone. This is to avoid a stale entry in the file descriptor table of the + child process. + DK_CXLFLASH_VERIFY ------------------ This ioctl is used to detect various changes such as the capacity of @@ -309,6 +338,11 @@ DK_CXLFLASH_RECOVER_AFU at which time the context/resources they held will be freed as part of the release fop. + When the DK_CXLFLASH_APP_CLOSE_ADAP_FD flag was returned on a successful + attach, the application _must_ unmap and close the fd2 associated with the + original context following this ioctl returning success and indicating that + the context was recovered (DK_CXLFLASH_RECOVER_AFU_CONTEXT_RESET). + DK_CXLFLASH_MANAGE_LUN ---------------------- This ioctl is used to switch a LUN from a mode where it is available diff --git a/drivers/scsi/cxlflash/superpipe.c b/drivers/scsi/cxlflash/superpipe.c index be7522a..b3bb90d 100644 --- a/drivers/scsi/cxlflash/superpipe.c +++ b/drivers/scsi/cxlflash/superpipe.c @@ -825,7 +825,6 @@ static void remove_context(struct kref *kref) { struct ctx_info *ctxi = container_of(kref, struct ctx_info, kref); struct cxlflash_cfg *cfg = ctxi->cfg; - int lfd; u64 ctxid = DECODE_CTXID(ctxi->ctxid); /* Remove context from table/error list */ @@ -842,19 +841,7 @@ static void remove_context(struct kref *kref) mutex_unlock(&ctxi->mutex); /* Context now completely uncoupled/unreachable */ - lfd = ctxi->lfd; destroy_context(cfg, ctxi); - - /* - * As a last step, clean up external resources when not - * already on an external cleanup thread, i.e.: close(adap_fd). - * - * NOTE: this will free up the context from the CXL services, - * allowing it to dole out the same context_id on a future - * (or even currently in-flight) disk_attach operation. - */ - if (lfd != -1) - sys_close(lfd); } /** @@ -949,34 +936,18 @@ static int cxlflash_disk_detach(struct scsi_device *sdev, * * This routine is the release handler for the fops registered with * the CXL services on an initial attach for a context. It is called - * when a close is performed on the adapter file descriptor returned - * to the user. Programmatically, the user is not required to perform - * the close, as it is handled internally via the detach ioctl when - * a context is being removed. Note that nothing prevents the user - * from performing a close, but the user should be aware that doing - * so is considered catastrophic and subsequent usage of the superpipe - * API with previously saved off tokens will fail. + * when a close (explicity by the user or as part of a process tear + * down) is performed on the adapter file descriptor returned to the + * user. The user should be aware that explicitly performing a close + * considered catastrophic and subsequent usage of the superpipe API + * with previously saved off tokens will fail. * - * When initiated from an external close (either by the user or via - * a process tear down), the routine derives the context reference - * and calls detach for each LUN associated with the context. The - * final detach operation will cause the context itself to be freed. - * Note that the saved off lfd is reset prior to calling detach to - * signify that the final detach should not perform a close. - * - * When initiated from a detach operation as part of the tear down - * of a context, the context is first completely freed and then the - * close is performed. This routine will fail to derive the context - * reference (due to the context having already been freed) and then - * call into the CXL release entry point. - * - * Thus, with exception to when the CXL process element (context id) - * lookup fails (a case that should theoretically never occur), every - * call into this routine results in a complete freeing of a context. - * - * As part of the detach, all per-context resources associated with the LUN - * are cleaned up. When detaching the last LUN for a context, the context - * itself is cleaned up and released. + * This routine derives the context reference and calls detach for + * each LUN associated with the context.The final detach operation + * causes the context itself to be freed. With exception to when the + * CXL process element (context id) lookup fails (a case that should + * theoretically never occur), every call into this routine results + * in a complete freeing of a context. * * Return: 0 on success */ @@ -1014,11 +985,8 @@ static int cxlflash_cxl_release(struct inode *inode, struct file *file) goto out; } - dev_dbg(dev, "%s: close(%d) for context %d\n", - __func__, ctxi->lfd, ctxid); + dev_dbg(dev, "%s: close for context %d\n", __func__, ctxid); - /* Reset the file descriptor to indicate we're on a close() thread */ - ctxi->lfd = -1; detach.context_id = ctxi->ctxid; list_for_each_entry_safe(lun_access, t, &ctxi->luns, list) _cxlflash_disk_detach(lun_access->sdev, ctxi, &detach); @@ -1391,7 +1359,6 @@ static int cxlflash_disk_attach(struct scsi_device *sdev, __func__, rctxid); kref_get(&ctxi->kref); list_add(&lun_access->list, &ctxi->luns); - fd = ctxi->lfd; goto out_attach; } @@ -1461,7 +1428,11 @@ static int cxlflash_disk_attach(struct scsi_device *sdev, fd_install(fd, file); out_attach: - attach->hdr.return_flags = 0; + if (fd != -1) + attach->hdr.return_flags = DK_CXLFLASH_APP_CLOSE_ADAP_FD; + else + attach->hdr.return_flags = 0; + attach->context_id = ctxi->ctxid; attach->block_size = gli->blk_len; attach->mmio_size = sizeof(afu->afu_map->hosts[0].harea); @@ -1526,7 +1497,7 @@ static int recover_context(struct cxlflash_cfg *cfg, struct ctx_info *ctxi) { struct device *dev = &cfg->dev->dev; int rc = 0; - int old_fd, fd = -1; + int fd = -1; int ctxid = -1; struct file *file; struct cxl_context *ctx; @@ -1574,7 +1545,6 @@ static int recover_context(struct cxlflash_cfg *cfg, struct ctx_info *ctxi) * No error paths after this point. Once the fd is installed it's * visible to user space and can't be undone safely on this thread. */ - old_fd = ctxi->lfd; ctxi->ctxid = ENCODE_CTXID(ctxi, ctxid); ctxi->lfd = fd; ctxi->ctx = ctx; @@ -1593,9 +1563,6 @@ static int recover_context(struct cxlflash_cfg *cfg, struct ctx_info *ctxi) cfg->ctx_tbl[ctxid] = ctxi; mutex_unlock(&cfg->ctx_tbl_list_mutex); fd_install(fd, file); - - /* Release the original adapter fd and associated CXL resources */ - sys_close(old_fd); out: dev_dbg(dev, "%s: returning ctxid=%d fd=%d rc=%d\n", __func__, ctxid, fd, rc); @@ -1707,7 +1674,7 @@ retry_recover: recover->context_id = ctxi->ctxid; recover->adap_fd = ctxi->lfd; recover->mmio_size = sizeof(afu->afu_map->hosts[0].harea); - recover->hdr.return_flags |= + recover->hdr.return_flags = DK_CXLFLASH_APP_CLOSE_ADAP_FD | DK_CXLFLASH_RECOVER_AFU_CONTEXT_RESET; goto out; } diff --git a/drivers/scsi/cxlflash/vlun.c b/drivers/scsi/cxlflash/vlun.c index 50f8e93..90c5d7f 100644 --- a/drivers/scsi/cxlflash/vlun.c +++ b/drivers/scsi/cxlflash/vlun.c @@ -1135,14 +1135,13 @@ int cxlflash_disk_clone(struct scsi_device *sdev, ctxid_dst = DECODE_CTXID(clone->context_id_dst), rctxid_src = clone->context_id_src, rctxid_dst = clone->context_id_dst; - int adap_fd_src = clone->adap_fd_src; int i, j; int rc = 0; bool found; LIST_HEAD(sidecar); - pr_debug("%s: ctxid_src=%llu ctxid_dst=%llu adap_fd_src=%d\n", - __func__, ctxid_src, ctxid_dst, adap_fd_src); + pr_debug("%s: ctxid_src=%llu ctxid_dst=%llu\n", + __func__, ctxid_src, ctxid_dst); /* Do not clone yourself */ if (unlikely(rctxid_src == rctxid_dst)) { @@ -1166,13 +1165,6 @@ int cxlflash_disk_clone(struct scsi_device *sdev, goto out; } - if (unlikely(adap_fd_src != ctxi_src->lfd)) { - pr_debug("%s: Invalid source adapter fd! (%d)\n", - __func__, adap_fd_src); - rc = -EINVAL; - goto out; - } - /* Verify there is no open resource handle in the destination context */ for (i = 0; i < MAX_RHT_PER_CONTEXT; i++) if (ctxi_dst->rht_start[i].nmask != 0) { @@ -1257,7 +1249,6 @@ int cxlflash_disk_clone(struct scsi_device *sdev, out_success: list_splice(&sidecar, &ctxi_dst->luns); - sys_close(adap_fd_src); /* fall through */ out: diff --git a/include/uapi/scsi/cxlflash_ioctl.h b/include/uapi/scsi/cxlflash_ioctl.h index 2302f3c..6bf1f8a 100644 --- a/include/uapi/scsi/cxlflash_ioctl.h +++ b/include/uapi/scsi/cxlflash_ioctl.h @@ -39,19 +39,28 @@ struct dk_cxlflash_hdr { * at this time, this provides future flexibility. */ #define DK_CXLFLASH_ALL_PORTS_ACTIVE 0x0000000000000001ULL +#define DK_CXLFLASH_APP_CLOSE_ADAP_FD 0x0000000000000002ULL /* - * Notes: - * ----- + * General Notes: + * ------------- * The 'context_id' field of all ioctl structures contains the context * identifier for a context in the lower 32-bits (upper 32-bits are not * to be used when identifying a context to the AFU). That said, the value * in its entirety (all 64-bits) is to be treated as an opaque cookie and * should be presented as such when issuing ioctls. + */ + +/* + * DK_CXLFLASH_ATTACH Notes: + * ------------------------ + * Read/write access permissions are specified via the O_RDONLY, O_WRONLY, + * and O_RDWR flags defined in the fcntl.h header file. * - * For DK_CXLFLASH_ATTACH ioctl, user specifies read/write access - * permissions via the O_RDONLY, O_WRONLY, and O_RDWR flags defined in - * the fcntl.h header file. + * A valid adapter file descriptor (fd >= 0) is only returned on the initial + * attach (successful) of a context. When a context is shared(reused), the user + * is expected to already 'know' the adapter file descriptor associated with the + * context. */ #define DK_CXLFLASH_ATTACH_REUSE_CONTEXT 0x8000000000000000ULL