From patchwork Thu Oct 1 17:48:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812071 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9C8C6CB for ; Thu, 1 Oct 2020 17:48:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D92A20759 for ; Thu, 1 Oct 2020 17:48:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NwU7MAkY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732858AbgJARs5 (ORCPT ); Thu, 1 Oct 2020 13:48:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732830AbgJARs4 (ORCPT ); Thu, 1 Oct 2020 13:48:56 -0400 Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C2ACC0613D0 for ; Thu, 1 Oct 2020 10:48:56 -0700 (PDT) Received: by mail-ot1-x341.google.com with SMTP id u25so6306240otq.6 for ; Thu, 01 Oct 2020 10:48:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l0EuencCLWSruZarBPiteoNhedhanBrOUgx+fqPMcAU=; b=NwU7MAkY1oWtzV/6hTtWm+MBpuDb2yi06oEpkP4lQig4kWO0toMbiHvDHVdQvYNXH4 jhtn7h8YeVErnQ90IvKUzY9qT8ddJnkrTUx+pVPXSpN3JKosbc8GaHO+jEhozGjIdVOK R7zD4R9HMfwIaM6OSqxBFybSCZ5LeaqF1Ni3Fq+ft1Snjb6sKz+sMP9MHsJRAdttm+T4 kfDCC7OtIU05fdfCE9BGYpr7owPj8HvhFEfvBFiOuaHDMDLloaJks4olI38cgIwPPhMM BGvIHzdHtCuk9+U4QdqslfNVhk3JP8pQfb0bj5pEg8LwRWJl8k/F4lq1Pc9RbosavFyP csLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l0EuencCLWSruZarBPiteoNhedhanBrOUgx+fqPMcAU=; b=A0XiicGfHDG18buPiXsA/fGzlZreXwWvl+OvVonaEL+bL3IrkkA+JsjzBScKZ670CY 8hKvjGIa+fz80Vt90ULx2l82tW2tAjen4W9X3uNgOe3fV12Dk8V1HL/xxex068uW+UQB KWIrZxNk9zYRGHaUK7E0b4bfGN5Jit/nuQnsWszeAw+jxKRfdRw9tvKRR4tdgKHmOhkh UjjDwlxoMA/YMR6MiGuEHo6BpiTRpFA5TPDjmJqXumdiTEu+dQls+3oYz0AlcEk4h80e Vds4ZRCf4M6r1khfhduVAnv5b+td3beVv/YZBlR1SZPoMnlRgYLfBgDrUeEtXTBBUTmz qyKg== X-Gm-Message-State: AOAM533lABbkHy6egWhCejt+CrcoDOqRirfTvoaIbKar/BGZs6dLMMBO J5idaSvzrQGNHCj9aOPU8Is= X-Google-Smtp-Source: ABdhPJyj7jsTUv2IXgF7yWRQIE8F/FoJbjlJsC/v0SOLvFkqrPNc1mqq9/heaAwuxLEBZV0IICgJ8A== X-Received: by 2002:a9d:4e1a:: with SMTP id p26mr5950859otf.197.1601574535429; Thu, 01 Oct 2020 10:48:55 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id j18sm1366569otr.12.2020.10.01.10.48.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:48:55 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 01/19] rdma_rxe: Separat MEM into MR and MW objects. Date: Thu, 1 Oct 2020 12:48:29 -0500 Message-Id: <20201001174847.4268-2-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In the original rxe implementation it was intended to use a common object to represent MRs and MWs but it became clear that they are different enough to separate these into two objects. This allows replacing the mem name with mr for MRs which is more consistent with the style for the other objects and less likely to be confusing. This is a long patch that mostly changes mem to mr where it makes sense and adds a new rxe_mw struct. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 4 +- drivers/infiniband/sw/rxe/rxe_loc.h | 26 +-- drivers/infiniband/sw/rxe/rxe_mr.c | 264 +++++++++++++------------- drivers/infiniband/sw/rxe/rxe_pool.c | 8 +- drivers/infiniband/sw/rxe/rxe_req.c | 6 +- drivers/infiniband/sw/rxe/rxe_resp.c | 30 +-- drivers/infiniband/sw/rxe/rxe_verbs.c | 18 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 51 ++--- 8 files changed, 204 insertions(+), 203 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 0a1e6393250b..5dc86c9e74c2 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -345,7 +345,7 @@ static inline enum comp_state do_read(struct rxe_qp *qp, ret = copy_data(qp->pd, IB_ACCESS_LOCAL_WRITE, &wqe->dma, payload_addr(pkt), - payload_size(pkt), to_mem_obj, NULL); + payload_size(pkt), to_mr_obj, NULL); if (ret) return COMPST_ERROR; @@ -365,7 +365,7 @@ static inline enum comp_state do_atomic(struct rxe_qp *qp, ret = copy_data(qp->pd, IB_ACCESS_LOCAL_WRITE, &wqe->dma, &atomic_orig, - sizeof(u64), to_mem_obj, NULL); + sizeof(u64), to_mr_obj, NULL); if (ret) return COMPST_ERROR; else diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 0d758760b9ae..9ec6bff6863f 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -72,40 +72,40 @@ int rxe_mmap(struct ib_ucontext *context, struct vm_area_struct *vma); /* rxe_mr.c */ enum copy_direction { - to_mem_obj, - from_mem_obj, + to_mr_obj, + from_mr_obj, }; -void rxe_mem_init_dma(struct rxe_pd *pd, - int access, struct rxe_mem *mem); +void rxe_mr_init_dma(struct rxe_pd *pd, + int access, struct rxe_mr *mr); -int rxe_mem_init_user(struct rxe_pd *pd, u64 start, +int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova, int access, struct ib_udata *udata, - struct rxe_mem *mr); + struct rxe_mr *mr); -int rxe_mem_init_fast(struct rxe_pd *pd, - int max_pages, struct rxe_mem *mem); +int rxe_mr_init_fast(struct rxe_pd *pd, + int max_pages, struct rxe_mr *mr); -int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr, +int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, enum copy_direction dir, u32 *crcp); int copy_data(struct rxe_pd *pd, int access, struct rxe_dma_info *dma, void *addr, int length, enum copy_direction dir, u32 *crcp); -void *iova_to_vaddr(struct rxe_mem *mem, u64 iova, int length); +void *iova_to_vaddr(struct rxe_mr *mr, u64 iova, int length); enum lookup_type { lookup_local, lookup_remote, }; -struct rxe_mem *lookup_mem(struct rxe_pd *pd, int access, u32 key, +struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, enum lookup_type type); -int mem_check_range(struct rxe_mem *mem, u64 iova, size_t length); +int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length); -void rxe_mem_cleanup(struct rxe_pool_entry *arg); +void rxe_mr_cleanup(struct rxe_pool_entry *arg); int advance_dma_data(struct rxe_dma_info *dma, unsigned int length); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 47f737736961..4baa2f498e9d 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -24,17 +24,17 @@ static u8 rxe_get_key(void) return key; } -int mem_check_range(struct rxe_mem *mem, u64 iova, size_t length) +int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) { - switch (mem->type) { - case RXE_MEM_TYPE_DMA: + switch (mr->type) { + case RXE_MR_TYPE_DMA: return 0; - case RXE_MEM_TYPE_MR: - case RXE_MEM_TYPE_FMR: - if (iova < mem->iova || - length > mem->length || - iova > mem->iova + mem->length - length) + case RXE_MR_TYPE_MR: + case RXE_MR_TYPE_FMR: + if (iova < mr->iova || + length > mr->length || + iova > mr->iova + mr->length - length) return -EFAULT; return 0; @@ -47,90 +47,90 @@ int mem_check_range(struct rxe_mem *mem, u64 iova, size_t length) | IB_ACCESS_REMOTE_WRITE \ | IB_ACCESS_REMOTE_ATOMIC) -static void rxe_mem_init(int access, struct rxe_mem *mem) +static void rxe_mr_init(int access, struct rxe_mr *mr) { - u32 lkey = mem->pelem.index << 8 | rxe_get_key(); + u32 lkey = mr->pelem.index << 8 | rxe_get_key(); u32 rkey = (access & IB_ACCESS_REMOTE) ? lkey : 0; - if (mem->pelem.pool->type == RXE_TYPE_MR) { - mem->ibmr.lkey = lkey; - mem->ibmr.rkey = rkey; + if (mr->pelem.pool->type == RXE_TYPE_MR) { + mr->ibmr.lkey = lkey; + mr->ibmr.rkey = rkey; } - mem->lkey = lkey; - mem->rkey = rkey; - mem->state = RXE_MEM_STATE_INVALID; - mem->type = RXE_MEM_TYPE_NONE; - mem->map_shift = ilog2(RXE_BUF_PER_MAP); + mr->lkey = lkey; + mr->rkey = rkey; + mr->state = RXE_MEM_STATE_INVALID; + mr->type = RXE_MR_TYPE_NONE; + mr->map_shift = ilog2(RXE_BUF_PER_MAP); } -void rxe_mem_cleanup(struct rxe_pool_entry *arg) +void rxe_mr_cleanup(struct rxe_pool_entry *arg) { - struct rxe_mem *mem = container_of(arg, typeof(*mem), pelem); + struct rxe_mr *mr = container_of(arg, typeof(*mr), pelem); int i; - ib_umem_release(mem->umem); + ib_umem_release(mr->umem); - if (mem->map) { - for (i = 0; i < mem->num_map; i++) - kfree(mem->map[i]); + if (mr->map) { + for (i = 0; i < mr->num_map; i++) + kfree(mr->map[i]); - kfree(mem->map); + kfree(mr->map); } } -static int rxe_mem_alloc(struct rxe_mem *mem, int num_buf) +static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf) { int i; int num_map; - struct rxe_map **map = mem->map; + struct rxe_map **map = mr->map; num_map = (num_buf + RXE_BUF_PER_MAP - 1) / RXE_BUF_PER_MAP; - mem->map = kmalloc_array(num_map, sizeof(*map), GFP_KERNEL); - if (!mem->map) + mr->map = kmalloc_array(num_map, sizeof(*map), GFP_KERNEL); + if (!mr->map) goto err1; for (i = 0; i < num_map; i++) { - mem->map[i] = kmalloc(sizeof(**map), GFP_KERNEL); - if (!mem->map[i]) + mr->map[i] = kmalloc(sizeof(**map), GFP_KERNEL); + if (!mr->map[i]) goto err2; } BUILD_BUG_ON(!is_power_of_2(RXE_BUF_PER_MAP)); - mem->map_shift = ilog2(RXE_BUF_PER_MAP); - mem->map_mask = RXE_BUF_PER_MAP - 1; + mr->map_shift = ilog2(RXE_BUF_PER_MAP); + mr->map_mask = RXE_BUF_PER_MAP - 1; - mem->num_buf = num_buf; - mem->num_map = num_map; - mem->max_buf = num_map * RXE_BUF_PER_MAP; + mr->num_buf = num_buf; + mr->num_map = num_map; + mr->max_buf = num_map * RXE_BUF_PER_MAP; return 0; err2: for (i--; i >= 0; i--) - kfree(mem->map[i]); + kfree(mr->map[i]); - kfree(mem->map); + kfree(mr->map); err1: return -ENOMEM; } -void rxe_mem_init_dma(struct rxe_pd *pd, - int access, struct rxe_mem *mem) +void rxe_mr_init_dma(struct rxe_pd *pd, + int access, struct rxe_mr *mr) { - rxe_mem_init(access, mem); + rxe_mr_init(access, mr); - mem->pd = pd; - mem->access = access; - mem->state = RXE_MEM_STATE_VALID; - mem->type = RXE_MEM_TYPE_DMA; + mr->pd = pd; + mr->access = access; + mr->state = RXE_MEM_STATE_VALID; + mr->type = RXE_MR_TYPE_DMA; } -int rxe_mem_init_user(struct rxe_pd *pd, u64 start, +int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova, int access, struct ib_udata *udata, - struct rxe_mem *mem) + struct rxe_mr *mr) { struct rxe_map **map; struct rxe_phys_buf *buf = NULL; @@ -148,23 +148,23 @@ int rxe_mem_init_user(struct rxe_pd *pd, u64 start, goto err1; } - mem->umem = umem; + mr->umem = umem; num_buf = ib_umem_num_pages(umem); - rxe_mem_init(access, mem); + rxe_mr_init(access, mr); - err = rxe_mem_alloc(mem, num_buf); + err = rxe_mr_alloc(mr, num_buf); if (err) { - pr_warn("err %d from rxe_mem_alloc\n", err); + pr_warn("err %d from rxe_mr_alloc\n", err); ib_umem_release(umem); goto err1; } - mem->page_shift = PAGE_SHIFT; - mem->page_mask = PAGE_SIZE - 1; + mr->page_shift = PAGE_SHIFT; + mr->page_mask = PAGE_SIZE - 1; num_buf = 0; - map = mem->map; + map = mr->map; if (length > 0) { buf = map[0]->buf; @@ -191,15 +191,15 @@ int rxe_mem_init_user(struct rxe_pd *pd, u64 start, } } - mem->pd = pd; - mem->umem = umem; - mem->access = access; - mem->length = length; - mem->iova = iova; - mem->va = start; - mem->offset = ib_umem_offset(umem); - mem->state = RXE_MEM_STATE_VALID; - mem->type = RXE_MEM_TYPE_MR; + mr->pd = pd; + mr->umem = umem; + mr->access = access; + mr->length = length; + mr->iova = iova; + mr->va = start; + mr->offset = ib_umem_offset(umem); + mr->state = RXE_MEM_STATE_VALID; + mr->type = RXE_MR_TYPE_MR; return 0; @@ -207,24 +207,24 @@ int rxe_mem_init_user(struct rxe_pd *pd, u64 start, return err; } -int rxe_mem_init_fast(struct rxe_pd *pd, - int max_pages, struct rxe_mem *mem) +int rxe_mr_init_fast(struct rxe_pd *pd, + int max_pages, struct rxe_mr *mr) { int err; - rxe_mem_init(0, mem); + rxe_mr_init(0, mr); /* In fastreg, we also set the rkey */ - mem->ibmr.rkey = mem->ibmr.lkey; + mr->ibmr.rkey = mr->ibmr.lkey; - err = rxe_mem_alloc(mem, max_pages); + err = rxe_mr_alloc(mr, max_pages); if (err) goto err1; - mem->pd = pd; - mem->max_buf = max_pages; - mem->state = RXE_MEM_STATE_FREE; - mem->type = RXE_MEM_TYPE_MR; + mr->pd = pd; + mr->max_buf = max_pages; + mr->state = RXE_MEM_STATE_FREE; + mr->type = RXE_MR_TYPE_MR; return 0; @@ -233,27 +233,27 @@ int rxe_mem_init_fast(struct rxe_pd *pd, } static void lookup_iova( - struct rxe_mem *mem, + struct rxe_mr *mr, u64 iova, int *m_out, int *n_out, size_t *offset_out) { - size_t offset = iova - mem->iova + mem->offset; + size_t offset = iova - mr->iova + mr->offset; int map_index; int buf_index; u64 length; - if (likely(mem->page_shift)) { - *offset_out = offset & mem->page_mask; - offset >>= mem->page_shift; - *n_out = offset & mem->map_mask; - *m_out = offset >> mem->map_shift; + if (likely(mr->page_shift)) { + *offset_out = offset & mr->page_mask; + offset >>= mr->page_shift; + *n_out = offset & mr->map_mask; + *m_out = offset >> mr->map_shift; } else { map_index = 0; buf_index = 0; - length = mem->map[map_index]->buf[buf_index].size; + length = mr->map[map_index]->buf[buf_index].size; while (offset >= length) { offset -= length; @@ -263,7 +263,7 @@ static void lookup_iova( map_index++; buf_index = 0; } - length = mem->map[map_index]->buf[buf_index].size; + length = mr->map[map_index]->buf[buf_index].size; } *m_out = map_index; @@ -272,48 +272,48 @@ static void lookup_iova( } } -void *iova_to_vaddr(struct rxe_mem *mem, u64 iova, int length) +void *iova_to_vaddr(struct rxe_mr *mr, u64 iova, int length) { size_t offset; int m, n; void *addr; - if (mem->state != RXE_MEM_STATE_VALID) { - pr_warn("mem not in valid state\n"); + if (mr->state != RXE_MEM_STATE_VALID) { + pr_warn("mr not in valid state\n"); addr = NULL; goto out; } - if (!mem->map) { + if (!mr->map) { addr = (void *)(uintptr_t)iova; goto out; } - if (mem_check_range(mem, iova, length)) { + if (mr_check_range(mr, iova, length)) { pr_warn("range violation\n"); addr = NULL; goto out; } - lookup_iova(mem, iova, &m, &n, &offset); + lookup_iova(mr, iova, &m, &n, &offset); - if (offset + length > mem->map[m]->buf[n].size) { + if (offset + length > mr->map[m]->buf[n].size) { pr_warn("crosses page boundary\n"); addr = NULL; goto out; } - addr = (void *)(uintptr_t)mem->map[m]->buf[n].addr + offset; + addr = (void *)(uintptr_t)mr->map[m]->buf[n].addr + offset; out: return addr; } /* copy data from a range (vaddr, vaddr+length-1) to or from - * a mem object starting at iova. Compute incremental value of - * crc32 if crcp is not zero. caller must hold a reference to mem + * a mr object starting at iova. Compute incremental value of + * crc32 if crcp is not zero. caller must hold a reference to mr */ -int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr, int length, +int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, enum copy_direction dir, u32 *crcp) { int err; @@ -329,43 +329,43 @@ int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr, int length, if (length == 0) return 0; - if (mem->type == RXE_MEM_TYPE_DMA) { + if (mr->type == RXE_MR_TYPE_DMA) { u8 *src, *dest; - src = (dir == to_mem_obj) ? + src = (dir == to_mr_obj) ? addr : ((void *)(uintptr_t)iova); - dest = (dir == to_mem_obj) ? + dest = (dir == to_mr_obj) ? ((void *)(uintptr_t)iova) : addr; memcpy(dest, src, length); if (crcp) - *crcp = rxe_crc32(to_rdev(mem->pd->ibpd.device), + *crcp = rxe_crc32(to_rdev(mr->pd->ibpd.device), *crcp, dest, length); return 0; } - WARN_ON_ONCE(!mem->map); + WARN_ON_ONCE(!mr->map); - err = mem_check_range(mem, iova, length); + err = mr_check_range(mr, iova, length); if (err) { err = -EFAULT; goto err1; } - lookup_iova(mem, iova, &m, &i, &offset); + lookup_iova(mr, iova, &m, &i, &offset); - map = mem->map + m; + map = mr->map + m; buf = map[0]->buf + i; while (length > 0) { u8 *src, *dest; va = (u8 *)(uintptr_t)buf->addr + offset; - src = (dir == to_mem_obj) ? addr : va; - dest = (dir == to_mem_obj) ? va : addr; + src = (dir == to_mr_obj) ? addr : va; + dest = (dir == to_mr_obj) ? va : addr; bytes = buf->size - offset; @@ -375,7 +375,7 @@ int rxe_mem_copy(struct rxe_mem *mem, u64 iova, void *addr, int length, memcpy(dest, src, bytes); if (crcp) - crc = rxe_crc32(to_rdev(mem->pd->ibpd.device), + crc = rxe_crc32(to_rdev(mr->pd->ibpd.device), crc, dest, bytes); length -= bytes; @@ -417,7 +417,7 @@ int copy_data( struct rxe_sge *sge = &dma->sge[dma->cur_sge]; int offset = dma->sge_offset; int resid = dma->resid; - struct rxe_mem *mem = NULL; + struct rxe_mr *mr = NULL; u64 iova; int err; @@ -430,8 +430,8 @@ int copy_data( } if (sge->length && (offset < sge->length)) { - mem = lookup_mem(pd, access, sge->lkey, lookup_local); - if (!mem) { + mr = lookup_mr(pd, access, sge->lkey, lookup_local); + if (!mr) { err = -EINVAL; goto err1; } @@ -441,9 +441,9 @@ int copy_data( bytes = length; if (offset >= sge->length) { - if (mem) { - rxe_drop_ref(mem); - mem = NULL; + if (mr) { + rxe_drop_ref(mr); + mr = NULL; } sge++; dma->cur_sge++; @@ -455,9 +455,9 @@ int copy_data( } if (sge->length) { - mem = lookup_mem(pd, access, sge->lkey, + mr = lookup_mr(pd, access, sge->lkey, lookup_local); - if (!mem) { + if (!mr) { err = -EINVAL; goto err1; } @@ -472,7 +472,7 @@ int copy_data( if (bytes > 0) { iova = sge->addr + offset; - err = rxe_mem_copy(mem, iova, addr, bytes, dir, crcp); + err = rxe_mr_copy(mr, iova, addr, bytes, dir, crcp); if (err) goto err2; @@ -486,14 +486,14 @@ int copy_data( dma->sge_offset = offset; dma->resid = resid; - if (mem) - rxe_drop_ref(mem); + if (mr) + rxe_drop_ref(mr); return 0; err2: - if (mem) - rxe_drop_ref(mem); + if (mr) + rxe_drop_ref(mr); err1: return err; } @@ -531,31 +531,31 @@ int advance_dma_data(struct rxe_dma_info *dma, unsigned int length) return 0; } -/* (1) find the mem (mr or mw) corresponding to lkey/rkey +/* (1) find the mr corresponding to lkey/rkey * depending on lookup_type - * (2) verify that the (qp) pd matches the mem pd - * (3) verify that the mem can support the requested access - * (4) verify that mem state is valid + * (2) verify that the (qp) pd matches the mr pd + * (3) verify that the mr can support the requested access + * (4) verify that mr state is valid */ -struct rxe_mem *lookup_mem(struct rxe_pd *pd, int access, u32 key, +struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, enum lookup_type type) { - struct rxe_mem *mem; + struct rxe_mr *mr; struct rxe_dev *rxe = to_rdev(pd->ibpd.device); int index = key >> 8; - mem = rxe_pool_get_index(&rxe->mr_pool, index); - if (!mem) + mr = rxe_pool_get_index(&rxe->mr_pool, index); + if (!mr) return NULL; - if (unlikely((type == lookup_local && mem->lkey != key) || - (type == lookup_remote && mem->rkey != key) || - mem->pd != pd || - (access && !(access & mem->access)) || - mem->state != RXE_MEM_STATE_VALID)) { - rxe_drop_ref(mem); - mem = NULL; + if (unlikely((type == lookup_local && mr->lkey != key) || + (type == lookup_remote && mr->rkey != key) || + mr->pd != pd || + (access && !(access & mr->access)) || + mr->state != RXE_MEM_STATE_VALID)) { + rxe_drop_ref(mr); + mr = NULL; } - return mem; + return mr; } diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index b374eb53e2fe..32ba47d143f3 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -8,8 +8,6 @@ #include "rxe_loc.h" /* info about object pools - * note that mr and mw share a single index space - * so that one can map an lkey to the correct type of object */ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_UC] = { @@ -50,15 +48,15 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { }, [RXE_TYPE_MR] = { .name = "rxe-mr", - .size = sizeof(struct rxe_mem), - .cleanup = rxe_mem_cleanup, + .size = sizeof(struct rxe_mr), + .cleanup = rxe_mr_cleanup, .flags = RXE_POOL_INDEX, .max_index = RXE_MAX_MR_INDEX, .min_index = RXE_MIN_MR_INDEX, }, [RXE_TYPE_MW] = { .name = "rxe-mw", - .size = sizeof(struct rxe_mem), + .size = sizeof(struct rxe_mw), .flags = RXE_POOL_INDEX, .max_index = RXE_MAX_MW_INDEX, .min_index = RXE_MIN_MW_INDEX, diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index e27585ce9eb7..57236d8c2146 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -465,7 +465,7 @@ static int fill_packet(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } else { err = copy_data(qp->pd, 0, &wqe->dma, payload_addr(pkt), paylen, - from_mem_obj, + from_mr_obj, &crc); if (err) return err; @@ -597,7 +597,7 @@ int rxe_requester(void *arg) if (wqe->mask & WR_REG_MASK) { if (wqe->wr.opcode == IB_WR_LOCAL_INV) { struct rxe_dev *rxe = to_rdev(qp->ibqp.device); - struct rxe_mem *rmr; + struct rxe_mr *rmr; rmr = rxe_pool_get_index(&rxe->mr_pool, wqe->wr.ex.invalidate_rkey >> 8); @@ -613,7 +613,7 @@ int rxe_requester(void *arg) wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; } else if (wqe->wr.opcode == IB_WR_REG_MR) { - struct rxe_mem *rmr = to_rmr(wqe->wr.wr.reg.mr); + struct rxe_mr *rmr = to_rmr(wqe->wr.wr.reg.mr); rmr->state = RXE_MEM_STATE_VALID; rmr->access = wqe->wr.wr.reg.access; diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index c7e3b6a4af38..69867bf39cfb 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -390,7 +390,7 @@ static enum resp_states check_length(struct rxe_qp *qp, static enum resp_states check_rkey(struct rxe_qp *qp, struct rxe_pkt_info *pkt) { - struct rxe_mem *mem = NULL; + struct rxe_mr *mr = NULL; u64 va; u32 rkey; u32 resid; @@ -429,18 +429,18 @@ static enum resp_states check_rkey(struct rxe_qp *qp, resid = qp->resp.resid; pktlen = payload_size(pkt); - mem = lookup_mem(qp->pd, access, rkey, lookup_remote); - if (!mem) { + mr = lookup_mr(qp->pd, access, rkey, lookup_remote); + if (!mr) { state = RESPST_ERR_RKEY_VIOLATION; goto err; } - if (unlikely(mem->state == RXE_MEM_STATE_FREE)) { + if (unlikely(mr->state == RXE_MEM_STATE_FREE)) { state = RESPST_ERR_RKEY_VIOLATION; goto err; } - if (mem_check_range(mem, va, resid)) { + if (mr_check_range(mr, va, resid)) { state = RESPST_ERR_RKEY_VIOLATION; goto err; } @@ -468,12 +468,12 @@ static enum resp_states check_rkey(struct rxe_qp *qp, WARN_ON_ONCE(qp->resp.mr); - qp->resp.mr = mem; + qp->resp.mr = mr; return RESPST_EXECUTE; err: - if (mem) - rxe_drop_ref(mem); + if (mr) + rxe_drop_ref(mr); return state; } @@ -483,7 +483,7 @@ static enum resp_states send_data_in(struct rxe_qp *qp, void *data_addr, int err; err = copy_data(qp->pd, IB_ACCESS_LOCAL_WRITE, &qp->resp.wqe->dma, - data_addr, data_len, to_mem_obj, NULL); + data_addr, data_len, to_mr_obj, NULL); if (unlikely(err)) return (err == -ENOSPC) ? RESPST_ERR_LENGTH : RESPST_ERR_MALFORMED_WQE; @@ -498,8 +498,8 @@ static enum resp_states write_data_in(struct rxe_qp *qp, int err; int data_len = payload_size(pkt); - err = rxe_mem_copy(qp->resp.mr, qp->resp.va, payload_addr(pkt), - data_len, to_mem_obj, NULL); + err = rxe_mr_copy(qp->resp.mr, qp->resp.va, payload_addr(pkt), + data_len, to_mr_obj, NULL); if (err) { rc = RESPST_ERR_RKEY_VIOLATION; goto out; @@ -521,7 +521,7 @@ static enum resp_states process_atomic(struct rxe_qp *qp, u64 iova = atmeth_va(pkt); u64 *vaddr; enum resp_states ret; - struct rxe_mem *mr = qp->resp.mr; + struct rxe_mr *mr = qp->resp.mr; if (mr->state != RXE_MEM_STATE_VALID) { ret = RESPST_ERR_RKEY_VIOLATION; @@ -700,8 +700,8 @@ static enum resp_states read_reply(struct rxe_qp *qp, if (!skb) return RESPST_ERR_RNR; - err = rxe_mem_copy(res->read.mr, res->read.va, payload_addr(&ack_pkt), - payload, from_mem_obj, &icrc); + err = rxe_mr_copy(res->read.mr, res->read.va, payload_addr(&ack_pkt), + payload, from_mr_obj, &icrc); if (err) pr_err("Failed copying memory\n"); @@ -883,7 +883,7 @@ static enum resp_states do_complete(struct rxe_qp *qp, } if (pkt->mask & RXE_IETH_MASK) { - struct rxe_mem *rmr; + struct rxe_mr *rmr; wc->wc_flags |= IB_WC_WITH_INVALIDATE; wc->ex.invalidate_rkey = ieth_rkey(pkt); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index f368dc16281a..664bcb025aae 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -867,7 +867,7 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access) { struct rxe_dev *rxe = to_rdev(ibpd->device); struct rxe_pd *pd = to_rpd(ibpd); - struct rxe_mem *mr; + struct rxe_mr *mr; mr = rxe_alloc(&rxe->mr_pool); if (!mr) @@ -875,7 +875,7 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access) rxe_add_index(mr); rxe_add_ref(pd); - rxe_mem_init_dma(pd, access, mr); + rxe_mr_init_dma(pd, access, mr); return &mr->ibmr; } @@ -889,7 +889,7 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, int err; struct rxe_dev *rxe = to_rdev(ibpd->device); struct rxe_pd *pd = to_rpd(ibpd); - struct rxe_mem *mr; + struct rxe_mr *mr; mr = rxe_alloc(&rxe->mr_pool); if (!mr) { @@ -901,7 +901,7 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, rxe_add_ref(pd); - err = rxe_mem_init_user(pd, start, length, iova, + err = rxe_mr_init_user(pd, start, length, iova, access, udata, mr); if (err) goto err3; @@ -918,7 +918,7 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, static int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) { - struct rxe_mem *mr = to_rmr(ibmr); + struct rxe_mr *mr = to_rmr(ibmr); mr->state = RXE_MEM_STATE_ZOMBIE; rxe_drop_ref(mr->pd); @@ -932,7 +932,7 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, { struct rxe_dev *rxe = to_rdev(ibpd->device); struct rxe_pd *pd = to_rpd(ibpd); - struct rxe_mem *mr; + struct rxe_mr *mr; int err; if (mr_type != IB_MR_TYPE_MEM_REG) @@ -948,7 +948,7 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, rxe_add_ref(pd); - err = rxe_mem_init_fast(pd, max_num_sg, mr); + err = rxe_mr_init_fast(pd, max_num_sg, mr); if (err) goto err2; @@ -964,7 +964,7 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, static int rxe_set_page(struct ib_mr *ibmr, u64 addr) { - struct rxe_mem *mr = to_rmr(ibmr); + struct rxe_mr *mr = to_rmr(ibmr); struct rxe_map *map; struct rxe_phys_buf *buf; @@ -984,7 +984,7 @@ static int rxe_set_page(struct ib_mr *ibmr, u64 addr) static int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, unsigned int *sg_offset) { - struct rxe_mem *mr = to_rmr(ibmr); + struct rxe_mr *mr = to_rmr(ibmr); int n; mr->nbuf = 0; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 560a610bb0aa..dbc649c9c43f 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -39,7 +39,7 @@ struct rxe_ucontext { }; struct rxe_pd { - struct ib_pd ibpd; + struct ib_pd ibpd; struct rxe_pool_entry pelem; }; @@ -156,7 +156,7 @@ struct resp_res { struct sk_buff *skb; } atomic; struct { - struct rxe_mem *mr; + struct rxe_mr *mr; u64 va_org; u32 rkey; u32 length; @@ -183,7 +183,7 @@ struct rxe_resp_info { /* RDMA read / atomic only */ u64 va; - struct rxe_mem *mr; + struct rxe_mr *mr; u32 resid; u32 rkey; u32 length; @@ -269,31 +269,27 @@ enum rxe_mem_state { RXE_MEM_STATE_VALID, }; -enum rxe_mem_type { - RXE_MEM_TYPE_NONE, - RXE_MEM_TYPE_DMA, - RXE_MEM_TYPE_MR, - RXE_MEM_TYPE_FMR, - RXE_MEM_TYPE_MW, +enum rxe_mr_type { + RXE_MR_TYPE_NONE, + RXE_MR_TYPE_DMA, + RXE_MR_TYPE_MR, + RXE_MR_TYPE_FMR, }; #define RXE_BUF_PER_MAP (PAGE_SIZE / sizeof(struct rxe_phys_buf)) struct rxe_phys_buf { - u64 addr; - u64 size; + u64 addr; + u64 size; }; struct rxe_map { struct rxe_phys_buf buf[RXE_BUF_PER_MAP]; }; -struct rxe_mem { +struct rxe_mr { struct rxe_pool_entry pelem; - union { - struct ib_mr ibmr; - struct ib_mw ibmw; - }; + struct ib_mr ibmr; struct rxe_pd *pd; struct ib_umem *umem; @@ -302,7 +298,7 @@ struct rxe_mem { u32 rkey; enum rxe_mem_state state; - enum rxe_mem_type type; + enum rxe_mr_type type; u64 va; u64 iova; size_t length; @@ -323,6 +319,18 @@ struct rxe_mem { struct rxe_map **map; }; +struct rxe_mw { + struct rxe_pool_entry pelem; + struct ib_mw ibmw; + struct rxe_qp *qp; /* type 2B only */ + struct rxe_mr *mr; + spinlock_t lock; + enum rxe_mem_state state; + u32 access; + u64 addr; + u64 length; +}; + struct rxe_mc_grp { struct rxe_pool_entry pelem; spinlock_t mcg_lock; /* guard group */ @@ -428,14 +436,9 @@ static inline struct rxe_cq *to_rcq(struct ib_cq *cq) return cq ? container_of(cq, struct rxe_cq, ibcq) : NULL; } -static inline struct rxe_mem *to_rmr(struct ib_mr *mr) -{ - return mr ? container_of(mr, struct rxe_mem, ibmr) : NULL; -} - -static inline struct rxe_mem *to_rmw(struct ib_mw *mw) +static inline struct rxe_mr *to_rmr(struct ib_mr *mr) { - return mw ? container_of(mw, struct rxe_mem, ibmw) : NULL; + return mr ? container_of(mr, struct rxe_mr, ibmr) : NULL; } int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name); From patchwork Thu Oct 1 17:48:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812073 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15A0A92C for ; Thu, 1 Oct 2020 17:48:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA59820848 for ; Thu, 1 Oct 2020 17:48:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BNrKjzqp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732830AbgJARs5 (ORCPT ); Thu, 1 Oct 2020 13:48:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732096AbgJARs4 (ORCPT ); Thu, 1 Oct 2020 13:48:56 -0400 Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0C99C0613E2 for ; Thu, 1 Oct 2020 10:48:56 -0700 (PDT) Received: by mail-ot1-x344.google.com with SMTP id c2so6313917otp.7 for ; Thu, 01 Oct 2020 10:48:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Orjx+oyJBYrvIhnGLKRbdepxwiKwvSv5S446kYztnyc=; b=BNrKjzqpPDyMPPE7ORmhjBTVLEgb7YpI+n1iup3o3XvsUWn+jZgBHi2sD7ke2A1CyO cto7JIGeKXorL/Q6yFapOTegd7VClXkyhh9mhwztw6obllD1x32847yZGy2FGXd3gIB4 ivFmdIUxicdp6fP74Abge/cOdCx/lrbYzThd5oCeYIczMul8owZxTTsLpGgJKdN3fMff 2CCQOz5YjSyEgloToD8uDQ3B5OYdlh8RNwzeP3TGUKvbQUwUfEe7GUUbb1SwvQC9t8p0 LU//ebqPIN/0F6S5+NlqriuJt+jxvIwrDSQakEwW4aW68NnR/TzlWtAh9LOQLEGpRo4g hCqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Orjx+oyJBYrvIhnGLKRbdepxwiKwvSv5S446kYztnyc=; b=hcxONY9RMXJkYUx6xDmYP+O3t8yGBH0pKQJJQ/+qNA+5ZB6Q2B++8KpAgZFuIIb1hO f7dZA/qkYNQdzWsJ+w30GDenFnJjhHxUIkhGbmRLTfXATLGTXQUuDC7iwonxd4EpiYHq 9uHjHfii/dbs01tGWP/3hiWx/w80+duUwOZyg3fFZoHqri9M+GftcAoJSXfQ7LD1MCUN Qyh+YlIFj9JrCRQqSatB/e3spSfyMJi/G5JtHUeC0e1IsNjJWFqnAnFMJt5paQ3pcclb Gg2gb7tdo5DKREWMMqJ4b91AqfO+jCzcFZoN8h3aHXWD0+q1jPzJ7hIdpOtWXrfzFPgc JLfA== X-Gm-Message-State: AOAM53112Mvu/Fkywfx6S1zvRcMrBDofdSkQZ8Nu+Rx6vfU0IGzMNqBK uO9w7+PX5M+DUpZLEbI0iVQ= X-Google-Smtp-Source: ABdhPJzF+HofadERLIjk9fjoUSRfy+43ZW/LnwTC6Esm4F6Z8teF11ZJ8OsLXrCvg/70OHqSBeTVbQ== X-Received: by 2002:a9d:4b18:: with SMTP id q24mr5536304otf.265.1601574536186; Thu, 01 Oct 2020 10:48:56 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id p20sm1360015oth.48.2020.10.01.10.48.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:48:55 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 02/19] rdma_rxe: Enable MW objects Date: Thu, 1 Oct 2020 12:48:30 -0500 Message-Id: <20201001174847.4268-3-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Change parameters in rxe_param.h so that MAX_MW is the same as MAX_MR. Set device attribute in rxe.c so max_mw = MAX_MW. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_param.h | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 95f0de0c8b49..8e0f9c489cab 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -54,6 +54,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->attr.max_cq = RXE_MAX_CQ; rxe->attr.max_cqe = (1 << RXE_MAX_LOG_CQE) - 1; rxe->attr.max_mr = RXE_MAX_MR; + rxe->attr.max_mw = RXE_MAX_MW; rxe->attr.max_pd = RXE_MAX_PD; rxe->attr.max_qp_rd_atom = RXE_MAX_QP_RD_ATOM; rxe->attr.max_res_rd_atom = RXE_MAX_RES_RD_ATOM; diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index 25ab50d9b7c2..4ebb3da8c07d 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -58,7 +58,8 @@ enum rxe_device_param { RXE_MAX_SGE_RD = 32, RXE_MAX_CQ = 16384, RXE_MAX_LOG_CQE = 15, - RXE_MAX_MR = 256 * 1024, + RXE_MAX_MR = 0x40000, + RXE_MAX_MW = 0x40000, RXE_MAX_PD = 0x7ffc, RXE_MAX_QP_RD_ATOM = 128, RXE_MAX_RES_RD_ATOM = 0x3f000, @@ -87,9 +88,10 @@ enum rxe_device_param { RXE_MAX_SRQ_INDEX = 0x00040000, RXE_MIN_MR_INDEX = 0x00000001, - RXE_MAX_MR_INDEX = 0x00040000, - RXE_MIN_MW_INDEX = 0x00040001, - RXE_MAX_MW_INDEX = 0x00060000, + RXE_MAX_MR_INDEX = RXE_MIN_MR_INDEX + RXE_MAX_MR - 1, + RXE_MIN_MW_INDEX = RXE_MIN_MR_INDEX + RXE_MAX_MR, + RXE_MAX_MW_INDEX = RXE_MIN_MW_INDEX + RXE_MAX_MW - 1, + RXE_MAX_PKT_PER_ACK = 64, RXE_MAX_UNACKED_PSNS = 128, From patchwork Thu Oct 1 17:48:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812075 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DC2B892C for ; Thu, 1 Oct 2020 17:48:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B2EE220848 for ; Thu, 1 Oct 2020 17:48:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="n/2+d46+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732949AbgJARs6 (ORCPT ); Thu, 1 Oct 2020 13:48:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732096AbgJARs5 (ORCPT ); Thu, 1 Oct 2020 13:48:57 -0400 Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 845A9C0613D0 for ; Thu, 1 Oct 2020 10:48:57 -0700 (PDT) Received: by mail-ot1-x342.google.com with SMTP id m12so6352507otr.0 for ; Thu, 01 Oct 2020 10:48:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J1TZIs7znNhkB5dcnBH7CahLIv96Wzdg/cqZF3s4VXM=; b=n/2+d46+iEMUko3OdXq0H+0kt1mix28YE/PGl9oYJYK42ukIIYmE2aryFwDAPani5m njOcXoTGd7g2AduqJX3fb8iJIqarRSGwWxxPXsXxxiiQTr1t7hn2Ukd3OWc856n198Kn 1xR7LMypnScj7X+h+RxFymdzQ+8zib0PVdVQpvdJIaPELlNehHZ6uQwZh8edFvSDrzph qI81vKn+kDWjuFO1ianq4rF8JxId4fcUTqfAaeQ10ArioFEHVsJ7HWIzjHZpK1h/l+bz SLwHcaxCM3H1dt3U7K8SSkqe+qA9zzYSfZV54Hh3q0w2lTNdqrd90ArOF0K2SrFbZufv YQWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J1TZIs7znNhkB5dcnBH7CahLIv96Wzdg/cqZF3s4VXM=; b=jadCPnAIymgfMXApOp+vwwuSjJ+usQDwljEA9P53oQiyL/lBOz/j/tcKPFRmgtlrcG A5jSN5rC9KCr4GJb8lKFbNpCyq9ELc6Fij8/pIw5GWMdVNRMDSAVsZQW629h0RmFE2CZ k+XtIoU1QNlALBFKOZc1d2VHKOkInFX91sCCTTunlL7xQZsgqEPeSQkM9wZlxMzKcYC4 k/kBrTd+CEaO2zFQxdG0QgNdEyLX91IcG4gmHdhNXShZoudQnTTKjJSutZI5ezOaVDc4 ew/HjWEelzM6W5J5/YYYfmmuY2Kl04zsTnju1H9CpEbdkYi0+tmPcKIzF6tjicNRogV2 BULw== X-Gm-Message-State: AOAM531ihWDpNI8HMtt82GbvCD0j5y4pJs6/W2c2WIEhLQZRZZiEtR3Q +rpJP85Ty7l30y4qn6Erdy0= X-Google-Smtp-Source: ABdhPJxuupXIz7a0D/RS/JTOiYOd2lJYE9FgFf/KlUnvgPtyQJ9jnDDXiXk9x5Y5MQE8wfriy7F+KQ== X-Received: by 2002:a9d:6950:: with SMTP id p16mr5562728oto.60.1601574536909; Thu, 01 Oct 2020 10:48:56 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id k10sm1370305ots.60.2020.10.01.10.48.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:48:56 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 03/19] rdma_rxe: Let pools support both keys and indices Date: Thu, 1 Oct 2020 12:48:31 -0500 Message-Id: <20201001174847.4268-4-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Allow both indices and keys to exist for objects in pools. Previously you were limited to one or the other. This will support allowing the keys on MWs to change. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_pool.c | 73 ++++++++++++++-------------- drivers/infiniband/sw/rxe/rxe_pool.h | 32 +++++++----- 2 files changed, 58 insertions(+), 47 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 32ba47d143f3..30b8f037ee20 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -92,18 +92,18 @@ static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) goto out; } - pool->max_index = max; - pool->min_index = min; + pool->index.max_index = max; + pool->index.min_index = min; size = BITS_TO_LONGS(max - min + 1) * sizeof(long); - pool->table = kmalloc(size, GFP_KERNEL); - if (!pool->table) { + pool->index.table = kmalloc(size, GFP_KERNEL); + if (!pool->index.table) { err = -ENOMEM; goto out; } - pool->table_size = size; - bitmap_zero(pool->table, max - min + 1); + pool->index.table_size = size; + bitmap_zero(pool->index.table, max - min + 1); out: return err; @@ -125,7 +125,8 @@ int rxe_pool_init( pool->max_elem = max_elem; pool->elem_size = ALIGN(size, RXE_POOL_ALIGN); pool->flags = rxe_type_info[type].flags; - pool->tree = RB_ROOT; + pool->index.tree = RB_ROOT; + pool->key.tree = RB_ROOT; pool->cleanup = rxe_type_info[type].cleanup; atomic_set(&pool->num_elem, 0); @@ -143,8 +144,8 @@ int rxe_pool_init( } if (rxe_type_info[type].flags & RXE_POOL_KEY) { - pool->key_offset = rxe_type_info[type].key_offset; - pool->key_size = rxe_type_info[type].key_size; + pool->key.key_offset = rxe_type_info[type].key_offset; + pool->key.key_size = rxe_type_info[type].key_size; } pool->state = RXE_POOL_STATE_VALID; @@ -158,7 +159,7 @@ static void rxe_pool_release(struct kref *kref) struct rxe_pool *pool = container_of(kref, struct rxe_pool, ref_cnt); pool->state = RXE_POOL_STATE_INVALID; - kfree(pool->table); + kfree(pool->index.table); } static void rxe_pool_put(struct rxe_pool *pool) @@ -183,27 +184,27 @@ void rxe_pool_cleanup(struct rxe_pool *pool) static u32 alloc_index(struct rxe_pool *pool) { u32 index; - u32 range = pool->max_index - pool->min_index + 1; + u32 range = pool->index.max_index - pool->index.min_index + 1; - index = find_next_zero_bit(pool->table, range, pool->last); + index = find_next_zero_bit(pool->index.table, range, pool->index.last); if (index >= range) - index = find_first_zero_bit(pool->table, range); + index = find_first_zero_bit(pool->index.table, range); WARN_ON_ONCE(index >= range); - set_bit(index, pool->table); - pool->last = index; - return index + pool->min_index; + set_bit(index, pool->index.table); + pool->index.last = index; + return index + pool->index.min_index; } static void insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) { - struct rb_node **link = &pool->tree.rb_node; + struct rb_node **link = &pool->index.tree.rb_node; struct rb_node *parent = NULL; struct rxe_pool_entry *elem; while (*link) { parent = *link; - elem = rb_entry(parent, struct rxe_pool_entry, node); + elem = rb_entry(parent, struct rxe_pool_entry, index_node); if (elem->index == new->index) { pr_warn("element already exists!\n"); @@ -216,25 +217,25 @@ static void insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) link = &(*link)->rb_right; } - rb_link_node(&new->node, parent, link); - rb_insert_color(&new->node, &pool->tree); + rb_link_node(&new->index_node, parent, link); + rb_insert_color(&new->index_node, &pool->index.tree); out: return; } static void insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) { - struct rb_node **link = &pool->tree.rb_node; + struct rb_node **link = &pool->key.tree.rb_node; struct rb_node *parent = NULL; struct rxe_pool_entry *elem; int cmp; while (*link) { parent = *link; - elem = rb_entry(parent, struct rxe_pool_entry, node); + elem = rb_entry(parent, struct rxe_pool_entry, key_node); - cmp = memcmp((u8 *)elem + pool->key_offset, - (u8 *)new + pool->key_offset, pool->key_size); + cmp = memcmp((u8 *)elem + pool->key.key_offset, + (u8 *)new + pool->key.key_offset, pool->key.key_size); if (cmp == 0) { pr_warn("key already exists!\n"); @@ -247,8 +248,8 @@ static void insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) link = &(*link)->rb_right; } - rb_link_node(&new->node, parent, link); - rb_insert_color(&new->node, &pool->tree); + rb_link_node(&new->key_node, parent, link); + rb_insert_color(&new->key_node, &pool->key.tree); out: return; } @@ -260,7 +261,7 @@ void rxe_add_key(void *arg, void *key) unsigned long flags; write_lock_irqsave(&pool->pool_lock, flags); - memcpy((u8 *)elem + pool->key_offset, key, pool->key_size); + memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); insert_key(pool, elem); write_unlock_irqrestore(&pool->pool_lock, flags); } @@ -272,7 +273,7 @@ void rxe_drop_key(void *arg) unsigned long flags; write_lock_irqsave(&pool->pool_lock, flags); - rb_erase(&elem->node, &pool->tree); + rb_erase(&elem->key_node, &pool->key.tree); write_unlock_irqrestore(&pool->pool_lock, flags); } @@ -295,8 +296,8 @@ void rxe_drop_index(void *arg) unsigned long flags; write_lock_irqsave(&pool->pool_lock, flags); - clear_bit(elem->index - pool->min_index, pool->table); - rb_erase(&elem->node, &pool->tree); + clear_bit(elem->index - pool->index.min_index, pool->index.table); + rb_erase(&elem->index_node, &pool->index.tree); write_unlock_irqrestore(&pool->pool_lock, flags); } @@ -400,10 +401,10 @@ void *rxe_pool_get_index(struct rxe_pool *pool, u32 index) if (pool->state != RXE_POOL_STATE_VALID) goto out; - node = pool->tree.rb_node; + node = pool->index.tree.rb_node; while (node) { - elem = rb_entry(node, struct rxe_pool_entry, node); + elem = rb_entry(node, struct rxe_pool_entry, index_node); if (elem->index > index) node = node->rb_left; @@ -432,13 +433,13 @@ void *rxe_pool_get_key(struct rxe_pool *pool, void *key) if (pool->state != RXE_POOL_STATE_VALID) goto out; - node = pool->tree.rb_node; + node = pool->key.tree.rb_node; while (node) { - elem = rb_entry(node, struct rxe_pool_entry, node); + elem = rb_entry(node, struct rxe_pool_entry, key_node); - cmp = memcmp((u8 *)elem + pool->key_offset, - key, pool->key_size); + cmp = memcmp((u8 *)elem + pool->key.key_offset, + key, pool->key.key_size); if (cmp > 0) node = node->rb_left; diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 432745ffc8d4..3d722aae5f15 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -56,8 +56,11 @@ struct rxe_pool_entry { struct kref ref_cnt; struct list_head list; - /* only used if indexed or keyed */ - struct rb_node node; + /* only used if keyed */ + struct rb_node key_node; + + /* only used if indexed */ + struct rb_node index_node; u32 index; }; @@ -74,15 +77,22 @@ struct rxe_pool { unsigned int max_elem; atomic_t num_elem; - /* only used if indexed or keyed */ - struct rb_root tree; - unsigned long *table; - size_t table_size; - u32 max_index; - u32 min_index; - u32 last; - size_t key_offset; - size_t key_size; + /* only used if indexed */ + struct { + struct rb_root tree; + unsigned long *table; + size_t table_size; + u32 last; + u32 max_index; + u32 min_index; + } index; + + /* only used if keyed */ + struct { + struct rb_root tree; + size_t key_offset; + size_t key_size; + } key; }; /* initialize a pool of objects with given limit on From patchwork Thu Oct 1 17:48:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812077 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7DFD96CB for ; Thu, 1 Oct 2020 17:49:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 542DB20848 for ; Thu, 1 Oct 2020 17:49:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="W0ZMU6oQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732967AbgJARtB (ORCPT ); Thu, 1 Oct 2020 13:49:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732961AbgJARs7 (ORCPT ); Thu, 1 Oct 2020 13:48:59 -0400 Received: from mail-ot1-x344.google.com (mail-ot1-x344.google.com [IPv6:2607:f8b0:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6978CC0613E2 for ; Thu, 1 Oct 2020 10:48:58 -0700 (PDT) Received: by mail-ot1-x344.google.com with SMTP id s66so6335780otb.2 for ; Thu, 01 Oct 2020 10:48:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3j4+tNhIrc4tYr+5hwVg3HjI7QSO00fdfiuuSJP/fpo=; b=W0ZMU6oQDhRhxPTw2c+d/s+NbWZ/UUOd/axLSP+v6L3eet6W2NzKw20YTb+C3TwaZ5 /jb0sNcesaBMJEw1lHu5pU5BPWeVE1PD7b6lTmd5f70ThRUh4ra+8uHMJ78T35yKYXMB 1MQsFV/rT5Lfti/IVqeM6p6jX2QLc2UNdNEsyf59jQ/s7HIpUslaQrmA2SHWnElYTZ52 ozWy6HCegcBMECDbRsDK851s2h/YKp5uynhs1ldfkdW/0W1Ou56G/EeLVZZRRlt6ZhZ0 MjiLeQLwRprhpKz+VMjHoDNreysWQNlNPndJrFwzaoRunXEcpyyAjWjgXfsgnbGuLNBj 4KmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3j4+tNhIrc4tYr+5hwVg3HjI7QSO00fdfiuuSJP/fpo=; b=W5ROOWmaUCllQhYYnjNkOcG2XGgV1W9ePVhKDVDw25pN0NyuIOCWE8RUonO4Z9UUzh H7OSuH2ofEtH8Y77MAnvefRp2ohIDiTZpqdO6T1Qj4fOPKEwTQ8Cxsy12v94B3SZgY5m lkuPwRrMgwfcz+1ENeM43zCpvp4AxUFOeVPxixtC2e34sHYXaMy1ugTwaJgbpJg4ZDxg 3NFpFFDGrfvVidyTbeKLDDM8jUu1EL6d7wED/Pha0AXlhqC/0J1zYicOXa46T6DM9PJh dIgIX2FSEt/JaKN63R60gvAJ+KQnHIJqS2PwO+HBvujlcESm6HtjtmiQjSYbshj+f5bs xuZQ== X-Gm-Message-State: AOAM531zoLBPrqUrbnqyvge2zZ6ZBVlVv65iI2nogXEb0Yxsp8GzIOv5 hjDGDtdLp3G4EE8wBvknSSs= X-Google-Smtp-Source: ABdhPJyU8LlkM8xYsYkU0QpVslYaPogmQ/pdEnWdm/JYQRpGz0cduMExzSi8A8Aey5Gw1PDe2pd3Ew== X-Received: by 2002:a9d:6219:: with SMTP id g25mr5807053otj.58.1601574537755; Thu, 01 Oct 2020 10:48:57 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id c124sm1188467oib.22.2020.10.01.10.48.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:48:57 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 04/19] rdma_rxe: make pool return values position independent Date: Thu, 1 Oct 2020 12:48:32 -0500 Message-Id: <20201001174847.4268-5-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org rxe pool manages objects which are based on a base type of rxe_pool_entry. Once apon a time these were always the first entry in the rxe object structs. Now the pool entries (usually named elem or pelem) occur in more than one position. In order to make the return values from lookup and allocate routines work independently of the position of the pelem the offset of pelem is added to the rxe_type_info struct and used to compute the return value from these routines. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mcast.c | 4 +-- drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++-- drivers/infiniband/sw/rxe/rxe_pool.c | 40 +++++++++++++++++++++------ drivers/infiniband/sw/rxe/rxe_pool.h | 5 ++-- drivers/infiniband/sw/rxe/rxe_recv.c | 6 ++-- drivers/infiniband/sw/rxe/rxe_req.c | 2 +- drivers/infiniband/sw/rxe/rxe_resp.c | 2 +- 7 files changed, 44 insertions(+), 21 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c index c02315aed8d1..b09c6594045a 100644 --- a/drivers/infiniband/sw/rxe/rxe_mcast.c +++ b/drivers/infiniband/sw/rxe/rxe_mcast.c @@ -18,7 +18,7 @@ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, goto err1; } - grp = rxe_pool_get_key(&rxe->mc_grp_pool, mgid); + grp = rxe_get_key(&rxe->mc_grp_pool, mgid); if (grp) goto done; @@ -98,7 +98,7 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, struct rxe_mc_grp *grp; struct rxe_mc_elem *elem, *tmp; - grp = rxe_pool_get_key(&rxe->mc_grp_pool, mgid); + grp = rxe_get_key(&rxe->mc_grp_pool, mgid); if (!grp) goto err1; diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 4baa2f498e9d..f40cf4df394f 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -10,7 +10,7 @@ /* * lfsr (linear feedback shift register) with period 255 */ -static u8 rxe_get_key(void) +static u8 get_key(void) { static u32 key = 1; @@ -49,7 +49,7 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) static void rxe_mr_init(int access, struct rxe_mr *mr) { - u32 lkey = mr->pelem.index << 8 | rxe_get_key(); + u32 lkey = mr->pelem.index << 8 | get_key(); u32 rkey = (access & IB_ACCESS_REMOTE) ? lkey : 0; if (mr->pelem.pool->type == RXE_TYPE_MR) { @@ -544,7 +544,7 @@ struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, struct rxe_dev *rxe = to_rdev(pd->ibpd.device); int index = key >> 8; - mr = rxe_pool_get_index(&rxe->mr_pool, index); + mr = rxe_get_index(&rxe->mr_pool, index); if (!mr) return NULL; diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 30b8f037ee20..c4f1318ea761 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -13,21 +13,25 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_UC] = { .name = "rxe-uc", .size = sizeof(struct rxe_ucontext), + .elem_offset = offsetof(struct rxe_ucontext, pelem), .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_PD] = { .name = "rxe-pd", .size = sizeof(struct rxe_pd), + .elem_offset = offsetof(struct rxe_pd, pelem), .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_AH] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), + .elem_offset = offsetof(struct rxe_ah, pelem), .flags = RXE_POOL_ATOMIC | RXE_POOL_NO_ALLOC, }, [RXE_TYPE_SRQ] = { .name = "rxe-srq", .size = sizeof(struct rxe_srq), + .elem_offset = offsetof(struct rxe_srq, pelem), .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, .min_index = RXE_MIN_SRQ_INDEX, .max_index = RXE_MAX_SRQ_INDEX, @@ -35,6 +39,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_QP] = { .name = "rxe-qp", .size = sizeof(struct rxe_qp), + .elem_offset = offsetof(struct rxe_qp, pelem), .cleanup = rxe_qp_cleanup, .flags = RXE_POOL_INDEX, .min_index = RXE_MIN_QP_INDEX, @@ -43,12 +48,14 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_CQ] = { .name = "rxe-cq", .size = sizeof(struct rxe_cq), + .elem_offset = offsetof(struct rxe_cq, pelem), .flags = RXE_POOL_NO_ALLOC, .cleanup = rxe_cq_cleanup, }, [RXE_TYPE_MR] = { .name = "rxe-mr", .size = sizeof(struct rxe_mr), + .elem_offset = offsetof(struct rxe_mr, pelem), .cleanup = rxe_mr_cleanup, .flags = RXE_POOL_INDEX, .max_index = RXE_MAX_MR_INDEX, @@ -57,6 +64,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_MW] = { .name = "rxe-mw", .size = sizeof(struct rxe_mw), + .elem_offset = offsetof(struct rxe_mw, pelem), .flags = RXE_POOL_INDEX, .max_index = RXE_MAX_MW_INDEX, .min_index = RXE_MIN_MW_INDEX, @@ -64,6 +72,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_MC_GRP] = { .name = "rxe-mc_grp", .size = sizeof(struct rxe_mc_grp), + .elem_offset = offsetof(struct rxe_mc_grp, pelem), .cleanup = rxe_mc_cleanup, .flags = RXE_POOL_KEY, .key_offset = offsetof(struct rxe_mc_grp, mgid), @@ -72,6 +81,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_MC_ELEM] = { .name = "rxe-mc_elem", .size = sizeof(struct rxe_mc_elem), + .elem_offset = offsetof(struct rxe_mc_elem, pelem), .flags = RXE_POOL_ATOMIC, }, }; @@ -304,6 +314,8 @@ void rxe_drop_index(void *arg) void *rxe_alloc(struct rxe_pool *pool) { struct rxe_pool_entry *elem; + struct rxe_type_info *info = &rxe_type_info[pool->type]; + u8 *obj = NULL; unsigned long flags; might_sleep_if(!(pool->flags & RXE_POOL_ATOMIC)); @@ -330,8 +342,9 @@ void *rxe_alloc(struct rxe_pool *pool) elem->pool = pool; kref_init(&elem->ref_cnt); + obj = (u8 *)elem - info->elem_offset; - return elem; + return obj; out_cnt: atomic_dec(&pool->num_elem); @@ -390,10 +403,12 @@ void rxe_elem_release(struct kref *kref) rxe_pool_put(pool); } -void *rxe_pool_get_index(struct rxe_pool *pool, u32 index) +void *rxe_get_index(struct rxe_pool *pool, u32 index) { struct rb_node *node = NULL; struct rxe_pool_entry *elem = NULL; + struct rxe_type_info *info = &rxe_type_info[pool->type]; + u8 *obj = NULL; unsigned long flags; read_lock_irqsave(&pool->pool_lock, flags); @@ -410,21 +425,25 @@ void *rxe_pool_get_index(struct rxe_pool *pool, u32 index) node = node->rb_left; else if (elem->index < index) node = node->rb_right; - else { - kref_get(&elem->ref_cnt); + else break; - } } + if (node) { + kref_get(&elem->ref_cnt); + obj = (u8 *)elem - info->elem_offset; + } out: read_unlock_irqrestore(&pool->pool_lock, flags); - return node ? elem : NULL; + return obj; } -void *rxe_pool_get_key(struct rxe_pool *pool, void *key) +void *rxe_get_key(struct rxe_pool *pool, void *key) { struct rb_node *node = NULL; struct rxe_pool_entry *elem = NULL; + struct rxe_type_info *info = &rxe_type_info[pool->type]; + u8 *obj = NULL; int cmp; unsigned long flags; @@ -449,10 +468,13 @@ void *rxe_pool_get_key(struct rxe_pool *pool, void *key) break; } - if (node) + if (node) { kref_get(&elem->ref_cnt); + obj = (u8 *)elem - info->elem_offset; + } + out: read_unlock_irqrestore(&pool->pool_lock, flags); - return node ? elem : NULL; + return obj; } diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 3d722aae5f15..68c6dbeb72d4 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -36,6 +36,7 @@ struct rxe_pool_entry; struct rxe_type_info { const char *name; size_t size; + size_t elem_offset; void (*cleanup)(struct rxe_pool_entry *obj); enum rxe_pool_flags flags; u32 max_index; @@ -128,10 +129,10 @@ void rxe_add_key(void *elem, void *key); void rxe_drop_key(void *elem); /* lookup an indexed object from index. takes a reference on object */ -void *rxe_pool_get_index(struct rxe_pool *pool, u32 index); +void *rxe_get_index(struct rxe_pool *pool, u32 index); /* lookup keyed object from key. takes a reference on the object */ -void *rxe_pool_get_key(struct rxe_pool *pool, void *key); +void *rxe_get_key(struct rxe_pool *pool, void *key); /* cleanup an object when all references are dropped */ void rxe_elem_release(struct kref *kref); diff --git a/drivers/infiniband/sw/rxe/rxe_recv.c b/drivers/infiniband/sw/rxe/rxe_recv.c index a3eed4da1540..e14cb20c4596 100644 --- a/drivers/infiniband/sw/rxe/rxe_recv.c +++ b/drivers/infiniband/sw/rxe/rxe_recv.c @@ -185,7 +185,7 @@ static int hdr_check(struct rxe_pkt_info *pkt) if (qpn != IB_MULTICAST_QPN) { index = (qpn == 1) ? port->qp_gsi_index : qpn; - qp = rxe_pool_get_index(&rxe->qp_pool, index); + qp = rxe_get_index(&rxe->qp_pool, index); if (unlikely(!qp)) { pr_warn_ratelimited("no qp matches qpn 0x%x\n", qpn); goto err1; @@ -242,7 +242,7 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb) memcpy(&dgid, &ipv6_hdr(skb)->daddr, sizeof(dgid)); /* lookup mcast group corresponding to mgid, takes a ref */ - mcg = rxe_pool_get_key(&rxe->mc_grp_pool, &dgid); + mcg = rxe_get_key(&rxe->mc_grp_pool, &dgid); if (!mcg) goto err1; /* mcast group not registered */ @@ -274,7 +274,7 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb) spin_unlock_bh(&mcg->mcg_lock); - rxe_drop_ref(mcg); /* drop ref from rxe_pool_get_key. */ + rxe_drop_ref(mcg); /* drop ref from rxe_get_key. */ err1: kfree_skb(skb); diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 57236d8c2146..ea08464e461d 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -599,7 +599,7 @@ int rxe_requester(void *arg) struct rxe_dev *rxe = to_rdev(qp->ibqp.device); struct rxe_mr *rmr; - rmr = rxe_pool_get_index(&rxe->mr_pool, + rmr = rxe_get_index(&rxe->mr_pool, wqe->wr.ex.invalidate_rkey >> 8); if (!rmr) { pr_err("No mr for key %#x\n", diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 69867bf39cfb..9f96b4b706c3 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -888,7 +888,7 @@ static enum resp_states do_complete(struct rxe_qp *qp, wc->wc_flags |= IB_WC_WITH_INVALIDATE; wc->ex.invalidate_rkey = ieth_rkey(pkt); - rmr = rxe_pool_get_index(&rxe->mr_pool, + rmr = rxe_get_index(&rxe->mr_pool, wc->ex.invalidate_rkey >> 8); if (unlikely(!rmr)) { pr_err("Bad rkey %#x invalidation\n", From patchwork Thu Oct 1 17:48:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812087 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0360D6CB for ; Thu, 1 Oct 2020 17:49:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D0AAD20848 for ; Thu, 1 Oct 2020 17:49:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DfFJh3Uw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732974AbgJARtB (ORCPT ); Thu, 1 Oct 2020 13:49:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732096AbgJARs7 (ORCPT ); Thu, 1 Oct 2020 13:48:59 -0400 Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36D3CC0613D0 for ; Thu, 1 Oct 2020 10:48:59 -0700 (PDT) Received: by mail-ot1-x341.google.com with SMTP id a2so6287386otr.11 for ; Thu, 01 Oct 2020 10:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H0Pa4DycvodekXTAXArEWJli50xpBUboI/r2vry1bLM=; b=DfFJh3Uwh/gUYyVOYRg9Bi5UdBK3TAGZvYcHXA0OjFw6dP4OLKU1HGC87DN3y0fsfa 9JUHfsX+7cz/ea/zYLgX2fbc25/tk9Rnc+jwGWUjk7iM75ZhAihbf94RWpbhKgwX0G3A aMfCdtH81qvaOptQkmG4sH2eGu4qqsLeMu1BiElZpw4Yrne4izCg4d4I4IlM3IA09j6n TZJn8u1kMRa3FWr5HsQPqqE17w93m2+JIvVkyqD5xr0KRSkJkkZv06FSOsGPDFiqfWRY hnPrKA9EfebnwZN5uIL8Xbw26/2cXtJUy4zUQ9m/tgZodGmRcGvjJKYrF20wHokBs/uo brMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H0Pa4DycvodekXTAXArEWJli50xpBUboI/r2vry1bLM=; b=rvXAU7m0pSAnGZFPOng/xGG1vl6hEg32oQRlL13ZK1zFAbpCPAx719hBe9sRbrvJQ9 oMFU4IEAUfieBDZfwm3aWuZU25Z5Oz31haQi7pAwHfvgxqZietkU8o8qWcSgXacDQ2g5 0V8hB92zoUv/Rb32mHER9FD35rBqiXZs5bzGDDylDFXuH4HOHk6f5gDhxDxKuofRwtF9 rYdWvZYl7qc7my6EighmPZ6NWIoq+dzlyfixZY33x8bTLmDX3p+AxRCnhryjY/s8rEeS GVlhhgcmtRqNnJL3wlPhEZ50iQJGOkxmiKllHjQMqpDcOPTuMDpwEL8Fc2CRoBPqOANn Qg+w== X-Gm-Message-State: AOAM530Z/vuDOrowT9mbzW3zNZpHVwFMicNb7aMNxAQ8zWIEr9PIWLBf Q3ntfIZ+4+fK0Tq/pDishDU= X-Google-Smtp-Source: ABdhPJwH98eHwyCWIRr5hy0mqonkAEJWakli8jFQMLQ8+D1PLDNHP90DDUmkuc1H6u+aIKqNG/sd3A== X-Received: by 2002:a05:6830:1e8a:: with SMTP id n10mr5917822otr.371.1601574538617; Thu, 01 Oct 2020 10:48:58 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id d83sm1192790oib.43.2020.10.01.10.48.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:48:58 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 05/19] rdma_rxe: remove void * parameters in pool APIs Date: Thu, 1 Oct 2020 12:48:33 -0500 Message-Id: <20201001174847.4268-6-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org More work to make rxe pools type safe. Replaced routines that took void * as parameters and and replaced them with rxe_pool_entry *elem. Then added macros to convert objects to pool entries. This makes these routines type safe against moving the pool entries around in the objects. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_pool.c | 14 +++++-------- drivers/infiniband/sw/rxe/rxe_pool.h | 29 ++++++++++++++------------- drivers/infiniband/sw/rxe/rxe_verbs.c | 10 ++++----- 3 files changed, 25 insertions(+), 28 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index c4f1318ea761..e40a64616d79 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -264,9 +264,8 @@ static void insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) return; } -void rxe_add_key(void *arg, void *key) +void __rxe_add_key(struct rxe_pool_entry *elem, void *key) { - struct rxe_pool_entry *elem = arg; struct rxe_pool *pool = elem->pool; unsigned long flags; @@ -276,9 +275,8 @@ void rxe_add_key(void *arg, void *key) write_unlock_irqrestore(&pool->pool_lock, flags); } -void rxe_drop_key(void *arg) +void __rxe_drop_key(struct rxe_pool_entry *elem) { - struct rxe_pool_entry *elem = arg; struct rxe_pool *pool = elem->pool; unsigned long flags; @@ -287,9 +285,8 @@ void rxe_drop_key(void *arg) write_unlock_irqrestore(&pool->pool_lock, flags); } -void rxe_add_index(void *arg) +void __rxe_add_index(struct rxe_pool_entry *elem) { - struct rxe_pool_entry *elem = arg; struct rxe_pool *pool = elem->pool; unsigned long flags; @@ -299,9 +296,8 @@ void rxe_add_index(void *arg) write_unlock_irqrestore(&pool->pool_lock, flags); } -void rxe_drop_index(void *arg) +void __rxe_drop_index(struct rxe_pool_entry *elem) { - struct rxe_pool_entry *elem = arg; struct rxe_pool *pool = elem->pool; unsigned long flags; @@ -354,7 +350,7 @@ void *rxe_alloc(struct rxe_pool *pool) return NULL; } -int rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) +int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { unsigned long flags; diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 68c6dbeb72d4..2f9be92ef6f9 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -109,24 +109,25 @@ void rxe_pool_cleanup(struct rxe_pool *pool); /* allocate an object from pool */ void *rxe_alloc(struct rxe_pool *pool); -/* connect already allocated object to pool */ -int rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); +int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); -/* assign an index to an indexed object and insert object into - * pool's rb tree - */ -void rxe_add_index(void *elem); +#define rxe_add_to_pool(pool, obj) __rxe_add_to_pool(pool, &(obj)->pelem) -/* drop an index and remove object from rb tree */ -void rxe_drop_index(void *elem); +void __rxe_add_index(struct rxe_pool_entry *elem); -/* assign a key to a keyed object and insert object into - * pool's rb tree - */ -void rxe_add_key(void *elem, void *key); +#define rxe_add_index(obj) __rxe_add_index(&(obj)->pelem) + +void __rxe_drop_index(struct rxe_pool_entry *elem); + +#define rxe_drop_index(obj) __rxe_drop_index(&(obj)->pelem) + +void __rxe_add_key(struct rxe_pool_entry *elem, void *key); + +#define rxe_add_key(obj, key) __rxe_add_key(&(obj)->pelem, key) + +void __rxe_drop_key(struct rxe_pool_entry *elem); -/* remove elem from rb tree */ -void rxe_drop_key(void *elem); +#define rxe_drop_key(obj) __rxe_drop_key(&(obj)->pelem) /* lookup an indexed object from index. takes a reference on object */ void *rxe_get_index(struct rxe_pool *pool, u32 index); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 664bcb025aae..db1d7f5ca2a7 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -111,7 +111,7 @@ static int rxe_alloc_ucontext(struct ib_ucontext *uctx, struct ib_udata *udata) struct rxe_dev *rxe = to_rdev(uctx->device); struct rxe_ucontext *uc = to_ruc(uctx); - return rxe_add_to_pool(&rxe->uc_pool, &uc->pelem); + return rxe_add_to_pool(&rxe->uc_pool, uc); } static void rxe_dealloc_ucontext(struct ib_ucontext *ibuc) @@ -145,7 +145,7 @@ static int rxe_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) struct rxe_dev *rxe = to_rdev(ibpd->device); struct rxe_pd *pd = to_rpd(ibpd); - return rxe_add_to_pool(&rxe->pd_pool, &pd->pelem); + return rxe_add_to_pool(&rxe->pd_pool, pd); } static int rxe_dealloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) @@ -169,7 +169,7 @@ static int rxe_create_ah(struct ib_ah *ibah, if (err) return err; - err = rxe_add_to_pool(&rxe->ah_pool, &ah->pelem); + err = rxe_add_to_pool(&rxe->ah_pool, ah); if (err) return err; @@ -275,7 +275,7 @@ static int rxe_create_srq(struct ib_srq *ibsrq, struct ib_srq_init_attr *init, if (err) goto err1; - err = rxe_add_to_pool(&rxe->srq_pool, &srq->pelem); + err = rxe_add_to_pool(&rxe->srq_pool, srq); if (err) goto err1; @@ -776,7 +776,7 @@ static int rxe_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, if (err) return err; - return rxe_add_to_pool(&rxe->cq_pool, &cq->pelem); + return rxe_add_to_pool(&rxe->cq_pool, cq); } static int rxe_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) From patchwork Thu Oct 1 17:48:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812085 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6B51513B2 for ; Thu, 1 Oct 2020 17:49:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4DAF5208B6 for ; Thu, 1 Oct 2020 17:49:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uxEEHF44" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732984AbgJARtB (ORCPT ); Thu, 1 Oct 2020 13:49:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732962AbgJARtA (ORCPT ); Thu, 1 Oct 2020 13:49:00 -0400 Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FB6EC0613E4 for ; Thu, 1 Oct 2020 10:49:00 -0700 (PDT) Received: by mail-ot1-x342.google.com with SMTP id n61so6291549ota.10 for ; Thu, 01 Oct 2020 10:49:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RVrthfqmmTxKfC664iHwanq0OmwqtLEiT1N3ENfO1s4=; b=uxEEHF44DuMO3/HU5S8ZeCzvwcG1dfApFesD0g8pu1kUtJ9w4QoXLKEB8EvhVRg/SU rukaHtzb1g59YU+yJkfqapXzL8HTXlGL4GeS6q9r2apFCgE/JO8ECe/KmxViovIVV8/D Jg/D/xSQWAyZvYQDtniTtOAcysTiDipqAueLfPcsziYCrMu/xdztNL/5qDd5NCsmsMV8 KM4arzSCWgI29Avlq8XHK9tJFX6Cpj//O4gzLGe7pCd43U/Eax5QBRVxFc4oF0pU9PVD Uuf7lTmj/lRmx4JRP4HQgELJ8tQt5R3A3DOOLuYntFrtKC2+dGRKiKdb00l0qAoWHyQO ZXNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RVrthfqmmTxKfC664iHwanq0OmwqtLEiT1N3ENfO1s4=; b=bv7SKRh6wdz8x3N6VmDLBP5sfyqhTO2hfWSvlFaFIwn0udbxK1ZabJeFMjH664W8T1 69mOebvu3OO0FbMT/WDt1Rr/iOhK1QR2d10Tfef8oBGfp2zYuL1b3xUKwYAbo0dQgKg9 eLUuvrvq9L38C7Rv0xpf1nDJoQ9cvh2q387+jEBo2N5m3wwqj24BIsi0jvk9Drz9WFhs tlAOVeVkr5xT856PAKAL93xrQJqA1L1z5d17emHIStPjEKsg1gTFs13dzfIfUZpM1vLF c7wehhkw37eBZGw29WmoyFkreMM4AuRKwVTAgldFJCxl9xiApjkDZf917fTYuOaSmwT9 /3Hg== X-Gm-Message-State: AOAM5324xjwaxTYw3IDJ/7GaTcWrclfa+NePWLBPgex1Y0ka7wdLXJiU ekWrW17od3Nr0fMrSzQd+6U= X-Google-Smtp-Source: ABdhPJzPlbMhWNIGMQIZPX93E21dW2wtMwMVg5jAvJBkPFkHzSaH6FOOWKKm/Jd3QwLrPy6Jipy+uA== X-Received: by 2002:a9d:1726:: with SMTP id i38mr5352060ota.252.1601574539555; Thu, 01 Oct 2020 10:48:59 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id l136sm1191792oig.7.2020.10.01.10.48.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:48:59 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 06/19] rdma_rxe: add alloc_mw and dealloc_mw verbs Date: Thu, 1 Oct 2020 12:48:34 -0500 Message-Id: <20201001174847.4268-7-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org v7: - Adjust for mw being allocated in core. - Made MW NO_ALLOC type in pool - Rearranged rxe_mw struct to make ibmw first v6: - Add a new file focused on memory windows, rxe_mw.c. - Add alloc_mw and dealloc_mw verbs and added them to the list of supported user space verbs. - Added indices AND keys for MW and MR. - Add indices to user api response for MW and MR - Replaced lookup by index with lookup by key in rxe_req.c and rxe_resp.c Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/Makefile | 1 + drivers/infiniband/sw/rxe/rxe_loc.h | 7 +++ drivers/infiniband/sw/rxe/rxe_mr.c | 80 +++++++++++-------------- drivers/infiniband/sw/rxe/rxe_mw.c | 85 +++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_pool.c | 50 ++++++++++------ drivers/infiniband/sw/rxe/rxe_pool.h | 2 +- drivers/infiniband/sw/rxe/rxe_req.c | 4 +- drivers/infiniband/sw/rxe/rxe_resp.c | 4 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 53 +++++++++++------ drivers/infiniband/sw/rxe/rxe_verbs.h | 10 +++- include/uapi/rdma/rdma_user_rxe.h | 10 ++++ 11 files changed, 219 insertions(+), 87 deletions(-) diff --git a/drivers/infiniband/sw/rxe/Makefile b/drivers/infiniband/sw/rxe/Makefile index 66af72dca759..1e24673e9318 100644 --- a/drivers/infiniband/sw/rxe/Makefile +++ b/drivers/infiniband/sw/rxe/Makefile @@ -15,6 +15,7 @@ rdma_rxe-y := \ rxe_qp.o \ rxe_cq.o \ rxe_mr.o \ + rxe_mw.o \ rxe_opcode.o \ rxe_mmap.o \ rxe_icrc.o \ diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 9ec6bff6863f..1502aa557621 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -109,6 +109,13 @@ void rxe_mr_cleanup(struct rxe_pool_entry *arg); int advance_dma_data(struct rxe_dma_info *dma, unsigned int length); +/* rxe_mw.c */ +int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata); + +int rxe_dealloc_mw(struct ib_mw *ibmw); + +void rxe_mw_cleanup(struct rxe_pool_entry *arg); + /* rxe_net.c */ void rxe_loopback(struct sk_buff *skb); int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index f40cf4df394f..af2d0d8877fe 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -7,21 +7,18 @@ #include "rxe.h" #include "rxe_loc.h" -/* - * lfsr (linear feedback shift register) with period 255 +/* choose a unique non zero random number for lkey + * use high order bit to indicate MR vs MW */ -static u8 get_key(void) +static void rxe_set_mr_lkey(struct rxe_mr *mr) { - static u32 key = 1; - - key = key << 1; - - key |= (0 != (key & 0x100)) ^ (0 != (key & 0x10)) - ^ (0 != (key & 0x80)) ^ (0 != (key & 0x40)); - - key &= 0xff; - - return key; + u32 lkey; +again: + get_random_bytes(&lkey, sizeof(lkey)); + lkey &= ~IS_MW; + if (likely(lkey && (rxe_add_key(mr, &lkey) == 0))) + return; + goto again; } int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) @@ -49,36 +46,18 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) static void rxe_mr_init(int access, struct rxe_mr *mr) { - u32 lkey = mr->pelem.index << 8 | get_key(); - u32 rkey = (access & IB_ACCESS_REMOTE) ? lkey : 0; + rxe_add_index(mr); + rxe_set_mr_lkey(mr); + if (access & IB_ACCESS_REMOTE) + mr->ibmr.rkey = mr->ibmr.lkey; - if (mr->pelem.pool->type == RXE_TYPE_MR) { - mr->ibmr.lkey = lkey; - mr->ibmr.rkey = rkey; - } - - mr->lkey = lkey; - mr->rkey = rkey; + mr->lkey = mr->ibmr.lkey; + mr->rkey = mr->ibmr.rkey; mr->state = RXE_MEM_STATE_INVALID; mr->type = RXE_MR_TYPE_NONE; mr->map_shift = ilog2(RXE_BUF_PER_MAP); } -void rxe_mr_cleanup(struct rxe_pool_entry *arg) -{ - struct rxe_mr *mr = container_of(arg, typeof(*mr), pelem); - int i; - - ib_umem_release(mr->umem); - - if (mr->map) { - for (i = 0; i < mr->num_map; i++) - kfree(mr->map[i]); - - kfree(mr->map); - } -} - static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf) { int i; @@ -531,20 +510,13 @@ int advance_dma_data(struct rxe_dma_info *dma, unsigned int length) return 0; } -/* (1) find the mr corresponding to lkey/rkey - * depending on lookup_type - * (2) verify that the (qp) pd matches the mr pd - * (3) verify that the mr can support the requested access - * (4) verify that mr state is valid - */ struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, enum lookup_type type) { struct rxe_mr *mr; struct rxe_dev *rxe = to_rdev(pd->ibpd.device); - int index = key >> 8; - mr = rxe_get_index(&rxe->mr_pool, index); + mr = rxe_get_key(&rxe->mr_pool, &key); if (!mr) return NULL; @@ -559,3 +531,21 @@ struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, return mr; } + +void rxe_mr_cleanup(struct rxe_pool_entry *elem) +{ + struct rxe_mr *mr = container_of(elem, typeof(*mr), pelem); + int i; + + ib_umem_release(mr->umem); + + if (mr->map) { + for (i = 0; i < mr->num_map; i++) + kfree(mr->map[i]); + + kfree(mr->map); + } + + rxe_drop_index(mr); + rxe_drop_key(mr); +} diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c new file mode 100644 index 000000000000..c10e8762243f --- /dev/null +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -0,0 +1,85 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* + * Copyright (c) 2020 Hewlett Packard Enterprise, Inc. All rights reserved. + */ + +#include "rxe.h" +#include "rxe_loc.h" + +/* choose a unique non zero random number for rkey + * use high order bit to indicate MR vs MW + */ +static void rxe_set_mw_rkey(struct rxe_mw *mw) +{ + u32 rkey; +again: + get_random_bytes(&rkey, sizeof(rkey)); + rkey |= IS_MW; + if (likely((rkey & ~IS_MW) && + (rxe_add_key(mw, &rkey) == 0))) + return; + goto again; +} + +int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) +{ + struct rxe_mw *mw = to_rmw(ibmw); + struct rxe_pd *pd = to_rpd(ibmw->pd); + struct rxe_dev *rxe = to_rdev(ibmw->device); + struct rxe_alloc_mw_resp __user *uresp = NULL; + int ret; + + if (udata) { + if (udata->outlen < sizeof(*uresp)) + return -EINVAL; + uresp = udata->outbuf; + } + + ret = rxe_add_to_pool(&rxe->mw_pool, mw); /* takes a ref on mw */ + if (ret) + return ret; + + rxe_add_ref(pd); + rxe_add_index(mw); + rxe_set_mw_rkey(mw); /* sets mw->ibmw.rkey */ + + spin_lock_init(&mw->lock); + mw->state = (mw->ibmw.type == IB_MW_TYPE_2) ? + RXE_MEM_STATE_FREE : + RXE_MEM_STATE_VALID; + + if (uresp) { + if (copy_to_user(&uresp->index, &mw->pelem.index, + sizeof(uresp->index))) { + rxe_drop_ref(mw); + rxe_drop_ref(pd); + return -EFAULT; + } + } + + return 0; +} + +int rxe_dealloc_mw(struct ib_mw *ibmw) +{ + struct rxe_mw *mw = to_rmw(ibmw); + struct rxe_pd *pd = to_rpd(ibmw->pd); + unsigned long flags; + + spin_lock_irqsave(&mw->lock, flags); + mw->state = RXE_MEM_STATE_INVALID; + spin_unlock_irqrestore(&mw->lock, flags); + + rxe_drop_ref(mw); + rxe_drop_ref(pd); + + return 0; +} + +void rxe_mw_cleanup(struct rxe_pool_entry *elem) +{ + struct rxe_mw *mw = container_of(elem, typeof(*mw), pelem); + + rxe_drop_index(mw); + rxe_drop_key(mw); +} diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index e40a64616d79..43b4f91d86be 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -14,7 +14,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-uc", .size = sizeof(struct rxe_ucontext), .elem_offset = offsetof(struct rxe_ucontext, pelem), - .flags = RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_PD] = { .name = "rxe-pd", @@ -26,13 +26,14 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), .elem_offset = offsetof(struct rxe_ah, pelem), - .flags = RXE_POOL_ATOMIC | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_NO_ALLOC, }, [RXE_TYPE_SRQ] = { .name = "rxe-srq", .size = sizeof(struct rxe_srq), .elem_offset = offsetof(struct rxe_srq, pelem), - .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX + | RXE_POOL_NO_ALLOC, .min_index = RXE_MIN_SRQ_INDEX, .max_index = RXE_MAX_SRQ_INDEX, }, @@ -41,7 +42,8 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .size = sizeof(struct rxe_qp), .elem_offset = offsetof(struct rxe_qp, pelem), .cleanup = rxe_qp_cleanup, - .flags = RXE_POOL_INDEX, + .flags = RXE_POOL_ATOMIC + | RXE_POOL_INDEX, .min_index = RXE_MIN_QP_INDEX, .max_index = RXE_MAX_QP_INDEX, }, @@ -49,7 +51,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-cq", .size = sizeof(struct rxe_cq), .elem_offset = offsetof(struct rxe_cq, pelem), - .flags = RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_NO_ALLOC, .cleanup = rxe_cq_cleanup, }, [RXE_TYPE_MR] = { @@ -57,25 +59,37 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .size = sizeof(struct rxe_mr), .elem_offset = offsetof(struct rxe_mr, pelem), .cleanup = rxe_mr_cleanup, - .flags = RXE_POOL_INDEX, + .flags = RXE_POOL_ATOMIC + | RXE_POOL_INDEX + | RXE_POOL_KEY, .max_index = RXE_MAX_MR_INDEX, .min_index = RXE_MIN_MR_INDEX, + .key_offset = offsetof(struct rxe_mr, ibmr.lkey) + - offsetof(struct rxe_mr, pelem), + .key_size = sizeof(u32), }, [RXE_TYPE_MW] = { .name = "rxe-mw", .size = sizeof(struct rxe_mw), .elem_offset = offsetof(struct rxe_mw, pelem), - .flags = RXE_POOL_INDEX, + .flags = RXE_POOL_NO_ALLOC + | RXE_POOL_INDEX + | RXE_POOL_KEY, .max_index = RXE_MAX_MW_INDEX, .min_index = RXE_MIN_MW_INDEX, + .key_offset = offsetof(struct rxe_mw, ibmw.rkey) + - offsetof(struct rxe_mw, pelem), + .key_size = sizeof(u32), }, [RXE_TYPE_MC_GRP] = { .name = "rxe-mc_grp", .size = sizeof(struct rxe_mc_grp), .elem_offset = offsetof(struct rxe_mc_grp, pelem), .cleanup = rxe_mc_cleanup, - .flags = RXE_POOL_KEY, - .key_offset = offsetof(struct rxe_mc_grp, mgid), + .flags = RXE_POOL_ATOMIC + | RXE_POOL_KEY, + .key_offset = offsetof(struct rxe_mc_grp, mgid) + - offsetof(struct rxe_mc_grp, pelem), .key_size = sizeof(union ib_gid), }, [RXE_TYPE_MC_ELEM] = { @@ -233,7 +247,7 @@ static void insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) return; } -static void insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) +static int insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) { struct rb_node **link = &pool->key.tree.rb_node; struct rb_node *parent = NULL; @@ -249,7 +263,7 @@ static void insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) if (cmp == 0) { pr_warn("key already exists!\n"); - goto out; + return -EAGAIN; } if (cmp > 0) @@ -260,19 +274,22 @@ static void insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) rb_link_node(&new->key_node, parent, link); rb_insert_color(&new->key_node, &pool->key.tree); -out: - return; + + return 0; } -void __rxe_add_key(struct rxe_pool_entry *elem, void *key) +int __rxe_add_key(struct rxe_pool_entry *elem, void *key) { struct rxe_pool *pool = elem->pool; unsigned long flags; + int ret; write_lock_irqsave(&pool->pool_lock, flags); memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); - insert_key(pool, elem); + ret = insert_key(pool, elem); write_unlock_irqrestore(&pool->pool_lock, flags); + + return ret; } void __rxe_drop_key(struct rxe_pool_entry *elem) @@ -354,8 +371,6 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { unsigned long flags; - might_sleep_if(!(pool->flags & RXE_POOL_ATOMIC)); - read_lock_irqsave(&pool->pool_lock, flags); if (pool->state != RXE_POOL_STATE_VALID) { read_unlock_irqrestore(&pool->pool_lock, flags); @@ -469,7 +484,6 @@ void *rxe_get_key(struct rxe_pool *pool, void *key) obj = (u8 *)elem - info->elem_offset; } - out: read_unlock_irqrestore(&pool->pool_lock, flags); return obj; diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 2f9be92ef6f9..e0e75faca96a 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -121,7 +121,7 @@ void __rxe_drop_index(struct rxe_pool_entry *elem); #define rxe_drop_index(obj) __rxe_drop_index(&(obj)->pelem) -void __rxe_add_key(struct rxe_pool_entry *elem, void *key); +int __rxe_add_key(struct rxe_pool_entry *elem, void *key); #define rxe_add_key(obj, key) __rxe_add_key(&(obj)->pelem, key) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index ea08464e461d..fe3fcc70411d 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -599,8 +599,8 @@ int rxe_requester(void *arg) struct rxe_dev *rxe = to_rdev(qp->ibqp.device); struct rxe_mr *rmr; - rmr = rxe_get_index(&rxe->mr_pool, - wqe->wr.ex.invalidate_rkey >> 8); + rmr = rxe_get_key(&rxe->mr_pool, + &wqe->wr.ex.invalidate_rkey); if (!rmr) { pr_err("No mr for key %#x\n", wqe->wr.ex.invalidate_rkey); diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 9f96b4b706c3..cc8a0bb7530b 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -888,8 +888,8 @@ static enum resp_states do_complete(struct rxe_qp *qp, wc->wc_flags |= IB_WC_WITH_INVALIDATE; wc->ex.invalidate_rkey = ieth_rkey(pkt); - rmr = rxe_get_index(&rxe->mr_pool, - wc->ex.invalidate_rkey >> 8); + rmr = rxe_get_key(&rxe->mr_pool, + &wc->ex.invalidate_rkey); if (unlikely(!rmr)) { pr_err("Bad rkey %#x invalidation\n", wc->ex.invalidate_rkey); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index db1d7f5ca2a7..f9425daad924 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -869,12 +869,14 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access) struct rxe_pd *pd = to_rpd(ibpd); struct rxe_mr *mr; + rxe_add_ref(pd); + mr = rxe_alloc(&rxe->mr_pool); - if (!mr) + if (!mr) { + rxe_drop_ref(pd); return ERR_PTR(-ENOMEM); + } - rxe_add_index(mr); - rxe_add_ref(pd); rxe_mr_init_dma(pd, access, mr); return &mr->ibmr; @@ -890,6 +892,17 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, struct rxe_dev *rxe = to_rdev(ibpd->device); struct rxe_pd *pd = to_rpd(ibpd); struct rxe_mr *mr; + struct rxe_reg_mr_resp __user *uresp = NULL; + + if (udata) { + if (udata->outlen < sizeof(*uresp)) { + err = -EINVAL; + goto err1; + } + uresp = udata->outbuf; + } + + rxe_add_ref(pd); mr = rxe_alloc(&rxe->mr_pool); if (!mr) { @@ -897,22 +910,25 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, goto err2; } - rxe_add_index(mr); - - rxe_add_ref(pd); - err = rxe_mr_init_user(pd, start, length, iova, - access, udata, mr); + access, udata, mr); if (err) goto err3; - return &mr->ibmr; + if (uresp) { + if (copy_to_user(&uresp->index, &mr->pelem.index, + sizeof(uresp->index))) { + err = -EFAULT; + goto err3; + } + } + return &mr->ibmr; err3: - rxe_drop_ref(pd); - rxe_drop_index(mr); rxe_drop_ref(mr); err2: + rxe_drop_ref(pd); +err1: return ERR_PTR(err); } @@ -922,7 +938,6 @@ static int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) mr->state = RXE_MEM_STATE_ZOMBIE; rxe_drop_ref(mr->pd); - rxe_drop_index(mr); rxe_drop_ref(mr); return 0; } @@ -938,16 +953,14 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, if (mr_type != IB_MR_TYPE_MEM_REG) return ERR_PTR(-EINVAL); + rxe_add_ref(pd); + mr = rxe_alloc(&rxe->mr_pool); if (!mr) { err = -ENOMEM; goto err1; } - rxe_add_index(mr); - - rxe_add_ref(pd); - err = rxe_mr_init_fast(pd, max_num_sg, mr); if (err) goto err2; @@ -955,10 +968,9 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, return &mr->ibmr; err2: - rxe_drop_ref(pd); - rxe_drop_index(mr); rxe_drop_ref(mr); err1: + rxe_drop_ref(pd); return ERR_PTR(err); } @@ -1105,9 +1117,12 @@ static const struct ib_device_ops rxe_dev_ops = { .reg_user_mr = rxe_reg_user_mr, .req_notify_cq = rxe_req_notify_cq, .resize_cq = rxe_resize_cq, + .alloc_mw = rxe_alloc_mw, + .dealloc_mw = rxe_dealloc_mw, INIT_RDMA_OBJ_SIZE(ib_ah, rxe_ah, ibah), INIT_RDMA_OBJ_SIZE(ib_cq, rxe_cq, ibcq), + INIT_RDMA_OBJ_SIZE(ib_mw, rxe_mw, ibmw), INIT_RDMA_OBJ_SIZE(ib_pd, rxe_pd, ibpd), INIT_RDMA_OBJ_SIZE(ib_srq, rxe_srq, ibsrq), INIT_RDMA_OBJ_SIZE(ib_ucontext, rxe_ucontext, ibuc), @@ -1166,6 +1181,8 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) | BIT_ULL(IB_USER_VERBS_CMD_DESTROY_AH) | BIT_ULL(IB_USER_VERBS_CMD_ATTACH_MCAST) | BIT_ULL(IB_USER_VERBS_CMD_DETACH_MCAST) + | BIT_ULL(IB_USER_VERBS_CMD_ALLOC_MW) + | BIT_ULL(IB_USER_VERBS_CMD_DEALLOC_MW) ; ib_set_device_ops(dev, &rxe_dev_ops); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index dbc649c9c43f..3253508a3b74 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -319,9 +319,12 @@ struct rxe_mr { struct rxe_map **map; }; +/* use high order bit to separate MW and MR rkeys */ +#define IS_MW (1 << 31) + struct rxe_mw { - struct rxe_pool_entry pelem; struct ib_mw ibmw; + struct rxe_pool_entry pelem; struct rxe_qp *qp; /* type 2B only */ struct rxe_mr *mr; spinlock_t lock; @@ -441,6 +444,11 @@ static inline struct rxe_mr *to_rmr(struct ib_mr *mr) return mr ? container_of(mr, struct rxe_mr, ibmr) : NULL; } +static inline struct rxe_mw *to_rmw(struct ib_mw *mw) +{ + return mw ? container_of(mw, struct rxe_mw, ibmw) : NULL; +} + int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name); void rxe_mc_cleanup(struct rxe_pool_entry *arg); diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index d8f2e0e46dab..4ad0fa0b2ab9 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -175,4 +175,14 @@ struct rxe_modify_srq_cmd { __aligned_u64 mmap_info_addr; }; +struct rxe_reg_mr_resp { + __u32 index; + __u32 reserved; +}; + +struct rxe_alloc_mw_resp { + __u32 index; + __u32 reserved; +}; + #endif /* RDMA_USER_RXE_H */ From patchwork Thu Oct 1 17:48:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812079 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35D7A92C for ; Thu, 1 Oct 2020 17:49:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0DF7220848 for ; Thu, 1 Oct 2020 17:49:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PvVVPgfQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732991AbgJARtC (ORCPT ); Thu, 1 Oct 2020 13:49:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732965AbgJARtB (ORCPT ); Thu, 1 Oct 2020 13:49:01 -0400 Received: from mail-oi1-x241.google.com (mail-oi1-x241.google.com [IPv6:2607:f8b0:4864:20::241]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EDAEC0613E5 for ; Thu, 1 Oct 2020 10:49:01 -0700 (PDT) Received: by mail-oi1-x241.google.com with SMTP id u126so6461893oif.13 for ; Thu, 01 Oct 2020 10:49:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kIIxjMiijQKzXq2Kbweyj7CBlorEnAYCrsUgCTrnYFA=; b=PvVVPgfQzxaqxbq4Tn6gIcrarPobO4PzzTziDPqFS31Q2o+EH0ay+pHdNL8Mnra0QR Y1SanmQoGRpG+IzZzpUxDAi/zQCz/8DN+e61LrSnAxRPcuzdjC7BxTmqwR09cExcSrfM vUa4hubbz4ob+2EZCpuZtd9qpLADR/0mvcAjO6H+u7U8dQtNAsbUU5nTw2+hPkNK+xv7 EBSiikF1hD2qlpRxg3s0okL0DLMimzVLMDmlu83MNvHlL7sZavkFsMud4Uth9qh3op+c lu8lJwNREtUCQCp2Exbcb9bVmdbXdAjhQXmgi+8C/Vkp7gR8eTxhVadLmsxSfyOOw1Wg ne2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kIIxjMiijQKzXq2Kbweyj7CBlorEnAYCrsUgCTrnYFA=; b=ElDlOHBDKIOgApKHYHc9qCFJBISx6z4eOrdsTeYFcEfd4lY0EafWBs/GNOJ9p5g+uy TBE0/cZE8LRw992foRlgn54yp0xsa2TAOokfs/QqTxA01dpfBguqqxHnvJ9a4N369zt7 2exxFMlDKpTHrBvQPlQXF/goHjG68yO6NL05q/Mfcs48bgIlG2yD1zy6GCkq9dYW8ThB aVGYr+N1TwD1524f/WVSpQe0ljMO/C1A8HRddI8vwU5vEsVSLI9ila82VpIMVGKXDCjy 1XN431dqhTP4HzZptT6PMaJmB1GsYAxhiYbWwfO6b6JZmlYidKoqAZiYPf1MEJGJ79Ra fgIA== X-Gm-Message-State: AOAM530K+raG3jlTl9cVZsVJvye0mgZQShADz1tu5WPvlWnzxVD8LyYF D1sCCfocClMkQo7aq3YlUno= X-Google-Smtp-Source: ABdhPJzj7sNRHHc/NK8Jc7PL+Vr2Go+gGRc0hmQtfGmazEtv9PkH0+Q2yD2KHTKh7uB4YRLuC+aV9Q== X-Received: by 2002:aca:aa84:: with SMTP id t126mr723402oie.5.1601574540374; Thu, 01 Oct 2020 10:49:00 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id 91sm1361258ott.55.2020.10.01.10.48.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:00 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 07/19] rdma_rxe: Add bind_mw and invalidate_mw verbs Date: Thu, 1 Oct 2020 12:48:35 -0500 Message-Id: <20201001174847.4268-8-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org v7: - adjust for new pool api changes v6: - Add code to implement ibv_bind_mw (for type 1 MWs) and post send queue bind_mw (for type 2 MWs). - Add code to implement local (post send) and remote (send with invalidate) invalidate operations. - Add rules checking for MW operations from IBA. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 1 + drivers/infiniband/sw/rxe/rxe_loc.h | 6 + drivers/infiniband/sw/rxe/rxe_mr.c | 13 +- drivers/infiniband/sw/rxe/rxe_mw.c | 289 ++++++++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_opcode.c | 11 +- drivers/infiniband/sw/rxe/rxe_opcode.h | 1 - drivers/infiniband/sw/rxe/rxe_req.c | 134 +++++++++--- drivers/infiniband/sw/rxe/rxe_resp.c | 81 +++++-- drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 7 + include/uapi/rdma/rdma_user_rxe.h | 34 ++- 11 files changed, 512 insertions(+), 67 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 5dc86c9e74c2..8b81d3b24a8a 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -103,6 +103,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opcode opcode) case IB_WR_RDMA_READ_WITH_INV: return IB_WC_RDMA_READ; case IB_WR_LOCAL_INV: return IB_WC_LOCAL_INV; case IB_WR_REG_MR: return IB_WC_REG_MR; + case IB_WR_BIND_MW: return IB_WC_BIND_MW; default: return 0xff; diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 1502aa557621..0916fc8c541e 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -109,6 +109,8 @@ void rxe_mr_cleanup(struct rxe_pool_entry *arg); int advance_dma_data(struct rxe_dma_info *dma, unsigned int length); +int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr); + /* rxe_mw.c */ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata); @@ -116,6 +118,10 @@ int rxe_dealloc_mw(struct ib_mw *ibmw); void rxe_mw_cleanup(struct rxe_pool_entry *arg); +int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe); + +int rxe_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw); + /* rxe_net.c */ void rxe_loopback(struct sk_buff *skb); int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index af2d0d8877fe..a1aa5e2267d6 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -532,12 +532,23 @@ struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, return mr; } +int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr) +{ + /* TODO there are API rules being ignored here + * cleanup later. Current project is not trying + * to fix MR + */ + mr->state = RXE_MEM_STATE_FREE; + return 0; +} + void rxe_mr_cleanup(struct rxe_pool_entry *elem) { struct rxe_mr *mr = container_of(elem, typeof(*mr), pelem); int i; - ib_umem_release(mr->umem); + if (mr->umem) + ib_umem_release(mr->umem); if (mr->map) { for (i = 0; i < mr->num_map; i++) diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index c10e8762243f..b09a880df35a 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -30,7 +30,7 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) int ret; if (udata) { - if (udata->outlen < sizeof(*uresp)) + if (unlikely(udata->outlen < sizeof(*uresp))) return -EINVAL; uresp = udata->outbuf; } @@ -49,10 +49,9 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) RXE_MEM_STATE_VALID; if (uresp) { - if (copy_to_user(&uresp->index, &mw->pelem.index, - sizeof(uresp->index))) { + if (unlikely(copy_to_user(&uresp->index, &mw->pelem.index, + sizeof(uresp->index)))) { rxe_drop_ref(mw); - rxe_drop_ref(pd); return -EFAULT; } } @@ -60,22 +59,298 @@ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata) return 0; } +/* cleanup mw in case someone is still holding a ref */ +static void do_dealloc_mw(struct rxe_mw *mw) +{ + if (mw->mr) { + rxe_drop_ref(mw->mr); + atomic_dec(&mw->mr->num_mw); + mw->mr = NULL; + } + + mw->qp = NULL; + mw->access = 0; + mw->addr = 0; + mw->length = 0; + mw->state = RXE_MEM_STATE_INVALID; +} + int rxe_dealloc_mw(struct ib_mw *ibmw) { struct rxe_mw *mw = to_rmw(ibmw); - struct rxe_pd *pd = to_rpd(ibmw->pd); unsigned long flags; spin_lock_irqsave(&mw->lock, flags); - mw->state = RXE_MEM_STATE_INVALID; + + do_dealloc_mw(mw); + spin_unlock_irqrestore(&mw->lock, flags); rxe_drop_ref(mw); - rxe_drop_ref(pd); return 0; } +/* Check the rules for bind MW oepration. */ +static int check_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, + struct rxe_mw *mw, struct rxe_mr *mr) +{ + /* check to see if bind operation came through + * ibv_bind_mw verbs API. + */ + switch (mw->ibmw.type) { + case IB_MW_TYPE_1: + /* o10-37.2.34 */ + if (unlikely(!(wqe->wr.wr.umw.flags & RXE_BIND_MW))) { + pr_err_once("attempt to bind type 1 MW with send WR\n"); + return -EINVAL; + } + break; + case IB_MW_TYPE_2: + /* o10-37.2.35 */ + if (unlikely(wqe->wr.wr.umw.flags & RXE_BIND_MW)) { + pr_err_once("attempt to bind type 2 MW with verbs API\n"); + return -EINVAL; + } + + /* C10-72 */ + if (unlikely(qp->pd != to_rpd(mw->ibmw.pd))) { + pr_err_once("attempt to bind type 2 MW with qp with different PD\n"); + return -EINVAL; + } + + /* o10-37.2.40 */ + if (unlikely(wqe->wr.wr.umw.length == 0)) { + pr_err_once("attempt to invalidate type 2 MW by binding with zero length\n"); + return -EINVAL; + } + + if (unlikely(!mr)) { + pr_err_once("attempt to bind MW to a NULL mr\n"); + return -EINVAL; + } + break; + default: + return -EINVAL; + } + + if (unlikely((mw->ibmw.type == IB_MW_TYPE_1) && + (mw->state != RXE_MEM_STATE_VALID))) { + pr_err_once("attempt to bind a type 1 MW not in the valid state\n"); + return -EINVAL; + } + + /* o10-36.2.2 */ + if (unlikely((mw->access & IB_ZERO_BASED) && + (mw->ibmw.type == IB_MW_TYPE_1))) { + pr_err_once("attempt to bind a zero based type 1 MW\n"); + return -EINVAL; + } + + if (unlikely((wqe->wr.wr.umw.rkey & 0xff) == (mw->ibmw.rkey & 0xff))) { + pr_err_once("attempt to bind MW with same key\n"); + return -EINVAL; + } + + /* remaining checks only apply to a nonzero MR */ + if (!mr) + return 0; + + if (unlikely(mr->access & IB_ZERO_BASED)) { + pr_err_once("attempt to bind MW to zero based MR\n"); + return -EINVAL; + } + + /* o10-37.2.30 */ + if (unlikely((mw->ibmw.type == IB_MW_TYPE_2) && + (mw->state != RXE_MEM_STATE_FREE))) { + pr_err_once("attempt to bind a type 2 MW not in the free state\n"); + return -EINVAL; + } + + /* C10-73 */ + if (unlikely(!(mr->access & IB_ACCESS_MW_BIND))) { + pr_err_once("attempt to bind an MW to an MR without bind access\n"); + return -EINVAL; + } + + /* C10-74 */ + if (unlikely((mw->access & (IB_ACCESS_REMOTE_WRITE | + IB_ACCESS_REMOTE_ATOMIC)) && + !(mr->access & IB_ACCESS_LOCAL_WRITE))) { + pr_err_once("attempt to bind an writeable MW to an MR without local write access\n"); + return -EINVAL; + } + + /* C10-75 */ + if (mw->access & IB_ZERO_BASED) { + if (unlikely(wqe->wr.wr.umw.length > mr->length)) { + pr_err_once("attempt to bind a ZB MW outside of the MR\n"); + return -EINVAL; + } + } else { + if (unlikely((wqe->wr.wr.umw.addr < mr->iova) || + ((wqe->wr.wr.umw.addr + wqe->wr.wr.umw.length) > + (mr->iova + mr->length)))) { + pr_err_once("attempt to bind a VA MW outside of the MR\n"); + return -EINVAL; + } + } + + return 0; +} + +static int do_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, + struct rxe_mw *mw, struct rxe_mr *mr) +{ + u32 rkey; + u32 new_rkey; + struct rxe_mw *duplicate_mw; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); + + /* key part of new rkey is provided by user for type 2 + * and ibv_bind_mw() for type 1 MWs + * there is a very rare chance that the new rkey will + * collide with an existing MW. Return an error if this + * occurs + */ + rkey = mw->ibmw.rkey; + new_rkey = (rkey & 0xffffff00) | (wqe->wr.wr.umw.rkey & 0x000000ff); + duplicate_mw = rxe_get_key(&rxe->mw_pool, &new_rkey); + if (duplicate_mw) { + pr_err_once("new MW key is a duplicate, try another\n"); + rxe_drop_ref(duplicate_mw); + return -EINVAL; + } + + rxe_drop_key(mw); + rxe_add_key(mw, &new_rkey); + + mw->access = wqe->wr.wr.umw.access; + mw->state = RXE_MEM_STATE_VALID; + mw->addr = wqe->wr.wr.umw.addr; + mw->length = wqe->wr.wr.umw.length; + + if (mw->mr) { + rxe_drop_ref(mw->mr); + atomic_dec(&mw->mr->num_mw); + mw->mr = NULL; + } + + if (mw->length) { + mw->mr = mr; + atomic_inc(&mr->num_mw); + rxe_add_ref(mr); + } + + if (mw->ibmw.type == IB_MW_TYPE_2) + mw->qp = qp; + + return 0; +} + +int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) +{ + int ret; + struct rxe_mw *mw; + struct rxe_mr *mr; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); + unsigned long flags; + + if (qp->is_user) { + mw = rxe_get_index(&rxe->mw_pool, + wqe->wr.wr.umw.mw_index); + if (!mw) { + pr_err_once("mw with index = %d not found\n", + wqe->wr.wr.umw.mw_index); + ret = -EINVAL; + goto err1; + } + mr = rxe_get_index(&rxe->mr_pool, + wqe->wr.wr.umw.mr_index); + if (!mr && wqe->wr.wr.umw.length) { + pr_err_once("mr with index = %d not found\n", + wqe->wr.wr.umw.mr_index); + ret = -EINVAL; + goto err2; + } + } else { + mw = to_rmw(wqe->wr.wr.kmw.mw); + rxe_add_ref(mw); + if (wqe->wr.wr.kmw.mr) { + mr = to_rmr(wqe->wr.wr.kmw.mr); + rxe_add_ref(mr); + } else { + mr = NULL; + } + } + + spin_lock_irqsave(&mw->lock, flags); + + ret = check_bind_mw(qp, wqe, mw, mr); + if (ret) + goto err3; + + ret = do_bind_mw(qp, wqe, mw, mr); +err3: + spin_unlock_irqrestore(&mw->lock, flags); + + if (mr) + rxe_drop_ref(mr); +err2: + rxe_drop_ref(mw); +err1: + return ret; +} + +static int check_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw) +{ + if (unlikely(mw->state != RXE_MEM_STATE_VALID)) { + pr_err_once("attempt to invalidate a MW that is not valid\n"); + return -EINVAL; + } + + /* o10-37.2.26 */ + if (unlikely(mw->ibmw.type == IB_MW_TYPE_1)) { + pr_err_once("attempt to invalidate a type 1 MW\n"); + return -EINVAL; + } + + return 0; +} + +static void do_invalidate_mw(struct rxe_mw *mw) +{ + mw->qp = NULL; + + rxe_drop_ref(mw->mr); + atomic_dec(&mw->mr->num_mw); + mw->mr = NULL; + + mw->access = 0; + mw->addr = 0; + mw->length = 0; + mw->state = RXE_MEM_STATE_FREE; +} + +int rxe_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw) +{ + int ret; + unsigned long flags; + + spin_lock_irqsave(&mw->lock, flags); + + ret = check_invalidate_mw(qp, mw); + if (ret) + goto err; + + do_invalidate_mw(mw); +err: + spin_unlock_irqrestore(&mw->lock, flags); + + return ret; +} + void rxe_mw_cleanup(struct rxe_pool_entry *elem) { struct rxe_mw *mw = container_of(elem, typeof(*mw), pelem); diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw/rxe/rxe_opcode.c index 0cb4b01fd910..5532f01ae5a3 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.c +++ b/drivers/infiniband/sw/rxe/rxe_opcode.c @@ -87,13 +87,20 @@ struct rxe_wr_opcode_info rxe_wr_opcode_info[] = { [IB_WR_LOCAL_INV] = { .name = "IB_WR_LOCAL_INV", .mask = { - [IB_QPT_RC] = WR_REG_MASK, + [IB_QPT_RC] = WR_LOCAL_MASK, }, }, [IB_WR_REG_MR] = { .name = "IB_WR_REG_MR", .mask = { - [IB_QPT_RC] = WR_REG_MASK, + [IB_QPT_RC] = WR_LOCAL_MASK, + }, + }, + [IB_WR_BIND_MW] = { + .name = "IB_WR_BIND_MW", + .mask = { + [IB_QPT_RC] = WR_LOCAL_MASK, + [IB_QPT_UC] = WR_LOCAL_MASK, }, }, }; diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h index 1041ac9a9233..440e34f446bd 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.h +++ b/drivers/infiniband/sw/rxe/rxe_opcode.h @@ -20,7 +20,6 @@ enum rxe_wr_mask { WR_READ_MASK = BIT(3), WR_WRITE_MASK = BIT(4), WR_LOCAL_MASK = BIT(5), - WR_REG_MASK = BIT(6), WR_READ_OR_WRITE_MASK = WR_READ_MASK | WR_WRITE_MASK, WR_READ_WRITE_OR_SEND_MASK = WR_READ_OR_WRITE_MASK | WR_SEND_MASK, diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index fe3fcc70411d..7efafb3e21eb 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -524,9 +524,9 @@ static void save_state(struct rxe_send_wqe *wqe, struct rxe_send_wqe *rollback_wqe, u32 *rollback_psn) { - rollback_wqe->state = wqe->state; + rollback_wqe->state = wqe->state; rollback_wqe->first_psn = wqe->first_psn; - rollback_wqe->last_psn = wqe->last_psn; + rollback_wqe->last_psn = wqe->last_psn; *rollback_psn = qp->req.psn; } @@ -556,9 +556,38 @@ static void update_state(struct rxe_qp *qp, struct rxe_send_wqe *wqe, jiffies + qp->qp_timeout_jiffies); } +static int invalidate_key(struct rxe_qp *qp, u32 key) +{ + int ret; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); + struct rxe_mw *mw; + struct rxe_mr *mr; + + if (key & IS_MW) { + mw = rxe_get_key(&rxe->mw_pool, &key); + if (!mw) { + pr_err("No mw for key %#x\n", key); + return -EINVAL; + } + ret = rxe_invalidate_mw(qp, mw); + rxe_drop_ref(mw); + } else { + mr = rxe_get_key(&rxe->mr_pool, &key); + if (!mr) { + pr_err("No mr for key %#x\n", key); + return -EINVAL; + } + ret = rxe_invalidate_mr(qp, mr); + rxe_drop_ref(mr); + } + + return ret; +} + int rxe_requester(void *arg) { struct rxe_qp *qp = (struct rxe_qp *)arg; + struct rxe_mr *mr; struct rxe_pkt_info pkt; struct sk_buff *skb; struct rxe_send_wqe *wqe; @@ -569,6 +598,7 @@ int rxe_requester(void *arg) int ret; struct rxe_send_wqe rollback_wqe; u32 rollback_psn; + u32 key; rxe_add_ref(qp); @@ -594,42 +624,47 @@ int rxe_requester(void *arg) if (unlikely(!wqe)) goto exit; - if (wqe->mask & WR_REG_MASK) { - if (wqe->wr.opcode == IB_WR_LOCAL_INV) { - struct rxe_dev *rxe = to_rdev(qp->ibqp.device); - struct rxe_mr *rmr; - - rmr = rxe_get_key(&rxe->mr_pool, - &wqe->wr.ex.invalidate_rkey); - if (!rmr) { - pr_err("No mr for key %#x\n", - wqe->wr.ex.invalidate_rkey); - wqe->state = wqe_state_error; + if (wqe->mask & WR_LOCAL_MASK) { + switch (wqe->wr.opcode) { + case IB_WR_LOCAL_INV: + key = wqe->wr.ex.invalidate_rkey; + ret = invalidate_key(qp, key); + if (ret) { + wqe->status = IB_WC_LOC_QP_OP_ERR; + goto err; + } + break; + case IB_WR_REG_MR: + mr = to_rmr(wqe->wr.wr.reg.mr); + mr->state = RXE_MEM_STATE_VALID; + mr->access = wqe->wr.wr.reg.access; + mr->lkey = wqe->wr.wr.reg.key; + mr->rkey = wqe->wr.wr.reg.key; + mr->iova = wqe->wr.wr.reg.mr->iova; + break; + case IB_WR_BIND_MW: + ret = rxe_bind_mw(qp, wqe); + if (ret) { + wqe->state = wqe_state_done; wqe->status = IB_WC_MW_BIND_ERR; - goto exit; + goto err; } - rmr->state = RXE_MEM_STATE_FREE; - rxe_drop_ref(rmr); - wqe->state = wqe_state_done; - wqe->status = IB_WC_SUCCESS; - } else if (wqe->wr.opcode == IB_WR_REG_MR) { - struct rxe_mr *rmr = to_rmr(wqe->wr.wr.reg.mr); - - rmr->state = RXE_MEM_STATE_VALID; - rmr->access = wqe->wr.wr.reg.access; - rmr->lkey = wqe->wr.wr.reg.key; - rmr->rkey = wqe->wr.wr.reg.key; - rmr->iova = wqe->wr.wr.reg.mr->iova; - wqe->state = wqe_state_done; - wqe->status = IB_WC_SUCCESS; - } else { - goto exit; + break; + default: + pr_err_once("unexpected LOCAL WR opcode = %d\n", + wqe->wr.opcode); + goto err; } + + wqe->state = wqe_state_done; + wqe->status = IB_WC_SUCCESS; + qp->req.wqe_index = next_index(qp->sq.queue, + qp->req.wqe_index); + if ((wqe->wr.send_flags & IB_SEND_SIGNALED) || qp->sq_sig_type == IB_SIGNAL_ALL_WR) rxe_run_task(&qp->comp.task, 1); - qp->req.wqe_index = next_index(qp->sq.queue, - qp->req.wqe_index); + goto next_wqe; } @@ -649,6 +684,7 @@ int rxe_requester(void *arg) opcode = next_opcode(qp, wqe, wqe->wr.opcode); if (unlikely(opcode < 0)) { wqe->status = IB_WC_LOC_QP_OP_ERR; + /* TODO this should be goto err */ goto exit; } @@ -678,8 +714,7 @@ int rxe_requester(void *arg) wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; __rxe_do_task(&qp->comp.task); - rxe_drop_ref(qp); - return 0; + goto again; } payload = mtu; } @@ -687,12 +722,14 @@ int rxe_requester(void *arg) skb = init_req_packet(qp, wqe, opcode, payload, &pkt); if (unlikely(!skb)) { pr_err("qp#%d Failed allocating skb\n", qp_num(qp)); + wqe->status = IB_WC_LOC_PROT_ERR; goto err; } if (fill_packet(qp, wqe, &pkt, skb, payload)) { pr_debug("qp#%d Error during fill packet\n", qp_num(qp)); kfree_skb(skb); + wqe->status = IB_WC_LOC_PROT_ERR; goto err; } @@ -716,6 +753,7 @@ int rxe_requester(void *arg) goto exit; } + wqe->status = IB_WC_LOC_PROT_ERR; goto err; } @@ -724,11 +762,35 @@ int rxe_requester(void *arg) goto next_wqe; err: - wqe->status = IB_WC_LOC_PROT_ERR; + /* we come here if an error occurred while processing + * a send wqe. The completer will put the qp in error + * state and no more wqes will be processed unless + * the qp is cleaned up and restarted. We do not want + * to be called again + */ wqe->state = wqe_state_error; __rxe_do_task(&qp->comp.task); + ret = -EAGAIN; + goto done; exit: + /* we come here if either there are no more wqes in the send + * queue or we are blocked waiting for some resource or event. + * The current wqe will be restarted or new wqe started when + * there is work to do or we can complete the current wqe. + */ + ret = -EAGAIN; + goto done; + +again: + /* we come here if we are done with the current wqe but want to + * get called again. Mostly we loop back to next wqe so should + * be all one way or the other + */ + ret = 0; + goto done; + +done: rxe_drop_ref(qp); - return -EAGAIN; + return ret; } diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index cc8a0bb7530b..351faa867181 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -35,6 +35,7 @@ enum resp_states { RESPST_ERR_TOO_MANY_RDMA_ATM_REQ, RESPST_ERR_RNR, RESPST_ERR_RKEY_VIOLATION, + RESPST_ERR_INVALIDATE_RKEY, RESPST_ERR_LENGTH, RESPST_ERR_CQ_OVERFLOW, RESPST_ERROR, @@ -68,6 +69,7 @@ static char *resp_state_name[] = { [RESPST_ERR_TOO_MANY_RDMA_ATM_REQ] = "ERR_TOO_MANY_RDMA_ATM_REQ", [RESPST_ERR_RNR] = "ERR_RNR", [RESPST_ERR_RKEY_VIOLATION] = "ERR_RKEY_VIOLATION", + [RESPST_ERR_INVALIDATE_RKEY] = "ERR_INVALIDATE_RKEY_VIOLATION", [RESPST_ERR_LENGTH] = "ERR_LENGTH", [RESPST_ERR_CQ_OVERFLOW] = "ERR_CQ_OVERFLOW", [RESPST_ERROR] = "ERROR", @@ -751,6 +753,39 @@ static void build_rdma_network_hdr(union rdma_network_hdr *hdr, memcpy(&hdr->ibgrh, ipv6_hdr(skb), sizeof(hdr->ibgrh)); } +static int invalidate_rkey(struct rxe_qp *qp, u32 rkey) +{ + int ret; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); + struct rxe_mw *mw; + struct rxe_mr *mr; + + if (rkey & IS_MW) { + mw = rxe_get_key(&rxe->mw_pool, &rkey); + if (!mw) { + pr_err("No mw for rkey %#x\n", rkey); + goto err; + } + ret = rxe_invalidate_mw(qp, mw); + rxe_drop_ref(mw); + } else { + mr = rxe_get_key(&rxe->mr_pool, &rkey); + if (!mr || mr->ibmr.rkey != rkey) { + pr_err("No mr for rkey %#x\n", rkey); + goto err; + } + ret = rxe_invalidate_mr(qp, mr); + rxe_drop_ref(mr); + } + + if (ret) + goto err; + + return 0; +err: + return RESPST_ERR_INVALIDATE_RKEY; +} + /* Executes a new request. A retried request never reach that function (send * and writes are discarded, and reads and atomics are retried elsewhere. */ @@ -790,6 +825,14 @@ static enum resp_states execute(struct rxe_qp *qp, struct rxe_pkt_info *pkt) WARN_ON_ONCE(1); } + if (pkt->mask & RXE_IETH_MASK) { + u32 rkey = ieth_rkey(pkt); + + err = invalidate_rkey(qp, rkey); + if (err) + return err; + } + /* next expected psn, read handles this separately */ qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK; qp->resp.ack_psn = qp->resp.psn; @@ -822,15 +865,20 @@ static enum resp_states do_complete(struct rxe_qp *qp, memset(&cqe, 0, sizeof(cqe)); if (qp->rcq->is_user) { - uwc->status = qp->resp.status; - uwc->qp_num = qp->ibqp.qp_num; - uwc->wr_id = wqe->wr_id; + uwc->status = qp->resp.status; + uwc->qp_num = qp->ibqp.qp_num; + uwc->wr_id = wqe->wr_id; } else { - wc->status = qp->resp.status; - wc->qp = &qp->ibqp; - wc->wr_id = wqe->wr_id; + wc->status = qp->resp.status; + wc->qp = &qp->ibqp; + wc->wr_id = wqe->wr_id; } + /* TODO nothing is returned for error WQEs but + * at least some of these have important things + * to report (for example send with invalidate but + * rkey fails) Fix this when I clean up MR logic + */ if (wc->status == IB_WC_SUCCESS) { rxe_counter_inc(rxe, RXE_CNT_RDMA_RECV); wc->opcode = (pkt->mask & RXE_IMMDT_MASK && @@ -883,20 +931,8 @@ static enum resp_states do_complete(struct rxe_qp *qp, } if (pkt->mask & RXE_IETH_MASK) { - struct rxe_mr *rmr; - wc->wc_flags |= IB_WC_WITH_INVALIDATE; wc->ex.invalidate_rkey = ieth_rkey(pkt); - - rmr = rxe_get_key(&rxe->mr_pool, - &wc->ex.invalidate_rkey); - if (unlikely(!rmr)) { - pr_err("Bad rkey %#x invalidation\n", - wc->ex.invalidate_rkey); - return RESPST_ERROR; - } - rmr->state = RXE_MEM_STATE_FREE; - rxe_drop_ref(rmr); } wc->qp = &qp->ibqp; @@ -1314,6 +1350,15 @@ int rxe_responder(void *arg) } break; + case RESPST_ERR_INVALIDATE_RKEY: + /* RC Only - Class C. */ + /* Class J */ + qp->resp.goto_error = 1; + /* is there a better choice */ + qp->resp.status = IB_WC_REM_INV_REQ_ERR; + state = RESPST_COMPLETE; + break; + case RESPST_ERR_LENGTH: if (qp_type(qp) == IB_QPT_RC) { /* Class C */ diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index f9425daad924..807c9a3b22ea 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -577,7 +577,7 @@ static int init_send_wqe(struct rxe_qp *qp, const struct ib_send_wr *ibwr, p += sge->length; } - } else if (mask & WR_REG_MASK) { + } else if (mask & WR_LOCAL_MASK) { wqe->mask = mask; wqe->state = wqe_state_posted; return 0; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 3253508a3b74..57849df8ccec 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -316,9 +316,16 @@ struct rxe_mr { u32 max_buf; u32 num_map; + atomic_t num_mw; + struct rxe_map **map; }; +enum rxe_send_flags { + /* flag indicaes bind call came through verbs API */ + RXE_BIND_MW = (1 << 0), +}; + /* use high order bit to separate MW and MR rkeys */ #define IS_MW (1 << 31) diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index 4ad0fa0b2ab9..d49125682359 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -93,7 +93,39 @@ struct rxe_send_wr { __u32 remote_qkey; __u16 pkey_index; } ud; - /* reg is only used by the kernel and is not part of the uapi */ + struct { + __aligned_u64 addr; + __aligned_u64 length; + union { + __u32 mr_index; + __aligned_u64 reserved1; + }; + union { + __u32 mw_index; + __aligned_u64 reserved2; + }; + __u32 rkey; + __u32 access; + __u32 flags; + } umw; + /* The following are only used by the kernel + * and are not part of the uapi + */ + struct { + __aligned_u64 addr; + __aligned_u64 length; + union { + struct ib_mr *mr; + __aligned_u64 reserved1; + }; + union { + struct ib_mw *mw; + __aligned_u64 reserved2; + }; + __u32 rkey; + __u32 access; + __u32 flags; + } kmw; struct { union { struct ib_mr *mr; From patchwork Thu Oct 1 17:48:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812081 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E832A92C for ; Thu, 1 Oct 2020 17:49:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BA33620848 for ; Thu, 1 Oct 2020 17:49:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PSaxvGQu" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732957AbgJARtD (ORCPT ); Thu, 1 Oct 2020 13:49:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732995AbgJARtC (ORCPT ); Thu, 1 Oct 2020 13:49:02 -0400 Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEB96C0613D0 for ; Thu, 1 Oct 2020 10:49:01 -0700 (PDT) Received: by mail-ot1-x341.google.com with SMTP id y5so6314140otg.5 for ; Thu, 01 Oct 2020 10:49:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lg8dH5/PFM0TkLth5dQu/aI/g2fJlghNghsOSlEAD3U=; b=PSaxvGQuvOaNd1fVO9+k1Qf1s4RhEbAft7/2EPZHyidREvqRrzbIUDoh7szZHTczkN 0d/CGeRrpmtCjo8i3MSXQ7YPaes2FShi3rQnIXJSk2gB2KlfBqoVsiBTA1NEfz2+JAwL erbCoHL8s0/KgXcXM+HsBGj8eLrFfmL/2uiFjU9/Qf6aRAyETbKsi4ZNXZq4pK3Xdw4f LGTIVM9QQVo4MKOXgB+YaczK1rng7FE3zh0BxJ9punVrdoMnaRWEl+kx51noDmGz67iJ QGNFjv+gdiP+mDhF3CnfWE25BodWb9YOuQN1s7A7LICNpbdcMkpMLzAVz3cNNDZmtIC7 nr5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lg8dH5/PFM0TkLth5dQu/aI/g2fJlghNghsOSlEAD3U=; b=HC45uQFCXzAZ+OoyPfI6FC1XK5VR/PxxRcROCb8+jWD0bo2Tpm+phvxw3WS1yXu7m9 1Tc3oxeWyKQZJ5bDebufhVxxOIPY0BfAPb/aJk1qAu9ybrNRnwqjyUmfL3/jz8Hatz2u lHCVreJkhXPBbv22h8AyA7++hX8jsyTR/1qa6pMLY3S5dl73j+ds+tXGe8MFq0UW0KJx bPzUpo1JswDZ+/eatGMVCecJwg1toA+dhcOMqCHiW5asFZ93WsvNXt5pwDMi06EZFFAZ 7dCgJos3MriDJONIB1R+L28OuHXpA4/8xNX378vnk+0lzGagaQXFbMC8nX2X8VnoKuh6 G7MA== X-Gm-Message-State: AOAM530tKdFkjvMQmm0lumjlIIcNoIEEJ1howcnh7hGylMimwqR3hSWM urlM+G//FMwqbY2oNLAyhxg= X-Google-Smtp-Source: ABdhPJwoHz7YCPyuEHtm5CwWdE2Ug/4K1kO3ZW7PLAaQ3Vg6AqFEDePt0CxWruV13MnQ5wIfLBg1QA== X-Received: by 2002:a9d:5509:: with SMTP id l9mr6199865oth.154.1601574541148; Thu, 01 Oct 2020 10:49:01 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id q7sm887359oth.57.2020.10.01.10.49.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:00 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 08/19] rdma_rxe: Add memory access through MWs Date: Thu, 1 Oct 2020 12:48:36 -0500 Message-Id: <20201001174847.4268-9-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org v7: - adjust for pool API changes v6: - Implement memory access through MWs. - Add rules checks from IBA. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_loc.h | 19 +++--- drivers/infiniband/sw/rxe/rxe_mr.c | 76 +++++++++++++++--------- drivers/infiniband/sw/rxe/rxe_mw.c | 57 +++++++++++++++--- drivers/infiniband/sw/rxe/rxe_pool.c | 14 +++-- drivers/infiniband/sw/rxe/rxe_pool.h | 3 +- drivers/infiniband/sw/rxe/rxe_req.c | 15 ++--- drivers/infiniband/sw/rxe/rxe_resp.c | 83 ++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 8 files changed, 189 insertions(+), 79 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 0916fc8c541e..fcffd5075b18 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -100,28 +100,29 @@ enum lookup_type { lookup_remote, }; -struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, - enum lookup_type type); - -int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length); - -void rxe_mr_cleanup(struct rxe_pool_entry *arg); - int advance_dma_data(struct rxe_dma_info *dma, unsigned int length); int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr); +int rxe_mr_check_access(struct rxe_qp *qp, struct rxe_mr *mr, + int access, u64 va, u32 resid); + +void rxe_mr_cleanup(struct rxe_pool_entry *arg); + /* rxe_mw.c */ int rxe_alloc_mw(struct ib_mw *ibmw, struct ib_udata *udata); int rxe_dealloc_mw(struct ib_mw *ibmw); -void rxe_mw_cleanup(struct rxe_pool_entry *arg); - int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe); int rxe_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw); +int rxe_mw_check_access(struct rxe_qp *qp, struct rxe_mw *mw, + int access, u64 va, u32 resid); + +void rxe_mw_cleanup(struct rxe_pool_entry *arg); + /* rxe_net.c */ void rxe_loopback(struct sk_buff *skb); int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index a1aa5e2267d6..c3ad89d85360 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -21,7 +21,7 @@ static void rxe_set_mr_lkey(struct rxe_mr *mr) goto again; } -int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) +static int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) { switch (mr->type) { case RXE_MR_TYPE_DMA: @@ -380,6 +380,25 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, return err; } +static struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 lkey) +{ + struct rxe_mr *mr; + struct rxe_dev *rxe = to_rdev(pd->ibpd.device); + + mr = rxe_get_key(&rxe->mr_pool, &lkey); + if (!mr) + return NULL; + + if (unlikely((mr->ibmr.lkey != lkey) || (mr->pd != pd) || + (access && !(access & mr->access)) || + (mr->state != RXE_MEM_STATE_VALID))) { + rxe_drop_ref(mr); + return NULL; + } + + return mr; +} + /* copy data in or out of a wqe, i.e. sg list * under the control of a dma descriptor */ @@ -409,7 +428,7 @@ int copy_data( } if (sge->length && (offset < sge->length)) { - mr = lookup_mr(pd, access, sge->lkey, lookup_local); + mr = lookup_mr(pd, access, sge->lkey); if (!mr) { err = -EINVAL; goto err1; @@ -434,8 +453,7 @@ int copy_data( } if (sge->length) { - mr = lookup_mr(pd, access, sge->lkey, - lookup_local); + mr = lookup_mr(pd, access, sge->lkey); if (!mr) { err = -EINVAL; goto err1; @@ -510,28 +528,6 @@ int advance_dma_data(struct rxe_dma_info *dma, unsigned int length) return 0; } -struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, - enum lookup_type type) -{ - struct rxe_mr *mr; - struct rxe_dev *rxe = to_rdev(pd->ibpd.device); - - mr = rxe_get_key(&rxe->mr_pool, &key); - if (!mr) - return NULL; - - if (unlikely((type == lookup_local && mr->lkey != key) || - (type == lookup_remote && mr->rkey != key) || - mr->pd != pd || - (access && !(access & mr->access)) || - mr->state != RXE_MEM_STATE_VALID)) { - rxe_drop_ref(mr); - mr = NULL; - } - - return mr; -} - int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr) { /* TODO there are API rules being ignored here @@ -542,6 +538,34 @@ int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr) return 0; } +int rxe_mr_check_access(struct rxe_qp *qp, struct rxe_mr *mr, + int access, u64 va, u32 resid) +{ + int ret; + struct rxe_pd *pd = to_rpd(mr->ibmr.pd); + + if (unlikely(mr->state != RXE_MEM_STATE_VALID)) { + pr_err("attempt to access a MR that is not in the valid state\n"); + return -EINVAL; + } + + /* C10-56 */ + if (unlikely(pd != qp->pd)) { + pr_err("attempt to access a MR with a different PD than the QP\n"); + return -EINVAL; + } + + /* C10-57 */ + if (unlikely(access && !(access & mr->access))) { + pr_err("attempt to access a MR without required access rights\n"); + return -EINVAL; + } + + ret = mr_check_range(mr, va, resid); + + return ret; +} + void rxe_mr_cleanup(struct rxe_pool_entry *elem) { struct rxe_mr *mr = container_of(elem, typeof(*mr), pelem); diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index b09a880df35a..22d3570ac184 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -305,11 +305,6 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) static int check_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw) { - if (unlikely(mw->state != RXE_MEM_STATE_VALID)) { - pr_err_once("attempt to invalidate a MW that is not valid\n"); - return -EINVAL; - } - /* o10-37.2.26 */ if (unlikely(mw->ibmw.type == IB_MW_TYPE_1)) { pr_err_once("attempt to invalidate a type 1 MW\n"); @@ -323,9 +318,11 @@ static void do_invalidate_mw(struct rxe_mw *mw) { mw->qp = NULL; - rxe_drop_ref(mw->mr); - atomic_dec(&mw->mr->num_mw); - mw->mr = NULL; + if (mw->mr) { + atomic_dec(&mw->mr->num_mw); + mw->mr = NULL; + rxe_drop_ref(mw->mr); + } mw->access = 0; mw->addr = 0; @@ -351,6 +348,50 @@ int rxe_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw) return ret; } +int rxe_mw_check_access(struct rxe_qp *qp, struct rxe_mw *mw, + int access, u64 va, u32 resid) +{ + struct rxe_pd *pd = to_rpd(mw->ibmw.pd); + + if (unlikely(mw->state != RXE_MEM_STATE_VALID)) { + pr_err_once("attempt to access a MW that is not valid\n"); + return -EINVAL; + } + + /* C10-76.2.1 */ + if (unlikely((mw->ibmw.type == IB_MW_TYPE_1) && (pd != qp->pd))) { + pr_err_once("attempt to access a type 1 MW with a different PD than the QP\n"); + return -EINVAL; + } + + /* o10-37.2.43 */ + if (unlikely((mw->ibmw.type == IB_MW_TYPE_2) && (mw->qp != qp))) { + pr_err_once("attempt to access a type 2 MW that is associated with a different QP\n"); + return -EINVAL; + } + + /* C10-77 */ + if (unlikely(access && !(access & mw->access))) { + pr_err_once("attempt to access a MW without sufficient access\n"); + return -EINVAL; + } + + if (mw->access & IB_ZERO_BASED) { + if (unlikely((va + resid) > mw->length)) { + pr_err_once("attempt to access a ZB MW out of bounds\n"); + return -EINVAL; + } + } else { + if (unlikely((va < mw->addr) || + ((va + resid) > (mw->addr + mw->length)))) { + pr_err_once("attempt to access a VA MW out of bounds\n"); + return -EINVAL; + } + } + + return 0; +} + void rxe_mw_cleanup(struct rxe_pool_entry *elem) { struct rxe_mw *mw = container_of(elem, typeof(*mw), pelem); diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 43b4f91d86be..5b83b6088975 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -72,6 +72,7 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-mw", .size = sizeof(struct rxe_mw), .elem_offset = offsetof(struct rxe_mw, pelem), + .cleanup = rxe_mw_cleanup, .flags = RXE_POOL_NO_ALLOC | RXE_POOL_INDEX | RXE_POOL_KEY, @@ -220,11 +221,12 @@ static u32 alloc_index(struct rxe_pool *pool) return index + pool->index.min_index; } -static void insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) +static int insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) { struct rb_node **link = &pool->index.tree.rb_node; struct rb_node *parent = NULL; struct rxe_pool_entry *elem; + int ret = 0; while (*link) { parent = *link; @@ -232,6 +234,7 @@ static void insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) if (elem->index == new->index) { pr_warn("element already exists!\n"); + ret = -EINVAL; goto out; } @@ -244,7 +247,7 @@ static void insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) rb_link_node(&new->index_node, parent, link); rb_insert_color(&new->index_node, &pool->index.tree); out: - return; + return ret; } static int insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) @@ -302,15 +305,18 @@ void __rxe_drop_key(struct rxe_pool_entry *elem) write_unlock_irqrestore(&pool->pool_lock, flags); } -void __rxe_add_index(struct rxe_pool_entry *elem) +int __rxe_add_index(struct rxe_pool_entry *elem) { struct rxe_pool *pool = elem->pool; unsigned long flags; + int ret; write_lock_irqsave(&pool->pool_lock, flags); elem->index = alloc_index(pool); - insert_index(pool, elem); + ret = insert_index(pool, elem); write_unlock_irqrestore(&pool->pool_lock, flags); + + return ret; } void __rxe_drop_index(struct rxe_pool_entry *elem) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index e0e75faca96a..5954fae6cf99 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -113,7 +113,7 @@ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); #define rxe_add_to_pool(pool, obj) __rxe_add_to_pool(pool, &(obj)->pelem) -void __rxe_add_index(struct rxe_pool_entry *elem); +int __rxe_add_index(struct rxe_pool_entry *elem); #define rxe_add_index(obj) __rxe_add_index(&(obj)->pelem) @@ -144,4 +144,5 @@ void rxe_elem_release(struct kref *kref); /* drop a reference on an object */ #define rxe_drop_ref(elem) kref_put(&(elem)->pelem.ref_cnt, rxe_elem_release) +#define rxe_read_ref(elem) kref_read(&(elem)->pelem.ref_cnt) #endif /* RXE_POOL_H */ diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 7efafb3e21eb..0428965c664b 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -645,7 +645,6 @@ int rxe_requester(void *arg) case IB_WR_BIND_MW: ret = rxe_bind_mw(qp, wqe); if (ret) { - wqe->state = wqe_state_done; wqe->status = IB_WC_MW_BIND_ERR; goto err; } @@ -653,6 +652,7 @@ int rxe_requester(void *arg) default: pr_err_once("unexpected LOCAL WR opcode = %d\n", wqe->wr.opcode); + wqe->status = IB_WC_LOC_QP_OP_ERR; goto err; } @@ -698,13 +698,7 @@ int rxe_requester(void *arg) payload = (mask & RXE_WRITE_OR_SEND) ? wqe->dma.resid : 0; if (payload > mtu) { if (qp_type(qp) == IB_QPT_UD) { - /* C10-93.1.1: If the total sum of all the buffer lengths specified for a - * UD message exceeds the MTU of the port as returned by QueryHCA, the CI - * shall not emit any packets for this message. Further, the CI shall not - * generate an error due to this condition. - */ - - /* fake a successful UD send */ + /* C10-93.1.1: fake a successful UD send */ wqe->first_psn = qp->req.psn; wqe->last_psn = qp->req.psn; qp->req.psn = (qp->req.psn + 1) & BTH_PSN_MASK; @@ -769,6 +763,8 @@ int rxe_requester(void *arg) * to be called again */ wqe->state = wqe_state_error; + qp->req.wqe_index = next_index(qp->sq.queue, + qp->req.wqe_index); __rxe_do_task(&qp->comp.task); ret = -EAGAIN; goto done; @@ -784,8 +780,7 @@ int rxe_requester(void *arg) again: /* we come here if we are done with the current wqe but want to - * get called again. Mostly we loop back to next wqe so should - * be all one way or the other + * get called again. */ ret = 0; goto done; diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 351faa867181..2aae311f9063 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -393,6 +393,8 @@ static enum resp_states check_rkey(struct rxe_qp *qp, struct rxe_pkt_info *pkt) { struct rxe_mr *mr = NULL; + struct rxe_mw *mw = NULL; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); u64 va; u32 rkey; u32 resid; @@ -400,6 +402,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, int mtu = qp->mtu; enum resp_states state; int access; + unsigned long flags; if (pkt->mask & (RXE_READ_MASK | RXE_WRITE_MASK)) { if (pkt->mask & RXE_RETH_MASK) { @@ -407,6 +410,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, qp->resp.rkey = reth_rkey(pkt); qp->resp.resid = reth_len(pkt); qp->resp.length = reth_len(pkt); + qp->resp.offset = 0; } access = (pkt->mask & RXE_READ_MASK) ? IB_ACCESS_REMOTE_READ : IB_ACCESS_REMOTE_WRITE; @@ -414,6 +418,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, qp->resp.va = atmeth_va(pkt); qp->resp.rkey = atmeth_rkey(pkt); qp->resp.resid = sizeof(u64); + qp->resp.offset = 0; access = IB_ACCESS_REMOTE_ATOMIC; } else { return RESPST_EXECUTE; @@ -431,20 +436,46 @@ static enum resp_states check_rkey(struct rxe_qp *qp, resid = qp->resp.resid; pktlen = payload_size(pkt); - mr = lookup_mr(qp->pd, access, rkey, lookup_remote); - if (!mr) { - state = RESPST_ERR_RKEY_VIOLATION; - goto err; - } + /* check rkey on each packet because someone could + * have invalidated, deallocated or unregistered it + * since the last packet + */ + if (rkey & IS_MW) { + mw = rxe_get_key(&rxe->mw_pool, &rkey); + if (!mw) { + pr_err_once("no MW found with rkey = 0x%08x\n", rkey); + state = RESPST_ERR_RKEY_VIOLATION; + goto err; + } - if (unlikely(mr->state == RXE_MEM_STATE_FREE)) { - state = RESPST_ERR_RKEY_VIOLATION; - goto err; - } + spin_lock_irqsave(&mw->lock, flags); + if (rxe_mw_check_access(qp, mw, access, va, resid)) { + spin_unlock_irqrestore(&mw->lock, flags); + rxe_drop_ref(mw); + state = RESPST_ERR_RKEY_VIOLATION; + goto err; + } - if (mr_check_range(mr, va, resid)) { - state = RESPST_ERR_RKEY_VIOLATION; - goto err; + mr = mw->mr; + rxe_add_ref(mr); + + if (mw->access & IB_ZERO_BASED) + qp->resp.offset = mw->addr; + + spin_unlock_irqrestore(&mw->lock, flags); + rxe_drop_ref(mw); + } else { + mr = rxe_get_key(&rxe->mr_pool, &rkey); + if (!mr || (mr->rkey != rkey)) { + pr_err_once("no MR found with rkey = 0x%08x\n", rkey); + state = RESPST_ERR_RKEY_VIOLATION; + goto err; + } + + if (rxe_mr_check_access(qp, mr, access, va, resid)) { + state = RESPST_ERR_RKEY_VIOLATION; + goto err; + } } if (pkt->mask & RXE_WRITE_MASK) { @@ -500,8 +531,8 @@ static enum resp_states write_data_in(struct rxe_qp *qp, int err; int data_len = payload_size(pkt); - err = rxe_mr_copy(qp->resp.mr, qp->resp.va, payload_addr(pkt), - data_len, to_mr_obj, NULL); + err = rxe_mr_copy(qp->resp.mr, qp->resp.va + qp->resp.offset, + payload_addr(pkt), data_len, to_mr_obj, NULL); if (err) { rc = RESPST_ERR_RKEY_VIOLATION; goto out; @@ -520,7 +551,6 @@ static DEFINE_SPINLOCK(atomic_ops_lock); static enum resp_states process_atomic(struct rxe_qp *qp, struct rxe_pkt_info *pkt) { - u64 iova = atmeth_va(pkt); u64 *vaddr; enum resp_states ret; struct rxe_mr *mr = qp->resp.mr; @@ -530,7 +560,7 @@ static enum resp_states process_atomic(struct rxe_qp *qp, goto out; } - vaddr = iova_to_vaddr(mr, iova, sizeof(u64)); + vaddr = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, sizeof(u64)); /* check vaddr is 8 bytes aligned. */ if (!vaddr || (uintptr_t)vaddr & 7) { @@ -655,8 +685,10 @@ static enum resp_states read_reply(struct rxe_qp *qp, res->type = RXE_READ_MASK; res->replay = 0; - res->read.va = qp->resp.va; - res->read.va_org = qp->resp.va; + res->read.va = qp->resp.va + + qp->resp.offset; + res->read.va_org = qp->resp.va + + qp->resp.offset; res->first_psn = req_pkt->psn; @@ -1336,7 +1368,10 @@ int rxe_responder(void *arg) /* Class C */ do_class_ac_error(qp, AETH_NAK_REM_ACC_ERR, IB_WC_REM_ACCESS_ERR); - state = RESPST_COMPLETE; + if (qp->resp.wqe) + state = RESPST_COMPLETE; + else + state = RESPST_ACKNOWLEDGE; } else { qp->resp.drop_msg = 1; if (qp->srq) { @@ -1364,7 +1399,10 @@ int rxe_responder(void *arg) /* Class C */ do_class_ac_error(qp, AETH_NAK_INVALID_REQ, IB_WC_REM_INV_REQ_ERR); - state = RESPST_COMPLETE; + if (qp->resp.wqe) + state = RESPST_COMPLETE; + else + state = RESPST_ACKNOWLEDGE; } else if (qp->srq) { /* UC/UD - class E */ qp->resp.status = IB_WC_REM_INV_REQ_ERR; @@ -1380,7 +1418,10 @@ int rxe_responder(void *arg) /* All, Class A. */ do_class_ac_error(qp, AETH_NAK_REM_OP_ERR, IB_WC_LOC_QP_OP_ERR); - state = RESPST_COMPLETE; + if (qp->resp.wqe) + state = RESPST_COMPLETE; + else + state = RESPST_ACKNOWLEDGE; break; case RESPST_ERR_CQ_OVERFLOW: diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 57849df8ccec..e8d9c8388345 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -183,6 +183,7 @@ struct rxe_resp_info { /* RDMA read / atomic only */ u64 va; + u64 offset; struct rxe_mr *mr; u32 resid; u32 rkey; From patchwork Thu Oct 1 17:48:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812091 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 70BD213B2 for ; Thu, 1 Oct 2020 17:49:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 49970208B6 for ; Thu, 1 Oct 2020 17:49:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OMXKU1Bd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732940AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732995AbgJARtE (ORCPT ); Thu, 1 Oct 2020 13:49:04 -0400 Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 030CBC0613D0 for ; Thu, 1 Oct 2020 10:49:03 -0700 (PDT) Received: by mail-oo1-xc30.google.com with SMTP id 4so1695596ooh.11 for ; Thu, 01 Oct 2020 10:49:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fc531sLyxOZPdtcwW7cnTNlX+F5YJyjNvQPM6F5HE4c=; b=OMXKU1Bdat5cdjzVsq2kluicKtb59hVmTs0fdAEZRY2Bc11T1hBjdCPvmm7XxIH/3F 6i4LOlsNZL6LakW5o0Pkk2yIMQyZ5PUM+gZAeRLAgi2Qe/ZKo/XQA3jq039mdEhgoBss 3S6dvC2YstzO31zGyEN2SQRtHNSAsOxB+wOyiCjbubXIVtdP8XDS7L5XNmtkIK+ZK4Sm OCn0Ndgl9YhR0UmzbekKu6f1oZp7RCvI1W65uakXiSr0vb4SQ80oyQ2Eit3pZumwONam 2IgNV6psN5hkggp7bOvKZxMnK5H6gms1lhPkYVzMg17833AWE9VdRLYXGrlKaIPTfgRh NDQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fc531sLyxOZPdtcwW7cnTNlX+F5YJyjNvQPM6F5HE4c=; b=LxOELU7AorVRH2KiPoN9AP3Fi4lTabXcTpQdRtVCtKtFcbwHafcyQfE9WsrWoaoiI/ NKOGeZyFZUuOVOu8o8C7TzgqEapp7Wpa07y7fGHizFGm1Xia823fkFTP0nRQqsTjm9ln lLBA1M/PaZadqp34BYRsqr/zlx4kbf5q4+HRWBGJazH3cj9O8FXNgL/Kx+LUhwprWD0p HCDTd6wb+DJOuLQv8cTgYpbJ7VtFpW6veWvMu8bw5cKbOuK+DcWVEJ1sRaARrdaFUGkS m+jHfqJD4ZcmU2clb2RiRQ7scg/m0wipGomL+1Kyt5vefbFj876T0PqzOXyU1UBsn38p a9Rg== X-Gm-Message-State: AOAM531VYIiXMgLQioOVq0y10+YK3AQI0aKFB3lzKbAR/MAVLiy3oWht EBZeHqewvGhJtD/efGcxYQ8fo40wpRo= X-Google-Smtp-Source: ABdhPJzQhR+JD5toawWp0cHuyNkjqcrWu1Q+xdHv23u6ad2Ll8tn53VchhtWi8yMjjf6Sng2J5Gj9w== X-Received: by 2002:a4a:dc99:: with SMTP id g25mr6443442oou.64.1601574542121; Thu, 01 Oct 2020 10:49:02 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id y25sm1355953oti.26.2020.10.01.10.49.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:01 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 09/19] rdma_rxe: Add locked and unlocked pool APIs Date: Thu, 1 Oct 2020 12:48:37 -0500 Message-Id: <20201001174847.4268-10-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org v7: - Current object pool APIs use the pool_lock to protect concurrent execution from race conditions. Most APIs take the pool_lock, execute one operation and then free the lock. This patch adds unlocked versions of the APIs so that a caller can take the pool_lock, combine sequences of operations and then free the lock. This will be required to fix some race conditions which occur without this fix. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_pool.c | 528 ++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_pool.h | 111 +++--- 2 files changed, 467 insertions(+), 172 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 5b83b6088975..9d53b4d71230 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -7,9 +7,8 @@ #include "rxe.h" #include "rxe_loc.h" -/* info about object pools - */ -struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { +/* info about object pools */ +static struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { [RXE_TYPE_UC] = { .name = "rxe-uc", .size = sizeof(struct rxe_ucontext), @@ -62,8 +61,8 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .flags = RXE_POOL_ATOMIC | RXE_POOL_INDEX | RXE_POOL_KEY, - .max_index = RXE_MAX_MR_INDEX, .min_index = RXE_MIN_MR_INDEX, + .max_index = RXE_MAX_MR_INDEX, .key_offset = offsetof(struct rxe_mr, ibmr.lkey) - offsetof(struct rxe_mr, pelem), .key_size = sizeof(u32), @@ -76,8 +75,8 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .flags = RXE_POOL_NO_ALLOC | RXE_POOL_INDEX | RXE_POOL_KEY, - .max_index = RXE_MAX_MW_INDEX, .min_index = RXE_MIN_MW_INDEX, + .max_index = RXE_MAX_MW_INDEX, .key_offset = offsetof(struct rxe_mw, ibmw.rkey) - offsetof(struct rxe_mw, pelem), .key_size = sizeof(u32), @@ -106,21 +105,18 @@ static inline const char *pool_name(struct rxe_pool *pool) return rxe_type_info[pool->type].name; } -static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) +static int init_index_table(struct rxe_pool *pool, u32 indices) { int err = 0; size_t size; - if ((max - min + 1) < pool->max_elem) { + if (indices < pool->max_elem) { pr_warn("not enough indices for max_elem\n"); err = -EINVAL; goto out; } - pool->index.max_index = max; - pool->index.min_index = min; - - size = BITS_TO_LONGS(max - min + 1) * sizeof(long); + size = BITS_TO_LONGS(indices) * sizeof(long); pool->index.table = kmalloc(size, GFP_KERNEL); if (!pool->index.table) { err = -ENOMEM; @@ -128,31 +124,37 @@ static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) } pool->index.table_size = size; - bitmap_zero(pool->index.table, max - min + 1); - + bitmap_zero(pool->index.table, indices); out: return err; } -int rxe_pool_init( - struct rxe_dev *rxe, - struct rxe_pool *pool, - enum rxe_elem_type type, - unsigned int max_elem) +/** + * rxe_pool_init - Initialize an object pool + * @rxe: The rxe device containing pool + * @pool: The pool to initialize + * @type: The object type held in pool + * @max_elem: The pool size in elements + * + * Initialize a pool of objects with given limit on + * number of elements. gets parameters from rxe_type_info + */ +int rxe_pool_init(struct rxe_dev *rxe, struct rxe_pool *pool, + enum rxe_elem_type type, unsigned int max_elem) { - int err = 0; - size_t size = rxe_type_info[type].size; + struct rxe_type_info *info = &rxe_type_info[type]; + int err = 0; memset(pool, 0, sizeof(*pool)); pool->rxe = rxe; pool->type = type; pool->max_elem = max_elem; - pool->elem_size = ALIGN(size, RXE_POOL_ALIGN); - pool->flags = rxe_type_info[type].flags; + pool->elem_size = ALIGN(info->size, RXE_POOL_ALIGN); + pool->flags = info->flags; pool->index.tree = RB_ROOT; pool->key.tree = RB_ROOT; - pool->cleanup = rxe_type_info[type].cleanup; + pool->cleanup = info->cleanup; atomic_set(&pool->num_elem, 0); @@ -160,26 +162,27 @@ int rxe_pool_init( rwlock_init(&pool->pool_lock); - if (rxe_type_info[type].flags & RXE_POOL_INDEX) { - err = rxe_pool_init_index(pool, - rxe_type_info[type].max_index, - rxe_type_info[type].min_index); + if (info->flags & RXE_POOL_INDEX) { + pool->index.min_index = info->min_index; + pool->index.max_index = info->max_index; + + err = init_index_table(pool, info->max_index + - info->min_index + 1); if (err) goto out; } - if (rxe_type_info[type].flags & RXE_POOL_KEY) { - pool->key.key_offset = rxe_type_info[type].key_offset; - pool->key.key_size = rxe_type_info[type].key_size; + if (info->flags & RXE_POOL_KEY) { + pool->key.key_offset = info->key_offset; + pool->key.key_size = info->key_size; } pool->state = RXE_POOL_STATE_VALID; - out: return err; } -static void rxe_pool_release(struct kref *kref) +static void pool_release(struct kref *kref) { struct rxe_pool *pool = container_of(kref, struct rxe_pool, ref_cnt); @@ -187,11 +190,19 @@ static void rxe_pool_release(struct kref *kref) kfree(pool->index.table); } -static void rxe_pool_put(struct rxe_pool *pool) -{ - kref_put(&pool->ref_cnt, rxe_pool_release); -} - +#define pool_get(pool) kref_get(&(pool)->ref_cnt) +#define pool_put(pool) kref_put(&(pool)->ref_cnt, pool_release) + +/** + * rxe_pool_cleanup - Cleanup an object pool + * @pool: The pool to cleanup + * + * Marks pool as invalid and drops a ref to the pool. + * When the ref count gets to zero pool_release will + * delete the table memory held by the pool. + * Pool entries hold references to the pool so it will + * hang around until they are all removed. + */ void rxe_pool_cleanup(struct rxe_pool *pool) { unsigned long flags; @@ -203,9 +214,15 @@ void rxe_pool_cleanup(struct rxe_pool *pool) pool_name(pool)); write_unlock_irqrestore(&pool->pool_lock, flags); - rxe_pool_put(pool); + pool_put(pool); } +/* + * Allocate a new index in the pool from the index bit table. + * Search from the last allocated index forward. + * Caller should hold pool->pool_lock to be thread safe. + * There should be more possible indices than objects in the pool. + */ static u32 alloc_index(struct rxe_pool *pool) { u32 index; @@ -215,12 +232,21 @@ static u32 alloc_index(struct rxe_pool *pool) if (index >= range) index = find_first_zero_bit(pool->index.table, range); - WARN_ON_ONCE(index >= range); + if (unlikely(index >= range)) + pr_err("Unable to allocate a new index in %s pool\n", + pool_name(pool)); + set_bit(index, pool->index.table); pool->index.last = index; + return index + pool->index.min_index; } +/* + * Insert index in index-red-black tree in pool. + * Used to lookup object by index. Returns an error + * if index is already inserted. + */ static int insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) { struct rb_node **link = &pool->index.tree.rb_node; @@ -232,8 +258,9 @@ static int insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) parent = *link; elem = rb_entry(parent, struct rxe_pool_entry, index_node); - if (elem->index == new->index) { - pr_warn("element already exists!\n"); + if (unlikely(elem->index == new->index)) { + pr_warn("index %d already exists in %s pool\n", + new->index, pool_name(pool)); ret = -EINVAL; goto out; } @@ -250,11 +277,79 @@ static int insert_index(struct rxe_pool *pool, struct rxe_pool_entry *new) return ret; } +/** + * __add_index - Add index to pool entry + * @elem: pool entry to add index to + * + * Allocate a new index and insert into index-red-black tree. + * Caller should hold pool->pool_lock. + */ +void __add_index(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + + elem->index = alloc_index(pool); + insert_index(pool, elem); +} + +/** + * __rxe_add_index - Lock and add index to pool entry + * @elem: pool entry to add index to + * + * Allocate a new index and insert into index-red-black tree. + */ +void __rxe_add_index(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + unsigned long flags; + + write_lock_irqsave(&pool->pool_lock, flags); + __add_index(elem); + write_unlock_irqrestore(&pool->pool_lock, flags); +} + +/** + * __drop_index - Remove index from pool entry + * @elem: pool entry to remove index from + * + * Remove index from bit table and delete entry from index-red-black tree. + * Caller should hold pool->pool_lock. + */ +void __drop_index(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + + clear_bit(elem->index - pool->index.min_index, pool->index.table); + rb_erase(&elem->index_node, &pool->index.tree); +} + +/** + * __rxe_drop_index - Lock and remove index from pool entry + * @elem: pool entry to remove index from + * + * Remove index from bit table and delete from index-red-black tree. + */ +void __rxe_drop_index(struct rxe_pool_entry *elem) +{ + struct rxe_pool *pool = elem->pool; + unsigned long flags; + + write_lock_irqsave(&pool->pool_lock, flags); + __drop_index(elem); + write_unlock_irqrestore(&pool->pool_lock, flags); +} + +/* + * Insert key in key red-black tree in pool. + * Used to lookup object by key. Returns an error + * if key is already inserted. + */ static int insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) { struct rb_node **link = &pool->key.tree.rb_node; struct rb_node *parent = NULL; struct rxe_pool_entry *elem; + int ret = 0; int cmp; while (*link) { @@ -262,11 +357,14 @@ static int insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) elem = rb_entry(parent, struct rxe_pool_entry, key_node); cmp = memcmp((u8 *)elem + pool->key.key_offset, - (u8 *)new + pool->key.key_offset, pool->key.key_size); + (u8 *)new + pool->key.key_offset, + pool->key.key_size); - if (cmp == 0) { - pr_warn("key already exists!\n"); - return -EAGAIN; + if (unlikely(cmp == 0)) { + pr_warn("key already exists in %s pool\n", + pool_name(pool)); + ret = -EAGAIN; + goto out; } if (cmp > 0) @@ -277,75 +375,98 @@ static int insert_key(struct rxe_pool *pool, struct rxe_pool_entry *new) rb_link_node(&new->key_node, parent, link); rb_insert_color(&new->key_node, &pool->key.tree); - - return 0; +out: + return ret; } -int __rxe_add_key(struct rxe_pool_entry *elem, void *key) +/** + * __add_key - Add key to pool entry + * @elem: pool entry to add index to + * @key: the key to add + * + * Copy key into entry and insert into key-red-black tree. + * Caller should hold pool->pool_lock. + */ +int __add_key(struct rxe_pool_entry *elem, void *key) { - struct rxe_pool *pool = elem->pool; - unsigned long flags; int ret; + struct rxe_pool *pool = elem->pool; - write_lock_irqsave(&pool->pool_lock, flags); memcpy((u8 *)elem + pool->key.key_offset, key, pool->key.key_size); ret = insert_key(pool, elem); - write_unlock_irqrestore(&pool->pool_lock, flags); return ret; } -void __rxe_drop_key(struct rxe_pool_entry *elem) +/** + * __rxe_add_key - Lock and add key to pool entry + * @elem: pool entry to add index to + * @key: the key to add + * + * Copy key into entry and insert into key-red-black tree. + */ +int __rxe_add_key(struct rxe_pool_entry *elem, void *key) { + int ret; struct rxe_pool *pool = elem->pool; unsigned long flags; write_lock_irqsave(&pool->pool_lock, flags); - rb_erase(&elem->key_node, &pool->key.tree); + ret = __add_key(elem, key); write_unlock_irqrestore(&pool->pool_lock, flags); + + return ret; } -int __rxe_add_index(struct rxe_pool_entry *elem) +/** + * __drop_key - Remove key from pool entry + * @elem: pool entry to remove key from + * + * Delete entry from key-red-black tree. + * Caller should hold pool->pool_lock. + */ +void __drop_key(struct rxe_pool_entry *elem) { struct rxe_pool *pool = elem->pool; - unsigned long flags; - int ret; - - write_lock_irqsave(&pool->pool_lock, flags); - elem->index = alloc_index(pool); - ret = insert_index(pool, elem); - write_unlock_irqrestore(&pool->pool_lock, flags); - return ret; + rb_erase(&elem->key_node, &pool->key.tree); } -void __rxe_drop_index(struct rxe_pool_entry *elem) +/** + * __rxe_drop_key - Lock and remove key from pool entry + * @elem: pool entry to remove key from + * + * Delete entry from key-red-black tree. + */ +void __rxe_drop_key(struct rxe_pool_entry *elem) { struct rxe_pool *pool = elem->pool; unsigned long flags; write_lock_irqsave(&pool->pool_lock, flags); - clear_bit(elem->index - pool->index.min_index, pool->index.table); - rb_erase(&elem->index_node, &pool->index.tree); + __drop_key(elem); write_unlock_irqrestore(&pool->pool_lock, flags); } -void *rxe_alloc(struct rxe_pool *pool) +/** + * __add_to_pool - Add pool entry allocated elsewhere to pool + * @pool: Pool to add entry to + * @elem: Pool entry to add + * + * Used to add objects allocated in rdma-core to the pool. + * Entry holds a reference to pool to protect the pool pointer + * Takes a reference to the ib_device so it can't be deleted + * before the pool is deleted. + * Returns an error if pool has been deleted or if there are + * too many objects in the pool. + * Caller should hold the pool->pool_lock. + */ +int __add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { - struct rxe_pool_entry *elem; - struct rxe_type_info *info = &rxe_type_info[pool->type]; - u8 *obj = NULL; - unsigned long flags; + if (pool->state != RXE_POOL_STATE_VALID) + return -EINVAL; - might_sleep_if(!(pool->flags & RXE_POOL_ATOMIC)); - - read_lock_irqsave(&pool->pool_lock, flags); - if (pool->state != RXE_POOL_STATE_VALID) { - read_unlock_irqrestore(&pool->pool_lock, flags); - return NULL; - } - kref_get(&pool->ref_cnt); - read_unlock_irqrestore(&pool->pool_lock, flags); + pool_get(pool); if (!ib_device_try_get(&pool->rxe->ib_dev)) goto out_put_pool; @@ -353,85 +474,153 @@ void *rxe_alloc(struct rxe_pool *pool) if (atomic_inc_return(&pool->num_elem) > pool->max_elem) goto out_cnt; - elem = kzalloc(rxe_type_info[pool->type].size, - (pool->flags & RXE_POOL_ATOMIC) ? - GFP_ATOMIC : GFP_KERNEL); - if (!elem) - goto out_cnt; - elem->pool = pool; kref_init(&elem->ref_cnt); - obj = (u8 *)elem - info->elem_offset; - - return obj; + return 0; out_cnt: atomic_dec(&pool->num_elem); ib_device_put(&pool->rxe->ib_dev); out_put_pool: - rxe_pool_put(pool); - return NULL; + pool_put(pool); + return -EINVAL; } +/** + * __rxe_add_to_pool - Lock and add pool entry allocated elsewhere to pool + * @pool: Pool to add entry to + * @elem: Pool entry to add + * + * Same as __add_to_pool except takes the pool_lock. + */ int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem) { + int err; unsigned long flags; - read_lock_irqsave(&pool->pool_lock, flags); + write_lock_irqsave(&pool->pool_lock, flags); + err = __add_to_pool(pool, elem); + write_unlock_irqrestore(&pool->pool_lock, flags); + + return err; +} + +/** + * __alloc - Create a new pool entry + * @pool: Pool to create entry in + * + * Entry holds a reference to pool to protect the pool pointer + * Takes a reference to the ib_device so it can't be deleted + * before the pool is deleted. + * Returns an error pointer if pool has been deleted or if there are + * too many objects in the pool or it cannot get memory for the + * object. + * Caller should hold the pool->pool_lock. + */ +void *__alloc(struct rxe_pool *pool) +{ + struct rxe_type_info *info = &rxe_type_info[pool->type]; + void *obj; + struct rxe_pool_entry *elem; + int ret = 0; + if (pool->state != RXE_POOL_STATE_VALID) { - read_unlock_irqrestore(&pool->pool_lock, flags); - return -EINVAL; + ret = -EINVAL; + goto out; } - kref_get(&pool->ref_cnt); - read_unlock_irqrestore(&pool->pool_lock, flags); - if (!ib_device_try_get(&pool->rxe->ib_dev)) + pool_get(pool); + + if (!ib_device_try_get(&pool->rxe->ib_dev)) { + ret = -EINVAL; goto out_put_pool; + } - if (atomic_inc_return(&pool->num_elem) > pool->max_elem) + if (atomic_inc_return(&pool->num_elem) > pool->max_elem) { + ret = -EINVAL; + goto out_cnt; + } + + obj = kzalloc(info->size, GFP_ATOMIC); + if (!obj) { + ret = -ENOMEM; goto out_cnt; + } + + elem = (struct rxe_pool_entry *)((u8 *)obj + info->elem_offset); elem->pool = pool; kref_init(&elem->ref_cnt); - return 0; + return obj; out_cnt: atomic_dec(&pool->num_elem); ib_device_put(&pool->rxe->ib_dev); out_put_pool: - rxe_pool_put(pool); - return -EINVAL; + pool_put(pool); +out: + return ERR_PTR(ret); } -void rxe_elem_release(struct kref *kref) +/** + * rxe_alloc - Lock and create a new pool entry + * @pool: Pool to create entry in + * + * Same as __alloc except holds the pool_lock. + */ +void *rxe_alloc(struct rxe_pool *pool) { - struct rxe_pool_entry *elem = - container_of(kref, struct rxe_pool_entry, ref_cnt); - struct rxe_pool *pool = elem->pool; + void *obj; + struct rxe_pool_entry *elem; + unsigned long flags; + int err; - if (pool->cleanup) - pool->cleanup(elem); + /* allocate object outside of lock in case it sleeps */ + obj = kzalloc(rxe_type_info[pool->type].size, + (pool->flags & RXE_POOL_ATOMIC) ? + GFP_ATOMIC : GFP_KERNEL); + if (!obj) + goto out; - if (!(pool->flags & RXE_POOL_NO_ALLOC)) - kfree(elem); - atomic_dec(&pool->num_elem); - ib_device_put(&pool->rxe->ib_dev); - rxe_pool_put(pool); + elem = (struct rxe_pool_entry *)((u8 *)obj + + rxe_type_info[pool->type].elem_offset); + + write_lock_irqsave(&pool->pool_lock, flags); + err = __add_to_pool(pool, elem); + write_unlock_irqrestore(&pool->pool_lock, flags); + if (err) { + kfree(obj); + obj = NULL; + } +out: + return obj; } -void *rxe_get_index(struct rxe_pool *pool, u32 index) +/** + * __get_index - Lookup index in pool and return object + * @pool: pool to lookup index in + * @index: index to lookup + * + * Lookup an index in pool's index-red-black tree and retrieve + * pool entry with that index. + * Convert entry address to object address with elem_offset. + * Returns the object address if found else NULL if not or + * an error pointer if the pool has been destroyed. + * Caller should hold the pool->pool_lock. + */ +void *__get_index(struct rxe_pool *pool, u32 index) { - struct rb_node *node = NULL; - struct rxe_pool_entry *elem = NULL; struct rxe_type_info *info = &rxe_type_info[pool->type]; - u8 *obj = NULL; - unsigned long flags; - - read_lock_irqsave(&pool->pool_lock, flags); + struct rb_node *node; + struct rxe_pool_entry *elem = NULL; + void *obj; + int ret = 0; - if (pool->state != RXE_POOL_STATE_VALID) + if (pool->state != RXE_POOL_STATE_VALID) { + ret = -EINVAL; goto out; + } node = pool->index.tree.rb_node; @@ -449,25 +638,59 @@ void *rxe_get_index(struct rxe_pool *pool, u32 index) if (node) { kref_get(&elem->ref_cnt); obj = (u8 *)elem - info->elem_offset; + } else { + obj = NULL; } + + return obj; out: + return ERR_PTR(ret); +} + +/** + * rxe_get_index - Lock and lookup index in pool and return object + * @pool: pool to lookup index in + * @index: index to lookup + * + * Same as __get_index except holds pool_lock. + */ +void *rxe_get_index(struct rxe_pool *pool, u32 index) +{ + void *obj; + unsigned long flags; + + read_lock_irqsave(&pool->pool_lock, flags); + obj = __get_index(pool, index); read_unlock_irqrestore(&pool->pool_lock, flags); + return obj; } -void *rxe_get_key(struct rxe_pool *pool, void *key) +/** + * __get_key - Lookup key in pool and return object + * @pool: pool to lookup key in + * @key: key to lookup + * + * Lookup a key in pool's key-red-black tree and retrieve + * pool entry with that key. + * Convert entry address to object address with elem_offset. + * Returns the object address if found else NULL if not or + * an error pointer if the pool has been destroyed. + * Caller should hold the pool->pool_lock. + */ +void *__get_key(struct rxe_pool *pool, void *key) { - struct rb_node *node = NULL; - struct rxe_pool_entry *elem = NULL; struct rxe_type_info *info = &rxe_type_info[pool->type]; - u8 *obj = NULL; + struct rb_node *node; + struct rxe_pool_entry *elem = NULL; int cmp; - unsigned long flags; - - read_lock_irqsave(&pool->pool_lock, flags); + void *obj; + int ret = 0; - if (pool->state != RXE_POOL_STATE_VALID) + if (pool->state != RXE_POOL_STATE_VALID) { + ret = -EINVAL; goto out; + } node = pool->key.tree.rb_node; @@ -488,9 +711,58 @@ void *rxe_get_key(struct rxe_pool *pool, void *key) if (node) { kref_get(&elem->ref_cnt); obj = (u8 *)elem - info->elem_offset; + } else { + obj = NULL; } + return obj; out: + return ERR_PTR(ret); +} + +/** + * rxe_get_key - Lock and lookup key in pool and return object + * @pool: pool to lookup key in + * @key: key to lookup + * + * Same as __get_key except holds pool_lock. + */ +void *rxe_get_key(struct rxe_pool *pool, void *key) +{ + void *obj; + unsigned long flags; + + read_lock_irqsave(&pool->pool_lock, flags); + obj = __get_key(pool, key); read_unlock_irqrestore(&pool->pool_lock, flags); + return obj; } + +/** + * rxe_elem_release - Delete pool object when ref count goes to zero + * @kref: Reference counter contained in pool entry + * + * If pool has an object cleanup routine call it. + * If entry memory was allocated in pool free it. + */ +void rxe_elem_release(struct kref *kref) +{ + struct rxe_pool_entry *elem = container_of(kref, + struct rxe_pool_entry, ref_cnt); + struct rxe_pool *pool = elem->pool; + struct rxe_type_info *info = &rxe_type_info[pool->type]; + void *obj; + + if (pool->cleanup) + pool->cleanup(elem); + + if (!(pool->flags & RXE_POOL_NO_ALLOC)) { + obj = (u8 *)elem - info->elem_offset; + kfree(obj); + } + + atomic_dec(&pool->num_elem); + ib_device_put(&pool->rxe->ib_dev); + pool_put(pool); +} diff --git a/drivers/infiniband/sw/rxe/rxe_pool.h b/drivers/infiniband/sw/rxe/rxe_pool.h index 5954fae6cf99..9a36b4d904fb 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.h +++ b/drivers/infiniband/sw/rxe/rxe_pool.h @@ -8,13 +8,12 @@ #define RXE_POOL_H #define RXE_POOL_ALIGN (16) -#define RXE_POOL_CACHE_FLAGS (0) enum rxe_pool_flags { RXE_POOL_ATOMIC = BIT(0), RXE_POOL_INDEX = BIT(1), RXE_POOL_KEY = BIT(2), - RXE_POOL_NO_ALLOC = BIT(4), + RXE_POOL_NO_ALLOC = BIT(3), }; enum rxe_elem_type { @@ -39,35 +38,20 @@ struct rxe_type_info { size_t elem_offset; void (*cleanup)(struct rxe_pool_entry *obj); enum rxe_pool_flags flags; - u32 max_index; u32 min_index; + u32 max_index; size_t key_offset; size_t key_size; }; -extern struct rxe_type_info rxe_type_info[]; - enum rxe_pool_state { RXE_POOL_STATE_INVALID, RXE_POOL_STATE_VALID, }; -struct rxe_pool_entry { - struct rxe_pool *pool; - struct kref ref_cnt; - struct list_head list; - - /* only used if keyed */ - struct rb_node key_node; - - /* only used if indexed */ - struct rb_node index_node; - u32 index; -}; - struct rxe_pool { struct rxe_dev *rxe; - rwlock_t pool_lock; /* protects pool add/del/search */ + rwlock_t pool_lock; size_t elem_size; struct kref ref_cnt; void (*cleanup)(struct rxe_pool_entry *obj); @@ -77,8 +61,6 @@ struct rxe_pool { unsigned int max_elem; atomic_t num_elem; - - /* only used if indexed */ struct { struct rb_root tree; unsigned long *table; @@ -87,8 +69,6 @@ struct rxe_pool { u32 max_index; u32 min_index; } index; - - /* only used if keyed */ struct { struct rb_root tree; size_t key_offset; @@ -96,53 +76,96 @@ struct rxe_pool { } key; }; -/* initialize a pool of objects with given limit on - * number of elements. gets parameters from rxe_type_info - * pool elements will be allocated out of a slab cache - */ +struct rxe_pool_entry { + struct rxe_pool *pool; + struct kref ref_cnt; + struct list_head list; + struct rb_node key_node; + struct rb_node index_node; + u32 index; +}; + int rxe_pool_init(struct rxe_dev *rxe, struct rxe_pool *pool, - enum rxe_elem_type type, u32 max_elem); + enum rxe_elem_type type, unsigned int max_elem); -/* free resources from object pool */ void rxe_pool_cleanup(struct rxe_pool *pool); -/* allocate an object from pool */ -void *rxe_alloc(struct rxe_pool *pool); - -int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); +/* + * In the following: + * rxe_xxx() take pool->pool_lock and xxx() do not. + * Sequences of xxx() calls may be protected by + * taking the pool->pool_lock by caller. + * + * __something(elem) take a pool entry argument, and + * something(obj) take a pool object (e.g. cq, mr, etc.) + */ +void __add_index(struct rxe_pool_entry *elem); -#define rxe_add_to_pool(pool, obj) __rxe_add_to_pool(pool, &(obj)->pelem) +#define add_index(obj) __add_index(&(obj)->pelem) -int __rxe_add_index(struct rxe_pool_entry *elem); +void __rxe_add_index(struct rxe_pool_entry *elem); #define rxe_add_index(obj) __rxe_add_index(&(obj)->pelem) +void __drop_index(struct rxe_pool_entry *elem); + +#define drop_index(obj) __drop_index(&(obj)->pelem) + void __rxe_drop_index(struct rxe_pool_entry *elem); #define rxe_drop_index(obj) __rxe_drop_index(&(obj)->pelem) +int __add_key(struct rxe_pool_entry *elem, void *key); + +#define add_key(obj, key) __add_key(&(obj)->pelem, key) + int __rxe_add_key(struct rxe_pool_entry *elem, void *key); #define rxe_add_key(obj, key) __rxe_add_key(&(obj)->pelem, key) +void __drop_key(struct rxe_pool_entry *elem); + +#define drop_key(obj) __drop_key(&(obj)->pelem) + void __rxe_drop_key(struct rxe_pool_entry *elem); #define rxe_drop_key(obj) __rxe_drop_key(&(obj)->pelem) -/* lookup an indexed object from index. takes a reference on object */ +int __add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); + +#define add_to_pool(pool, obj) __add_to_pool(pool, &(obj)->pelem) + +int __rxe_add_to_pool(struct rxe_pool *pool, struct rxe_pool_entry *elem); + +#define rxe_add_to_pool(pool, obj) __rxe_add_to_pool(pool, &(obj)->pelem) + +/* + * The following routines allocate new objects or + * lookup objects from an index or key and return + * the address if found. the rxe_XXX() routines take the + * pool->pool_lock the __xxx() do not. Sequences of + * unprotected pool operations may be protected by + * taking the pool->pool_lock by the caller + */ +void *__alloc(struct rxe_pool *pool); + +void *rxe_alloc(struct rxe_pool *pool); + +void *__get_index(struct rxe_pool *pool, u32 index); + void *rxe_get_index(struct rxe_pool *pool, u32 index); -/* lookup keyed object from key. takes a reference on the object */ +void *__get_key(struct rxe_pool *pool, void *key); + void *rxe_get_key(struct rxe_pool *pool, void *key); -/* cleanup an object when all references are dropped */ -void rxe_elem_release(struct kref *kref); +/* ref counting for pool objects */ +#define rxe_add_ref(obj) kref_get(&(obj)->pelem.ref_cnt) + +#define rxe_drop_ref(obj) kref_put(&(obj)->pelem.ref_cnt, rxe_elem_release) -/* take a reference on an object */ -#define rxe_add_ref(elem) kref_get(&(elem)->pelem.ref_cnt) +#define rxe_read_ref(obj) kref_read(&(obj)->pelem.ref_cnt) -/* drop a reference on an object */ -#define rxe_drop_ref(elem) kref_put(&(elem)->pelem.ref_cnt, rxe_elem_release) +void rxe_elem_release(struct kref *kref); -#define rxe_read_ref(elem) kref_read(&(elem)->pelem.ref_cnt) #endif /* RXE_POOL_H */ From patchwork Thu Oct 1 17:48:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812083 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B9906CB for ; Thu, 1 Oct 2020 17:49:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1A4CC20848 for ; Thu, 1 Oct 2020 17:49:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ca7mI3MB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732998AbgJARtE (ORCPT ); Thu, 1 Oct 2020 13:49:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732956AbgJARtD (ORCPT ); Thu, 1 Oct 2020 13:49:03 -0400 Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3141C0613E2 for ; Thu, 1 Oct 2020 10:49:03 -0700 (PDT) Received: by mail-ot1-x32d.google.com with SMTP id o8so6326193otl.4 for ; Thu, 01 Oct 2020 10:49:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T3Wkw5kn/NaubmiIUIn5KnbZxsuaZTT2cqCWA5dwL8M=; b=Ca7mI3MB0TpFBF1lmA1HjQ08mifKg4u6UKWFy9NTXThIeLKrp9OTS/ppRqK/7hK0ME vwc22FYLxRvE4/eMa7+SY1b1i7cAz9f062ByepTC7TOXkaHCmchDmANVdU7Wt8+oSEsu Wghcni1rLklvqVySxr+0AyqkSVypMqr+GI3kXXosV1Yq562ONBC41cfIt/PjIdcUWdGs +v0+Ih5Qmj0UDjtKwWH5RoyDt1m0kZi9mvMhQMxTcfcjI5G7ijHNL6iqFtNeVSphyORe TDoi3mL6S206HOyeXi0V9iB+mgjZijpG2kCqMYJbLCDfLSmwwwt14gYmGWeykKIDNDes JsUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T3Wkw5kn/NaubmiIUIn5KnbZxsuaZTT2cqCWA5dwL8M=; b=Yd5WWK4J6I8+Rql9Ncfh+ISdFvS87mV5UhXUaUx/9+5WDj8kYJOjV3FSYJH38X1plq TMsxF0UEHTsxWxZQL13/wPU2gCb77I6AyDy7OR0qDiBWMmROMO312CH6VM6i/sgxQZ+A C7SL+SPZYZinONAhswzfYWWc+2sPWg5KNh04O3m6h8tdofeYR+sjmCJTvAPgHpb01NQK +MD3vHfqC3W7eflLi/ZsA2jzS1DUpGVG6V4dyGLK3S2plPHJHhos061uCJwO9hZ5a6yB NyvOvOAJD+KagJ+JLouLJrZFAEB00P66EYHwnrlKTR0xcO4LyxIEso/YI59i4ftAgKze 3eVw== X-Gm-Message-State: AOAM5338xsjoX1QleCOba8Q9kIwKWt6R/ySoGhPVJtVrQLmT+K3tmlbe vEcZB66DSAFOHy6/A+Txhtk= X-Google-Smtp-Source: ABdhPJzepuxuyhjeZc/4XqgNfLFOSeDB2eCS3GtLQVf4hnpqw/aJXzFjCpe/UeZNr/MrluTk5jMIMQ== X-Received: by 2002:a05:6830:1f09:: with SMTP id u9mr5270818otg.175.1601574542997; Thu, 01 Oct 2020 10:49:02 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id o205sm1196061oig.8.2020.10.01.10.49.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:02 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 10/19] rdma_rxe: Add support for ibv_query_device_ex Date: Thu, 1 Oct 2020 12:48:38 -0500 Message-Id: <20201001174847.4268-11-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add code to initialize new struct members in ib_device_attr as place holders. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe.c | 101 ++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_verbs.c | 7 +- 2 files changed, 75 insertions(+), 33 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 8e0f9c489cab..ecc61f960c58 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -40,40 +40,77 @@ void rxe_dealloc(struct ib_device *ib_dev) /* initialize rxe device parameters */ static void rxe_init_device_param(struct rxe_dev *rxe) { - rxe->max_inline_data = RXE_MAX_INLINE_DATA; - - rxe->attr.vendor_id = RXE_VENDOR_ID; - rxe->attr.max_mr_size = RXE_MAX_MR_SIZE; - rxe->attr.page_size_cap = RXE_PAGE_SIZE_CAP; - rxe->attr.max_qp = RXE_MAX_QP; - rxe->attr.max_qp_wr = RXE_MAX_QP_WR; - rxe->attr.device_cap_flags = RXE_DEVICE_CAP_FLAGS; - rxe->attr.max_send_sge = RXE_MAX_SGE; - rxe->attr.max_recv_sge = RXE_MAX_SGE; - rxe->attr.max_sge_rd = RXE_MAX_SGE_RD; - rxe->attr.max_cq = RXE_MAX_CQ; - rxe->attr.max_cqe = (1 << RXE_MAX_LOG_CQE) - 1; - rxe->attr.max_mr = RXE_MAX_MR; - rxe->attr.max_mw = RXE_MAX_MW; - rxe->attr.max_pd = RXE_MAX_PD; - rxe->attr.max_qp_rd_atom = RXE_MAX_QP_RD_ATOM; - rxe->attr.max_res_rd_atom = RXE_MAX_RES_RD_ATOM; - rxe->attr.max_qp_init_rd_atom = RXE_MAX_QP_INIT_RD_ATOM; - rxe->attr.atomic_cap = IB_ATOMIC_HCA; - rxe->attr.max_mcast_grp = RXE_MAX_MCAST_GRP; - rxe->attr.max_mcast_qp_attach = RXE_MAX_MCAST_QP_ATTACH; - rxe->attr.max_total_mcast_qp_attach = RXE_MAX_TOT_MCAST_QP_ATTACH; - rxe->attr.max_ah = RXE_MAX_AH; - rxe->attr.max_srq = RXE_MAX_SRQ; - rxe->attr.max_srq_wr = RXE_MAX_SRQ_WR; - rxe->attr.max_srq_sge = RXE_MAX_SRQ_SGE; - rxe->attr.max_fast_reg_page_list_len = RXE_MAX_FMR_PAGE_LIST_LEN; - rxe->attr.max_pkeys = RXE_MAX_PKEYS; - rxe->attr.local_ca_ack_delay = RXE_LOCAL_CA_ACK_DELAY; - addrconf_addr_eui48((unsigned char *)&rxe->attr.sys_image_guid, - rxe->ndev->dev_addr); + struct ib_device_attr *a = &rxe->attr; + rxe->max_inline_data = RXE_MAX_INLINE_DATA; rxe->max_ucontext = RXE_MAX_UCONTEXT; + + a->atomic_cap = IB_ATOMIC_HCA; + a->cq_caps.max_cq_moderation_count = 0; + a->cq_caps.max_cq_moderation_period = 0; + a->device_cap_flags = RXE_DEVICE_CAP_FLAGS; + a->fw_ver = 0; + a->hca_core_clock = 0; + a->hw_ver = 0; + a->local_ca_ack_delay = RXE_LOCAL_CA_ACK_DELAY; + a->masked_atomic_cap = 0; + a->max_ah = RXE_MAX_AH; + a->max_cqe = (1 << RXE_MAX_LOG_CQE) - 1; + a->max_cq = RXE_MAX_CQ; + a->max_dm_size = 0; + a->max_ee_init_rd_atom = 0; + a->max_ee = 0; + a->max_ee_rd_atom = 0; + a->max_fast_reg_page_list_len = RXE_MAX_FMR_PAGE_LIST_LEN; + a->max_mcast_grp = RXE_MAX_MCAST_GRP; + a->max_mcast_qp_attach = RXE_MAX_MCAST_QP_ATTACH; + a->max_mr = RXE_MAX_MR; + a->max_mr_size = RXE_MAX_MR_SIZE; + a->max_mw = RXE_MAX_MW; + a->max_pd = RXE_MAX_PD; + a->max_pi_fast_reg_page_list_len = 0; + a->max_pkeys = RXE_MAX_PKEYS; + a->max_qp_init_rd_atom = RXE_MAX_QP_INIT_RD_ATOM; + a->max_qp_rd_atom = RXE_MAX_QP_RD_ATOM; + a->max_qp = RXE_MAX_QP; + a->max_qp_wr = RXE_MAX_QP_WR; + a->max_raw_ethy_qp = 0; + a->max_raw_ipv6_qp = 0; + a->max_rdd = 0; + a->max_recv_sge = RXE_MAX_SGE; + a->max_res_rd_atom = RXE_MAX_RES_RD_ATOM; + a->max_send_sge = RXE_MAX_SGE; + a->max_sge_rd = RXE_MAX_SGE_RD; + a->max_sgl_rd = 0; + a->max_srq = RXE_MAX_SRQ; + a->max_srq_sge = RXE_MAX_SRQ_SGE; + a->max_srq_wr = RXE_MAX_SRQ_WR; + a->max_total_mcast_qp_attach = RXE_MAX_TOT_MCAST_QP_ATTACH; + a->max_wq_type_rq = 0; + a->odp_caps.general_caps = 0; + a->odp_caps.per_transport_caps.rc_odp_caps = 0; + a->odp_caps.per_transport_caps.uc_odp_caps = 0; + a->odp_caps.per_transport_caps.ud_odp_caps = 0; + a->odp_caps.per_transport_caps.xrc_odp_caps = 0; + a->page_size_cap = RXE_PAGE_SIZE_CAP; + a->raw_packet_caps = 0; + a->rss_caps.supported_qpts = 0; + a->rss_caps.max_rwq_indirection_tables = 0; + a->rss_caps.max_rwq_indirection_table_size = 0; + a->sig_guard_cap = 0; + a->sig_prot_cap = 0; + a->sys_image_guid = 0; + a->timestamp_mask = 0; + a->tm_caps.max_rndv_hdr_size = 0; + a->tm_caps.max_num_tags = 0; + a->tm_caps.flags = 0; + a->tm_caps.max_ops = 0; + a->tm_caps.max_sge = 0; + a->vendor_id = RXE_VENDOR_ID; + a->vendor_part_id = 0; + + addrconf_addr_eui48((unsigned char *)&a->sys_image_guid, + rxe->ndev->dev_addr); } /* initialize port attributes */ diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 807c9a3b22ea..2695b286cd8e 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1150,7 +1150,8 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) dma_coerce_mask_and_coherent(&dev->dev, dma_get_required_mask(&dev->dev)); - dev->uverbs_cmd_mask = BIT_ULL(IB_USER_VERBS_CMD_GET_CONTEXT) + dev->uverbs_cmd_mask = + BIT_ULL(IB_USER_VERBS_CMD_GET_CONTEXT) | BIT_ULL(IB_USER_VERBS_CMD_CREATE_COMP_CHANNEL) | BIT_ULL(IB_USER_VERBS_CMD_QUERY_DEVICE) | BIT_ULL(IB_USER_VERBS_CMD_QUERY_PORT) @@ -1185,6 +1186,10 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) | BIT_ULL(IB_USER_VERBS_CMD_DEALLOC_MW) ; + dev->uverbs_ex_cmd_mask = + BIT_ULL(IB_USER_VERBS_EX_CMD_QUERY_DEVICE) + ; + ib_set_device_ops(dev, &rxe_dev_ops); err = ib_device_set_netdev(&rxe->ib_dev, rxe->ndev, 1); if (err) From patchwork Thu Oct 1 17:48:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812097 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB29292C for ; Thu, 1 Oct 2020 17:49:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ABFEA20848 for ; Thu, 1 Oct 2020 17:49:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VtUIk8r5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733003AbgJARtL (ORCPT ); Thu, 1 Oct 2020 13:49:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732958AbgJARtE (ORCPT ); Thu, 1 Oct 2020 13:49:04 -0400 Received: from mail-oi1-x242.google.com (mail-oi1-x242.google.com [IPv6:2607:f8b0:4864:20::242]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6536EC0613E2 for ; Thu, 1 Oct 2020 10:49:04 -0700 (PDT) Received: by mail-oi1-x242.google.com with SMTP id 185so6464878oie.11 for ; Thu, 01 Oct 2020 10:49:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OTM7HhrEOp3XZ1ea8pFtsfLCSR50LhwxJ1vlV2pEwrI=; b=VtUIk8r59EqJkXofNIfHGE+xc9fKqiH9rzEeneftln8yKKoJU3w6vIQ2tlnMN6iB0N amxWM79mRypMJdnjACWZNKZLyEYUEafMqfjodhQr1rZ0JpNcH1RG4qdRed5g9JeJO4kp k7ouxtpL5agW722xGvT55ISydyt70tO22HUapE0GakNSeJOo2PxHpxgo/2IHaQiWQFeg ryL0dBWsMNI2meyYBtv6xJKA16BDE41XUEUubOvM2V8+dEeZTulNItNeq29T4o+LQ74P Dd41XdjdxjpE5NusU1QxtvVLbwq3KwWUa82wyYnxUuL107wm4D+3dpt66B0/6rePmpfh kM2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OTM7HhrEOp3XZ1ea8pFtsfLCSR50LhwxJ1vlV2pEwrI=; b=tghSSB/NYLz0hlSNz7IXlsQm3PyX2eIw1Md7lM89vStBTgx0R4WlyJplXB5n0Dd2Zv 9UqzHgMlbOENs/xnXNoD0NP2jPgTzXUf8iXXKAeAgW+KYYR0ptHhmCvKOQZcBgD4rTX8 iR/ORHbG9jSIl67F0VQ+5b+NzhFYqPWw/6snCXqQBNuSawnI8vEJqmu8CprJ9WKTAe9x ZM0LYpFqhivMcX9TwVZwu2nhm4F+oyNc6rTrC93ISVt+VA6b7rzDNuNj1TnHHX8/HP8m 5SnPS5+7Pj4VhHZ8VJofOn5JVe/x4V4psKEier2/ojnwNRuMFRw0Oh3bsAUtnWCrlAQe EnFA== X-Gm-Message-State: AOAM533j9pjK7kyGOzuxWNbmd4eTN4Aim79rk4iIP78CkuM8oDF4QODy oNUZXg7tq9Ap+XmbxAg53VCsyEPbacE= X-Google-Smtp-Source: ABdhPJzw2qmJSk+AH8tOz7dtKNSTLRNYUjkC6nv5f3UauObLhcivyPhG+Jb71ka8OS8Ea98BN5Sm9A== X-Received: by 2002:aca:d845:: with SMTP id p66mr671781oig.47.1601574543823; Thu, 01 Oct 2020 10:49:03 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id k2sm1191923oiw.20.2020.10.01.10.49.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:03 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 11/19] rdma_rxe: Add support for extended CQ operations Date: Thu, 1 Oct 2020 12:48:39 -0500 Message-Id: <20201001174847.4268-12-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add private members to user/kernel wc struct to carry extensions used by cq_ex. Add timestamps on completion. Add ignore overrun support. Add commands to user API bitmasks. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_comp.c | 7 ++++- drivers/infiniband/sw/rxe/rxe_resp.c | 8 +++++- drivers/infiniband/sw/rxe/rxe_verbs.c | 10 ++++--- drivers/infiniband/sw/rxe/rxe_verbs.h | 3 ++- include/uapi/rdma/rdma_user_rxe.h | 38 ++++++++++++++++++++++----- 5 files changed, 52 insertions(+), 14 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 8b81d3b24a8a..72745ffcf118 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -390,7 +390,7 @@ static void make_send_cqe(struct rxe_qp *qp, struct rxe_send_wqe *wqe, wc->byte_len = wqe->dma.length; wc->qp = &qp->ibqp; } else { - struct ib_uverbs_wc *uwc = &cqe->uibwc; + struct rxe_uverbs_wc *uwc = &cqe->ruwc; uwc->wr_id = wqe->wr.wr_id; uwc->status = wqe->status; @@ -400,6 +400,11 @@ static void make_send_cqe(struct rxe_qp *qp, struct rxe_send_wqe *wqe, uwc->wc_flags = IB_WC_WITH_IMM; uwc->byte_len = wqe->dma.length; uwc->qp_num = qp->ibqp.qp_num; + if (qp->scq->flags & + IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION) { + uwc->timestamp = (u64)ktime_get(); + uwc->realtime = (u64)ktime_get_real(); + } } } diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 2aae311f9063..58d3783063bb 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -887,7 +887,7 @@ static enum resp_states do_complete(struct rxe_qp *qp, { struct rxe_cqe cqe; struct ib_wc *wc = &cqe.ibwc; - struct ib_uverbs_wc *uwc = &cqe.uibwc; + struct rxe_uverbs_wc *uwc = &cqe.ruwc; struct rxe_recv_wqe *wqe = qp->resp.wqe; struct rxe_dev *rxe = to_rdev(qp->ibqp.device); @@ -943,6 +943,12 @@ static enum resp_states do_complete(struct rxe_qp *qp, uwc->src_qp = deth_sqp(pkt); uwc->port_num = qp->attr.port_num; + + if (qp->rcq->flags & + IB_UVERBS_CQ_FLAGS_TIMESTAMP_COMPLETION) { + uwc->timestamp = (u64)ktime_get(); + uwc->realtime = (u64)ktime_get_real(); + } } else { struct sk_buff *skb = PKT_TO_SKB(pkt); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 2695b286cd8e..6be0aa9cf1f3 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -749,7 +749,8 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, return err; } -static int rxe_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, +static int rxe_create_cq(struct ib_cq *ibcq, + const struct ib_cq_init_attr *attr, struct ib_udata *udata) { int err; @@ -764,13 +765,12 @@ static int rxe_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, uresp = udata->outbuf; } - if (attr->flags) - return -EINVAL; - err = rxe_cq_chk_attr(rxe, NULL, attr->cqe, attr->comp_vector); if (err) return err; + cq->flags = attr->flags; + err = rxe_cq_from_init(rxe, cq, attr->cqe, attr->comp_vector, udata, uresp); if (err) @@ -1188,6 +1188,8 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) dev->uverbs_ex_cmd_mask = BIT_ULL(IB_USER_VERBS_EX_CMD_QUERY_DEVICE) + | BIT_ULL(IB_USER_VERBS_EX_CMD_CREATE_CQ) + | BIT_ULL(IB_USER_VERBS_EX_CMD_MODIFY_CQ) ; ib_set_device_ops(dev, &rxe_dev_ops); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index e8d9c8388345..54429d8a06fb 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -53,7 +53,7 @@ struct rxe_ah { struct rxe_cqe { union { struct ib_wc ibwc; - struct ib_uverbs_wc uibwc; + struct rxe_uverbs_wc ruwc; }; }; @@ -62,6 +62,7 @@ struct rxe_cq { struct rxe_pool_entry pelem; struct rxe_queue *queue; spinlock_t cq_lock; + u32 flags; u8 notify; bool is_dying; int is_user; diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index d49125682359..95352e050ab4 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -98,29 +98,27 @@ struct rxe_send_wr { __aligned_u64 length; union { __u32 mr_index; - __aligned_u64 reserved1; + __aligned_u64 pad1; }; union { __u32 mw_index; - __aligned_u64 reserved2; + __aligned_u64 pad2; }; __u32 rkey; __u32 access; __u32 flags; } umw; - /* The following are only used by the kernel - * and are not part of the uapi - */ + /* below are only used by the kernel */ struct { __aligned_u64 addr; __aligned_u64 length; union { struct ib_mr *mr; - __aligned_u64 reserved1; + __aligned_u64 reserved1; }; union { struct ib_mw *mw; - __aligned_u64 reserved2; + __aligned_u64 reserved2; }; __u32 rkey; __u32 access; @@ -184,6 +182,32 @@ struct rxe_recv_wqe { struct rxe_dma_info dma; }; +struct rxe_uverbs_wc { + /* keep these the same as ib_uverbs_wc */ + __aligned_u64 wr_id; + __u32 status; + __u32 opcode; + __u32 vendor_err; + __u32 byte_len; + union { + __be32 imm_data; + __u32 invalidate_rkey; + } ex; + __u32 qp_num; + __u32 src_qp; + __u32 wc_flags; + __u16 pkey_index; + __u16 slid; + __u8 sl; + __u8 dlid_path_bits; + __u8 port_num; + __u8 reserved; + + /* any extras go here */ + __aligned_u64 timestamp; + __aligned_u64 realtime; +}; + struct rxe_create_cq_resp { struct mminfo mi; }; From patchwork Thu Oct 1 17:48:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812089 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39F1C92C for ; Thu, 1 Oct 2020 17:49:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1D4BE20759 for ; Thu, 1 Oct 2020 17:49:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nmtwCFXw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733011AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732940AbgJARtG (ORCPT ); Thu, 1 Oct 2020 13:49:06 -0400 Received: from mail-oi1-x244.google.com (mail-oi1-x244.google.com [IPv6:2607:f8b0:4864:20::244]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B58AC0613E3 for ; Thu, 1 Oct 2020 10:49:05 -0700 (PDT) Received: by mail-oi1-x244.google.com with SMTP id x14so6476093oic.9 for ; Thu, 01 Oct 2020 10:49:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=igfUxFQ4CgEZ4GzuGjFQLf1aWZAvqukG+0VbuhxdjoY=; b=nmtwCFXwZzwLcTlmCpj+V82j4S4IYLxJHDQZLifq1JIGssjEkolGpJtsnVNVdPKGiK 77+c/LpFdswiNxwe38RSq8t2P2AZpAJE/K3Z61ocGXxGKtQoz1q+8GesV+492tgDmWJ/ /tykEH7hVckZeEoxwv9Ph0brngW2X8+NA6aIL6H8Onh214lJk4CeKZJy1awuy1f0aOnu cN/onq6homiORysRDSRnQaL+3w/jnpZVWehOB9lgLS0HkZRUTGfrxlc8TYt3UaHTHjLx +IcGD3+CkBB9Zxc3zwd3IpqRpzn3nHa+QzbkmBtY7lf3bugLzs6fS4X7Z/V8YbeJmn4U vuDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=igfUxFQ4CgEZ4GzuGjFQLf1aWZAvqukG+0VbuhxdjoY=; b=sMHZdYcQkoNlliBdxJ2Xnex0ZCfd8wAhbvrydb0/8YvwkChYI0AXDLmGuzQPxTkhpt zKNx3En2tyXzBzite7E3j+lfjPt5BZTkNqnBE+ULO1MQwY1O6ufd/TK10eRN/AT9Vc5G 0pYjeuhOuZhWlPn46bjZo1xnELosAtm4RDgcg5pyHTvMh88i0ioxBMnTFxSWbsz9tqR/ dgSp7ndRlvCW8Bf8ExeQi5Evnogpsyh9H2tUvXvcs/+x9gDp2xs/v2KULkegFpCx+xAF 5ad5+/juTb22dYgVhge7vfDS6R5UDNhlKinW3QLedE9SqPVcD5676xOz4X08BRqSiihp yyyw== X-Gm-Message-State: AOAM530WpSA9z+7zOu+5+xcPENYD90rFyP+S/cAsI6LFykfbTLoo9DRv MiaQWMXHxJdxsnrM8YTntOEFjqCzL6M= X-Google-Smtp-Source: ABdhPJztfxyKxe2IT0rRg+M4ZdFqLlGuwlqZN1bhB0P70pUWTJfmRBIYNawF3NBnCNIr4+1YiY3NAA== X-Received: by 2002:a05:6808:95:: with SMTP id s21mr738109oic.120.1601574544657; Thu, 01 Oct 2020 10:49:04 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id z16sm1365866oth.43.2020.10.01.10.49.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:04 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 12/19] rdma_rxe: Add support for extended QP operations Date: Thu, 1 Oct 2020 12:48:40 -0500 Message-Id: <20201001174847.4268-13-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add bits to user api command bitmask. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 6be0aa9cf1f3..4de263e41b07 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1188,6 +1188,8 @@ int rxe_register_device(struct rxe_dev *rxe, const char *ibdev_name) dev->uverbs_ex_cmd_mask = BIT_ULL(IB_USER_VERBS_EX_CMD_QUERY_DEVICE) + | BIT_ULL(IB_USER_VERBS_EX_CMD_CREATE_QP) + | BIT_ULL(IB_USER_VERBS_EX_CMD_MODIFY_QP) | BIT_ULL(IB_USER_VERBS_EX_CMD_CREATE_CQ) | BIT_ULL(IB_USER_VERBS_EX_CMD_MODIFY_CQ) ; From patchwork Thu Oct 1 17:48:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1397813B2 for ; Thu, 1 Oct 2020 17:49:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ECB7220848 for ; Thu, 1 Oct 2020 17:49:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VYLtb3/D" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732956AbgJARtL (ORCPT ); Thu, 1 Oct 2020 13:49:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732999AbgJARtG (ORCPT ); Thu, 1 Oct 2020 13:49:06 -0400 Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CBA0C0613E4 for ; Thu, 1 Oct 2020 10:49:06 -0700 (PDT) Received: by mail-ot1-x342.google.com with SMTP id a2so6287832otr.11 for ; Thu, 01 Oct 2020 10:49:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FYwKkNRp9uCds/YjAuYWixK/yGwzj9/mfnhHuXBMfrg=; b=VYLtb3/D+7tOidUZM3PdLF2E6Wnqnd8TRbW1EaWUskXAEeDmCxPzJ5OffvCtZp7xjY Sr2xOQKIOQYFGHRLUuZ2tC2+B4Wk3StQWIdapbtAq6AGm3u+tfUn5nR2/1AczSOQJWXb i4PTnA5gFH7UMQfClFW8PXwdbubIU3PfwL33xBtBrrHEzUQOXQ6zwy1vQdSE/bNU3TAo QQDa39hj4JtqS3bNHzm/UvHEi49l4ELdEb/uNzvPT2VXCH1H/rjRSpw2JE5N+HbkXeUO UbbdjRzwceR7hsOS1RB6Q1NuPQ2VWYRu1QoJ0dqYrPPHEgLX0OQdszuBRLByICrK/sbn 781g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FYwKkNRp9uCds/YjAuYWixK/yGwzj9/mfnhHuXBMfrg=; b=HTIoeZe4SpZYfZwj2OjmWlMIq0pu2qdEz4+99SaVqLA6Qwr39nryp4W5ex50HotOwD zIl3Vbd/Xlj6J3O4chZ5wmZLH2kz3yBLtHIDjEHXyokE+H94++BPJrtEnzp7ROjtu9IW ry/kN/VAQ4yl3K5I3IdFXhz76BLXkSkKlqBTEzBu+dk/Xl67l+3vcs2a7hzBb2DLES4b RGERUN9eG66/Dr21JmnNpP0oBTZRF8IAm8w+vUVwv0OgZM2ACqjrJOLu4nUdDR/noFe4 44fp5/u6Xd0W87KT1ya98uBxmyJWGw7YwT5RHJTcDGZCrsd90anF07exlomh34jdahUP Qweg== X-Gm-Message-State: AOAM5319B5aQgz+hcazO6NpGwni2aPLySi1H0WzBiaIOruztH30paNvm emjlllIjIIYhKDCrOwNYXkE= X-Google-Smtp-Source: ABdhPJwIq0WDTrkEYPUeKpKBaZ5J1Ke9fGow7QQOuWRT39tescTOYa946rigBb9l8567mZgY6v2bPA== X-Received: by 2002:a9d:2382:: with SMTP id t2mr5392100otb.1.1601574545419; Thu, 01 Oct 2020 10:49:05 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id e30sm1374758otf.49.2020.10.01.10.49.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:05 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 13/19] rdma_rxe: Fix mcast group allocation bug Date: Thu, 1 Oct 2020 12:48:41 -0500 Message-Id: <20201001174847.4268-14-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This patch does the following: - Cleans up multicast group create to use an atomic sequence of lookup followed by allocate. This fixes an error that occurred when two QPs were attempting to attach to the same mcast address at the same time. - Fixes a bug in rxe_mcast_get_grp not initializing err = 0. If the group is found in get_key the routine can return an uninitialized value to caller. - Changes the variable elem to mce (for mcast elem). This is less likely to confuse readers because of the ambiguity with elem as pool element which is also used. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mcast.c | 108 ++++++++++++++------------ 1 file changed, 60 insertions(+), 48 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c index b09c6594045a..541cc68a8a94 100644 --- a/drivers/infiniband/sw/rxe/rxe_mcast.c +++ b/drivers/infiniband/sw/rxe/rxe_mcast.c @@ -7,44 +7,56 @@ #include "rxe.h" #include "rxe_loc.h" -int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, - struct rxe_mc_grp **grp_p) +/* caller should hold mc_grp_pool->pool_lock */ +static struct rxe_mc_grp *create_grp(struct rxe_dev *rxe, + struct rxe_pool *pool, + union ib_gid *mgid) { int err; struct rxe_mc_grp *grp; - if (rxe->attr.max_mcast_qp_attach == 0) { - err = -EINVAL; - goto err1; - } - - grp = rxe_get_key(&rxe->mc_grp_pool, mgid); - if (grp) - goto done; - - grp = rxe_alloc(&rxe->mc_grp_pool); - if (!grp) { - err = -ENOMEM; - goto err1; - } + grp = __alloc(&rxe->mc_grp_pool); + if (unlikely(!grp)) + return NULL; INIT_LIST_HEAD(&grp->qp_list); spin_lock_init(&grp->mcg_lock); grp->rxe = rxe; - - rxe_add_key(grp, mgid); + add_key(grp, mgid); err = rxe_mcast_add(rxe, mgid); - if (err) - goto err2; + if (unlikely(err)) { + drop_key(grp); + rxe_drop_ref(grp); + return NULL; + } + return grp; +} + +/* atomically lookup or create mc group */ +int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, + struct rxe_mc_grp **grp_p) +{ + int err = 0; + struct rxe_mc_grp *grp; + struct rxe_pool *pool = &rxe->mc_grp_pool; + unsigned long flags; + + if (unlikely(rxe->attr.max_mcast_qp_attach == 0)) + return -EINVAL; + + write_lock_irqsave(&pool->pool_lock, flags); + grp = __get_key(pool, mgid); + if (grp) + goto done; + + grp = create_grp(rxe, pool, mgid); + if (unlikely(!grp)) + err = -ENOMEM; done: + write_unlock_irqrestore(&pool->pool_lock, flags); *grp_p = grp; - return 0; - -err2: - rxe_drop_ref(grp); -err1: return err; } @@ -52,13 +64,13 @@ int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, struct rxe_mc_grp *grp) { int err; - struct rxe_mc_elem *elem; + struct rxe_mc_elem *mce; /* check to see of the qp is already a member of the group */ spin_lock_bh(&qp->grp_lock); spin_lock_bh(&grp->mcg_lock); - list_for_each_entry(elem, &grp->qp_list, qp_list) { - if (elem->qp == qp) { + list_for_each_entry(mce, &grp->qp_list, qp_list) { + if (mce->qp == qp) { err = 0; goto out; } @@ -69,8 +81,8 @@ int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, goto out; } - elem = rxe_alloc(&rxe->mc_elem_pool); - if (!elem) { + mce = rxe_alloc(&rxe->mc_elem_pool); + if (!mce) { err = -ENOMEM; goto out; } @@ -79,11 +91,11 @@ int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, rxe_add_ref(grp); grp->num_qp++; - elem->qp = qp; - elem->grp = grp; + mce->qp = qp; + mce->grp = grp; - list_add(&elem->qp_list, &grp->qp_list); - list_add(&elem->grp_list, &qp->grp_list); + list_add(&mce->qp_list, &grp->qp_list); + list_add(&mce->grp_list, &qp->grp_list); err = 0; out: @@ -96,7 +108,7 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, union ib_gid *mgid) { struct rxe_mc_grp *grp; - struct rxe_mc_elem *elem, *tmp; + struct rxe_mc_elem *mce, *tmp; grp = rxe_get_key(&rxe->mc_grp_pool, mgid); if (!grp) @@ -105,15 +117,15 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, spin_lock_bh(&qp->grp_lock); spin_lock_bh(&grp->mcg_lock); - list_for_each_entry_safe(elem, tmp, &grp->qp_list, qp_list) { - if (elem->qp == qp) { - list_del(&elem->qp_list); - list_del(&elem->grp_list); + list_for_each_entry_safe(mce, tmp, &grp->qp_list, qp_list) { + if (mce->qp == qp) { + list_del(&mce->qp_list); + list_del(&mce->grp_list); grp->num_qp--; spin_unlock_bh(&grp->mcg_lock); spin_unlock_bh(&qp->grp_lock); - rxe_drop_ref(elem); + rxe_drop_ref(mce); rxe_drop_ref(grp); /* ref held by QP */ rxe_drop_ref(grp); /* ref from get_key */ return 0; @@ -130,7 +142,7 @@ int rxe_mcast_drop_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, void rxe_drop_all_mcast_groups(struct rxe_qp *qp) { struct rxe_mc_grp *grp; - struct rxe_mc_elem *elem; + struct rxe_mc_elem *mce; while (1) { spin_lock_bh(&qp->grp_lock); @@ -138,24 +150,24 @@ void rxe_drop_all_mcast_groups(struct rxe_qp *qp) spin_unlock_bh(&qp->grp_lock); break; } - elem = list_first_entry(&qp->grp_list, struct rxe_mc_elem, + mce = list_first_entry(&qp->grp_list, struct rxe_mc_elem, grp_list); - list_del(&elem->grp_list); + list_del(&mce->grp_list); spin_unlock_bh(&qp->grp_lock); - grp = elem->grp; + grp = mce->grp; spin_lock_bh(&grp->mcg_lock); - list_del(&elem->qp_list); + list_del(&mce->qp_list); grp->num_qp--; spin_unlock_bh(&grp->mcg_lock); rxe_drop_ref(grp); - rxe_drop_ref(elem); + rxe_drop_ref(mce); } } -void rxe_mc_cleanup(struct rxe_pool_entry *arg) +void rxe_mc_cleanup(struct rxe_pool_entry *pelem) { - struct rxe_mc_grp *grp = container_of(arg, typeof(*grp), pelem); + struct rxe_mc_grp *grp = container_of(pelem, struct rxe_mc_grp, pelem); struct rxe_dev *rxe = grp->rxe; rxe_drop_key(grp); From patchwork Thu Oct 1 17:48:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812093 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF7F56CB for ; Thu, 1 Oct 2020 17:49:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C292320759 for ; Thu, 1 Oct 2020 17:49:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JdJeV2FC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733012AbgJARtL (ORCPT ); Thu, 1 Oct 2020 13:49:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732956AbgJARtH (ORCPT ); Thu, 1 Oct 2020 13:49:07 -0400 Received: from mail-oi1-x244.google.com (mail-oi1-x244.google.com [IPv6:2607:f8b0:4864:20::244]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBEE2C0613E5 for ; Thu, 1 Oct 2020 10:49:06 -0700 (PDT) Received: by mail-oi1-x244.google.com with SMTP id x14so6476186oic.9 for ; Thu, 01 Oct 2020 10:49:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q/EVf/acTWOTeXlExnzmqEgiMNsr5DangUbYAmxCMVM=; b=JdJeV2FCKs8be345iDNAgWKbm7vq7HOfRnoGBXA++/UphYiB/LPhw9vYCFQV3jbE1y g7jDzqTmiEICtYoSB6f/BriReeE6G/W/plTnveXsGqzhXDd+IFJ5ksFwREbYx959pNGz EX4HVjdcRtJqrXOO/nwslvbcygEFTQepeqWuj0cSoqNssu3NPrGytJvP6el85OaTvxMi 5iTJvFjyegFSnwiHWLQV0S8P0euE08ftecB5pbis/WO2XYVy/KvYUoEgOTjDjqc5eFQq B/oOnZAvv9o6UloJLrO3u+wTrRN1R8WdUbUaZ5gDlUvgt02t5pEOnD5uyCWQj0RVEKNO fKtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q/EVf/acTWOTeXlExnzmqEgiMNsr5DangUbYAmxCMVM=; b=kphEZ5wg1nV4od13zya/G8ZYvvQaXsg9rYEirhEzCkzpgtrYJmmk6AiApYToNnj7C7 F8+lTyuZpxJmxprsif7TuVlYJNO7qzqUhktb8NgeDm2Ifvpgn7qBd9dHON+mVeBZE1Tv QVgCv4A5zliI6r3lX1AZfqISTZBhOZ+Zhxhk69e0IVw6UIxAVqBJoTOQhyVOO7ABw0Uq vayXwZOPKxrRi3MoRCBULSPRTAeB3jqJWI+hLqUOat7YztMaUg9oZnLrYvtUYiGIGa+7 YYHg0Iem3L8gOcU+EI2QHPRIwK/J8wAkyxG+lBHcdO/J7ET6T6viO1Us0hLAZbejwTIX kPsQ== X-Gm-Message-State: AOAM531UchODLP5oTv+jeZqVrwFtTXwMoW0ZWpY0S/7ybDLPwijdlewd sqUwjSRIR/vUwv232TCpnS8= X-Google-Smtp-Source: ABdhPJzsdGIjC7NAKQmGmOhLJK0ltXlNSyyzq4MTPF4rADxxa+v22AO5bSzvkWUx3Cg/EQ2Vz2q7bA== X-Received: by 2002:aca:abc7:: with SMTP id u190mr679098oie.146.1601574546192; Thu, 01 Oct 2020 10:49:06 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id q184sm1492910ooq.40.2020.10.01.10.49.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:05 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 14/19] rdma_rxe: Fix bugs in the multicast receive path Date: Thu, 1 Oct 2020 12:48:42 -0500 Message-Id: <20201001174847.4268-15-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This patch does the following - Fix a bug in rxe_rcv. The current code calls rxe_match_dgid which checks to see if the destination ip address (dgid) matches one of the addresses in the gid table. This is ok for unicast adfdresses but not mcast addresses. Because of this all mcast packets were previously dropped. - Fix a bug in rxe_rcv_mcast_pkt. The current code is just wrong. It assumed that it could pass the same skb to rxe_rcv_pkt changing the qp pointer as it went when multiple QPs were attached to the same mcast address. In fact each QP needs a separate clone of the skb which it will delete later. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_recv.c | 61 ++++++++++++++++------------ 1 file changed, 36 insertions(+), 25 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_recv.c b/drivers/infiniband/sw/rxe/rxe_recv.c index e14cb20c4596..dbb623ec5eaa 100644 --- a/drivers/infiniband/sw/rxe/rxe_recv.c +++ b/drivers/infiniband/sw/rxe/rxe_recv.c @@ -233,6 +233,8 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb) struct rxe_mc_elem *mce; struct rxe_qp *qp; union ib_gid dgid; + struct sk_buff *per_qp_skb; + struct rxe_pkt_info *per_qp_pkt; int err; if (skb->protocol == htons(ETH_P_IP)) @@ -261,42 +263,36 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb) if (err) continue; - /* if *not* the last qp in the list - * increase the users of the skb then post to the next qp + /* for all but the last qp create a new clone of the + * skb and pass to the qp. + * This effectively reverts an earlier change + * which did not work. The pkt struct is contained + * in the skb so each time you changed pkt you also + * changed all the earlier pkts as well. Caused a mess. */ if (mce->qp_list.next != &mcg->qp_list) - skb_get(skb); + per_qp_skb = skb_clone(skb, GFP_ATOMIC); + else + per_qp_skb = skb; - pkt->qp = qp; + per_qp_pkt = SKB_TO_PKT(per_qp_skb); + per_qp_pkt->qp = qp; rxe_add_ref(qp); - rxe_rcv_pkt(pkt, skb); + rxe_rcv_pkt(per_qp_pkt, per_qp_skb); } spin_unlock_bh(&mcg->mcg_lock); - - rxe_drop_ref(mcg); /* drop ref from rxe_get_key. */ + rxe_drop_ref(mcg); /* drop ref from rxe_pool_get_key. */ + return; err1: kfree_skb(skb); } -static int rxe_match_dgid(struct rxe_dev *rxe, struct sk_buff *skb) +static int rxe_match_dgid(struct rxe_dev *rxe, struct sk_buff *skb, + union ib_gid *pdgid) { - struct rxe_pkt_info *pkt = SKB_TO_PKT(skb); const struct ib_gid_attr *gid_attr; - union ib_gid dgid; - union ib_gid *pdgid; - - if (pkt->mask & RXE_LOOPBACK_MASK) - return 0; - - if (skb->protocol == htons(ETH_P_IP)) { - ipv6_addr_set_v4mapped(ip_hdr(skb)->daddr, - (struct in6_addr *)&dgid); - pdgid = &dgid; - } else { - pdgid = (union ib_gid *)&ipv6_hdr(skb)->daddr; - } gid_attr = rdma_find_gid_by_port(&rxe->ib_dev, pdgid, IB_GID_TYPE_ROCE_UDP_ENCAP, @@ -314,17 +310,32 @@ void rxe_rcv(struct sk_buff *skb) int err; struct rxe_pkt_info *pkt = SKB_TO_PKT(skb); struct rxe_dev *rxe = pkt->rxe; + union ib_gid dgid; + union ib_gid *pdgid; __be32 *icrcp; u32 calc_icrc, pack_icrc; + int is_mc; pkt->offset = 0; if (unlikely(skb->len < pkt->offset + RXE_BTH_BYTES)) goto drop; - if (rxe_match_dgid(rxe, skb) < 0) { - pr_warn_ratelimited("failed matching dgid\n"); - goto drop; + if (skb->protocol == htons(ETH_P_IP)) { + ipv6_addr_set_v4mapped(ip_hdr(skb)->daddr, + (struct in6_addr *)&dgid); + pdgid = &dgid; + } else { + pdgid = (union ib_gid *)&ipv6_hdr(skb)->daddr; + } + + is_mc = rdma_is_multicast_addr((struct in6_addr *)pdgid); + + if (!(pkt->mask & RXE_LOOPBACK_MASK) && !is_mc) { + if (rxe_match_dgid(rxe, skb, pdgid) < 0) { + pr_warn_ratelimited("failed matching dgid\n"); + goto drop; + } } pkt->opcode = bth_opcode(pkt); From patchwork Thu Oct 1 17:48:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812099 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04BE713B2 for ; Thu, 1 Oct 2020 17:49:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA2C5208B6 for ; Thu, 1 Oct 2020 17:49:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZArVcGV5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732999AbgJARtL (ORCPT ); Thu, 1 Oct 2020 13:49:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733003AbgJARtH (ORCPT ); Thu, 1 Oct 2020 13:49:07 -0400 Received: from mail-oi1-x241.google.com (mail-oi1-x241.google.com [IPv6:2607:f8b0:4864:20::241]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DB2BC0613E6 for ; Thu, 1 Oct 2020 10:49:07 -0700 (PDT) Received: by mail-oi1-x241.google.com with SMTP id 185so6465077oie.11 for ; Thu, 01 Oct 2020 10:49:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IDhkTZQlDlaso3OBshhU5cB4uZ8LfUV5njrw8rLn0xo=; b=ZArVcGV57zYG54G8FREMW9ey3Tm8CA+uZ260dWzk3tcFZlNeOkOindXOkZ9zwLZ7eB F932fpIBUY+7rvhmwh8hrAG+8RfSnpOoETv/Jiz6tCUPtdcGcF9H3eyQDsR95tZHget4 A/u1oP6tWqkocEH4Ycr7w6iL1obKdATVvGraJx1AHL63l1yuxLs1Kan0hAWnF2Av4De3 aX+w4hTv2FTUGYFNPKbQ7ptzWLdHFopqUMdSCzI0t/c/Qe5mtgD4bdmTH/lWw2qHgxm8 O0Gxtzz+yd7Hnwu/wTaswcPkje0PUI0cYOEhJWvquRJ86S61xPPL+3Ch8j/hfjblT2i0 t1KA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IDhkTZQlDlaso3OBshhU5cB4uZ8LfUV5njrw8rLn0xo=; b=TopyDaWgd4NYMNrgjPBAZifFFJ0hqxS8tS5OeJyim15rEMHn9jKYAk9oUAWPW4rmp/ j/90RMWrlEJMpZvqukOAFNVcE52eTGxf6LZZt9vP7vPFP13Lczf8dPlW5d7jcyade3+R 1KEubkW2EFhw3fSns7WCJHQUm5UyGmqUmdq2o0W+fknvuLUb1+CwwZBIeiNHUB3ITFUE gMNb3W8R2qkPKwTHYO1YgkwqNDAFBcncYPSJAox3GdeDG4/gKwBMe3U2DawLmxPBFr0G ccTXCSUIVLqVms+i/MB/OirCbMOEJNtVnsK95Nn88RWThUh6yJ+Oc1woVaS8LaDY0EFI 4hAw== X-Gm-Message-State: AOAM530abvqx8iRdk2FnnIw0HKgVIVxlrYZstQlmHD6dTuj9HGzDnSX6 vAa8sKAmcBr2ZrHb+X+YJPg= X-Google-Smtp-Source: ABdhPJzD9Wbejjav7fs788hyVk1mh84/+OMqMrHpXu+Pcffr8tY6xELaD78JcIUL5B/jXYhrBEWytw== X-Received: by 2002:a54:4619:: with SMTP id p25mr739247oip.18.1601574546980; Thu, 01 Oct 2020 10:49:06 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id z5sm1364942otp.16.2020.10.01.10.49.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:06 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 15/19] rdma_rxe: handle ERR_PTR returns from pool Date: Thu, 1 Oct 2020 12:48:43 -0500 Message-Id: <20201001174847.4268-16-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org v7: - rxe_pool.c has been extended to return errors from alloc, get_index and get_key using ERR_PTR. Add code to detect these as needed. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mcast.c | 28 ++++++++++++++++++--------- drivers/infiniband/sw/rxe/rxe_mr.c | 2 +- drivers/infiniband/sw/rxe/rxe_mw.c | 11 +++++++++++ drivers/infiniband/sw/rxe/rxe_pool.c | 6 +++--- drivers/infiniband/sw/rxe/rxe_recv.c | 4 +++- drivers/infiniband/sw/rxe/rxe_req.c | 4 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 8 ++++---- drivers/infiniband/sw/rxe/rxe_verbs.c | 16 +++++++-------- 8 files changed, 51 insertions(+), 28 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mcast.c b/drivers/infiniband/sw/rxe/rxe_mcast.c index 541cc68a8a94..0dd8abefc13f 100644 --- a/drivers/infiniband/sw/rxe/rxe_mcast.c +++ b/drivers/infiniband/sw/rxe/rxe_mcast.c @@ -16,8 +16,8 @@ static struct rxe_mc_grp *create_grp(struct rxe_dev *rxe, struct rxe_mc_grp *grp; grp = __alloc(&rxe->mc_grp_pool); - if (unlikely(!grp)) - return NULL; + if (IS_ERR(grp)) + return grp; INIT_LIST_HEAD(&grp->qp_list); spin_lock_init(&grp->mcg_lock); @@ -28,7 +28,7 @@ static struct rxe_mc_grp *create_grp(struct rxe_dev *rxe, if (unlikely(err)) { drop_key(grp); rxe_drop_ref(grp); - return NULL; + return ERR_PTR(err); } return grp; @@ -43,20 +43,30 @@ int rxe_mcast_get_grp(struct rxe_dev *rxe, union ib_gid *mgid, struct rxe_pool *pool = &rxe->mc_grp_pool; unsigned long flags; - if (unlikely(rxe->attr.max_mcast_qp_attach == 0)) - return -EINVAL; + if (unlikely(rxe->attr.max_mcast_qp_attach == 0)) { + err = -EINVAL; + goto err; + } write_lock_irqsave(&pool->pool_lock, flags); grp = __get_key(pool, mgid); + if (IS_ERR(grp)) + goto err_ptr; if (grp) goto done; grp = create_grp(rxe, pool, mgid); - if (unlikely(!grp)) - err = -ENOMEM; + if (IS_ERR(grp)) + goto err_ptr; done: write_unlock_irqrestore(&pool->pool_lock, flags); *grp_p = grp; + return 0; + +err_ptr: + write_unlock_irqrestore(&pool->pool_lock, flags); + err = PTR_ERR(grp); +err: return err; } @@ -82,8 +92,8 @@ int rxe_mcast_add_grp_elem(struct rxe_dev *rxe, struct rxe_qp *qp, } mce = rxe_alloc(&rxe->mc_elem_pool); - if (!mce) { - err = -ENOMEM; + if (IS_ERR(mce)) { + err = PTR_ERR(mce); goto out; } diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index c3ad89d85360..e3eb0f4bb2a0 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -386,7 +386,7 @@ static struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 lkey) struct rxe_dev *rxe = to_rdev(pd->ibpd.device); mr = rxe_get_key(&rxe->mr_pool, &lkey); - if (!mr) + if (IS_ERR(mr) || !mr) return NULL; if (unlikely((mr->ibmr.lkey != lkey) || (mr->pd != pd) || diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index 22d3570ac184..d2d09502a28d 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -217,6 +217,8 @@ static int do_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, rkey = mw->ibmw.rkey; new_rkey = (rkey & 0xffffff00) | (wqe->wr.wr.umw.rkey & 0x000000ff); duplicate_mw = rxe_get_key(&rxe->mw_pool, &new_rkey); + if (IS_ERR(duplicate_mw)) + return PTR_ERR(duplicate_mw); if (duplicate_mw) { pr_err_once("new MW key is a duplicate, try another\n"); rxe_drop_ref(duplicate_mw); @@ -260,14 +262,23 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) if (qp->is_user) { mw = rxe_get_index(&rxe->mw_pool, wqe->wr.wr.umw.mw_index); + if (IS_ERR(mw)) { + ret = PTR_ERR(mw); + goto err1; + } if (!mw) { pr_err_once("mw with index = %d not found\n", wqe->wr.wr.umw.mw_index); ret = -EINVAL; goto err1; } + mr = rxe_get_index(&rxe->mr_pool, wqe->wr.wr.umw.mr_index); + if (IS_ERR(mr)) { + ret = PTR_ERR(mr); + goto err2; + } if (!mr && wqe->wr.wr.umw.length) { pr_err_once("mr with index = %d not found\n", wqe->wr.wr.umw.mr_index); diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 9d53b4d71230..76e6609dcf02 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -581,7 +581,7 @@ void *rxe_alloc(struct rxe_pool *pool) (pool->flags & RXE_POOL_ATOMIC) ? GFP_ATOMIC : GFP_KERNEL); if (!obj) - goto out; + return ERR_PTR(-ENOMEM); elem = (struct rxe_pool_entry *)((u8 *)obj + rxe_type_info[pool->type].elem_offset); @@ -591,9 +591,9 @@ void *rxe_alloc(struct rxe_pool *pool) write_unlock_irqrestore(&pool->pool_lock, flags); if (err) { kfree(obj); - obj = NULL; + return ERR_PTR(err); } -out: + return obj; } diff --git a/drivers/infiniband/sw/rxe/rxe_recv.c b/drivers/infiniband/sw/rxe/rxe_recv.c index dbb623ec5eaa..213c979b6b46 100644 --- a/drivers/infiniband/sw/rxe/rxe_recv.c +++ b/drivers/infiniband/sw/rxe/rxe_recv.c @@ -186,6 +186,8 @@ static int hdr_check(struct rxe_pkt_info *pkt) index = (qpn == 1) ? port->qp_gsi_index : qpn; qp = rxe_get_index(&rxe->qp_pool, index); + if (IS_ERR(qp)) + goto err1; if (unlikely(!qp)) { pr_warn_ratelimited("no qp matches qpn 0x%x\n", qpn); goto err1; @@ -245,7 +247,7 @@ static void rxe_rcv_mcast_pkt(struct rxe_dev *rxe, struct sk_buff *skb) /* lookup mcast group corresponding to mgid, takes a ref */ mcg = rxe_get_key(&rxe->mc_grp_pool, &dgid); - if (!mcg) + if (IS_ERR(mcg) || !mcg) goto err1; /* mcast group not registered */ spin_lock_bh(&mcg->mcg_lock); diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 0428965c664b..3e815158df55 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -565,7 +565,7 @@ static int invalidate_key(struct rxe_qp *qp, u32 key) if (key & IS_MW) { mw = rxe_get_key(&rxe->mw_pool, &key); - if (!mw) { + if (IS_ERR(mw) || !mw) { pr_err("No mw for key %#x\n", key); return -EINVAL; } @@ -573,7 +573,7 @@ static int invalidate_key(struct rxe_qp *qp, u32 key) rxe_drop_ref(mw); } else { mr = rxe_get_key(&rxe->mr_pool, &key); - if (!mr) { + if (IS_ERR(mr) || !mr) { pr_err("No mr for key %#x\n", key); return -EINVAL; } diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 58d3783063bb..47d863c4dd48 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -442,7 +442,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, */ if (rkey & IS_MW) { mw = rxe_get_key(&rxe->mw_pool, &rkey); - if (!mw) { + if (IS_ERR(mw) || !mw) { pr_err_once("no MW found with rkey = 0x%08x\n", rkey); state = RESPST_ERR_RKEY_VIOLATION; goto err; @@ -466,7 +466,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, rxe_drop_ref(mw); } else { mr = rxe_get_key(&rxe->mr_pool, &rkey); - if (!mr || (mr->rkey != rkey)) { + if (IS_ERR(mr) || !mr || (mr->rkey != rkey)) { pr_err_once("no MR found with rkey = 0x%08x\n", rkey); state = RESPST_ERR_RKEY_VIOLATION; goto err; @@ -794,7 +794,7 @@ static int invalidate_rkey(struct rxe_qp *qp, u32 rkey) if (rkey & IS_MW) { mw = rxe_get_key(&rxe->mw_pool, &rkey); - if (!mw) { + if (IS_ERR(mw) || !mw) { pr_err("No mw for rkey %#x\n", rkey); goto err; } @@ -802,7 +802,7 @@ static int invalidate_rkey(struct rxe_qp *qp, u32 rkey) rxe_drop_ref(mw); } else { mr = rxe_get_key(&rxe->mr_pool, &rkey); - if (!mr || mr->ibmr.rkey != rkey) { + if (IS_ERR(mr) || !mr || mr->ibmr.rkey != rkey) { pr_err("No mr for rkey %#x\n", rkey); goto err; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 4de263e41b07..58a3b66e6283 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -397,8 +397,8 @@ static struct ib_qp *rxe_create_qp(struct ib_pd *ibpd, goto err1; qp = rxe_alloc(&rxe->qp_pool); - if (!qp) { - err = -ENOMEM; + if (IS_ERR(qp)) { + err = PTR_ERR(qp); goto err1; } @@ -872,9 +872,9 @@ static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access) rxe_add_ref(pd); mr = rxe_alloc(&rxe->mr_pool); - if (!mr) { + if (IS_ERR(mr)) { rxe_drop_ref(pd); - return ERR_PTR(-ENOMEM); + return (void *)mr; } rxe_mr_init_dma(pd, access, mr); @@ -905,8 +905,8 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, rxe_add_ref(pd); mr = rxe_alloc(&rxe->mr_pool); - if (!mr) { - err = -ENOMEM; + if (IS_ERR(mr)) { + err = PTR_ERR(mr); goto err2; } @@ -956,8 +956,8 @@ static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, rxe_add_ref(pd); mr = rxe_alloc(&rxe->mr_pool); - if (!mr) { - err = -ENOMEM; + if (IS_ERR(mr)) { + err = PTR_ERR(mr); goto err1; } From patchwork Thu Oct 1 17:48:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812103 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D95E1668 for ; Thu, 1 Oct 2020 17:49:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7CE94208B6 for ; Thu, 1 Oct 2020 17:49:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lWwrjsg9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733007AbgJARtM (ORCPT ); Thu, 1 Oct 2020 13:49:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733004AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from mail-ot1-x341.google.com (mail-ot1-x341.google.com [IPv6:2607:f8b0:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70EB7C0613E7 for ; Thu, 1 Oct 2020 10:49:08 -0700 (PDT) Received: by mail-ot1-x341.google.com with SMTP id s66so6336403otb.2 for ; Thu, 01 Oct 2020 10:49:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dkz4VuuQcCYH1Nquft/UukRxtklX3/y0RNHvRWeURDw=; b=lWwrjsg9/DTGO+J/jBBWCp9DyXf6ukh0Qq/wUBRrDQTIVQL4UDedyGb1sNkvz4raPS mKyP+OiXf0718hBOnGbPMpuLw3H3q0lx518sDS+ZmEjM+SdqyPdOvrT4L7JmukjCM/9N ac+D29Z75no0eZK028lKm0lC7cfTcm9tzqDZ0nQi9la6dckVvesc0rAqvNndYRPpYREM hbJIX8Q2pkFY3OfdrVIkzGk0X/z30L2h+0Y/OG+cRWkNJ5qKKEaov1Lx3OzIGqyzvGii GUuXxFliFLg4ErkpUeBSgg4l/C6x6rf/RueKIJxkJFs6K6XxCG8Hdozn6CSkvJEjI0Su rjOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dkz4VuuQcCYH1Nquft/UukRxtklX3/y0RNHvRWeURDw=; b=bh5oHav6/Hs5sxCqOuEK9kL+bfDDZk0V3oKaub9Tf9cnkriAhT5gcYdbpRIiNRx8JQ jMQ9IrdbKlZbdGoA62yO0YcCwiADG+YLHdfE76NtT7cf60BOYBAbbw6Y6WaPcAFbKgcq 8MJSXp+7xh+ZTVaZINtXcO5uBQp4MKItc3a67jFIklPh0Ah0UODlCkvruc2+U6jvmllr 0tYrTDCirvfxc/51iBYPlzwkFaNKRKEgPEwgqmlLwukTo4NIweeUusWV3K874ty99bcN khx+CgWXTPQkkdEJUZGZtLmjuQuiY9r8GhYJroKQYpvKpaZlLMsAPsKFU2n34fRwNBz/ fFlQ== X-Gm-Message-State: AOAM530a9tmPpnvt/8+HcTYPk0nwoR9yu+DZ6gJZYOgAOMMkrEQqqmoZ AJNUIsO+HAOZ+rQ5SuvpdlXeo8X3m90= X-Google-Smtp-Source: ABdhPJwXAH/VV+ree2utOeLQbG86cH0vPf18PKxfc6oskxs7szNBhWaQCLAkHLlEpNT/Rr+QwoqEJQ== X-Received: by 2002:a05:6830:1191:: with SMTP id u17mr5622650otq.335.1601574547876; Thu, 01 Oct 2020 10:49:07 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id k16sm1179724oij.56.2020.10.01.10.49.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:07 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 16/19] rdma_rxe: remove duplicate entries in struct rxe_mr Date: Thu, 1 Oct 2020 12:48:44 -0500 Message-Id: <20201001174847.4268-17-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org v7: - Struct rxe_mr had lkey and rkey values both in itself and in the struct ib_mr. These are deleted and references replaced to the ones in ibmr. - Removed the struct rxe_pd pointer from struct rxe_mr. This duplicates the pd pointer in struct ib_mr which was also present. Added a mr_pd() macro which extracts the pd pointer from mr. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mr.c | 11 +++-------- drivers/infiniband/sw/rxe/rxe_req.c | 4 ++-- drivers/infiniband/sw/rxe/rxe_resp.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 8 ++++---- 5 files changed, 11 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index e3eb0f4bb2a0..f96f908644b1 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -51,8 +51,6 @@ static void rxe_mr_init(int access, struct rxe_mr *mr) if (access & IB_ACCESS_REMOTE) mr->ibmr.rkey = mr->ibmr.lkey; - mr->lkey = mr->ibmr.lkey; - mr->rkey = mr->ibmr.rkey; mr->state = RXE_MEM_STATE_INVALID; mr->type = RXE_MR_TYPE_NONE; mr->map_shift = ilog2(RXE_BUF_PER_MAP); @@ -101,7 +99,6 @@ void rxe_mr_init_dma(struct rxe_pd *pd, { rxe_mr_init(access, mr); - mr->pd = pd; mr->access = access; mr->state = RXE_MEM_STATE_VALID; mr->type = RXE_MR_TYPE_DMA; @@ -170,7 +167,6 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, } } - mr->pd = pd; mr->umem = umem; mr->access = access; mr->length = length; @@ -200,7 +196,6 @@ int rxe_mr_init_fast(struct rxe_pd *pd, if (err) goto err1; - mr->pd = pd; mr->max_buf = max_pages; mr->state = RXE_MEM_STATE_FREE; mr->type = RXE_MR_TYPE_MR; @@ -320,7 +315,7 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, memcpy(dest, src, length); if (crcp) - *crcp = rxe_crc32(to_rdev(mr->pd->ibpd.device), + *crcp = rxe_crc32(to_rdev(mr->ibmr.device), *crcp, dest, length); return 0; @@ -354,7 +349,7 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, memcpy(dest, src, bytes); if (crcp) - crc = rxe_crc32(to_rdev(mr->pd->ibpd.device), + crc = rxe_crc32(to_rdev(mr->ibmr.device), crc, dest, bytes); length -= bytes; @@ -389,7 +384,7 @@ static struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 lkey) if (IS_ERR(mr) || !mr) return NULL; - if (unlikely((mr->ibmr.lkey != lkey) || (mr->pd != pd) || + if (unlikely((mr_lkey(mr) != lkey) || (mr_pd(mr) != pd) || (access && !(access & mr->access)) || (mr->state != RXE_MEM_STATE_VALID))) { rxe_drop_ref(mr); diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 3e815158df55..128ec5d79501 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -638,8 +638,8 @@ int rxe_requester(void *arg) mr = to_rmr(wqe->wr.wr.reg.mr); mr->state = RXE_MEM_STATE_VALID; mr->access = wqe->wr.wr.reg.access; - mr->lkey = wqe->wr.wr.reg.key; - mr->rkey = wqe->wr.wr.reg.key; + mr->ibmr.lkey = wqe->wr.wr.reg.key; + mr->ibmr.rkey = wqe->wr.wr.reg.key; mr->iova = wqe->wr.wr.reg.mr->iova; break; case IB_WR_BIND_MW: diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index 47d863c4dd48..b8eb4f43c312 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -466,7 +466,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, rxe_drop_ref(mw); } else { mr = rxe_get_key(&rxe->mr_pool, &rkey); - if (IS_ERR(mr) || !mr || (mr->rkey != rkey)) { + if (IS_ERR(mr) || !mr || (mr_rkey(mr) != rkey)) { pr_err_once("no MR found with rkey = 0x%08x\n", rkey); state = RESPST_ERR_RKEY_VIOLATION; goto err; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 58a3b66e6283..fc00231627e7 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -937,7 +937,7 @@ static int rxe_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) struct rxe_mr *mr = to_rmr(ibmr); mr->state = RXE_MEM_STATE_ZOMBIE; - rxe_drop_ref(mr->pd); + rxe_drop_ref(mr_pd(mr)); rxe_drop_ref(mr); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 54429d8a06fb..847f80d7a1b6 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -293,12 +293,8 @@ struct rxe_mr { struct rxe_pool_entry pelem; struct ib_mr ibmr; - struct rxe_pd *pd; struct ib_umem *umem; - u32 lkey; - u32 rkey; - enum rxe_mem_state state; enum rxe_mr_type type; u64 va; @@ -323,6 +319,10 @@ struct rxe_mr { struct rxe_map **map; }; +#define mr_pd(mr) to_rpd((mr)->ibmr.pd) +#define mr_lkey(mr) ((mr)->ibmr.lkey) +#define mr_rkey(mr) ((mr)->ibmr.rkey) + enum rxe_send_flags { /* flag indicaes bind call came through verbs API */ RXE_BIND_MW = (1 << 0), From patchwork Thu Oct 1 17:48:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812095 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 309E913B2 for ; Thu, 1 Oct 2020 17:49:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1646F20848 for ; Thu, 1 Oct 2020 17:49:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uNBLrykm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732995AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733009AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from mail-oi1-x241.google.com (mail-oi1-x241.google.com [IPv6:2607:f8b0:4864:20::241]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D53CC0613E8 for ; Thu, 1 Oct 2020 10:49:09 -0700 (PDT) Received: by mail-oi1-x241.google.com with SMTP id x69so6498259oia.8 for ; Thu, 01 Oct 2020 10:49:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7ZpZLp9uHExOAGYVe2DGT9DP+CSVCfXtjYnapnR4Lz0=; b=uNBLrykmAvTCFWjpIKYtSIqAViXtyLaSDdJ9ckPoPoGWez1qGdx5zGM2ulQo5wPkSP xirf+iGkvQK3HcEmUJChXR2XT62oOaK1JYQm3OTCyoaoFmTzJeWGHss9NqWDYffqLqT7 gNs7dg4KcYaH74gfu8OwQlm42YRuy9o33N10D2e9ceaIc/oIYMfQ0x6cSK7rOo186220 EXbqdGkmmuhE002NwrSoyIrhrnWn/ix97EeL1ZBmG01aoSs2P/jZxB4AIPw78GaFu/dF iNoMlDoyA7CxzYlvhSeDWbekyq3naXp35Z5vcwTLGO8CmogD4hoXc8gn5yIOFjQo4sLH eiWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7ZpZLp9uHExOAGYVe2DGT9DP+CSVCfXtjYnapnR4Lz0=; b=eHEeYWAl3pxX+PfAXZl0DjDgKRvLxtt80cQ25RVyfu1O4Uks7SfyUOhMJlXybfchAj UObCvXWxuEpPHOaVZJbe6ChjN+dWryXGWBUTAsdByL2MzIpQzsbiwdl3n0TjKPw4NH/T 9oTzWzDaYdfLHA47Bzfg52ExjNUjUy7LdHn4P+SuJVeZVerU2UhNDefm2+PucBqtBZmc ZHmZrxJsmp7LI5V73eF56/7iakZJ82uVD6yTYHr5kqplPMxt3VsoPFn6Ch4+tdPWXJu0 MquIbFee2D00NgmzVMlrrzNWXxFak0ZXX/494qdYTGJaUWrZpbCc/u5A0r2Exn+izPER 3GyA== X-Gm-Message-State: AOAM533JsztMsEy+usbFl24j6W/wnFuMHHlIj1hNKpaQREtOgy+/Y5ue amyszAUt/SrvCkkSFd002G7yBdrS1Z4= X-Google-Smtp-Source: ABdhPJzY+1G6COKyfRSvVSedz4lVbgTi3nSEhd1rBUUMV000sBNUcXz9iNhv6iSOWkCM3LYfwjJ2/w== X-Received: by 2002:aca:ecc4:: with SMTP id k187mr677844oih.138.1601574548716; Thu, 01 Oct 2020 10:49:08 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id 71sm1450315otm.81.2020.10.01.10.49.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:08 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 17/19] rdma_rme: removed unused RXE_MR_TYPE_FMR Date: Thu, 1 Oct 2020 12:48:45 -0500 Message-Id: <20201001174847.4268-18-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This is a left over from the past. A step in replacing rxe mr types with ib mr types. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mr.c | 1 - drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 2 files changed, 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index f96f908644b1..1901f388c747 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -28,7 +28,6 @@ static int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) return 0; case RXE_MR_TYPE_MR: - case RXE_MR_TYPE_FMR: if (iova < mr->iova || length > mr->length || iova > mr->iova + mr->length - length) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 847f80d7a1b6..ea9861998413 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -275,7 +275,6 @@ enum rxe_mr_type { RXE_MR_TYPE_NONE, RXE_MR_TYPE_DMA, RXE_MR_TYPE_MR, - RXE_MR_TYPE_FMR, }; #define RXE_BUF_PER_MAP (PAGE_SIZE / sizeof(struct rxe_phys_buf)) From patchwork Thu Oct 1 17:48:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812101 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6FAD26CB for ; Thu, 1 Oct 2020 17:49:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 55B9D20848 for ; Thu, 1 Oct 2020 17:49:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wz5Rkhw+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732958AbgJARtM (ORCPT ); Thu, 1 Oct 2020 13:49:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733007AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 031E8C0613E9 for ; Thu, 1 Oct 2020 10:49:10 -0700 (PDT) Received: by mail-oo1-xc30.google.com with SMTP id y25so1701269oog.4 for ; Thu, 01 Oct 2020 10:49:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nY3c0i5sSwIiPTOZv8oxbrrUu6cJ8Pc0aLiiNAKAzRk=; b=Wz5Rkhw+FMvBi8arQbMveOy5RZ7E55W4G/xJDvG/oKd+HGfmSyyZ+WCfglpZVT/p5/ 7C7dYmNyhR17KTlzZ8yb1Kpnjuct0GNaBHv+SvMgn1ClZumEorzzdKj/a13sKpJUqjd6 dwMIhST42iHWuF1uOAjGi89Yf9y27zoAPYc5Bxm/YkZsMSzpBBwhqMq4U0H1Hgm9Neqk IXiZH1c0VY75z6mYaBnlL3kWPVlMQFDH7Y34mABgD6npUg1Q6g72DK+IUOeL1he3Wb8V +2ZCTNmJPQ1wWefdpSgQroatvwwQt+LcFGcjlfJddXtK1TFYYio1r2lix3DZvZwDJfXq 4GkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nY3c0i5sSwIiPTOZv8oxbrrUu6cJ8Pc0aLiiNAKAzRk=; b=XjkpHYiqdUQXmBNUlPe+PDiwevKY/B5PRhbkoKK3urFlL5ILHvQsYRSn1P/CYkswaA NZBUepNUYsDE58rT9GbF5YxgAql1sl1n2cK9zFnjRuAolqUDTMdxLLZDXtEawWxUbiwq 8OEKi55/PhukrVXCqEO6p9+wPJL3ZpdeRDVUAjsuP1VZPzkZDLuT/5B1HdgGpoO4bPX6 1pp87HTNRhFsIhAuScNuEMvy/xkohe94cMP0ptvz+l8RKceYPufiDRSfMm0WtOJrgLI0 6K9dywJhbfNkxmKbiF/wOMMXkawu9P+2o2CLBhtKZgzW8aMSaJynwHyC9Rw19HvJbNaK 9j4g== X-Gm-Message-State: AOAM530qMbxN2PzynkhjW5awpua+TU+yM4YbxrilYjaV/aKW8uhsU87D Jfsk55V73H/fOxKczZD06b4= X-Google-Smtp-Source: ABdhPJyk3igjrzCaCPqK4/wfrj6YBX6xshv5H1Kw7SV6UtaO2XhnUhMOO3feS0h6FKgRl/V2wyVpQg== X-Received: by 2002:a4a:9b01:: with SMTP id a1mr6537406ook.51.1601574549442; Thu, 01 Oct 2020 10:49:09 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id p1sm1361299otq.7.2020.10.01.10.49.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:09 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 18/19] rdma_rxe: add rkey validation checks for MR and MW Date: Thu, 1 Oct 2020 12:48:46 -0500 Message-Id: <20201001174847.4268-19-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Added checks from 9.4.1.1.3 r_key validation for remote memory invalidate to MR and MW invalidate routines. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mr.c | 49 ++++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_mw.c | 10 +++++- 2 files changed, 58 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 1901f388c747..2cc3487d3f77 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -522,8 +522,57 @@ int advance_dma_data(struct rxe_dma_info *dma, unsigned int length) return 0; } +static int check_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr) +{ + int type = mr->ibmr.type; + int qpt = qp_type(qp); + + if (unlikely(type > IB_MR_TYPE_INTEGRITY)) { + pr_err("invalidate: MR of unknown type = %d\n", + mr->ibmr.type); + return -EINVAL; + } + + /* o10-37.2.13 */ + if (unlikely(type == IB_MR_TYPE_MEM_REG || + type == IB_MR_TYPE_USER)) { + pr_err("invalidate: MR created by reg_mr or user_reg_mr\n"); + return -EINVAL; + } + + /* o10-37.2.17 */ + if (unlikely(atomic_read(&mr->num_mw))) { + pr_err("invalidate: MR with bound MWs\n"); + return -EINVAL; + } + + if (unlikely(!((qpt == IB_QPT_RC) || (qpt == IB_QPT_UC) || + (qpt == IB_QPT_XRC_INI) || (qpt == IB_QPT_XRC_TGT)))) { + pr_err("invalidate: MR with invalid QP type\n"); + return -EINVAL; + } + + if (unlikely(qp->ibqp.pd != mr->ibmr.pd)) { + pr_err("invalidate: MR and QP have different PDs\n"); + return -EINVAL; + } + + if (unlikely(mr->state == RXE_MEM_STATE_INVALID)) { + pr_err("invalidate: MR in invalid state\n"); + return -EINVAL; + } + + return 0; +} + int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr) { + int ret; + + ret = check_invalidate_mr(qp, mr); + if (ret) + return ret; + /* TODO there are API rules being ignored here * cleanup later. Current project is not trying * to fix MR diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index d2d09502a28d..ecda634ab7f0 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -316,9 +316,17 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) static int check_invalidate_mw(struct rxe_qp *qp, struct rxe_mw *mw) { + int qpt = qp_type(qp); + /* o10-37.2.26 */ if (unlikely(mw->ibmw.type == IB_MW_TYPE_1)) { - pr_err_once("attempt to invalidate a type 1 MW\n"); + pr_err_once("invalidate a type 1 MW\n"); + return -EINVAL; + } + + if (unlikely(!((qpt == IB_QPT_RC) || (qpt == IB_QPT_UC) || + (qpt == IB_QPT_XRC_INI) || (qpt == IB_QPT_XRC_TGT)))) { + pr_err("invalidate Mw with invalid QP type\n"); return -EINVAL; } From patchwork Thu Oct 1 17:48:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 11812107 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A9966CB for ; Thu, 1 Oct 2020 17:49:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BD8F20848 for ; Thu, 1 Oct 2020 17:49:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="p/WwbWFS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1733004AbgJARtM (ORCPT ); Thu, 1 Oct 2020 13:49:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1733009AbgJARtK (ORCPT ); Thu, 1 Oct 2020 13:49:10 -0400 Received: from mail-ot1-x343.google.com (mail-ot1-x343.google.com [IPv6:2607:f8b0:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBA10C0613D0 for ; Thu, 1 Oct 2020 10:49:10 -0700 (PDT) Received: by mail-ot1-x343.google.com with SMTP id o8so6326575otl.4 for ; Thu, 01 Oct 2020 10:49:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C62gDt66uknWUy7fAfUfQKtDdScjtj2a0ful2t+NuZ8=; b=p/WwbWFSbKfVA5zQIGnBDdu7C1cUUF+bAUcJZYflfWTJs3LVrHryvzKUK2rksl30KA qZfgBjqo/V5LELncDH8kc3szisi0inrz5o/t+xYXH9sqFDJhaOZqpV+qppC6Nw0BMyll Z7b3iIjBjcZQvz8Ot+FEQUagYVczl2F56FTTMBfq7qXTwC9UA8FSrCs6QhIoMW4M+jc7 GUJI+ziyObjoLA7Sp8+Yt+pOE60VGBzzDNcbL+pZumBC98aDvO38hBVAupQ/0qzcUBgR PfUNj7boVyx4w2anF5CRrI3YwCJQvFJiSx8wzooos2DQHtsD42GoKak50GesIpoojMIY 8Acw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C62gDt66uknWUy7fAfUfQKtDdScjtj2a0ful2t+NuZ8=; b=ED2GsFnM/vMKfXGlRYigExt+IIFJP4dEn2cUAk+FBk4Xnjbfw4ktISIFqOBosFInLk zqEm47RcGf79zUAS/bpuyuarCIpVHmkVijYqTxwLm4d3VM8Ezxrh0tOgLuSRBTyKD/BW D9HhYYcIJr5OwnbLOaPvCrTFKW+CKPwkMy6q0WMeoRzpijD8mH3r8fxc0dm9pFN5VLuH r0tVdEerIqJjqMY2iFKn82/fsOceDBYpZPQSVislb7ivrNpEVhObxkhp00vHf6f9Sxzo 5vEIDgyPbk3gFl3Loho1l+8a/X/G5kswKhq6CBvIvFZw/x3LEkfmk6yZkmN9pMI+lkLM DRKA== X-Gm-Message-State: AOAM533oUaHrV/mHcjR4yUvNE2/yx11+6Zv2fjrjp5j1wXNfdUtd0bPd vjvvF4jXtRZXkjx//wxP1mU= X-Google-Smtp-Source: ABdhPJxbJWXNNmYtxzdG5TYfonOGDINFPj/BOkxwNxNOzMAKMF/OLIgY8AaqxtZFU0JFw5OUGxUeeg== X-Received: by 2002:a05:6830:1e89:: with SMTP id n9mr5592654otr.274.1601574550171; Thu, 01 Oct 2020 10:49:10 -0700 (PDT) Received: from localhost ([2605:6000:8b03:f000:d01f:9a3e:d22f:7a6]) by smtp.gmail.com with ESMTPSA id s13sm1374025otq.5.2020.10.01.10.49.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Oct 2020 10:49:09 -0700 (PDT) From: Bob Pearson X-Google-Original-From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v7 19/19] rdma_rxe: moved rxe_xmit_packet to rxe_net.c Date: Thu, 1 Oct 2020 12:48:47 -0500 Message-Id: <20201001174847.4268-20-rpearson@hpe.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201001174847.4268-1-rpearson@hpe.com> References: <20201001174847.4268-1-rpearson@hpe.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org No good reason to stick such a large subroutine in a header file as an inline function. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_loc.h | 47 ++--------------------------- drivers/infiniband/sw/rxe/rxe_net.c | 47 +++++++++++++++++++++++++++-- 2 files changed, 47 insertions(+), 47 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index fcffd5075b18..9d3c59694e7b 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -124,8 +124,6 @@ int rxe_mw_check_access(struct rxe_qp *qp, struct rxe_mw *mw, void rxe_mw_cleanup(struct rxe_pool_entry *arg); /* rxe_net.c */ -void rxe_loopback(struct sk_buff *skb); -int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb); struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av, int paylen, struct rxe_pkt_info *pkt); int rxe_prepare(struct rxe_pkt_info *pkt, struct sk_buff *skb, u32 *crc); @@ -133,6 +131,8 @@ const char *rxe_parent_name(struct rxe_dev *rxe, unsigned int port_num); struct device *rxe_dma_device(struct rxe_dev *rxe); int rxe_mcast_add(struct rxe_dev *rxe, union ib_gid *mgid); int rxe_mcast_delete(struct rxe_dev *rxe, union ib_gid *mgid); +int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, + struct sk_buff *skb); /* rxe_qp.c */ int rxe_qp_chk_init(struct rxe_dev *rxe, struct ib_qp_init_attr *init); @@ -230,47 +230,4 @@ static inline unsigned int wr_opcode_mask(int opcode, struct rxe_qp *qp) return rxe_wr_opcode_info[opcode].mask[qp->ibqp.qp_type]; } -static inline int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, - struct sk_buff *skb) -{ - int err; - int is_request = pkt->mask & RXE_REQ_MASK; - struct rxe_dev *rxe = to_rdev(qp->ibqp.device); - - if ((is_request && (qp->req.state != QP_STATE_READY)) || - (!is_request && (qp->resp.state != QP_STATE_READY))) { - pr_info("Packet dropped. QP is not in ready state\n"); - goto drop; - } - - if (pkt->mask & RXE_LOOPBACK_MASK) { - memcpy(SKB_TO_PKT(skb), pkt, sizeof(*pkt)); - rxe_loopback(skb); - err = 0; - } else { - err = rxe_send(pkt, skb); - } - - if (err) { - rxe->xmit_errors++; - rxe_counter_inc(rxe, RXE_CNT_SEND_ERR); - return err; - } - - if ((qp_type(qp) != IB_QPT_RC) && - (pkt->mask & RXE_END_MASK)) { - pkt->wqe->state = wqe_state_done; - rxe_run_task(&qp->comp.task, 1); - } - - rxe_counter_inc(rxe, RXE_CNT_SENT_PKTS); - goto done; - -drop: - kfree_skb(skb); - err = 0; -done: - return err; -} - #endif /* RXE_LOC_H */ diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 31b93e7e1e2f..759d2c751e5d 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -394,7 +394,7 @@ static void rxe_skb_tx_dtor(struct sk_buff *skb) rxe_drop_ref(qp); } -int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb) +static int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb) { int err; @@ -424,11 +424,54 @@ int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb) return 0; } -void rxe_loopback(struct sk_buff *skb) +static void rxe_loopback(struct sk_buff *skb) { rxe_rcv(skb); } +int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, + struct sk_buff *skb) +{ + int err; + int is_request = pkt->mask & RXE_REQ_MASK; + struct rxe_dev *rxe = to_rdev(qp->ibqp.device); + + if ((is_request && (qp->req.state != QP_STATE_READY)) || + (!is_request && (qp->resp.state != QP_STATE_READY))) { + pr_info("Packet dropped. QP is not in ready state\n"); + goto drop; + } + + if (pkt->mask & RXE_LOOPBACK_MASK) { + memcpy(SKB_TO_PKT(skb), pkt, sizeof(*pkt)); + rxe_loopback(skb); + err = 0; + } else { + err = rxe_send(pkt, skb); + } + + if (err) { + rxe->xmit_errors++; + rxe_counter_inc(rxe, RXE_CNT_SEND_ERR); + return err; + } + + if ((qp_type(qp) != IB_QPT_RC) && + (pkt->mask & RXE_END_MASK)) { + pkt->wqe->state = wqe_state_done; + rxe_run_task(&qp->comp.task, 1); + } + + rxe_counter_inc(rxe, RXE_CNT_SENT_PKTS); + goto done; + +drop: + kfree_skb(skb); + err = 0; +done: + return err; +} + struct sk_buff *rxe_init_packet(struct rxe_dev *rxe, struct rxe_av *av, int paylen, struct rxe_pkt_info *pkt) {