From patchwork Tue Oct 6 08:37:40 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 7333701 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 0FF4EBEEA4 for ; Tue, 6 Oct 2015 08:37:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 2638E20630 for ; Tue, 6 Oct 2015 08:37:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 200DF20616 for ; Tue, 6 Oct 2015 08:37:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752163AbbJFIhu (ORCPT ); Tue, 6 Oct 2015 04:37:50 -0400 Received: from mail-wi0-f181.google.com ([209.85.212.181]:35406 "EHLO mail-wi0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750818AbbJFIhq (ORCPT ); Tue, 6 Oct 2015 04:37:46 -0400 Received: by wicge5 with SMTP id ge5so155659340wic.0 for ; Tue, 06 Oct 2015 01:37:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=Mj3PhAbVP21rsct4IwtlZnx28J1O6lsHjjwuV0AToOc=; b=iSVvUsVVcqHVh0ZhiLlgWk2kCSezuoCnyL82Jp7MPUQyY5Wtk2q9eJQpsXVJBqlGbU cDhPFAmX4RK3a6bVFhyCLlsyYVbZLSK343qF78bbNdk7aS1/CUyrh1hzGjrwGDF4tjVg brTAnuKtw9lA8a1X8uPQviWfdr3pW3dzv/2rftX3ScoUXYHCWUaqTS6R5y28rnWz3iem FgYGlmKg+NunifbchJLiRtT9vmj4K+lB4NUqOs+m5ItmIrdqvLpTPu+W/B/d+ofXRRgZ A3CCSEWdN+q0wOx01nxZg1My5QyF5fAVS0iiGiYNXVOawzaRY2lTXrtAvR2cd10GZpIa Ag3A== X-Gm-Message-State: ALoCoQlr0pmfkAjt05oKo39b2OTSOY/JKRzargpEKoCLb6MTN4HBZR6ITsayrSPz/hY79uBU2aMs X-Received: by 10.180.19.41 with SMTP id b9mr15279627wie.71.1444120665003; Tue, 06 Oct 2015 01:37:45 -0700 (PDT) Received: from [10.223.0.123] ([193.47.165.251]) by smtp.googlemail.com with ESMTPSA id qq4sm31139864wjc.14.2015.10.06.01.37.43 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Oct 2015 01:37:44 -0700 (PDT) Subject: Re: [PATCH v1 00/24] New fast registration API To: Bart Van Assche References: <1442482947-27785-1-git-send-email-sagig@mellanox.com> <560AE099.2080004@sandisk.com> <560AFB71.3010003@dev.mellanox.co.il> <560C30F4.50900@sandisk.com> <560C42CE.6090000@sandisk.com> <560CDDBC.8000400@dev.mellanox.co.il> <560D730C.9000302@sandisk.com> <560D9E69.70604@sandisk.com> <35618B90-4D6E-4036-A69B-4405F020D440@dev.mellanox.co.il> <560EA4A1.3080709@sandisk.com> Cc: Sagi Grimberg , "linux-rdma@vger.kernel.org" , "linux-nfs@vger.kernel.org" , "Nicholas A. Bellinger" From: Sagi Grimberg Message-ID: <56138854.4040209@dev.mellanox.co.il> Date: Tue, 6 Oct 2015 11:37:40 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <560EA4A1.3080709@sandisk.com> Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 10/2/2015 6:37 PM, Bart Van Assche wrote: > On 10/01/2015 11:14 PM, Sagi Grimberg wrote: >> Would you mind sending me your .config? > > Hello Sagi, Hi Bart, > > I just sent this .config file to you off-list. I see now the error you are referring to. The issue is that the device requires the MR page array to have an alignment (0x40 for mlx4 and 0x400 for mlx5). When I modified the page array allocation to be non-coherent I didn't take care of alignment. Taking care of this alignment may result in a higher order allocation as we'd need to add (alignment - 1) to the allocation size. e.g. a 512 pages on mlx4 will become: 512 * 8 + 0x40 - 1 = 4159 I'm leaning towards this approach. Any preference? I think this patch should take care of mlx4: } --- Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index de6eab3..4c69247 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -129,6 +129,8 @@ struct mlx4_ib_cq { struct list_head recv_qp_list; }; +#define MLX4_MR_PAGES_ALIGN 0x40 + struct mlx4_ib_mr { struct ib_mr ibmr; __be64 *pages; @@ -137,6 +139,7 @@ struct mlx4_ib_mr { u32 max_pages; struct mlx4_mr mmr; struct ib_umem *umem; + void *pages_alloc; }; struct mlx4_ib_mw { diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c index fa01f75..d3f8175 100644 --- a/drivers/infiniband/hw/mlx4/mr.c +++ b/drivers/infiniband/hw/mlx4/mr.c @@ -279,10 +279,14 @@ mlx4_alloc_priv_pages(struct ib_device *device, int size = max_pages * sizeof(u64); int ret; - mr->pages = kzalloc(size, GFP_KERNEL); - if (!mr->pages) + size += max_t(int, MLX4_MR_PAGES_ALIGN - ARCH_KMALLOC_MINALIGN, 0); + + mr->pages_alloc = kzalloc(size, GFP_KERNEL); + if (!mr->pages_alloc) return -ENOMEM; + mr->pages = PTR_ALIGN(mr->pages_alloc, MLX4_MR_PAGES_ALIGN); + mr->page_map = dma_map_single(device->dma_device, mr->pages, size, DMA_TO_DEVICE); @@ -293,20 +297,22 @@ mlx4_alloc_priv_pages(struct ib_device *device, return 0; err: - kfree(mr->pages); + kfree(mr->pages_alloc); return ret; } static void mlx4_free_priv_pages(struct mlx4_ib_mr *mr) { - struct ib_device *device = mr->ibmr.device; - int size = mr->max_pages * sizeof(u64); - if (mr->pages) { + struct ib_device *device = mr->ibmr.device; + int size = mr->max_pages * sizeof(u64); + + size += max_t(int, MLX4_MR_PAGES_ALIGN - ARCH_KMALLOC_MINALIGN, 0); + dma_unmap_single(device->dma_device, mr->page_map, size, DMA_TO_DEVICE); - kfree(mr->pages); + kfree(mr->pages_alloc); mr->pages = NULL; }