From patchwork Tue May 2 13:06:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sunil Kovvuri X-Patchwork-Id: 9707991 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A5E2260245 for ; Tue, 2 May 2017 13:10:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95DC728449 for ; Tue, 2 May 2017 13:10:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 894CD2844C; Tue, 2 May 2017 13:10:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FROM autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9384B28449 for ; Tue, 2 May 2017 13:10:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=yKonhumiqssDjFWYAjalYGNSe02lU+S8V12+g5LhmJc=; b=tXZyYdOO0er8N6h/fvJgvmdktt DzczBOfaQ1yT7IG3/zKhmpVy1Q3Zf8EqVHVxG6gtnXubUJ+4I26iniXNBAgPGAVECjy/ZOWtBD7Lf mYLgpaplX64w6V+XPY+FOCdnsuozNHNx/n4aYRL0Fhw/RrJ3hrdxKZhm/sz15DH567E5baufMvmY3 MI7iXnDEnGV3jU3uBQraIfGLI1W1BTPvQNmN0vJ3S9IdT40myy0m9maTy39+Gu51Ck6N+1B0kyKMI sBFlf8d44kji90D1wqUdsfVtZ/EyWg0n1ueRjNAsCFf6kQjXHsfKv+2yFCokVl5/6NG56k3XkMGQd aqWBFUyQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1d5XZU-0008I6-Bi; Tue, 02 May 2017 13:10:28 +0000 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1d5XWt-000593-J1 for linux-arm-kernel@lists.infradead.org; Tue, 02 May 2017 13:07:51 +0000 Received: by mail-pf0-x242.google.com with SMTP id b23so17175740pfc.0 for ; Tue, 02 May 2017 06:07:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QjO8gTvNUBmMmL0UdFvXtJd1fvGYzQnBQA9VgSc06S8=; b=f/7UiV8SxGWIYWoSZTHJ6/WoUeugIODMjzKT6BT+7f9tr4Td7+1unWIkMWoK6c8TVE 6QLH6PZpfYjTBOdxTzah/bAivRUGERXUwF2q6KbTKI9nHIowdcuV7L1i2STBRTr2fk00 LGMI2NRg8/89yt6ibv1OrHP1irAc+TBGXooukjQDmWGlUPYIDyJbynT2J1YLgL+QfUqF uQ/6qGv48JM51BfuOzQOVfopboIQz+alrHuagcokgjZW9BkVmx3DjpeqitF0Xtl71ZNG TJ/5SOPVf2Tpotts4uuGrPJL2eThEPYNzBQYy8/leVlr7HrGP38vdOqnpqXRr8sfRCh+ NTYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=QjO8gTvNUBmMmL0UdFvXtJd1fvGYzQnBQA9VgSc06S8=; b=c1C5DDuSp0CLjOXHcXdtAU+1vsH7CqskO/xN40Y4ORMaJqQEhP53mVMYKem7v39juB PeUk7BN1grKPbAy/bY/X3x3wZ2cDpkMqFWxDHLAIFhisHz1S6pIuDPjRptOGr4uzNp7F TfXYxh/o7HdOnabJB2uMNFc+eBNTcNhuRKodckWiPqwWMzhkNITg2F+kcf7bpLXckaP3 mH3gg9i/SlBeiMyRSQSGp/Vmw7l4D8Pfh02WRGCmh8iNJCBiC4Wk8h+lePb3Eu5TOHp6 RfbThPmrR9tpJwrlGtqGWA/qkPVI8tqwcNqgNJsGh8lWYWHZZN2E/SdKGB+DZVJfHP9L owJA== X-Gm-Message-State: AN3rC/4vJvbTLVOep5EzrDHvqbx2w1iDQUfjw8Khxt3POgCKUsuwCjum GV5khp+1P+syXg== X-Received: by 10.98.153.216 with SMTP id t85mr32471377pfk.178.1493730446704; Tue, 02 May 2017 06:07:26 -0700 (PDT) Received: from machine421.in.caveonetworks.com ([14.140.2.178]) by smtp.googlemail.com with ESMTPSA id k198sm24610689pga.54.2017.05.02.06.07.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 May 2017 06:07:25 -0700 (PDT) From: sunil.kovvuri@gmail.com To: netdev@vger.kernel.org Subject: [PATCH 1/9] net: thunderx: Support for page recycling Date: Tue, 2 May 2017 18:36:50 +0530 Message-Id: <1493730418-24606-2-git-send-email-sunil.kovvuri@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com> References: <1493730418-24606-1-git-send-email-sunil.kovvuri@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170502_060747_754151_475DCDEC X-CRM114-Status: GOOD ( 25.73 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sunil Goutham , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Sunil Goutham Adds support for page recycling for allocating receive buffers to reduce cost of refilling RBDR ring. Also got rid of using compound pages when pagesize is 4K, only order-0 pages now. Only page is recycled, DMA mappings still needs to be done for every receive buffer allocated due to following constraints - Cannot have just one receive buffer per 64KB page. - There is just one buffer ring shared across 8 Rx queues, so buffers of same page can go to any Rx queue. - HW gives buffer address where packet has been DMA'ed and not the index into buffer ring. This makes it not possible to resue DMA mapping info. So unfortunately have to go through costly mapping route for every buffer. Signed-off-by: Sunil Goutham --- drivers/net/ethernet/cavium/thunder/nic.h | 4 +- .../net/ethernet/cavium/thunder/nicvf_ethtool.c | 3 +- drivers/net/ethernet/cavium/thunder/nicvf_queues.c | 121 ++++++++++++++++++--- drivers/net/ethernet/cavium/thunder/nicvf_queues.h | 11 ++ 4 files changed, 119 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/cavium/thunder/nic.h b/drivers/net/ethernet/cavium/thunder/nic.h index 6fb4421..dca6aed 100644 --- a/drivers/net/ethernet/cavium/thunder/nic.h +++ b/drivers/net/ethernet/cavium/thunder/nic.h @@ -252,12 +252,14 @@ struct nicvf_drv_stats { u64 tx_csum_overflow; /* driver debug stats */ - u64 rcv_buffer_alloc_failures; u64 tx_tso; u64 tx_timeout; u64 txq_stop; u64 txq_wake; + u64 rcv_buffer_alloc_failures; + u64 page_alloc; + struct u64_stats_sync syncp; }; diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c index 02a986c..a89db5f 100644 --- a/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c +++ b/drivers/net/ethernet/cavium/thunder/nicvf_ethtool.c @@ -100,11 +100,12 @@ static const struct nicvf_stat nicvf_drv_stats[] = { NICVF_DRV_STAT(tx_csum_overlap), NICVF_DRV_STAT(tx_csum_overflow), - NICVF_DRV_STAT(rcv_buffer_alloc_failures), NICVF_DRV_STAT(tx_tso), NICVF_DRV_STAT(tx_timeout), NICVF_DRV_STAT(txq_stop), NICVF_DRV_STAT(txq_wake), + NICVF_DRV_STAT(rcv_buffer_alloc_failures), + NICVF_DRV_STAT(page_alloc), }; static const struct nicvf_stat nicvf_queue_stats[] = { diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c index 7b0fd8d..12f9709 100644 --- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.c +++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.c @@ -19,8 +19,6 @@ #include "q_struct.h" #include "nicvf_queues.h" -#define NICVF_PAGE_ORDER ((PAGE_SIZE <= 4096) ? PAGE_ALLOC_COSTLY_ORDER : 0) - static inline u64 nicvf_iova_to_phys(struct nicvf *nic, dma_addr_t dma_addr) { /* Translation is installed only when IOMMU is present */ @@ -90,33 +88,88 @@ static void nicvf_free_q_desc_mem(struct nicvf *nic, struct q_desc_mem *dmem) dmem->base = NULL; } -/* Allocate buffer for packet reception - * HW returns memory address where packet is DMA'ed but not a pointer - * into RBDR ring, so save buffer address at the start of fragment and - * align the start address to a cache aligned address +/* Allocate a new page or recycle one if possible + * + * We cannot optimize dma mapping here, since + * 1. It's only one RBDR ring for 8 Rx queues. + * 2. CQE_RX gives address of the buffer where pkt has been DMA'ed + * and not idx into RBDR ring, so can't refer to saved info. + * 3. There are multiple receive buffers per page */ -static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp, - u32 buf_len, u64 **rbuf) +static struct pgcache *nicvf_alloc_page(struct nicvf *nic, + struct rbdr *rbdr, gfp_t gfp) { - int order = NICVF_PAGE_ORDER; + struct page *page = NULL; + struct pgcache *pgcache, *next; + + /* Check if page is already allocated */ + pgcache = &rbdr->pgcache[rbdr->pgidx]; + page = pgcache->page; + /* Check if page can be recycled */ + if (page && (page_ref_count(page) != 1)) + page = NULL; + + if (!page) { + page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN, 0); + if (!page) + return NULL; + + this_cpu_inc(nic->pnicvf->drv_stats->page_alloc); + + /* Check for space */ + if (rbdr->pgalloc >= rbdr->pgcnt) { + /* Page can still be used */ + nic->rb_page = page; + return NULL; + } + + /* Save the page in page cache */ + pgcache->page = page; + rbdr->pgalloc++; + } + + /* Take extra page reference for recycling */ + page_ref_add(page, 1); + + rbdr->pgidx++; + rbdr->pgidx &= (rbdr->pgcnt - 1); + + /* Prefetch refcount of next page in page cache */ + next = &rbdr->pgcache[rbdr->pgidx]; + page = next->page; + if (page) + prefetch(&page->_refcount); + + return pgcache; +} + +/* Allocate buffer for packet reception */ +static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, struct rbdr *rbdr, + gfp_t gfp, u32 buf_len, u64 **rbuf) +{ + struct pgcache *pgcache = NULL; /* Check if request can be accomodated in previous allocated page */ if (nic->rb_page && - ((nic->rb_page_offset + buf_len) < (PAGE_SIZE << order))) { + ((nic->rb_page_offset + buf_len) <= PAGE_SIZE)) { nic->rb_pageref++; goto ret; } nicvf_get_page(nic); + nic->rb_page = NULL; - /* Allocate a new page */ - nic->rb_page = alloc_pages(gfp | __GFP_COMP | __GFP_NOWARN, - order); - if (!nic->rb_page) { + /* Get new page, either recycled or new one */ + pgcache = nicvf_alloc_page(nic, rbdr, gfp); + if (!pgcache && !nic->rb_page) { this_cpu_inc(nic->pnicvf->drv_stats->rcv_buffer_alloc_failures); return -ENOMEM; } + nic->rb_page_offset = 0; + /* Check if it's recycled */ + if (pgcache) + nic->rb_page = pgcache->page; ret: /* HW will ensure data coherency, CPU sync not required */ *rbuf = (u64 *)((u64)dma_map_page_attrs(&nic->pdev->dev, nic->rb_page, @@ -125,7 +178,7 @@ static inline int nicvf_alloc_rcv_buffer(struct nicvf *nic, gfp_t gfp, DMA_ATTR_SKIP_CPU_SYNC)); if (dma_mapping_error(&nic->pdev->dev, (dma_addr_t)*rbuf)) { if (!nic->rb_page_offset) - __free_pages(nic->rb_page, order); + __free_pages(nic->rb_page, 0); nic->rb_page = NULL; return -ENOMEM; } @@ -177,10 +230,26 @@ static int nicvf_init_rbdr(struct nicvf *nic, struct rbdr *rbdr, rbdr->head = 0; rbdr->tail = 0; + /* Initialize page recycling stuff. + * + * Can't use single buffer per page especially with 64K pages. + * On embedded platforms i.e 81xx/83xx available memory itself + * is low and minimum ring size of RBDR is 8K, that takes away + * lots of memory. + */ + rbdr->pgcnt = ring_len / (PAGE_SIZE / buf_size); + rbdr->pgcnt = roundup_pow_of_two(rbdr->pgcnt); + rbdr->pgcache = kzalloc(sizeof(*rbdr->pgcache) * + rbdr->pgcnt, GFP_KERNEL); + if (!rbdr->pgcache) + return -ENOMEM; + rbdr->pgidx = 0; + rbdr->pgalloc = 0; + nic->rb_page = NULL; for (idx = 0; idx < ring_len; idx++) { - err = nicvf_alloc_rcv_buffer(nic, GFP_KERNEL, RCV_FRAG_LEN, - &rbuf); + err = nicvf_alloc_rcv_buffer(nic, rbdr, GFP_KERNEL, + RCV_FRAG_LEN, &rbuf); if (err) { /* To free already allocated and mapped ones */ rbdr->tail = idx - 1; @@ -201,6 +270,7 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr) { int head, tail; u64 buf_addr, phys_addr; + struct pgcache *pgcache; struct rbdr_entry_t *desc; if (!rbdr) @@ -234,6 +304,18 @@ static void nicvf_free_rbdr(struct nicvf *nic, struct rbdr *rbdr) if (phys_addr) put_page(virt_to_page(phys_to_virt(phys_addr))); + /* Sync page cache info */ + smp_rmb(); + + /* Release additional page references held for recycling */ + head = 0; + while (head < rbdr->pgcnt) { + pgcache = &rbdr->pgcache[head]; + if (pgcache->page && page_ref_count(pgcache->page) != 0) + put_page(pgcache->page); + head++; + } + /* Free RBDR ring */ nicvf_free_q_desc_mem(nic, &rbdr->dmem); } @@ -269,13 +351,16 @@ static void nicvf_refill_rbdr(struct nicvf *nic, gfp_t gfp) else refill_rb_cnt = qs->rbdr_len - qcount - 1; + /* Sync page cache info */ + smp_rmb(); + /* Start filling descs from tail */ tail = nicvf_queue_reg_read(nic, NIC_QSET_RBDR_0_1_TAIL, rbdr_idx) >> 3; while (refill_rb_cnt) { tail++; tail &= (rbdr->dmem.q_len - 1); - if (nicvf_alloc_rcv_buffer(nic, gfp, RCV_FRAG_LEN, &rbuf)) + if (nicvf_alloc_rcv_buffer(nic, rbdr, gfp, RCV_FRAG_LEN, &rbuf)) break; desc = GET_RBDR_DESC(rbdr, tail); diff --git a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h index 10cb4b8..da48366 100644 --- a/drivers/net/ethernet/cavium/thunder/nicvf_queues.h +++ b/drivers/net/ethernet/cavium/thunder/nicvf_queues.h @@ -213,6 +213,11 @@ struct q_desc_mem { void *unalign_base; }; +struct pgcache { + struct page *page; + u64 dma_addr; +}; + struct rbdr { bool enable; u32 dma_size; @@ -222,6 +227,12 @@ struct rbdr { u32 head; u32 tail; struct q_desc_mem dmem; + + /* For page recycling */ + int pgidx; + int pgcnt; + int pgalloc; + struct pgcache *pgcache; } ____cacheline_aligned_in_smp; struct rcv_queue {