From patchwork Wed May 13 18:35:55 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dan Williams X-Patchwork-Id: 6399461 X-Patchwork-Delegate: dan.j.williams@gmail.com Return-Path: X-Original-To: patchwork-linux-nvdimm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 9009F9F32E for ; Wed, 13 May 2015 18:36:00 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 3906920435 for ; Wed, 13 May 2015 18:35:59 +0000 (UTC) Received: from ml01.01.org (ml01.01.org [198.145.21.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A0AFE20437 for ; Wed, 13 May 2015 18:35:57 +0000 (UTC) Received: from ml01.vlan14.01.org (localhost [IPv6:::1]) by ml01.01.org (Postfix) with ESMTP id 6AF21182684; Wed, 13 May 2015 11:35:57 -0700 (PDT) X-Original-To: linux-nvdimm@lists.01.org Delivered-To: linux-nvdimm@lists.01.org Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by ml01.01.org (Postfix) with ESMTP id E054C182682 for ; Wed, 13 May 2015 11:35:56 -0700 (PDT) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga103.jf.intel.com with ESMTP; 13 May 2015 11:35:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,422,1427785200"; d="scan'208";a="570900898" Received: from orsmsx106.amr.corp.intel.com ([10.22.225.133]) by orsmga003.jf.intel.com with ESMTP; 13 May 2015 11:35:56 -0700 Received: from orsmsx158.amr.corp.intel.com (10.22.240.20) by ORSMSX106.amr.corp.intel.com (10.22.225.133) with Microsoft SMTP Server (TLS) id 14.3.224.2; Wed, 13 May 2015 11:35:56 -0700 Received: from orsmsx107.amr.corp.intel.com ([169.254.1.103]) by ORSMSX158.amr.corp.intel.com ([169.254.10.149]) with mapi id 14.03.0224.002; Wed, 13 May 2015 11:35:56 -0700 From: "Williams, Dan J" To: "hch@lst.de" , "axboe@fb.com" Subject: Re: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries Thread-Topic: [PATCH v3 06/11] scatterlist: support "page-less" (__pfn_t only) entries Thread-Index: AQHQjGy7YpNy8ibWSU6+xloLIzm5BJ16s8OA Date: Wed, 13 May 2015 18:35:55 +0000 Message-ID: <1431542149.31415.10.camel@intel.com> References: <20150512042629.11521.70356.stgit@dwillia2-desk3.amr.corp.intel.com> <20150512043001.11521.3526.stgit@dwillia2-desk3.amr.corp.intel.com> In-Reply-To: <20150512043001.11521.3526.stgit@dwillia2-desk3.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.138] Content-ID: <1FFD39A6595B33489C3782FA877A827C@intel.com> MIME-Version: 1.0 Cc: "axboe@kernel.dk" , "linux-arch@vger.kernel.org" , "riel@redhat.com" , "linux-nvdimm@lists.01.org" , "linux-fsdevel@vger.kernel.org" , "david@fromorbit.com" , "linux-kernel@vger.kernel.org" , "dmaengine@vger.kernel.org" , "mgorman@suse.de" , "j.glisse@gmail.com" , "akpm@linux-foundation.org" , "mingo@kernel.org" X-BeenThere: linux-nvdimm@lists.01.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: "Linux-nvdimm developer list." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP [ adding willy (yes, need to fix my scripts), and dmaengine folks] Jens, Christoph, I've rebased this patch series block/for-next. With commit 84be456f883c "remove " I think we can take the next step of removing all references to page_link and just use __pfn_t by default. v4 patch below. Jens, I'm wondering if you want to take this series(.) as patches or prepare a git branch to pull? 8<-------- Subject: scatterlist: support "page-less" (__pfn_t only) entries From: Matthew Wilcox Given that an offset will never be more than PAGE_SIZE, steal the unused bits of the offset to implement a flags field. Move the existing "this is a sg_chain() or sg_last() entry" flags to the new 'flags' field. [djbw: rebase on block/for-4.2] Signed-off-by: Dan Williams Signed-off-by: Matthew Wilcox --- block/blk-merge.c | 2 - drivers/crypto/omap-sham.c | 2 - drivers/dma/imx-dma.c | 7 +-- drivers/dma/ste_dma40.c | 5 -- drivers/mmc/card/queue.c | 4 +- include/crypto/scatterwalk.h | 9 +--- include/linux/scatterlist.h | 92 ++++++++++++++++++++++++++---------------- samples/kfifo/dma-example.c | 8 ++-- 8 files changed, 69 insertions(+), 60 deletions(-) diff --git a/block/blk-merge.c b/block/blk-merge.c index 218ad1e57a49..82a688551b72 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -267,7 +267,7 @@ int blk_rq_map_sg(struct request_queue *q, struct request *rq, if (rq->cmd_flags & REQ_WRITE) memset(q->dma_drain_buffer, 0, q->dma_drain_size); - sg->page_link &= ~0x02; + sg_unmark_end(sg); sg = sg_next(sg); sg_set_page(sg, virt_to_page(q->dma_drain_buffer), q->dma_drain_size, diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c index 4d63e0d4da9a..21736afd3320 100644 --- a/drivers/crypto/omap-sham.c +++ b/drivers/crypto/omap-sham.c @@ -582,7 +582,7 @@ static int omap_sham_xmit_dma(struct omap_sham_dev *dd, dma_addr_t dma_addr, * the dmaengine may try to DMA the incorrect amount of data. */ sg_init_table(&ctx->sgl, 1); - ctx->sgl.page_link = ctx->sg->page_link; + ctx->sgl.pfn = ctx->sg->pfn; ctx->sgl.offset = ctx->sg->offset; sg_dma_len(&ctx->sgl) = len32; sg_dma_address(&ctx->sgl) = sg_dma_address(ctx->sg); diff --git a/drivers/dma/imx-dma.c b/drivers/dma/imx-dma.c index eed405976ea9..a767727bcfef 100644 --- a/drivers/dma/imx-dma.c +++ b/drivers/dma/imx-dma.c @@ -886,7 +886,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic( sg_init_table(imxdmac->sg_list, periods); for (i = 0; i < periods; i++) { - imxdmac->sg_list[i].page_link = 0; + imxdmac->sg_list[i].pfn.page = NULL; imxdmac->sg_list[i].offset = 0; imxdmac->sg_list[i].dma_address = dma_addr; sg_dma_len(&imxdmac->sg_list[i]) = period_len; @@ -894,10 +894,7 @@ static struct dma_async_tx_descriptor *imxdma_prep_dma_cyclic( } /* close the loop */ - imxdmac->sg_list[periods].offset = 0; - sg_dma_len(&imxdmac->sg_list[periods]) = 0; - imxdmac->sg_list[periods].page_link = - ((unsigned long)imxdmac->sg_list | 0x01) & ~0x02; + sg_chain(imxdmac->sg_list, periods + 1, imxdmac->sg_list); desc->type = IMXDMA_DESC_CYCLIC; desc->sg = imxdmac->sg_list; diff --git a/drivers/dma/ste_dma40.c b/drivers/dma/ste_dma40.c index 3c10f034d4b9..e8c00642cacb 100644 --- a/drivers/dma/ste_dma40.c +++ b/drivers/dma/ste_dma40.c @@ -2562,10 +2562,7 @@ dma40_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t dma_addr, dma_addr += period_len; } - sg[periods].offset = 0; - sg_dma_len(&sg[periods]) = 0; - sg[periods].page_link = - ((unsigned long)sg | 0x01) & ~0x02; + sg_chain(sg, periods + 1, sg); txd = d40_prep_sg(chan, sg, sg, periods, direction, DMA_PREP_INTERRUPT); diff --git a/drivers/mmc/card/queue.c b/drivers/mmc/card/queue.c index 236d194c2883..127f76294e71 100644 --- a/drivers/mmc/card/queue.c +++ b/drivers/mmc/card/queue.c @@ -469,7 +469,7 @@ static unsigned int mmc_queue_packed_map_sg(struct mmc_queue *mq, sg_set_buf(__sg, buf + offset, len); offset += len; remain -= len; - (__sg++)->page_link &= ~0x02; + sg_unmark_end(__sg++); sg_len++; } while (remain); } @@ -477,7 +477,7 @@ static unsigned int mmc_queue_packed_map_sg(struct mmc_queue *mq, list_for_each_entry(req, &packed->list, queuelist) { sg_len += blk_rq_map_sg(mq->queue, req, __sg); __sg = sg + (sg_len - 1); - (__sg++)->page_link &= ~0x02; + sg_unmark_end(__sg++); } sg_mark_end(sg + (sg_len - 1)); return sg_len; diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index 20e4226a2e14..4529889b0f07 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -25,13 +25,8 @@ #include #include -static inline void scatterwalk_sg_chain(struct scatterlist *sg1, int num, - struct scatterlist *sg2) -{ - sg_set_page(&sg1[num - 1], (void *)sg2, 0, 0); - sg1[num - 1].page_link &= ~0x02; - sg1[num - 1].page_link |= 0x01; -} +#define scatterwalk_sg_chain(prv, num, sgl) sg_chain(prv, num, sgl) +#define scatterwalk_sg_next(sgl) sg_next(sgl) static inline void scatterwalk_crypto_chain(struct scatterlist *head, struct scatterlist *sg, diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index eca1ec93775c..98c21b460292 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -9,14 +9,18 @@ struct scatterlist { #ifdef CONFIG_DEBUG_SG - unsigned long sg_magic; + unsigned long sg_magic; #endif - unsigned long page_link; - unsigned int offset; - unsigned int length; - dma_addr_t dma_address; + union { + __pfn_t pfn; + struct scatterlist *next; + }; + unsigned short offset; + unsigned short sg_flags; + unsigned int length; + dma_addr_t dma_address; #ifdef CONFIG_NEED_SG_DMA_LENGTH - unsigned int dma_length; + unsigned int dma_length; #endif }; @@ -44,30 +48,29 @@ struct sg_table { /* * Notes on SG table design. * - * We use the unsigned long page_link field in the scatterlist struct to place - * the page pointer AND encode information about the sg table as well. The two - * lower bits are reserved for this information. + * We use the fact that a given sg entry will never be larger than a + * page to pack 'offset' and 'sg_flags' into 32-bits. This supports a + * PAGE_SIZE up to 64K and flags to encode information about the sg + * entry. * - * If bit 0 is set, then the page_link contains a pointer to the next sg - * table list. Otherwise the next entry is at sg + 1. + * If SG_FLAGS_CHAIN is set, then the 'next' member of the entry is a + * pointer to the next sg table list. Otherwise the next entry is at sg + * + 1. * - * If bit 1 is set, then this sg entry is the last element in a list. + * If SG_FLAGS_LAST is set, then this sg entry is the last element in a + * list. * * See sg_next(). * */ #define SG_MAGIC 0x87654321 +#define SG_FLAGS_CHAIN 0x0001 +#define SG_FLAGS_LAST 0x0002 -/* - * We overload the LSB of the page pointer to indicate whether it's - * a valid sg entry, or whether it points to the start of a new scatterlist. - * Those low bits are there for everyone! (thanks mason :-) - */ -#define sg_is_chain(sg) ((sg)->page_link & 0x01) -#define sg_is_last(sg) ((sg)->page_link & 0x02) -#define sg_chain_ptr(sg) \ - ((struct scatterlist *) ((sg)->page_link & ~0x03)) +#define sg_is_chain(sg) ((sg)->sg_flags & SG_FLAGS_CHAIN) +#define sg_is_last(sg) ((sg)->sg_flags & SG_FLAGS_LAST) +#define sg_chain_ptr(sg) ((sg)->next) /** * sg_assign_page - Assign a given page to an SG entry @@ -81,18 +84,15 @@ struct sg_table { **/ static inline void sg_assign_page(struct scatterlist *sg, struct page *page) { - unsigned long page_link = sg->page_link & 0x3; - - /* - * In order for the low bit stealing approach to work, pages - * must be aligned at a 32-bit boundary as a minimum. - */ - BUG_ON((unsigned long) page & 0x03); #ifdef CONFIG_DEBUG_SG BUG_ON(sg->sg_magic != SG_MAGIC); BUG_ON(sg_is_chain(sg)); #endif - sg->page_link = page_link | (unsigned long) page; + sg->pfn = page_to_pfn_t(page); +#ifdef CONFIG_DEBUG_SG + BUG_ON(sg->sg_magic != SG_MAGIC); + BUG_ON(sg_is_chain(sg)); +#endif } /** @@ -113,17 +113,32 @@ static inline void sg_set_page(struct scatterlist *sg, struct page *page, unsigned int len, unsigned int offset) { sg_assign_page(sg, page); + BUG_ON(offset > 65535); sg->offset = offset; sg->length = len; } +static inline void sg_set_pfn(struct scatterlist *sg, __pfn_t pfn, + unsigned int len, unsigned int offset) +{ +#ifdef CONFIG_DEBUG_SG + BUG_ON(sg->sg_magic != SG_MAGIC); + BUG_ON(sg_is_chain(sg)); +#endif + sg->pfn = pfn; + BUG_ON(offset > 65535); + sg->offset = offset; + sg->sg_flags = 0; + sg->length = len; +} + static inline struct page *sg_page(struct scatterlist *sg) { #ifdef CONFIG_DEBUG_SG BUG_ON(sg->sg_magic != SG_MAGIC); BUG_ON(sg_is_chain(sg)); #endif - return (struct page *)((sg)->page_link & ~0x3); + return __pfn_t_to_page(sg->pfn); } /** @@ -175,7 +190,8 @@ static inline void sg_chain(struct scatterlist *prv, unsigned int prv_nents, * Set lowest bit to indicate a link pointer, and make sure to clear * the termination bit if it happens to be set. */ - prv[prv_nents - 1].page_link = ((unsigned long) sgl | 0x01) & ~0x02; + prv[prv_nents - 1].next = sgl; + prv[prv_nents - 1].sg_flags = SG_FLAGS_CHAIN; } /** @@ -195,8 +211,8 @@ static inline void sg_mark_end(struct scatterlist *sg) /* * Set termination bit, clear potential chain bit */ - sg->page_link |= 0x02; - sg->page_link &= ~0x01; + sg->sg_flags |= SG_FLAGS_LAST; + sg->sg_flags &= ~SG_FLAGS_CHAIN; } /** @@ -212,7 +228,7 @@ static inline void sg_unmark_end(struct scatterlist *sg) #ifdef CONFIG_DEBUG_SG BUG_ON(sg->sg_magic != SG_MAGIC); #endif - sg->page_link &= ~0x02; + sg->sg_flags &= ~SG_FLAGS_LAST; } /** @@ -227,7 +243,7 @@ static inline void sg_unmark_end(struct scatterlist *sg) **/ static inline dma_addr_t sg_phys(struct scatterlist *sg) { - return page_to_phys(sg_page(sg)) + sg->offset; + return __pfn_t_to_phys(sg->pfn) + sg->offset; } /** @@ -242,7 +258,11 @@ static inline dma_addr_t sg_phys(struct scatterlist *sg) **/ static inline void *sg_virt(struct scatterlist *sg) { - return page_address(sg_page(sg)) + sg->offset; + struct page *page; + + page = __pfn_t_to_page(sg->pfn) + sg->offset; + BUG_ON(!page); /* don't use sg_virt() on unmapped memory */ + return page_address(page) + sg->offset; } int sg_nents(struct scatterlist *sg); diff --git a/samples/kfifo/dma-example.c b/samples/kfifo/dma-example.c index aa243db93f01..3eeff9a56e0e 100644 --- a/samples/kfifo/dma-example.c +++ b/samples/kfifo/dma-example.c @@ -75,8 +75,8 @@ static int __init example_init(void) for (i = 0; i < nents; i++) { printk(KERN_INFO "sg[%d] -> " - "page_link 0x%.8lx offset 0x%.8x length 0x%.8x\n", - i, sg[i].page_link, sg[i].offset, sg[i].length); + "pfn_data 0x%.8lx offset 0x%.8x length 0x%.8x\n", + i, sg[i].pfn.data, sg[i].offset, sg[i].length); if (sg_is_last(&sg[i])) break; @@ -104,8 +104,8 @@ static int __init example_init(void) for (i = 0; i < nents; i++) { printk(KERN_INFO "sg[%d] -> " - "page_link 0x%.8lx offset 0x%.8x length 0x%.8x\n", - i, sg[i].page_link, sg[i].offset, sg[i].length); + "pfn_data 0x%.8lx offset 0x%.8x length 0x%.8x\n", + i, sg[i].pfn.data, sg[i].offset, sg[i].length); if (sg_is_last(&sg[i])) break;