From patchwork Sat Jul 30 17:08:44 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jim Rees X-Patchwork-Id: 1022752 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter1.kernel.org (8.14.4/8.14.4) with ESMTP id p6UH8scF030932 for ; Sat, 30 Jul 2011 17:09:28 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752512Ab1G3RJ1 (ORCPT ); Sat, 30 Jul 2011 13:09:27 -0400 Received: from merit-proxy01.merit.edu ([207.75.116.193]:49632 "EHLO merit-proxy01.merit.edu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752511Ab1G3RJ0 (ORCPT ); Sat, 30 Jul 2011 13:09:26 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by merit-proxy01.merit.edu (Postfix) with ESMTP id E3DB0203986E; Sat, 30 Jul 2011 13:09:25 -0400 (EDT) X-Virus-Scanned: amavisd-new at merit-proxy01.merit.edu Received: from merit-proxy01.merit.edu ([127.0.0.1]) by localhost (merit-proxy01.merit.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cF81H+eX2boZ; Sat, 30 Jul 2011 13:09:25 -0400 (EDT) Received: from merit.edu (74-126-0-171.static.123.net [74.126.0.171]) by merit-proxy01.merit.edu (Postfix) with ESMTPSA id 906AF203986C; Sat, 30 Jul 2011 13:09:24 -0400 (EDT) From: Jim Rees To: Trond Myklebust Cc: linux-nfs@vger.kernel.org, peter honeyman Subject: [PATCH v6 21/26] pnfsblock: encode_layoutcommit Date: Sat, 30 Jul 2011 13:08:44 -0400 Message-Id: <1312045729-27554-22-git-send-email-rees@umich.edu> X-Mailer: git-send-email 1.7.4.1 In-Reply-To: <1312045729-27554-1-git-send-email-rees@umich.edu> References: <1312045729-27554-1-git-send-email-rees@umich.edu> Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter1.kernel.org [140.211.167.41]); Sat, 30 Jul 2011 17:09:28 +0000 (UTC) From: Fred Isaman In blocklayout driver. There are two things happening while layoutcommit/cleanup. 1. the modified extents are encoded. 2. On cleanup the extents are put back on the layout rw extents list, for reads. In the new system where actual xdr encoding is done in encode_layoutcommit() directly into xdr buffer, these are the new commit stages: 1. On setup_layoutcommit, the range is adjusted as before and a structure is allocated for communication with bl_encode_layoutcommit && bl_cleanup_layoutcommit (Generic layer provides a void-star to hang it on) 2. bl_encode_layoutcommit is called to do the actual encoding directly into xdr. The commit-extent-list is not freed and is stored on above structure. FIXME: The code is not yet converted to the new XDR cleanup 3. On cleanup the commit-extent-list is put back by a call to set_to_rw() as before, but with no need for XDR decoding of the list as before. And the commit-extent-list is freed. Finally allocated structure is freed. [rm inode and pnfs_layout_hdr args from cleanup_layoutcommit()] [pnfsblock: get rid of deprecated xdr macros] Signed-off-by: Jim Rees Signed-off-by: Peng Tao Signed-off-by: Fred Isaman [blocklayout: encode_layoutcommit implementation] Signed-off-by: Boaz Harrosh [pnfsblock: fix bug setting up layoutcommit.] Signed-off-by: Tao Guo [pnfsblock: prevent commit list corruption] [pnfsblock: fix layoutcommit with an empty opaque] Signed-off-by: Fred Isaman Signed-off-by: Benny Halevy Signed-off-by: Benny Halevy --- fs/nfs/blocklayout/blocklayout.c | 2 + fs/nfs/blocklayout/blocklayout.h | 12 +++ fs/nfs/blocklayout/extents.c | 176 ++++++++++++++++++++++++++++---------- fs/nfs/nfs4proc.c | 2 +- fs/nfs/pnfs.c | 8 +- fs/nfs/pnfs.h | 6 +- 6 files changed, 152 insertions(+), 54 deletions(-) diff --git a/fs/nfs/blocklayout/blocklayout.c b/fs/nfs/blocklayout/blocklayout.c index c1bc67e..1fc1a2f 100644 --- a/fs/nfs/blocklayout/blocklayout.c +++ b/fs/nfs/blocklayout/blocklayout.c @@ -155,6 +155,8 @@ static void bl_encode_layoutcommit(struct pnfs_layout_hdr *lo, struct xdr_stream *xdr, const struct nfs4_layoutcommit_args *arg) { + dprintk("%s enter\n", __func__); + encode_pnfs_block_layoutupdate(BLK_LO2EXT(lo), xdr, arg); } static void diff --git a/fs/nfs/blocklayout/blocklayout.h b/fs/nfs/blocklayout/blocklayout.h index fcf47b5..3caaefc 100644 --- a/fs/nfs/blocklayout/blocklayout.h +++ b/fs/nfs/blocklayout/blocklayout.h @@ -91,6 +91,15 @@ struct pnfs_block_extent { struct pnfs_inval_markings *be_inval; /* tracks INVAL->RW transition */ }; +/* Shortened extent used by LAYOUTCOMMIT */ +struct pnfs_block_short_extent { + struct list_head bse_node; + struct nfs4_deviceid bse_devid; + struct block_device *bse_mdev; + sector_t bse_f_offset; /* the starting offset in the file */ + sector_t bse_length; /* the size of the extent */ +}; + static inline void BL_INIT_INVAL_MARKS(struct pnfs_inval_markings *marks, sector_t blocksize) { @@ -184,6 +193,9 @@ int bl_mark_sectors_init(struct pnfs_inval_markings *marks, void bl_put_extent(struct pnfs_block_extent *be); struct pnfs_block_extent *bl_alloc_extent(void); int bl_is_sector_init(struct pnfs_inval_markings *marks, sector_t isect); +int encode_pnfs_block_layoutupdate(struct pnfs_block_layout *bl, + struct xdr_stream *xdr, + const struct nfs4_layoutcommit_args *arg); int bl_add_merge_extent(struct pnfs_block_layout *bl, struct pnfs_block_extent *new); diff --git a/fs/nfs/blocklayout/extents.c b/fs/nfs/blocklayout/extents.c index 292aadf..84bf240 100644 --- a/fs/nfs/blocklayout/extents.c +++ b/fs/nfs/blocklayout/extents.c @@ -286,6 +286,49 @@ int bl_mark_sectors_init(struct pnfs_inval_markings *marks, return -ENOMEM; } +/* Marks sectors in [offest, offset+length) as having been written to disk. + * All lengths should be block aligned. + */ +static int mark_written_sectors(struct pnfs_inval_markings *marks, + sector_t offset, sector_t length) +{ + int status; + + dprintk("%s(offset=%llu,len=%llu) enter\n", __func__, + (u64)offset, (u64)length); + spin_lock(&marks->im_lock); + status = _set_range(&marks->im_tree, EXTENT_WRITTEN, offset, length); + spin_unlock(&marks->im_lock); + return status; +} + +static void print_short_extent(struct pnfs_block_short_extent *be) +{ + dprintk("PRINT SHORT EXTENT extent %p\n", be); + if (be) { + dprintk(" be_f_offset %llu\n", (u64)be->bse_f_offset); + dprintk(" be_length %llu\n", (u64)be->bse_length); + } +} + +static void print_clist(struct list_head *list, unsigned int count) +{ + struct pnfs_block_short_extent *be; + unsigned int i = 0; + + ifdebug(FACILITY) { + printk(KERN_DEBUG "****************\n"); + printk(KERN_DEBUG "Extent list looks like:\n"); + list_for_each_entry(be, list, bse_node) { + i++; + print_short_extent(be); + } + if (i != count) + printk(KERN_DEBUG "\n\nExpected %u entries\n\n\n", count); + printk(KERN_DEBUG "****************\n"); + } +} + static void print_bl_extent(struct pnfs_block_extent *be) { dprintk("PRINT EXTENT extent %p\n", be); @@ -378,65 +421,67 @@ bl_add_merge_extent(struct pnfs_block_layout *bl, /* Scan for proper place to insert, extending new to the left * as much as possible. */ - list_for_each_entry_safe(be, tmp, list, be_node) { - if (new->be_f_offset < be->be_f_offset) + list_for_each_entry_safe_reverse(be, tmp, list, be_node) { + if (new->be_f_offset >= be->be_f_offset + be->be_length) break; - if (end <= be->be_f_offset + be->be_length) { - /* new is a subset of existing be*/ + if (new->be_f_offset >= be->be_f_offset) { + if (end <= be->be_f_offset + be->be_length) { + /* new is a subset of existing be*/ + if (extents_consistent(be, new)) { + dprintk("%s: new is subset, ignoring\n", + __func__); + bl_put_extent(new); + return 0; + } else { + goto out_err; + } + } else { + /* |<-- be -->| + * |<-- new -->| */ + if (extents_consistent(be, new)) { + /* extend new to fully replace be */ + new->be_length += new->be_f_offset - + be->be_f_offset; + new->be_f_offset = be->be_f_offset; + new->be_v_offset = be->be_v_offset; + dprintk("%s: removing %p\n", __func__, be); + list_del(&be->be_node); + bl_put_extent(be); + } else { + goto out_err; + } + } + } else if (end >= be->be_f_offset + be->be_length) { + /* new extent overlap existing be */ if (extents_consistent(be, new)) { - dprintk("%s: new is subset, ignoring\n", - __func__); - bl_put_extent(new); - return 0; - } else + /* extend new to fully replace be */ + dprintk("%s: removing %p\n", __func__, be); + list_del(&be->be_node); + bl_put_extent(be); + } else { goto out_err; - } else if (new->be_f_offset <= - be->be_f_offset + be->be_length) { - /* new overlaps or abuts existing be */ - if (extents_consistent(be, new)) { + } + } else if (end > be->be_f_offset) { + /* |<-- be -->| + *|<-- new -->| */ + if (extents_consistent(new, be)) { /* extend new to fully replace be */ - new->be_length += new->be_f_offset - - be->be_f_offset; - new->be_f_offset = be->be_f_offset; - new->be_v_offset = be->be_v_offset; + new->be_length += be->be_f_offset + be->be_length - + new->be_f_offset - new->be_length; dprintk("%s: removing %p\n", __func__, be); list_del(&be->be_node); bl_put_extent(be); - } else if (new->be_f_offset != - be->be_f_offset + be->be_length) + } else { goto out_err; + } } } /* Note that if we never hit the above break, be will not point to a * valid extent. However, in that case &be->be_node==list. */ - list_add_tail(&new->be_node, &be->be_node); + list_add(&new->be_node, &be->be_node); dprintk("%s: inserting new\n", __func__); print_elist(list); - /* Scan forward for overlaps. If we find any, extend new and - * remove the overlapped extent. - */ - be = list_prepare_entry(new, list, be_node); - list_for_each_entry_safe_continue(be, tmp, list, be_node) { - if (end < be->be_f_offset) - break; - /* new overlaps or abuts existing be */ - if (extents_consistent(be, new)) { - if (end < be->be_f_offset + be->be_length) { - /* extend new to fully cover be */ - end = be->be_f_offset + be->be_length; - new->be_length = end - new->be_f_offset; - } - dprintk("%s: removing %p\n", __func__, be); - list_del(&be->be_node); - bl_put_extent(be); - } else if (end != be->be_f_offset) { - list_del(&new->be_node); - goto out_err; - } - } - dprintk("%s: after merging\n", __func__); - print_elist(list); /* FIXME - The per-list consistency checks have all been done, * should now check cross-list consistency. */ @@ -494,6 +539,49 @@ bl_find_get_extent(struct pnfs_block_layout *bl, sector_t isect, return ret; } +int +encode_pnfs_block_layoutupdate(struct pnfs_block_layout *bl, + struct xdr_stream *xdr, + const struct nfs4_layoutcommit_args *arg) +{ + struct pnfs_block_short_extent *lce, *save; + unsigned int count = 0; + __be32 *p, *xdr_start; + + dprintk("%s enter\n", __func__); + /* BUG - creation of bl_commit is buggy - need to wait for + * entire block to be marked WRITTEN before it can be added. + */ + spin_lock(&bl->bl_ext_lock); + /* Want to adjust for possible truncate */ + /* We now want to adjust argument range */ + + /* XDR encode the ranges found */ + xdr_start = xdr_reserve_space(xdr, 8); + if (!xdr_start) + goto out; + list_for_each_entry_safe(lce, save, &bl->bl_commit, bse_node) { + p = xdr_reserve_space(xdr, 7 * 4 + sizeof(lce->bse_devid.data)); + if (!p) + break; + p = xdr_encode_opaque_fixed(p, lce->bse_devid.data, NFS4_DEVICEID4_SIZE); + p = xdr_encode_hyper(p, lce->bse_f_offset << SECTOR_SHIFT); + p = xdr_encode_hyper(p, lce->bse_length << SECTOR_SHIFT); + p = xdr_encode_hyper(p, 0LL); + *p++ = cpu_to_be32(PNFS_BLOCK_READWRITE_DATA); + list_del(&lce->bse_node); + list_add_tail(&lce->bse_node, &bl->bl_committing); + bl->bl_count--; + count++; + } + xdr_start[0] = cpu_to_be32((xdr->p - xdr_start - 1) * 4); + xdr_start[1] = cpu_to_be32(count); +out: + spin_unlock(&bl->bl_ext_lock); + dprintk("%s found %i ranges\n", __func__, count); + return 0; +} + /* Helper function to set_to_rw that initialize a new extent */ static void _prep_new_extent(struct pnfs_block_extent *new, diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 6cb84b4..8c77039 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -5963,7 +5963,7 @@ static void nfs4_layoutcommit_release(void *calldata) struct nfs4_layoutcommit_data *data = calldata; struct pnfs_layout_segment *lseg, *tmp; - pnfs_cleanup_layoutcommit(data->args.inode, data); + pnfs_cleanup_layoutcommit(data); /* Matched by references in pnfs_set_layoutcommit */ list_for_each_entry_safe(lseg, tmp, &data->lseg_list, pls_lc_list) { list_del_init(&lseg->pls_lc_list); diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index c1cc216..e550e88 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -1411,14 +1411,12 @@ pnfs_set_layoutcommit(struct nfs_write_data *wdata) } EXPORT_SYMBOL_GPL(pnfs_set_layoutcommit); -void pnfs_cleanup_layoutcommit(struct inode *inode, - struct nfs4_layoutcommit_data *data) +void pnfs_cleanup_layoutcommit(struct nfs4_layoutcommit_data *data) { - struct nfs_server *nfss = NFS_SERVER(inode); + struct nfs_server *nfss = NFS_SERVER(data->args.inode); if (nfss->pnfs_curr_ld->cleanup_layoutcommit) - nfss->pnfs_curr_ld->cleanup_layoutcommit(NFS_I(inode)->layout, - data); + nfss->pnfs_curr_ld->cleanup_layoutcommit(data); } /* diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index 82dde37..e0b5d80 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -113,8 +113,7 @@ struct pnfs_layoutdriver_type { struct xdr_stream *xdr, const struct nfs4_layoutreturn_args *args); - void (*cleanup_layoutcommit) (struct pnfs_layout_hdr *layoutid, - struct nfs4_layoutcommit_data *data); + void (*cleanup_layoutcommit) (struct nfs4_layoutcommit_data *data); void (*encode_layoutcommit) (struct pnfs_layout_hdr *layoutid, struct xdr_stream *xdr, @@ -198,8 +197,7 @@ void pnfs_roc_release(struct inode *ino); void pnfs_roc_set_barrier(struct inode *ino, u32 barrier); bool pnfs_roc_drain(struct inode *ino, u32 *barrier); void pnfs_set_layoutcommit(struct nfs_write_data *wdata); -void pnfs_cleanup_layoutcommit(struct inode *inode, - struct nfs4_layoutcommit_data *data); +void pnfs_cleanup_layoutcommit(struct nfs4_layoutcommit_data *data); int pnfs_layoutcommit_inode(struct inode *inode, bool sync); int _pnfs_return_layout(struct inode *); int pnfs_ld_write_done(struct nfs_write_data *);