From patchwork Thu Nov 15 14:20:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Aur=C3=A9lien_Aptel?= X-Patchwork-Id: 10684401 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8002F13BF for ; Thu, 15 Nov 2018 14:21:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D2D02C810 for ; Thu, 15 Nov 2018 14:21:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5FC212C820; Thu, 15 Nov 2018 14:21:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C2542C810 for ; Thu, 15 Nov 2018 14:21:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387548AbeKPA3U (ORCPT ); Thu, 15 Nov 2018 19:29:20 -0500 Received: from mx2.suse.de ([195.135.220.15]:42522 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1729035AbeKPA3U (ORCPT ); Thu, 15 Nov 2018 19:29:20 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 232CFAD70; Thu, 15 Nov 2018 14:21:16 +0000 (UTC) From: Aurelien Aptel To: linux-cifs@vger.kernel.org Cc: smfrench@gmail.com, Aurelien Aptel Subject: [PATCH v1 00/14] DFS failover Date: Thu, 15 Nov 2018 15:20:49 +0100 Message-Id: <20181115142103.24617-1-aaptel@suse.com> X-Mailer: git-send-email 2.13.7 Sender: linux-cifs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-cifs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi linux-cifs, This patchset adds * a DFS cache so that DFS links can be resolved even when hosts are down * DFS failover so that if the DFS target we are connected to is down cifs.ko will try to reconnect to a different target if there are any. This is 90% Paulo's work, I gave him the task and a general roadmap to go about it thinking it would not be too hard and it turned into this massive, intricate 2k lines beast as we both slowly realized all the ins and outs and subtleties of the problem and resulting solutions. So congrats to him. * * * * What is DFS? DFS is basically symbolic links across servers. You can have links that points to a UNC path to directories on other servers. The client has to resolve the link, get the target from the result, and connect to the target. There is no proxy-ing involved. A share that can take resolving requests and respond to them is called a DFS root. A host with such a share is in a "standalone" setup. A domain can be setup so that when accessing a share, the share *itself* is redirected to a DFS root on a separate server. Essentially the share is a symbolic link itself and the domain doesn't host it, even though you typed \\domain\dfsshare to access it. This is called a "domain-based" setup. Each of those links can have multiple targets so that if one fails you can try another. So how do you know a file is a link or not? Well if you try to access it the server will reply with the error STATUS_PATH_NOT_COVERED. Then you are supposed to issue a DFS Referral Request on that path and you get the targets in the response. You are then supposed to connect to one of the target and repeat. Microsoft has a dedicated document [MS-DFSC] about it which I suggest you take a look. * How does DFS work in cifs.ko? There are 2 entry points for DFS currently: - cifs_mount(): the "static" codepath - dentry automount: the dynamic codepath The static code path is when you mount a UNC path \\FOO\d1\d2\d3. cifs_mount() will follow all the links until you reach the final host and mount that as a regular mount point. The dynamic code path is when you mount a DFS root (= a share that has links) and, *after* mounting, access one of those links. In this case cifs.ko will set an AUTOMOUNT flag on the inode&dentry of the file which is a linux VFS thing that instruct the VFS upper-layer to lazily call the d_automount() dentry operations when needed. That operation does a VFS sub-mount on that dentry. It ends up calling cifs_mount() and it will get its own superblock and everything. The problem is we pass the resolved path when mounting which means if it fails, we cannot resolve again and use a different target. Note that the original mount path can have multiple nested links. This is why we need a cache to store results to do failover properly. * What this patch adds - a DFS cache so that DFS links can be resolved even when hosts are down - DFS failover so that if the DFS target we are connected to is down cifs.ko will try to reconnect to a different target if there are any +- cifs: Refactor out cifs_mount() | cifs: Skip any trailing backslashes from UNC refactor | cifs: Fix separator when building path from dentry &bugfix | cifs: Make devname param optional in cifs_compose_mount_options() | cifs: Respect -EAGAIN when querying paths | cifs: Save TTL value when parsing DFS referrals +- cifs: auto disable 'serverino' in dfs mounts new impl,| cifs: Add DFS cache routines <------ main new code replace, | cifs: Make use of DFS cache to get new DFS referrals reco | cifs: Add support for failover in cifs_mount() & | cifs: Add support for failover in cifs_reconnect() failover | cifs: start DFS cache refresher in cifs_mount() | cifs: Add support for failover in smb2_reconnect() +- cifs: Add support for failover in cifs_reconnect_tcon() ** The DFS cache The DFS cache is a hashtable that maps UNC paths to cache entries. A cache entry contains: - the UNC path it is mapped on - how much the the UNC path the entry consumes - flags - a Time-To-Live after which the entry expires - a list of possible targets (linked lists of UNC paths) - a "hint target" pointing the last known working target or the first target if none were tried. This hint lets cifs.ko remember and try working targets first. * Looking for an entry in the cache is done with dfs_cache_find() - if no valid entries are found, a DFS query is made, stored in the cache and returned - the full target list can be copied and returned to avoid race conditions and looped on with the help with the dfs_cache_tgt_iterator * Updating the target hint to the next target is done with dfs_cache_update_tgthint() These functions have a dfs_cache_noreq_XXX() version that doesn't fetches referrals if no entries are found. These versions don't require the tcp/ses/tcon/cifs_sb parameters as a result. ** Refreshing expired cache entries Expired entries cannot be used and since they have a pretty short TTL in order for them to be useful for failover the DFS cache adds a delayed work called periodically to keep them fresh. Since we might not have available connections to issue the referral request when refreshing we need to store volume_info structs with credentials and other needed info to be able to connect to the right server. ** Mount failover The static and dynamic codepaths were patched to use the DFS cache to try alternative targets. We store the initial user-provided mount path in the superblock as origin_fullpath. ** Reconnect failover As you know the reconnect logic isn't the simplest to follow and we had to tweak some things: Since we might try to reconnect to multiple targets and we do this sequentially threads waiting for tcp reconnection should wait the socket timeout x number of targets. ** Server file id When following a DFS link you connect to a different server with a different set of file ids. Those 2 sets of ids can overlap as a result. Similarly if you failover to a different server, you will get a different set file ids than the ones you initially got from your original servers. We decided to disable server inode in case of failover and had to tweak the logic of dentry revalidation in order to not return -ESTALE on syscalls. ** Remaining issues We hit problem sometime where we suspect if the reconnect codepath is triggered *while* mounting a DFS link you hit a NULL-ptr deref in the reconnection code. This may be an already existing bug. * Testing This was tested in various ways: - static and dynamic path at mount time with random target initially down - dropping packets from an already mounted connection and waiting for IO timeout or echo-thread timeout - soft and hard mount (hard means only fail to userspace if there is no other way) We have a little testsuite that tries mounting every weird combinations of links and paths in the static or dynamic code path and a simple reconnect test. We used iptables on the client to drop packets to/from the server to simulate failure. All of our testing was on very short lived sessions though, and we need more testing on long living ones. Our test setups were the following: * a 3 VM Windows Server "domain-based" setup (SMB1, SMB3): (share link) DOM/dfstest -> DFSROOT1/dfstest [files and links to] -> {DFSROOT1/share1,DFSROOT2/share2} -> DFSROOT2/dfstest [files and links to] -> {DFSROOT1/share1,DFSROOT2/share2} * a 3 VM "standalone" samba setup (SMB1+unix extension, SMB3): ROOT/dfstest [files and links to] -> {TARGET1/share1, TARGET2/share2} * * * Paulo Alcantara (14): cifs: Refactor out cifs_mount() cifs: Skip any trailing backslashes from UNC cifs: Fix separator when building path from dentry cifs: Make devname param optional in cifs_compose_mount_options() cifs: Respect -EAGAIN when querying paths cifs: Save TTL value when parsing DFS referrals cifs: auto disable 'serverino' in dfs mounts cifs: Add DFS cache routines cifs: Make use of DFS cache to get new DFS referrals cifs: Add support for failover in cifs_mount() cifs: Add support for failover in cifs_reconnect() cifs: start DFS cache refresher in cifs_mount() cifs: Add support for failover in smb2_reconnect() cifs: Add support for failover in cifs_reconnect_tcon() fs/cifs/Makefile | 2 +- fs/cifs/cifs_debug.c | 12 + fs/cifs/cifs_dfs_ref.c | 141 +++-- fs/cifs/cifs_fs_sb.h | 9 + fs/cifs/cifsfs.c | 17 +- fs/cifs/cifsglob.h | 14 +- fs/cifs/cifsproto.h | 28 +- fs/cifs/cifssmb.c | 88 ++- fs/cifs/connect.c | 889 +++++++++++++++++++++++-------- fs/cifs/dfs_cache.c | 1379 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/cifs/dfs_cache.h | 97 ++++ fs/cifs/dir.c | 2 +- fs/cifs/inode.c | 49 +- fs/cifs/misc.c | 34 +- fs/cifs/smb1ops.c | 15 +- fs/cifs/smb2ops.c | 23 +- fs/cifs/smb2pdu.c | 88 ++- 17 files changed, 2565 insertions(+), 322 deletions(-) create mode 100644 fs/cifs/dfs_cache.c create mode 100644 fs/cifs/dfs_cache.h