From patchwork Wed Feb 7 23:41:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10206341 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 43BC060327 for ; Wed, 7 Feb 2018 23:42:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 339C4290F6 for ; Wed, 7 Feb 2018 23:42:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 27BAF2916B; Wed, 7 Feb 2018 23:42:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68D62290F6 for ; Wed, 7 Feb 2018 23:42:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750745AbeBGXmG (ORCPT ); Wed, 7 Feb 2018 18:42:06 -0500 Received: from mail-he1eur01on0068.outbound.protection.outlook.com ([104.47.0.68]:32992 "EHLO EUR01-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750729AbeBGXmF (ORCPT ); Wed, 7 Feb 2018 18:42:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=2Rm+Zr5njrKgp9nyZKTZfDYxk9BayLWyiIs9A+xznhI=; b=IM3irvUnEuwpPV1nOu2LlZnrzKdulFasCnK4pq/cOudWTV/XbXENkjE+4rFvvp4CnLqHgIyJOBXSdjEcI5bTN+Cxqzw7ICOuErl9YgexjrSoyNZe1YUkKhaMqzH+6RNlaJRTZLL4IypRRlI0RjS2mENjaVCAVroVpp5L1n+SXT4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; Received: from mlx.ziepe.ca (70.74.179.152) by HE1PR0501MB2860.eurprd05.prod.outlook.com (2603:10a6:3:c2::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.464.11; Wed, 7 Feb 2018 23:42:01 +0000 Received: from jgg by mlx.ziepe.ca with local (Exim 4.86_2) (envelope-from ) id 1ejZLd-0003i1-JJ; Wed, 07 Feb 2018 16:41:53 -0700 Date: Wed, 7 Feb 2018 16:41:53 -0700 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org, Moni Shoua , Matan Barak Subject: [RFC PATCH] rxe: Fix structure layouts for 32/64 bit compat Message-ID: <20180207234153.GA14067@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Originating-IP: [70.74.179.152] X-ClientProxiedBy: VI1PR08CA0222.eurprd08.prod.outlook.com (2603:10a6:802:15::31) To HE1PR0501MB2860.eurprd05.prod.outlook.com (2603:10a6:3:c2::8) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: c0f07ef0-2a3e-485e-3177-08d56e8462aa X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(4534165)(4627221)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:HE1PR0501MB2860; X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2860; 3:si7A3SkCCnWv6oWRI1e1WSI0pP1CGLuKH6UR/0JJJhyI3snQ8a1/qTnxPWRWiS8adRhj4njh+Axe2elQz/R0Qp9vutzlNgPKv2amnH6GGq5/lIVHk3sRd3Hh77+P15Eoz+sSc8p6IlA8/GKw18JwFDNQfAZ6+1rQKYtawarpAseisaNkUgkuO5ikzVpgPUNjPH4LNXvi8xTWRUg0+uhfnTRr3vNvKJC585Fu0jOU42PmXtR742rzR9qpQlDLJ+hF; 25:aySdyjjWnRT9rqDp6oDfRCzoFrhUWvQ6ZeOAJnrtRolCnkvprsFWkdX53zPPP6j9EEUWO+QVN7wSC1MrMBFsUZSeCNOsPli8RQGSnBKT9E4v+yh1Z9pJdnf3IVACRre5w728wRlC3myICH1Rm4+STlOVaKTltth1z1x8FyMOOV1kPCWrPG++plu+UnbRR+r7NT9SiZjnkwySJOOu8gkS++p5toihGuomfr+RO0yJbYp1ZGd8XjIVYl9UYYLnriStKR5/cHk1Xz4uexJMQRU53Pgnr5FhqsGpcAhlj5/NvLUrbB4qoX+2eBLDYLqH7yaQ9TwmyOBe8WGwSq24PBaj9g==; 31:9aa43wNhhxp0hHGrvecpvUm46tr9ZIQjImMVA7Ux04ov4uIzwVYTvwSCEnyInuYnJ0i/maUSkdYZFr6NOkCnJUcmjgEAGfA81tITP6RuJJOcybfn5rVCNvJCi0ippOsiQFimdmUY/me8Jx6u8hvwHt6zrajyp/6rKEN6nH+rsKPss1UUhlZpI2O8wOzUk8xY8Eg+zL4vGPb+R9X+0E7/OKuhzTCUc5D/fY2/+zI/K2w= X-MS-TrafficTypeDiagnostic: HE1PR0501MB2860: X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2860; 20:djTpkwcenOHBjw8ro4q0lBSOMiuDLc3MmvPyU/E4tqOirB+j6d7aIejvbAHt5OhejfLJBWgNYgmfUSJmm+HLgUZDFafmGhHgEJ5ETx0FgtoRco686dV/jqIHJGT3RnkBDDcGsBpQ8UNDgaT3gPSKJi2nLlx/m7v0DP86YawNhX7YTpoWwvJaQ1SKOFLu8HkpJOME2jBC6a1+lhiNU8xUuDUSWsa+nswWiS0juXn9SXZf6NThAuL5d0X6sqvcwyjzUFNh5hIzd76jCy1aNXP9IU85Tnr8J3JYW7valme/wNwPHg3fjmBVk+T0Wtls++06iYO1o/jFqmY7bDOlcftey8/X9YarAqjD1YZqYteeXOh5Tg7e8Q1UfHvfiIpDf1XoBe53KzPY3TBJ6UYeoIfkRY2joq88GHJNLJVRGGu39w4jbrk/yd1mXlOdVyRDJs4pLq1uksKdhMtx+yi3OlQ7v51wdFhW/BkfdozXxW0JTvQYPe9hm4uyyC5wUcD8bbS7; 4:bGcaJKZvtTx48eN+HmBS+tfKrnF37z4c3GDWr7C2aRVe72Sg9/fwlnhLhEvpiLmp4ljdsA7X3+BOkGpDLhm1x4UIMm34D4RXlXFIIG5PtKJLqljJ+O1LkVIZkCpbm0o1BS129U4lBf5ubJI7d8YTBDBmhHnIqS90uoWJ9slLuot4/mF9NmlAOfaKu1FH1AF37d4Pj2Bfpl1eMnvCuCNEsQK603OT2s4uRxCQ1mOXjnwNdSKUkE6n9c4mzpiwrpPnn7UahNWX+0ifSelRyYYPBw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040501)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231101)(2400082)(944501161)(3002001)(10201501046)(6055026)(6041288)(20161123558120)(20161123562045)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:HE1PR0501MB2860; BCL:0; PCL:0; RULEID:; SRVR:HE1PR0501MB2860; X-Forefront-PRVS: 0576145E86 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(346002)(39380400002)(366004)(376002)(39860400002)(396003)(189003)(199004)(59450400001)(50466002)(33656002)(86362001)(9786002)(305945005)(83506002)(9746002)(69596002)(1076002)(83796002)(57986006)(97736004)(52116002)(6116002)(2906002)(68736007)(3846002)(106356001)(36756003)(122856001)(316002)(6636002)(386003)(5660300001)(53936002)(23726003)(26005)(66066001)(46656002)(47776003)(105586002)(110136005)(9686003)(58126008)(478600001)(81166006)(186003)(81156014)(33896004)(8676002)(8936002)(7736002)(16586007)(18370500001)(24400500001)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0501MB2860; H:mlx.ziepe.ca; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; HE1PR0501MB2860; 23:+EF9A2fgpHzFI5DgEBNgbAVDeGb7LpeBba+NcKz?= =?us-ascii?Q?FzHTX0t52yZmsJHvecFOYk6TZ8TNPsjxP4m+SGJGxiL42Z2ufanOrXdakqwY?= =?us-ascii?Q?xEF4FLT40QU7O2O2dGpm2dXmDPqbb5IrxzdVp6qEYLxhX0kYf1V1x9CH8E9G?= =?us-ascii?Q?JUkZffXx0pi4P9TMZ6vJS75ho50MJkO0cnwIAlFebmVPOTMecnaOJEb/HSTE?= =?us-ascii?Q?a2ssXpywGTeScoJG3NZ3PapCHp3s4980KcMMF9BDC9x59yjwqtZriGQ1pJtr?= =?us-ascii?Q?axN+fBg0S9/TCjRDwiz7cWbZqw1rEONm8j5ur+qJREtR8DyDtFlc44pWcw/x?= =?us-ascii?Q?1tpKazIj6IVmU3prRydJs9svk8lc9otfDIu9diUhKuByju96VudspfQjMiOo?= =?us-ascii?Q?HDWKwFKpGKniv3eQeBqNynzcCUpNtaKSj1LcLaV91IqukUJ3EKcsIgGZ3tbx?= =?us-ascii?Q?dQZK4jXbf/zfaAfv/0X+73UFsWzJEnOKZQmxSVvyEule4oAUNDAUhEvOZ8HP?= =?us-ascii?Q?zLifVIYFkJUSPc2f35RNnXxovWxMZ1zUBb4dPgU4RhxElFEzbpLABhP2Ncf2?= =?us-ascii?Q?uhwZ48iUvHoD7mchoKFSjQLqFpoP4k9yqQEB4WgQLrNmU9hVWTi6DLPYWmwX?= =?us-ascii?Q?oO2xWMCoypu2/tAtL9Qbfh4rQFx7Pn1yoYSVCTyIoaEXKjeM0HlYm7ROilPz?= =?us-ascii?Q?0jNVKcvxHQy+ppRblmhURRFfi/p1TBZGDsQs/gViIlAYxLUKIp/XBZshFkjK?= =?us-ascii?Q?TTB3+OGenLh4/YHFjG5GyrlgKsrDH807dI2xwuVScSdUh6NEE1GWbfpCHZzI?= =?us-ascii?Q?KpkXTYCXsQCYebinEUqXazv5MLDhG/7mVHDUsvoKqG1o/JOPMZZ3frw5wrdV?= =?us-ascii?Q?rnpULX1130Bp/LHzUjZXoNCKsTO5dOO3zhzoZg2Cq35CZw+HjLBR8Hx6qLP9?= =?us-ascii?Q?VXbxCPxSO5bBdvzgItQeHPoISdTT7cuIM7nlDRhN2Df49lvDW9mXi/KSX0GO?= =?us-ascii?Q?P28aijGLZyrpe/UT07kEE2+a9qKfbMbr+RX/Acx3Fkz20hTG+2Popq+HKlGh?= =?us-ascii?Q?y51cPFoOssmVSqzpgkB0Mnl6VgW8v9U91V6TXihBaSxpXfULvBm6Kp3zuCjm?= =?us-ascii?Q?J601Koz8gRwhuHlPifSut9W86zVlUM5YK5V1pJFPVFmW7XNSs7Vlbyy171i3?= =?us-ascii?Q?dfelKlYM4BJB99hAgmJaPcb8TpXIY3289BaxBvXJtmFYF8elR6T9aGG8XGC7?= =?us-ascii?Q?Yhl72nS+tbEX5Cfy6cvTgPWsdvLSnh4i+UcvrUfdV?= X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2860; 6:YXMS+/oXtITzzoUb96Dg2/D6mgdXrWkSfor+FxubjXVfHZDfJL/nzpKp4WWeLF5bY7CZX0u2kgSWWJzPqVjKh0fNisrjWFvqCtpPoxeCPO4wC8XqEUXHj2QoBM1TlVz23T73STbuHc6wxm92mfTnbnSBMuoAP5VrIOGK4W7IZvlJ1Im0ibWRM9S4R1ZxlJqV714rWRtX5TJRhZ6DVCRwf+LoYkXojwe9Eyz2cv9m+rkCryX4UUEUBMtpRcYSgNcQ9G+KZPRETtS7nCS/tpQ27o7To+J7WPDUtzOx0Fqa1ZZg4DPdX02/CYEhYCzTiyS8IyZL8Vurjv6vWPrj/eOXQYJmUF5pMiKaINL4wh0GKVk=; 5:mfHwFEy6Fqokq9JXZNKg8Yek3OPWs+arF5hchtSPgHS6dw+wUNnt0+Ii5aNpyEWue5m88wKVatrn2jnwwCJwhvb4jhR5Ktw3Wi4/CrwYSLnPMHR8n+iFc3/M4jp44vSO0vhUiCJ7jcGJvbShVk/32EMmCpvaAfyz/JmAj/kAxkA=; 24:Tm0qlUit/27jc7bkDj2dXzLJXqp3EJcoGE/3cbIEpIo1X7he/793NbCE3T/Jrc2LGFOKSmVQC6S/w4rY9ZRSBO6M9xbkhsrzKezm5A4llFM=; 7:/AAmufVfNW5EMPoLnmfigyUQFSdW1zKVLgFBZH16wcc24Ni6aQuyTznK3S275dA2Kz+s7KAy5Q4GoqsCh+NvYrAjpL79G+pAU0bzjwf/Uwwvme/vragMyubcjQxI70NvIiFCWK0Qfm5kMf7VL1G4TJpsyBVEoR2yMCfVDFLU9e5OOhj4QNgm0Hdd48+tLje5VQGIwvcKEou8RkWgsSayEPc+WaiwfPP9aWg5Q9p/fZHifI54LIIuXX2BPVhZDjtm SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2018 23:42:01.0665 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c0f07ef0-2a3e-485e-3177-08d56e8462aa X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2860 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The rxe driver structure layouts have implicit padding which differs depending on 32 bit or 64 bit mode, meaning rxe does not work if a 32 bit userspace is used on a 64 bit kernel. They do work if the kernel and user space are the same bit width. Since this is an ABI break change the ABI version. Unfortunately, the userspace driver does not handle the ABI version properly, so this will not stop any broken user spaces, but it does let us fix userspace to work properly in future. Signed-off-by: Jason Gunthorpe --- drivers/infiniband/sw/rxe/rxe.h | 6 +++++- include/uapi/rdma/rdma_user_rxe.h | 6 +++++- 2 files changed, 10 insertions(+), 2 deletions(-) This is marked RFC to pause and give some thought to the best solution, here is my first suggestion. If we go this way, the userspace patch would be something like: static const struct verbs_device_ops rxe_dev_ops = { .name = "rxe", - .match_min_abi_version = 0, - .match_max_abi_version = INT_MAX, + .match_min_abi_version = (sizeof(void *) == 8?1:2), + .match_max_abi_version = 2, Then 32 builds of the kernel and rdma-core would demand to both be updated to work correctly. I can't see any easy solution that lets existing 32 user/32 kernel users survive unchanged while still allowing compat for 32 user/64 kernel mode. The few places I know of that are likely to be 32 bit, like ARM cores, already generally don't work with rxe because rxe has broken use of the cache APIs. So I don't actually think there is a 32 bit user of rxe.. diff --git a/drivers/infiniband/sw/rxe/rxe.h b/drivers/infiniband/sw/rxe/rxe.h index 7d232611303f40..d6cce6b6d88034 100644 --- a/drivers/infiniband/sw/rxe/rxe.h +++ b/drivers/infiniband/sw/rxe/rxe.h @@ -59,7 +59,11 @@ #include "rxe_verbs.h" #include "rxe_loc.h" -#define RXE_UVERBS_ABI_VERSION (1) +/* + * Version 1 and Version 2 are identical on 64 bit machines, but on 32 bit + * machines Version 2 has a different struct layout. + */ +#define RXE_UVERBS_ABI_VERSION (sizeof(void *) == 8?1:2) #define IB_PHYS_STATE_LINK_UP (5) #define IB_PHYS_STATE_LINK_DOWN (3) diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index e3e6852b58eb45..b501deb18d8184 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -58,6 +58,8 @@ struct rxe_global_route { struct rxe_av { __u8 port_num; __u8 network_type; + __u16 reserved1; + __u32 reserved2; struct rxe_global_route grh; union { struct sockaddr_in _sockaddr_in; @@ -84,6 +86,7 @@ struct rxe_send_wr { __u64 compare_add; __u64 swap; __u32 rkey; + __u32 reserved; } atomic; struct { __u32 remote_qpn; @@ -93,7 +96,7 @@ struct rxe_send_wr { struct { struct ib_mr *mr; __u32 key; - int access; + __u32 access; } reg; } wr; }; @@ -116,6 +119,7 @@ struct rxe_dma_info { __u32 cur_sge; __u32 num_sge; __u32 sge_offset; + __u32 reserved; union { __u8 inline_data[0]; struct rxe_sge sge[0];