From patchwork Thu Jun 23 15:09:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12892845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8A44C43334 for ; Thu, 23 Jun 2022 15:09:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231229AbiFWPJh (ORCPT ); Thu, 23 Jun 2022 11:09:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230480AbiFWPJg (ORCPT ); Thu, 23 Jun 2022 11:09:36 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C00063336D for ; Thu, 23 Jun 2022 08:09:35 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25N0wZAZ008889 for ; Thu, 23 Jun 2022 08:09:35 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=message-id : date : to : from : subject : content-type : content-transfer-encoding : mime-version; s=facebook; bh=mzXJZL/rHy91qVVeOqSpti15aJOAvxebufhUzLQQh7E=; b=lFJXVEoh2ajvjXt/3AsAZzUCRJ0/vVLRaDf6wwcaIaBJFq04ZY9Q4YAWr9ZD20/wF6/B YmqqceCJ+8IqxvCRxqf2UPAtuQDs8Lo+INNvUpSijlFmruzQtK1Sw0xIyMStU6LyLD4H aAzQGaGYAGmkHOMiyB5+7B/0kyVIOgUBRKw= Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2170.outbound.protection.outlook.com [104.47.55.170]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gv4qg7pwx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 23 Jun 2022 08:09:35 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L1Cg5LsMzX7cDelweSXZcuIHB/MBVcMhw7tw9ErzoqlMhXDQXlHm0f7TIJb4RAYd9KjqBNxthnCrLO5EgV5N59icT8GuQzAc31VpvVKYUvWCKW6ug/hEGf8CdfEgw+/NOCGOGHqIioSodKKj27pVfzUWGibtXRUdOJbhj8ZQDY+9T7cQaEJ1/yKgNHmp7ACbVzoRkh8TI60ROMnfPI3S58j4/apMrKpBJV+MSfhZw22IfzHaedWyxor5eUhOlElrL+OpjMH8sJH5TiUwen1jokNyh9jpHsqi4mZMhQnJnT43Nc3poiVxL1LChDDqC2Bzj7dnqkB/gNaIgKwQGfa/7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mzXJZL/rHy91qVVeOqSpti15aJOAvxebufhUzLQQh7E=; b=jp5nHRUZz+h0bdFQKVtkfUH7EuABMYgu1bmW/VWx66m9zsz0g/UhTuQQpefGTK50Wh6rifkfK9xPPuwSbLI8zp1a9OaePG1eoOlHbvvSBOtzUFZhkhvctNZ3B6uTn6jUCpA602WMGdxwEZzHLRqHwKJw72fHtvtLScR+GAudwtRMha3YYBYR3syzAXPY4H659GKhlfzY28WP1XjuAd6wzMpqwYX6dNRB72O3KrsNr8UmLtx5Zcp1OnqHnUcfGoZ6ghvAHeVf3AFnLQuL7xjhFCB9UOrKRzyYTujI3w0+87JFB7/Y4kANchbUiO9fmoi/LfFHfx7jCuKYeVcqbUjFgw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fb.com; dmarc=pass action=none header.from=fb.com; dkim=pass header.d=fb.com; arc=none Received: from BYAPR15MB2566.namprd15.prod.outlook.com (2603:10b6:a03:150::24) by MN2PR15MB2990.namprd15.prod.outlook.com (2603:10b6:208:f2::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5373.15; Thu, 23 Jun 2022 15:09:32 +0000 Received: from BYAPR15MB2566.namprd15.prod.outlook.com ([fe80::f900:19ea:3d56:9231]) by BYAPR15MB2566.namprd15.prod.outlook.com ([fe80::f900:19ea:3d56:9231%6]) with mapi id 15.20.5373.016; Thu, 23 Jun 2022 15:09:31 +0000 Message-ID: Date: Thu, 23 Jun 2022 09:09:28 -0600 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:91.0) Gecko/20100101 Thunderbird/91.10.0 Content-Language: en-US To: io-uring From: Jens Axboe Subject: [PATCH for-next] io_uring: move POLLFREE handling to separate function X-ClientProxiedBy: CY5PR14CA0020.namprd14.prod.outlook.com (2603:10b6:930:2::9) To BYAPR15MB2566.namprd15.prod.outlook.com (2603:10b6:a03:150::24) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 12da6c16-f0ed-463c-51c5-08da552a5fa6 X-MS-TrafficTypeDiagnostic: MN2PR15MB2990:EE_ X-Microsoft-Antispam-PRVS: X-FB-Source: Internal X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5OQw9uqIPGWyci3XNZcYbu7i18t0ArqoG9u3jOoM79Va1W3oYlcdJhaoIs9cU0Ov3DLqlDsM/X3ZJRS70KHU3sMvUtBhzQFwH3SEBqn+hTk9fghQZsIh4wcAXLvabtSZEOSgvWGzaeTWEPgkVXXbnCnIr12kME8UTSz1BLi87HOeIaI/3W6x+jqXMmMGonpC0djTrtmXrzU9UcM8CNeOvCgWcxfihDefwc5Gc7OH9TbgYCNhMd57qIVkK4ViUXY8TbVLolQevhjk4HeeThvhKfN2Be/bK24IdOZoFyOflwYzSnh91uPkfNI2iu05srmFVWr1kTjNZT6iuStp5fQk1+WeTXnwhgtZonb+PF2Mmx5VLuKzm9JXP0b7dWoB8VRRLO5nObT7JnLyIKHOSLvbEVJ3kNRILHCF/DM2XGqcR38OH2czZtu4os/wuoDfO8IKYhSCzaJc5Mjhrr8NHiCYBcbynNE2BWsmOwl+j93r1eES+VdohMyACQBXipaMV6es1hGXoAOENRTLhFUwl2PsS1oAe+t4Dztyv4cxf8bndRAL9zeOURSmzqCKsWOdYlqa+dGF9SD72plrOy2mIrcqDOSabY1t4Ryadvmwx28RX2JvAP2e7g5qT5y4CnLyhJLV/y5xTRsE+hTbbg0pDYGmWAe1I9Gxg0Xq7ZZIMibxUT3UvbXFnqFgZhZC9497HZvadXducW0mIwPa76bVBCzthTtA/C38q6alLDL/vTzLv8QNcljZwUbb9zKQKOGq113MKu1efzIhbQTGeZdPAMQvq7tiigV37UlyloWxLLJ0LT4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR15MB2566.namprd15.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(346002)(136003)(39860400002)(396003)(376002)(366004)(6506007)(38100700002)(41300700001)(6666004)(26005)(186003)(31696002)(6512007)(2616005)(83380400001)(8936002)(8676002)(31686004)(36756003)(66476007)(2906002)(6916009)(316002)(66556008)(86362001)(5660300002)(478600001)(6486002)(66946007)(45980500001)(43740500002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?tHj0ZG/4N+ORR5yuNk6hw/Z5H+mS?= =?utf-8?q?WrY6yZOxuN/X4EGrznRnJ3lzGVgF1TlpxtGi9QS5aXOC1lkUn7e81pl1AuEru6Zz+?= =?utf-8?q?cCbiaMNcBxktGvx++711CMBr+tR4cvKHRZrojdL9CF0EZfxHvvPFZNsTBBOv9x+gH?= =?utf-8?q?9nmNd/H1pZjpGcQmdEamTIBaS/9R2q67yEVMp3dgfh+fW6VAYbX+9pllfpmMtgh5q?= =?utf-8?q?ZmVOS+svCKgZlWoLKd6svTnWU6WxW7FIExfD7/vyGBYILtGj6d8bkmeJwgASRVaMP?= =?utf-8?q?xNC4clx/3+ak+tgR/ITNI+sLkWgakHBUtCdVMO9FfHhruep/wHqvQMp/AiIQMgCHS?= =?utf-8?q?Rj8uNQvkP3TizVpkTVYJy3FM8QZNbNXnLwPc499onlbOs+i5nLfWmQnnB2rD9RK5T?= =?utf-8?q?MufutKbD6DnfLNUhJBcgCGoIafxN0LT9TGwzjnSQNOqvixOSrQEEbD5i5gf2OU5qx?= =?utf-8?q?GcdJTnSiEuFxlrNjNQISV5UXPjZBoRRIicoielaVQHrZo90dk1beZa1v2saIX2zyM?= =?utf-8?q?mbiFaZqc78T7T9lJqcOVzImpyYVQTDtgCwFmvKMvWlceezRcQLQ6eHR1tky2JhfUL?= =?utf-8?q?luRSS7Hm/jErSI+12xB9/6z8RWRf+ZAN2G5Q1vAVFK/KJTljBw2GNbXxB4n/9fa9k?= =?utf-8?q?ZqXH1tQmhrqgDj7+LIFJmAKkqvKqmCpPg2QEsRwizqE5JInBGUatjpc6TC8q0mfgZ?= =?utf-8?q?sL6+szaOmAaT7VAFLZ6kBnCP77xq8++RcwBke9LNJIepF6Obyb0nI35DBYahe3irg?= =?utf-8?q?IB/KCZbbXdeHAFL9KKqcjua8I2McOdC4kmm+x/gMwkrAmB/gaL1i8gMhiyvZ6spTR?= =?utf-8?q?xt+xqV0GXRzet1uDGr6KEuV52sFJxDyYxChc9zZSMRWw/F8A18It7hNL9dDa7KPG9?= =?utf-8?q?d0wMIUbmaCerz7+gFA6YC3HYue8+RyiFA8L+80vVV/ctTgFwfDtnPZ8B//OdmzUQD?= =?utf-8?q?IpeT6p8ggxeELx5sKqVNmzrV0bT780MrmkEUWomo7eLxshBcNQI9SnAH1HWR9dR4X?= =?utf-8?q?Yovq49ZIgZtY83MEmU9F5paLk53iso4zSi6KRUzF4DYncBQYH89Lz7KFg1hh1Hmar?= =?utf-8?q?TBOZ3yYbKhXafeOmzXPaTI3JT8RZngWiaq/quD55/dQfztKCJS01Y0LSF1kevI4G0?= =?utf-8?q?8km4Drxa4FuyG27QffNHkEq8veT+ma6Xm+jvPcJ9Kw9mGwsNiPqfUIODCgkZ6BNey?= =?utf-8?q?zzDmVHbsXo2I1Qtp/0hMgcQ8bw/CDPjJyO9UFRBVGzI2IaY5O4t0kYp57xgd+vgHL?= =?utf-8?q?2rtqipkdE/mwS5EiyAVwsAoUeUL7uFlwFS6pBh5y0yz0QCkaKR7rxF7VOqEAnyH5j?= =?utf-8?q?1qc+3KmlXB9nLaspUzOg5VubJ34S0Hkp4eykIItTd12zre5ycONzkLrOAPigICTU8?= =?utf-8?q?N+YOZKvFWf3q1rfOPHKZYlUNcgSWVfXEZys9yWjdX1Ltbtjj/3aCVHEqcLYYUrA+G?= =?utf-8?q?rHs6DUI0vAVZTBrt7DgA82xeDRwRSa8NobUJn9iT1hkMvE2yqHgL909BIK8gcJYdA?= =?utf-8?q?BUUahxcjOkqECpY02Fvtb8NBRv3YQZQXw5+tHT8B3FHGaU1e6zrS+1aRERlpZ1GKO?= =?utf-8?q?9/66P7xtHohncs7Zz1RTTZBFleTHTHIfQVyRkTR59/cbBxKsPc1DX8304N4EDKiic?= =?utf-8?q?pMS7VDL2GtxxrMb4i6roEUtjBRDt2xEQ=3D=3D?= X-OriginatorOrg: fb.com X-MS-Exchange-CrossTenant-Network-Message-Id: 12da6c16-f0ed-463c-51c5-08da552a5fa6 X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB2566.namprd15.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jun 2022 15:09:31.6506 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JnV669ctjUK1dZWPBg/jNCOwInlvUw6kJjF3ehEnpF4QnHQcn4mLjnamdXXNutZU X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR15MB2990 X-Proofpoint-ORIG-GUID: plPVaoF2Wu2NAAFK9d92WJmP8AvGnZHx X-Proofpoint-GUID: plPVaoF2Wu2NAAFK9d92WJmP8AvGnZHx X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-06-23_06,2022-06-23_01,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We really don't care about this at all in terms of performance. Outside of having it already be marked unlikely(), shove it into a separate __cold function. Signed-off-by: Jens Axboe diff --git a/io_uring/poll.c b/io_uring/poll.c index eba767594dee..fa25b88a7b93 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -325,6 +325,31 @@ static void io_poll_cancel_req(struct io_kiocb *req) #define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI) +static __cold int io_pollfree_wake(struct io_kiocb *req, struct io_poll *poll) +{ + io_poll_mark_cancelled(req); + /* we have to kick tw in case it's not already */ + io_poll_execute(req, 0); + + /* + * If the waitqueue is being freed early but someone is already + * holds ownership over it, we have to tear down the request as + * best we can. That means immediately removing the request from + * its waitqueue and preventing all further accesses to the + * waitqueue via the request. + */ + list_del_init(&poll->wait.entry); + + /* + * Careful: this *must* be the last step, since as soon + * as req->head is NULL'ed out, the request can be + * completed and freed, since aio_poll_complete_work() + * will no longer need to take the waitqueue lock. + */ + smp_store_release(&poll->head, NULL); + return 1; +} + static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, void *key) { @@ -332,29 +357,8 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, struct io_poll *poll = container_of(wait, struct io_poll, wait); __poll_t mask = key_to_poll(key); - if (unlikely(mask & POLLFREE)) { - io_poll_mark_cancelled(req); - /* we have to kick tw in case it's not already */ - io_poll_execute(req, 0); - - /* - * If the waitqueue is being freed early but someone is already - * holds ownership over it, we have to tear down the request as - * best we can. That means immediately removing the request from - * its waitqueue and preventing all further accesses to the - * waitqueue via the request. - */ - list_del_init(&poll->wait.entry); - - /* - * Careful: this *must* be the last step, since as soon - * as req->head is NULL'ed out, the request can be - * completed and freed, since aio_poll_complete_work() - * will no longer need to take the waitqueue lock. - */ - smp_store_release(&poll->head, NULL); - return 1; - } + if (unlikely(mask & POLLFREE)) + return io_pollfree_wake(req, poll); /* for instances that support it check for an event match first */ if (mask && !(mask & (poll->events & ~IO_ASYNC_POLL_COMMON)))