From patchwork Mon Jun 8 11:40:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: SeongJae Park X-Patchwork-Id: 11593195 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 52E67739 for ; Mon, 8 Jun 2020 11:49:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B851D2074B for ; Mon, 8 Jun 2020 11:49:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="P8Ug86t1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B851D2074B Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amazon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8BE686B0006; Mon, 8 Jun 2020 07:49:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 86F156B0007; Mon, 8 Jun 2020 07:49:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 675536B0008; Mon, 8 Jun 2020 07:49:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id C20696B0006 for ; Mon, 8 Jun 2020 07:49:31 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 7965B801295D for ; Mon, 8 Jun 2020 11:49:31 +0000 (UTC) X-FDA: 76905874542.28.fact42_15029bc26db9 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 4FB983C670 for ; Mon, 8 Jun 2020 11:49:31 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,prvs=421c5bd8b=sjpark@amazon.com,,RULES_HIT:30003:30004:30005:30007:30012:30017:30034:30044:30045:30046:30051:30054:30055:30056:30064:30070:30074:30075:30080:30090:30091,0,RBL:207.171.184.29:@amazon.com:.lbl8.mailshell.net-66.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:1:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: fact42_15029bc26db9 X-Filterd-Recvd-Size: 174491 Received: from smtp-fw-9102.amazon.com (smtp-fw-9102.amazon.com [207.171.184.29]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Mon, 8 Jun 2020 11:49:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1591616971; x=1623152971; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=M2mHQXBhWz/OZ/fKBNU7BYTL4rzfZCY6oyNpr/06tAw=; b=P8Ug86t1uagudxky2zt/u6WablaQV8fArlVeybxjx6TuVGlWVR/jaSbs Pq2wS8M6F2M7g8xyQOry3un8F+RhIKEPt0UcbooNJWbl45PjUhmBUbz3+ pe12NNFINk1c80iidU0PBWHb0BQBnBWe+3v/tHGBrx3FGxiiaWIvgB0Hd 0=; IronPort-SDR: rXukmPwVB5d+NKqoZvv+M+R1jgIWuPbdQMEB1IIAibqDbCmLgHjf50uoJcb9WsMDAzddyNIiBI EhV6uzEEIZTQ== X-IronPort-AV: E=Sophos;i="5.73,487,1583193600"; d="scan'208";a="50571682" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1d-98acfc19.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 08 Jun 2020 11:49:25 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1d-98acfc19.us-east-1.amazon.com (Postfix) with ESMTPS id 33144A1FEE; Mon, 8 Jun 2020 11:49:13 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 8 Jun 2020 11:49:13 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.162.53) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 8 Jun 2020 11:48:55 +0000 From: SeongJae Park To: CC: SeongJae Park , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH v15 11/14] Documentation/admin-guide/mm: Add a document for DAMON Date: Mon, 8 Jun 2020 13:40:44 +0200 Message-ID: <20200608114047.26589-12-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200608114047.26589-1-sjpark@amazon.com> References: <20200608114047.26589-1-sjpark@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.53] X-ClientProxiedBy: EX13D46UWB004.ant.amazon.com (10.43.161.204) To EX13D31EUA001.ant.amazon.com (10.43.165.15) X-Rspamd-Queue-Id: 4FB983C670 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: SeongJae Park This commit adds a simple document for DAMON under `Documentation/admin-guide/mm`. Signed-off-by: SeongJae Park --- Documentation/admin-guide/mm/damon/api.rst | 20 ++ .../admin-guide/mm/damon/damon_heatmap.png | Bin 0 -> 8366 bytes .../admin-guide/mm/damon/damon_wss_change.png | Bin 0 -> 7211 bytes .../admin-guide/mm/damon/damon_wss_dist.png | Bin 0 -> 6173 bytes Documentation/admin-guide/mm/damon/eval.rst | 215 ++++++++++++ Documentation/admin-guide/mm/damon/faq.rst | 46 +++ .../admin-guide/mm/damon/freqmine_heatmap.png | Bin 0 -> 8687 bytes .../admin-guide/mm/damon/freqmine_wss_sz.png | Bin 0 -> 4986 bytes .../mm/damon/freqmine_wss_time.png | Bin 0 -> 6283 bytes Documentation/admin-guide/mm/damon/guide.rst | 196 +++++++++++ Documentation/admin-guide/mm/damon/index.rst | 36 +++ .../admin-guide/mm/damon/mechanisms.rst | 111 +++++++ Documentation/admin-guide/mm/damon/plans.rst | 49 +++ Documentation/admin-guide/mm/damon/start.rst | 119 +++++++ .../mm/damon/streamcluster_heatmap.png | Bin 0 -> 37916 bytes .../mm/damon/streamcluster_wss_sz.png | Bin 0 -> 5522 bytes .../mm/damon/streamcluster_wss_time.png | Bin 0 -> 6322 bytes Documentation/admin-guide/mm/damon/usage.rst | 305 ++++++++++++++++++ Documentation/admin-guide/mm/index.rst | 1 + 19 files changed, 1098 insertions(+) create mode 100644 Documentation/admin-guide/mm/damon/api.rst create mode 100644 Documentation/admin-guide/mm/damon/damon_heatmap.png create mode 100644 Documentation/admin-guide/mm/damon/damon_wss_change.png create mode 100644 Documentation/admin-guide/mm/damon/damon_wss_dist.png create mode 100644 Documentation/admin-guide/mm/damon/eval.rst create mode 100644 Documentation/admin-guide/mm/damon/faq.rst create mode 100644 Documentation/admin-guide/mm/damon/freqmine_heatmap.png create mode 100644 Documentation/admin-guide/mm/damon/freqmine_wss_sz.png create mode 100644 Documentation/admin-guide/mm/damon/freqmine_wss_time.png create mode 100644 Documentation/admin-guide/mm/damon/guide.rst create mode 100644 Documentation/admin-guide/mm/damon/index.rst create mode 100644 Documentation/admin-guide/mm/damon/mechanisms.rst create mode 100644 Documentation/admin-guide/mm/damon/plans.rst create mode 100644 Documentation/admin-guide/mm/damon/start.rst create mode 100644 Documentation/admin-guide/mm/damon/streamcluster_heatmap.png create mode 100644 Documentation/admin-guide/mm/damon/streamcluster_wss_sz.png create mode 100644 Documentation/admin-guide/mm/damon/streamcluster_wss_time.png create mode 100644 Documentation/admin-guide/mm/damon/usage.rst diff --git a/Documentation/admin-guide/mm/damon/api.rst b/Documentation/admin-guide/mm/damon/api.rst new file mode 100644 index 000000000000..649409828eab --- /dev/null +++ b/Documentation/admin-guide/mm/damon/api.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============= +API Reference +============= + +Kernel space programs can use every feature of DAMON using below APIs. All you +need to do is including ``damon.h``, which is located in ``include/linux/`` of +the source tree. + +Structures +========== + +.. kernel-doc:: include/linux/damon.h + + +Functions +========= + +.. kernel-doc:: mm/damon.c diff --git a/Documentation/admin-guide/mm/damon/damon_heatmap.png b/Documentation/admin-guide/mm/damon/damon_heatmap.png new file mode 100644 index 0000000000000000000000000000000000000000..ff485f9f59ca2717f5221b2824eaaed616b22c0e GIT binary patch literal 8366 zcmZu$2|QHa+rRfV!ysEip^VCs>|`m+$Wmj^E^B2;mZWTDxsghevSl~P8nW-(s3>cp zvW01}Wt+&p%zH<_|Nr;CZ_Q`SxaXYbe4p)n&+`ywpm&;)o{Js;07m>7bwdDvU;qF~ zm|e&S#4Pd&0CoWeI>s6l3I!P<5D3&SA^^Y?B0vG==7>1p?OQ0}wR89{CJ96n;o7+-&DLeE)tAL=^z&yxTPm zP*tP=UEP$u(Uc$qG7M8RfOXh&D((h({6T%COVu5QrKQgV7)!|n=7Q&9fZk!k@FN49 ziHkVA7l8Nfg>lFmhV6*39R;T9kE0Nfw;hGxK_L<;h&>cyXJsmxLMBvnHc^^7iADYt z9D#yuXlOvJ+Y8Vi0QMdR4hjK6qJWeXpePGyDggRQfCUb?pa!^T0@v|CfDUj+4|sSM zcwz*knF3knz>9NWxiwg43%)uJwq69^Ie;IXz(Hql)CK(R22EapX0AeHPiXNvwCoM7 z`tI8B+qHRf*H*x;?LZU-Mp15|C?V+WQ1sRv^u|4!^)Q;1aN4B@vyV^gNCE*;AIJbk^u+toJka1ZT1NW$*RK zVRy>quzt>IoWD=2V87}MF1bQ((ITG1#RvA69%L@#MVDiNibFu=P%Qw|2m*CN zK)o=~a13Y^1&CrmlLYWm5_lyIygm*z%K|O(z?&1mTScH%8E8`h+EsxLHK0=i=sFF& z!vpVifNnjY#{lRx1U?u8AI*SI7C@gB&~F0_*a3qVfgwlW^Ce)|1sHJ$Mm&JgYXHd` z_~HkA^#{HM0pEjxu~1AvFaRImvp45&Ywvi7vXOo`F^83y<0oM?h>G;A(blNnD<8NV~&G^ zQ&EYIMsod|{bu-v(QI`s73`TDg{Z6B_I?Apfk$8Y#n74CL?qEp)j87~#J?^H)P+V+a(jvkjyXJAUa zeRPZ|Xa#!XadpBFjwYy=HM7-Xccsk3=-q9j!>cupmYX93F@2>TR zOb4Gt{&pV*?i!N7zI{@?l}2us0%M&W&h52FaxFD11k(Gr%4qp?JGu=b#imxZtoD#8 zu9tZ`82Dl=&h%MJh;PWX8diE~TzjJbDxMY3QyKVRo+u`btW=DKg9So5`?l7@HdN0% zP8nD_+1axy;3b$}b~ntgQjvA|_J3A}!_Fh($D}{mchM$g4kOb$|ZUTr`iMs_`LnEoi$?mt{!oo)(G$g|fK?6OZwOls5wR+&Ib$I~Yb{ooRRj^DBE+CJ<&v-!)-16Y4n*i*grM?2 z5E}dcKuB}@13~{!gwy{I0Wmz}Ps2z4Fg(cV{~G>}4V8Z)aQ}%={s)5oAMWsV{Rd!A zBJ3=JDr}Hx(8}NQl(_Hwmj zl%J~b*O@*;=a@57*&V`ktIdLGn@cOpS|{kiTrleDRO_DNh3|@OD3SVCorY~%FHQE4 z=eoN;GA~hAgsFf3rZab*`K(iy?zr=(cTwgK!RY*NV{C6+L2d~X>mv9H&)mq#l_uw=<4}XY|Mh$o`=I$rk_TmS5?30O z&U@V|`w^Y*)N-ocO@9wx58pZ`j=Sq?BVT?dHL8fZ36NaG111*6$Gf@^4=p9FmQ@1=W(uodD;9XE3*T&xez|O+9G`-c06gFBF!nL=$2}` zYJBshGNd)wky>MCVgFWr`~F9?oNBCf9V{N`IB5D)m=tLfP!;6i>FMw5;qT#}!Fgga z`dePF;W*S#7gXfq>*?v~>yd7hnrzSOA>;MrFef>7xW0bt*;akM(w6;7(?}{)(zxf% zm#1^83*58ojm*b}A6zJT)sD57nvcGBLHRw`Z5dT|i#qtq$-U%jN#cz|w#4bvr|&AE zKPxH!I7V998bCh~7A2+Xx;ItKQ^+8x07(imG{CQWe%-dCgj?RJ5DSX46S^O+wQUjU?~G2wui zY>78_WQSOam{5wlp3kJC6cbg6L*Mq04`3!~p3l5QD=JtLD_i(UN~6$7`7M}*1?plz zL{H&t4IKDLWVC-D4f24I7e}ys<6t>&J_2`62@6>SoS)y|AiHs6Fd!ePZpZQ@M0$O% z2QyWm>N+^zkHaCe6QRNkARj>-k@8rD8XW$kXg)0%vU?9PHO!+&u??FnQ*CYuV%xb3IYN>?)}b)FsTr})jmM`cw|Y_GG^Z5^ ziD?sKB+uVC-?wrgYe(=|N95wh3!|MNZ01Hx4@+nAIN1?)=9)2QxexuJ+!BUW zimwXWy^M;En{KRi4t$!W==DpKe{;($_Q*|x zZ$q?-cVK=^VM2_~9tbjp;da(X+8j;~3K{Oe4jcj`+kq-2p7yc3Pj*x_7xWHN-b2l0 zmZFFDBHjTagj+~D+!ZH9y72^Q$7wZvzu*J*n%_(OxZ~lZ(8sP%Z9|J>49nS+c6=uL zG%9{dR$FvtQ76phz)6#eT3BxM4jhQu)aHI;rb#Mx-;NpKw)za4uQWP$Iz1Ob?N-Aj z1bA`x(suYA$=!bBPg}HfPsDFs_o`Z4bFEcTYoKkL`j&{G76w#fkzl_nM$kEaS@YZF zm>X^5P2?lBEh2*Ro-HO(^D9Rm_vr7ZZm5dudghuWe)%&YOG~l!g=G8X^9we~dQx#8 zvl4z=(=O}XSA_#(GcM@NzI6kuou?O~!={E^-?x{h_jDJqNce&Y4<^!-9v|vXxqK*V zzppUqi1xMP?VIWXH~Oq>TSWdT9&K@~i!uo-20i|jqA9W;#L8ifxcgUABd<*j=g%G! zpp2*8y8}0%HW`v+!7%$SeB$Ty+&8;E9m)P(P9`b80>D>n64M*&hf5>1YP^~ zEvJIHRgjsxLIexbN+XMgwpvC_kHG~cyv(V$>Mceq{4S7T+zD&L8jt&UiSLhc4raHVBC?>2Z}bX|D;^T)U8Ci5 zGPHN{-qI_TX~3VoL9H!O2$$T!+AVwSGjhyt{SngHEZy4mK~s10EirrlkoB>9j^8Frz!b;a7VGKRX6cOGMnSCXW@gnfu~e;57?f&Rj< z8o*s?rupawO5}BLWNYP9>fxAVKq7VfCHFMaJ$L@4}MJ9 z+Rp719ueOMy?6i%zf>eJqO9WJcv@F`FZ5MJgSKT& zidfWcGaNDjflv6c;~+OOTsT-YqGZ%|m4;?VZw{;YV=g*+20myi3lmQy_omLiCAyn4f@T`HtV!BM*aS<8`hC8C(61ig&V~d;@D^>HH%dDM z5ava==ZNv{zHL<;re7=ZMB1G@@H1BOkgPi`UsvJJq=}0#^?)wY5qWmB@bVSOsJ6z? z@KK4%jk~f}0xc#ThQ*?O?f&oOfJNmS)R=JAkm&P{k>Vmwf|;6_bGuzw7D$YkDtbtB z{7%9z>|K6RlPIR@5aeBbe(BKfm{d7`TTXg3<5G3q{O(0nVl1l!hoZ?*&Q~Wy*>)US z`{i=JmE#^&VmGGh{I~V84&%yO?xpwsi%z5uNkLA*2+FF`qABbWriu&FjKAl;!=c4G z>b5r-AwBJC39!#qf|U_`#qJ@(*4<}Q&kSZuoCDuTu?Eb>-emEhXJ8|vxpmalh>y@m zKDW*hj(C71_lX_ z!1cakd)Ot%MDEfd^o872X_Z$6vEjM17uzhDvMu%_I_F~(9vwxw0+5&n@yuDljt{{q zvJs&<)?VS%q$Z#l)i@L}pPq4@8&b zNmF9lmPayVXjb<&vUlrBBJYF=4q?{w7@{~LYiJT&#o%1ILWZjF>7T#LjQ-@qI>;#dX{eaizetFV_cdjxiM>th$5s{8aL2C zARjHhm7&{~oYb*cyW2AJY3&FO#|&++MNV*#Es{7uqpV8`r+JfOCXhFs-m(B`_AFh0cq^gOBj3n5r{o&l>Q+SO zPJvy}8Bq{XzEm)~RIrq?l-=mPz1cUJX0flUJ2-Hog<;y#tfI0)?)s?|cbAJd!(UXC zmb%5+est-%Y?~P~(dEEB2&`*_rRW z>rh-oou>ewAN| z!8Cli)r!FnYJ+sd?4+s_Pi6NQXkZB!L2>29OBZ``Ax$Mq;+c3!oZ^JcZl^+ZuPTK=j^Bs6xf?3KGm4|sOFjbd8O26ZNYcipbb!%<;?I|J- z#q;ZUIv7}s8N_H#|SUUSFCPEF&v~N}?)aqdF??NshlPdc)IljqYKCD~z82vtt_L?H7 z8Kl8%`7f5vJ`RP$ zImmgkOjk^Ek^`!>{m&rYSViyPC=g!p)P2W*nzx87-`x8PZ-zKs9dQL7ezUtIjhVw?{Y=y9%R zKWbAci3MF}!bTnrgKKZH6z#{zGw;1?1kRhF{TEx*i6?9S8KY)kz(3j7i*#BCkmp+J zGs}BU#Iw}N`vV0LJet+4AjSJEcf0$3N>sABUXgpHk)tO`<=r+ zZY@U%2C=q?$0ShT#r3!R!%lH{m9V5f+ju(> zKSwP}ujfiw3}t(Q4)y+swfll$^{-y;cRVN3_bnm~GSv@Fn0PPWhPR$hhbZMw87io$ z%SV67G{-$#Q15)KkN%gk>HyVh!SsGMw|%$G5gsD3>@?tPvgK~iJ5H_t2o|jVr}^F| zJ$G;V*(h{h%?FI+BQ)bs#yVNsG<@0B+@l$Dw_}$j@V)EmNRIF1aN|WImzkcb&Dg$@ zqn`R57;|T~()scD=K<_ycTIy~K8(EH|4B5Y$Nc|^h7>#K-v@RS={5h7SV{jzT!^8{ zlATOL=er?*az#U3yM7Dm_y4ggwa{h!Yew=`{uM>??~9RM{Vyt2F``^)k-Efv=S1%4 z|L>_e{`Z|K{<;(2Ul%uC-USX(8*lC%;`mSQhy3e_|CRf^e?O1lUw5AW(|T$x|4%gS z{|X!Mci6wX#^$fEjhBo#A2A558qOv>&Vmeiuyz;-Y5rasPH-({6~n*N`cOB+{*x;? zauMi^9qwJdr)z@KaAV+|dwBjDNIj~+P8rm%Tvfdz0>c))>35FV?Mxzsz8$<17cpz1 zq)(Gwj?SoFtf)edDh4UAmku3Vkm-2OfEmA#%+`(N(Ofg(0d`(n_=JH*V@WpY$ZMNN z4%miG7JjlzNu|YQ4ggKUMN8gNE-$X2g`X>42bfOpB6s{=&Ss z$!OcdSohJ0j6*Nz<^7%}Ht?N@`YCf%FqJQtbaej7het&v@NjPn^3N1yeC(t9;WqrT zep!aU@FyNoGG;=HJ^2VO-yw^J8Cq1PFO&(~=Dtx^(F7NPsY4^`F zQ-Lf#0tBY3XBX@6BAM@eMrJK~Zs!K#mwe%s_K$UVs4La<^_tCe2R`wZZFhpMsX#O`lYsHq^K{VS{(>6wzpeB4aFS z?WlF$b-Z}EkCQ%E!fM_@d@f@vYUYXCprEA&{gM_}toPy#cHfK+d5k1?d4IS2pql#y zsSt^4h7KHiDGAYS{^vxejJ^uB)u$;;lcjVZo_;Q?jfD*LC(tH#!8L&*~Ep zPfoCTU*=wzA*rk0u8gPq5$vRw=kvYCl||}iMXp}^27k&7w46V4Y3i4WRdSHNy4(F8 zRxulS>43Hyf)Ozt4fm%X$4EfSXOdb5=mSbR8}Cm?%M9I`v9|b~qks8)ANtjpxuu|; z^TQeSS1wNveqxFX)#^2VJ>Dj2rE6Wh_QUTgi&-fU$Z9Qe*@lkO3$++somp$w#C8q3 z)1e=KM$1XhK5L2{70#ZL+jOC32AznL5YLA0tMiMDYe{K5u{BUsDHdtWh_kr$BA=LH zc-`+&mXaFa_|=#w-?)z^MN@m?JqM?Kd!J+o)BcHU?>tO^ggKB6agFIDIia){r)hJ9 zf-21uHTC5}2b{~F+MGv=JeCF9pHy|$d<}!2_p2E1p6Cl!aCQTyJbBhmeB_Xj60A;Y zVHU;4Cpm)8w1#9Y?xKtbGMD6+!6xVus|2f6X0wd8_tV@{L}TVhlJqnT=^D61+8AL= z!8BTt8oP&nxFRdEE{06sz(5n&)GhH}ugQl{Xi>Q3^R3}_KGZ?HhMxM1Q+C1s1NaT3 A^#A|> literal 0 HcmV?d00001 diff --git a/Documentation/admin-guide/mm/damon/damon_wss_change.png b/Documentation/admin-guide/mm/damon/damon_wss_change.png new file mode 100644 index 0000000000000000000000000000000000000000..36699547c342cd05c9f416c78bc3fc86b9dc415e GIT binary patch literal 7211 zcmeHMcQ~8xyH6sBhKf-!T7uG|_G&3&SJkKzYDR}mDX|*$+FFzj+N#;sru192N7VRI zwMSKJS8Za|mh(n`<6PG{=fCsUxlXR+eV=hZ&;5-1bFVxwHq>Q0#(fL|fiUS^(lUWS zU;qRH9Y@fBEimSDgJ4C*_=>4Eg+c*aGBPr#FA@X-P)HC8WOG9GQb;5UxE~6sqdbjFA!n3z zG*B8kNF_uHHiLqxtE=PVP-PAv2DHWMikk6s5sXS{m+)z%jIjic@??lV^-OZ?#*JcKJ;CPsYY7|J%jk#O>|jo zTpG*e3@F^7`xDoNOJziq^yiraf!p;-ToV) z`ho*%Au;Z$Z6j{1Qp|y=mn1z`2T-We#)Sczm7FtA0iB@@+0E)69iN7bQzkfoNL#Ihc<=xQlieMwTZurVu~j(>4zhFk-0z6QLGhJ`8tTJ6tIE(7sCH$EVXHU zYJSv0k`v}jx($yN%Pt;_3jbhWXAdB;U?V?hXMg&kHtx!M-8hKW`;9B>MEQHW2|^%cg-|TgY)l zzEM_`czHNHIcZchrVob(^o0Nnc^dZ|Zy3GW{dOA9yIIA)*nd-Ge0_(#>rDdhXO`{6 z+!M8k*)XfonX!0vEStFjv)Y@;t1aHudM# zi~|Cl0{~hqRtbZL9yNn^b1q0%sVMt(sq6>LI5)!Ig#&I)12yUh1za)XdDl^%Oz?~H zNZ7j&Ajd(SB#$EZgY(kC(voSRU$LZX;GpEc{C=9ALE)$@?5UIgFBeJiv_GsQD|it@ zjzjUZpoGQ5o#-_1Vpz@r)R$@kJ69Bf3zhO0wd?X(*qbWGE>00o> zH9IOjZ!AvXHL{MtnF!do@<(!XRobV#;+-kGld|8awK39|3%?SUu;JB3)pSGev`{5o zQUXx--Tvpd*<9)fJ+yb01x@&vpnFJVivh}#7rPWP>DE)16lNk{xZCoSg&cfQHnxVA zQS{M2)n2&-C_l1=nSjCzVFOQt$kg2pE@Q{BN#b}ZL6v2sgvz2MdA_WLtGfPMj)cla z)m74J!FeuSl0s=v-(a)3mtB$*9d(L|sv=w)CX3=#LW?5f?6U@nBA&Ncy1qs4uD2VV z?KuWsfQ3Eb9%Z-N;gCyR4gwn$Ei9>;R*UDR0goJab;KF=jU^a%QB^`RP(QFp*w?EXBzk-{~|ps@TKtP={b55J$z9RtVYsFDTF5Nin1bxG>7iMpUp#| z?amekLiRzt*A37qhQ)VhL-j)d+I~ij@V}Vbs0>zlK%5O$;ZDmuX?WH9!CR z6C;?a#Od*MFEgkW?{#g0@(Ck{OjR@##%ll)d@lITuoMcU{WgUo{LdG435pdj`yaPU zL7K)r*#ZFizTJy|ID8GwPl9 zwcKW`8rQ(!8BXNfl)tUy0CJVKu^Lwcv~ucWc7nD=dO<`qm0?zp_v$=y^5rtV#I zIX0saVUc&>y+(z3QLnET-%IBOVlAHMR@t$1=$(^;o~X$muFV_7-h62utEY*L7mDVR z_5v?_WTNT}U-o@gPq%$*w}nWng`A>*?uEM^RbP4FHQt<$0o-}<-UqT??q4U$>YQDz z-9--Nc5csx-r>~|&bT|+9Hp+U80+0Pc;wG8oq4T4`oyDPuFC!vtVPGDrM6n{JrmR_ z3Dh*4gTAQ=0qA|sYOBm&R%jdYFI55gEbb7XHo^=?Xi(xWR&Ui;$ZLVN?O|?t^Gyp` z!l3sUU8@aS^-1S{Un+2oEc0+%j&y$E<+Mh4fWwk_8@b=H-3W7*F+6CFzpvx;l#ahg zvB<${hOK8T(TNtmC|x+A#IH)2^SVB<{bb?~@?V^Hw;`U`GW#>ZyAOs4_@d0B(gFxB ziN=3a3-hbu`0G`K;ESgV%`Afm-nIIw@R|zXQ;vvJS;1`$QUUTsmrW#)WvvXt*z%Ml zW5`h%4O5LZp@SveT`EaOa27}Rp-K{x229xprymb6aMrg)01;RctLSoq(*A2(aRw@W z*IMrrT^})lXeyD2FWl%vD2!x^_Xzu=5&m+8?Sf*pim~@bGK&nz5-75h#dFR6@dXbk?1lLTbIs>$@Hgl-yzWWY(l`T6Z(W{m&aT+xS@ag&0sC<=lJpbsFLVpiJ_(i^^;ZOG8jCId>ATo(LYCCgNkx+ zUD;JrjIl!XqVPYQJmgt|^4Sn#Fqlb5i2w$3;-5uPorWFs&ea*)y%g;-A?e&PEImE7_F{gC5!`u^w*3kiA8tvf%lx z7f2^<>$3+zuE}kK*QaiMuuU62V*SvgS-I?ARI)XhJ7|zpKXGT7rNm}fb6T9AE)u9!I0gI7 z;#&nM1V3CK4D7A;#)2e=mi}Q*yonGZCfgvok4(2TK z^_EppWKWBD-N*o}C@B1>gkxZjJwbmjLD1mtRtON)7X{sHzUWlTw9^Ie;Vv})$gmzzxDt|k5WSK*9`B%=WqiKg^Bfsqw`4Br)`SsykeE+U<{d+=PbF*FGw>+vQ z7%=mDLawhBSGG9{rgRswsn&1Jwl@#-UplF$W+J#fHkt%$fyQk0(Gr{s^c)9X{Giw6 z>7gA{0^J{!q^hB|G~2}Y;hn=Yk%96$Yw%TY1P2KRJK(VDdQhE z7Yd{i#|PWH8%d@jb9Mg?PrhU0)9`99`eaNL5EKz~b>QB;ln$Cl5B20b>FbU_dyyCR z8?7Uh!E=E8Pf(4L8GANM+4^()notm@+-yAcxi25w8Y@k9Pc_mya*-uK7a6pI6N0v= z?&ygSQ|Hs??5xms&GJrrH{aH zj1>d|0BB)JG8haCs)P}72@XF;bJSds3{YytPMKe`w}{fEgFXO&+gPjs2G4~`Kp-sW z;HP1*Bq-F`-(w#gWErLFD+mql6;>xPTN{!N%BQJO7XtnkY9E2ca-(pq&G7vdg*?zV z0!@JMIF0zYo_+dCv^CukiM27Qb}MTJM99!4{6~;o{%GZ$G3zozqD{(mix|x?fS{Zw z*+igqyX%B1@i?;?0cOMQszbL%pRc*NNPU_T!N+8Z!S`Fg!jfv=l2o4_Sq4$@ z_g9#zzOAWAIBx8Ps^i`dro9!R4`?~|l8_KJj*^MM2H9h3vD6`IHxm-$NJdYb{iT``8-uJL(){M6LY>e2D)lF!< zNY8q@E~7FkL9%(wNvGN;O>!)|>Vb8EJ(KKa$60V#^PDCKMMnJd=vD*LhAI$TC` zp;LPYKSrwVuWtVpOnYv7_1L-nn57vsjH7fZLKhc&5qx|+69_Hx^x&L5ZgAtig0rqH zOD!RHozjKI^(n3_!|zpDD98k0*;=QqSl%* zk-(^Oo%n2KPucc zR`P&qS}7{{HOgH%`PjdH&)l>*1Sn_irdxU|{SEYFjlKxKE?6Z3oO!y%|9yO>ckrhc=Co1h$nKI=ksf%a}7^;5BHUfuh9sI$s#& zz>IJ^ZPWK|l)zBc|%=5){xZZ#MBc_!}h+z(_GqM(%U zaN}%tyNHv(sAMO8eR^opEvTyYT8qWwcbXY7mz3k|oO`veJipehRb!W;ZnZL<;`KV# zv{c$Q68<@QwO`;<^i^zC^mM>gH^YQo&wM;vjc^>pZ*AzDK%u8)6Wc{Z{EeEFP>CA> z4zpaFwoOq_v#c`V|5s;{$ zIF~jy=h|iU0QRYA_Hwp_8j-=`^-{eXy#wYyZ9IoRWIaOQe!N3wA4)03Zp_`-@XQQ* z%gY}w`X=UuL{7yBCVbm1Nb%4v_FRijuSkWD?h{{btLs&7z)nnzg-uGsFHxpp2E8nU za3$f->YVoB3Xgo1f~rzNg(nXt85P)i-rBWsdUa|;^{~VfW!Cu0LKDBB zle9IT022%%H6N5_g|%yHTb%H9se{|}ePyMMKF>q_Rpu7vCL*razDeh8n7*`?MP!(b zqP&1_G`h=A-X3e8{&|f#$di__)hB`%Ag>hP7+J z>c(k|DotnzF?~wPVL%^R`|%F!@^s-#2jkw3$98YVy{rxowV2a;{{i;^r5cP>$TZ+5%b?DJOu7 szI*`uV@MpKGS3h7Qegb&4SI4NLMj77KkXD{OHkYNv<!8&_r>`Anc3kx7f(}GdK+my%t3qNA(E1jz%m{(UvvOydz4>lh* z{}9$k$|faVz?(QrgX}PnogKtO7YN!CL0cNQCqJH+4PCahY=Ho?U*HN zGjdDx_@fe896GEW3BBs~hb*h^Bei+}XyC$1c~F6Ab-BKC`Ap%~`g3mXFMZh;+-x&E z)+lMw^2=Lt8CS&XZPRmraH9obWUa1Gpb<){%Pr|_kJLa=&pj$~;q>d~<;}J*-Gv6Hl!f!xNQ(wy-c~7TgJR?nia~y`hS+Ev=*~YlTHkg@$qdON+&?fh&&lf) zHl|#B-`6eotP^GGGJLAWjc@xJj;HhOd~HR0T7~GECLY8t1`yuJzn9eEPa``V+7L5b z>6#}DeKV_!XHfA`&t&qus>!}+p!KXiWH_y`#G@66$I2) z@+@pLaZwFkGchPcPqJ$eCKN~QB}faRHEn8EwGd&`=PDbb@aP~)KNkj-D^2>3gV+aV zzUbT3w&x))e_miJL74TeTBls z0rm4}bj^v;(7t`G5$$}o`%3C&YY;`YkV0hBGQ&_x`uwYFR8oTnYJ+1ME%SJM`6ZJk zu@oN(CKv9o-cV}cLv66SBwd*7y_Ipr_l5PsazxYC+=;s$5zfT1YJTgIgh+nt@5mKl zq_Yn_eezUgE$iK<%&X39KzyV&+<9)wm-lHwg==cCQqsqw^+*9t;^M|Hz*SW&&2=W9 zD)sR3n>Xbq#Oi^xn@w$2Xm_uYkn6ingI-f@g!Cy@=@>9)_K8Y8Cu>ZsT$6@Tkuu?A zwq%J`Qg3e$0qFD|Jwa{7uS{FU1nVIyJX_LEwelka0pP&9)r+Sk0qU3I6aCHe&~Zx% zvov-Wj4Z0A#k;(idy(Qj<44vI`R-(V6%z`}l5)m9p0A+FfAMzqu4HL>$bYu8G#Cw! zR40I)kt_o*K7p_Kf@&(V)6U)UU+>!+t!`;0bQ0t%z> zJSKp()Omz{UJ0|OtVKdc#y7d-v=Jm+N4pgCxoKA9d*{XXSB)90H@a3RuiwQ#<9^A5F#p&trB|q}`l2Q-S`0pM z{RwN5vk$e~vJ}2EYr)Mn(mBtNdrt^(8e%%6!L+zau}oVj{2~dIRF1n);H`!8!=3!D z?}83tyz+Cvp4yv<%c=@~^w9RrAy$Nk(brf`@zQpOc{LGCj>5K`l`(G?G}y)w(0)E) zbPVzHJps)$P&(obJI!bV7bIV(U@wamD=6ez@`p_xhGv;FrQG8CT%OAXav=^!KL!jr zYzq-jD{v4j4p}iD29`a&g|#d&J^IA8KfF9)G<)HD)6jt7{>iCp00@d z<9GM*`GBygaJg=m7M}PIM^W1oDh7c4U?B>nz%GfD@Z9)lmc3Q=DQgd|e8A5R$8*5z zIeg|S{R12sU%}qZ!>g}r=q1$;{a53jwB2(xw=y0`&j|nrG>zcxf*MCYBxnke734VF zH#?_Qq*vr8i7+|#?tlsL&LHdQ1RONPE^{Zz!7Uz!3R;&9{-Ks(LA$8hB;dLMawvXB}x zEG(c7y}M!#nYIdC@X_W3;x#7hQszo?GwdKp{{XmHfV|WNK!`%HXMkeXn-IRI4_}ne zsc>69wxdmY!xXdjALhzfK->UnsZGn>s_Pa|BYHRvBf5O?5(YBG71(Y=g(*Up@Pl@2`)5){L)-@szlBOo5vW%k}(|$yQ6WwLlt8Z=$ z=Bh4^tw7$=x4afomFHAbQ)(eLy*O((@U*CZsr`p_Bh!KMEfVB3*PV9-A#JR>o$X|KG@s#CfU(lK9gv(N%<}(krb`>h( zz=`a0#3iRR(0MXh&#F>_VAIvPXJZxv{u)HP9&uZ_wlqbL{Mor0i5ZKo$E3#}ol_d) zg#E6|F;8F8pPYMk)tp+rI%~O3xEsQvned5mw~f(E?2LuVnLlRstQvG%XB?DoN`75^ z(lunQ8=<-r+Z=MA$$zmBq#gg#_*LY4KVx;9J!$C3o!M~Zo!C(wM39pO-L8FBVj7tD z>!R;K`p=rzH5L(e{UvR!7&V6>rVXF0M1@rQ_HLW%73TMmLyV#gcPf=?1)_E(1gG^a z;G^@{2hK^>Z#bkzUyVeU=P?`>!B&<>T6hR(Au=eJ=w6*i6<2U12C0+d-?zxjVga2e z33L&-2IZsB`OH!}Qv868IWC~7`hZ%wg>bPPr+a2?ELhg-f4DoDK?9Ax{De)DVG016 z^|q9b$+n>k9`Rwu=PW!;yFH+U{T{3gi=k5n6totEw9!qjEwfT#L7hQvf3Qn4rn z7FMJe<&F9f01AsN@GF0vzs$-#F#8~!S>;eSAKW$FmL!g+D^g~ zFFKw%u;KdqaoULPm7a=(9A~1*(wOdywM^0XmbjTutywK%n+vAq1K>e%p)$KwVnt^B z(U&KhSDe-ZOogDe8n`6NPN?(swY7?&p;dMi$eZFqd%0-qWj?4~5B|EWz0j(%MEcyc zIvZTuUpA|5HX$)6YHuBO$^J(8L#Y7xQ&HYL z)l7oWU~qrc&}>20kSUm1{A~%pW{!}vj|ED2ijc3{U=|(} z=((mgn{-9gjOwNgldlRHEIfbCS@zrn87rKZmj$7`M-?}qG@>efJSzpv>n#EdJ7!;e z#lqhS0mqG*&BW`k5Zn8?gI)f_oN%nLV0z|H1H9WOMc?-|-t+_4M7~oT(s?jZJvH=} z>xFH*-a39*8BUi%`hptQKQXA(ZTbiqSHbE2qU~ohN~Q4-Q*4sc8~v}Z?04?BL%|5W zABB!IIS}1;0!rCcrO;w57Otu!MoXu zh99`6V^EpBb&PANe0^iuTZU1UkBmMBu>uFw3+xVgpxx1mteP=*!Moktm(w?1$jG7; zPz0pDlqV}n-ru10cEQ6f9>ih9dw7n>)JDNxY8vKH!O{Ib#_DtdIT=Sj1T~J9z{g;F z22%~TOHe;(pS<8GMc&r8$YzYt^Mq>zHFT7j9yT7bq0>$lxSYfegSTdDi2P;WXR8hX z@oEzuMD8SRhVaz)6QEM3=*Cf$0(~r&Q$@r#6BF0?z#&?w`T zEIq~FYs+yZ8lq-%uz(fih^J|8)*Zk{ytrEz*eGCKYHXb2_z(*!8p zKXrx;aX4%ZkiufF4@}kL$T%oCCLi@DojuwJz$3Hk!Tv$?U2jXe=DbL-eO_l={yW+o zrHJ})u-^Tt2KEybgl-~l6|y=@V1v?r#)@)H$2SGX3OVcskCV!-QNfie`Z{(-Qfk;| zwReHV0LC3v`2p9_)E zE0l0!#t>@7wWrP`g;{Rnxp@LK|1N%WP!DPQpWLQE)n$eH*rxpy$>ks1-B z(B3H9$0yR~g34PGe{ydx3$HET3UwTd!n4EyKZc&2Pxea!n0$d^at)$nS3Q0ydUiVe+#_7J zV?X#$;=Hm^x>%MV1>;~8s^e>HZRV`C)*gdfnRKyAC?^%|lg)X2zK(^G{v0rb>BQj_OY$BbN9*#K6xJ%U&mS2 zGijqGK@zo!2V5B}@2q;auNYg`yJ0ad+h&_5YqyJZ^J}S(^{lvl_`Kmy38*jROowuR z8j%gC4>;|=7Hj{=M$3|TF7syftQ&*+;K?9=Nght_tW!-PJHC$sHw&T_J`qCJdYSIP zpF)*{X(^NOz>1seH*w}3KO>~a7suj5EQo~}R(t{M3nrS1Mg>o!1kAf^a^KeL(ZQwP z&#}#>ks0zs_?9?>%W6R%F zDy5~MUtDD4+x7G-JGb%On!-JOOahz3 z^Xk(+DFo?t%7b^BMp+t(;<>?&)p??(Z~sjA@MCAaBIVe%4431BhDUWdUm8jtl(@=q zp(JXe?$}A}vv2ww%J*iqVp3ptQi?Zfgw3c|y(pA+j)`OjzG5#nJH1p(%W}NlaSRAZZQfcecU)uUGf($$NjY9nPoE9V_i!chW`>2eu;M@tAE z5_mA{PW_|wbM7e4`lDF=v`F9uI1(MukrXLgG4jt1|FiMXeIxmptopCDTBM_fSl^fs zpf0lg3Mv6Kb03EwWCO=NG>? zKKT9B`DFkxd9i1BY;x--z#Ib(p9LGoe?G0VubfR4kL(D>_V`x?W_2;Z1$;6YHcG4R z9KTDGWQM)nRGak{jdoYEY$#R!atFKMbC6sr0ql;h*}kfkjC&Pgy;%qjYgE8Ot`{!k z^OTj4UUFA|0k>M%PLgj!Q(Xo?XPf2@=jGzzV*&>RkagWOFk#TYN$OqMM-wiCQofOi zF)`bCk2z$i2z)nsIDw8j%op#?__$@PrY_qv(gS}U%OXHzW&OKQ_4menTcnlxbfjHh Rl5p7TIjd!;S)y(m_Fp#ycijL0 literal 0 HcmV?d00001 diff --git a/Documentation/admin-guide/mm/damon/eval.rst b/Documentation/admin-guide/mm/damon/eval.rst new file mode 100644 index 000000000000..02a36a0d6145 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/eval.rst @@ -0,0 +1,215 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========== +Evaluation +========== + +DAMON is lightweight. It increases system memory usage by only -0.39% and +consumes less than 1% CPU time in most case. It slows target workloads down by +only 0.63%. + +DAMON is accurate and useful for memory management optimizations. An +experimental DAMON-based operation scheme for THP, 'ethp', removes 69.43% of +THP memory overheads while preserving 37.11% of THP speedup. Another +experimental DAMON-based 'proactive reclamation' implementation, 'prcl', +reduces 89.30% of residential sets and 22.40% of system memory footprint while +incurring only 1.98% runtime overhead in the best case (parsec3/freqmine). + +Setup +===== + +On a QEMU/KVM based virtual machine hosted by an Intel i7 host machine running +Ubuntu 18.04, I measure runtime and consumed system memory while running +various realistic workloads with several configurations. I use 13 and 12 +workloads in PARSEC3 [3]_ and SPLASH-2X [4]_ benchmark suites, respectively. I +use another wrapper scripts [5]_ for convenient setup and run of the workloads. + +Measurement +----------- + +For the measurement of the amount of consumed memory in system global scope, I +drop caches before starting each of the workloads and monitor 'MemFree' in the +'/proc/meminfo' file. To make results more stable, I repeat the runs 5 times +and average results. + +Configurations +-------------- + +The configurations I use are as below. + +- orig: Linux v5.5 with 'madvise' THP policy +- rec: 'orig' plus DAMON running with record feature +- thp: same with 'orig', but use 'always' THP policy +- ethp: 'orig' plus a DAMON operation scheme, 'efficient THP' +- prcl: 'orig' plus a DAMON operation scheme, 'proactive reclaim [6]_' + +I use 'rec' for measurement of DAMON overheads to target workloads and system +memory. The remaining configs including 'thp', 'ethp', and 'prcl' are for +measurement of DAMON monitoring accuracy. + +'ethp' and 'prcl' are simple DAMON-based operation schemes developed for +proof of concepts of DAMON. 'ethp' reduces memory space waste of THP by using +DAMON for the decision of promotions and demotion for huge pages, while 'prcl' +is as similar as the original work. Those are implemented as below:: + + # format: + # ethp: Use huge pages if a region >2MB shows >5% access rate, use regular + # pages if a region >2MB shows <5% access rate for >2 seconds + 2M null 5 null null null hugepage + 2M null null 5 13s null nohugepage + + # prcl: If a region >4KB shows <5% access rate for >7 seconds, page out. + 4K null null 5 7s null pageout + +Note that both 'ethp' and 'prcl' are designed with my only straightforward +intuition because those are for only proof of concepts and monitoring accuracy +of DAMON. In other words, those are not for production. For production use, +those should be more tuned. + +.. [1] "Redis latency problems troubleshooting", https://redis.io/topics/latency +.. [2] "Disable Transparent Huge Pages (THP)", + https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/ +.. [3] "The PARSEC Becnhmark Suite", https://parsec.cs.princeton.edu/index.htm +.. [4] "SPLASH-2x", https://parsec.cs.princeton.edu/parsec3-doc.htm#splash2x +.. [5] "parsec3_on_ubuntu", https://github.com/sjp38/parsec3_on_ubuntu +.. [6] "Proactively reclaiming idle memory", https://lwn.net/Articles/787611/ + +Results +======= + +Below two tables show the measurement results. The runtimes are in seconds +while the memory usages are in KiB. Each configuration except 'orig' shows +its overhead relative to 'orig' in percent within parenthesizes.:: + + runtime orig rec (overhead) thp (overhead) ethp (overhead) prcl (overhead) + parsec3/blackscholes 106.888 108.158 (1.19) 107.082 (0.18) 107.429 (0.51) 115.307 (7.88) + parsec3/bodytrack 79.235 79.631 (0.50) 79.379 (0.18) 79.368 (0.17) 80.493 (1.59) + parsec3/canneal 139.014 140.668 (1.19) 122.524 (-11.86) 134.214 (-3.45) 143.914 (3.52) + parsec3/dedup 11.883 11.933 (0.42) 11.737 (-1.23) 11.904 (0.18) 13.236 (11.39) + parsec3/facesim 208.675 209.123 (0.21) 205.758 (-1.40) 207.150 (-0.73) 210.075 (0.67) + parsec3/ferret 190.099 191.667 (0.82) 190.354 (0.13) 191.256 (0.61) 192.305 (1.16) + parsec3/fluidanimate 211.297 212.724 (0.68) 209.383 (-0.91) 212.653 (0.64) 212.622 (0.63) + parsec3/freqmine 290.441 292.775 (0.80) 289.952 (-0.17) 291.501 (0.36) 296.205 (1.98) + parsec3/raytrace 119.029 119.661 (0.53) 119.551 (0.44) 120.368 (1.12) 133.943 (12.53) + parsec3/streamcluster 323.952 328.647 (1.45) 284.653 (-12.13) 289.881 (-10.52) 328.274 (1.33) + parsec3/swaptions 154.964 155.210 (0.16) 155.238 (0.18) 155.839 (0.56) 156.107 (0.74) + parsec3/vips 59.052 58.983 (-0.12) 58.950 (-0.17) 58.967 (-0.14) 59.978 (1.57) + parsec3/x264 72.536 69.273 (-4.50) 63.240 (-12.82) 71.688 (-1.17) 68.089 (-6.13) + splash2x/barnes 80.328 81.241 (1.14) 74.921 (-6.73) 78.784 (-1.92) 110.985 (38.16) + splash2x/fft 33.250 33.527 (0.83) 23.007 (-30.81) 32.634 (-1.85) 40.968 (23.21) + splash2x/lu_cb 85.438 85.634 (0.23) 85.112 (-0.38) 85.972 (0.63) 90.131 (5.49) + splash2x/lu_ncb 92.407 93.775 (1.48) 90.297 (-2.28) 93.443 (1.12) 94.041 (1.77) + splash2x/ocean_cp 44.578 44.827 (0.56) 43.087 (-3.35) 43.977 (-1.35) 45.840 (2.83) + splash2x/ocean_ncp 82.248 81.990 (-0.31) 51.128 (-37.84) 64.951 (-21.03) 112.929 (37.30) + splash2x/radiosity 91.234 91.869 (0.70) 90.466 (-0.84) 91.623 (0.43) 104.004 (14.00) + splash2x/radix 31.340 31.533 (0.61) 25.151 (-19.75) 29.634 (-5.44) 41.771 (33.28) + splash2x/raytrace 84.232 84.939 (0.84) 82.813 (-1.69) 83.432 (-0.95) 85.422 (1.41) + splash2x/volrend 87.056 88.313 (1.44) 86.126 (-1.07) 87.473 (0.48) 88.180 (1.29) + splash2x/water_nsquared 232.762 234.777 (0.87) 220.799 (-5.14) 222.883 (-4.24) 235.231 (1.06) + splash2x/water_spatial 89.835 89.925 (0.10) 89.730 (-0.12) 89.643 (-0.21) 97.167 (8.16) + total 3001.780 3020.810 (0.63) 2860.450 (-4.71) 2936.640 (-2.17) 3157.220 (5.18) + + + memused.avg orig rec (overhead) thp (overhead) ethp (overhead) prcl (overhead) + parsec3/blackscholes 1820134.000 1830056.000 (0.55) 1820762.000 (0.03) 1833280.400 (0.72) 1620112.200 (-10.99) + parsec3/bodytrack 1419200.200 1430358.200 (0.79) 1415382.400 (-0.27) 1431830.400 (0.89) 1428315.400 (0.64) + parsec3/canneal 1041700.200 1054783.600 (1.26) 1039941.200 (-0.17) 1049565.600 (0.76) 1051594.600 (0.95) + parsec3/dedup 2403454.400 2413097.800 (0.40) 2411733.600 (0.34) 2404252.000 (0.03) 2412274.600 (0.37) + parsec3/facesim 539662.800 552780.800 (2.43) 540933.200 (0.24) 550971.400 (2.10) 548370.000 (1.61) + parsec3/ferret 320565.800 331712.000 (3.48) 319322.600 (-0.39) 332973.200 (3.87) 331142.000 (3.30) + parsec3/fluidanimate 572426.400 584585.200 (2.12) 575249.600 (0.49) 584782.000 (2.16) 574169.400 (0.30) + parsec3/freqmine 985943.600 998553.400 (1.28) 1028516.600 (4.32) 998298.000 (1.25) 765104.200 (-22.40) + parsec3/raytrace 1747157.200 1757628.000 (0.60) 1743396.200 (-0.22) 1754790.600 (0.44) 1569932.000 (-10.14) + parsec3/streamcluster 121307.000 134103.200 (10.55) 118746.200 (-2.11) 133971.600 (10.44) 132233.400 (9.01) + parsec3/swaptions 14363.400 26875.400 (87.11) 14055.000 (-2.15) 26496.200 (84.47) 27033.800 (88.21) + parsec3/vips 2945431.400 2963379.200 (0.61) 2956226.200 (0.37) 2943295.800 (-0.07) 2957360.400 (0.41) + parsec3/x264 3191271.200 3196683.800 (0.17) 3161188.400 (-0.94) 3200352.000 (0.28) 3184628.600 (-0.21) + splash2x/barnes 1213799.200 1216094.000 (0.19) 1217834.800 (0.33) 1212186.200 (-0.13) 928636.400 (-23.49) + splash2x/fft 9462681.600 9183411.000 (-2.95) 9265091.400 (-2.09) 9166673.600 (-3.13) 9424556.600 (-0.40) + splash2x/lu_cb 515952.400 521722.600 (1.12) 520563.600 (0.89) 520948.600 (0.97) 335968.400 (-34.88) + splash2x/lu_ncb 515638.000 522689.600 (1.37) 521192.200 (1.08) 523513.600 (1.53) 523046.000 (1.44) + splash2x/ocean_cp 3349483.000 3290003.000 (-1.78) 3380084.400 (0.91) 3340689.800 (-0.26) 3292473.200 (-1.70) + splash2x/ocean_ncp 3895485.600 3871371.400 (-0.62) 7066323.600 (81.40) 4970278.200 (27.59) 3637734.400 (-6.62) + splash2x/radiosity 1472652.200 1470431.600 (-0.15) 1482235.000 (0.65) 1471199.800 (-0.10) 513667.600 (-65.12) + splash2x/radix 1692530.000 1686171.200 (-0.38) 1382322.000 (-18.33) 1583722.600 (-6.43) 1892434.400 (11.81) + splash2x/raytrace 46488.400 58600.600 (26.05) 51538.200 (10.86) 60059.400 (29.19) 55203.000 (18.75) + splash2x/volrend 150022.400 166788.400 (11.18) 151466.000 (0.96) 163678.600 (9.10) 161593.400 (7.71) + splash2x/water_nsquared 49073.200 62559.000 (27.48) 60601.800 (23.49) 64762.800 (31.97) 53323.000 (8.66) + splash2x/water_spatial 666074.200 672698.600 (0.99) 668054.000 (0.30) 673098.200 (1.05) 536452.200 (-19.46) + total 40152600.000 39997300.000 (-0.39) 42912754.000 (6.87) 40995700.000 (2.10) 37957300.000 (-5.47) + + +DAMON Overheads +--------------- + +In total, DAMON recording feature incurs 0.63% runtime overhead and -0.39% +memory space overhead. + +For a convenience test run of 'rec', I use a Python wrapper. The wrapper +constantly consumes about 10-15MB of memory. This becomes a high memory +overhead if the target workload has a small memory footprint. Nonetheless, the +overheads are not from DAMON, but from the wrapper, and thus should be ignored. +This fake memory overhead continues in 'ethp' and 'prcl', as those +configurations are also using the Python wrapper. + + +Efficient THP +------------- + +THP 'always' enabled policy achieves 4.71% speedup but incurs 6.87% memory +overhead. It achieves 37.84% speedup in the best case, but 81.40% memory +overhead in the worst case. Interestingly, both the best and worst-case are +with 'splash2x/ocean_ncp'). + +The 2-lines implementation of data access monitoring based THP version ('ethp') +shows 2.17% speedup and 2.10% memory overhead. In other words, 'ethp' removes +69.43% of THP memory waste while preserving 46.07% of THP speedup in total. In +the case of the 'splash2x/ocean_ncp', 'ethp' removes 66.10% of THP memory waste +while preserving 55.57% of THP speedup. + + +Proactive Reclamation +--------------------- + +As same to the original work, I use 'zram' swap device for this configuration. + +In total, our 1 line implementation of Proactive Reclamation, 'prcl', incurred +5.18% runtime overhead in total while achieving 5.47% system memory usage +reduction. + +Nonetheless, as the memory usage is calculated with 'MemFree' in +'/proc/meminfo', it contains the SwapCached pages. As the swapcached pages can +be easily evicted, I also measured the residential set size of the workloads:: + + rss.avg orig rec (overhead) thp (overhead) ethp (overhead) prcl (overhead) + parsec3/blackscholes 590255.400 590427.600 (0.03) 592919.200 (0.45) 589924.600 (-0.06) 272620.800 (-53.81) + parsec3/bodytrack 32331.400 32280.200 (-0.16) 32304.600 (-0.08) 32295.600 (-0.11) 25386.400 (-21.48) + parsec3/canneal 839327.800 840517.600 (0.14) 837729.400 (-0.19) 839485.000 (0.02) 838981.200 (-0.04) + parsec3/dedup 1214897.200 1205565.200 (-0.77) 1231421.600 (1.36) 1206902.400 (-0.66) 909662.200 (-25.12) + parsec3/facesim 311656.600 310912.800 (-0.24) 314467.200 (0.90) 312564.200 (0.29) 308407.800 (-1.04) + parsec3/ferret 99738.600 99711.000 (-0.03) 101068.200 (1.33) 100018.200 (0.28) 90036.000 (-9.73) + parsec3/fluidanimate 531879.200 531871.200 (-0.00) 531891.000 (0.00) 532066.000 (0.04) 521523.000 (-1.95) + parsec3/freqmine 552169.800 553773.200 (0.29) 556495.800 (0.78) 553485.800 (0.24) 59071.000 (-89.30) + parsec3/raytrace 894327.600 894141.000 (-0.02) 890956.000 (-0.38) 892035.400 (-0.26) 303397.200 (-66.08) + parsec3/streamcluster 110943.200 110954.000 (0.01) 111407.400 (0.42) 111340.000 (0.36) 109841.000 (-0.99) + parsec3/swaptions 5680.000 5563.800 (-2.05) 5660.200 (-0.35) 5658.200 (-0.38) 3846.000 (-32.29) + parsec3/vips 31890.400 32098.200 (0.65) 32154.600 (0.83) 32321.600 (1.35) 29437.400 (-7.69) + parsec3/x264 81907.200 81854.000 (-0.06) 83120.400 (1.48) 82805.600 (1.10) 81796.200 (-0.14) + splash2x/barnes 1216312.800 1215561.000 (-0.06) 1224936.000 (0.71) 1219227.600 (0.24) 576351.600 (-52.61) + splash2x/fft 9587848.800 9587352.000 (-0.01) 9646184.400 (0.61) 9625447.600 (0.39) 7134239.600 (-25.59) + splash2x/lu_cb 510697.400 510591.800 (-0.02) 514411.400 (0.73) 510292.400 (-0.08) 314724.000 (-38.37) + splash2x/lu_ncb 510252.600 510340.200 (0.02) 513900.400 (0.71) 510359.200 (0.02) 510247.400 (-0.00) + splash2x/ocean_cp 3411545.800 3408327.400 (-0.09) 3441360.800 (0.87) 3394161.600 (-0.51) 3333649.000 (-2.28) + splash2x/ocean_ncp 3931731.200 3921194.600 (-0.27) 7178746.200 (82.58) 5099068.400 (29.69) 3157059.400 (-19.70) + splash2x/radiosity 1476138.800 1472629.200 (-0.24) 1485801.000 (0.65) 1477227.000 (0.07) 229545.600 (-84.45) + splash2x/radix 1757003.200 1760448.800 (0.20) 1435953.000 (-18.27) 1661870.000 (-5.41) 1456666.000 (-17.09) + splash2x/raytrace 23309.600 23310.400 (0.00) 28946.400 (24.18) 26803.600 (14.99) 15641.800 (-32.90) + splash2x/volrend 44090.600 44095.800 (0.01) 44491.400 (0.91) 44292.200 (0.46) 32785.000 (-25.64) + splash2x/water_nsquared 29437.600 29428.000 (-0.03) 29851.200 (1.41) 29746.600 (1.05) 26154.600 (-11.15) + splash2x/water_spatial 656379.400 656150.200 (-0.03) 656559.000 (0.03) 656055.200 (-0.05) 484139.600 (-26.24) + total 28451700.000 28429000.000 (-0.08) 31522759.000 (10.79) 29545600.000 (3.84) 20825100.000 (-26.81) + +In total, 26.81% of residential sets were reduced. + +With parsec3/freqmine, 'prcl' reduced 89.30% of residential sets and 22.40% of +system memory usage while incurring only 1.98% runtime overhead. diff --git a/Documentation/admin-guide/mm/damon/faq.rst b/Documentation/admin-guide/mm/damon/faq.rst new file mode 100644 index 000000000000..02f7581b05f6 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/faq.rst @@ -0,0 +1,46 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================== +Frequently Asked Questions +========================== + +Why a new module, instead of extending perf or other user space tools? +====================================================================== + +First, because it needs to be lightweight as much as possible so that it can be +used online, any unnecessary overhead such as kernel - user space context +switching cost should be avoided. Second, DAMON aims to be used by other +programs including the kernel. Therefore, having a dependency on specific +tools like perf is not desirable. These are the two biggest reasons why DAMON +is implemented in the kernel space. + + +Can 'idle pages tracking' or 'perf mem' substitute DAMON? +========================================================= + +Idle page tracking is a low level primitive for access check of each physical +page frame. 'perf mem' is similar, though it can use sampling to minimize the +overhead. DAMON is a higher-level framework for access patterns of data objects +that focused on memory management optimization and providing sophisticated +features for that including the overhead handling. Therefore, 'idle pages +tracking' and 'perf mem' could provide a subset of DAMON's output, but cannot +substitute DAMON. Rather than that, DAMON could use those as it's low-level +primitives. + + +How can I optimize my system's memory management using DAMON? +============================================================= + +Because there are several ways for the DAMON-based optimizations, we wrote a +separate document, :doc:`guide`. Please refer to that. + + +Does DAMON support virtual memory only? +======================================== + +For now, yes. But, DAMON will be able to support various address spaces +including physical memory in near future. An RFC patchset [1]_ for this +extension is already available. Please refer :doc:`plans` for detailed plan +for this. + +.. [1] https://lore.kernel.org/linux-mm/20200409094232.29680-1-sjpark@amazon.com/ diff --git a/Documentation/admin-guide/mm/damon/freqmine_heatmap.png b/Documentation/admin-guide/mm/damon/freqmine_heatmap.png new file mode 100644 index 0000000000000000000000000000000000000000..9641bade4e2e52cfaf77ac2ef50607239ab90899 GIT binary patch literal 8687 zcma)g2|Sct`~Q9424jh_3o#^O1!s0jc- zFaUrgGy@`mIOX2~00Urj+EkB9r6Q7?oE+K*5ddH+5ugHOG7$%KbO1-3C#Xa$0ON=- zRS6Hr000gKs4z8{`u35mETW1dy+QPZgoF+pIACC4;OOWW92}gMmiF}NQzDVr-Q7Jo zIf=NSC@aU45()tH zME1@Av=FI4Zy%L6ks547fnlm1um)fImh4ZNy3vB%J7*B-pRG1b&j+%qK@zfkoDv?M<@}UxYYO^R*N={8r3$>+( zNVrMG>y z=&5@wpAuQdl30h6+4@u1y3^QSq_YzsiF-^qAkeSirPY@MPJ}BjrMeDufjXA`(w_?R+ZAStZ6;jRk6U1GN%Bog`2%1vKmh zo=F2uvOu#O@LT~PDgrG^K>?Evtn3ec_!yifyP{sDAo0;a%1(Lw35SR2*DkIn8I^@@ zhF~jCPzd$toMPn)`1Z-EvyGem8;`Cq%eY)u5WMlm>V3V|mq|W?ct-JJpPA3$_jA7X zcgePT{;2LXR#BmIz=*294H$k|w!baL$I_s&jW2Z0H>N4ytdUuMeIK!HYBx^K)&{g( z&zJ2vQX?+2b~vkXv+EvrDzheAU}6S2_zGBU(kgOk^hkt=Td*`%jng{Omm#b>65nv8 zLQ>;Zt7wQidN$j#r@hK`FI?q%Ns~Ni4vuo`3g(R={=;{r}7hC-@nc& zT@gvDVYpn|W!UC9Tb0Wd9#^RgTF&LC-Fat^8Qm$r8htt9!MTIE6k{?kL4*8$%YuCN z>h(iABzl4jzm=(vnyXkFwqM=WytX1rQ4(pO>}}zr1n=Y+i-=U*IDsq?XR+8}UUMSV zPR(0b6nP9nAqKY)u@y+on0VuUW4bjPg+{9~BI2uz8~Y-3YbPGE1#{E15OF&nC4)Wk zN}I00mLv{GkilTwZp)E^l|>qMa9Xb>0$@u4O)&GG|0Vv!x?1s1ABP10kwfr*$sz5ZIqdys4q`MwHrZmcJ*WZgc`p}N ze|+w8jc@#tH}P|6hJ3((iZ5>Mt>D6|&rTaFa-(Kx72W*n28X{%P+VgVXGN*B;N5)~ z+Bn6uyHX~8c!xz_{CMl5z9Bbsw`3%#D~l_?B^pxm#PLcSEhu?K5BZ^rc*Xpn2fu8-1s_DQM`)MAOb#{<&1cud zc7UN1aSHfoE$l~COnkeC#NJ;j5yFt{BDnWG6I8kC{zXy(#VeVzBE@QEE~y{>+dw6} z6vRQSHqz62ziUNX2x4lRKZffu-?b`o7w!q#kSjKci+W3Xx^+yKh+o}AubM@~!v@P- zgoELt+r6Wex=^=2+;nRYd^nEOaZhEJoJy!g6syUX~a#gNZ`E2JA zgxX+cl8SsdHe_mDe0Tqhp&%GDBU*Pbc%8cAXc^OjZu+*bJeE2sb5PFM%COPmb;JH+ zD;3IZKa8`hsC^hEK~cz1;+4%jc68T+;)cUnd<09khTY!>JGas%UdpxFFw{G7Z@OAo{0K6;wEMB&B4jL{?d^8#{MW>gq_>dh{fQPGFQcp2ucNqD0IOAfd_Ni1sY2zU-Lz3Qx3P;6P8hm=hBlyza-y2K?P?;N^+ECD~i6bn>?ZIP% zb1cUbV?ECjGc;LCwN4R9{PWVib+WBuABC?&bpMORJE0f)4~PhzSimNi2>6NdUdm|4 zi7kb;CyW~{r>-9%elx@36{nL@zK6gAB{-s$+8`rXXg(4P2e$#>FV;x-Dvf*MIp#eW zCDre!AYt_r=xMV9s5`NL(5Ayrum4EPe~=xt&od;|CM1TzHY&Qr;UXTIzINCD$=HGa zlQC*K2>s;a06)`G(ZN-ui{stivTUecyN7vG;)oXfJA1!o*mog{v)g@M9tyhs|I%7v zxe;hvvEDFl{fUOLjl&_RxtlcUIX)CHqrj5!nJ%2PfcSgaN|v%WJj(9Ae5{C8Yw9#+ zeDchrt#H)LFJR7Z8(8&HxOP0*pEK8G zZKLw>QP06&zvb0%&zDx{Z}i*aH!+z!g$-EJqHH6V(Si85o+qo={5O4)OyZxX&3ku~ zy0u6q14<5M_FM`cIcKdgp8iQ==v9^o7(Qiilx=EAS6K=u5Erbxb zT6zh-Fr|UgC8B#Ja=7y~?(P1FMM3(WqF{jOMV~H%R5A^?u|`r zkuLx3x)*Vs`)vyP<(yh6#5!%2pq0roB;ftn&W7%60e!q!j;T?`EGbWmv~;?gWR+8t zE@(uN61)eh=L|;~HO1M;hzB;SkTsPHAhH*!Qb2Y+n3AH#lSGB$9rN$qz zZcWNir_Zs%HMw&YUvnEDU3qwJQ}M@bx*aAc=#E`%^yOk97nIDDMqr=Zl0b7!=;W28 z7?{lWkhb~vZbiqOv5VfQ=xn8yRy&c$#3;P2I`?VJGW z4xSM_mi;Dsf9X|~?&*0x=NN@0LA_ydrbB4EC$#W}@tk!D@WHj2K#yhEo%zk}mxgTw z^>sRQhB)==y$2vw|9#R8W=ON3=w$mFd}&=Du)^VUfnB%I^+RlV>tWRaddoehD&3KJ zj_%!!A271F*r-QiDDRM%_i_<->0OF(`>WEOo~bjSx*L6{L{5h%WsX1xDgmBbm1wi) zVnyGW>GB+2V%*?~Ff(jdF^h0|%C~LmD;Hr}(aR?2&}DkgieS(E@lR(oYw4Nbh}w=8 z$A~6IgVHkm&460{}?`-+c4noVm3s=^|OwrSmR<=ZKyv64i-d@}m?_7R#)HGo6 zCq;4a?4suG7=9O2~7cI4Z zj&(Ll(<2DrGrH#|vMJ79MaQuO9%DFBGI{2MBazM^1~51hgpk?p^#fT3$M@tEcZ;jN zetM9fZXI6%@tfFcct$B=QJSgn^By9H4lk4VE{vxlvV^~CM9pRm$R|JZImzd*sML0Z zUKKn8sKW58PGosf#AUxh=LKy_ma?}Tojj@-O08hQGR&4G-Zvx>)gXQK#FnWYHb-}4 z2^4tde}B!_`cpyv<5nClrTt91*_uj9;=WSDmcIf`w2LkL5@X^kf7SfMB|9ws+;*Vc&arxyB zv62V{JtWevC@Q%N74R1q9TP`GI>gcV@3#|bWHu|AAL-U7h?5?oHuxfzP?uKVHLdbI zhLWKAg75dH2&09$=gKZYg1}EQYr^qD-*N5N5qAxL4)k^!aRe$%5Hm668%4RPM@F`#$ zj{jFfb<2-fZyV88zK^tTf2o%i9<71qN^vbX4~}$TFwpqal!MaP!@#$-j6<-S{OVpeKY&*Jx2~-+=GY-EDc7U z8|Uq;WwlO7#E{cUH+3hFrHd&9DJM%{xxwp-)boN`Bj%$ocSVtBNE_k^WTQG5iV?K6 zTlWeW-u9`51p!`IaG|LGX$R)b*fsqAXFoGbTQa|52ycdJi*&MIbzCzSLETYoTZ5WA z2rrISDF^c|!{lU!n2DsXmv&5P#=`mCVo8&IA{eCx_ChYgcPSu4K5hzz`y4E6iE&D$ zKW?aU8;nG;&o_(HRvuhBo5_M^4mYHqGNB3apU>(tMmF{Hq3S~dls0xX9&QNgr8VmH z;jkJ{D+mtpVVra}6qxzjaK5w#V!%k;xWIcTrBF(G|ZK*TkVuiuSzDbEPb5G z6lZT3dyR+%dxhp+SEa9OP!J3{>FD5y^|YR8p4X@*G=lNtyW0oEXaAMPhZ5Mq^|niL z6{C|*>c2aiccNS?Vh<=rcFBnnKM8O%}GYw7BHzt?PMo$$@W#cmq6cDE%wz~Wo=edTgQ6{04m zPA^LMmS{5jgupDqV~^q1o7^`f2Q?ReJnQ{g+zmkm?%sjh3&>I>WD=|t2~TW?u*7)t zR!f(7iWVy4b43VT%gVj!zM=0ci=W9lnAmsf;W6wq!?^76#o8ajVHCwiLyoTonV#gWoib$bTG0Sey-MDO_O|0bk9nJB%&oqV0xr?t8Y~Q?A zGijSeqH3ktRw_M&o)E2bDDO}C>`@9o`ZcPJH>%R;{Bf3z1(FIwV5>f&Eg65zxm`BU zd(NyX>-u^(+15eSQq3d@i`N+*K;K|nP|_K>P*hwV=fpV6zkZDEd-=tYU^}D?7@t5e zINdG)G#g@eJAS)4&X}j!Y6Flda>Mf zcSpO$OSbO5X$9+rE)AKtKVxa=9fYPvQQI*oy=?5)K!_4XD>%}H^od+oWcZDa+2}yY z!I)!ozz;%>fw(=WI~<6452xWRD^G|9vkvhRO?m7ItP=3Dyh zv%yPBra*DL>F@!T;a%Y=vFCBbD#8rg zDnG(r(=oM6Em8c)tg>5?Cxu*Izt;07Evf(xr9z#fhr3ixE*3VIY?=u|#d z7PWhO&mJfTP#+Y2(=S61OCsNb8);)K(5dV+yb&+<>=@4#9C1tsqa-emIvuec^M)}K zx-I}qx7IfoOdERVV~G7hetA80cZ$D3H7FAK6w-!x5s{;YVaO*y?sOk!r$PxFs<(;76^0DW)?tIMG+gzh+kQ93hxmet~(VBkSD zlotlo&Eq3ZJ20&%&FT}8yVFR)57;oIu33b%Ww?=_FbCK)hHl^H93g7j)$$rWKW`SN zHk4ai*0&31TrUQF8}e4?JCq;{ zO`#hf!Bg;FnXej50--b~maLbJZrvTai_mzln-usw=9U2APtiBPc5R+n;;Ju*Dea(T zcSRv5$8DdwE${DCHFm0jGxj2|IKGCEhI-5X#omjz$9N+Yu5Z24!0dK+KZMM=ZXVS{ z_+txcQabLl7^<7k=M-3?X!Kw(Go0pdHW?CS`E71=Cl-%Js|nCAcQV=jH(!3avvVFH zIw2dnoj56wW|$uuN7g>gzkVWqDxCgp`}0)XZca`hHn+`wnGyW|ySVV*hc> zQC?Nc@^dSlUEn;#^P!`Si=y)1SpCb0@unf$st79cB@@(~KGA|+ea1zxtT6k5N8OZK zbkM{o?FxZ^;E3L6=*81p@aqFO;_oKv?>HqBZ#Fr z&Y-pGGtU-n=t_l~EoYuDMswP63brAc-kYEx))V(_gz!@aF`lA`yKAVMTBuBB)M?sK zZD#t3#YvE3AIV93Wy=>iv?Dh288;T0v3LZ@2g}Ubo1HJmSMI7X-c*(9E$hiVN_@#4 z#DUxho9eq?B(V){kqT`i;H$Ieh$o9`>YHAS^EVsOg8CNs902WyOW-}<9LGm2OZKwr z`SWf(CYvHvc*O)AzDT+Dc5vzAn38c3&52I1;_410!~i}5_u5BC+t9>looemXZvwVk z$W*nfh2B|PmP{_5aLP5xUxCE}L5POf$5zi9p!Px^vbqZT$dR6KwBYAIxRhVD6m!Nv zqS#~U5g=y*gTc|z$f#oHxWZ397Sh1>%Od1`?_E{CL0OfPJ>AviyXGd6CvWXG7(AF{ zqLZ&*zh#|!`bnSYmGh#K)`O-F#?mIy#-bPV<{!%oE6FiokX|rg8{h!cP+ka{F_VB*4c350^b~*Q6cxiF;huTiMW@W;f)vYgHA7#6ar1=J= zmQ2hr+T>Xb1mKo+s*UbnwCJlvJu6q<^0TQbuoOy~AFsS~ZEy0U(C>AXBKQ5wJFHxX zt=9Wi^M9p|C?C_|-EK9JnrX(E|4d8510)A-)pLPt%O zX-a*a=tZU{I(ekSU1P>N&^0|FLGp%X^T^4uw+Rm4y!M$Tld;0q$)ESn_q&fsk9{?k zis226{9>YarL^C>BPoq}ky%}}tI1fgkEz5^;9LK6X!grvt{Dxt_r>%y9Y23mBP@Z% zK^@Dr!m${cSyQ=2R<_i$Q0}YFUy=Ao`lBdzt`g4zrx|| zce0ZCU3BQts8d?{dOs}ftnweXi5_$3d}!?OxCuOW@_gznS9O5oMa6+bW_>=!Zi8pM z6Lp8>uNq6Cyh9AGZ;aprvx+JYW(CVQ^hL?m$VWQCH~>!ef6 z4D!76uH}xgp1WT1?y5um7n^)}wn4?;6#eH%>X)DA?tCu~@N(IDXl2m)w1N1oo;szZ z(qQ{jsI@T;4F^0Xj|Usf0!w8-*^(GD<7Y$(kMCm_ZWM2wi}p##EX+t-7aC1+`R?O4 z^~3RxJ%v88z=uEi2+3upmFGG&RdB``E{fN#r+-Uo$mnM6bCK;yv=kk0ip8slfYrzV z&dRsetz=dGtAV+=l?tENX#b{g&B=ro(w90Y-C-`-Pw_&nZKF5ErBZJBg_+soh6>al zx3K&x7`J2%(4ldkNOOIa03jeJyk5cFm#fDH1BG0cViZ}KO@9K1`+d4LBy3eyM|!wl zfplC>1Ksy+=FXibyrCFOPZjLXYgIJ0+gy7`Ug8DZp8TMkJoauQ2G2xiu+gdHU}w3r xPe8Az(d{5n3{M|^ho>xu8=Ip3zabb&qJlrac^-KsBdbLdpU^WrTCRf+`5y|rTh0Ig literal 0 HcmV?d00001 diff --git a/Documentation/admin-guide/mm/damon/freqmine_wss_sz.png b/Documentation/admin-guide/mm/damon/freqmine_wss_sz.png new file mode 100644 index 0000000000000000000000000000000000000000..28049dec5fabfca0ca33896ee6b667dc116aacbf GIT binary patch literal 4986 zcmds4c|4Te+rMX!ea$X=A!RQKnMw96rFvA-3`&%2d9n?2C*d)+P*k>%;wO|sPhxB# zdxS#9PHCF#40FF{c%I+qec$K(>-Xn7pP9MubDeX2&vmZrI@kH0KTg=1^KwaYK@h}i zX<=dyL1-9)P!tXpAVKr*?*k3(6V|6q5d;B}?CfmjgA741LWU5CN+n~V=4R-q$0$OU zgJ3KfMl|s-34*XNgun<9>3yQA3RJO_E}$nVDXF8QV`F3E=H^Bu5+6KxP*zq(CX?IR z+9oF_!4x!Y?L=zg^MNVa85#vasKen%d2I=miu6NZHs})gCn}H>)JYCoZIXn94=@FS z+M_zYLrfD9sG}PZPDY3)XfTYJLhG>K?EPTcRP4KoYn74QOG`u9upIIPqJ~pbpq}CE z;lvZzGuJd=VF(r$hOyua!+0``M_{J?SR@;K@kq80LM9_%JqWqIB9n&Dvdi0Rk=k~0 zDFMM|BXZT%)gvP#I2=w^Ru+T7WMyU5)YPO){#FFm%;{rs?gj+$H86jud;9pLAV`pE zX>#;*VET_??uWUBqLI$hQ40mWlsjv7&fM-vRt17D;n$CThISMjPvTpg^42gp=fn=6 zm>)6{Bk?q>{B_7Jx45#|_XRgKx{5r`RfNAQ80Ot@oU5XqO&#Hn+sxFd85|Ci>63l` z2H)QW4JPDEt@pEj36fZbY|GB+RBmdoY%&z(TxW*+J;I2+t3OG|SAIt7XY)@f-=!pJ zJlXt@c?au`Tp3MQBpiHaz(P&U4L}J!Xz}hZYW-Tu5Y@dC0@atu=VgRU)`<$doasTJzuoR;qAF zL#)BB^p#Vwxygy+kk!&SSKLN@oTnrCE8@OPk+!YB^rJ~caxF^gT%8uB>&~gs#3pK> zg8mguaaVKAP&4CX`qJRxo&@@>jFCf+UJwbwsTHzf?s?s2UR@vgh4U{{UN5rT@sJ2q z+r&Db-*NFc=0w{4TmApPx(@DOJaf@Ud7h5Dg(*z4g{027QW8}2caqIoL<)lnT1rXJ zn9@B4v}FgiytJ5E*3y$NSuoY9^;J$<8IZ2kP0FT>UI7!LyR*}-hJl;g6wNBk+Zvd| zf{6+^Ua$plm{yH_oP1(S$B&YFZKSc7*)H=|^0irZt{!dL`Sn|MuXw8Nhh=N3v6MWKRk8k$y1s8C zeo%m)#Xvc#=;%TYxx)XvQK*W_bJ4$f25r&kKUc__7&ALyxudINbMdVZuSn0YTX@Ln z`ENQ^zoGJ7^o{OvX9xWYD}6#sI&}iQS`=Z~=s%<+Mh=q656f}U&uM1!vB^3}8fmfD zk`4eUrmDg%0NhJxh>ae=C7A1?PdyuTK^-2~flpgEIr*VpbvkK^^!TPyOco8#VI*t- zobuYqA@O|YXKC?+VZB@4dNu%kan8&SD$T-4kRaW?>Tv5Ts+mLRYQr&Ml&Ae^Xxc+_ zRN26KVeP78WrbrNFP0*qLg#;j6d|l@Hx>1u|6vROj?{gyE*5zUosEFsRYNxWJb%>U6AY1 z6c9Wjc-{DPT6AP|1N4$DB6h}!Hj}FzKX!yUZ)P<0>0{Zc1rgmOu!=?;zuroJg+smq zY-9BxT6cbUh6A6QQxX|OSC0aAjRuo(Ba;GTr9C^k0914(w9Ic^k2SdfMLm`^{Pnxw z*68StUVdOlCANOUnkBf5!>kRDTJ=9EF{10zzsdJ1Q^kdisQx^~lkayB`RTccZHGkk>?wWRNt)rtChKiw+Yu%B&Bv3>n76UtT&*mR?I zt^2-c4Lt4NwS4NQ)#FoI(iK2qa9NFAEbq(GVuLNCD?VnnQkk(|k6@5-Tg!P6BZ$U4t!$f+9Kw~ZLf_ea-Zxs>G`9~Jo{r1ZEKt6WyEbizy>f^8rXaSf31A> z{cL9p418p-l+#(+`DNn(4>SEPK{a_`KZ49EITbK=&zJ*G8bj+XF9fIezd1a8Zllj1 zJ&@DvR9YjfUlfbadut3i3KMzUqkP*=s;5D^eHib@#piIuN3Y-zd{

+Ym-wN z;(j?S7Vq#DF*V8&!L_&N>AVBTQHq{rEwq6&y!EqogLH@nzZ3%C^ zx;R>aMUQ8=k*ffw0Q0(t#WG8Z6-V~$kkEBpai>|Mf8a0F1v3oy05KJ-nqN=i&mQz@ zcdZ}<92OPn8Ql{qG)8)!xl!=z`mDQZ{Z4_p&)$E2wR67xd&xHzW>Aw2`Q77XUR^6DKm}>>A(xbMoNT_8a)TW5 z66DRs)DOg?fZSj=i~EMwuqYEaUbgmV1^v$X`yfLs87>i}Y|L_FnHC*JXLPWGT2`sN zl%KbNnPgVCp_E_$ zPXMdB=cp~PC581U=Sswhy_|kRQz+hhNpebuVNaC2ytA00{a#KNAu8u%d!dQE{q0Ae z?2fj46sBE7)eO1lYKLRRmnk@M+W@65P4-rDEl<9Q~q6ab~ps=&)Gu*7DM>tVc z7t?2?w%;zK&wx-fW&eH~67LQR1hQFF8y1}8eE&y|J-HBDD$)3pM+yF6$%V(f5qaN_ za1+Iip}}MUtii*MP`NApDj#k}LgmVrx!pGo=>(YRX#vQ+{VbA5`;!D{ObT;!ZV7fi zndU51qd)@VCNmSmWbW6BIE&Yu%}a`z^|7BqJ+IMNZPmep@LQ%w>*5YsH~Q<ADH|I zD0~SUMlvi)ig0_2A5WS!a3~LO=w8%Nz`LTvn36rAqPy}}%tZG#_1mK>R}%)zzC}<% zESq4jcnxy~2mXdJBZoaKeQMVlK(OS&Xa0~ydD;=6r`a`sN-9a9JZ%V2l-Ss?JpA19 z3tQIL*u{hBfroj}6p0h`*+V+LGhYPAeY@Fc^+)7!a$pn}r76T1)O#RHEW=w^W<+IX41$T9h*=+Mf79&*mn2Apg*UkoXp`W5Be4kqKr9T^{64{#{L zaq{ppRW$`c$YPwQZ4fk_vW z0(n$M?(zb`*a2${e~_^SB$X_REsH&sDIi3CPw*9b*(Py+r!vsx%a)$1zM>$Bv*OO? z$KB7xq6gA?lM`zRBf26zcg3**eDj;4O#ZOoFewh67W{Y%%1Bgu={=9Y(7j;G@7IEj z2+9AlTxc@<+`*`J2DD5snW^U)pEX~>HCUnD3&QlGSB(-i4Y-GIt5_54AS$*GBt)(u z>cA)ybal5Kv~q0;gqZ&&tItrvcMy1f0(6HxbYQ`}(Vvjy%&lkmjd_e@HM9E!<&l7j zg%Ny8CBX>f&Y?>W*{z>~QhyuD$v0N<$teucJr$HHbK?aZ{A+=B@^gm=TFCQ*)8%bmz8@AyA40y zw3z_q+|LrIt=aBMs2?uL-MYU>q^A=Dw*rvZIJ@ISJCj5E=@}ukt%2B~*|n&rTLZ_` zb%`07OwUFeU>G;HyZb(Imj2y+2;N$r+>c=XA~)vM=p%>R)j#92wsgR@VUp;4yS0C5 z(n?e_lz3|`m7NdTuuDSO7NRsD-}kn`!>beza%~ol_kfhMqT2eJMak3jR15N*}dO(81hqqOo%I*Epzi7 zoKGk2G*}ANV{QA?EJJRhi&>EGq1!vDIRB2w4MBef^k+?@;pOWM84=<{p^vsunK#_z zvzS@(*e5tKRBR}!Lie21gXNO;x$rXCaK6FMuGNuqVWivbYZVf*sJ4M-naJ2UcLUCr zP$8o^?~J#Phk7)ZUvPH}G?%l@Nj;2gG30$?h`2nMHYl?*d3$(i&&aM96%npvc~`|R zN;x{Xe9WDU@w9i3y$}_3oZ`4HZ*tk){)92V145Cmj3vg@`VeCeROE`+fx(|3B^iK&2$m_3$$6&IrD zNl_leIp3^#`cCTa zFTQb7m4;dI<5|@Umy+3M7xdF^7i(fJX7{{uc|sYbgQq^T40olr#4a|>*$tz%5G=au zBHAP3jhEh zwA0LZ9{@lI0Dvq!4<>;`)^}k8jlH{VOgJ14CSfob+#eJG5Dp4(fXPV|2AZ3JZSLcOt6yZ>5NHhSz2*5!&!JM~QDk_*N%zBOKNlQy>YHC_oSh%>j1P2G-x^=6xv=l|r zmX?;0krB)UTSFsuGB&^GBiokE;&3LJp`1tcg_Dz)=O>U0M|th(ALfoi{}LIWg`g31eiz8e7YIP{&=OP+@s>6;P>zS41~<8VBL}4|5pUo5o>yaZr?l#lt~c%hT8#Hseui zJ*U1EEehhm3=X-rwsvrE(9zLRUS8hN&@eqcy{@h@?nHLr+`+1^EcOx2Oyyy(Vf(C0Z(o5gd9dNu1(?dG#H-`)?cx2jdpu-Yi8 z+HLvNFR_2bZHwHFr*iDX2f&x%U8vq6acRM;Gx7G*clQSz_f+j#3)WkGgp&LD-m1(M zM0FF~ntsW|c*9=MS+GW6qDr8{J{y9(1T^n>cIQ1vNT1Ni4-LM94sGtMyMkSL^5#yKaMk5)G_N^RNHZl3}yA>$;s3G?mhzNZtf3BglJ zRV{pXH$+W-z_Iy{q{0Q==T#XmqqVO-X)HF_GLWUQU+CV4g2>5f>~?<|N}-$bTO$4v zGuMPufT{H#Wo322frBku^t_L$yNYgAv`rE2jGre^7jz| zpn?YM9sqW~mu2!G$T<7jBnYFv^kpEV*zzaUzDc?A5(M(+=_L4e_y*lQx0dLIc3P~v z-{LHrJ|_2KhIhsO)=cR_^UhPh4(iC~rB4cgQ32IReRrS|i=Mz`t3BKN@uvb}Y;u!e z4^`5uEZ1_%{2uS=*6vzjY?35l?C6BF&DKMmKQ>Q4uqx(HDR-1&z6`jI+*vnvWoF3s z;0u>^8?+Q@S4G)}9296Xa@Csgwr$sLUTM;0ja2VMd!FMvY)N1+h?z}59!{6q#;eRp zsu)T2jX$WZ5E-mKn@_O2U|xpBrfU`i7~ z%9=%AZ#g-AkCE!g{Jx${Rx821jTUr<;X}>@I!|>*e2hM5A+AdEF_H__1wCX&yFa_u zDDY!AmZ)xeVY%&j3rQTbZ;d4^?%F5N9R-5!E0r0V7D9ixIw)Rt5yoYjq##}0-Z&~l z-~!(VoR-7W=eAq*D6ilYMD61+St0}}GZSZP&`p))10cSsMv`8S>q|sBCApF<;p5XfYa90xH0@1eg{5Lu$Oj2-t~^(Wu@c@;;O$x`HCwGx z)qDzB4Kwz_J!>)V?vXWqw-;?5(0(O%^|*|C6$Lt!7{P}c2~v9^?iXt)hL&W%i4Rqq zJLC`PotEbR)$lW5N*W7r4|-zMotS#fur4fOxUOB`L^B9 z4&M(AHkN$1lG4-E3QfKmVDh|9J*uS5f4;l<@L2dlq}S;{n&cC6H2C#RMCH6zDqXOF znrJ#3*hmgp@fs|}dv_5!ysj2qYwn}SntUr|e9M2dkb+$e$ zNVi)qK#@kT{SVJa0$6$Xpel~Lib|yxbg=lZL)n36U-1(7G5VN^75#YOPFhf$&ECG8 z(+7k7B<;kCKY8#u{mtP!xFFyIuDXD8`%ln%+;-wK2!jcLp52mG{{tV+E%sj3!8IrmPGtjE0Sbp2_pAQoDJ#4xS_~AU zu*sV&8{w0vD|r{ZHB_BJz~B!;1HjZ+<4n%-8cK?xf=%(7N8XBDL^AQD*D8*;=O}J< zi(TKg!|=XvLehqawmmzjHeYv#Aem;QqRSEK54$ZSS73Br%X-SbW~oms!eMYc;I4mX zux2mZBDtV2Oo;p$Dj%T_Zgga4#43|Vg6a)Bs@A3^uHFydKwiOV$lDp-SD3|Sk{2ha zz6n{rnhmPw$7Fc1qTbDp>}_Q7x@p{(Y^|n@&2l07VAW^I25Jn@%abnR}KQ%48uAi=^qu%NxYhuLMo`yWM z$UrWCCWQF;-4eo_NJGAEY#-RasdMDO32hV)?!2x{)0g}sx=wabz~Qm`z0Ezf*_*nq z78Wig<5og(tM|aJ$VF9_kminhc194NHB{PX)> z&JJCwqa;1<<${FEvqA(l3r$<+cJ&wDV^0Ah|rRf{YADvR`id~i_!->zntgM7HJ5# zX}hJ{>QVbeaTDjm!tH?+aoL`z10avw-k*-+#X| zF+BOj-jsoeyO{SrWxlJr27?wRKfLZbVhu4&FkAGxySapS!Ff{n*s1_%pJ9qyTMjhv z)p{_sW*noAyoj`%S_4}@!YMSB)UH_aA0?kXDYfr7m z=3a+XMpLH;?9ZC)r&OLF?IyEUgg|^AxpX6SEtR-%O&&$EBEYFQI?gR&Z!A5)u25IAZkM(qhw30F$mK;|0COPJ9%5^GHnf=(@g6xJUhQ00TP z);g1z7Th2&9 zZ6SyWCU}FhIgzw6NuA*ymnBPPffyCa7J43HcCbr24>xnVlamGaEVDph9gr_RImKQ`qTpVI$ox7Dy=*;!ha z%&gzMT~!Wt%Bv*34(QNztF*Wuo*^X{osxEstrdfYAO*0jsfKGp(DCNGvucS!q7?W%R(slD2aDc5%k zYOLV9VF*8QEo(p`W9P#YsiZB?k%BHz_M(g< z$;!jIm&%s;*?cOEh}AX<+TZe#5P}7-!b-sU1SdfMy5=snlta9py}|S5gk9|u71@$p zec1Z2e^&T1@ptT=c&E=0h2*wnW2e8YiWSD&(KUE6>T`c5RU((|a8}u!sAH6xCw_Q3 zE(CgJfHy~5VN}D7M-&2Iewliu#6wv9lXsauI}Kbtwv$>Umyw?t6x|^V`v*o$U};Z( z-e=!zfNhf^WGnPR2ojDV5Mzn>1AGt{28NA(N6G_YF&rn8Da4``I0YPZ^PK|gQ96-J z_LX@#FddTqX~63V_~#pbh|jLJ%3E z8tWQ{dI9_!<6f7|lRi~`FND!GFnapSm-;ZSyK`4_U0;q@1-nc=wWqKm?eK18)@{kp zE4fF;M$;wX!yY=n^l%Ep;!vPuE=$RQ*Kynu9<>-^e46e0g%Q(~9)Uc+Pc1yS1swlY zpf~X(PY?ok21FnPlK9!;;P%`ef*X^4=)6C7GoM;nbgehynbA` zg$GXSvCw^6_Q|t_lCS@9Fuc)?AG38E>=O}$mHpe<<2*7-EtJbj14 zS~)3M6k9XO*uug7qtANZAx#UJ!$y;mR3kOUB>zE7s66n5+M;t9x8N+`4Wk>dO^(@@ zb&ThA)sNEWs;)fkj*R8dDub>@TVznZjbHmE)?|K6FFp~Za_Y1Xk&Q1TvawU@-&a*z zpyIu0IO6!P#gCNmG8OzT@tXi=lM(-B04xW<=Lm8bL7ZV!%}jzlGi2>#ht~!Erd~z6 zFQ57<8H!wDt(3P(M7BIT zQUr>tUlW%2{(n+i;9Xe1WJF{W^*@9m&g^EEiGid1DG-VS|J6Mh)*!Pa$mCT-FF|7L z3Va3c59nFC>wD!;#1!Kig80Cw5De>)S$x`p2j;~oEXlv;C!^PKboMk3i~)XgHFiB~ zv`N_#!$?tNaW@(ZyacyX0Ca9j zr=5i2e$^YpqeBnUOf|*X*5~VX`yB}SnK!Y2=_5xcH#m4E87^Nub3^7a-fifCrTEv# ze8?|;GmNInbl2iQ|Cgyab zjL$6WN8k5^bLSPrZWbbO9CD5G3ga+HdG#Y1J88t&;kkUIp*Uw|-f{hr>|p6)i41dv z`?0(VdbP8~;)mlEX-9HHdD_1o1}`169{pZ-pu&89{q7SyqKcv-x4E;<)W0ppE1Q#b zv(AuWKg^{fi^Wkh^T9;^ks*$b)#{wh7OJ!r16y)&N0l2hEt@eNwVNkTJ&RX}tHNj6 z7QMK%$(TUzuBojMP&%3P`H{@dIJ2=yf4og__ylj}P6jD-Y)(Uua>M>CQ}!lx^I#Cv|ac$k&{T(wOw%Aj8etr=I^)W&ZVYZPBL-GUFo3XXo%NcehQ|1;^f8mAe#| zbHu(!r26W_thAg8(C4gGDDx6EadWLMc}Je=C~2Mb?4fhQUS7zoc@y?{dQqe`ZhCu8 z;*?nHmJ#Jb-@cjmGny?=Jhxlkv&gQo7p@kX@Ep5bw4PR)l9sPg5IWSm<1X!LR18G@ z*;U$>)RkIhUHSfGPnt-2+AWDM-4_>f)Z893d-3WN4=qtI`ef4qw4$pqG&k2acqV@; zW~0FknbX~iC()>%7n3ZN8KyeJSBJmI3-+8(7LYPjQVY{K>C{=?nAa=XG?_ySv`v{Tue=vgV4#sY#Qd+{8vD$g=$2hGa|h=fxYsxS&me-#kg>R%pOk z3ywN?so|A5YE>y`%YONMwF2QvuL`Rrp@5`O1}=VMZQNpiUl$1)$z?I zE5nAFO+yIt#}D=*d(dz_|15dk*+xZ(w + + + (small mmap()-ed regions and munmap()-ed regions) + + + + +To further minimize dynamic mapping changes applying overhead, DAMON checks the +dynamic memory mapping changes and applies it to the abstracted target area +only for each of a user-specified time interval (``regions update interval``). diff --git a/Documentation/admin-guide/mm/damon/plans.rst b/Documentation/admin-guide/mm/damon/plans.rst new file mode 100644 index 000000000000..96251752b3f0 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/plans.rst @@ -0,0 +1,49 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +Future Plans +============ + +DAMON is still on its first stage. Below plans are still under development. + + +Automate Data Access Monitoring-based Memory Operation Schemes Execution +======================================================================== + +The ultimate goal of DAMON is being used as a building block of the data access +pattern aware kernel memory management subsystem optimization. However, as +always, some users having very special workloads will want to do their +optimization. DAMON will automate most of the tasks for such manual +optimizations soon. Users will be required to only describe what kind of data +access pattern-based operation schemes they want in a simple form. + +By applying a very simple scheme for THP promotion/demotion with a prototype +implementation, DAMON reduced 60% of THP memory footprint overhead while +preserving 50% of the THP performance benefit. The detailed results can be +seen on an external web page [1]_. + +Several RFC patchsets for this plan are available [2]_. + + +Support Various Address Spaces +============================== + +Currently, DAMON supports only virtual memory address spaces because it +utilizes PTE Accessed bits as its low-level access check primitive and ``struct +vma`` as a way to address the monitoring target regions. However, the core +idea of DAMON is in a separate higher layer. Therefore, DAMON can support +other various address spaces by changing the two low primitives to others for +the address spaces. + +In the future, DAMON will make the lower level primitives configurable so that +it can support various address spaces including physical memory. The +configuration will be highly flexible so that users can even implement the +primitives by themselves for their special use cases. Monitoring of +clean/dirty/entire page cache, NUMA nodes, specific files, or block devices +would be examples of such use cases. + +An RFC patchset for this plan is available [3]_. + +.. [1] https://damonitor.github.io/test/result/perf/latest/html/ +.. [2] https://lore.kernel.org/linux-mm/20200429124540.32232-1-sjpark@amazon.com/ +.. [3] https://lore.kernel.org/linux-mm/20200409094232.29680-1-sjpark@amazon.com/ diff --git a/Documentation/admin-guide/mm/damon/start.rst b/Documentation/admin-guide/mm/damon/start.rst new file mode 100644 index 000000000000..0f1a366ac245 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/start.rst @@ -0,0 +1,119 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Getting Started +=============== + +This document briefly describes how you can use DAMON by demonstrating the user +space tool. Please note that this document describes only a part of the +features for brevity. Please refer to :doc:`usage` for more details. + + +TL; DR +====== + +Simply follow below 5 commands. Don't forget replacing ```` +with your *real* workload, though. :: + + $ git clone https://github.com/sjp38/linux -b damon/master + /* build the kernel with CONFIG_DAMON=y, install, reboot */ + $ cd linux/tools/damon + $ ./damo record $(pidof ) + $ ./damo report heats --heatmap access_pattern.pdf + $ ./damo report wss --range 0 101 1 --plot wss_dist.pdf + + +Prerequisites +============= + +Kernel +------ + +You should first ensure your system is running on a kernel built with +``CONFIG_DAMON``. If the value is set as ``m``, you should load the module +first:: + + # modprobe damon + + +User Space Tool +--------------- + +For the demonstration, we will use a user space tool for the convenient use of +DAMON, called DAMON Operator (DAMO). It is located at ``tools/damon/damo`` of +the kernel source tree. We assume that you set ``$PATH`` to point it. The +``$PATH`` setting is not mandatory but we make the assumption here for the +brevity of the below examples. + +Because DAMO is using the debugfs interface (refer to :doc:`usage` for the +detail) of DAMON, you should also ensure debugfs is mounted. Mount it manually +as below:: + + # mount -t debugfs none /sys/kernel/debug/ + +or append below line to your ``/etc/fstab`` file so that your system can +automatically mount debugfs from next booting:: + + debugfs /sys/kernel/debug debugfs defaults 0 0 + + +Recording Data Access Patterns +============================== + +Below commands record memory access pattern of a program, ``masim``, and save +it in a file, ``damon.data``. The program will access two 100 MiB memory +regions one by one. :: + + $ git clone https://github.com/sjp38/masim + $ cd masim; make; ./masim ./configs/zigzag.cfg & + $ sudo damo record -o damon.data $(pidof masim) + +The first two lines of commands start the monitoring target process in the +background. You can substitute this with your real workload. The last line +asks ``damo record`` to record the access pattern. + + +Visualizing Recorded Patterns +============================= + +Below three commands visualize the recorded access patterns into three +image files, ``access_pattern_heatmap.png``, ``wss_dist.png``, and +``wss_chron_change.png``. :: + + $ damo report heats --heatmap -i damon.data access_pattern_heatmap.png + $ damo report wss --range 0 101 1 --plot wss_dist.png + $ damo report wss --range 0 101 1 --sortby time --plot wss_chron_change.png + +- ``access_pattern_heatmap.png`` will show the data access pattern in a + heatmap, which shows when (x-axis) what memory region (y-axis) is how + frequently accessed (color). +- ``wss_dist.png`` will show the distribution of the working set size. +- ``wss_chron_change.png`` will show how the working set size has + chronologically changed. + +Below are the three images. You can show those made with other realistic +workloads at external web pages [1]_ [2]_ [3]_. + +.. list-table:: + + * - .. kernel-figure:: damon_heatmap.png + :alt: the access pattern in heatmap format + :align: center + + The access pattern in heatmap format. + + - .. kernel-figure:: damon_wss_dist.png + :alt: the distribution of working set size + :align: center + + The distribution of working set size. + + - .. kernel-figure:: damon_wss_change.png + :alt: the chronological changes of working set size + :align: center + + The chronological changes of working set size. + +.. [1] https://damonitor.github.io/test/result/visual/latest/heatmap.1.html +.. [2] https://damonitor.github.io/test/result/visual/latest/wss_sz.html +.. [3] https://damonitor.github.io/test/result/visual/latest/wss_time.html diff --git a/Documentation/admin-guide/mm/damon/streamcluster_heatmap.png b/Documentation/admin-guide/mm/damon/streamcluster_heatmap.png new file mode 100644 index 0000000000000000000000000000000000000000..c9186f63bca4292a3f2e7602b2095c46914b9ae0 GIT binary patch literal 37916 zcma&NXH*nR)HYf@149~;1R0PZIY`ckI7-eSN%Ba}B9bNaF(4U45CnlCBN+rl6c|v+ z8Ho}F6hz4)g20_~-tVsO&wbarYxRWg>gw7Rde`31es+wJfhIL2D3s-K;m5wGC!_!z;iN3mjB|rv~uX2H|=j z$^ax9gLD&+YYvKUf+{Ph!4`UF552zy^*BO9&d|6kG~*5}ctR`Qh_%~@A9oPjeuzDP z#6ck9Fc^6pg8X|Ac@mC1jUYLLNzVQuIeS2Q`iS%-iu5=dbrgg89Y^*no@_UfY%_`c zdouag6pAlTDdy5Br=C%brc?DlM|WnT30c%{vT5paXv%VF3-ai)^68%z(8m^@yI;%@ zQo`t4%H;Zz*|zMwX$6Z;CFWWct9&(^Xw3zlT6WAU4%&K7(grTj$PHfefF@q>h95Kw zfOmqRMHsY-fHqM;5CiQJphFV8mj)kX!ACjJsQ|i^K({jJQ31VILEkmduL=g#!JsA> z(gwqNV8j578iBDJ;FBpBw*V8CVA2Lm*@0;sm~jN3UBIjxnDYelK49J#EZhZ)fnX^F zdef&_IURs zGHJg*(@CR(xVgjL97j~&rU%{qn<)u0SKEaxGs{JE}65Uh6DIU4*@z;T z+<*vYTRzIbcM!gMrhnutd87$7Jqp_L$vFP<>O$7SnMp(#Y*@m&U7~sPI`*64(6H^w z%W1Q z{?|6<|Cct}{=XalPYyD07AnuN^A9v=mFcdls$1g34lh`QaS_a6gVp0mTQpR^2w9p=cx<)um*=+u1q*V>n*ohk8_ z7B*V&ur=M)@M?R&^@v+xfUy2s6K<2?;7B?deT?W&r}*nM8GAgvV+YPIp4}xm6#4jm zxz_$j0v=OuR%z?I(R!Y6XQ>2mEF+hfb`)4JAF{&#ehPQJJCsBmw1}PU;2-Jpvk7FN z_3yiq#iz$$^9x~`eo;K}|Mce8)@qaVk#rmnA&KKZ*}wYCwruSuFFQlIk%MnHoW-1w<@Mn6L zC-V8(?*O~;3x6g1y{#@5_@@VaYyHV?*{Wv8y~B^Wt*1wZW2#{%6B~AmwPRTS6q@lxoE~JlcM>;m- zz)NhWaluqEJ0c_E`EB{t58*HVZfWm1uO%$nIxa})L@{RKc>=*}IM2t<;W8jXrdN{v`anm@>Q0X zZe#pzv=3JG6wOK3*}DRtd|!JJ;aKkjl6(G9JY5`)-~fh(KGK=4PWvpM7Z#|tTNN;5 zns4Dk&9DW48(}G61NAoQ7j3x>WFExC4qMe|Vz<_pV?lk%)Q9J1s*}Al9JPwC)a968m(@pg zJ4RDt_BwCoZ=^|EX*sIZRTtD`D%k$hK!!z?RU|$#QGGNU6_EdK^u{M%b?&0Z{MYUU zie!Y?EUOAO1MLsY`Lg4)pYwgVFs~5C>y$tKtjg;J+?Vq~9IAe{qyO$d*wjutFU0Ew zR$WRJA{xtE29)_v~s+hPo}f8j40G>Uoj zrfi=ht5&e0MUuLUxqt5+1FP4XzN^wT!B?u{KB&BgK-s$+!U94E557BoIr@7K6E+CD z|AQX=ieA&d5ixf!1@hw>kS^X(y-};Dd_Dh{=#s$;TJGAs!g#mlFTtiIN5cX>c@^h` zT;Oz59032%@jAqbI3a|f(N8n?&KHRR&I(F=9QY?c}=DGJqmYO$BJE= zCh;DWFeO-THcr*l_4Hn^i8V8pop_iaS)?Nt>}sFO@HyI#qzQ$kufAYTFY^9AhjK=N z!cL>y18(+tst4~b)d-vB-zXh#zQc@>xHE6EHwJFM78Lk7cNh-B?)=UY;E`d3-?q(8 zKAdF}2xNbAgDC}Pou9&BrLmggUa{Qpi0tFN&3EVN)0i}PZLBA6DeYgG_v3LK&F^&M zyBfWx>-z9#L$tbtmBTLk#kjX6_8G-E;u!QC z%+E>u{upC5DfmLfp1E>c3c~+%%m6eGO*)^TD^qEfA5#b*FKBs8QF|z+Zn5zRHazg0 z+zE5G(4D?qs}WMo&94spuh<6-CYsbx?xg*AsFri1R8f(wemqJ`e_zChhux^Wt%F)5 zE50t2>mT#F75-H;HGI9oi0+`WZtYrz>T!O-7?`?k)c?k)4#PFBh)e`&JW)-mI*$DCd|4itZ zo8<<3q~CMUblQ|XB=0NudP`N>P6SD_D+_f-C`Bx2#!_Xc4iaiJA4Qo8=B9X_D&{;L zZDnpbhitaw_%-@+IJ3cA8p{U_3%l2U%lD?3K@YvGNM2d9m%tpg&*Z-^ox6G8wZ`*{ zrw_UNaI*{SrrWpZy};$lY8))`QsY4H$ww8ukavs>54K){?UB&@2xOTm_%5?IPPr|X z%dIrw?ZdYk<^kBTr4MKcdJjH|7FTZ@PfjiJZyAtqtW0>%g<_m?TCuw{Z!L;2%<@{} zk3)jlPd-|=?eX!3$IZ?5^dUbHecH8`Kr(s#OzjMb$O0Sgu6zV7$ko3mojU~fT)mfl zGSYeaU;TajVC#GKm8;W{j6wDQUu$s5>7xs*v}=$j?HiCSD2%Cn0cfUz-`IpW>M}m6 zr!QR!@J$^ajiTqeaZAXmnKYOJZm~S3$BmaJLw@_QX_fr&XhCJgu*>jOi%O)j$(qts zLO{F!5~U}ivM?L!#Fq_EsZ_w0j0vwXTwP5XF;Q}jYr0Fm2o|@x#6ao`)4U`4gs1Jr z&K9+a%pjq%Z{`oSAqP4T^cMMIF7u|Tm^o-WeSZKKb8f&75v?<6;~j<7fJ;p!Erajd zxk)4Zzt#S7n;`B58*Y=7wtjg7=o+R6$A6*I-aFi^js-U}SiwJro+C5BJ-royeOI1e zHH_04&+B*aSbaPD{M={D)$B{8;M`p>oQ~nL+7KlKQqd)d0lz#-%}hCm{7>`ewJpD{ z=cXlmrOcLaY%l4sOrP1Pef&Y@ybIjO)2R=A$r@oJQO@Mu5JXz;sq+3 zdU$FK#Cmrrz*DJ_^l(T}@~6XkGX;);^|AKhzvll*B}k=KYIr0B8R4)`?c{8XLojhK zAU=LZnOEZpMyB$qJjhtNEq2St#eq+>uA@;y*FGsy`yp;Zx}QYE_84D4&Gh(`owPFJ z?$?eTCRCIdm>M|N^`0=jnGh_M2_P&FTzPYyv9hGEZ;iTzy!7PVf=8hC^mtGb!O&9WTj1 zz#S@BYC?Vh*0xho<~T0$m*?Qa(pQd^vlBFO1sq~7X~Ngi^Ibs3!wwmj{$7(YnV9b? zMKja+4yGp4TxkGz6{-9iu$b$4K^jRFOaZ9UBg7mR9mCA5_R4=i8TQ0(=*EAbm zN@D|;Qlw8(GSEhO@6;G>xWG0wMnSgAYxIjh73h^CnawvS@Z0S#3eMSRD;J~g4_gvIxb|NWt9}s&f zUP{O;czoS9&^_Urq>e~9IbDm6U`I*{Jx*5xPT*EAX)7>@tMj%>hm=NbW7LM5>0@bB zWnD-FS4riJ#>`Ef;HX`C95oy}nOLwF*bwla-*=k?C;qOwJ0%^>qe@CUk*;HUzxlp1 zY$~%#ZWJw-x2Y*m8kS#HGSg%xdPL7eJr27+*_U++yPf;vje z>V|qF?$OS|w%>!WJbgz6e%&c&C#qroaVJ>V+lE}n84g2bNMVbZTwib?Ryse-cm2wM zWY6msi#z)%2kr8C*uMQTBKX7ynOY~D$&Qsp?8`;^{+^5FL3Vahfp@iGC+3AI;`IbJ z2xbLO33urLn#qiGlp`;`g0jaG!)9h8T9Y3oI{y}h4qBE(T##TeTR{K}kZLer?O^e5 z=SS{?(m&+M_P#C`yt$=Ok=S#lrPJ!S6&n)H$%E)3bWNq>yO{6#Wv;LbnQx37AyZ=z zH0Wp0kGweL5t3(pnt?7JeJE%r+(^2{kg`BtQ|Vt~h8r+)hKncIJ)&frFScHHjQ~bX zAMWj!VsPH(We%Zv!S;3+1#I*GQRl9Z_=ChF2`F2A2;OAI%0X%;QEyB?)8mH1NwBpL z;lB7oLi}4YHw!04#0n3V26pohhG6w!zXZxUvH5$Hr`AwufJLQ8L_0Y!|Kq5b13zSC z9LZ4glij=$+H3LXtJ@{ETV83w=OsSb%WVFCG58`^YC+Oji4rE3<83+(M-tTny(qKK^-j) z!uw$eH%c#tqPtYWbWT*`sz%6TU^2T-Jx-vdvBc@5?lcygaPpexug5Ivq($ zE7cj+#}JlDNUnKX)1M63$54LV*vrYv1@c6d#B=onFS@r2&LzKZpx|CH^t=K>>%#$y z7&I*M(XAI5=pRQDRTd)*u2bJzKHDO{NY}#i-fh5fEHU7 z!q1}mx~DD~>_IiW27&i9xz3eqa=XsA7@iD@@()ZQ6JWQQ0^6Y!u01I?$3L~&cl zR}u^(TNv1F3Ev$Ubfe<+PJSsN z-(6SrCJ}%6JKL>;nM-p*+=xHDuMkLRDI(PPliqyz(0ROpP(D1Jekx#?8=eDrK37!T zMjug#36J138noO%hDjHB&3eA;3_}lzZPde#uI@;)sOc0V+gg~WW(a)jaD|I~_ciX( z!gHdqoX0{%FiMO2$!YVs@9#==e~M*X3mU$ofwAb;Sw{ftTu4}Tlw|m|AZZEuW;`e| zU=EI&5NGc72?XJ9Vhj60deJC|c8ReX82m^rker;<-3MUk{Jev!`3|K)vz4N0wjy$}kCBPi?T<9}UC3?y zp^5>{sdw_&AbAcY&HbmZh`_8N{E#iL`UEXe@qI%9s))~*#^$D4YtE*Mc`wUipxtD0 zii9ZjP0#2NM{R?Xhtr$*n z{tuWO7IC!lie!rSrf%&L1Gc(6fg%FclnIJ2Hu#J^KL^q{uTGH>dRKWg^yUV^8wH>x z(|8ZqbbSU{?ATv{q9K2%6{7 z0k=H&Zji3VlxjD_sn}wgNH>n)a{12uXW9i;L3)WYFc8$HxcFhDS>-+3tWn9$U}tIU zo8jPdV5R%P99qKdMiyEkU+ESPHdQdM_)WBLuv?$4_1FR>uGQeZD}~X)+PP+~BMRIo zchg$SEaJmLy=tYIOa`sxBZul$yLce`6{jhMO$vE??nqW69+&AeO>G9;_+hI_C)5<4 zOE-|jZD#40vvM`$`-h!R8vF6S%7>y}s=13@-(=?%q`FvQL1UTSacc;@wvM4-!Q zIS}{yoV;jrnY>+vxw!?aes|V&+Z$$N{Tm$XHjN3__NsUDi1EvcKnAR?mVGB~IGR@n zbw*kK_*8I&0^h-Vp#eY3%hm`aB`>Wh^hvqP09Fw1kr#hgExL~PhX45SBz)kbOd=6^ z_U#pko2qU|dN3HKsU3dt-c8iu7fKV# z>A5~5{O_QqT;Tw}Z<&l5XNq~j^4vFz)vAp&M0N8Wq{2%JTVnxn_fZ*{$fW0*j1j3H zs|MRQSXxFI2&)~w=2?_r8l7BGs@L&MyG_UlDg1LcJy9s)GmXY9BKr99Kc{BHIj(T< zj*7-Z}@~k4oo4ug!q-^ERzC$e}2Ky7avEHJ0MU-?y); zkT3eTd>=Z8yCM0Vu@Q(gBB$;~#rPtco_Q7V27%~j9})Ofv-ye=R$qjm%n+bXU{P55 zLoP9g5PUHW>8X0ZM-Ml8P)V#Hxy_{srtabF<)T=<=(&s7H>7J4L=V1R$S8M&g6p-q zJ$0vazCU+P0|PdZR74)6EANw}MrWz-tey^kof6*ke z6&|(VKe>+<1jMf07l5->pGRLFyB@N45!}rnhKAg)@WbM-1x|xQhV&zb?&MpWeVt(u${4va6a&nn*c8*jbV%~fATS9BzQQF2NLVYFe0+iXQH7a<6zkQ0_$V9IlM2p^P zpwDc$3=cLo-UT;bmUo5&>|J?|ANO;^iZR14CKIcUKRLlKPz8l1JcP1hksEk&hlm2N zU8W<=XH-u%afY= zil3oIJlM+!8+L>fR*290jR}dZID4DO%;m1fu(fB)cLkwkLAw3_mkYon%4hCHvbLva z;6w$hN#y<|9}c{M!1aF1!ukdEa6Df@buvPsw5^35ztg4OxnkiH*ZjRkyj8e$#-B*Kgr2BvlTR z|KdhopaBQj0k44v=oekv{US!FTKlN+P4#!vllfmq`ubu|Xx)k0u>fFsOYVWPGJa6H zp7y0qhsx8`sWYhO>BU2qR9k-VBVd?43pjjPHRf;d&~0bb>7ZW>C@A=uN`f88_MDHr zif?A4wG-Lwaitv2Ie*dH$ZrFk9BhTaZ2=?48C*lZCSS$fP+qOjPDv5rmm2VSKArL_ zgwddGX&ypvi73{&!UQJ+6b{T&8{R>I*&HnOl zw|Bw>g{>!_G9peL#HFwdDH|IYuDeQKdCvI{c-E2%Zc7cbc+$daPv)x8+pm%s7-DgS zYZh?3vB|fI(U$?JXkm$r*l)CtvKyER-i5GxZ?R4@^AOtIi@M2c0x|d@d+u`~yHV+A zi3Gp^Vkz(ec9w>5eneO+rAuem|Dw^V=TzyVKgOCJ0`UCPFuERciC8+xW1mnqAEc>8 zrI8+ESd0uQ+{Zy$T0TE9yMdSsCzqKJ(`mC505<4+cU|{s7fZAEM*!DG7A?wUS^rAO zChgB>j%uybO@<@O$!~jp>LLUA%XMF`C*PtVLh|lv*SvY{{Z3C-(aHXzlHIU!WTC}$ zn9}3_g1_>JVN`$p;OnAEc2I{N?xnzqvYJ?i{G-nXNSD_eXxanj#UUfJfA$J{o2L<< z#E?!vD}0Xz`xYtaUn-5wmgYJY~Wuy*;V4J#F@eGk@KPfejA9$r^vL=p-+o;Qk z3RgI1%&T<@Y3f3GA>!T-TRI?mHn$eV|WN zlf0H)XN*?WM%HDSyJ54p5`3O)MN0I7+gQN<${R*#b7pv;DQLZ+4=8gf;LXPTuaz8Q&$u#sU4C}g@cgUqgun@x_V#Hjj38!C%#~5F< zQMd%NyQ|RA06v~R64(x7eA00K4y{o%1Y2{)zx&%=kbb5*DK4%{{M1A70PmTR%9CHrlP~(Ce`u znV{6SC~lPPb0sCBma;cB$=3N{6;=iTnA)X{f@@ruF4oGd!5TIn0_yoduyh6O{OB{p z+an7~fl)67KAa41;mrccVQ}vXyS-TTV?KqAk8%F(Ek!54j>^m4<-zs?>LdA4zoSq@ zMv^F*^Gwg${<*RDj{#%QvN1wUGsEZl66LSv3@B3w$-VP3Z3?-PIH|uz^3e^#_Y^g0 zGNcd#9`E{FddVe>ZUFqq&n%*0&0iUP`&|Hrh+k$5RjFt3WBMD?MIOZTu{?krKNw%t zZ+HGZ_lD&YT7uWbA%zCKwFZ5i)045cpG&OpN{{q6KhDT$ld3<_IC^(V6Q-EH$P`4Y9cj^(QSXC{+eM=a6e!N z`Nc@=v#VsJSZTS741^+Gt5Q|i{KIRGr)#3HfxMoAJwESc{k8KWETBFlNiq|c*?Pso z^wkvt(Q6z4f6%_;I(Mt4nDJ{w5Qk1g8PUHNtR~(}rU395D3+dUT@!XeutZzw#xjSI z+%)g)Y}@9J{gm8s{EO5q+n$1GD!MH18xdzT&nI)O(4HmAKt1oq%RTy150RGa#SGQwsxiB_bjBD95+1sHwuU-yY! zfoi^2xCK-Tz!V_OnljhQzuK%w??{?Qo8^w>IX2zO@SF4qn6dgM`qN8y$||D_9V<$m zx1VnH0qp*`()V86`&A5{eE;7VLG9Ze7Kk5sMS97|`;e^lhEZnR(Ughk@kTWf&5oS; z6m+uERmA>=5Hf!Pfi<&5-J@EJs>sX}*HbBPu@p#$4o*bjnNjACOTJ>mm8^`}N)qa` ztibZL*|sn$U7zT#NnsiHrRGoydccz*5PS}kdh*%@b(2Kb)>)F&o&!>Ui#9P*MF4LSzAmT_l(6jq%{WNo9z53No1kRUz)eqDra~!4S};W4@VId z%37p&F_Z2!|BpkiP+n!Qm-KggH>wkh58%?6D6J&2L`PFi`gzZt+`d6t&7)PVSn1;AwUdc z`2#mic;*5kl?Q8UY32hRp=eCs zHVVQS$K*}$V8y|9Lw}b}Nt0a`1y{B0hvJ7lHooj}3OG&l>9|1O~-$BMe8MN(#RRGidv~57<|d-%iGMQq>`&E(*8zhYfn>D_b(Yzd3~mZ z;)z310zRGQoUh~eRw1@q{dEEn&OIcdfsfNfAYeC4~snVLhSWvtDC^a z4X7zTfnqUSlw*q`UaTpFGSCX>_6VInT~tL)5#KWQ-{7uv8YFi(`Fx*p%aH5g+C*c_ z?q4&$H)2KPA(UdPa>&fSFp5|kwTz8-j%KN4Vj$Tp3PWTe)N8`85g-7?QZb{Yi2Na| zFU9$KvfJPpk&c4Ix$gzZ0*Un%qK%?G(8W5O2q=yoh+@fz9{dj=k!vNAJGdxExF(uS z!8#`kn1zrR+%ykBk%nrr@Z3vwY&z7uKg7QkJr5N=(ve+27V=MZSqbMt%v-t8`dn~4XzdB(sEeZ>Vau zQ{O}ae5dEi1$CkXBJLE9VzI)^7bioVFHvZvEwFRr^4@?SPyI`e$M;X^xbDZk1(Czw z0qzs%eld$!q4EC2fm4~g-33W(f^!RM?vR94q0dqk->t!l%d2}YOG2(7`b{M6%3fIC zkL>RB)U;mc-MEOXH$la7fb(G}EwAAO)xN|~ygVOAk!i*~$E?HNKXi#;OGAbCs!S;) zqKI4p(ejc=r_9WMJY*r8J8WK*^k9x;yK>?N-4iJ43c>0w*vAm%Z@dq1T`q8O7GRaZ z>l9G1_D7ZrKN-5cet3~Ux0l}EEefyQx7}BY>|N|Ufq4k<9>?`k^&x@ohQ`?(>U&biR;Mvla7Wl5n9dfezZ$*>a816rU) z+l40(r1`jeNMY^FuhKj))lJY5Mmq5jOozaG^tMM8WZAlX>kE;ATQ%Dc(*9KA7TL;n ze@&7vxucpGuL)tsKXTwRfzErO6Cv2P?lp9v18^BqTrOqSBAzkn17*_VRYQc$N!3Gy>RB==7Wp_UEKbOco+i$SVi&r@CKhQ)Z!RP(Tv!}) z6%Z(ie)`lksEX|O=MNNfM)uX!m1^SoCrrPqx_q7B_cJu^Eu?Nj!d@~Vm6abz=(PPL zk!Wdx4AP4qey<-ZWc%Gf#z4em+8Sh~I!Vo>X$hR^8|Sm~`HFyjZG9ke>vHFPr3FO2 ze3-I`(2Re5wfbd9Sv=GNqr}0_9@m%bKMR>QRstgk&tnZ=!)G*oH+`$-4%4*C?%Kjy zT|5M~65*$iohz)DZu_cZf^kaN43wUmMuZLQ>`!F^iD4^wwK*9IAo*MOy+S095i)8) zmN4cjK3H6oF5Z})bDF?|OfBEkIDD8eQO(r2WNvyH_9LTnKL1q&Tv}s?BM>a_MPZkw zm_hk;rM#P6^%m0`?AXK-Pe6=k(4C6A<>>Qh!hCmH2Ej{bS@4Vuyr<3r6K*2Cs_?|* zi$qXL^Yn|W@gLC&0qfOvX}HY%gX^)W78hE0D$2Y-MKrwgo? zv*#2W*DGjk+cVRs*5X}dW`FLTguU3Iwdn(OkH{fl|96G9&C&_3kI&Q49;dkd0V3cY zbDc2bYn!d-P#OI2+CxoPdC;WB!xf%GCo|5_>uvpH#;q8~TS%zC%}&}oiYo)jpV2j3 zR_4HM*PzoHP)pB8DIj_MNv1^J3bV@aHwbV8F2QoNkrhoX19pYaVVS%JK`^+Px%)=y zyjuioj(BA==~@tQN3A)&6qbcn>Q@z!ejRB@ELl?p8&u_~KDd{z$$Z^B!2Q_=Xc5(7 zdunCI<88UE5@~4JayBVdPPF>6=+XrF+ApE$vk$%=i(;=St)fhAIt;+Aa%(5JC|;W2 zLS}SKzvwqhn?b9Q0lrqlG8yMWDwhobTIyS6TQ?JsdYQUJTD!mm)|4$!KzyBKTNad+ zMu}pJr3iIq{7A0@perJcy^Qq#g_h{1!1V&{1%>?cFJ4OEM54M-GeZ9;3R zj67Jq!36|vel8jjiy$gbmOV|Le_Hq%2&B?XNglRCV4Q}l4>-;Zfl9pOAb2wS!v8fC zMUQiW8;Om0K16RL)_CuTM=R8<1-Dl=DJ z=tt5t{{M(6hRE&py)gBOq@dWOr((vbmqlBcQE8d>6w9ipZegbPHEOr2Rga&UPEgI= zp$`O4qHVIFR#Ji#l2KA#SelRueT_TfMpBCDwR)9#@4cLV^c-Z;2?3?D;KM}Y_DpTa9W2U)<|K|@hzC6y zsl1U?!bax4Ifk+O|7HQizTKe*IbY*WRU&uym+QG0urk>(O1$a(WsGHRovg%g4s*(F z=&L6}IE0NW8_KVLLv7TAu3_lL4HONXztR<4&w#C>g+et)z{%wJ{Sp353a(M-3y>06 zVR?Zvt*a<`^lVYydnuK!d$1 zkLH~-f}0SyFJa!6{QtTkI#hySM1<0s^sKm zjrV@^h1`H;>@85F=E43v+s&-0i<#AELv|}w-bVHZ6`Xm!Xr)eefe&Ub`@BWhdWs&XTn*3sP0TQfHt`j*Iq%Q|NkI4y zE=mYSkny|R2NM3M33sYx5<_g9?t1gm*c>Rz8ic(0Wbv0fB(3dNcIIcm#2@hVY6Tf# z>zOnmVH){kFP)gE^rt|fV|aAC8zj5JOtct$Ce%D}Z+#Q|aDn;ht7)P9CL{s2_;!Ip zqxLd5AxU;r^%DPu9!{Op2RVKf$YkFOma6OPR_noLPYu)YlKxj#0{U0B`mj#_>IYpo*Nw5Xb-BuZ| zoksAU@frb~$>)f^PvdgY6yzjL^m$4b`hjlZ z&Q(BkQZmprzMouSc?_=LXYf*Bp;&HKUQrIjnjDZRou|*+wCsl{{SEX{Q!R49EpAXB zV1CM1jq?yv$>(d>2?=B2m!VYQnc^2f*vrMg=2O*3*WKGk|3SI#VDQdCmh#%3tH4&T zO3$&K449ay_wFDU3?;AO#t;`SrMf1Zg;z%jRvrq$5hMz@8!}f~Zr)oOAAbP_5Y!YA z7pf`DMe475Af`MyB)&Sg4xU^jgsm16HEIaJ5>a19?}QQ0uja%MnZk@Ke7jRf0V(78 zK%Mi^)3^L2fGyoHi|>B>k1$%SDH4N65N;Yt>w!L(W$tll!atnAYsB>;bpbu4Sgw-` zBEwL`?cRo-jBZrr&lX5*tJAfW=jhM+^wV!!jOidcRR&R*_mhz4A|ea@kiy0o1d&W- zSx7a6P$u3;J?JVe$kXpOse!O}$c+*~L%+RJ{`r$f1h;R?MaEwFJMn+8l>l}?2x_6G zK0TmmHd91)ey78^H3HGlOwadH1hHIJF=Tw~a6Wx#rAM_=cC4;C5ohT0G`%-YZgi|r z7@PqW^nF|MFrk3mwfmXzJwR2K2u<*I)>OUJ=fTR%W-Vqj{vsb9L;GGq`d9qA;ShIl zb*hOpnz{dw&Fg+1gw_6+gQW)Sb5!1sd0k~VHRIu2^bPboegX}?WL0MI;G_dr10S0$YV zoH}o;=sLN;8F33xbbfscX>w`Un?ZDqNx*SDnGQv)3?bwq1`y!_d$r}BLLz4PvZEqE@qVGlZQk$0;6{PlcQ{2qMBfGeV`D3WEL6%p5?$jx zy~GcDcWYiGZtjze;3QxQ>2ytrF_D+rLtBz`?_X3!?oYlk1{e7pig?TzD|>g%0njNL zGkQOfk$k~~!QFihO$2O+)PEUa;Ut23j34|Wo#$~`#`-6@ ztDP)kM6Y6<4zGTEi~?@5D7&C8xIF=U9cYw@VUi*^} ziJbbnz)6QkQWN~`yNFdpT%tn2M>Fi8e==%vQYz%DhbLkV+80Kq20bpiTPg#z|Bjml)WRxL#P? z`kyr7F2kES?_R}P!=jz_J&!Ls!9}em+ zR=08zm65zXG^$t%y5TFI{*eJhiV&qNv_B{`BHUw%g!RFi7S)$Zi*i+1kn8Yk76qMSRZv`TkzdzrZkipR>>2 zYhBlStplneMguhVKE{bggKeZf^D)r<^MSaHY$s{CJoR)ezjQAZuO0G823Ag;zx({b zI-Oow+50C#A()q0;7!$-)6a6q`>49MKddC=SfOZ!3%v6X0T5~bo`y>dVg8AD(ZY=w zd4{9VDhDOY*r}c9c@yFmrKmsyvnWs5Z`>6NsBu5{MWQDH`%L^JWDIvc=aY!oG2fM^ z)X!pCZh^1Pua8ku@9L|ie-+n|&X+-+%Fcg{I@++j%n-8z`fZzaaM`^tX!t&F-)lek z+!L3FEnCFSC8FoB+o~&Uu4Fg;_i)kXPy`O>F<>gxW6*l_{q7dAe)pMc{geLoFW4v# z1Z+m9t9NiRvI%UtlaLk||JN5FLzLook=1`3p5Ov9S6qs2PoMhMzp-_-?I)B@2b9Lp z_WNhYPp7}&5k+(siJyFlcolTbT+hi^%!V=iZOkYadVQXey03|vGx$wkksXX{j|5V& z@7D`e6( zAAcFL8$OSA+Yv_RVcz|i`2td#nc0BhP3R-InaTHb~iTjbC)AFIB3eR)X{B;udFk{8+po6 z=I*UJ8On{>U@Rb-%jUjDDNACY79u+v^`F1dTp1xX|XavLaI zV4*%`MQ{q6W8kD&B-1;H^L^#lF;FG|N&FNTp~oltWLys+VX1I z-@`XM-t+qv;BE87KTo{ATd>rAOCwOLFP%2Or6=$P zy}F}}@?_AWle`z`R^}VTaqt=3p;UyaW&~dpRsk!(IV4*_Dj6%U{}RzNph*Bpn`Hh8E0~-Dn*f`!ODw*=>O*H`d1C%Yd+$nQGp{{Vw{ znx#4u7(i`pJ$ zI}Zh*!@r;3rG&4T7n34w|HgCX820K-#9n%S`)6nUrO2T(N6#kRYZ3S;`iaTI>uBuY z>m}KYE1Pmc*a^_CO9A$ilu`sRT^$i*WsrDMSqe9$lr$S zUI@JrhoiD&uZlFlvf!aUtapq|qsywBe{%==%CNX05K$snVYTl-=9biqgHWo($<_^M zE$TKA#d*EobJdqOHrhz}eIkVsqLrG-Ex$^SjN3kkp9AsQIrQALqzGlH8Km1W(!F?PtBM|pUf6#U z%jU`O#d315BRLq>>ukXl=4w#_V-zrA-1aOsT3_<=sD|@^L&9@WQrIN}cvkv~p#6AJ zsw!}f$*2^wuQI=_b`vmAX9RKdZJz<_mI>dOFXnSVlMx2QxuGdo@Tw%-!q5W!0SZHH z9HVUTc#@VOgTE~V8_g(pTel>7kSUB63`zi!XeBJHs22$jXI$)dbU+q<#Tr1E00}u$ zt$^$=_B?^YNrEISY^#RL6WIk;4B*-mlnuj|87#$5;cRa(tZnLpdC9ZG@98!UzG_k2 zq45B4GoNpLsrokHI0^aUTPSc5ofN1bjV4RzT>qQO=jn!)dv6rlD>1v6%#~ zHF)x8v?zOVHPP6X5H(8b=|@y9+7501`FRqZ*}vR2D9RdS4#heD+BIQe9m>wIN(-?; z``%Ln{F&+fuF{Qkd&X7{{I~>F)mHSzjfd=UM9ODZiWd9Xe<5q}4?9~9$8em26{BC< zk`q_HGNI=Fb+i-~%8-R|P-iz28;Ru%%n2phGl<5rke3XzS@2!@80WAuvl$KCp(bxR z&Z`T+!eUIw>U8k93CMS$`wSIaShM&tv~M>fv-zP3706S)|Lbw;a4Nh0( zOiKuoB>gq%3fHGce~AsC3F~0rxv#u43^h(zs=I^ulV2R!UHpXq@zQ6Sj)A4c zDr5nNLB_Zy7~kfcUPKiaF?izWBLfL-%Hc3OG^B~kn>9jFIR5IWGQy>_!bxSRlGIo( z(v==e3v&OGIk0pVAYqJDwhf4_jIclE96xIADVP95Y<5fZO(SRr3*cz?B_Df@1k*it zY_>wi>Y3fnwm@K2_G)|!bNj79{(_#iQ0a6H%DEZ%?m3p;hVnm{;;{Vr1@zAV`8L!y zG=%mI#)X`M;@SL}Snsr8DNj{OkFkPG@$?B9Z8#;VTZ?ArCbI z)d4kWD}xDL@J(_n@*mPt=5>&k;#EC|j?D#>B4dZczGD==CxopLIdf4XnqnE9^Dm_e z2eDH093#w@(ypd(Buyu#sK9zQjDNeJJ*q9EN{dEvq2q-pJ=Sul@r%Z6o~c01x12gp zF~Ni5w|Y(iU;K7fuuSwqpXTR!DHSz}LXou{F51{Pg({93cPOO@3URi61L1e^C{^tV ztXGjR1JX-HZZ|wX#4BU#hB&<)&`cm)QdM8qWV6TGBz1bAOS$7$YMb#k&ZdtXdu8GL zQFh4veXI_5)>Y>$e;_D$DbxB%z;MRfk}puEqXT-}2bj(#&jGVvxi%-UoIIgyrEcC& zH0h+yz?+UR#IJH`zx*;>RF^AErOj7)L7GOPq zy;A!EXs4o$e%O~b@{w**uaPKhDVjX9I=c_<+hPjo7$9ql+8jA)2N%eJi;@N=0>xAn zTI8?6gd|E#FB{2Guz~_>V_|Zd`iZF!9wr^e5NZ#@9|4iG2bG!JFEos{rDuAdImKdLb z(f~SjMY}x3LA+6Cn;_}^(kG~ip&|}bV1{W>TN*Q8K>5c;1WktOUQBx0vi{1BF*(=E z32>Bh$k|S(Isto(=HW{ya_zx4LDgB*Yb^={tgvJV4nxtK_R9351P^sr~_~(j@LxQduH>jugNCny`L6CWx{bdXah3H1PBiTPrr)0 zn4lp`=bu;P9lw4nVN{1!*!MKaSi5QWv40?alAb7Nkby<%0Y}NN+;~8)lbfj0EQ99o z)8|;We|~&6Z~$j{446m3!YOXpmh({W3&pd`RUrh!u=9vq{fv*a!AgLdZX;*IVglVB zGtTpmPL@`g=6bvM72wPVc#kY|%X`LR7rU@EfJ)2+MT}g-gV6E9WI1lY+PR8~8RlCQ z(xLGl%3URRyS2BVc#U=9eJSe+oZhKG9;_ny4M>w3)M_?dK-K-xqVWP56?2u82dhq? zgVO?3qMcEu3Je(2V-zR!>TMvox0NT_y`l2t{P$p;1u9Q`(#Gme3=YVe2s7 zt|?#+_Kf40|5_z0<88*F!|yVbF!~@l5aGqwpfjavt)o!Y=+Ms()-D9Yn0iB3FsDYz zmeh32$QYb+#@%h>NS8RdEzI@FfbO1K=6}xwF4S^ir!g%tF1uQf*?uUFp+eAT4n0uP z%$@-<=yjS0PN+m!wx5Ai z{`yN24Nt%;P7)0KHVYb@(^h*&2e;c9#z~@%(Tt+tYkB0V-px_A#pDxb<5zAE zQ2&c`OimhtI*2K5^rz3nSi40~Gm47nsFWz+AWT0q=#ZjLcbft%m4l-0^h%Q5YU}8w; zS*Q}ap^Pr7(kvMWEQMh!2Dn*cT=GqA(QOP+T_#U7JuoqQ`%|6-x{}wx!=A%8=!pBM zLQ$H5)yipb7*=m)Y!uA3fPUb{00O?1t`UBjMo-rYVM0ybfM(PAfh?#>N5QURSj6rKlc65P! z#q2|L%6ziTJX}WYN*6kCd%6N0PL(olU#r>wXi<2lhL-JubW}JVP=@yT7YERvc!Fpl z{`yhMPI@g4%t%DSN}3lj^HpyG+#PO6uzpa?xQSuO)n}r{z|6+U_!dvL(>P62{wv|T zJI*@4$l&9nt(I1L8K|O`8HrXa3ZV$FU1pZUYNs`JjADxx!?X7EGs&ao&J#V^*?O1A zCxYRi^E>(Myq_$R!cCa&@M+}$nm)IAlLMCQ#YGFX-hPtbcIW;gtFNUD%=S;Ri_`;t zzps(SMPsu3T^KLe72iZ`ES$%0V= z$+tTeXG5Y%4frMg-cEdJ`tq3+YxFO6$XW(wFAh?>&OJSMI)qwKOIL`Zm}cw~JYCRp zt#qb0?ZaWMww>T_)?Kx~p_c9ok2{EV;;0&19J=&p$Rb;Xjrr5JXt}$VR_th60VmXL z2oFtyx@i+M1=YcT${Ry}%Dquix-p>4($Y3P)Q9&t&LoP?S=G8+p$F0mh9RDKo-%A%iMb>?_hh z!i(QT(H!OLm!<5EUx}sB`SkZM0@$+EH(=0)qzEW&|h!Sv0Gsh zR`g)ObOB{8jZ&2bNlHL2z5#@Vy7*qV7^`GfHOq?&6;)7D-T~kFNt-8Xjo^OrX)vCs zu7-Ue4F+7%lgHgDziYqY;8)WPrk&x{zD(0EKthg@AH<{v2qA+xQXyQFVrlFVQ&dpr zO}q`h-;2`6#G`t(6|DQ@t6dfU{R z?&L}Yj1T!{?akvxz79C}h(NO)MYXEQFwG0{dq}sJFGATESwf#+Mbf!S9>rZPn$+db zGYrUC9D^k#t?~eEpsCD73ljq63WK6NYK$y4N*Jm4AtU;=WHm}-966jvu%hjXJ;bli zqoeDDpHA1I$)(QmSFo`IiMN<(tSn2*-}mO{lKZ3M!Cl(mRasa`Op{s?1?PX@qUiFY zC;?|+qrtige295jsv_@d9qKA7$d0S8MvbNf(oN`9@81ijk{@wVc!r~Tt2fXTH+4QB zp(H6&6*VIPV65zM=>_6#$CFS5bw^6!?(7P?dxF#{$X6CrzL?o74gc6+cHNbXmSR^+ zZodFArQBTttf%Jh;M%_r;;;Q9cw#3fww#i6oYm7UtdhRqHjuU@KsiLpOV0=h?8{KN z|2m$6EJL#cR7m4!xz!Vb6_Fhw81+2u%15Y%0%!=ycmc3h2l1cOQj27~E*A~o=WMc3 zf&b)W`Rees*s}8#4Gyx??&G9_26QMqXj&Ie6Bk~~AHWchU<|81p42b7WxcDsl0U=e zDdDJFgkE(cQIzPC3(nL}B5Cpa>%M9!DM%Osd1^y&%rwd^gE=XW{Nh;$T>}MP`-+16 z$^QMf^=_I%Br!fUiaZnfc>yQ5kEMwn$})h;2OJIw(jeWVPk8Hu>XogUxa-x`#H7;s zmXHk{8kIz$2NPK{Gtby5-C!giFhGh*{`5{A*CSqWo~##p0I2!ZY-rfq$YV&9=Ly52 zQ0224@yAssOp3mObHps41(bvi-(UnkO|lMPT&x&Ps}3bqet#6iOuqvv z8~51esm%~GQiF3HI^cxrG#`u;JSF4`?DJR0tO~8)HlPCWU=E9#10mGt?1*GPd_utu#uXT z9${GJ`fU&zJ{kgkOw*%jpP|Ft?@?!A$+DGx1zpXf$GW%Vhg^-5RPOp_f+RuYOj3nF z1U-Sl-VUvb;AKNQ{_q%jF3+GDO8QwFsR9!d&cE^y$?|AqR`~+qlu)aNn2iV*v<@9y zoigEr2|V5c3Iln!ay6zU%=gZ~wxX&-f+T@iogW~LZ9j-=AGpCv3UoR=$+_{+Yul8S z^h|TVg3h$cZbo^;!pt&Xo6U?bFuzI+eOfN5H24lNA$|E?08$W1ifewQ0L6Ql+d>24 z2I1B^l;Smd3*lSMQ1`C}=wyL`)1~%HLQn3vAH#R1#YOG8-$+%dQ8YJT=oCmwHsf>BE@s=a3_ZY;aQ$jLPfS1B^r;m^H``oQT%^$bL zwgN}g1qkKK%nqGgWt;k)Gdp9)!?{i4Y`sqwjND#ViEq}e*C)-$g-7|Z0^iE(t*4o& zTrXq{J)^YIn(8)b%8elS_QSP!lau|!d2AM5o}X@*F|SPB+IW$hcMT8Iz1%A+RekKX zP4K)Ak-qVHHDV@txKNb9h#Wpo>0=b$42TlU4K5C8wnc_E49N0y=ix&v>qf;a+OhDd z^mc(aUfkzF0}n{hhdD5&!SDrRELN0=^aH21!nINJ8Yuo2!YOSGFPG?!A?Hx*&N{S| ztE3m^z#mzOx3M_82RtDlzWQOmBU3eQAindrtubQ(xQI1%7~L z436AfLP=KM0o{XZw#I@LMG%2>-7F)y#p*7ED_zqN${MWhIh9pk81KwNZAOi^#jZdG z>qiU~(}A!n@da@C1ogF_6CnB4LoDbY43Y^3@C6+Txo_r1eWHKIQ4?65nBxaq%7Z#i zDASAog7-2cqao9WbYJ3!%()w>^rahjQ4N*%nh|-GP9*B=G0I(%%Jml$70$&maHDuH zFT^aaxSd1u%J$DUkXa35?nz1%B)pbek({lp%-*4_dCRpb z1S_7aKYxoBYaF}JN7?~?j3H{;MVO1oL5k{%5Uhw-;+)D!e;J>8l_|u?9y2oT2)p?E z1o>Fh#EK_!(SH80v-VwBN;hIiIy3gi4IQ6voJORe|NP^O-X))iB zy9+6^P#*%Pe)Kr2=T&UQ_0VwqD-z*qM2>z=7btA)Ri@ZM2j_y7d5=iz@4W7G@_l&= zQf)>_S^95`ya3UuKqWfgMqMn9ia<}E;cY5n{M&$K;}lYAbfVUBp;BiQ2;?WNy#!GmE;TfyKxxmqx?p zSq4uGNK&KmO9oS=-AzOV)C@A_7$_pgDBkwhoz@hA#2fU%)9+C;AdN`#y2>U>NvQX_ z!!Xie-Qrl?hLci}zQVpNgNP4%N6+UH*!4w|ly2Tbfd+g;?~kyvZmKxAkYl>f5ir&3 z+=ZGJ-iXAd$KY)4D=GXv{?HdHgB%*~W3xuY%#7{wI_&aVvz73yl4QAd^FD(dA;Xso z)=j@}?2g!prl^PDH_}A-t}n#Xc*$b6=9Y`5xIel(Kpb$&(;mJAa1P`S zx$PscGM@XL*Cthlz6dR*Mlqjo_EYAvZ*s>9150W$5*zBWFtJd$lqf6Z}J(8 zX1Zu76#c}MOGw;y%=0Z>HB;Q}YwY@TsUoyMXd21kR(FfYGLvm|4HCr2Ug9cQ(9?iq zqLFPemTYX8_&|X~v&7P0fSL!`k#vf4A6xbi78RZ9_zJ8n>C`A9DRg+^N}eKQ8K~~q zn4E5Aq_+r=T&?9qjSL*mjjW=3pk@;R1Njm%rI_!XMn^~}xGAuah!WDMbk>Rs&@LY$ zNReNB1JXAQe?H-Ye=!9JODdybF+#M*P^gEqgn8ieYW~O+;b*3Ks+M3^SR7Aw19_Z) zIjuGE8#l=G!HlGt`Lk8V7FeiwGx+(SpoMUQ1oP*JyQjJhuG~3Irktk(}B@*M(=L`*m_T9()LEP~pm_VWzJ2^4dt`xk09Eq;F|zl%p9Ox5JGM zCc*ngIz$O!Gl~foX1-nSmg`I(~wF+v0u8aDa^5}aenz~w=XFuh{0z_c@ z_+;@83;JuCkxJLy%7Hd=sv_dx*#s(|1Z!%@#gppD&W^!GubVyxO7v_WqR)=mYAcvt zMWEH10~_4vls-{NhhH&)-f4*sWQa{(CdZ#I%2+irR3}l0?ou4eSOERt z_Y~^s#V1k=k{L+v#|h{g-HSeFtzUIixOognPjF=C&|kj2?0iOrcY5=L+dugZ11d08 zg9U@?k_t&-;79-9z(_6pBh=G6gU&On+h#N>uEpD2hNH5@P+~e!a`0mXvo`tU7{{kW z=gke^A#{!4;u#$92*wRF9S)rCTL&wbVmT;#EAC*93TCs_L)&qzmrB;SC?_F;77CHH zd)X>@CxL;Fc*tj#5et4bz6(ZSphc*+g(cuB{T&?Kf^ryEgKf><_X2baEUdff~1t?cFTRm}_1VQWt(`#Qx**{UY{rs1C_H^uRfmL#4`>;%6oL}*)Ozh$Rrs;|*dsswr5pK273X7q0yuR7 z@(+rPd*g87dmvaG-4Rx=n|Q}nP#e|x5U2SA_yDQ42vrrb^Ak^=nJ74yNC`$Xm9sD~ zW7>ZF0v+#W3uupv+3eAX(aR6MzWlQaF{|V%K(S`;?fQ!73iW5>8F?*;!AJr7b4ZpK z#V=D(7d3diHfB5MP0T|cTCeYy(38KarQ~kL6C)1HD&~=bQ=* z3DQB5s#aIfhNmV@yBm&8dj8C7(zV{@s^2*?dO=q(&E&~LnB^j0-Tp4DMx&htQV&!K zd2}PyNPxkut$6AzhL`ieE%#|_C*J#56XdV(!kYU^m_DI7;bv=PR|7yqvd~_ZJ_NNA zn2K10>_%8vJr9F(lqKcM)o*o~8FNv*44#%rnHB@eX()E{B)|bG1|M<*l%;UCHFm^j zVXM^&lWf*%L-uqre zXDS0{#yMmgRlGnGOf>bIa=0ObWgv);lNh(X&Cn&C(=SOcvc3B{!;*=X1FZMa7*qLMNTNYKz@O$rL=%efREfETT`9Nqco1gLfz=_S=f!G$o!YF>&6ACYQ5`*i|SXeShp|D5w`p43FRr@qst{R z&?)A)&o(z-I{<-35XOSQC3f^z?xEpMmJS)>5B4$PdE1$Ofee+ z7|42rl<3XRnYw$Jzb@Kx(iAk58N);%C#NCy@r`o9^i#?7^dWNR2wrl%f8+xfO{YkN zHbN*WLi1~F-1yxXH>Jo{(G1uzQq|#5kSUft&Ul0j8uY^8j9YYakY^%z;AJg<%KNtL zH}p`@oi$~*>}v=iTBN>m%uC75_}{N;v+c1>W4}`75ItRtQ(+{G$w%qP z1k}U{eq39M8Q3n~$XWK=7CV+yjT`^;0PIi6+T3`;?EQ_fdSSeWMXH2{d z1|uIj`j6_1CO9R%0MGT-EP#DjW?DDmYXk|@pfaQ%lHLsYP%wT*RN)kF?X6N|M$g@?ny9S?Z|EvRD4+ zZdQOJY1TzcH=W4lf`ixz0Thy}Y_NjYO;|aH8^8oAf$}&^W!go;ioYj}2g6jUPN}Jj z$g2r?sqzqiw?>Tbj>Ri>$=fFIDK}q0wzxZbIpq7{1OxEq8Ba_Zt4LCCDNXaF z4htgCh!0E1*=7AyAp8e{iG&MjiLSLo%vnN5UVbrYeK4r-)5$M-}os;__$ zPQy;R4mpMEiQwE~VG7ui-s8r(YOI8`}(Br7u*ooi;4PI$&k+H)BQ(ItM{8Z?z? z!@%hV;pMLn_}_aX;4HD)#oz7H!Ao(7=M$mpMM zJ~ZWLxKn$6uvnzn5zVjMifR*!DsPB1n{>mGazkbEBs!kWk~FKymvT42c&`W9QfoW4 zhxZb`R*$o6Z1J@#w|}y{BkeVf2FP9}^=kY_I?@>e!mNB8a|R(B|&MRp0!~YtB5Vu_u0xsr?nuVd!K&8m zO_dj())eEtc5c5SQo8OjDW~SZTsIGSwS(`Jatl}5VW?s0_AF%U+-04pMo;|7fy$@9 zQv|n_p6eui5&Y#?HLhn?Au#c@wbrMf{(U6R2}ZZ_RVIz=2C(!8ECi3glw#^&O~{32 zwaNl@H;*3w6>p@Bnd+I>)DSvwgp(bhK!FgRK9A!yw4?LJ{mwMX<5IKD&hec0aFbSSp> z=UEkia&fzGIwTs<7NM2!zW>mcM01$#0~GElXoyhYrI3h7w|eu{TCheNoOWdA;KcL1 znfY_~!|uIDz!4{Fa^x7>Y6$9yC;dYl*tvYF%fy>~|G8qCbE^?Xbi|>`W55x15chS! zQN`-NCRN&M75E9-@I1h0?mI`pJcu4^Tw=Fo$65Cy;gGphUAHGLlw{T!Ac$ERD1GV^Rh} z>%+f?r7#yhn_gZlax<@|p7Awu7en$0Hx`W>=*b3u^?CCH<>x`g!MEZ$i#u5#TH2F43|CDUnPawyZ7=Y+*6 zGffdu5pNcoG#wdD2g>p`!NWVMwCnQL*BdV{X-P8%4nY-`g;+Qp@jlg2Dw@jP^EX^IJpZ02=L{3D~Nj}mb`dK~)qUB|VH&GOwGwL;owZit1 z1c4nz>9c4vQxsqty!|o3Lb(JhQg-Ul5M||4B`lxKFweXL5CJVD%8WmzHv6ePU4o06 zW6_9lQ4LN}NBU~tVl#QBajqTl6HXJb)aA^5FddB z<{SwGTp9VGeP>S*I?zLogIV&smC17j&(GpE?v3hx^l1)KwRdZ1D@e%J~zIR^CUJ{V6h<}q&=gywYA5|EneT-?sn)6Tb)@(5Zzc| z(On^GuIcVsEmqT<_`!GxeJN$(bCH3ww%N{FSE;P%LX8ZuFG_@1Bix2~TpgS>f-nAQ zrCOAbyeBhir;CKAPaF*~*-?#^F&Vg_prFX>lpEz-ELS+o6Qa4y)WIjyg= zjph1=^T)K^U-^DFhTeV&3GyHWLTS8*Ux_~4;84CMT!c@|r<|+OmwK#@n!tTeu|5}W zw9qNx%L>=_3rp8fIc^yRv$ei@vny3n7?rN_O=7jw+B@*$02YiNFE3*>anX*-{YOyJ z+*(s(I?6fofzxqx?Nu8~cfujHiJTIDBaa5rXqr-S9@9DOKU{T^rB|OxJY{OWeQvoj zGu<>oyLXhaH&=N__4&RmTtFi|vnjBX6>apsD+A9lGc~I{HIX)Bl3k9U&Kd2JA}yG! zB&cMmc&)TJtVD@<@987j;k5B>B8A~}C!A#isZOqGRde5G7B#OPO7AO2K=HlH&0q-M z!k>D#hw+dM9v2Z~u!})I7e9?;eX>3pF#JIH{g7*I=kSq9bko~GZf4cE@eJ#FV*0ro z(B(9s)&`TCge-FY2J=Zy@3GMV6*sZ6XA#@nuN7)lMk1tD6@ZOzqlipc%R3fw zot;;Czp0)E!)tb45Bk8Un_eql0b-GjDM-Z^8DBRI_l5rmf7(Hd`V!_ zpN}W2yH*Ah=3EzU&+nTgrYd1$izYs_ttN~+R=ua}vIZag>(Nvo9$~7r$`ny$IVLak zW{0!hZ@dzH!+KQAoa2Z3FE%^&O}Tl^-bQtQ)Zt^KdWO|f`FdZdia8j?jnm*zfvFCgQ zXNJ0kJn!aNHE;wwpTyt?hWQYsj&`-GoiEUBi&>l5TRPoLft+;^hIG!Pco@=&$e(XQk>e)T2|7!aU z&(!TJ?mE9GZY>@@mt7K6Jn%WVdF|*_&y}4C#vb{u=`^0H?)@K60`?OBuxtzM2_0<9 z?$NogzS6jM$gh6w@L2byC^~008**&bg;vg}qJW-0o&)Xh7a6x>*PsPJyT}uHA zdc%*upE;QNxv-Kncxh!p;Nb7$scRXou}MdIqTC&a?=A7n3#Hxs{T|6qf?NAaR`_5t zC=KBKZ?!}}+pnU#_50(bh;Gu~{=bpkO^5pjyXDvFcK3M>hi^{&o8%!Exit}a@9LjH zp0v~4Cw~4b#8dKX9=_I7df~wF`$l-|XGQK>-QO`g=6ly}O}PJkFnRX()VeF}Z;Z!e zHxA*I5JW027|9&cY^`!7h`MvIPCqmbTQ+uf6%>1o+q3~MkruqFD zy`90cJ@>SaT)69|jp&|#-^8ef9{<|^P;+-&$ATXfxum8u(-z%71;isTy|I;k4 zeZo88p~AJX533>h@ukaPdoR#0_s!PxGmtxWZ@->t`$vQBn@}d!o-@1kUX!s$=f5+` zO=xudasP(}yZzLWgN1&(B#V>OeJ=J{N!FL;XTuEJUTEm*3)BHGba?OM&PE#7|3Q10 zJ;Ir~OG2>6HBHz<7ex!02M1qs!Aq^a-N6j>!6#jCOM`xU9(f~rrzCCREZAG-gFW8S zF9!TZZ-aZjn}Um%xHH~yJ!Ya^5M*9-x^=PqTBH!XpayNHy@RfX3(yoR_SP>!dp!uH zDOvU8@YWZT=P2FF$d_GvvH#;04!*Ut^tb8KQJM$xkvCI9+>4KYqg!Ub|BX0OQtX4< z9(05~ey#xzH^vs-wT1V%kjW#Et_R)T2jp|t4kt`d+fNh)xfk#Kpj))Ph}v#zZPWs! z*T2`CE}{0m=Y#d6561Ad!&Y!Gfkq#EN4V;#kIw)Nd5kufHg|5`{voKyx_Idl-OCPv zMGaSU>|ub%In}oX=uu37?Iom4T~{(80gip#0)NhChq=Hu?%j z*cryX5<2WgQiv0@98J3oZe76b4eCM%oybH5z+u+M)<+jE-raeI4&L`%h>vXo$ji)| z2a@(AR-a~J7Sqmwt532(PqZBTPILjQ=NbNd^Q7Y2MGfI2%e&E7g*4abuCF2DF;41x z-m`U1VDj+E{iSsay(_Iz2ZJ*u`?117%pv=JCqbk-bf?^->-mL$lfvQMbKSpU_WxXY z7&UsY;BhA^#PI*Lo}}+Dsqy^1vbw8gD?>f>uXyEY$Dm|0)1ON@)Hs%HtC68V=nFmV zU+(mkq_fS%hZ4R%Z9oqN^ZpHCJMy#a-K95IY&cnd2Wt!u$^6QLuDbO6VL^72=HZL- zi~RwFfH#)Uc34|mQ+55|2d96c7roSQv5WL&PwGt`cTFUvWjM|u3mqK({J8VdW6&#t z#fx>XMKRj`?42s^#rGg~gEo3G>r2m4(eU5TF{*RQWo2#a(E9&AMWUfUr5=$9^x=94F3@TI24(g1 zEv+w|kIU2EKv%o>*0aH(#>Z!f4|}OwEy2&h0w@0bWE2BJAJ&mRf!1s-25$+BO)_zd;!nJVpX@G-$Z!G*O^t5yEUcQqf?MKG8v<=iwD)+4t! zI#*^2=+~Qt*t(%44?s3a92IT_30;?M0xECMRUVZW8 z*LXeDa=C5dO>8?djsN$jCzuyULm$r`JKFnvfiES)G2-1$5Y+s6HyQj*`_Hqqo$pEN zXSwXLtu|bbZGSYCi+5YUL99!obhP_j9t-v~Gp9O5_x*tNZ^ocI9D7rfdHNQ|F9Zj-@RE+8M{n%&8`|5NjICBJ8o2+c0#+mG-XvI01qG-85;zHv>j;T40VjzNIX@V?n;0EV)Njul|ecy>cxSr>I z-{-!6_wQbwhYOwudos1u_ir{v8GZvHq7OiyTrKkU+2>2)+>#V{vX%&fTS`(Y;LSfl z5NBz(=-&kCA8R$X>mV4<08y+PnTCYOMJi4!A^w3Ghgo+Va6`r0kcD-#fX;EQ zLTH}zM<1u(qJj<6Y2hdpJU(-qw;Y>t+ME zDaG)J>1-6vrjnS!3A(kr-N=Ajk1L)cy!q;7vqO~MqPpLdDofN{L|Y-X3aq;)OBD ze}++izJ;Z;rnd?~xjuCZzMn8ZS?l;<+p_&V2_CvNs&JO@HvHiW(a;k^_BV1E_e!nQ zcR*V_-BrJ(esQ3MIH9IkG1jjXB$VX;Des?sT0X!WP_q;jqIV}37$_FUx7NXEHc}Jn-Yd{ z9Zn?v39a0fpb^eighQqV;jnTbdO;~zwtOiT0c1CCC5aPigN93Ls^PNfzJ2!2+{^_C z)^qur8wg)atv1`~~Z8n|V98xmrUkKl<2%!cp zDFC-CR-F|>d)8D<50ipDz+?mVqnuqGl zb+laOX^t50{HZsCJHNY1rJ76LvL}gK;ImEOY*=}4Y~99MZxfsa{nDq(x%<`zTcVn{pmCVQbgWs2*xHJ4!6Nv^}X5(Ae$vdt?`R|F=Z5UwY|=TKk6; zc75UF&(h=3Le#DyTf#uit{_pd;hkM}JSv6t|qUi&eR9;T-w`kR4lTW4KjarWg>AQ_~^ z5<{c6wE6J&>_!d3kJX(67zi{bdgj8Xe?Z67G0+*CPAPo_QGjwsj?gRHTVBVYx-$(t z{B`G@4bMZSz+iH#=-8+vk}h)3k=H+5R+yI0K*?V=5W!CovbMcHYCwv3)doSc*nyl- zD%IA`6u?}mjQIK1c=&?-VPeSC+*jXdfU$Lr5qSrZpLNxrYP7j>j=%siVuPGjY-H@I70o2_2sV)|enyG%TDaVd0rSGU*^`&N;}(V^M$X}j63_@RUlcRh2AErt@DUJU@?Ha#GA)9 zvbgBKKjZQdj)?{f<9~DB!2gk?OTGr}omvqF_&{cQT=mwz9?}h+>RN~SPWax zv3mJZ7N|EtBRLXRxNDT>c0cNn&e0w#kCW@q-*ufpPUyU1*H6Pl=`FgX)hxH@vY%W9 z;q;|9X{Y-?XkVaELs_xPVrk>m(E{oiwAp47;@O{TgQ-5}ORW$|rARqjtD!h3N8aDE zKGtjtgWJX;MIu5Z#B;1K=&W{A<5yyz%Ww%oTFj1q|WhgcVl^uxAWv1K9(#);PS z=yFDg=?( z7yv}il88rEJ2T%^Je~g5gst9CmWi(u_n6|;P(Lv%<Z4G&*p=$x5vl~MVMUCm6E@PhRv;) zD~8@4tE`-UfH+K;jps@ge;gMo+*2;eOtKM~ziO&<(4SVvd+1*U-XYH55F?xF@Pt*d z3{NT1Goj~1eZ;mDk`TrZ2``j1^UiIe$edZEBA(yRU9>cL_R+wo;J(OIWQO*a(@n@v z^PA0)CGAc8*zJC&L*j%Ab_n0kq?$>%SKnB$M^yeIow2K_Wq;2DJcZ&m!#;Q&?O0xU zNbF~5dib2T?B1z~U+-cU=n?2E)RQx;alFtvzT#~(e}jc8vc=q?g}_G74dNWC8T$O> zwWn)3;#ghDy=~M zamqYhO;TJ`Mg^)#4b z%04Hc8~+ba()xb%s;tQ?9rYI>|pit{A&A--dv%`R~- zbH5@8ISNzE*b6_%o8=7pe_LHU-#VSK#l|JJG`@|0=1a(AGJbNs6E;0vTj8g88dO;) z5P4%upI~jVB2sWW&?g$P7z+_-SA?}ev*B@6q&LPGQ*ymY!fR1K>=e%Dq!!t1If;UvJ@zJ)ghPm|ie@(WiETfK!FnFBlq1Kt;z37>V23Pow zJHQ48GpAiye9vlS%EijNzucd45sdD&am!Y^N8r|Dx13&kU%wVR`K&n~kM-mt-@SA# zsE5H|ct$9Ub%pDySl1L>tbYb28ew~fHb2_Qz2Iw-65(uane5aBjg+>;ICB|)<+L*Z zixB_j3d!rtTwOG(E2j6ie%c$Dz?^6%`dTGc#9LS3Dm7{Q2{; zvN95h)Ya8BIXMYWv81e=OiwQSJhf!AM1vqYH43U|DyGw+Apo8YxBhV+?ePELaHyd*+q-Xh~bGVeNZIK~u-Gi0WRSBG^IqF9q_W!6P9=u|r3#}jzaw-)m zp*ddFl>P0#Bm&-nBv}rzC&HGjJ7JH?dPA2CY>d??>Gi&+4>t95LDo@3> z)Rit<7)*;(SH(#RzCVxzkJ_<9zeuVoZ;pdv*k)^;v)ekvF!yr&(?PN3JFFpV1G#0e zHj@m_8_I_H8)MfbQ?xaYV!{ME#YFcf*T=Xd@o{~;+FEK9&2~fK`f!duOR4bt`M$5V zN$vXDkZtIpMFG-W$A1-&Kn%y({qObuKMiYH6uWcqeX)`Qc_ukJ3t*rrlkt2ZoZMM^ zB%?W?L6taCkxSor0k-+G*wywUwA6{}-R|ayeIB@DV7aL%$=bQ5Fh-=ule8xxy9sU= z?JQJV1t8)*nIfJ0Nrg((@&h$Mn;KD&^ktHhR-{A?xR76+s+p~n4h+g00YM3~bWZ0n zWc;quPJWVubLirZAD@%IbLdY2y_o4{GjURcbIsM8!{z;L{z55Q;DRTD@qy#%Fd?z| z0&;1T;9U)Hd*O_(y=C3_L>R*%o_>KjZ0YedsWATyQG|gN8^56dR>%;r68b;yaB*W2 zW`m9%2UUsVkF$VJgh-?3A{1@*IaQ+bvT=QkMZ#?HZs%AC0Jjy`=a!Q^aM4e+Ky!Qn z-VX)0jWU45_CwXts>Bv4Q($w#e0(dp&A;xi{jh5oI}!=A!H=at>TT9Kom?f+V-&u>RBJOZpq3rA91fYPQG-A^G1(;?w2f&GJS%KfE-3xcelF zU3CeN*dsT$l>1ZzUz^5-D6qE#kQDN!XN_9R^pK1ma8g)^w6<{X7N^tXY0k4&O^6R8 zPTglY_jB>mJ`Z=e1Za(5q+m6@0Ne2Y2BT@XMxd$AU-EQ)Z2lnl9&|#lSwacU_ z!DLv89_we`oODG0t*FNa?!ab~gNDY8Y)*(FJdp-!W0ox?7ZxJKoP%fTzZbF-xAx8k z16#E;Yj?9`h`)R&To4E33u_5b&He7;S}>1jk7gpd*#-NIC6&Pf#N%^IGF5)BW@>N~ z8dBgho{X4FM4N@BudZkh0n%R&GEt}Vbc=FvvA`vHS4l?ciW*xvZ%N=q8O!^j1 zY8kJF1r0eY%Qy`7Uqf6xNxYV9Un|D5OkEg%lto;D<0eQ5 zO}Z42BUWp%9VMplZ1#%n=!$zN)|s)Or6>HOjxKV~+&H2T=g3;+Q$SSOj*MbG-Kcqs zZ>n6GDndeKMw_rBeS7J}S!Z%XPf+U9oW#+Td{-U{s{$^_k>$HQ4g1;lIsFXhk0eER z|Bdncg@}a7!Shp_5m+CWnjmyrG%zS93oei=p*F|o^vsww3>d&OhTJ!mj~ho*paM8! z1ayj%#5=@9BM%%To%MkttgJZ^1y}5VGivePes4#np4 zH4he9=PJGz<@3nE`+;Hm?~b~FVBZS6R5}g#Bpm;=xtQyf!Xd5)jt7h!75B&TVmz@& z1sb@BFpV9V+~AZeq{FMvSe8TB;gV_U$fV>v-_kmNM@>p=Q?8CvrqsVpJu) zdj>_qq9s^jgs#(38(MtcT$n#%YR|g7;ebh!i^-Z-11ZKVc9ly$dQ*)>bDc-QfQfi_ zA$XC~4}+@J_oX@I@`pzE%WUwy)s+MToCV{vIWe)Zr$zJMSGrR)IP{$v+iXbR`B7Qg zr{JeT3A5{h15Z``y9pEivS`ZUvAJ5OT1FUD92hV|f8~WaGG2efkznr3;C)c(^5xus zh3%gU-+LGICx6dtS69$j>xV<@OcP@MVc)@hk+L42k-l>hs?(`G3L2*Z`UsJ_U&q-| z-qLC_sNx?;Uq?bi-QT@7#;~?O$*hre@*d)(v8sU=ItPgg3K|vVaE8;pER&5&v-0Zv z);8Pjd5wW4={u^yMn2I7*D$%6D7}hie>@Pu=|n9>`o42Z-x=kmZ1087!7;*4LIjW+ zvl`v(zj!tVf4{>Bc{bxD$#W2Vry)e#d~+?F(`lfY)!iJw*)@B|T7KdK>a!RRMFcYV zfJ7Ah9t;f3`t+Pk(OmH$`}{iLZA)nI#6kIr@Z}<>3BC}yD0#ORD6+k8f6>+qPA_QE zTDAJLeSnP&99J7M8}HP{qkv0tJ*qep_)vAOXv`5Sn zyZ}{cSpCjca#LXF{#Jlrb}PIvNiuTl+myj?0;G3_I0O7`zwDW_2 zN$_|qkbRsbh8YoioxGSQ0}?(KtXa{e7Y z7tX^8!GN04eO0(XW#7ZLmCeTi4vGo0l}{$plHnWQ!R5so)zlr=SV=JRRgXCEbM2S2 zGsExA?OTkLHvf&*Bliif%5>_cdW8$>+E2ijSZv0_Uo+)O)NNBV<@Q0xfz^&z1-|YH z%0*>yZT>F}?S%`V$nIA~HixN6-cqK{nqJVQTLp-hM0<`L>J3%+raW+nG(8n6 zw{K@D8$LvStlx>(+;Q#sBtU0<4djt%2O<8O(5#QEBAUiJxkWafxPmUcoWb@7F z_+E8hBT`s^)YttzVC6pME+GPYCOybuhq*<3dYUTssk-oXSLM)nhVX19-megf8(?$4>Ow`5F|B((SblXecdJGIK6)89b6sv`zfZ}GT72D z*nGHe!=X!E;)+>|G^bNHGj$BJS-44kxE1xIHU_xqVs8;cRuV(OQxH4|@!AqP@X*JC zeW_+Fu#)RZkCvFgcf+hp$0_l4I2Q0csKT2AgM&kzwLP^?6`)Kc#>U|PQXbkSRV2T+nYAs#}=Qk!aM}Q(LnZ7Ba%Rg^ltg5CFadHwK>$uEbfq~R2`NW z$&^9`gn@*_6-T38{YSKSPH&Ex?zO$98`<^>Mx&-Q$30?!o*Q&(cbcZ($r=gg9i<|e zyI%0CxiW^TyQ`#T&|O0yWm0~FB9+Y{!T-ksEapSy%Xdq6l5};^wTFw+l$bd#OL4u4{|9)aO?c$ zA$z{r&EX&^Vh=uk-E8x(=An@hxXDt0ev(pIR5jbTLUVAg>0hEf6}Kid)&bZ<7~;gEZOhFDf_b8S8vuU7jjW9QkjzUIJW zI9~fWWoi@f>M=tv*Fk}1ZEIt{QdUtvBP^5;QQe*IM1FK9090Q^tBp&T4&E7 z&(j_YoRvNFe%pDEb*X0D#Ka+)Aq`F+o9E@QK#bP2ao)MgT9aA*GJOyn5iCXs1x<5r zsRwR?-^J`}$H2cdbiZ<8!tgPF^W7TCp%HyXq}`%(XxoD6wYp{A=0;5vQxLNmf(^4y zt#Cxi3|7}u1W$o<4#mw7(xfz&Khgf5e;juG5&l)2hT`XsiQ>^^_Rkxd8x-j|Mg1FI CrybG& literal 0 HcmV?d00001 diff --git a/Documentation/admin-guide/mm/damon/streamcluster_wss_time.png b/Documentation/admin-guide/mm/damon/streamcluster_wss_time.png new file mode 100644 index 0000000000000000000000000000000000000000..b4f19cce225e43e7b0c23563245c19ebe5ed3f0c GIT binary patch literal 6322 zcmc&(c|26_+n#d_&5(@7UNR%u%91T2#)RyoMRt`)*6fm;kQ8NP&-ztTRAi4ZN<~Q7 zN;QnVtYgWNFY`X*x4nP8f4;x@%$)N)&*xe1>%Ol0K4)%Onjhol6yZdnP~1j_dR8bD zjG#~u9m57%V4eegprT}HdQzXsWP+Bwygb&8ib5ewDvF7kou!gcEiEWr_aP>gfI>)A zgsJF^#Gz0m1jR&{5zL+fSy?cYME?ZFiHL|OD=V9snYp^UMnptpWMq_=ms6?K_V)I% zu`#d&LrIA;OL_KnoN=jwjPdyTYLA*2uHV0Z7eSn=oS>Zk4)68gzNb)I z(p1#V6Md2ge_woa>4WSP`f1znl=FJNY4?48g5ExlYv~pDUw!91(?9scD>-?P|H3;- zq2UuMw|hJKDo?#wY!~&f$<@Eqpi@``?=HMlq2$CB_ibh*5Ux$LnRLAR1>yG~#hMQ- zL*+fmGui_d)T@jlX$>CC&g7txOFT+H#I8Y*UQ69Zt?Z0>S;(O#ZL1Ki`~{nOTV{hM zLd$K4!YK^*8Yv(OzQ5tJWeF31(xaE1izS_B*3+GXMzK95Y6Ff-QiedR=)55Z5USDTF_T* zYyF)B`h`_L^3us*0ZfQ+Ll*8xZ}#i=mji_h^9$p`HHcS=G~Sv* zGsdC`^pQmVPomK^f32>k;~J5PlZB#aEmJiO?h|g_3MC2Hd7sGo-fUipziwM&rMv4F zRc-rY)nc*pd!kqO$2(>RIqq|QgN18HzU!LCGF93&dfm4{Jl)?eSiDSsY8X%J-SsY7 zViW(laBJSfjq>Q`?8UapikI&+;3Onu#7S& zA3wwL9|&hDSX+!oK_yXH6f{&4{hw=savOHd+q$?wax~ghml_q~N51E&wRA*-xSv43 z90S)51y6Mif&s#qkll^QXtd|>Vyy|W3>K(#cqZ&eR*OX53n$!jDtuyBoQN-DM{9jN zAocJFhA8F2h6%~Cv9elE%A#<;G9j&HQOsXNW>LIenNVX+^datK^A%}5SEwCx-zgn3}fBJMvNm;3~9XNG)VO;THvo-~H zTkD~|gCx-KAg=AONA8enzh}FVFpe=l{d3rJ{)P}{HDoE^pau~ObGTPE8H^LQtbX)l zs50GBr_05(1#5&;)N{Z8c{=IR8qOsPZ?(+EyLO*~S7L{vp3PibjyqxINB$8u!i#Ff z+Cb2kCz;o$Ge4VKe``m#yG4k3%&Zd*;#1ci^(R)Dcs3z?%gCIq1%&a$Y+k+s2OFJi6%cObfQ3 zT8I8IST8BJ%c-n#e-j?6OPp`T_P?$}dwvZSKkPpNSLepaTywJlT~RQHt)Tbq$|o@z z#QMAcnC@BJClTL!_^unLzz(_5_J!>;e4+9uJNN*}H7)cC15;hcKHN&G0nF_W({c8X zXq!B(EDNl)96ay7ybrSq2Mus5ugP%(veQ=BHqRF^qmYCNCs?EpArt7AzKdnW@Pbpe zl&O?wsmQ}{1CkxA$93uj8rI&kC6^G+H+!MtP6->(4|AyTy_mrxoC0eB$J=d_-S1pY za18e8xnWo5sPfZMK6pkF!8v_gz9GrmkOgO5T(Q!#^4H0Q@s_uc1hlZ8sR z)(4TSiR9M2Y1WsI7H!AWJo~LN;k*-l6aFzEt-#soN(KcnJzNa8AZs^}Tz0;og=SXg3;}8_t@)q2SA~=#w$ClfzNShffX; zSG8NGw)e0CUmOn{^OYjF8xO2IzZiKQ*h=b}qpvAoQgbKQT`X{euc1ws0rieLmP+Cj zQK3y`1ZT`bq`oedn<98%8_XeHC-oIAJAa*J<>S6}Yhx<$+c83y911G?kMuHGqzw+K z%J-gKJW3MADgO8qwmXZMhr`qq1m>h-q}gEaOAgt~k0roL+1BK5PHx;u2}r^*R*aU{ z#?*XMF{^!X_A9y#K|EY+cJ3jn$^&?}!t}om1BmP|R=u%Po?$%ZXHtr*Wlr22fPUo74aGfcv>XZl?T3Ip<8svwBPW{wDlvueq{<|7pLmWZhcUbA;Eg??eV> z^=i;SJmKWqo&U=&L__19><}9zS>nUQ%E%E)@2 zVvLOL5$FP75E~SQ*_VeiNfo1Kw=huH{CkN z@|I1y-a%Hn5#gJk&g#D9RXgBIM}q@D>yo^FK9@g?3E9WuQ2_-XiKMYmF>nTfhGMa8 z0T>U9-hRAl)Wm|r!jIf=^0yvo%iy>R8GRjRh6kCM1#0J^xZ3+X7xc_y#n9W&dXAEh z-!Hjs;`MH4_@ScHpt~A^5I@KtADU{j%1lJYi&OXkNZbVH(!l-BwMs*VRLNUJuWq3h zEcQA=yj0zo+Pgp4g`2(Nn9z(iL4&9Z{3j*VCPg)rYi*d2!uA(uoyiwYLvuq)HFh;& zR(1KMb9tW&A@5hrk}AM!$O~3$j*t(jG79tni@zo~ZwJ_U^M-ow7{}jj~id0XX&dluElhu+o~-$uBl64PLV^O?+|=%hI-7F7192tFQu1 zsXW^iU6jERE+;`ZQF1Rx4&Fat<*S0C=pMl1{3v9~tXuowndf=);uf}dcjA0g=5oZ= zk$gg)T7b1+b|-zpZY#5m+X2V?RI87t$igs(U7g)iza3;fu&I>r=h9MAdK~EOT|d5F z&4~bRv>bHVF>PY)=9Z}D)@&Z73YI>I^Emc_rf5Z=%l4yQMjEj89D1z6lHUZBhB9#a z1n1bUSiTn^Ioy+jJ&uNBC|4z;0oS&Rhw_;Ms2w>qh=3kIhkCac`=qiWEC&>IHRZ|F z)lgC~Ximv+#T+RA1c-MKBb0C0t)$?V6Q7>2rp3iclxha@ByEz0FAF+Waf}+DMKNQ+ zD%M@va*|a2Pp$NfK-!CRaQHop;V_uvum*oELi-JAK{4M3F6vQ=mv=@SIQ&J0eim|k+!_W;+bYiw8}zn2YZ!2 zD;_e9TQZIj0kh9Jaoq*<+lsNZUkQ1K0m{43CvVP9o{K^zfu?)nEJcBybHhb|c`tNH zfq;2xn8!P4y>B_7-oNVp??GEIg~WPh54dPyqD1tn)Ozl>NL>mw| zG;DBII!`pL%d||e1#5$X9w_c2Rst^X!U)~SKu(?K-!dm>c{)cm48rELA}IW|3#dBu9G(bBN|g2toQ8()w{<1=iI zFf2}h9Wk&ty?_7;obZ4>b*<;R#wC6RJt8&`=gW)MqweDRnJEHIfXOhi2quIGu4$2= zurC)4DLk9?H@)!yc6?5RB|#Ggh++!CdBF-h?Ts^^*~c`fbCTPoq(v#UPCKy|zTr9w=g z_Zr0Rw4{}j6VbczpAMJT|`h6^Vw-#T0J(Bx5VVx_a4w%q%&p!GLD|&{eBEfCoL2>jfJa9ltA`(ez)oJ~^ zer=+4j&Hb!)H+Iad{F<^a}&nYpx_*N^R?uJb2<+~ z;V0&BjJyq|bJYQ~Cy824A}OR57BrrwZm#d8?|WrT>fEb`x%l!W@_@CuBZDd*`jDow zFXPo`#nDTPN|0NErFb4YWC24P1iDfC?0}{;eVKU`^K%;{HMf`k(q5LXb0~~Cp=>7v z+vdpSMHDPbK!;3kAlGoE8#j=+*dQLbsuAa#7{;aE_{TRJ$vK}R1joF)Se{Fu|7mBw zq{w+t-3Kr7F&Zn|C7nD2tf7LN^V;Kx&twFbbO{lbKlt3c%W0r^B5}2SQ~mJTbcw~n z{K`uX$hwg_t|8f7qVD*2W+OyONjFD`ix9NEHFY>Se+?L`p>O=_)fWxy`|6X@f`V-pr`!_WDJSn--$kf3^{$Wuz zLS8KCnq2REe9sm!n%Vdzz9^=Z^tsb+MD<|P{ZzNm$+69FE5@BwMLfemGl?G3k2?^-Qu^rxVa(csl@e@oi8I|zq5`{%8EYwoPd^L zIe(IEtkE4j=y0Kb`t9j+T+)~7uxjK*F>NWA2|3yqrg%u>(m~ET9_b2-Yn3_aueHKT znIczy`3lsD!YAL^lhY%l=sp9l!#8xI<78*pU?r{>b=_s~{bm|XvK$#FJS^Qt^bxQO zsLZOPg(TjnIg0*l<~!+XhmoVwYTVf)<|NMqyjSty*{d&bu69B+z9+3E0{(Cy4^O_h zxuT9q)KclJ?dO!&TTqE#^NPG6@aZ|!ZJis#tEFM{+v$`yF<&Y_^Te-|_FKa-thW{9 z#m4uyE;j92_Nczmt}h5)StMmd4CKV{+EN)-_S0)Coi0m@<3rp34bXoI2r4tDI^$9A z=J|s+Cr}o9P1bX#_#E!_Fa?u(PUVTdx--)$vd0#xifauxa)v`Hu~$E1{9|5OMT!4P zyTO1K(^6B%bHAoyTG-~$`la;Fude;y47^7xXZ;U_9g4}wxJpm$mVXp(zhl7Q^u!~( z8yZk!$;?;TH)g(zv)1BCg8IsQq@g`WlxMVTgFW}IQ|*{9GwVf_J4VZSemKqCKK=c{ z9bZoK2iK8}8nL5LW^kOVe#*&Tb3@;?Utd-)zOov+9Jc+d(ng+*xs-G3*_K_GX*ztR z>mCQ37Ylp49_rVX{h6s8`{jJP_ox5LfX|rPMxN6!)hY!4F@H#Mb>&J0)_c>FUFEGr zX@CTM)nD`MXRP`LRel@fWTqzc{o#N#Js?@N0x9CHn#!@{ao$lhBj4dD9v1iqE ziAFE2BaNNqgU+W?e{DMoX<$P4nWMu3?pvreor*Z1#ars0!`I)LKC5Cnuw%U|Vpeju zC_9f%OPlw>_vWr|yxbEcipJ4{dL+%;an__T$C8iN_U6}&zga|R6Eo8hg-l7Yuh^M$ z>b%^{_s{C@$}|_Kuj+W`&c`A@;!(%W?q&TK0X_`dO;p@7v2F+F!Dpf F{{WW|bk+a> literal 0 HcmV?d00001 diff --git a/Documentation/admin-guide/mm/damon/usage.rst b/Documentation/admin-guide/mm/damon/usage.rst new file mode 100644 index 000000000000..1aa4f66e4320 --- /dev/null +++ b/Documentation/admin-guide/mm/damon/usage.rst @@ -0,0 +1,305 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=============== +Detailed Usages +=============== + +DAMON provides below three interfaces for various use cases. + +- *DAMON user space tool.* + This is for privileged people such as system administrators who want a + just-working human-friendly interface. This interface is a reference + implementation of the DAMON debugfs wrapper user space tool. Using this + tool, you can easily use the DAMON’s major features in a human-friendly way, + though it may not be highly tuned for your specific use-cases. +- *debugfs interface.* + This is for user space programmers who want optimized use of DAMON. Using + this interface, you can use DAMON’s major features by reading from and + writing to specific debugfs files. Of course, you can write and use your + personalized DAMON debugfs wrapper user space tool that reads/writes the + debugfs files instead of you and provides a more human-friendly interface. +- *Kernel Space Programming Interface.* + This is for kernel space programmers. Using this, you can utilize every + feature of DAMON most flexibly and efficiently by writing kernel space + DAMON application programs for you. + +We recommend you to start with the DAMON user space tool and move to debugfs +interface only if the real requirement is found, and prohibit the use of the +kernel space programming interface unless you need it, for the following +reasons. First of all, there will be no big difference between the overheads +of these three interfaces, unless the use case is so special. Also, all these +three interfaces support all major features of DAMON. + +This document, therefore, does not describe the kernel space programming +interface in detail. For the programming interface, please refer to :doc:`api` +or ``include/linux/damon.h`` file in the kernel source tree. + + + +User Space Tool for DAMON +========================= + +A reference implementation of the DAMON user space tool which provides a +convenient user interface is located at ``tools/damon/damo`` in the kernel +source tree. Please note that this is initially aimed to be used for minimal +reference of the DAMON's debugfs interfaces and for tests of the DAMON itself. +Based on the debugfs interface, you can create another cool and more convenient +user space tools. + +The interface of the tool is basically subcommand based. You can almost always +use ``-h`` option to get the help of the use of each subcommand. Currently, it +supports two subcommands, ``record`` and ``report``. + +Below example commands assume you set ``$PATH`` to points ``tools/damon/`` for +brevity. It is not mandatory for use of ``damo``, though. + + +Recording Data Access Pattern +----------------------------- + +The ``record`` subcommand records the data access pattern of target processes +in a file (``./damon.data`` by default). You can specify the target as either +pid of running target or a command for execution of the process. Below example +shows a command target usage:: + + # cd /tools/damon/ + # damo record "sleep 5" + +The tool will execute ``sleep 5`` by itself and record the data access patterns +of the process. Below example shows a pid target usage:: + + # sleep 5 & + # damo record `pidof sleep` + +You can tune this by setting the monitoring attributes and path to the record +file using optional arguments to the subcommand. To know about the monitoring +attributes in detail, please refer to :doc:`mechanisms`. + + +Analyzing Data Access Pattern +----------------------------- + +The ``report`` subcommand reads a data access pattern record file (if not +explicitly specified, reads ``./damon.data`` file by default) and generates +human-readable reports of various types. You can specify what type of report +you want using a sub-subcommand to ``report`` subcommand. ``raw``, ``heats``, +and ``wss`` report types are supported for now. + + +raw +~~~ + +``raw`` sub-subcommand simply transforms the binary record into human-readable +text. For example:: + + $ damo report raw + start_time: 193485829398 + rel time: 0 + nr_tasks: 1 + pid: 1348 + nr_regions: 4 + 560189609000-56018abce000( 22827008): 0 + 7fbdff59a000-7fbdffaf1a00( 5601792): 0 + 7fbdffaf1a00-7fbdffbb5000( 800256): 1 + 7ffea0dc0000-7ffea0dfd000( 249856): 0 + + rel time: 100000731 + nr_tasks: 1 + pid: 1348 + nr_regions: 6 + 560189609000-56018abce000( 22827008): 0 + 7fbdff59a000-7fbdff8ce933( 3361075): 0 + 7fbdff8ce933-7fbdffaf1a00( 2240717): 1 + 7fbdffaf1a00-7fbdffb66d99( 480153): 0 + 7fbdffb66d99-7fbdffbb5000( 320103): 1 + 7ffea0dc0000-7ffea0dfd000( 249856): 0 + +The first line shows the recording started timestamp (nanosecond). Records of +data access patterns are following this. Each record is separated by a blank +line. Each record first specifies the recorded time (``rel time``), the number +of monitored tasks in this record (``nr_tasks``). A numbers of records of data +access patterns for each task follow. Each data access pattern for each task +shows it's pid (``pid``) and a number of monitored virtual address regions in +this access pattern (``nr_regions``) first. After that, each line shows the +start/end address, size, and the number of monitored accesses to the region for +each of the regions. + + +heats +~~~~~ + +The ``raw`` output is very detailed but hard to manually read and analyze. +``heats`` sub-subcommand plots the data in 3-dimensional form, which represents +the time in x-axis, virtual address in y-axis, and the access frequency in +z-axis. Users can set the resolution of the map (``--tres`` and ``--ares``) +and start/end point of each axis (``--tmin``, ``--tmax``, ``--amin``, and +``--amax``) via optional arguments. For example:: + + $ damo report heats --tres 3 --ares 3 + 0 0 0.0 + 0 7609002 0.0 + 0 15218004 0.0 + 66112620851 0 0.0 + 66112620851 7609002 0.0 + 66112620851 15218004 0.0 + 132225241702 0 0.0 + 132225241702 7609002 0.0 + 132225241702 15218004 0.0 + +This command shows a recorded access pattern in heatmap of 3x3 resolution. +Therefore it shows 9 data points in total. Each line shows each of the data +points. The three numbers in each line represent time in nanosecond, virtual +address in bytes, and the observed access frequency. + +Users can easily convert this text output into a heatmap image (represent z-axis +values with colors) or other 3D representations using various tools such as +'gnuplot'. ``heats`` sub-subcommand also provides 'gnuplot' based heatmap +image creation. For this, you can use ``--heatmap`` option. Also, note that +because it uses 'gnuplot' internally, it will fail if 'gnuplot' is not +installed on your system. For example:: + + $ ./damo report heats --heatmap heatmap.png + +Creates ``heatmap.png`` file containing the heatmap image. It supports +``pdf``, ``png``, ``jpeg``, and ``svg``. + +If the target address space is virtual memory address space and you plot the +entire address space, the huge unmapped regions will make the picture looks +only black. Therefore you should do proper zoom in / zoom out using the axis +boundary-setting optional arguments. To make this effort minimal, you can use +``--guide`` option. For example:: + + $ ./damo report heats --guide + pid:1348 + time: 193485829398-198337863555 (4852034157) + region 0: 00000094564599762944-00000094564622589952 (22827008) + region 1: 00000140454009610240-00000140454016012288 (6402048) + region 2: 00000140731597193216-00000140731597443072 (249856) + +The output shows unions of monitored regions (start and end addresses in byte) +and union of monitored time duration (start and end time in nanoseconds) of +each target task. Therefore, it would be wise to plot the data points in each +union. If no axis boundary option is given, it will automatically find the +biggest union in ``--guide`` output and plot for it. + + +wss +~~~ + +The ``wss`` type extracts the distribution and chronological working set size +changes from the records. For example:: + + $ ./damo report wss + # + # pid 1348 + # avr: 66228 + 0 0 + 25 0 + 50 0 + 75 0 + 100 1920615 + +Without any option, it shows the distribution of the working set sizes as +above. It shows 0th, 25th, 50th, 75th, and 100th percentile and the average of +the measured working set sizes in the access pattern records. In this case, +the working set size was zero for 75th percentile but 1,920,615 bytes in max +and 66,228 bytes on average. + +By setting the sort key of the percentile using '--sortby', you can show how +the working set size has chronologically changed. For example:: + + $ ./damo report wss --sortby time + # + # pid 1348 + # avr: 66228 + 0 0 + 25 0 + 50 0 + 75 0 + 100 0 + +The average is still 66,228. And, because the access was spiked in very short +duration but we use only 4 data points, we cannot show when the access spikes +made. Users can specify the resolution of the distribution (``--range``). By +giving more fine resolution, users will be able to see the short duration +spikes. + +Similar to that of ``heats --heatmap``, it also supports 'gnuplot' based simple +visualization of the distribution via ``--plot`` option. + + +debugfs Interface +================= + +DAMON exports four files, ``attrs``, ``pids``, ``record``, and ``monitor_on`` +under its debugfs directory, ``/damon/``. + + +Attributes +---------- + +Users can get and set the ``sampling interval``, ``aggregation interval``, +``regions update interval``, and min/max number of monitoring target regions by +reading from and writing to the ``attrs`` file. To know about the monitoring +attributes in detail, please refer to :doc:`mechanisms`. For example, below +commands set those values to 5 ms, 100 ms, 1,000 ms, 10 and 1000, and then +check it again:: + + # cd /damon + # echo 5000 100000 1000000 10 1000 > attrs + # cat attrs + 5000 100000 1000000 10 1000 + + +Target PIDs +----------- + +Users can get and set the pids of monitoring target processes by reading from +and writing to the ``pids`` file. For example, below commands set processes +having pids 42 and 4242 as the processes to be monitored and check it again:: + + # cd /damon + # echo 42 4242 > pids + # cat pids + 42 4242 + +Note that setting the pids doesn't start the monitoring. + + +Record +------ + +This debugfs file allows you to record monitored access patterns in a regular +binary file. The recorded results are first written to an in-memory buffer and +flushed to a file in batch. Users can get and set the size of the buffer and +the path to the result file by reading from and writing to the ``record`` file. +For example, below commands set the buffer to be 4 KiB and the result to be +saved in ``/damon.data``. :: + + # cd /damon + # echo "4096 /damon.data" > record + # cat record + 4096 /damon.data + + +Turning On/Off +-------------- + +Setting the attributes as described above doesn't incur effect unless you +explicitly start the monitoring. You can start, stop, and check the current +status of the monitoring by writing to and reading from the ``monitor_on`` +file. Writing ``on`` to the file make DAMON start monitoring of the target +processes with the attributes. Recording will also start if requested before. +Writing ``off`` to the file stops those. DAMON also stops if every target +process is terminated. Below example commands turn on, off, and check the +status of DAMON:: + + # cd /damon + # echo on > monitor_on + # echo off > monitor_on + # cat monitor_on + off + +Please note that you cannot write to the above-mentioned debugfs files while +the monitoring is turned on. If you write to the files while DAMON is running, +an error code such as ``-EBUSY`` will be returned. diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst index 11db46448354..d3d0ba373eb6 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -27,6 +27,7 @@ the Linux memory management. concepts cma_debugfs + data_access_monitor hugetlbpage idle_page_tracking ksm