| Message ID | 20241119135104.595630-1-zengheng4@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | arm_mpam: Introduce the Narrow-PARTID feature for MPAM driver | expand |
Hi, On Tue, Nov 19, 2024 at 09:50:58PM +0800, Zeng Heng wrote: > The patch set is applied for mpam/snapshot/v6.12-rc1 branch of > https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git > repository. > > This patch set is fully compatible with x86 RDT functionality. > > The narrow-partid feature in MPAM allows for a more efficient use of > PARTIDs by enabling a many-to-one mapping of reqpartids (requested PARTIDs) > to intpartids (internal PARTIDs). This mapping reduces the number of unique > PARTIDs needed, thus allowing more tasks or processes to be monitored and > managed with the available resources. > > Intpartid(Internal PARTID) is an internal identifier used by the hardware > to represent a specific resource partition. It is a low-level identifier > that the hardware uses to track and manage resource allocation and > monitoring. > > Reqpartid(Request PARTID) is an identifier provided by the software when > requesting resources from the memory system. It indicates the desired > partition for resource monitoring. By using reqpartids, software can > monitor specific resources or allow the system to subdivide smaller > granularity partitions within existing partitions to serve as monitoring > partitions. > > For the new rmid allocation strategy, it will check whether there is an > available rmid of any reqPARTID which belongs to the input intPARTID. > > The MPAM driver statically assigns all reqPARTIDs to respective intPARTIDs, > with a specific illustration as follows: > > m - Indicates the number of reqPARTIDs per intPARTID > n - Indicates the total number of intPARTIDs > (m * n) - Represents the total number of reqPARTIDs > > intPARTID_1 = 0 > ├── reqPARTID_1_1 = 0 > ├── reqPARTID_1_2 = 0 + n > ├── ... > └── reqPARTID_1_m = 0 + n * (m - 1) > > intPARTID_2 = 1 > ├── reqPARTID_2_1 = 1 > ├── reqPARTID_2_2 = 1 + n > ├── ... > └── reqPARTID_2_m = 1 + n * (m - 1) > > ... > > intPARTID_n = (n - 1) > > Each intPARTID has m reqPARTIDs, which are used to expand the number of > monitoring groups under the control group. Therefore, the number of > monitoring groups is no longer limited by the range of MPAM PMG, which > enhances the extensibility of the system's monitoring capabilities. The idea of mapping multiple reqPARTIDs to each resctrl control group looks like it can work, but I think that there are some issues that need to be considered: 1) There may be a mixture of MSCs in the system, some of which support PARTID Narrowing and some of which do not. Affected MSCs will not be able to regulate resource consumption for a single resctrl control group as a single unit, if multiple reqPARTIDs are used. This matters when an MSC that does not support PARTID Narrowing also has resource controls that are not of the "partition bitmap" type. (Consider a resctrl control partition that throttles the partition to 30% of memory bandwidth. How can the same behaviour be achieved if the partition is split arbitrarily across multiple reqPARTIDs?) Because the MPAM driver needs to be as general as possible, it may be hard to make the "right" decision about whether to group reqPARTIDs to provide more monitoring groups. because different use cases may have different requirments (e.g., number of control groups versus number of monitoring groups, and which types of control are useful). 2) The resctrl core code uses CLOSIDs and RMIDs to identify control groups and monitoring groups. If a particular driver wants to translate these into other values (reqPARTID, intPARTID, PMG) then it can do so, but this mapping logic should be encapsulated in the driver. This should be better for maintainability, since the details of the remapping will be arch-specific -- and in general not all arches are going to require it. With this in mind, I think that changes in the resctrl core code would be minimal (perhaps no changes at all). 3) How should the amount of reqPARTID grouping (your "n" parameter) be determined, in general? As with (1), the right answer may depend on the use case as well as on the hardware. From my investigations into this, I feel that some configuration parameters will probably be needed, at least at boot time. 4) If the mapping between reqPARTIDs and (CLOSID,RMID) pairs is static, is it necessary to track which reqPARTIDs are in use? Would it be simpler to treat all n reqPARTIDs as permanently assigned to the corresponding CLOSID? If reqPARTID usage is not tracked, then every control change on MSCs that do not support PARTID Narrowing would need to be replicated across all reqPARTIDs corresponding to the affected resctrl control partition. But control changes are a relatively rare event, so this approach feels acceptable as a way of keeping the driver complexity down. It partly depends on how large the "n" parameter can become. (Since PARTID Narrowing allows any arbitrary set of reqPARTIDs to be mapped to a given intPARTID, it might be possible to allocate reqPARTIDs completely dynamically. But this probably would require a change to the resctrl core interface. It is also not clear to me whether the "num_closids" and "num_rmids" values advertised to userspace can be satisfied. For now, static allocation seems the most straightforward way to to get better monitoring, but perhaps it could be enhanced later on.) [...] Cheers ---Dave
Hi, Thanks for comments! On 2024/11/19 23:31, Dave Martin wrote: > > 1) There may be a mixture of MSCs in the system, some of which support > PARTID Narrowing and some of which do not. Affected MSCs will not be > able to regulate resource consumption for a single resctrl control > group as a single unit, if multiple reqPARTIDs are used. > > This matters when an MSC that does not support PARTID Narrowing also > has resource controls that are not of the "partition bitmap" type. > > (Consider a resctrl control partition that throttles the partition to > 30% of memory bandwidth. How can the same behaviour be achieved if the > partition is split arbitrarily across multiple reqPARTIDs?) > > Because the MPAM driver needs to be as general as possible, it may be > hard to make the "right" decision about whether to group reqPARTIDs to > provide more monitoring groups. because different use cases may have > different requirments (e.g., number of control groups versus number of > monitoring groups, and which types of control are useful). 1. The patch set solution is designed considering mixed MSC scenarios. Regarding the definition of the quantity 'n', here is a detailed explanation: n - Indicates the total number of intPARTIDs l - Represents the total number of reqPARTIDs m - Indicates the number of reqPARTIDs per intPARTID The values of n/l/m are derived from the following formula: n = min(intPARTID-np, PARTID-nnp) l = min(reqPARTID-np, PARTID-nnp) m = l // n reqPARTID-np -- The number of reqPARTIDs supported by MSCs that support narrow-partid. intPARTID-np -- The number of intPARTIDs supported by MSCs that support narrow partid. PARTID-nnp -- The number of PARTIDs supported by MSCs that do not support narrow partid. The software needs to ensure that 'm' is an integer, meaning the number of supported reqPARTIDs is an integer multiple of 'n'. To illustrate how to determine n, l, and m through examples, we can assume a specific platform: L3 - Supports the narrow PARTID feature, supports 32 intPARTIDs, and supports 256 reqPARTIDs. mata - Does not support the narrow PARTID feature, supports a range of 256 PARTIDs. Then， n = min(intPARTID-l3, PARTID-mata) = min(32, 256) = 32 l = min(reqPARTID-l3, PARTID-mata) = min(256,256) = 256 m = 256 / 32 = 8 The mapping relationships between each group's closid and the respective MSCs' intPARTID/reqPartid/PARTID are illustrated: P - partition group M - monitoring group Group: Closid MSCs with narrow-partid MSCs without narrow-partid P1 : 0 intPARTID_1 PARTID_1 M1_1 : 0 ├── reqPARTID_1_1 ├── PARTID_1_1 M1_2 : 0+n ├── reqPARTID_1_2 ├── PARTID_1_2 ... ├── ... ├── ... M1_m : 0+n*(m-1) └── reqPARTID_1_m └── PARTID_1_m P2 : 1 intPARTID_2 PARTID_2 M2_1 : 1 ├── reqPARTID_2_1 ├── PARTID_2_1 M2_2 : 1+n ├── reqPARTID_2_2 ├── PARTID_2_2 ... ├── ... ├── ... M2_m : 1+n*(m-1) └── reqPARTID_2_m └── PARTID_2_m Pn : (n-1) intPARTID_n PARTID_n Mn_1 : (n-1) ├── reqPARTID_n_1 ├── PARTID_n_1 Mn_2 : (n-1)+n ├── reqPARTID_n_2 ├── PARTID_n_2 ... ├── ... ├── ... Mn_m : (n-1)+n*(m-1) = n*m-1 └── reqPARTID_n_m └── PARTID_n_m The advantages of doing this are: 1. There is no need to modify or disrupt the existing resctrl layer interface, ensuring that each control group has same resource control functionality; 2. MSCs that support narrow-partid (including intPARTID and reqPARTID) and MSCs that do not support (only including PARTID) can share the same PARTID space; 3. On the premise of ensuring the (1) point, the number of control groups can be maximized, because users can always choose to make a control group act as a sub-monitoring group under another control group; > 2) The resctrl core code uses CLOSIDs and RMIDs to identify control > groups and monitoring groups. If a particular driver wants to > translate these into other values (reqPARTID, intPARTID, PMG) then it > can do so, but this mapping logic should be encapsulated in the driver. > This should be better for maintainability, since the details of the > remapping will be arch-specific -- and in general not all arches are > going to require it. With this in mind, I think that changes in the > resctrl core code would be minimal (perhaps no changes at all). Yes, maintaining the interface of the resctrl core code unchanged is, in essence, the (first) important constraint of the current MPAM code. We try the best to keep all resctrl interfaces and ensure the existing functionality of x86 RDT. The only thing that falls short of being ideal (forgive me), is that it introduces the sole new function resctrl_arch_alloc_rmid() into the resctrl code (resctrl_arch_free_rmid() will be optimized away in the next version, and there are no other new functions any more). The resctrl_arch_alloc_rmid() is the result of several restructuring iterations and it is one of the most critical points in the patch series. > 3) How should the amount of reqPARTID grouping (your "n" parameter) be > determined, in general? > > As with (1), the right answer may depend on the use case as well as on > the hardware. > > >From my investigations into this, I feel that some configuration > parameters will probably be needed, at least at boot time. As mentioned earlier, Total number of intPARTIDs: n = min(intPARTID-np, PARTID-nnp) Total number of reqPARTIDs: l = min(reqPARTID-np, PARTID-nnp) We maximize the number of control groups because users can always choose to make a control group act as a sub-monitoring group any time. > 4) If the mapping between reqPARTIDs and (CLOSID,RMID) pairs is static, > is it necessary to track which reqPARTIDs are in use? Would it be > simpler to treat all n reqPARTIDs as permanently assigned to the > corresponding CLOSID? > > If reqPARTID usage is not tracked, then every control change on MSCs > that do not support PARTID Narrowing would need to be replicated across > all reqPARTIDs corresponding to the affected resctrl control partition. > But control changes are a relatively rare event, so this approach feels > acceptable as a way of keeping the driver complexity down. It partly > depends on how large the "n" parameter can become. Yes, totally agree. I will try to remove the reqPARTID bitmap and the resctrl_arch_free_rmid(). As mentioned, this will simplify the code logic and reduce changes to the resctrl layer code. Initially, to reduce the number of IPI interrupt, keep this resource tracking until now, and I will prioritize optimization for the next version. (In fact, the initial version of the patch set was dynamically allocated, and during the code restructuring process, it was inevitable to retain some of the original ideas.) Best regards, Zeng Heng
The patch set is applied for mpam/snapshot/v6.12-rc1 branch of https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git repository. This patch set is fully compatible with x86 RDT functionality. The narrow-partid feature in MPAM allows for a more efficient use of PARTIDs by enabling a many-to-one mapping of reqpartids (requested PARTIDs) to intpartids (internal PARTIDs). This mapping reduces the number of unique PARTIDs needed, thus allowing more tasks or processes to be monitored and managed with the available resources. Intpartid(Internal PARTID) is an internal identifier used by the hardware to represent a specific resource partition. It is a low-level identifier that the hardware uses to track and manage resource allocation and monitoring. Reqpartid(Request PARTID) is an identifier provided by the software when requesting resources from the memory system. It indicates the desired partition for resource monitoring. By using reqpartids, software can monitor specific resources or allow the system to subdivide smaller granularity partitions within existing partitions to serve as monitoring partitions. For the new rmid allocation strategy, it will check whether there is an available rmid of any reqPARTID which belongs to the input intPARTID. The MPAM driver statically assigns all reqPARTIDs to respective intPARTIDs, with a specific illustration as follows: m - Indicates the number of reqPARTIDs per intPARTID n - Indicates the total number of intPARTIDs (m * n) - Represents the total number of reqPARTIDs intPARTID_1 = 0 ├── reqPARTID_1_1 = 0 ├── reqPARTID_1_2 = 0 + n ├── ... └── reqPARTID_1_m = 0 + n * (m - 1) intPARTID_2 = 1 ├── reqPARTID_2_1 = 1 ├── reqPARTID_2_2 = 1 + n ├── ... └── reqPARTID_2_m = 1 + n * (m - 1) ... intPARTID_n = (n - 1) Each intPARTID has m reqPARTIDs, which are used to expand the number of monitoring groups under the control group. Therefore, the number of monitoring groups is no longer limited by the range of MPAM PMG, which enhances the extensibility of the system's monitoring capabilities. --- compared with v1: - Rebase this patch set on latest MPAM driver of the v6.12-rc1 branch. --- Dave Martin (1): arm_mpam: Set INTERNAL as needed when setting MSC controls Zeng Heng (5): arm_mpam: Introduce the definitions of intPARTID and reqPARTID arm_mpam: Create reqPARTIDs resource bitmap arm_mpam: Enhance the rmid allocation strategy arm_mpam: Call resctrl_sync_config() when allocate new reqPARTID fs/resctrl: Add the helper to check if the task exists in the target group arch/x86/kernel/cpu/resctrl/core.c | 20 +++ drivers/platform/arm64/mpam/mpam_devices.c | 80 +++++++++-- drivers/platform/arm64/mpam/mpam_internal.h | 6 + drivers/platform/arm64/mpam/mpam_resctrl.c | 145 +++++++++++++++++++- fs/resctrl/internal.h | 4 - fs/resctrl/monitor.c | 16 ++- fs/resctrl/pseudo_lock.c | 7 +- fs/resctrl/rdtgroup.c | 84 ++++++++---- include/linux/resctrl.h | 30 ++++ 9 files changed, 342 insertions(+), 50 deletions(-) -- 2.25.1