| Message ID | 20220705102740.29337-7-yishaih@nvidia.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Add device DMA logging support for mlx5 driver | expand |
Hi Yishai,
I love your patch! Yet something to improve:
[auto build test ERROR on awilliam-vfio/next]
[also build test ERROR on next-20220705]
[cannot apply to linus/master v5.19-rc5]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119
base: https://github.com/awilliam/linux-vfio.git next
config: openrisc-randconfig-r036-20220703 (https://download.01.org/0day-ci/archive/20220705/202207052209.x00Iykkp-lkp@intel.com/config)
compiler: or1k-linux-gcc (GCC) 11.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/12299d61a80f6adf70a2e76b8f4721f3b7bcd95a
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119
git checkout 12299d61a80f6adf70a2e76b8f4721f3b7bcd95a
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=openrisc SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
or1k-linux-ld: certs/system_keyring.o: in function `load_system_certificate_list':
system_keyring.c:(.init.text+0xc0): undefined reference to `x509_load_certificate_list'
system_keyring.c:(.init.text+0xc0): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `x509_load_certificate_list'
or1k-linux-ld: or1k-linux-ld: DWARF error: could not find abbrev number 36676422
drivers/vfio/vfio_main.o: in function `vfio_ioctl_device_feature_logging_start':
vfio_main.c:(.text+0x18c8): undefined reference to `interval_tree_iter_first'
vfio_main.c:(.text+0x18c8): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `interval_tree_iter_first'
>> or1k-linux-ld: vfio_main.c:(.text+0x18e4): undefined reference to `interval_tree_insert'
vfio_main.c:(.text+0x18e4): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `interval_tree_insert'
Hi Yishai,
I love your patch! Yet something to improve:
[auto build test ERROR on awilliam-vfio/next]
[also build test ERROR on next-20220705]
[cannot apply to linus/master v5.19-rc5]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119
base: https://github.com/awilliam/linux-vfio.git next
config: m68k-randconfig-r035-20220703 (https://download.01.org/0day-ci/archive/20220705/202207052211.ARFEDBEi-lkp@intel.com/config)
compiler: m68k-linux-gcc (GCC) 11.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/12299d61a80f6adf70a2e76b8f4721f3b7bcd95a
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119
git checkout 12299d61a80f6adf70a2e76b8f4721f3b7bcd95a
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=m68k SHELL=/bin/bash
If you fix the issue, kindly add following tag where applicable
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
m68k-linux-ld: arch/m68k/kernel/machine_kexec.o: in function `machine_kexec':
machine_kexec.c:(.text+0x58): undefined reference to `m68k_mmutype'
m68k-linux-ld: machine_kexec.c:(.text+0x60): undefined reference to `m68k_cputype'
m68k-linux-ld: arch/m68k/kernel/relocate_kernel.o:(.m68k_fixup+0x0): undefined reference to `M68K_FIXUP_MEMOFFSET'
m68k-linux-ld: arch/m68k/kernel/relocate_kernel.o:(.m68k_fixup+0x8): undefined reference to `M68K_FIXUP_MEMOFFSET'
m68k-linux-ld: drivers/vfio/vfio_main.o: in function `vfio_ioctl_device_feature_logging_start':
vfio_main.c:(.text+0x105e): undefined reference to `interval_tree_iter_first'
>> m68k-linux-ld: vfio_main.c:(.text+0x1074): undefined reference to `interval_tree_insert'
On 7/5/22 15:11, kernel test robot wrote: > Hi Yishai, > > I love your patch! Yet something to improve: > > [auto build test ERROR on awilliam-vfio/next] > [also build test ERROR on next-20220705] > [cannot apply to linus/master v5.19-rc5] > [If your patch is applied to the wrong git tree, kindly drop us a note. > And when submitting patch, we suggest to use '--base' as documented in > https://git-scm.com/docs/git-format-patch] > > url: https://github.com/intel-lab-lkp/linux/commits/Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119 > base: https://github.com/awilliam/linux-vfio.git next > config: openrisc-randconfig-r036-20220703 (https://download.01.org/0day-ci/archive/20220705/202207052209.x00Iykkp-lkp@intel.com/config) > compiler: or1k-linux-gcc (GCC) 11.3.0 > reproduce (this is a W=1 build): > wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross > chmod +x ~/bin/make.cross > # https://github.com/intel-lab-lkp/linux/commit/12299d61a80f6adf70a2e76b8f4721f3b7bcd95a > git remote add linux-review https://github.com/intel-lab-lkp/linux > git fetch --no-tags linux-review Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119 > git checkout 12299d61a80f6adf70a2e76b8f4721f3b7bcd95a > # save the config file > mkdir build_dir && cp config build_dir/.config > COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=openrisc SHELL=/bin/bash > > If you fix the issue, kindly add following tag where applicable > Reported-by: kernel test robot <lkp@intel.com> > > All errors (new ones prefixed by >>): > > or1k-linux-ld: certs/system_keyring.o: in function `load_system_certificate_list': > system_keyring.c:(.init.text+0xc0): undefined reference to `x509_load_certificate_list' > system_keyring.c:(.init.text+0xc0): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `x509_load_certificate_list' > or1k-linux-ld: or1k-linux-ld: DWARF error: could not find abbrev number 36676422 > drivers/vfio/vfio_main.o: in function `vfio_ioctl_device_feature_logging_start': > vfio_main.c:(.text+0x18c8): undefined reference to `interval_tree_iter_first' > vfio_main.c:(.text+0x18c8): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `interval_tree_iter_first' >>> or1k-linux-ld: vfio_main.c:(.text+0x18e4): undefined reference to `interval_tree_insert' > vfio_main.c:(.text+0x18e4): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `interval_tree_insert' > Sounds like this needs: select INTERVAL_TREE in drivers/vfio/pci/Kconfig VFIO_PCI_CORE
On 7/5/22 15:28, Joao Martins wrote: > On 7/5/22 15:11, kernel test robot wrote: >> Hi Yishai, >> >> I love your patch! Yet something to improve: >> >> [auto build test ERROR on awilliam-vfio/next] >> [also build test ERROR on next-20220705] >> [cannot apply to linus/master v5.19-rc5] >> [If your patch is applied to the wrong git tree, kindly drop us a note. >> And when submitting patch, we suggest to use '--base' as documented in >> https://git-scm.com/docs/git-format-patch] >> >> url: https://github.com/intel-lab-lkp/linux/commits/Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119 >> base: https://github.com/awilliam/linux-vfio.git next >> config: openrisc-randconfig-r036-20220703 (https://download.01.org/0day-ci/archive/20220705/202207052209.x00Iykkp-lkp@intel.com/config) >> compiler: or1k-linux-gcc (GCC) 11.3.0 >> reproduce (this is a W=1 build): >> wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross >> chmod +x ~/bin/make.cross >> # https://github.com/intel-lab-lkp/linux/commit/12299d61a80f6adf70a2e76b8f4721f3b7bcd95a >> git remote add linux-review https://github.com/intel-lab-lkp/linux >> git fetch --no-tags linux-review Yishai-Hadas/Add-device-DMA-logging-support-for-mlx5-driver/20220705-183119 >> git checkout 12299d61a80f6adf70a2e76b8f4721f3b7bcd95a >> # save the config file >> mkdir build_dir && cp config build_dir/.config >> COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.3.0 make.cross W=1 O=build_dir ARCH=openrisc SHELL=/bin/bash >> >> If you fix the issue, kindly add following tag where applicable >> Reported-by: kernel test robot <lkp@intel.com> >> >> All errors (new ones prefixed by >>): >> >> or1k-linux-ld: certs/system_keyring.o: in function `load_system_certificate_list': >> system_keyring.c:(.init.text+0xc0): undefined reference to `x509_load_certificate_list' >> system_keyring.c:(.init.text+0xc0): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `x509_load_certificate_list' >> or1k-linux-ld: or1k-linux-ld: DWARF error: could not find abbrev number 36676422 >> drivers/vfio/vfio_main.o: in function `vfio_ioctl_device_feature_logging_start': >> vfio_main.c:(.text+0x18c8): undefined reference to `interval_tree_iter_first' >> vfio_main.c:(.text+0x18c8): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `interval_tree_iter_first' >>>> or1k-linux-ld: vfio_main.c:(.text+0x18e4): undefined reference to `interval_tree_insert' >> vfio_main.c:(.text+0x18e4): relocation truncated to fit: R_OR1K_INSN_REL_26 against undefined symbol `interval_tree_insert' >> > > Sounds like this needs: > > select INTERVAL_TREE > > in drivers/vfio/pci/Kconfig VFIO_PCI_CORE Err... I meant drivers/vfio/Kconfig VFIO
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 2efa06b1fafa..b6dabf398251 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1862,6 +1862,11 @@ int vfio_pci_core_register_device(struct vfio_pci_core_device *vdev) return -EINVAL; } + if (vdev->vdev.log_ops && !(vdev->vdev.log_ops->log_start && + vdev->vdev.log_ops->log_stop && + vdev->vdev.log_ops->log_read_and_clear)) + return -EINVAL; + /* * Prevent binding to PFs with VFs enabled, the VFs might be in use * by the host or other users. We cannot capture the VFs if they diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index bd84ca7c5e35..2414d827e3c8 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -32,6 +32,8 @@ #include <linux/vfio.h> #include <linux/wait.h> #include <linux/sched/signal.h> +#include <linux/interval_tree.h> +#include <linux/iova_bitmap.h> #include "vfio.h" #define DRIVER_VERSION "0.3" @@ -1603,6 +1605,153 @@ static int vfio_ioctl_device_feature_migration(struct vfio_device *device, return 0; } +#define LOG_MAX_RANGES 1024 + +static int +vfio_ioctl_device_feature_logging_start(struct vfio_device *device, + u32 flags, void __user *arg, + size_t argsz) +{ + size_t minsz = + offsetofend(struct vfio_device_feature_dma_logging_control, + ranges); + struct vfio_device_feature_dma_logging_range __user *ranges; + struct vfio_device_feature_dma_logging_control control; + struct vfio_device_feature_dma_logging_range range; + struct rb_root_cached root = RB_ROOT_CACHED; + struct interval_tree_node *nodes; + u32 nnodes; + int i, ret; + + if (!device->log_ops) + return -ENOTTY; + + ret = vfio_check_feature(flags, argsz, + VFIO_DEVICE_FEATURE_SET, + sizeof(control)); + if (ret != 1) + return ret; + + if (copy_from_user(&control, arg, minsz)) + return -EFAULT; + + nnodes = control.num_ranges; + if (!nnodes || nnodes > LOG_MAX_RANGES) + return -EINVAL; + + ranges = u64_to_user_ptr(control.ranges); + nodes = kmalloc_array(nnodes, sizeof(struct interval_tree_node), + GFP_KERNEL); + if (!nodes) + return -ENOMEM; + + for (i = 0; i < nnodes; i++) { + if (copy_from_user(&range, &ranges[i], sizeof(range))) { + ret = -EFAULT; + goto end; + } + if (!IS_ALIGNED(range.iova, control.page_size) || + !IS_ALIGNED(range.length, control.page_size)) { + ret = -EINVAL; + goto end; + } + nodes[i].start = range.iova; + nodes[i].last = range.iova + range.length - 1; + if (interval_tree_iter_first(&root, nodes[i].start, + nodes[i].last)) { + /* Range overlapping */ + ret = -EINVAL; + goto end; + } + interval_tree_insert(nodes + i, &root); + } + + ret = device->log_ops->log_start(device, &root, nnodes, + &control.page_size); + if (ret) + goto end; + + if (copy_to_user(arg, &control, sizeof(control))) { + ret = -EFAULT; + device->log_ops->log_stop(device); + } + +end: + kfree(nodes); + return ret; +} + +static int +vfio_ioctl_device_feature_logging_stop(struct vfio_device *device, + u32 flags, void __user *arg, + size_t argsz) +{ + int ret; + + if (!device->log_ops) + return -ENOTTY; + + ret = vfio_check_feature(flags, argsz, + VFIO_DEVICE_FEATURE_SET, 0); + if (ret != 1) + return ret; + + return device->log_ops->log_stop(device); +} + +static int +vfio_ioctl_device_feature_logging_report(struct vfio_device *device, + u32 flags, void __user *arg, + size_t argsz) +{ + size_t minsz = + offsetofend(struct vfio_device_feature_dma_logging_report, + bitmap); + struct vfio_device_feature_dma_logging_report report; + struct iova_bitmap_iter iter; + int ret; + + if (!device->log_ops) + return -ENOTTY; + + ret = vfio_check_feature(flags, argsz, + VFIO_DEVICE_FEATURE_GET, + sizeof(report)); + if (ret != 1) + return ret; + + if (copy_from_user(&report, arg, minsz)) + return -EFAULT; + + if (report.page_size < PAGE_SIZE) + return -EINVAL; + + iova_bitmap_init(&iter.dirty, report.iova, ilog2(report.page_size)); + ret = iova_bitmap_iter_init(&iter, report.iova, report.length, + u64_to_user_ptr(report.bitmap)); + if (ret) + return ret; + + for (; !iova_bitmap_iter_done(&iter); + iova_bitmap_iter_advance(&iter)) { + ret = iova_bitmap_iter_get(&iter); + if (ret) + break; + + ret = device->log_ops->log_read_and_clear(device, + iova_bitmap_iova(&iter), + iova_bitmap_length(&iter), &iter.dirty); + + iova_bitmap_iter_put(&iter); + + if (ret) + break; + } + + iova_bitmap_iter_free(&iter); + return ret; +} + static int vfio_ioctl_device_feature(struct vfio_device *device, struct vfio_device_feature __user *arg) { @@ -1636,6 +1785,18 @@ static int vfio_ioctl_device_feature(struct vfio_device *device, return vfio_ioctl_device_feature_mig_device_state( device, feature.flags, arg->data, feature.argsz - minsz); + case VFIO_DEVICE_FEATURE_DMA_LOGGING_START: + return vfio_ioctl_device_feature_logging_start( + device, feature.flags, arg->data, + feature.argsz - minsz); + case VFIO_DEVICE_FEATURE_DMA_LOGGING_STOP: + return vfio_ioctl_device_feature_logging_stop( + device, feature.flags, arg->data, + feature.argsz - minsz); + case VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT: + return vfio_ioctl_device_feature_logging_report( + device, feature.flags, arg->data, + feature.argsz - minsz); default: if (unlikely(!device->ops->device_feature)) return -EINVAL; diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 4d26e149db81..feed84d686ec 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -14,6 +14,7 @@ #include <linux/workqueue.h> #include <linux/poll.h> #include <uapi/linux/vfio.h> +#include <linux/iova_bitmap.h> struct kvm; @@ -33,10 +34,11 @@ struct vfio_device { struct device *dev; const struct vfio_device_ops *ops; /* - * mig_ops is a static property of the vfio_device which must be set - * prior to registering the vfio_device. + * mig_ops/log_ops is a static property of the vfio_device which must + * be set prior to registering the vfio_device. */ const struct vfio_migration_ops *mig_ops; + const struct vfio_log_ops *log_ops; struct vfio_group *group; struct vfio_device_set *dev_set; struct list_head dev_set_list; @@ -104,6 +106,21 @@ struct vfio_migration_ops { enum vfio_device_mig_state *curr_state); }; +/** + * @log_start: Optional callback to ask the device start DMA logging. + * @log_stop: Optional callback to ask the device stop DMA logging. + * @log_read_and_clear: Optional callback to ask the device read + * and clear the dirty DMAs in some given range. + */ +struct vfio_log_ops { + int (*log_start)(struct vfio_device *device, + struct rb_root_cached *ranges, u32 nnodes, u64 *page_size); + int (*log_stop)(struct vfio_device *device); + int (*log_read_and_clear)(struct vfio_device *device, + unsigned long iova, unsigned long length, + struct iova_bitmap *dirty); +}; + /** * vfio_check_feature - Validate user input for the VFIO_DEVICE_FEATURE ioctl * @flags: Arg from the device_feature op
Introduce the DMA logging feature support in the vfio core layer. It includes the processing of the device start/stop/report DMA logging UAPIs and calling the relevant driver 'op' to do the work. Specifically, Upon start, the core translates the given input ranges into an interval tree, checks for unexpected overlapping, non aligned ranges and then pass the translated input to the driver for start tracking the given ranges. Upon report, the core translates the given input user space bitmap and page size into an IOVA kernel bitmap iterator. Then it iterates it and call the driver to set the corresponding bits for the dirtied pages in a specific IOVA range. Upon stop, the driver is called to stop the previous started tracking. The next patches from the series will introduce the mlx5 driver implementation for the logging ops. Signed-off-by: Yishai Hadas <yishaih@nvidia.com> --- drivers/vfio/pci/vfio_pci_core.c | 5 + drivers/vfio/vfio_main.c | 161 +++++++++++++++++++++++++++++++ include/linux/vfio.h | 21 +++- 3 files changed, 185 insertions(+), 2 deletions(-)