| Message ID | 20190306084832.57753-1-talgi@mellanox.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | net/dim: Support for multiple implementations | expand |
On Wed, 2019-03-06 at 10:48 +0200, Tal Gilboa wrote: > net_dim.h lib exposes an implementation of the DIM algorithm for dynamically-tuned interrupt > moderation for networking interfaces. > > We need the same behavior for any block CQ. The main motivation is two benefit from maximized > completion rate and reduced interrupt overhead that DIM may provide. What is a "block CQ"? How does net_dim compare to lib/irq_poll? Which approach results in the best performance and lowest latency? Thanks, Bart.
>> net_dim.h lib exposes an implementation of the DIM algorithm for dynamically-tuned interrupt >> moderation for networking interfaces. >> >> We need the same behavior for any block CQ. The main motivation is two benefit from maximized >> completion rate and reduced interrupt overhead that DIM may provide. > > What is a "block CQ"? There is no such thing... Also, this has no difference if a block/file/whatever is using the rdma cq. The naming should really be something like rdma_dim as it accounts for completions and not bytes/packets. > How does net_dim compare to lib/irq_poll? Its orthogonal, its basically adaptive interrupt moderation for RDMA devices. Its sort of below the irq_poll code. It basically configures interrupt moderation based on stats collected by the rdma driver. > Which approach results in the best performance and lowest latency? I guess it depends on what is the test case. This approach tries to apply some time or completion count limit to when the HW should fire an interrupt based on the load in an adaptive fashion. The scheme is to try and detect what are the load characteristics and come up with a moderation parameters that fit. For high interrupt rate (usually seen with small size high queue-depth workloads) it configures the device to aggregate some more before firing an interrupt - so less interrupts, better efficiency per interrupt (finds more completions). For low interrupt rate (low queue depth) the load is probably low to moderate and aggregating before firing an interrupt is just added latency for no benefit. So the algorithm tries to transition between a number of pre-defined levels according to the load it samples. This has been widely used by the network drivers for the past decade. Now, this algorithm while trying to adjust itself by learning the load, also adds entropy to the overall system performance and latency. So this is not a trivial trade-off for any workload. I took a stab at this once (came up with something very similar), and while for large queue-depth workloads I got up to 2x IOPs as the algorithm chose aggressive moderation parameters which improved the efficiency a lot, but when the workload varied the algorithm wasn't very successful detecting the load and the step direction (I used a variation of the same basic algorithm from mlx5 driver that net_dim is based on). Also, QD=1 resulted in higher latency as the algorithm was dangling between the two lowest levels. So I guess this needs to undergo a thorough performance evaluation for steady and varying workloads before we can consider this. Overall, I think its a great idea to add that to the rdma subsystem but we cannot make it the default and especially without being able to turn it off. So this needs to be opt in with a sysctl option. Moreover, not every device support cq moderation so you need to check the device capabilities before you apply any of this.
On 3/7/2019 3:56 AM, Sagi Grimberg wrote: > >>> net_dim.h lib exposes an implementation of the DIM algorithm for >>> dynamically-tuned interrupt >>> moderation for networking interfaces. >>> >>> We need the same behavior for any block CQ. The main motivation is >>> two benefit from maximized >>> completion rate and reduced interrupt overhead that DIM may provide. >> >> What is a "block CQ"? > > There is no such thing... Also, this has no difference > if a block/file/whatever is using the rdma cq. > > The naming should really be something like rdma_dim as it accounts > for completions and not bytes/packets. Sagi, I think that in the future we could use it in nvme since there is an option to set the interrupt coalescing in NVMe spec. This might improve performance for NVMe driver. We already see some bottlenecks in performance (maybe driver ones) while developing the NVMe SNAP feature in Bluefield (NVMe emulation using Smart NIC). We're trying to get 2.5-2.7 MIOPs OOB from 1 controller and it's not trivial for today's driver. So let's take this into consideration when we set the naming. > > >> How does net_dim compare to lib/irq_poll? > > Its orthogonal, its basically adaptive interrupt moderation for > RDMA devices. Its sort of below the irq_poll code. It basically > configures interrupt moderation based on stats collected by > the rdma driver. > >> Which approach results in the best performance and lowest latency? > > I guess it depends on what is the test case. This approach tries to > apply some time or completion count limit to when the HW should fire > an interrupt based on the load in an adaptive fashion. > > The scheme is to try and detect what are the load characteristics and > come up with a moderation parameters that fit. For high interrupt rate > (usually seen with small size high queue-depth workloads) it configures > the device to aggregate some more before firing an interrupt - so less > interrupts, better efficiency per interrupt (finds more completions). > For low interrupt rate (low queue depth) the load is probably low to > moderate and aggregating before firing an interrupt is just added > latency for no benefit. So the algorithm tries to transition between a > number of pre-defined levels according to the load it samples. > > This has been widely used by the network drivers for the past decade. > > Now, this algorithm while trying to adjust itself by learning the load, > also adds entropy to the overall system performance and latency. > So this is not a trivial trade-off for any workload. > > I took a stab at this once (came up with something very similar), > and while for large queue-depth workloads I got up to 2x IOPs as the > algorithm chose aggressive moderation parameters which improved the > efficiency a lot, but when the workload varied the algorithm wasn't very > successful detecting the load and the step direction (I used a variation > of the same basic algorithm from mlx5 driver that net_dim is based on). > > Also, QD=1 resulted in higher latency as the algorithm was dangling > between the two lowest levels. So I guess this needs to undergo a > thorough performance evaluation for steady and varying workloads before > we can consider this. > > Overall, I think its a great idea to add that to the rdma subsystem > but we cannot make it the default and especially without being able > to turn it off. So this needs to be opt in with a sysctl option. We can add flag in create_cq command that will try_coalescing_is_possible instead of module parameter of course. Storage ULPs can set it to True. Also in the internal review Yamin added a table that summarize all the testing that were done using NVMeoF (I guess it somehow didn't get to this RFC). I guess we can do the same for iSER to get more confidence and then set both to create modifiable cq (if HCA supports, of course). Agreed ? > > > Moreover, not every device support cq moderation so you need to check > the device capabilities before you apply any of this. for sure.
>>> What is a "block CQ"? >> >> There is no such thing... Also, this has no difference >> if a block/file/whatever is using the rdma cq. >> >> The naming should really be something like rdma_dim as it accounts >> for completions and not bytes/packets. > > Sagi, > > I think that in the future we could use it in nvme since there is an > option to set the interrupt coalescing in NVMe spec. > > This might improve performance for NVMe driver. That would require changing the spec to make moderation config per-queue and not controller-wide. This does not apply specifically to block so naming it with blk does not make sense. >> Overall, I think its a great idea to add that to the rdma subsystem >> but we cannot make it the default and especially without being able >> to turn it off. So this needs to be opt in with a sysctl option. > > We can add flag in create_cq command that will > try_coalescing_is_possible instead of module parameter of course. > > Storage ULPs can set it to True. The point is that it can't be universally on. > Also in the internal review Yamin added a table that summarize all the > testing that were done using NVMeoF (I guess it somehow didn't get to > this RFC). > > I guess we can do the same for iSER to get more confidence and then set > both to create modifiable cq (if HCA supports, of course). > > Agreed ? Sure.
On 3/14/2019 1:45 PM, Max Gurtovoy wrote: > > On 3/7/2019 3:56 AM, Sagi Grimberg wrote: >> >>>> net_dim.h lib exposes an implementation of the DIM algorithm for >>>> dynamically-tuned interrupt >>>> moderation for networking interfaces. >>>> >>>> We need the same behavior for any block CQ. The main motivation is >>>> two benefit from maximized >>>> completion rate and reduced interrupt overhead that DIM may provide. >>> >>> What is a "block CQ"? >> >> There is no such thing... Also, this has no difference >> if a block/file/whatever is using the rdma cq. >> >> The naming should really be something like rdma_dim as it accounts >> for completions and not bytes/packets. > > Sagi, > > I think that in the future we could use it in nvme since there is an > option to set the interrupt coalescing in NVMe spec. > > This might improve performance for NVMe driver. > > We already see some bottlenecks in performance (maybe driver ones) > while developing the NVMe SNAP feature in Bluefield (NVMe emulation > using Smart NIC). > > We're trying to get 2.5-2.7 MIOPs OOB from 1 controller and it's not > trivial for today's driver. > > So let's take this into consideration when we set the naming. > > I agree that blk is not the most successful name, we were trying to find something that would work for general storage applications. I think rdma_dim would work as it is completion based but then when we want to use it for nvme it will probably require code duplication. >> >> >>> How does net_dim compare to lib/irq_poll? >> >> Its orthogonal, its basically adaptive interrupt moderation for >> RDMA devices. Its sort of below the irq_poll code. It basically >> configures interrupt moderation based on stats collected by >> the rdma driver. >> >>> Which approach results in the best performance and lowest latency? >> >> I guess it depends on what is the test case. This approach tries to >> apply some time or completion count limit to when the HW should fire >> an interrupt based on the load in an adaptive fashion. >> >> The scheme is to try and detect what are the load characteristics and >> come up with a moderation parameters that fit. For high interrupt rate >> (usually seen with small size high queue-depth workloads) it configures >> the device to aggregate some more before firing an interrupt - so less >> interrupts, better efficiency per interrupt (finds more completions). >> For low interrupt rate (low queue depth) the load is probably low to >> moderate and aggregating before firing an interrupt is just added >> latency for no benefit. So the algorithm tries to transition between a >> number of pre-defined levels according to the load it samples. >> >> This has been widely used by the network drivers for the past decade. >> >> Now, this algorithm while trying to adjust itself by learning the load, >> also adds entropy to the overall system performance and latency. >> So this is not a trivial trade-off for any workload. >> >> I took a stab at this once (came up with something very similar), >> and while for large queue-depth workloads I got up to 2x IOPs as the >> algorithm chose aggressive moderation parameters which improved the >> efficiency a lot, but when the workload varied the algorithm wasn't very >> successful detecting the load and the step direction (I used a variation >> of the same basic algorithm from mlx5 driver that net_dim is based on). >> >> Also, QD=1 resulted in higher latency as the algorithm was dangling >> between the two lowest levels. So I guess this needs to undergo a >> thorough performance evaluation for steady and varying workloads before >> we can consider this. >> >> Overall, I think its a great idea to add that to the rdma subsystem >> but we cannot make it the default and especially without being able >> to turn it off. So this needs to be opt in with a sysctl option. > > We can add flag in create_cq command that will > try_coalescing_is_possible instead of module parameter of course. > > Storage ULPs can set it to True. > > Also in the internal review Yamin added a table that summarize all the > testing that were done using NVMeoF (I guess it somehow didn't get to > this RFC). > > I guess we can do the same for iSER to get more confidence and then > set both to create modifiable cq (if HCA supports, of course). > > Agreed ? > I think that adding a flag in create_cq will be less clean as it will require more work for anyone writing applications that should not have to consider this feature. Based on the results I saw during testing I would set it to work by default as I could not find a use case where it significantly reduces performance and in many cases it is a large improvement. It should be more of an opt out situation. Performance improvement (ConnectX-5 100GbE, x86) running FIO benchmark over NVMf between two equal end-hosts with 56 cores across a Mellanox switch using null_blk device: IO READS before: blk size | BW | IOPS | 99th percentile latency 512B | 3.2GiB | 6.6M | 1549 usec 4k | 7.2GiB | 1.8M | 7177 usec 64k | 10.7GiB | 176k | 82314 usec IO READS after: blk size | BW | IOPS | 99th percentile latency 512B | 4.2GiB | 8.6M | 1729 usec 4k | 10.5GiB | 2.7M | 5669 usec 64k | 10.7GiB | 176k | 102000 usec IO WRITES before: blk size | BW | IOPS | 99th percentile latency 512B | 3GiB | 6.2M | 2573 usec 4k | 7.2GiB | 1.8M | 5342 usec 64k | 10.7GiB | 176k | 62129 usec IO WRITES after: blk size | BW | IOPS | 99th percentile latency 512B | 4.2GiB | 8.6M | 938 usec 4k | 10.2GiB | 2.68M | 2769 usec 64k | 10.6GiB | 173k | 87557 usec It doesn't really make a difference to me how the option is implemented but I think it makes more sense to have it dealt with by us such as in a module parameter and not something like a flag that has a larger radius of effect. >> >> >> Moreover, not every device support cq moderation so you need to check >> the device capabilities before you apply any of this. > > for sure. > >
On 3/18/2019 11:24 AM, Yamin Friedman wrote: > On 3/14/2019 1:45 PM, Max Gurtovoy wrote: >> On 3/7/2019 3:56 AM, Sagi Grimberg wrote: >>>>> net_dim.h lib exposes an implementation of the DIM algorithm for >>>>> dynamically-tuned interrupt >>>>> moderation for networking interfaces. >>>>> >>>>> We need the same behavior for any block CQ. The main motivation is >>>>> two benefit from maximized >>>>> completion rate and reduced interrupt overhead that DIM may provide. >>>> What is a "block CQ"? >>> There is no such thing... Also, this has no difference >>> if a block/file/whatever is using the rdma cq. >>> >>> The naming should really be something like rdma_dim as it accounts >>> for completions and not bytes/packets. >> Sagi, >> >> I think that in the future we could use it in nvme since there is an >> option to set the interrupt coalescing in NVMe spec. >> >> This might improve performance for NVMe driver. >> >> We already see some bottlenecks in performance (maybe driver ones) >> while developing the NVMe SNAP feature in Bluefield (NVMe emulation >> using Smart NIC). >> >> We're trying to get 2.5-2.7 MIOPs OOB from 1 controller and it's not >> trivial for today's driver. >> >> So let's take this into consideration when we set the naming. >> >> > I agree that blk is not the most successful name, we were trying to find > something that would work for general storage applications. I think > rdma_dim would work as it is completion based but then when we want to > use it for nvme it will probably require code duplication. agreed for rdma_dim. Yamin/Idan, let's discuss internally regarding Sagi's proposal for moderation/coalescing per NVMe queue (and not per controller). maybe need to update specification. > >>> >>>> How does net_dim compare to lib/irq_poll? >>> Its orthogonal, its basically adaptive interrupt moderation for >>> RDMA devices. Its sort of below the irq_poll code. It basically >>> configures interrupt moderation based on stats collected by >>> the rdma driver. >>> >>>> Which approach results in the best performance and lowest latency? >>> I guess it depends on what is the test case. This approach tries to >>> apply some time or completion count limit to when the HW should fire >>> an interrupt based on the load in an adaptive fashion. >>> >>> The scheme is to try and detect what are the load characteristics and >>> come up with a moderation parameters that fit. For high interrupt rate >>> (usually seen with small size high queue-depth workloads) it configures >>> the device to aggregate some more before firing an interrupt - so less >>> interrupts, better efficiency per interrupt (finds more completions). >>> For low interrupt rate (low queue depth) the load is probably low to >>> moderate and aggregating before firing an interrupt is just added >>> latency for no benefit. So the algorithm tries to transition between a >>> number of pre-defined levels according to the load it samples. >>> >>> This has been widely used by the network drivers for the past decade. >>> >>> Now, this algorithm while trying to adjust itself by learning the load, >>> also adds entropy to the overall system performance and latency. >>> So this is not a trivial trade-off for any workload. >>> >>> I took a stab at this once (came up with something very similar), >>> and while for large queue-depth workloads I got up to 2x IOPs as the >>> algorithm chose aggressive moderation parameters which improved the >>> efficiency a lot, but when the workload varied the algorithm wasn't very >>> successful detecting the load and the step direction (I used a variation >>> of the same basic algorithm from mlx5 driver that net_dim is based on). >>> >>> Also, QD=1 resulted in higher latency as the algorithm was dangling >>> between the two lowest levels. So I guess this needs to undergo a >>> thorough performance evaluation for steady and varying workloads before >>> we can consider this. >>> >>> Overall, I think its a great idea to add that to the rdma subsystem >>> but we cannot make it the default and especially without being able >>> to turn it off. So this needs to be opt in with a sysctl option. >> We can add flag in create_cq command that will >> try_coalescing_is_possible instead of module parameter of course. >> >> Storage ULPs can set it to True. >> >> Also in the internal review Yamin added a table that summarize all the >> testing that were done using NVMeoF (I guess it somehow didn't get to >> this RFC). >> >> I guess we can do the same for iSER to get more confidence and then >> set both to create modifiable cq (if HCA supports, of course). >> >> Agreed ? >> > I think that adding a flag in create_cq will be less clean as it will > require more work for anyone writing applications that should not have > to consider this feature. As we discussed, let's check with RDMA maintainers if it's better to extend alloc_cq API or create alloc_cq_dim API function. Sagi/Christoph, how about adding a module param per ULP ? as we use register_always today, create a use_dimm module param for iSER/NVMe-RDMA ? > > Based on the results I saw during testing I would set it to work by > default as I could not find a use case where it significantly reduces > performance and in many cases it is a large improvement. It should be > more of an opt out situation. > > Performance improvement (ConnectX-5 100GbE, x86) running FIO benchmark over > NVMf between two equal end-hosts with 56 cores across a Mellanox switch > using null_blk device: > > IO READS before: > blk size | BW | IOPS | 99th percentile latency > 512B | 3.2GiB | 6.6M | 1549 usec > 4k | 7.2GiB | 1.8M | 7177 usec > 64k | 10.7GiB | 176k | 82314 usec > > IO READS after: > blk size | BW | IOPS | 99th percentile latency > 512B | 4.2GiB | 8.6M | 1729 usec > 4k | 10.5GiB | 2.7M | 5669 usec > 64k | 10.7GiB | 176k | 102000 usec > > IO WRITES before: > blk size | BW | IOPS | 99th percentile latency > 512B | 3GiB | 6.2M | 2573 usec > 4k | 7.2GiB | 1.8M | 5342 usec > 64k | 10.7GiB | 176k | 62129 usec > > IO WRITES after: > blk size | BW | IOPS | 99th percentile latency > 512B | 4.2GiB | 8.6M | 938 usec > 4k | 10.2GiB | 2.68M | 2769 usec > 64k | 10.6GiB | 173k | 87557 usec > > It doesn't really make a difference to me how the option is implemented > but I think it makes more sense to have it dealt with by us such as in a > module parameter and not something like a flag that has a larger radius > of effect. > >>> >>> Moreover, not every device support cq moderation so you need to check >>> the device capabilities before you apply any of this. >> for sure. >> >>
On 3/18/2019 1:08 PM, Max Gurtovoy wrote: > > On 3/18/2019 11:24 AM, Yamin Friedman wrote: >> On 3/14/2019 1:45 PM, Max Gurtovoy wrote: >>> On 3/7/2019 3:56 AM, Sagi Grimberg wrote: >>>>>> net_dim.h lib exposes an implementation of the DIM algorithm for >>>>>> dynamically-tuned interrupt >>>>>> moderation for networking interfaces. >>>>>> >>>>>> We need the same behavior for any block CQ. The main motivation is >>>>>> two benefit from maximized >>>>>> completion rate and reduced interrupt overhead that DIM may provide. >>>>> What is a "block CQ"? >>>> There is no such thing... Also, this has no difference >>>> if a block/file/whatever is using the rdma cq. >>>> >>>> The naming should really be something like rdma_dim as it accounts >>>> for completions and not bytes/packets. >>> Sagi, >>> >>> I think that in the future we could use it in nvme since there is an >>> option to set the interrupt coalescing in NVMe spec. >>> >>> This might improve performance for NVMe driver. >>> >>> We already see some bottlenecks in performance (maybe driver ones) >>> while developing the NVMe SNAP feature in Bluefield (NVMe emulation >>> using Smart NIC). >>> >>> We're trying to get 2.5-2.7 MIOPs OOB from 1 controller and it's not >>> trivial for today's driver. >>> >>> So let's take this into consideration when we set the naming. >>> >>> >> I agree that blk is not the most successful name, we were trying to find >> something that would work for general storage applications. I think >> rdma_dim would work as it is completion based but then when we want to >> use it for nvme it will probably require code duplication. > > agreed for rdma_dim. > > Yamin/Idan, > > let's discuss internally regarding Sagi's proposal for > moderation/coalescing per NVMe queue (and not per controller). > > maybe need to update specification. > > >> >>>> >>>>> How does net_dim compare to lib/irq_poll? >>>> Its orthogonal, its basically adaptive interrupt moderation for >>>> RDMA devices. Its sort of below the irq_poll code. It basically >>>> configures interrupt moderation based on stats collected by >>>> the rdma driver. >>>> >>>>> Which approach results in the best performance and lowest latency? >>>> I guess it depends on what is the test case. This approach tries to >>>> apply some time or completion count limit to when the HW should fire >>>> an interrupt based on the load in an adaptive fashion. >>>> >>>> The scheme is to try and detect what are the load characteristics and >>>> come up with a moderation parameters that fit. For high interrupt rate >>>> (usually seen with small size high queue-depth workloads) it >>>> configures >>>> the device to aggregate some more before firing an interrupt - so less >>>> interrupts, better efficiency per interrupt (finds more completions). >>>> For low interrupt rate (low queue depth) the load is probably low to >>>> moderate and aggregating before firing an interrupt is just added >>>> latency for no benefit. So the algorithm tries to transition between a >>>> number of pre-defined levels according to the load it samples. >>>> >>>> This has been widely used by the network drivers for the past decade. >>>> >>>> Now, this algorithm while trying to adjust itself by learning the >>>> load, >>>> also adds entropy to the overall system performance and latency. >>>> So this is not a trivial trade-off for any workload. >>>> >>>> I took a stab at this once (came up with something very similar), >>>> and while for large queue-depth workloads I got up to 2x IOPs as the >>>> algorithm chose aggressive moderation parameters which improved the >>>> efficiency a lot, but when the workload varied the algorithm wasn't >>>> very >>>> successful detecting the load and the step direction (I used a >>>> variation >>>> of the same basic algorithm from mlx5 driver that net_dim is based >>>> on). >>>> >>>> Also, QD=1 resulted in higher latency as the algorithm was dangling >>>> between the two lowest levels. So I guess this needs to undergo a >>>> thorough performance evaluation for steady and varying workloads >>>> before >>>> we can consider this. >>>> >>>> Overall, I think its a great idea to add that to the rdma subsystem >>>> but we cannot make it the default and especially without being able >>>> to turn it off. So this needs to be opt in with a sysctl option. >>> We can add flag in create_cq command that will >>> try_coalescing_is_possible instead of module parameter of course. >>> >>> Storage ULPs can set it to True. >>> >>> Also in the internal review Yamin added a table that summarize all the >>> testing that were done using NVMeoF (I guess it somehow didn't get to >>> this RFC). >>> >>> I guess we can do the same for iSER to get more confidence and then >>> set both to create modifiable cq (if HCA supports, of course). >>> >>> Agreed ? >>> >> I think that adding a flag in create_cq will be less clean as it will >> require more work for anyone writing applications that should not have >> to consider this feature. > > > As we discussed, let's check with RDMA maintainers if it's better to > extend alloc_cq API or create alloc_cq_dim API function. > > Sagi/Christoph, > > how about adding a module param per ULP ? as we use register_always > today, create a use_dimm module param for iSER/NVMe-RDMA ? another option is to add it to each ULP user space utility (nvme-cli, iscsiadm, etc...) thoughts ? > > >> >> Based on the results I saw during testing I would set it to work by >> default as I could not find a use case where it significantly reduces >> performance and in many cases it is a large improvement. It should be >> more of an opt out situation. >> >> Performance improvement (ConnectX-5 100GbE, x86) running FIO >> benchmark over >> NVMf between two equal end-hosts with 56 cores across a >> Mellanox switch >> using null_blk device: >> >> IO READS before: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 3.2GiB | 6.6M | 1549 usec >> 4k | 7.2GiB | 1.8M | 7177 usec >> 64k | 10.7GiB | 176k | 82314 usec >> >> IO READS after: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 4.2GiB | 8.6M | 1729 usec >> 4k | 10.5GiB | 2.7M | 5669 usec >> 64k | 10.7GiB | 176k | 102000 usec >> >> IO WRITES before: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 3GiB | 6.2M | 2573 usec >> 4k | 7.2GiB | 1.8M | 5342 usec >> 64k | 10.7GiB | 176k | 62129 usec >> >> IO WRITES after: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 4.2GiB | 8.6M | 938 usec >> 4k | 10.2GiB | 2.68M | 2769 usec >> 64k | 10.6GiB | 173k | 87557 usec >> >> It doesn't really make a difference to me how the option is implemented >> but I think it makes more sense to have it dealt with by us such as in a >> module parameter and not something like a flag that has a larger radius >> of effect. >> >>>> >>>> Moreover, not every device support cq moderation so you need to check >>>> the device capabilities before you apply any of this. >>> for sure. >>> >>>
> I agree that blk is not the most successful name, we were trying to find > something that would work for general storage applications. I think > rdma_dim would work as it is completion based but then when we want to > use it for nvme it will probably require code duplication. Lets worry about reuse NVMe when it is actually applicable. >> Also in the internal review Yamin added a table that summarize all the >> testing that were done using NVMeoF (I guess it somehow didn't get to >> this RFC). >> >> I guess we can do the same for iSER to get more confidence and then >> set both to create modifiable cq (if HCA supports, of course). >> >> Agreed ? >> > I think that adding a flag in create_cq will be less clean as it will > require more work for anyone writing applications that should not have > to consider this feature. > > Based on the results I saw during testing I would set it to work by > default as I could not find a use case where it significantly reduces > performance and in many cases it is a large improvement. It should be > more of an opt out situation. By detailed performance results I meant: 1. Full latency histogram for QD=1 both for single queue and multi-queue (including max, 99.99% and 99.999% percentiles) 2. latency vs. IOPs graph/table for both single queue and multi-queue 3. At least some measurement/analysis of how well the algorithm is adapting to workload change dynamically and how quickly. 4. Test also with real NVMe devices. Also, we need to separate the host side moderation and the target side moderation to understand if/how they effect each other. Its very easy to show that for high stress workloads you can get an improvement as obviously there is a clear win for interrupt moderation, however, if this was the only metric that is interesting, we wouldn't need it to be adaptive. As I said before, this adds entropy to the equation which in certain use cases can make more harm than good, and we need to quantify where is the impact and understand how important they are compared to the extremely niche use-case of a single host pushing 2M-8M IOPs. > Performance improvement (ConnectX-5 100GbE, x86) running FIO benchmark over > NVMf between two equal end-hosts with 56 cores across a Mellanox switch > using null_blk device: > > IO READS before: > blk size | BW | IOPS | 99th percentile latency > 512B | 3.2GiB | 6.6M | 1549 usec > 4k | 7.2GiB | 1.8M | 7177 usec > 64k | 10.7GiB | 176k | 82314 usec I've seen this before, why are we not getting 100Gb/s for 4k with CX5? I recall we used to get it with CX4. > IO READS after: > blk size | BW | IOPS | 99th percentile latency > 512B | 4.2GiB | 8.6M | 1729 usec > 4k | 10.5GiB | 2.7M | 5669 usec > 64k | 10.7GiB | 176k | 102000 usec > > IO WRITES before: > blk size | BW | IOPS | 99th percentile latency > 512B | 3GiB | 6.2M | 2573 usec > 4k | 7.2GiB | 1.8M | 5342 usec > 64k | 10.7GiB | 176k | 62129 usec > > IO WRITES after: > blk size | BW | IOPS | 99th percentile latency > 512B | 4.2GiB | 8.6M | 938 usec > 4k | 10.2GiB | 2.68M | 2769 usec > 64k | 10.6GiB | 173k | 87557 usec The fact that the 64k 99% latency is substantially higher (20+ milliseconds) without any BW benefit, while its not a very interesting measurement, gives me an indication that a more detailed analysis needs to be made here to understand where are the trade-offs. > It doesn't really make a difference to me how the option is implemented > but I think it makes more sense to have it dealt with by us such as in a > module parameter and not something like a flag that has a larger radius > of effect. I was suggesting a sysctl global parameter for global behavior and of someone wants to override it it can add a CQ flag (which follows the common net params exactly).
> As we discussed, let's check with RDMA maintainers if it's better to > extend alloc_cq API or create alloc_cq_dim API function. > > Sagi/Christoph, > > how about adding a module param per ULP ? as we use register_always > today, create a use_dimm module param for iSER/NVMe-RDMA ? I would say that its better (and simpler) to do a global sysctl knob for it. No need for a per-ULP param for a starting point.
On 3/18/2019 11:34 PM, Sagi Grimberg wrote: > >> As we discussed, let's check with RDMA maintainers if it's better to >> extend alloc_cq API or create alloc_cq_dim API function. >> >> Sagi/Christoph, >> >> how about adding a module param per ULP ? as we use register_always >> today, create a use_dimm module param for iSER/NVMe-RDMA ? > > I would say that its better (and simpler) to do a global sysctl knob for > it. No need for a per-ULP param for a starting point. for sure it's simpler, but in this case all should agree that in case we run more than 1 ULP in a server, all of them will have the same configuration (no QoS). are you suggesting something like: /proc/sys/rdma/mlx5_0/dim Jason/Leon/Doug, thoughts about the best way to configure this feature from your perspective ?
On Wed, Mar 20, 2019 at 11:17:36AM +0200, Max Gurtovoy wrote: > > On 3/18/2019 11:34 PM, Sagi Grimberg wrote: > > > > > As we discussed, let's check with RDMA maintainers if it's better to > > > extend alloc_cq API or create alloc_cq_dim API function. > > > > > > Sagi/Christoph, > > > > > > how about adding a module param per ULP ? as we use register_always > > > today, create a use_dimm module param for iSER/NVMe-RDMA ? > > > > I would say that its better (and simpler) to do a global sysctl knob for > > it. No need for a per-ULP param for a starting point. > > for sure it's simpler, but in this case all should agree that in case we run > more than 1 ULP in a server, all of them will have the same configuration > (no QoS). > > are you suggesting something like: > > /proc/sys/rdma/mlx5_0/dim > > Jason/Leon/Doug, > > thoughts about the best way to configure this feature from your perspective > ? It doesn't sound reasonable that we have per-ULP feature (DIM is per-ULP) to be configured globally. Especially given the fact that users can find themselves running different workloads with different requirements on the same system. Currently each ULP has some sort of tool to configure itself and I think that once ULP is converted to use DIM, it should have on/off knob in the tool used by their users. Thanks >
> It doesn't sound reasonable that we have per-ULP feature (DIM is > per-ULP) to be configured globally. Especially given the fact that > users can find themselves running different workloads with different > requirements on the same system. > > Currently each ULP has some sort of tool to configure itself and I think > that once ULP is converted to use DIM, it should have on/off knob in the > tool used by their users. Its not any different than socket options that can have a global sysctl knob that can be overridden by individual socket consumers.
On 3/18/2019 11:32 PM, Sagi Grimberg wrote: > >> I agree that blk is not the most successful name, we were trying to find >> something that would work for general storage applications. I think >> rdma_dim would work as it is completion based but then when we want to >> use it for nvme it will probably require code duplication. > > Lets worry about reuse NVMe when it is actually applicable. > >>> Also in the internal review Yamin added a table that summarize all the >>> testing that were done using NVMeoF (I guess it somehow didn't get to >>> this RFC). >>> >>> I guess we can do the same for iSER to get more confidence and then >>> set both to create modifiable cq (if HCA supports, of course). >>> >>> Agreed ? >>> >> I think that adding a flag in create_cq will be less clean as it will >> require more work for anyone writing applications that should not have >> to consider this feature. >> >> Based on the results I saw during testing I would set it to work by >> default as I could not find a use case where it significantly reduces >> performance and in many cases it is a large improvement. It should be >> more of an opt out situation. > > By detailed performance results I meant: > 1. Full latency histogram for QD=1 both for single queue and multi-queue > (including max, 99.99% and 99.999% percentiles) > 2. latency vs. IOPs graph/table for both single queue and multi-queue > 3. At least some measurement/analysis of how well the algorithm is > adapting to workload change dynamically and how quickly. > 4. Test also with real NVMe devices. > > Also, we need to separate the host side moderation and the target > side moderation to understand if/how they effect each other. > > Its very easy to show that for high stress workloads you can get an > improvement as obviously there is a clear win for interrupt moderation, > however, if this was the only metric that is interesting, we wouldn't > need it to be adaptive. > > As I said before, this adds entropy to the equation which in certain use > cases can make more harm than good, and we need to quantify where is the > impact and understand how important they are compared to the extremely > niche use-case of a single host pushing 2M-8M IOPs. I ran extensive tests between two hosts using FIO and NVMeoF with and without DIM on each side separately and together. I am attaching the results of the tests. The main point I would like to make is that when used on both sides especially we see improvements across the board in both IOPS and latency. > >> Performance improvement (ConnectX-5 100GbE, x86) running FIO >> benchmark over >> NVMf between two equal end-hosts with 56 cores across a >> Mellanox switch >> using null_blk device: >> >> IO READS before: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 3.2GiB | 6.6M | 1549 usec >> 4k | 7.2GiB | 1.8M | 7177 usec >> 64k | 10.7GiB | 176k | 82314 usec > > I've seen this before, why are we not getting 100Gb/s for 4k with CX5? > I recall we used to get it with CX4. It depends on the host server and whether or not it causes backpressure on the PCI because of the amount of interrupts. > >> IO READS after: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 4.2GiB | 8.6M | 1729 usec >> 4k | 10.5GiB | 2.7M | 5669 usec >> 64k | 10.7GiB | 176k | 102000 usec >> >> IO WRITES before: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 3GiB | 6.2M | 2573 usec >> 4k | 7.2GiB | 1.8M | 5342 usec >> 64k | 10.7GiB | 176k | 62129 usec >> >> IO WRITES after: >> blk size | BW | IOPS | 99th percentile latency >> 512B | 4.2GiB | 8.6M | 938 usec >> 4k | 10.2GiB | 2.68M | 2769 usec >> 64k | 10.6GiB | 173k | 87557 usec > > The fact that the 64k 99% latency is substantially higher (20+ > milliseconds) without any BW benefit, while its not a very interesting > measurement, gives me an indication that a more detailed analysis needs > to be made here to understand where are the trade-offs. > >> It doesn't really make a difference to me how the option is implemented >> but I think it makes more sense to have it dealt with by us such as in a >> module parameter and not something like a flag that has a larger radius >> of effect. > > I was suggesting a sysctl global parameter for global behavior and of > someone wants to override it it can add a CQ flag (which follows the > common net params exactly). With DIM Without DIM Type=read,Size=512B,NumJobs=1,depth=1 Type=read,Size=512B,NumJobs=1,depth=1 read: IOPS=56.7k, BW=27.7MiB/s (29.0MB/s)(277MiB/10001msec | read: IOPS=47.0k, BW=22.0MiB/s (24.1MB/s)(230MiB/10001msec slat (nsec): min=2087, max=43251, avg=2462.19, stdev=1301 | slat (nsec): min=1981, max=53196, avg=3048.29, stdev=2391 clat (nsec): min=623, max=108655, avg=14747.79, stdev=468 | clat (nsec): min=442, max=107156, avg=17658.37, stdev=647 lat (usec): min=12, max=110, avg=17.25, stdev= 5.05 | lat (usec): min=13, max=109, avg=20.78, stdev= 7.18 | 1.00th=[12992], 5.00th=[12992], 10.00th=[13120], 20. | | 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20. | 30.00th=[13120], 40.00th=[13248], 50.00th=[13248], 60. | | 30.00th=[14144], 40.00th=[14144], 50.00th=[14400], 60. | 70.00th=[13504], 80.00th=[13888], 90.00th=[16768], 95. | | 70.00th=[17024], 80.00th=[19072], 90.00th=[27776], 95. | 99.00th=[35584], 99.50th=[36608], 99.90th=[42240], 99. | | 99.00th=[40704], 99.50th=[45824], 99.90th=[51456], 99. | 99.99th=[53504] | | 99.99th=[58624] Type=read,Size=512B,NumJobs=1,depth=16 Type=read,Size=512B,NumJobs=1,depth=16 read: IOPS=299k, BW=146MiB/s (153MB/s)(1459MiB/10001msec) | read: IOPS=334k, BW=163MiB/s (171MB/s)(1629MiB/10001msec) slat (nsec): min=1634, max=45257, avg=1958.38, stdev=657. | slat (nsec): min=1564, max=28780, avg=2162.48, stdev=588. clat (usec): min=38, max=171, avg=51.28, stdev=13.39 | clat (usec): min=17, max=118, avg=45.50, stdev= 6.30 lat (usec): min=40, max=172, avg=53.27, stdev=13.41 | lat (usec): min=20, max=120, avg=47.70, stdev= 6.64 | 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 41], 20. | | 1.00th=[41728], 5.00th=[42752], 10.00th=[42752], 20. | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 46], 60. | | 30.00th=[43264], 40.00th=[43776], 50.00th=[43776], 60. | 70.00th=[ 57], 80.00th=[ 62], 90.00th=[ 70], 95. | | 70.00th=[44288], 80.00th=[44288], 90.00th=[46336], 95. | 99.00th=[ 97], 99.50th=[ 109], 99.90th=[ 127], 99. | | 99.00th=[72192], 99.50th=[73216], 99.90th=[77312], 99. | 99.99th=[ 149] | | 99.99th=[86528] Type=read,Size=512B,NumJobs=1,depth=64 Type=read,Size=512B,NumJobs=1,depth=64 read: IOPS=370k, BW=181MiB/s (189MB/s)(1806MiB/10001msec) | read: IOPS=344k, BW=168MiB/s (176MB/s)(1682MiB/10001msec) slat (nsec): min=1601, max=43074, avg=1927.33, stdev=232. | slat (nsec): min=1607, max=30650, avg=2093.56, stdev=233. clat (usec): min=56, max=378, avg=170.80, stdev= 4.52 | clat (usec): min=19, max=238, avg=183.42, stdev= 2.22 lat (usec): min=57, max=380, avg=172.77, stdev= 4.55 | lat (usec): min=21, max=241, avg=185.55, stdev= 2.23 | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20. | | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 182], 20. | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60. | | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 184], 60. | 70.00th=[ 172], 80.00th=[ 172], 90.00th=[ 176], 95. | | 70.00th=[ 184], 80.00th=[ 186], 90.00th=[ 186], 95. | 99.00th=[ 184], 99.50th=[ 194], 99.90th=[ 221], 99. | | 99.00th=[ 190], 99.50th=[ 192], 99.90th=[ 196], 99. | 99.99th=[ 245] | | 99.99th=[ 210] Type=read,Size=512B,NumJobs=16,depth=1 Type=read,Size=512B,NumJobs=16,depth=1 read: IOPS=917k, BW=448MiB/s (469MB/s)(4478MiB/10002msec) | read: IOPS=907k, BW=443MiB/s (465MB/s)(4431MiB/10001msec) slat (nsec): min=2003, max=73317, avg=2440.64, stdev=588. | slat (nsec): min=1925, max=69082, avg=2505.79, stdev=869. clat (nsec): min=597, max=172124, avg=14561.65, stdev=210 | clat (nsec): min=476, max=112521, avg=14672.56, stdev=182 lat (usec): min=11, max=174, avg=17.05, stdev= 2.20 | lat (usec): min=12, max=131, avg=17.23, stdev= 2.08 | 1.00th=[12096], 5.00th=[13120], 10.00th=[13376], 20. | | 1.00th=[11584], 5.00th=[13248], 10.00th=[13376], 20. | 30.00th=[13888], 40.00th=[14016], 50.00th=[14144], 60. | | 30.00th=[13888], 40.00th=[14016], 50.00th=[14272], 60. | 70.00th=[14528], 80.00th=[14912], 90.00th=[15680], 95. | | 70.00th=[14784], 80.00th=[15296], 90.00th=[16512], 95. | 99.00th=[27008], 99.50th=[28288], 99.90th=[33536], 99. | | 99.00th=[22400], 99.50th=[24448], 99.90th=[32384], 99. | 99.99th=[35584] | 99.99th=[35584] Type=read,Size=512B,NumJobs=16,depth=16 Type=read,Size=512B,NumJobs=16,depth=16 read: IOPS=4627k, BW=2259MiB/s (2369MB/s)(22.1GiB/10002mse | read: IOPS=4055k, BW=1980MiB/s (2076MB/s)(19.3GiB/10003mse slat (nsec): min=1620, max=95729, avg=2408.03, stdev=483. | slat (nsec): min=1572, max=239521, avg=2836.25, stdev=282 clat (usec): min=22, max=466, avg=52.52, stdev= 8.65 | clat (usec): min=9, max=616, avg=59.85, stdev=55.40 lat (usec): min=24, max=474, avg=54.98, stdev= 8.95 | lat (usec): min=15, max=630, avg=62.74, stdev=57.98 | 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20. | | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 53], 60. | | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60. | 70.00th=[ 57], 80.00th=[ 59], 90.00th=[ 61], 95. | | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 60], 95. | 99.00th=[ 73], 99.50th=[ 79], 99.90th=[ 139], 99. | | 99.00th=[ 326], 99.50th=[ 343], 99.90th=[ 367], 99. | 99.99th=[ 281] | | 99.99th=[ 519] Type=read,Size=512B,NumJobs=16,depth=64 Type=read,Size=512B,NumJobs=16,depth=64 read: IOPS=4878k, BW=2382MiB/s (2497MB/s)(23.3GiB/10001mse | read: IOPS=3451k, BW=1685MiB/s (1767MB/s)(16.5GiB/10001mse slat (nsec): min=1611, max=181900, avg=2357.92, stdev=159 | slat (nsec): min=1569, max=204726, avg=3322.74, stdev=274 clat (usec): min=28, max=934, avg=207.19, stdev=38.57 | clat (usec): min=5, max=1543, avg=292.82, stdev=220.59 lat (usec): min=29, max=944, avg=209.60, stdev=39.02 | lat (usec): min=14, max=1561, avg=296.20, stdev=223.03 | 1.00th=[ 172], 5.00th=[ 174], 10.00th=[ 174], 20. | | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20. | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 202], 60. | | 30.00th=[ 188], 40.00th=[ 227], 50.00th=[ 243], 60. | 70.00th=[ 215], 80.00th=[ 233], 90.00th=[ 251], 95. | | 70.00th=[ 249], 80.00th=[ 285], 90.00th=[ 685], 95. | 99.00th=[ 314], 99.50th=[ 322], 99.90th=[ 338], 99. | | 99.00th=[ 1270], 99.50th=[ 1319], 99.90th=[ 1385], 99. | 99.99th=[ 783] | | 99.99th=[ 1450] Type=read,Size=512B,NumJobs=56,depth=1 Type=read,Size=512B,NumJobs=56,depth=1 read: IOPS=2311k, BW=1128MiB/s (1183MB/s)(11.0GiB/10002mse | read: IOPS=2351k, BW=1148MiB/s (1204MB/s)(11.2GiB/10002mse slat (usec): min=2, max=860, avg= 3.68, stdev= 2.24 | slat (nsec): min=1744, max=502891, avg=3525.21, stdev=199 clat (nsec): min=604, max=4342.7k, avg=19742.21, stdev=70 | clat (nsec): min=456, max=781278, avg=19563.02, stdev=418 lat (usec): min=11, max=4366, avg=23.53, stdev= 7.52 | lat (usec): min=11, max=785, avg=23.18, stdev= 4.79 | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20. | | 1.00th=[13120], 5.00th=[14912], 10.00th=[15936], 20. | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17792], 40.00th=[18304], 50.00th=[19072], 60. | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20608], 80.00th=[21632], 90.00th=[23680], 95. | 99.00th=[ 35], 99.50th=[ 39], 99.90th=[ 49], 99. | | 99.00th=[31872], 99.50th=[35072], 99.90th=[41728], 99. | 99.99th=[ 330] | | 99.99th=[56064] Type=read,Size=512B,NumJobs=56,depth=16 Type=read,Size=512B,NumJobs=56,depth=16 read: IOPS=8152k, BW=3981MiB/s (4174MB/s)(38.9GiB/10002mse | read: IOPS=7234k, BW=3532MiB/s (3704MB/s)(34.5GiB/10002mse slat (nsec): min=1658, max=2406.5k, avg=5086.35, stdev=72 | slat (nsec): min=1588, max=13073k, avg=5296.03, stdev=602 clat (usec): min=10, max=3565, avg=104.02, stdev=51.20 | clat (usec): min=9, max=13311, avg=117.66, stdev=77.29 lat (usec): min=23, max=3568, avg=109.21, stdev=53.58 | lat (usec): min=18, max=13322, avg=123.07, stdev=80.61 | 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 69], 20. | | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20. | 30.00th=[ 73], 40.00th=[ 74], 50.00th=[ 76], 60. | | 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 81], 60. | 70.00th=[ 91], 80.00th=[ 163], 90.00th=[ 188], 95. | | 70.00th=[ 108], 80.00th=[ 184], 90.00th=[ 247], 95. | 99.00th=[ 221], 99.50th=[ 231], 99.90th=[ 359], 99. | | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 494], 99. | 99.99th=[ 799] | | 99.99th=[ 594] Type=read,Size=512B,NumJobs=56,depth=64 Type=read,Size=512B,NumJobs=56,depth=64 read: IOPS=8218k, BW=4013MiB/s (4208MB/s)(39.2GiB/10002mse | read: IOPS=7769k, BW=3793MiB/s (3978MB/s)(37.1GiB/10002mse slat (nsec): min=1673, max=1678.1k, avg=5096.07, stdev=74 | slat (nsec): min=1583, max=3785.4k, avg=5319.20, stdev=81 clat (usec): min=34, max=2534, avg=430.13, stdev=202.15 | clat (usec): min=2, max=5212, avg=455.02, stdev=334.71 lat (usec): min=61, max=2582, avg=435.33, stdev=204.56 | lat (usec): min=17, max=5214, avg=460.45, stdev=338.64 | 1.00th=[ 277], 5.00th=[ 285], 10.00th=[ 289], 20. | | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 281], 20. | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 310], 60. | | 30.00th=[ 293], 40.00th=[ 297], 50.00th=[ 302], 60. | 70.00th=[ 322], 80.00th=[ 725], 90.00th=[ 758], 95. | | 70.00th=[ 310], 80.00th=[ 437], 90.00th=[ 1074], 95. | 99.00th=[ 816], 99.50th=[ 824], 99.90th=[ 848], 99. | | 99.00th=[ 1401], 99.50th=[ 1991], 99.90th=[ 2245], 99. | 99.99th=[ 889] | | 99.99th=[ 2442] Type=read,Size=4K,NumJobs=1,depth=1 Type=read,Size=4K,NumJobs=1,depth=1 read: IOPS=52.8k, BW=206MiB/s (216MB/s)(2065MiB/10001msec) | read: IOPS=54.4k, BW=213MiB/s (223MB/s)(2126MiB/10001msec) slat (nsec): min=2019, max=42477, avg=2671.11, stdev=1827 | slat (nsec): min=2044, max=28781, avg=2556.03, stdev=1441 clat (nsec): min=605, max=96758, avg=15743.80, stdev=5503 | clat (nsec): min=621, max=70566, avg=15359.54, stdev=5250 lat (nsec): min=13458, max=99251, avg=18489.01, stdev=61 | lat (nsec): min=13302, max=73823, avg=17966.25, stdev=56 | 1.00th=[13248], 5.00th=[13376], 10.00th=[13376], 20. | | 1.00th=[13120], 5.00th=[13248], 10.00th=[13248], 20. | 30.00th=[13504], 40.00th=[13632], 50.00th=[13632], 60. | | 30.00th=[13376], 40.00th=[13376], 50.00th=[13376], 60. | 70.00th=[13888], 80.00th=[14912], 90.00th=[19584], 95. | | 70.00th=[13632], 80.00th=[14144], 90.00th=[19328], 95. | 99.00th=[36608], 99.50th=[38656], 99.90th=[48384], 99. | | 99.00th=[35584], 99.50th=[36608], 99.90th=[45312], 99. | 99.99th=[69120] | | 99.99th=[53504] Type=read,Size=4K,NumJobs=1,depth=16 Type=read,Size=4K,NumJobs=1,depth=16 read: IOPS=306k, BW=1197MiB/s (1255MB/s)(11.7GiB/10001msec | read: IOPS=334k, BW=1306MiB/s (1370MB/s)(12.8GiB/10001msec slat (nsec): min=1621, max=44989, avg=2060.53, stdev=583. | slat (nsec): min=1696, max=45538, avg=2168.76, stdev=208. clat (usec): min=36, max=179, avg=49.85, stdev=11.62 | clat (usec): min=18, max=148, avg=45.39, stdev= 1.47 lat (usec): min=38, max=181, avg=51.95, stdev=11.64 | lat (usec): min=20, max=192, avg=47.59, stdev= 1.48 | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 43], 20. | | 1.00th=[43776], 5.00th=[44288], 10.00th=[44288], 20. | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 44], 60. | | 30.00th=[44800], 40.00th=[45312], 50.00th=[45312], 60. | 70.00th=[ 53], 80.00th=[ 58], 90.00th=[ 64], 95. | | 70.00th=[45312], 80.00th=[45824], 90.00th=[45824], 95. | 99.00th=[ 95], 99.50th=[ 103], 99.90th=[ 127], 99. | | 99.00th=[49408], 99.50th=[54528], 99.90th=[64256], 99. | 99.99th=[ 149] | | 99.99th=[70144] Type=read,Size=4K,NumJobs=1,depth=64 Type=read,Size=4K,NumJobs=1,depth=64 read: IOPS=353k, BW=1381MiB/s (1448MB/s)(13.5GiB/10001msec | read: IOPS=338k, BW=1319MiB/s (1383MB/s)(12.9GiB/10001msec slat (nsec): min=1636, max=31675, avg=2040.73, stdev=224. | slat (nsec): min=1621, max=52632, avg=2145.29, stdev=209. clat (usec): min=44, max=244, avg=178.71, stdev= 2.85 | clat (usec): min=36, max=236, avg=187.02, stdev= 2.05 lat (usec): min=46, max=246, avg=180.79, stdev= 2.87 | lat (usec): min=38, max=238, avg=189.20, stdev= 2.06 | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 178], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20. | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 180], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60. | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 182], 95. | | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 190], 95. | 99.00th=[ 190], 99.50th=[ 194], 99.90th=[ 202], 99. | | 99.00th=[ 192], 99.50th=[ 196], 99.90th=[ 200], 99. | 99.99th=[ 210] | | 99.99th=[ 217] Type=read,Size=4K,NumJobs=16,depth=1 Type=read,Size=4K,NumJobs=16,depth=1 read: IOPS=878k, BW=3430MiB/s (3597MB/s)(33.5GiB/10001msec | read: IOPS=869k, BW=3396MiB/s (3561MB/s)(33.2GiB/10002msec slat (nsec): min=1947, max=53618, avg=2438.66, stdev=659. | slat (nsec): min=1890, max=70947, avg=2497.58, stdev=813. clat (nsec): min=593, max=420818, avg=15328.26, stdev=153 | clat (nsec): min=590, max=79209, avg=15456.14, stdev=1968 lat (usec): min=13, max=423, avg=17.82, stdev= 1.67 | lat (usec): min=12, max=110, avg=18.01, stdev= 2.18 | 1.00th=[12736], 5.00th=[14144], 10.00th=[14272], 20. | | 1.00th=[12352], 5.00th=[13888], 10.00th=[14144], 20. | 30.00th=[14656], 40.00th=[14912], 50.00th=[15040], 60. | | 30.00th=[14528], 40.00th=[14784], 50.00th=[15040], 60. | 70.00th=[15424], 80.00th=[15936], 90.00th=[16768], 95. | | 70.00th=[15552], 80.00th=[16192], 90.00th=[17280], 95. | 99.00th=[20352], 99.50th=[22912], 99.90th=[27264], 99. | | 99.00th=[23936], 99.50th=[25728], 99.90th=[33536], 99. | 99.99th=[70144] | | 99.99th=[36096] Type=read,Size=4K,NumJobs=16,depth=16 Type=read,Size=4K,NumJobs=16,depth=16 read: IOPS=2677k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec | read: IOPS=1843k, BW=7201MiB/s (7551MB/s)(70.3GiB/10002mse slat (nsec): min=1634, max=169095, avg=2282.01, stdev=978 | slat (nsec): min=1582, max=104435, avg=3170.92, stdev=170 clat (usec): min=23, max=416, avg=92.99, stdev=14.19 | clat (usec): min=40, max=554, avg=135.18, stdev=32.57 lat (usec): min=29, max=419, avg=95.32, stdev=14.22 | lat (usec): min=42, max=557, avg=138.42, stdev=32.59 | 1.00th=[ 75], 5.00th=[ 78], 10.00th=[ 80], 20. | | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 104], 20. | 30.00th=[ 87], 40.00th=[ 90], 50.00th=[ 94], 60. | | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 135], 60. | 70.00th=[ 98], 80.00th=[ 99], 90.00th=[ 101], 95. | | 70.00th=[ 141], 80.00th=[ 147], 90.00th=[ 161], 95. | 99.00th=[ 169], 99.50th=[ 174], 99.90th=[ 192], 99. | | 99.00th=[ 262], 99.50th=[ 306], 99.90th=[ 392], 99. | 99.99th=[ 273] | | 99.99th=[ 461] Type=read,Size=4K,NumJobs=16,depth=64 Type=read,Size=4K,NumJobs=16,depth=64 read: IOPS=2673k, BW=10.2GiB/s (10.9GB/s)(102GiB/10001msec | read: IOPS=1917k, BW=7487MiB/s (7850MB/s)(73.1GiB/10001mse slat (nsec): min=1620, max=148710, avg=2532.78, stdev=332 | slat (nsec): min=1566, max=653412, avg=3347.14, stdev=385 clat (usec): min=19, max=1229, avg=380.20, stdev=82.66 | clat (usec): min=43, max=1445, avg=530.39, stdev=127.84 lat (usec): min=23, max=1232, avg=382.77, stdev=83.63 | lat (usec): min=45, max=1619, avg=533.81, stdev=128.49 | 1.00th=[ 318], 5.00th=[ 330], 10.00th=[ 343], 20. | | 1.00th=[ 188], 5.00th=[ 351], 10.00th=[ 420], 20. | 30.00th=[ 351], 40.00th=[ 355], 50.00th=[ 363], 60. | | 30.00th=[ 486], 40.00th=[ 502], 50.00th=[ 519], 60. | 70.00th=[ 371], 80.00th=[ 383], 90.00th=[ 388], 95. | | 70.00th=[ 553], 80.00th=[ 578], 90.00th=[ 644], 95. | 99.00th=[ 725], 99.50th=[ 742], 99.90th=[ 750], 99. | | 99.00th=[ 1012], 99.50th=[ 1037], 99.90th=[ 1090], 99. | 99.99th=[ 857] | | 99.99th=[ 1205] Type=read,Size=4K,NumJobs=56,depth=1 Type=read,Size=4K,NumJobs=56,depth=1 read: IOPS=1606k, BW=6273MiB/s (6578MB/s)(61.3GiB/10002mse | read: IOPS=1571k, BW=6138MiB/s (6436MB/s)(59.9GiB/10001mse slat (nsec): min=1955, max=745694, avg=2980.95, stdev=139 | slat (nsec): min=1693, max=505053, avg=2771.65, stdev=110 clat (nsec): min=633, max=1012.2k, avg=31216.71, stdev=89 | clat (nsec): min=479, max=1348.1k, avg=32311.24, stdev=36 lat (usec): min=13, max=1015, avg=34.28, stdev= 9.05 | lat (usec): min=15, max=1350, avg=35.15, stdev= 3.81 | 1.00th=[ 22], 5.00th=[ 27], 10.00th=[ 29], 20. | | 1.00th=[26752], 5.00th=[28800], 10.00th=[29568], 20. | 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60. | | 30.00th=[30848], 40.00th=[31360], 50.00th=[31872], 60. | 70.00th=[ 32], 80.00th=[ 33], 90.00th=[ 35], 95. | | 70.00th=[33024], 80.00th=[33536], 90.00th=[35584], 95. | 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 70], 99. | | 99.00th=[42240], 99.50th=[43776], 99.90th=[49920], 99. | 99.99th=[ 474] | | 99.99th=[59136] Type=read,Size=4K,NumJobs=56,depth=16 Type=read,Size=4K,NumJobs=56,depth=16 read: IOPS=2591k, BW=9.88GiB/s (10.6GB/s)(98.9GiB/10002mse | read: IOPS=2009k, BW=7846MiB/s (8227MB/s)(76.6GiB/10001mse slat (nsec): min=1655, max=626114, avg=3005.44, stdev=179 | slat (nsec): min=1566, max=379585, avg=3204.59, stdev=230 clat (usec): min=25, max=1308, avg=342.16, stdev=55.44 | clat (usec): min=44, max=1761, avg=442.22, stdev=244.27 lat (usec): min=27, max=1311, avg=345.24, stdev=55.45 | lat (usec): min=46, max=1764, avg=445.50, stdev=244.24 | 1.00th=[ 151], 5.00th=[ 192], 10.00th=[ 318], 20. | | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 215], 20. | 30.00th=[ 330], 40.00th=[ 338], 50.00th=[ 343], 60. | | 30.00th=[ 273], 40.00th=[ 306], 50.00th=[ 351], 60. | 70.00th=[ 359], 80.00th=[ 367], 90.00th=[ 388], 95. | | 70.00th=[ 506], 80.00th=[ 644], 90.00th=[ 832], 95. | 99.00th=[ 478], 99.50th=[ 506], 99.90th=[ 635], 99. | | 99.00th=[ 1156], 99.50th=[ 1221], 99.90th=[ 1336], 99. | 99.99th=[ 955] | | 99.99th=[ 1450] Type=read,Size=4K,NumJobs=56,depth=64 Type=read,Size=4K,NumJobs=56,depth=64 read: IOPS=2652k, BW=10.1GiB/s (10.9GB/s)(101GiB/10003msec | read: IOPS=1801k, BW=7036MiB/s (7378MB/s)(68.7GiB/10002mse slat (nsec): min=1638, max=1266.7k, avg=14402.06, stdev=5 | slat (nsec): min=1586, max=2410.2k, avg=20136.27, stdev=8 clat (usec): min=23, max=7119, avg=1336.05, stdev=712.32 | clat (usec): min=58, max=8215, avg=1968.75, stdev=947.22 lat (usec): min=30, max=7224, avg=1350.51, stdev=725.25 | lat (usec): min=60, max=8632, avg=1988.96, stdev=962.84 | 1.00th=[ 570], 5.00th=[ 644], 10.00th=[ 652], 20. | | 1.00th=[ 449], 5.00th=[ 734], 10.00th=[ 865], 20. | 30.00th=[ 685], 40.00th=[ 1221], 50.00th=[ 1303], 60. | | 30.00th=[ 1418], 40.00th=[ 1729], 50.00th=[ 1909], 60. | 70.00th=[ 1401], 80.00th=[ 1926], 90.00th=[ 2409], 95. | | 70.00th=[ 2212], 80.00th=[ 2507], 90.00th=[ 3326], 95. | 99.00th=[ 3359], 99.50th=[ 3589], 99.90th=[ 4047], 99. | | 99.00th=[ 4817], 99.50th=[ 5080], 99.90th=[ 5735], 99. | 99.99th=[ 5080] | | 99.99th=[ 6587] Type=read,Size=64K,NumJobs=1,depth=1 Type=read,Size=64K,NumJobs=1,depth=1 read: IOPS=29.6k, BW=1849MiB/s (1939MB/s)(18.1GiB/10001mse | read: IOPS=29.8k, BW=1864MiB/s (1955MB/s)(18.2GiB/10001mse slat (nsec): min=4727, max=45982, avg=6068.77, stdev=2849 | slat (nsec): min=4768, max=95484, avg=6119.74, stdev=2779 clat (usec): min=3, max=110, avg=27.22, stdev= 7.91 | clat (nsec): min=1555, max=75556, avg=26881.73, stdev=781 lat (usec): min=25, max=122, avg=33.34, stdev= 9.17 | lat (usec): min=25, max=102, avg=33.05, stdev= 9.07 | 1.00th=[21888], 5.00th=[21888], 10.00th=[21888], 20. | | 1.00th=[21632], 5.00th=[21632], 10.00th=[21632], 20. | 30.00th=[22144], 40.00th=[22400], 50.00th=[22400], 60. | | 30.00th=[21888], 40.00th=[22144], 50.00th=[22144], 60. | 70.00th=[28288], 80.00th=[35072], 90.00th=[41216], 95. | | 70.00th=[28032], 80.00th=[34048], 90.00th=[40704], 95. | 99.00th=[47872], 99.50th=[50432], 99.90th=[63744], 99. | | 99.00th=[47360], 99.50th=[49920], 99.90th=[59648], 99. | 99.99th=[78336] | | 99.99th=[70144] Type=read,Size=64K,NumJobs=1,depth=16 Type=read,Size=64K,NumJobs=1,depth=16 read: IOPS=168k, BW=10.3GiB/s (11.0GB/s)(103GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec) slat (nsec): min=3795, max=48145, avg=4685.87, stdev=473. | slat (nsec): min=3756, max=65673, avg=4930.28, stdev=334. clat (usec): min=78, max=259, avg=90.18, stdev= 7.63 | clat (usec): min=83, max=243, avg=87.86, stdev= 2.04 lat (usec): min=83, max=300, avg=94.90, stdev= 7.60 | lat (usec): min=87, max=310, avg=92.83, stdev= 2.05 | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 85], 20. | | 1.00th=[ 86], 5.00th=[ 87], 10.00th=[ 87], 20. | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 89], 60. | | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60. | 70.00th=[ 92], 80.00th=[ 93], 90.00th=[ 96], 95. | | 70.00th=[ 88], 80.00th=[ 89], 90.00th=[ 91], 95. | 99.00th=[ 127], 99.50th=[ 137], 99.90th=[ 174], 99. | | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 108], 99. | 99.99th=[ 196] | | 99.99th=[ 114] Type=read,Size=64K,NumJobs=1,depth=64 Type=read,Size=64K,NumJobs=1,depth=64 read: IOPS=171k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec) slat (usec): min=2, max=384, avg= 3.57, stdev= 1.13 | slat (usec): min=2, max=390, avg= 4.15, stdev= 1.56 clat (usec): min=50, max=983, avg=370.03, stdev=11.50 | clat (usec): min=37, max=947, avg=376.95, stdev=15.70 lat (usec): min=53, max=1059, avg=373.64, stdev=11.40 | lat (usec): min=42, max=988, avg=381.15, stdev=16.25 | 1.00th=[ 351], 5.00th=[ 359], 10.00th=[ 363], 20. | | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 367], 20. | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 70.00th=[ 371], 80.00th=[ 375], 90.00th=[ 379], 95. | | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 400], 95. | 99.00th=[ 400], 99.50th=[ 408], 99.90th=[ 433], 99. | | 99.00th=[ 412], 99.50th=[ 416], 99.90th=[ 429], 99. | 99.99th=[ 644] | | 99.99th=[ 668] Type=read,Size=64K,NumJobs=16,depth=1 Type=read,Size=64K,NumJobs=16,depth=1 read: IOPS=171k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec) slat (nsec): min=4077, max=68679, avg=5394.03, stdev=2968 | slat (nsec): min=3480, max=74756, avg=5273.21, stdev=2961 clat (nsec): min=686, max=194443, avg=87548.33, stdev=162 | clat (nsec): min=616, max=453786, avg=87271.90, stdev=161 lat (usec): min=27, max=208, avg=93.00, stdev=16.25 | lat (usec): min=26, max=458, avg=92.60, stdev=16.20 | 1.00th=[ 42], 5.00th=[ 61], 10.00th=[ 71], 20. | | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 68], 20. | 30.00th=[ 81], 40.00th=[ 84], 50.00th=[ 87], 60. | | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60. | 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 111], 95. | | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 104], 95. | 99.00th=[ 127], 99.50th=[ 133], 99.90th=[ 143], 99. | | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 143], 99. | 99.99th=[ 161] | 99.99th=[ 161] Type=read,Size=64K,NumJobs=16,depth=16 Type=read,Size=64K,NumJobs=16,depth=16 read: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=4066, max=75561, avg=5259.83, stdev=1080 | slat (nsec): min=3866, max=57337, avg=5204.47, stdev=1220 clat (usec): min=118, max=3688, avg=1460.39, stdev=456.30 | clat (usec): min=116, max=3985, avg=1453.77, stdev=349.92 lat (usec): min=132, max=3693, avg=1465.70, stdev=456.22 | lat (usec): min=128, max=3990, avg=1459.03, stdev=349.93 | 1.00th=[ 1172], 5.00th=[ 1172], 10.00th=[ 1172], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1270], 60. | | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60. | 70.00th=[ 1352], 80.00th=[ 1401], 90.00th=[ 2376], 95. | | 70.00th=[ 1369], 80.00th=[ 1401], 90.00th=[ 1467], 95. | 99.00th=[ 2606], 99.50th=[ 2737], 99.90th=[ 2769], 99. | | 99.00th=[ 2737], 99.50th=[ 2769], 99.90th=[ 2769], 99. | 99.99th=[ 2802] | 99.99th=[ 2802] Type=read,Size=64K,NumJobs=16,depth=64 Type=read,Size=64K,NumJobs=16,depth=64 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) slat (usec): min=2, max=890, avg= 4.27, stdev= 3.77 | slat (usec): min=2, max=1185, avg= 9.71, stdev=60.22 clat (usec): min=108, max=11645, avg=5850.95, stdev=186.5 | clat (usec): min=116, max=16490, avg=5824.81, stdev=1345. lat (usec): min=113, max=12034, avg=5855.28, stdev=186.3 | lat (usec): min=133, max=16494, avg=5834.58, stdev=1367. | 1.00th=[ 5800], 5.00th=[ 5800], 10.00th=[ 5800], 20. | | 1.00th=[ 5407], 5.00th=[ 5407], 10.00th=[ 5473], 20. | 30.00th=[ 5866], 40.00th=[ 5866], 50.00th=[ 5866], 60. | | 30.00th=[ 5473], 40.00th=[ 5473], 50.00th=[ 5473], 60. | 70.00th=[ 5866], 80.00th=[ 5866], 90.00th=[ 5866], 95. | | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[ 5538], 95. | 99.00th=[ 5932], 99.50th=[ 6063], 99.90th=[ 6980], 99. | | 99.00th=[10945], 99.50th=[10945], 99.90th=[11076], 99. | 99.99th=[11207] | | 99.99th=[11338] Type=read,Size=64K,NumJobs=56,depth=1 Type=read,Size=64K,NumJobs=56,depth=1 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (usec): min=4, max=111, avg= 4.93, stdev= 1.11 | slat (nsec): min=3801, max=63985, avg=4896.45, stdev=1489 clat (usec): min=41, max=531, avg=314.40, stdev=75.66 | clat (usec): min=29, max=550, avg=314.45, stdev=76.21 lat (usec): min=50, max=535, avg=319.39, stdev=75.67 | lat (usec): min=35, max=555, avg=319.41, stdev=76.15 | 1.00th=[ 174], 5.00th=[ 176], 10.00th=[ 178], 20. | | 1.00th=[ 174], 5.00th=[ 178], 10.00th=[ 194], 20. | 30.00th=[ 306], 40.00th=[ 306], 50.00th=[ 306], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60. | 70.00th=[ 310], 80.00th=[ 318], 90.00th=[ 461], 95. | | 70.00th=[ 310], 80.00th=[ 322], 90.00th=[ 457], 95. | 99.00th=[ 478], 99.50th=[ 482], 99.90th=[ 490], 99. | | 99.00th=[ 474], 99.50th=[ 482], 99.90th=[ 490], 99. | 99.99th=[ 502] | | 99.99th=[ 506] Type=read,Size=64K,NumJobs=56,depth=16 Type=read,Size=64K,NumJobs=56,depth=16 read: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10006msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) slat (usec): min=4, max=176, avg= 5.48, stdev= 2.61 | slat (usec): min=3, max=251, avg= 5.40, stdev= 2.34 clat (usec): min=718, max=14322, avg=5122.23, stdev=2028. | clat (usec): min=236, max=14486, avg=5106.35, stdev=1241. lat (usec): min=744, max=14328, avg=5127.78, stdev=2028. | lat (usec): min=265, max=14491, avg=5111.80, stdev=1241. | 1.00th=[ 2343], 5.00th=[ 2376], 10.00th=[ 2442], 20. | | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 4883], 20. | 30.00th=[ 4752], 40.00th=[ 4752], 50.00th=[ 4948], 60. | | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5080], 60. | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 9503], 95. | | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 7373], 95. | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[10028], 99. | | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[ 9896], 99. | 99.99th=[10028] | 99.99th=[10028] Type=read,Size=64K,NumJobs=56,depth=64 Type=read,Size=64K,NumJobs=56,depth=64 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10017msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10018msec) slat (usec): min=2, max=9087, avg=220.21, stdev=768.63 | slat (usec): min=2, max=10184, avg=189.22, stdev=637.78 clat (usec): min=172, max=64296, avg=20233.61, stdev=8148 | clat (usec): min=347, max=61952, avg=20252.15, stdev=7056 lat (usec): min=186, max=64319, avg=20453.88, stdev=8308 | lat (usec): min=351, max=65141, avg=20441.44, stdev=7203 | 1.00th=[ 9110], 5.00th=[ 9110], 10.00th=[ 9110], 20. | | 1.00th=[ 9503], 5.00th=[ 9765], 10.00th=[ 9896], 20. | 30.00th=[18220], 40.00th=[18744], 50.00th=[19006], 60. | | 30.00th=[19530], 40.00th=[19530], 50.00th=[19792], 60. | 70.00th=[27395], 80.00th=[27395], 90.00th=[28705], 95. | | 70.00th=[20317], 80.00th=[20579], 90.00th=[29492], 95. | 99.00th=[38011], 99.50th=[40109], 99.90th=[44303], 99. | | 99.00th=[41157], 99.50th=[58983], 99.90th=[59507], 99. | 99.99th=[54789] | | 99.99th=[61080] Type=write,Size=512B,NumJobs=1,depth=1 Type=write,Size=512B,NumJobs=1,depth=1 write: IOPS=57.7k, BW=28.2MiB/s (29.6MB/s)(282MiB/10001msec | write: IOPS=56.0k, BW=27.4MiB/s (28.7MB/s)(274MiB/10001msec slat (nsec): min=2139, max=32784, avg=2527.92, stdev=1359 | slat (nsec): min=2112, max=43957, avg=2561.65, stdev=1423 clat (nsec): min=624, max=106340, avg=14332.09, stdev=474 | clat (nsec): min=633, max=60031, avg=14812.68, stdev=5638 lat (usec): min=12, max=108, avg=16.90, stdev= 5.16 | lat (nsec): min=12369, max=72124, avg=17416.39, stdev=60 | 1.00th=[12480], 5.00th=[12480], 10.00th=[12608], 20. | | 1.00th=[12352], 5.00th=[12352], 10.00th=[12480], 20. | 30.00th=[12608], 40.00th=[12736], 50.00th=[12736], 60. | | 30.00th=[12480], 40.00th=[12608], 50.00th=[12608], 60. | 70.00th=[12864], 80.00th=[13376], 90.00th=[16512], 95. | | 70.00th=[12864], 80.00th=[14656], 90.00th=[20096], 95. | 99.00th=[34560], 99.50th=[35584], 99.90th=[42240], 99. | | 99.00th=[35072], 99.50th=[36608], 99.90th=[47360], 99. | 99.99th=[53504] | | 99.99th=[54016] Type=write,Size=512B,NumJobs=1,depth=16 Type=write,Size=512B,NumJobs=1,depth=16 write: IOPS=306k, BW=149MiB/s (156MB/s)(1492MiB/10001msec) | write: IOPS=328k, BW=160MiB/s (168MB/s)(1601MiB/10001msec) slat (nsec): min=1650, max=31443, avg=2000.75, stdev=584. | slat (nsec): min=1641, max=32941, avg=2199.97, stdev=236. clat (usec): min=38, max=193, avg=50.03, stdev=12.10 | clat (usec): min=14, max=113, avg=46.29, stdev= 1.27 lat (usec): min=40, max=195, avg=52.07, stdev=12.12 | lat (usec): min=16, max=115, avg=48.52, stdev= 1.28 | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20. | | 1.00th=[43776], 5.00th=[44800], 10.00th=[45312], 20. | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60. | | 30.00th=[45824], 40.00th=[45824], 50.00th=[46336], 60. | 70.00th=[ 52], 80.00th=[ 59], 90.00th=[ 69], 95. | | 70.00th=[46336], 80.00th=[46848], 90.00th=[47360], 95. | 99.00th=[ 93], 99.50th=[ 100], 99.90th=[ 126], 99. | | 99.00th=[50432], 99.50th=[53504], 99.90th=[57600], 99. | 99.99th=[ 145] | | 99.99th=[63744] Type=write,Size=512B,NumJobs=1,depth=64 Type=write,Size=512B,NumJobs=1,depth=64 write: IOPS=363k, BW=177MiB/s (186MB/s)(1771MiB/10001msec) | write: IOPS=337k, BW=164MiB/s (172MB/s)(1644MiB/10001msec) slat (nsec): min=1644, max=41021, avg=1961.04, stdev=238. | slat (nsec): min=1642, max=43437, avg=2146.30, stdev=234. clat (usec): min=71, max=241, avg=174.18, stdev= 2.48 | clat (usec): min=16, max=235, avg=187.62, stdev= 2.46 lat (usec): min=73, max=243, avg=176.18, stdev= 2.50 | lat (usec): min=18, max=237, avg=189.80, stdev= 2.48 | 1.00th=[ 172], 5.00th=[ 172], 10.00th=[ 172], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20. | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60. | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 178], 95. | | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 192], 95. | 99.00th=[ 186], 99.50th=[ 190], 99.90th=[ 192], 99. | | 99.00th=[ 194], 99.50th=[ 196], 99.90th=[ 200], 99. | 99.99th=[ 198] | | 99.99th=[ 204] Type=write,Size=512B,NumJobs=16,depth=1 Type=write,Size=512B,NumJobs=16,depth=1 write: IOPS=938k, BW=458MiB/s (480MB/s)(4582MiB/10002msec) | write: IOPS=914k, BW=446MiB/s (468MB/s)(4462MiB/10001msec) slat (nsec): min=2061, max=61336, avg=2520.86, stdev=499. | slat (nsec): min=1818, max=83376, avg=2566.34, stdev=824. clat (nsec): min=595, max=164756, avg=14051.16, stdev=104 | clat (nsec): min=465, max=343253, avg=14459.07, stdev=181 lat (usec): min=11, max=168, avg=16.62, stdev= 1.19 | lat (usec): min=11, max=345, avg=17.08, stdev= 2.09 | 1.00th=[11712], 5.00th=[12864], 10.00th=[13248], 20. | | 1.00th=[12224], 5.00th=[12992], 10.00th=[13248], 20. | 30.00th=[13632], 40.00th=[13760], 50.00th=[13888], 60. | | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60. | 70.00th=[14272], 80.00th=[14528], 90.00th=[15040], 95. | | 70.00th=[14528], 80.00th=[15040], 90.00th=[16320], 95. | 99.00th=[17280], 99.50th=[18048], 99.90th=[23680], 99. | | 99.00th=[21888], 99.50th=[23936], 99.90th=[32384], 99. | 99.99th=[29568] | | 99.99th=[35584] Type=write,Size=512B,NumJobs=16,depth=16 Type=write,Size=512B,NumJobs=16,depth=16 write: IOPS=4631k, BW=2261MiB/s (2371MB/s)(22.1GiB/10001mse | write: IOPS=3218k, BW=1571MiB/s (1648MB/s)(15.3GiB/10001mse slat (nsec): min=1666, max=183449, avg=2373.16, stdev=548 | slat (nsec): min=1621, max=154997, avg=3548.60, stdev=323 clat (usec): min=18, max=440, avg=52.52, stdev= 9.27 | clat (usec): min=14, max=473, avg=75.44, stdev=65.86 lat (usec): min=20, max=444, avg=54.94, stdev= 9.57 | lat (usec): min=17, max=489, avg=79.05, stdev=68.82 | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 46], 20. | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 52], 60. | | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 52], 60. | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 62], 95. | | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 178], 95. | 99.00th=[ 74], 99.50th=[ 79], 99.90th=[ 89], 99. | | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99. | 99.99th=[ 355] | | 99.99th=[ 367] Type=write,Size=512B,NumJobs=16,depth=64 Type=write,Size=512B,NumJobs=16,depth=64 write: IOPS=5021k, BW=2452MiB/s (2571MB/s)(23.9GiB/10002mse | write: IOPS=3973k, BW=1940MiB/s (2034MB/s)(18.0GiB/10002mse slat (nsec): min=1670, max=80008, avg=2293.73, stdev=923. | slat (nsec): min=1627, max=171980, avg=2926.70, stdev=209 clat (usec): min=24, max=1715, avg=201.23, stdev=28.44 | clat (usec): min=10, max=1606, avg=254.29, stdev=176.36 lat (usec): min=26, max=1733, avg=203.56, stdev=28.76 | lat (usec): min=16, max=1615, avg=257.27, stdev=178.32 | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20. | | 1.00th=[ 184], 5.00th=[ 186], 10.00th=[ 188], 20. | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 208], 60. | | 30.00th=[ 194], 40.00th=[ 202], 50.00th=[ 237], 60. | 70.00th=[ 212], 80.00th=[ 215], 90.00th=[ 219], 95. | | 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 293], 95. | 99.00th=[ 293], 99.50th=[ 297], 99.90th=[ 314], 99. | | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1467], 99. | 99.99th=[ 898] | | 99.99th=[ 1500] Type=write,Size=512B,NumJobs=56,depth=1 Type=write,Size=512B,NumJobs=56,depth=1 write: IOPS=2332k, BW=1139MiB/s (1194MB/s)(11.1GiB/10003mse | write: IOPS=2369k, BW=1157MiB/s (1213MB/s)(11.3GiB/10002mse slat (usec): min=2, max=720, avg= 3.74, stdev= 2.14 | slat (nsec): min=1823, max=501953, avg=3567.43, stdev=201 clat (nsec): min=624, max=5616.3k, avg=19430.25, stdev=63 | clat (nsec): min=449, max=1722.5k, avg=19318.77, stdev=42 lat (usec): min=10, max=5623, avg=23.28, stdev= 6.86 | lat (usec): min=11, max=1724, avg=22.98, stdev= 4.83 | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20. | | 1.00th=[12992], 5.00th=[14656], 10.00th=[15680], 20. | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 19], 60. | | 30.00th=[17536], 40.00th=[18048], 50.00th=[18816], 60. | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20352], 80.00th=[21632], 90.00th=[23424], 95. | 99.00th=[ 33], 99.50th=[ 37], 99.90th=[ 47], 99. | | 99.00th=[31872], 99.50th=[35584], 99.90th=[41728], 99. | 99.99th=[ 310] | | 99.99th=[54016] Type=write,Size=512B,NumJobs=56,depth=16 Type=write,Size=512B,NumJobs=56,depth=16 write: IOPS=8099k, BW=3955MiB/s (4147MB/s)(38.6GiB/10002mse | write: IOPS=6237k, BW=3046MiB/s (3193MB/s)(29.7GiB/10002mse slat (nsec): min=1742, max=11042k, avg=5139.82, stdev=693 | slat (nsec): min=1640, max=2381.5k, avg=5462.05, stdev=67 clat (nsec): min=1973, max=13062k, avg=104665.53, stdev=5 | clat (usec): min=5, max=10435, avg=137.19, stdev=82.68 lat (usec): min=31, max=13064, avg=109.91, stdev=54.98 | lat (usec): min=15, max=10447, avg=142.77, stdev=85.58 | 1.00th=[ 66], 5.00th=[ 69], 10.00th=[ 71], 20. | | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 70], 20. | 30.00th=[ 74], 40.00th=[ 75], 50.00th=[ 76], 60. | | 30.00th=[ 85], 40.00th=[ 96], 50.00th=[ 104], 60. | 70.00th=[ 88], 80.00th=[ 167], 90.00th=[ 188], 95. | | 70.00th=[ 149], 80.00th=[ 210], 90.00th=[ 265], 95. | 99.00th=[ 215], 99.50th=[ 223], 99.90th=[ 375], 99. | | 99.00th=[ 363], 99.50th=[ 396], 99.90th=[ 553], 99. | 99.99th=[ 898] | | 99.99th=[ 873] Type=write,Size=512B,NumJobs=56,depth=64 Type=write,Size=512B,NumJobs=56,depth=64 write: IOPS=8134k, BW=3972MiB/s (4165MB/s)(38.8GiB/10002mse | write: IOPS=7488k, BW=3656MiB/s (3834MB/s)(35.7GiB/10002mse slat (nsec): min=1797, max=599123, avg=5146.84, stdev=693 | slat (nsec): min=1666, max=7929.3k, avg=5382.53, stdev=12 clat (usec): min=42, max=1474, avg=434.58, stdev=198.63 | clat (usec): min=14, max=12517, avg=472.19, stdev=331.30 lat (usec): min=49, max=1477, avg=439.83, stdev=201.01 | lat (usec): min=17, max=12520, avg=477.70, stdev=334.84 | 1.00th=[ 281], 5.00th=[ 289], 10.00th=[ 297], 20. | | 1.00th=[ 265], 5.00th=[ 277], 10.00th=[ 289], 20. | 30.00th=[ 310], 40.00th=[ 310], 50.00th=[ 314], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 314], 60. | 70.00th=[ 326], 80.00th=[ 725], 90.00th=[ 758], 95. | | 70.00th=[ 347], 80.00th=[ 562], 90.00th=[ 1057], 95. | 99.00th=[ 799], 99.50th=[ 816], 99.90th=[ 840], 99. | | 99.00th=[ 1434], 99.50th=[ 1696], 99.90th=[ 2040], 99. | 99.99th=[ 922] | | 99.99th=[ 2474] Type=write,Size=4K,NumJobs=1,depth=1 Type=write,Size=4K,NumJobs=1,depth=1 write: IOPS=54.1k, BW=211MiB/s (222MB/s)(2115MiB/10001msec) | write: IOPS=55.8k, BW=218MiB/s (229MB/s)(2180MiB/10000msec) slat (nsec): min=2154, max=32419, avg=2620.30, stdev=1579 | slat (nsec): min=2138, max=44613, avg=2552.61, stdev=1404 clat (nsec): min=639, max=171519, avg=15378.78, stdev=558 | clat (nsec): min=627, max=624863, avg=14910.38, stdev=517 lat (usec): min=13, max=173, avg=18.04, stdev= 6.03 | lat (usec): min=13, max=627, avg=17.50, stdev= 5.55 | 1.00th=[12864], 5.00th=[12992], 10.00th=[12992], 20. | | 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20. | 30.00th=[13120], 40.00th=[13120], 50.00th=[13248], 60. | | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60. | 70.00th=[13376], 80.00th=[15424], 90.00th=[20864], 95. | | 70.00th=[13248], 80.00th=[13632], 90.00th=[20352], 95. | 99.00th=[36096], 99.50th=[38144], 99.90th=[48384], 99. | | 99.00th=[35584], 99.50th=[37120], 99.90th=[44800], 99. | 99.99th=[69120] | | 99.99th=[53504] Type=write,Size=4K,NumJobs=1,depth=16 Type=write,Size=4K,NumJobs=1,depth=16 write: IOPS=285k, BW=1112MiB/s (1166MB/s)(10.9GiB/10001msec | write: IOPS=325k, BW=1271MiB/s (1332MB/s)(12.4GiB/10001msec slat (nsec): min=1784, max=29113, avg=2129.85, stdev=618. | slat (nsec): min=1697, max=32986, avg=2232.67, stdev=205. clat (usec): min=40, max=179, avg=53.75, stdev=12.39 | clat (nsec): min=14620, max=89670, avg=46638.22, stdev=11 lat (usec): min=42, max=182, avg=55.91, stdev=12.40 | lat (nsec): min=16814, max=91833, avg=48907.48, stdev=11 | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | | 1.00th=[44800], 5.00th=[45312], 10.00th=[45824], 20. | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 50], 60. | | 30.00th=[46336], 40.00th=[46336], 50.00th=[46336], 60. | 70.00th=[ 58], 80.00th=[ 64], 90.00th=[ 72], 95. | | 70.00th=[46848], 80.00th=[46848], 90.00th=[47360], 95. | 99.00th=[ 96], 99.50th=[ 105], 99.90th=[ 129], 99. | | 99.00th=[50944], 99.50th=[53504], 99.90th=[58112], 99. | 99.99th=[ 147] | | 99.99th=[63744] Type=write,Size=4K,NumJobs=1,depth=64 Type=write,Size=4K,NumJobs=1,depth=64 write: IOPS=340k, BW=1327MiB/s (1391MB/s)(12.0GiB/10001msec | write: IOPS=322k, BW=1259MiB/s (1320MB/s)(12.3GiB/10001msec slat (nsec): min=1701, max=34828, avg=2128.31, stdev=233. | slat (nsec): min=1697, max=33655, avg=2273.54, stdev=208. clat (usec): min=52, max=233, avg=185.96, stdev= 1.94 | clat (usec): min=18, max=238, avg=195.95, stdev= 2.07 lat (usec): min=54, max=235, avg=188.13, stdev= 1.95 | lat (usec): min=20, max=241, avg=198.26, stdev= 2.08 | 1.00th=[ 182], 5.00th=[ 184], 10.00th=[ 184], 20. | | 1.00th=[ 192], 5.00th=[ 194], 10.00th=[ 194], 20. | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60. | | 30.00th=[ 196], 40.00th=[ 196], 50.00th=[ 196], 60. | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95. | | 70.00th=[ 198], 80.00th=[ 198], 90.00th=[ 198], 95. | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 196], 99. | | 99.00th=[ 202], 99.50th=[ 204], 99.90th=[ 208], 99. | 99.99th=[ 202] | | 99.99th=[ 212] Type=write,Size=4K,NumJobs=16,depth=1 Type=write,Size=4K,NumJobs=16,depth=1 write: IOPS=883k, BW=3449MiB/s (3616MB/s)(33.7GiB/10002msec | write: IOPS=855k, BW=3339MiB/s (3502MB/s)(32.6GiB/10001msec slat (nsec): min=2093, max=58113, avg=2489.62, stdev=384. | slat (nsec): min=1832, max=65363, avg=2579.38, stdev=826. clat (nsec): min=601, max=198998, avg=15168.57, stdev=132 | clat (nsec): min=472, max=134403, avg=15646.36, stdev=196 lat (usec): min=12, max=201, avg=17.70, stdev= 1.39 | lat (usec): min=12, max=136, avg=18.28, stdev= 2.21 | 1.00th=[12608], 5.00th=[13760], 10.00th=[14144], 20. | | 1.00th=[13120], 5.00th=[13760], 10.00th=[14016], 20. | 30.00th=[14656], 40.00th=[14784], 50.00th=[15040], 60. | | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60. | 70.00th=[15424], 80.00th=[15808], 90.00th=[16320], 95. | | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95. | 99.00th=[18816], 99.50th=[20096], 99.90th=[25984], 99. | | 99.00th=[23680], 99.50th=[25728], 99.90th=[33536], 99. | 99.99th=[44800] | | 99.99th=[37120] Type=write,Size=4K,NumJobs=16,depth=16 Type=write,Size=4K,NumJobs=16,depth=16 write: IOPS=2677k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec | write: IOPS=1826k, BW=7133MiB/s (7480MB/s)(69.7GiB/10001mse slat (nsec): min=1663, max=57295, avg=2380.16, stdev=1242 | slat (nsec): min=1639, max=113071, avg=3312.71, stdev=236 clat (usec): min=37, max=345, avg=92.86, stdev=12.80 | clat (usec): min=40, max=454, avg=136.35, stdev=35.22 lat (usec): min=41, max=351, avg=95.28, stdev=12.74 | lat (usec): min=43, max=456, avg=139.72, stdev=35.40 | 1.00th=[ 69], 5.00th=[ 75], 10.00th=[ 78], 20. | | 1.00th=[ 72], 5.00th=[ 91], 10.00th=[ 101], 20. | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 96], 60. | | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60. | 70.00th=[ 99], 80.00th=[ 101], 90.00th=[ 104], 95. | | 70.00th=[ 143], 80.00th=[ 149], 90.00th=[ 182], 95. | 99.00th=[ 133], 99.50th=[ 151], 99.90th=[ 178], 99. | | 99.00th=[ 258], 99.50th=[ 273], 99.90th=[ 314], 99. | 99.99th=[ 212] | | 99.99th=[ 367] Type=write,Size=4K,NumJobs=16,depth=64 Type=write,Size=4K,NumJobs=16,depth=64 write: IOPS=2691k, BW=10.3GiB/s (11.0GB/s)(103GiB/10001msec | write: IOPS=1875k, BW=7325MiB/s (7680MB/s)(71.5GiB/10001mse slat (nsec): min=1672, max=299471, avg=2451.55, stdev=255 | slat (nsec): min=1638, max=627057, avg=3232.91, stdev=324 clat (usec): min=70, max=965, avg=377.69, stdev=68.69 | clat (usec): min=49, max=1357, avg=542.32, stdev=162.71 lat (usec): min=77, max=980, avg=380.18, stdev=69.11 | lat (usec): min=52, max=1384, avg=545.61, stdev=162.80 | 1.00th=[ 223], 5.00th=[ 285], 10.00th=[ 314], 20. | | 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20. | 30.00th=[ 351], 40.00th=[ 363], 50.00th=[ 375], 60. | | 30.00th=[ 453], 40.00th=[ 498], 50.00th=[ 537], 60. | 70.00th=[ 396], 80.00th=[ 412], 90.00th=[ 433], 95. | | 70.00th=[ 635], 80.00th=[ 685], 90.00th=[ 758], 95. | 99.00th=[ 693], 99.50th=[ 717], 99.90th=[ 758], 99. | | 99.00th=[ 922], 99.50th=[ 955], 99.90th=[ 1029], 99. | 99.99th=[ 807] | | 99.99th=[ 1123] Type=write,Size=4K,NumJobs=56,depth=1 Type=write,Size=4K,NumJobs=56,depth=1 write: IOPS=1417k, BW=5534MiB/s (5803MB/s)(54.1GiB/10002mse | write: IOPS=1416k, BW=5533MiB/s (5802MB/s)(54.0GiB/10002mse slat (usec): min=2, max=722, avg= 3.06, stdev= 1.31 | slat (nsec): min=1846, max=533231, avg=2881.71, stdev=111 clat (nsec): min=647, max=796809, avg=35776.52, stdev=697 | clat (nsec): min=498, max=4453.2k, avg=36057.87, stdev=38 lat (usec): min=14, max=799, avg=38.92, stdev= 7.08 | lat (usec): min=15, max=4456, avg=39.01, stdev= 4.01 | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 33], 20. | | 1.00th=[29312], 5.00th=[32128], 10.00th=[33024], 20. | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 36], 60. | | 30.00th=[34560], 40.00th=[35072], 50.00th=[35584], 60. | 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 40], 95. | | 70.00th=[37120], 80.00th=[37632], 90.00th=[38656], 95. | 99.00th=[ 50], 99.50th=[ 53], 99.90th=[ 69], 99. | | 99.00th=[49408], 99.50th=[52480], 99.90th=[56576], 99. | 99.99th=[ 367] | | 99.99th=[64768] Type=write,Size=4K,NumJobs=56,depth=16 Type=write,Size=4K,NumJobs=56,depth=16 write: IOPS=2470k, BW=9649MiB/s (10.1GB/s)(94.3GiB/10002mse | write: IOPS=1616k, BW=6314MiB/s (6621MB/s)(61.7GiB/10001mse slat (nsec): min=1714, max=353380, avg=3021.76, stdev=221 | slat (nsec): min=1649, max=738008, avg=3245.81, stdev=163 clat (usec): min=44, max=1490, avg=359.06, stdev=129.14 | clat (usec): min=44, max=5046, avg=550.36, stdev=143.07 lat (usec): min=47, max=1493, avg=362.15, stdev=129.13 | lat (usec): min=46, max=5049, avg=553.68, stdev=143.03 | 1.00th=[ 212], 5.00th=[ 235], 10.00th=[ 247], 20. | | 1.00th=[ 215], 5.00th=[ 343], 10.00th=[ 404], 20. | 30.00th=[ 277], 40.00th=[ 285], 50.00th=[ 302], 60. | | 30.00th=[ 490], 40.00th=[ 510], 50.00th=[ 529], 60. | 70.00th=[ 392], 80.00th=[ 478], 90.00th=[ 570], 95. | | 70.00th=[ 586], 80.00th=[ 660], 90.00th=[ 758], 95. | 99.00th=[ 717], 99.50th=[ 750], 99.90th=[ 840], 99. | | 99.00th=[ 938], 99.50th=[ 996], 99.90th=[ 1106], 99. | 99.99th=[ 1004] | | 99.99th=[ 1221] Type=write,Size=4K,NumJobs=56,depth=64 Type=write,Size=4K,NumJobs=56,depth=64 write: IOPS=2093k, BW=8177MiB/s (8574MB/s)(79.9GiB/10003mse | write: IOPS=1625k, BW=6348MiB/s (6656MB/s)(62.0GiB/10003mse slat (nsec): min=1735, max=4573.4k, avg=18118.68, stdev=6 | slat (nsec): min=1656, max=1933.3k, avg=21620.58, stdev=7 clat (usec): min=40, max=8076, avg=1693.04, stdev=804.74 | clat (usec): min=66, max=8663, avg=2182.95, stdev=879.90 lat (usec): min=44, max=8079, avg=1711.22, stdev=817.33 | lat (usec): min=69, max=8669, avg=2204.64, stdev=895.51 | 1.00th=[ 375], 5.00th=[ 644], 10.00th=[ 766], 20. | | 1.00th=[ 693], 5.00th=[ 971], 10.00th=[ 1074], 20. | 30.00th=[ 1205], 40.00th=[ 1516], 50.00th=[ 1647], 60. | | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60. | 70.00th=[ 1909], 80.00th=[ 2278], 90.00th=[ 2835], 95. | | 70.00th=[ 2343], 80.00th=[ 2507], 90.00th=[ 3392], 95. | 99.00th=[ 4015], 99.50th=[ 4293], 99.90th=[ 4883], 99. | | 99.00th=[ 4883], 99.50th=[ 5211], 99.90th=[ 5800], 99. | 99.99th=[ 5735] | | 99.99th=[ 6587] Type=write,Size=64K,NumJobs=1,depth=1 Type=write,Size=64K,NumJobs=1,depth=1 write: IOPS=23.3k, BW=1455MiB/s (1526MB/s)(14.2GiB/10001mse | write: IOPS=23.3k, BW=1454MiB/s (1525MB/s)(14.2GiB/10001mse slat (usec): min=4, max=109, avg= 6.71, stdev= 3.24 | slat (nsec): min=4849, max=39702, avg=6880.02, stdev=3240 clat (usec): min=14, max=124, avg=35.67, stdev= 9.52 | clat (usec): min=17, max=133, avg=35.53, stdev= 9.39 lat (usec): min=33, max=159, avg=42.42, stdev=10.65 | lat (usec): min=33, max=146, avg=42.46, stdev=10.52 | 1.00th=[29312], 5.00th=[29312], 10.00th=[29312], 20. | | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20. | 30.00th=[29568], 40.00th=[29824], 50.00th=[30080], 60. | | 30.00th=[29568], 40.00th=[29824], 50.00th=[29824], 60. | 70.00th=[36608], 80.00th=[44800], 90.00th=[49920], 95. | | 70.00th=[36608], 80.00th=[43776], 90.00th=[49408], 95. | 99.00th=[68096], 99.50th=[72192], 99.90th=[86528], 99. | | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99. | 99.99th=[91648] | | 99.99th=[90624] Type=write,Size=64K,NumJobs=1,depth=16 Type=write,Size=64K,NumJobs=1,depth=16 write: IOPS=141k, BW=8811MiB/s (9239MB/s)(86.1GiB/10001msec | write: IOPS=146k, BW=9109MiB/s (9552MB/s)(88.0GiB/10001msec slat (nsec): min=3960, max=44206, avg=5332.37, stdev=776. | slat (nsec): min=3787, max=44724, avg=5539.44, stdev=622. clat (usec): min=83, max=307, avg=107.82, stdev=13.00 | clat (usec): min=76, max=228, avg=103.90, stdev=11.08 lat (usec): min=88, max=312, avg=113.19, stdev=13.00 | lat (usec): min=82, max=233, avg=109.48, stdev=11.10 | 1.00th=[ 92], 5.00th=[ 96], 10.00th=[ 98], 20. | | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20. | 30.00th=[ 102], 40.00th=[ 103], 50.00th=[ 105], 60. | | 30.00th=[ 98], 40.00th=[ 99], 50.00th=[ 101], 60. | 70.00th=[ 110], 80.00th=[ 113], 90.00th=[ 121], 95. | | 70.00th=[ 105], 80.00th=[ 109], 90.00th=[ 120], 95. | 99.00th=[ 163], 99.50th=[ 174], 99.90th=[ 204], 99. | | 99.00th=[ 141], 99.50th=[ 149], 99.90th=[ 182], 99. | 99.99th=[ 241] | | 99.99th=[ 200] Type=write,Size=64K,NumJobs=1,depth=64 Type=write,Size=64K,NumJobs=1,depth=64 write: IOPS=144k, BW=8971MiB/s (9407MB/s)(87.6GiB/10001msec | write: IOPS=139k, BW=8708MiB/s (9131MB/s)(85.0GiB/10001msec slat (nsec): min=2732, max=48101, avg=4533.01, stdev=1509 | slat (nsec): min=2676, max=64177, avg=5388.79, stdev=1847 clat (usec): min=162, max=1015, avg=440.98, stdev=74.73 | clat (usec): min=39, max=894, avg=453.54, stdev=56.76 lat (usec): min=174, max=1022, avg=445.55, stdev=74.68 | lat (usec): min=45, max=898, avg=458.97, stdev=57.24 | 1.00th=[ 322], 5.00th=[ 326], 10.00th=[ 338], 20. | | 1.00th=[ 338], 5.00th=[ 355], 10.00th=[ 367], 20. | 30.00th=[ 400], 40.00th=[ 412], 50.00th=[ 437], 60. | | 30.00th=[ 408], 40.00th=[ 457], 50.00th=[ 478], 60. | 70.00th=[ 482], 80.00th=[ 502], 90.00th=[ 537], 95. | | 70.00th=[ 494], 80.00th=[ 498], 90.00th=[ 506], 95. | 99.00th=[ 635], 99.50th=[ 693], 99.90th=[ 799], 99. | | 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 734], 99. | 99.99th=[ 898] | | 99.99th=[ 848] Type=write,Size=64K,NumJobs=16,depth=1 Type=write,Size=64K,NumJobs=16,depth=1 write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) slat (nsec): min=4432, max=41303, avg=5462.66, stdev=782. | slat (nsec): min=4383, max=73755, avg=5607.60, stdev=1278 clat (usec): min=23, max=214, avg=85.24, stdev= 5.53 | clat (usec): min=22, max=424, avg=85.24, stdev= 8.07 lat (usec): min=37, max=220, avg=90.75, stdev= 5.58 | lat (usec): min=38, max=430, avg=90.90, stdev= 8.07 | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 82], 20. | | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 79], 20. | 30.00th=[ 84], 40.00th=[ 85], 50.00th=[ 86], 60. | | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60. | 70.00th=[ 87], 80.00th=[ 88], 90.00th=[ 90], 95. | | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 98], 95. | 99.00th=[ 103], 99.50th=[ 110], 99.90th=[ 123], 99. | | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 124], 99. | 99.99th=[ 137] | 99.99th=[ 137] Type=write,Size=64K,NumJobs=16,depth=16 Type=write,Size=64K,NumJobs=16,depth=16 write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=4024, max=70542, avg=6729.45, stdev=1425 | slat (nsec): min=3611, max=70931, avg=6507.12, stdev=1389 clat (usec): min=103, max=4145, avg=1447.24, stdev=446.02 | clat (usec): min=109, max=3915, avg=1451.13, stdev=409.41 lat (usec): min=120, max=4152, avg=1454.01, stdev=446.19 | lat (usec): min=129, max=3920, avg=1457.68, stdev=409.50 | 1.00th=[ 474], 5.00th=[ 1237], 10.00th=[ 1254], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1270], 60. | | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1287], 60. | 70.00th=[ 1369], 80.00th=[ 1450], 90.00th=[ 2540], 95. | | 70.00th=[ 1352], 80.00th=[ 1369], 90.00th=[ 2540], 95. | 99.00th=[ 2671], 99.50th=[ 2704], 99.90th=[ 2933], 99. | | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99. | 99.99th=[ 3916] | | 99.99th=[ 2769] Type=write,Size=64K,NumJobs=16,depth=64 Type=write,Size=64K,NumJobs=16,depth=64 write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) slat (usec): min=2, max=6195, avg=13.26, stdev=111.30 | slat (nsec): min=3042, max=96073, avg=6866.24, stdev=2170 clat (usec): min=112, max=16931, avg=5822.91, stdev=1645. | clat (usec): min=124, max=10103, avg=5850.40, stdev=798.1 lat (usec): min=118, max=17733, avg=5836.21, stdev=1665. | lat (usec): min=131, max=10109, avg=5857.31, stdev=798.1 | 1.00th=[ 4015], 5.00th=[ 4293], 10.00th=[ 4424], 20. | | 1.00th=[ 4752], 5.00th=[ 4883], 10.00th=[ 4948], 20. | 30.00th=[ 4883], 40.00th=[ 5080], 50.00th=[ 5276], 60. | | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5669], 60. | 70.00th=[ 6063], 80.00th=[ 6587], 90.00th=[ 7570], 95. | | 70.00th=[ 6259], 80.00th=[ 6587], 90.00th=[ 7046], 95. | 99.00th=[11600], 99.50th=[12387], 99.90th=[14091], 99. | | 99.00th=[ 7767], 99.50th=[ 7898], 99.90th=[ 8455], 99. | 99.99th=[15795] | | 99.99th=[ 8848] Type=write,Size=64K,NumJobs=56,depth=1 Type=write,Size=64K,NumJobs=56,depth=1 write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec) slat (nsec): min=4406, max=47421, avg=6031.08, stdev=1752 | slat (nsec): min=4029, max=63780, avg=5771.34, stdev=1537 clat (usec): min=49, max=915, avg=312.16, stdev=47.66 | clat (usec): min=32, max=979, avg=314.28, stdev=57.25 lat (usec): min=57, max=924, avg=318.25, stdev=47.66 | lat (usec): min=40, max=985, avg=320.11, stdev=57.29 | 1.00th=[ 194], 5.00th=[ 281], 10.00th=[ 289], 20. | | 1.00th=[ 186], 5.00th=[ 293], 10.00th=[ 297], 20. | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60. | | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60. | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95. | | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95. | 99.00th=[ 469], 99.50th=[ 482], 99.90th=[ 611], 99. | | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99. | 99.99th=[ 644] | | 99.99th=[ 750] Type=write,Size=64K,NumJobs=56,depth=16 Type=write,Size=64K,NumJobs=56,depth=16 write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) slat (usec): min=3, max=102, avg= 7.88, stdev= 2.26 | slat (nsec): min=3542, max=89310, avg=7585.26, stdev=2460 clat (usec): min=157, max=13620, avg=5092.13, stdev=1280. | clat (usec): min=190, max=14367, avg=5116.17, stdev=1396. lat (usec): min=183, max=13626, avg=5100.08, stdev=1280. | lat (usec): min=198, max=14375, avg=5123.82, stdev=1396. | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 2540], 20. | | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2573], 20. | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5080], 60. | | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5145], 60. | 70.00th=[ 5145], 80.00th=[ 5538], 90.00th=[ 6652], 95. | | 70.00th=[ 5145], 80.00th=[ 5669], 90.00th=[ 6849], 95. | 99.00th=[ 8586], 99.50th=[ 9241], 99.90th=[10552], 99. | | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[11207], 99. | 99.99th=[12256] | | 99.99th=[12780] Type=write,Size=64K,NumJobs=56,depth=64 Type=write,Size=64K,NumJobs=56,depth=64 write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10013msec) | write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(107GiB/10011msec) slat (usec): min=2, max=36257, avg=176.13, stdev=892.90 | slat (usec): min=3, max=32839, avg=176.71, stdev=925.53 clat (usec): min=184, max=83223, avg=20322.48, stdev=6000 | clat (usec): min=189, max=132076, avg=20366.61, stdev=728 lat (usec): min=188, max=83231, avg=20498.69, stdev=6090 | lat (usec): min=218, max=132086, avg=20543.39, stdev=740 | 1.00th=[ 7832], 5.00th=[10028], 10.00th=[13173], 20. | | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[10945], 20. | 30.00th=[18220], 40.00th=[19792], 50.00th=[20317], 60. | | 30.00th=[17957], 40.00th=[19530], 50.00th=[20317], 60. | 70.00th=[21103], 80.00th=[22938], 90.00th=[26870], 95. | | 70.00th=[21103], 80.00th=[23462], 90.00th=[28705], 95. | 99.00th=[40633], 99.50th=[45351], 99.90th=[54264], 99. | | 99.00th=[45876], 99.50th=[50594], 99.90th=[63177], 99. | 99.99th=[67634] | | 99.99th=[84411] With DIM on initiator side Without DIM Type=read,Size=512B,NumJobs=1,depth=1 Type=read,Size=512B,NumJobs=1,depth=1 read: IOPS=46.7k, BW=22.8MiB/s (23.9MB/s)(228MiB/10001msec | read: IOPS=47.0k, BW=22.0MiB/s (24.1MB/s)(230MiB/10001msec slat (nsec): min=2008, max=43561, avg=3175.68, stdev=2353 | slat (nsec): min=1981, max=53196, avg=3048.29, stdev=2391 clat (nsec): min=456, max=103914, avg=17680.68, stdev=638 | clat (nsec): min=442, max=107156, avg=17658.37, stdev=647 lat (usec): min=13, max=106, avg=20.93, stdev= 7.10 | lat (usec): min=13, max=109, avg=20.78, stdev= 7.18 | 1.00th=[12864], 5.00th=[12992], 10.00th=[14016], 20. | | 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20. | 30.00th=[14144], 40.00th=[14272], 50.00th=[14400], 60. | | 30.00th=[14144], 40.00th=[14144], 50.00th=[14400], 60. | 70.00th=[17280], 80.00th=[19072], 90.00th=[28032], 95. | | 70.00th=[17024], 80.00th=[19072], 90.00th=[27776], 95. | 99.00th=[40192], 99.50th=[44800], 99.90th=[50944], 99. | | 99.00th=[40704], 99.50th=[45824], 99.90th=[51456], 99. | 99.99th=[60160] | | 99.99th=[58624] Type=read,Size=512B,NumJobs=1,depth=16 Type=read,Size=512B,NumJobs=1,depth=16 read: IOPS=281k, BW=137MiB/s (144MB/s)(1371MiB/10001msec) | read: IOPS=334k, BW=163MiB/s (171MB/s)(1629MiB/10001msec) slat (nsec): min=1558, max=56900, avg=2362.92, stdev=2075 | slat (nsec): min=1564, max=28780, avg=2162.48, stdev=588. clat (usec): min=32, max=158, avg=54.17, stdev= 9.58 | clat (usec): min=17, max=118, avg=45.50, stdev= 6.30 lat (usec): min=34, max=160, avg=56.59, stdev= 9.72 | lat (usec): min=20, max=120, avg=47.70, stdev= 6.64 | 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 43], 20. | | 1.00th=[41728], 5.00th=[42752], 10.00th=[42752], 20. | 30.00th=[ 47], 40.00th=[ 53], 50.00th=[ 56], 60. | | 30.00th=[43264], 40.00th=[43776], 50.00th=[43776], 60. | 70.00th=[ 60], 80.00th=[ 62], 90.00th=[ 65], 95. | | 70.00th=[44288], 80.00th=[44288], 90.00th=[46336], 95. | 99.00th=[ 80], 99.50th=[ 85], 99.90th=[ 99], 99. | | 99.00th=[72192], 99.50th=[73216], 99.90th=[77312], 99. | 99.99th=[ 133] | | 99.99th=[86528] Type=read,Size=512B,NumJobs=1,depth=64 Type=read,Size=512B,NumJobs=1,depth=64 read: IOPS=377k, BW=184MiB/s (193MB/s)(1840MiB/10001msec) | read: IOPS=344k, BW=168MiB/s (176MB/s)(1682MiB/10001msec) slat (nsec): min=1591, max=41737, avg=1885.11, stdev=245. | slat (nsec): min=1607, max=30650, avg=2093.56, stdev=233. clat (usec): min=29, max=222, avg=167.69, stdev= 1.31 | clat (usec): min=19, max=238, avg=183.42, stdev= 2.22 lat (usec): min=32, max=224, avg=169.61, stdev= 1.33 | lat (usec): min=21, max=241, avg=185.55, stdev= 2.23 | 1.00th=[ 165], 5.00th=[ 167], 10.00th=[ 167], 20. | | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 182], 20. | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60. | | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 184], 60. | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95. | | 70.00th=[ 184], 80.00th=[ 186], 90.00th=[ 186], 95. | 99.00th=[ 172], 99.50th=[ 172], 99.90th=[ 178], 99. | | 99.00th=[ 190], 99.50th=[ 192], 99.90th=[ 196], 99. | 99.99th=[ 196] | | 99.99th=[ 210] Type=read,Size=512B,NumJobs=16,depth=1 Type=read,Size=512B,NumJobs=16,depth=1 read: IOPS=906k, BW=442MiB/s (464MB/s)(4423MiB/10001msec) | read: IOPS=907k, BW=443MiB/s (465MB/s)(4431MiB/10001msec) slat (nsec): min=1919, max=71597, avg=2515.26, stdev=515. | slat (nsec): min=1925, max=69082, avg=2505.79, stdev=869. clat (nsec): min=678, max=1006.2k, avg=14689.14, stdev=14 | clat (nsec): min=476, max=112521, avg=14672.56, stdev=182 lat (usec): min=12, max=1008, avg=17.26, stdev= 1.66 | lat (usec): min=12, max=131, avg=17.23, stdev= 2.08 | 1.00th=[11712], 5.00th=[12736], 10.00th=[13376], 20. | | 1.00th=[11584], 5.00th=[13248], 10.00th=[13376], 20. | 30.00th=[14144], 40.00th=[14400], 50.00th=[14656], 60. | | 30.00th=[13888], 40.00th=[14016], 50.00th=[14272], 60. | 70.00th=[15168], 80.00th=[15424], 90.00th=[15936], 95. | | 70.00th=[14784], 80.00th=[15296], 90.00th=[16512], 95. | 99.00th=[18048], 99.50th=[18816], 99.90th=[23424], 99. | | 99.00th=[22400], 99.50th=[24448], 99.90th=[32384], 99. | 99.99th=[62208] | | 99.99th=[35584] Type=read,Size=512B,NumJobs=16,depth=16 Type=read,Size=512B,NumJobs=16,depth=16 read: IOPS=4317k, BW=2108MiB/s (2211MB/s)(20.6GiB/10001mse | read: IOPS=4055k, BW=1980MiB/s (2076MB/s)(19.3GiB/10003mse slat (nsec): min=1572, max=75553, avg=2642.25, stdev=1686 | slat (nsec): min=1572, max=239521, avg=2836.25, stdev=282 clat (usec): min=21, max=1043, avg=56.25, stdev=12.46 | clat (usec): min=9, max=616, avg=59.85, stdev=55.40 lat (usec): min=23, max=1046, avg=58.94, stdev=13.08 | lat (usec): min=15, max=630, avg=62.74, stdev=57.98 | 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20. | | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 53], 60. | | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60. | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95. | | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 60], 95. | 99.00th=[ 87], 99.50th=[ 112], 99.90th=[ 124], 99. | | 99.00th=[ 326], 99.50th=[ 343], 99.90th=[ 367], 99. | 99.99th=[ 289] | | 99.99th=[ 519] Type=read,Size=512B,NumJobs=16,depth=64 Type=read,Size=512B,NumJobs=16,depth=64 read: IOPS=4484k, BW=2189MiB/s (2296MB/s)(21.4GiB/10001mse | read: IOPS=3451k, BW=1685MiB/s (1767MB/s)(16.5GiB/10001mse slat (nsec): min=1566, max=71087, avg=2603.51, stdev=2276 | slat (nsec): min=1569, max=204726, avg=3322.74, stdev=274 clat (usec): min=19, max=1784, avg=225.36, stdev=53.73 | clat (usec): min=5, max=1543, avg=292.82, stdev=220.59 lat (usec): min=26, max=1807, avg=228.01, stdev=54.38 | lat (usec): min=14, max=1561, avg=296.20, stdev=223.03 | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 176], 20. | | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20. | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 231], 60. | | 30.00th=[ 188], 40.00th=[ 227], 50.00th=[ 243], 60. | 70.00th=[ 247], 80.00th=[ 269], 90.00th=[ 302], 95. | | 70.00th=[ 249], 80.00th=[ 285], 90.00th=[ 685], 95. | 99.00th=[ 375], 99.50th=[ 388], 99.90th=[ 490], 99. | | 99.00th=[ 1270], 99.50th=[ 1319], 99.90th=[ 1385], 99. | 99.99th=[ 1631] | | 99.99th=[ 1450] Type=read,Size=512B,NumJobs=56,depth=1 Type=read,Size=512B,NumJobs=56,depth=1 read: IOPS=2301k, BW=1123MiB/s (1178MB/s)(10.0GiB/10002mse | read: IOPS=2351k, BW=1148MiB/s (1204MB/s)(11.2GiB/10002mse slat (nsec): min=1723, max=958519, avg=3130.18, stdev=300 | slat (nsec): min=1744, max=502891, avg=3525.21, stdev=199 clat (nsec): min=456, max=987494, avg=20564.41, stdev=170 | clat (nsec): min=456, max=781278, avg=19563.02, stdev=418 lat (usec): min=12, max=989, avg=23.77, stdev=17.43 | lat (usec): min=11, max=785, avg=23.18, stdev= 4.79 | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20. | | 1.00th=[13120], 5.00th=[14912], 10.00th=[15936], 20. | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17792], 40.00th=[18304], 50.00th=[19072], 60. | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20608], 80.00th=[21632], 90.00th=[23680], 95. | 99.00th=[ 37], 99.50th=[ 45], 99.90th=[ 367], 99. | | 99.00th=[31872], 99.50th=[35072], 99.90th=[41728], 99. | 99.99th=[ 545] | | 99.99th=[56064] Type=read,Size=512B,NumJobs=56,depth=16 Type=read,Size=512B,NumJobs=56,depth=16 read: IOPS=7830k, BW=3823MiB/s (4009MB/s)(37.3GiB/10002mse | read: IOPS=7234k, BW=3532MiB/s (3704MB/s)(34.5GiB/10002mse slat (nsec): min=1591, max=2609.7M, avg=5126.50, stdev=10 | slat (nsec): min=1588, max=13073k, avg=5296.03, stdev=602 clat (usec): min=16, max=3527.6k, avg=108.61, stdev=6148. | clat (usec): min=9, max=13311, avg=117.66, stdev=77.29 lat (usec): min=25, max=3527.6k, avg=113.83, stdev=6235. | lat (usec): min=18, max=13322, avg=123.07, stdev=80.61 | 1.00th=[ 47], 5.00th=[ 54], 10.00th=[ 63], 20. | | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20. | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 80], 60. | | 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 81], 60. | 70.00th=[ 97], 80.00th=[ 100], 90.00th=[ 105], 95. | | 70.00th=[ 108], 80.00th=[ 184], 90.00th=[ 247], 95. | 99.00th=[ 457], 99.50th=[ 519], 99.90th=[ 611], 99. | | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 494], 99. | 99.99th=[ 889] | | 99.99th=[ 594] Type=read,Size=512B,NumJobs=56,depth=64 Type=read,Size=512B,NumJobs=56,depth=64 read: IOPS=8456k, BW=4129MiB/s (4330MB/s)(40.3GiB/10002mse | read: IOPS=7769k, BW=3793MiB/s (3978MB/s)(37.1GiB/10002mse slat (nsec): min=1607, max=2432.8k, avg=4995.25, stdev=87 | slat (nsec): min=1583, max=3785.4k, avg=5319.20, stdev=81 clat (usec): min=28, max=3427, avg=418.02, stdev=272.37 | clat (usec): min=2, max=5212, avg=455.02, stdev=334.71 lat (usec): min=34, max=3431, avg=423.12, stdev=275.63 | lat (usec): min=17, max=5214, avg=460.45, stdev=338.64 | 1.00th=[ 281], 5.00th=[ 293], 10.00th=[ 302], 20. | | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 281], 20. | 30.00th=[ 314], 40.00th=[ 322], 50.00th=[ 383], 60. | | 30.00th=[ 293], 40.00th=[ 297], 50.00th=[ 302], 60. | 70.00th=[ 408], 80.00th=[ 412], 90.00th=[ 429], 95. | | 70.00th=[ 310], 80.00th=[ 437], 90.00th=[ 1074], 95. | 99.00th=[ 1909], 99.50th=[ 1991], 99.90th=[ 2180], 99. | | 99.00th=[ 1401], 99.50th=[ 1991], 99.90th=[ 2245], 99. | 99.99th=[ 2442] | 99.99th=[ 2442] Type=read,Size=4K,NumJobs=1,depth=1 Type=read,Size=4K,NumJobs=1,depth=1 read: IOPS=43.6k, BW=170MiB/s (178MB/s)(1701MiB/10001msec) | read: IOPS=54.4k, BW=213MiB/s (223MB/s)(2126MiB/10001msec) slat (nsec): min=2041, max=41259, avg=3359.33, stdev=2676 | slat (nsec): min=2044, max=28781, avg=2556.03, stdev=1441 clat (nsec): min=449, max=95097, avg=18989.95, stdev=6853 | clat (nsec): min=621, max=70566, avg=15359.54, stdev=5250 lat (nsec): min=14883, max=99275, avg=22427.07, stdev=76 | lat (nsec): min=13302, max=73823, avg=17966.25, stdev=56 | 1.00th=[14400], 5.00th=[14656], 10.00th=[14656], 20. | | 1.00th=[13120], 5.00th=[13248], 10.00th=[13248], 20. | 30.00th=[14912], 40.00th=[15040], 50.00th=[15296], 60. | | 30.00th=[13376], 40.00th=[13376], 50.00th=[13376], 60. | 70.00th=[19072], 80.00th=[22144], 90.00th=[30592], 95. | | 70.00th=[13632], 80.00th=[14144], 90.00th=[19328], 95. | 99.00th=[43776], 99.50th=[46848], 99.90th=[52480], 99. | | 99.00th=[35584], 99.50th=[36608], 99.90th=[45312], 99. | 99.99th=[63744] | | 99.99th=[53504] Type=read,Size=4K,NumJobs=1,depth=16 Type=read,Size=4K,NumJobs=1,depth=16 read: IOPS=345k, BW=1346MiB/s (1411MB/s)(13.1GiB/10001msec | read: IOPS=334k, BW=1306MiB/s (1370MB/s)(12.8GiB/10001msec slat (nsec): min=1584, max=67990, avg=2073.10, stdev=588. | slat (nsec): min=1696, max=45538, avg=2168.76, stdev=208. clat (usec): min=32, max=154, avg=44.06, stdev= 4.57 | clat (usec): min=18, max=148, avg=45.39, stdev= 1.47 lat (usec): min=35, max=156, avg=46.17, stdev= 4.62 | lat (usec): min=20, max=192, avg=47.59, stdev= 1.48 | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 43], 20. | | 1.00th=[43776], 5.00th=[44288], 10.00th=[44288], 20. | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60. | | 30.00th=[44800], 40.00th=[45312], 50.00th=[45312], 60. | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 45], 95. | | 70.00th=[45312], 80.00th=[45824], 90.00th=[45824], 95. | 99.00th=[ 67], 99.50th=[ 74], 99.90th=[ 86], 99. | | 99.00th=[49408], 99.50th=[54528], 99.90th=[64256], 99. | 99.99th=[ 117] | | 99.99th=[70144] Type=read,Size=4K,NumJobs=1,depth=64 Type=read,Size=4K,NumJobs=1,depth=64 read: IOPS=257k, BW=1004MiB/s (1053MB/s)(9.80GiB/10001msec | read: IOPS=338k, BW=1319MiB/s (1383MB/s)(12.9GiB/10001msec slat (nsec): min=1562, max=50661, avg=2614.87, stdev=2515 | slat (nsec): min=1621, max=52632, avg=2145.29, stdev=209. clat (usec): min=49, max=325, avg=245.85, stdev=11.72 | clat (usec): min=36, max=236, avg=187.02, stdev= 2.05 lat (usec): min=50, max=328, avg=248.54, stdev=11.76 | lat (usec): min=38, max=238, avg=189.20, stdev= 2.06 | 1.00th=[ 219], 5.00th=[ 227], 10.00th=[ 231], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20. | 30.00th=[ 241], 40.00th=[ 243], 50.00th=[ 245], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60. | 70.00th=[ 253], 80.00th=[ 255], 90.00th=[ 262], 95. | | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 190], 95. | 99.00th=[ 273], 99.50th=[ 273], 99.90th=[ 281], 99. | | 99.00th=[ 192], 99.50th=[ 196], 99.90th=[ 200], 99. | 99.99th=[ 285] | | 99.99th=[ 217] Type=read,Size=4K,NumJobs=16,depth=1 Type=read,Size=4K,NumJobs=16,depth=1 read: IOPS=849k, BW=3318MiB/s (3479MB/s)(32.4GiB/10002msec | read: IOPS=869k, BW=3396MiB/s (3561MB/s)(33.2GiB/10002msec slat (nsec): min=1894, max=65881, avg=2560.26, stdev=657. | slat (nsec): min=1890, max=70947, avg=2497.58, stdev=813. clat (nsec): min=516, max=708489, avg=15808.69, stdev=196 | clat (nsec): min=590, max=79209, avg=15456.14, stdev=1968 lat (usec): min=13, max=710, avg=18.42, stdev= 2.19 | lat (usec): min=12, max=110, avg=18.01, stdev= 2.18 | 1.00th=[13120], 5.00th=[14016], 10.00th=[14272], 20. | | 1.00th=[12352], 5.00th=[13888], 10.00th=[14144], 20. | 30.00th=[15040], 40.00th=[15424], 50.00th=[15680], 60. | | 30.00th=[14528], 40.00th=[14784], 50.00th=[15040], 60. | 70.00th=[16320], 80.00th=[16768], 90.00th=[17280], 95. | | 70.00th=[15552], 80.00th=[16192], 90.00th=[17280], 95. | 99.00th=[19840], 99.50th=[21120], 99.90th=[34560], 99. | | 99.00th=[23936], 99.50th=[25728], 99.90th=[33536], 99. | 99.99th=[76288] | | 99.99th=[36096] Type=read,Size=4K,NumJobs=16,depth=16 Type=read,Size=4K,NumJobs=16,depth=16 read: IOPS=2593k, BW=9.89GiB/s (10.6GB/s)(98.9GiB/10002mse | read: IOPS=1843k, BW=7201MiB/s (7551MB/s)(70.3GiB/10002mse slat (nsec): min=1568, max=163092, avg=2599.50, stdev=145 | slat (nsec): min=1582, max=104435, avg=3170.92, stdev=170 clat (usec): min=34, max=4287.1k, avg=95.73, stdev=9113.1 | clat (usec): min=40, max=554, avg=135.18, stdev=32.57 lat (usec): min=36, max=4287.1k, avg=98.38, stdev=9113.1 | lat (usec): min=42, max=557, avg=138.42, stdev=32.59 | 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 57], 20. | | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 104], 20. | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 73], 60. | | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 135], 60. | 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 96], 95. | | 70.00th=[ 141], 80.00th=[ 147], 90.00th=[ 161], 95. | 99.00th=[ 104], 99.50th=[ 106], 99.90th=[ 126], 99. | | 99.00th=[ 262], 99.50th=[ 306], 99.90th=[ 392], 99. | 99.99th=[ 176] | | 99.99th=[ 461] Type=read,Size=4K,NumJobs=16,depth=64 Type=read,Size=4K,NumJobs=16,depth=64 read: IOPS=2646k, BW=10.1GiB/s (10.8GB/s)(101GiB/10001msec | read: IOPS=1917k, BW=7487MiB/s (7850MB/s)(73.1GiB/10001mse slat (nsec): min=1566, max=100447, avg=2310.80, stdev=263 | slat (nsec): min=1566, max=653412, avg=3347.14, stdev=385 clat (usec): min=21, max=1440, avg=384.23, stdev=61.42 | clat (usec): min=43, max=1445, avg=530.39, stdev=127.84 lat (usec): min=24, max=1443, avg=386.58, stdev=62.18 | lat (usec): min=45, max=1619, avg=533.81, stdev=128.49 | 1.00th=[ 330], 5.00th=[ 347], 10.00th=[ 347], 20. | | 1.00th=[ 188], 5.00th=[ 351], 10.00th=[ 420], 20. | 30.00th=[ 363], 40.00th=[ 371], 50.00th=[ 379], 60. | | 30.00th=[ 486], 40.00th=[ 502], 50.00th=[ 519], 60. | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 396], 95. | | 70.00th=[ 553], 80.00th=[ 578], 90.00th=[ 644], 95. | 99.00th=[ 717], 99.50th=[ 725], 99.90th=[ 750], 99. | | 99.00th=[ 1012], 99.50th=[ 1037], 99.90th=[ 1090], 99. | 99.99th=[ 766] | | 99.99th=[ 1205] Type=read,Size=4K,NumJobs=56,depth=1 Type=read,Size=4K,NumJobs=56,depth=1 read: IOPS=1514k, BW=5912MiB/s (6200MB/s)(57.7GiB/10002mse | read: IOPS=1571k, BW=6138MiB/s (6436MB/s)(59.9GiB/10001mse slat (nsec): min=1722, max=911960, avg=2705.80, stdev=141 | slat (nsec): min=1693, max=505053, avg=2771.65, stdev=110 clat (nsec): min=466, max=4359.5k, avg=33747.42, stdev=23 | clat (nsec): min=479, max=1348.1k, avg=32311.24, stdev=36 lat (usec): min=13, max=4362, avg=36.52, stdev=23.37 | lat (usec): min=15, max=1350, avg=35.15, stdev= 3.81 | 1.00th=[ 20], 5.00th=[ 24], 10.00th=[ 27], 20. | | 1.00th=[26752], 5.00th=[28800], 10.00th=[29568], 20. | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 34], 60. | | 30.00th=[30848], 40.00th=[31360], 50.00th=[31872], 60. | 70.00th=[ 35], 80.00th=[ 36], 90.00th=[ 37], 95. | | 70.00th=[33024], 80.00th=[33536], 90.00th=[35584], 95. | 99.00th=[ 48], 99.50th=[ 143], 99.90th=[ 433], 99. | | 99.00th=[42240], 99.50th=[43776], 99.90th=[49920], 99. | 99.99th=[ 594] | | 99.99th=[59136] Type=read,Size=4K,NumJobs=56,depth=16 Type=read,Size=4K,NumJobs=56,depth=16 read: IOPS=2381k, BW=9299MiB/s (9751MB/s)(97.7GiB/10756mse | read: IOPS=2009k, BW=7846MiB/s (8227MB/s)(76.6GiB/10001mse slat (nsec): min=1573, max=1580.7M, avg=2676.88, stdev=44 | slat (nsec): min=1566, max=379585, avg=3204.59, stdev=230 clat (usec): min=17, max=4303.4k, avg=355.77, stdev=27892 | clat (usec): min=44, max=1761, avg=442.22, stdev=244.27 lat (usec): min=20, max=4303.4k, avg=358.50, stdev=27901 | lat (usec): min=46, max=1764, avg=445.50, stdev=244.24 | 1.00th=[ 44], 5.00th=[ 55], 10.00th=[ 62], 20. | | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 215], 20. | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 81], 60. | | 30.00th=[ 273], 40.00th=[ 306], 50.00th=[ 351], 60. | 70.00th=[ 109], 80.00th=[ 135], 90.00th=[ 167], 95. | | 70.00th=[ 506], 80.00th=[ 644], 90.00th=[ 832], 95. | 99.00th=[ 285], 99.50th=[ 400], 99.90th=[ 506], 99. | | 99.00th=[ 1156], 99.50th=[ 1221], 99.90th=[ 1336], 99. | 99.99th=[ 1172] | | 99.99th=[ 1450] Type=read,Size=4K,NumJobs=56,depth=64 Type=read,Size=4K,NumJobs=56,depth=64 read: IOPS=2496k, BW=9749MiB/s (10.2GB/s)(95.2GiB/10003mse | read: IOPS=1801k, BW=7036MiB/s (7378MB/s)(68.7GiB/10002mse slat (nsec): min=1568, max=2721.3M, avg=14509.11, stdev=1 | slat (nsec): min=1586, max=2410.2k, avg=20136.27, stdev=8 clat (usec): min=35, max=2725.6k, avg=1420.77, stdev=1481 | clat (usec): min=58, max=8215, avg=1968.75, stdev=947.22 lat (usec): min=42, max=2725.9k, avg=1435.33, stdev=1491 | lat (usec): min=60, max=8632, avg=1988.96, stdev=962.84 | 1.00th=[ 570], 5.00th=[ 611], 10.00th=[ 676], 20. | | 1.00th=[ 449], 5.00th=[ 734], 10.00th=[ 865], 20. | 30.00th=[ 742], 40.00th=[ 1221], 50.00th=[ 1369], 60. | | 30.00th=[ 1418], 40.00th=[ 1729], 50.00th=[ 1909], 60. | 70.00th=[ 1434], 80.00th=[ 1500], 90.00th=[ 2147], 95. | | 70.00th=[ 2212], 80.00th=[ 2507], 90.00th=[ 3326], 95. | 99.00th=[ 3163], 99.50th=[ 3261], 99.90th=[ 3654], 99. | | 99.00th=[ 4817], 99.50th=[ 5080], 99.90th=[ 5735], 99. | 99.99th=[ 4686] | | 99.99th=[ 6587] Type=read,Size=64K,NumJobs=1,depth=1 Type=read,Size=64K,NumJobs=1,depth=1 read: IOPS=29.7k, BW=1854MiB/s (1944MB/s)(18.1GiB/10001mse | read: IOPS=29.8k, BW=1864MiB/s (1955MB/s)(18.2GiB/10001mse slat (nsec): min=4746, max=53468, avg=6046.72, stdev=2806 | slat (nsec): min=4768, max=95484, avg=6119.74, stdev=2779 clat (usec): min=3, max=132, avg=27.14, stdev= 7.87 | clat (nsec): min=1555, max=75556, avg=26881.73, stdev=781 lat (usec): min=25, max=137, avg=33.23, stdev= 9.11 | lat (usec): min=25, max=102, avg=33.05, stdev= 9.07 | 1.00th=[21888], 5.00th=[21888], 10.00th=[21888], 20. | | 1.00th=[21632], 5.00th=[21632], 10.00th=[21632], 20. | 30.00th=[22144], 40.00th=[22144], 50.00th=[22400], 60. | | 30.00th=[21888], 40.00th=[22144], 50.00th=[22144], 60. | 70.00th=[28288], 80.00th=[35072], 90.00th=[41216], 95. | | 70.00th=[28032], 80.00th=[34048], 90.00th=[40704], 95. | 99.00th=[47360], 99.50th=[49920], 99.90th=[61696], 99. | | 99.00th=[47360], 99.50th=[49920], 99.90th=[59648], 99. | 99.99th=[70144] | 99.99th=[70144] Type=read,Size=64K,NumJobs=1,depth=16 Type=read,Size=64K,NumJobs=1,depth=16 read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec) read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec) slat (nsec): min=3300, max=48724, avg=4702.06, stdev=397. | slat (nsec): min=3756, max=65673, avg=4930.28, stdev=334. clat (usec): min=59, max=241, avg=88.09, stdev= 4.24 | clat (usec): min=83, max=243, avg=87.86, stdev= 2.04 lat (usec): min=65, max=277, avg=92.83, stdev= 4.22 | lat (usec): min=87, max=310, avg=92.83, stdev= 2.05 | 1.00th=[ 82], 5.00th=[ 83], 10.00th=[ 83], 20. | | 1.00th=[ 86], 5.00th=[ 87], 10.00th=[ 87], 20. | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60. | | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60. | 70.00th=[ 91], 80.00th=[ 92], 90.00th=[ 94], 95. | | 70.00th=[ 88], 80.00th=[ 89], 90.00th=[ 91], 95. | 99.00th=[ 98], 99.50th=[ 99], 99.90th=[ 111], 99. | | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 108], 99. | 99.99th=[ 145] | | 99.99th=[ 114] Type=read,Size=64K,NumJobs=1,depth=64 Type=read,Size=64K,NumJobs=1,depth=64 read: IOPS=172k, BW=10.5GiB/s (11.2GB/s)(105GiB/10001msec) | read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec) slat (usec): min=2, max=420, avg= 4.14, stdev= 3.46 | slat (usec): min=2, max=390, avg= 4.15, stdev= 1.56 clat (usec): min=299, max=1058, avg=368.25, stdev=18.60 | clat (usec): min=37, max=947, avg=376.95, stdev=15.70 lat (usec): min=306, max=1099, avg=372.44, stdev=18.47 | lat (usec): min=42, max=988, avg=381.15, stdev=16.25 | 1.00th=[ 326], 5.00th=[ 343], 10.00th=[ 351], 20. | | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 367], 20. | 30.00th=[ 359], 40.00th=[ 367], 50.00th=[ 367], 60. | | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 70.00th=[ 375], 80.00th=[ 379], 90.00th=[ 388], 95. | | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 400], 95. | 99.00th=[ 433], 99.50th=[ 445], 99.90th=[ 465], 99. | | 99.00th=[ 412], 99.50th=[ 416], 99.90th=[ 429], 99. | 99.99th=[ 758] | | 99.99th=[ 668] Type=read,Size=64K,NumJobs=16,depth=1 Type=read,Size=64K,NumJobs=16,depth=1 read: IOPS=150k, BW=9390MiB/s (9847MB/s)(105GiB/11428msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec) slat (nsec): min=3750, max=58669, avg=5120.99, stdev=2896 | slat (nsec): min=3480, max=74756, avg=5273.21, stdev=2961 clat (nsec): min=494, max=2276.2M, avg=88365.74, stdev=17 | clat (nsec): min=616, max=453786, avg=87271.90, stdev=161 lat (usec): min=26, max=2276.2k, avg=93.55, stdev=1737.1 | lat (usec): min=26, max=458, avg=92.60, stdev=16.20 | 1.00th=[ 36], 5.00th=[ 55], 10.00th=[ 68], 20. | | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 68], 20. | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 89], 60. | | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60. | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 103], 95. | | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 104], 95. | 99.00th=[ 124], 99.50th=[ 129], 99.90th=[ 141], 99. | | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 143], 99. | 99.99th=[ 165] | | 99.99th=[ 161] Type=read,Size=64K,NumJobs=16,depth=16 Type=read,Size=64K,NumJobs=16,depth=16 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10003msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=3284, max=53892, avg=4935.30, stdev=1716 | slat (nsec): min=3866, max=57337, avg=5204.47, stdev=1220 clat (usec): min=102, max=4744, avg=1455.23, stdev=493.14 | clat (usec): min=116, max=3985, avg=1453.77, stdev=349.92 lat (usec): min=116, max=4749, avg=1460.22, stdev=493.10 | lat (usec): min=128, max=3990, avg=1459.03, stdev=349.93 | 1.00th=[ 1172], 5.00th=[ 1172], 10.00th=[ 1188], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1352], 60. | | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60. | 70.00th=[ 1369], 80.00th=[ 1385], 90.00th=[ 1401], 95. | | 70.00th=[ 1369], 80.00th=[ 1401], 90.00th=[ 1467], 95. | 99.00th=[ 3818], 99.50th=[ 3851], 99.90th=[ 3851], 99. | | 99.00th=[ 2737], 99.50th=[ 2769], 99.90th=[ 2769], 99. | 99.99th=[ 3884] | | 99.99th=[ 2802] Type=read,Size=64K,NumJobs=16,depth=64 Type=read,Size=64K,NumJobs=16,depth=64 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) slat (usec): min=2, max=803, avg=16.08, stdev=83.27 | slat (usec): min=2, max=1185, avg= 9.71, stdev=60.22 clat (usec): min=109, max=15039, avg=5820.28, stdev=1740. | clat (usec): min=116, max=16490, avg=5824.81, stdev=1345. lat (usec): min=116, max=15042, avg=5836.41, stdev=1769. | lat (usec): min=133, max=16494, avg=5834.58, stdev=1367. | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4752], 20. | | 1.00th=[ 5407], 5.00th=[ 5407], 10.00th=[ 5473], 20. | 30.00th=[ 4752], 40.00th=[ 5080], 50.00th=[ 5080], 60. | | 30.00th=[ 5473], 40.00th=[ 5473], 50.00th=[ 5473], 60. | 70.00th=[ 5473], 80.00th=[ 5866], 90.00th=[ 9503], 95. | | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[ 5538], 95. | 99.00th=[10552], 99.50th=[10945], 99.90th=[10945], 99. | | 99.00th=[10945], 99.50th=[10945], 99.90th=[11076], 99. | 99.99th=[11076] | | 99.99th=[11338] Type=read,Size=64K,NumJobs=56,depth=1 Type=read,Size=64K,NumJobs=56,depth=1 read: IOPS=160k, BW=9979MiB/s (10.5GB/s)(107GiB/10962msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=3668, max=66111, avg=4685.24, stdev=1778 | slat (nsec): min=3801, max=63985, avg=4896.45, stdev=1489 clat (usec): min=36, max=3544.7k, avg=317.51, stdev=6531. | clat (usec): min=29, max=550, avg=314.45, stdev=76.21 lat (usec): min=42, max=3544.7k, avg=322.25, stdev=6531. | lat (usec): min=35, max=555, avg=319.41, stdev=76.15 | 1.00th=[ 161], 5.00th=[ 169], 10.00th=[ 174], 20. | | 1.00th=[ 174], 5.00th=[ 178], 10.00th=[ 194], 20. | 30.00th=[ 285], 40.00th=[ 293], 50.00th=[ 297], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60. | 70.00th=[ 306], 80.00th=[ 318], 90.00th=[ 445], 95. | | 70.00th=[ 310], 80.00th=[ 322], 90.00th=[ 457], 95. | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99. | | 99.00th=[ 474], 99.50th=[ 482], 99.90th=[ 490], 99. | 99.99th=[ 644] | | 99.99th=[ 506] Type=read,Size=64K,NumJobs=56,depth=16 Type=read,Size=64K,NumJobs=56,depth=16 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10007msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) slat (usec): min=3, max=171, avg= 4.99, stdev= 2.81 | slat (usec): min=3, max=251, avg= 5.40, stdev= 2.34 clat (usec): min=784, max=14310, avg=5117.02, stdev=1736. | clat (usec): min=236, max=14486, avg=5106.35, stdev=1241. lat (usec): min=884, max=14315, avg=5122.07, stdev=1736. | lat (usec): min=265, max=14491, avg=5111.80, stdev=1241. | 1.00th=[ 2409], 5.00th=[ 2442], 10.00th=[ 2474], 20. | | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 4883], 20. | 30.00th=[ 4883], 40.00th=[ 4948], 50.00th=[ 4948], 60. | | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5080], 60. | 70.00th=[ 5014], 80.00th=[ 5145], 90.00th=[ 7439], 95. | | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 7373], 95. | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[10159], 99. | | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[ 9896], 99. | 99.99th=[10290] | | 99.99th=[10028] Type=read,Size=64K,NumJobs=56,depth=64 Type=read,Size=64K,NumJobs=56,depth=64 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10016msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10018msec) slat (usec): min=2, max=3494.5k, avg=194.57, stdev=10422. | slat (usec): min=2, max=10184, avg=189.22, stdev=637.78 clat (usec): min=138, max=3513.0k, avg=20256.38, stdev=11 | clat (usec): min=347, max=61952, avg=20252.15, stdev=7056 lat (usec): min=178, max=3513.0k, avg=20451.00, stdev=11 | lat (usec): min=351, max=65141, avg=20441.44, stdev=7203 | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20. | | 1.00th=[ 9503], 5.00th=[ 9765], 10.00th=[ 9896], 20. | 30.00th=[ 10], 40.00th=[ 15], 50.00th=[ 19], 60. | | 30.00th=[19530], 40.00th=[19530], 50.00th=[19792], 60. | 70.00th=[ 20], 80.00th=[ 20], 90.00th=[ 28], 95. | | 70.00th=[20317], 80.00th=[20579], 90.00th=[29492], 95. | 99.00th=[ 40], 99.50th=[ 40], 99.90th=[ 3473], 99. | | 99.00th=[41157], 99.50th=[58983], 99.90th=[59507], 99. | 99.99th=[ 3507] | | 99.99th=[61080] Type=write,Size=512B,NumJobs=1,depth=1 Type=write,Size=512B,NumJobs=1,depth=1 write: IOPS=46.5k, BW=22.7MiB/s (23.8MB/s)(227MiB/10001msec | write: IOPS=56.0k, BW=27.4MiB/s (28.7MB/s)(274MiB/10001msec slat (nsec): min=2038, max=46160, avg=3144.53, stdev=2438 | slat (nsec): min=2112, max=43957, avg=2561.65, stdev=1423 clat (nsec): min=449, max=133529, avg=17730.81, stdev=670 | clat (nsec): min=633, max=60031, avg=14812.68, stdev=5638 lat (usec): min=13, max=136, avg=20.95, stdev= 7.40 | lat (nsec): min=12369, max=72124, avg=17416.39, stdev=60 | 1.00th=[13504], 5.00th=[13504], 10.00th=[13632], 20. | | 1.00th=[12352], 5.00th=[12352], 10.00th=[12480], 20. | 30.00th=[13760], 40.00th=[13888], 50.00th=[14016], 60. | | 30.00th=[12480], 40.00th=[12608], 50.00th=[12608], 60. | 70.00th=[17792], 80.00th=[20608], 90.00th=[28800], 95. | | 70.00th=[12864], 80.00th=[14656], 90.00th=[20096], 95. | 99.00th=[41728], 99.50th=[45824], 99.90th=[51456], 99. | | 99.00th=[35072], 99.50th=[36608], 99.90th=[47360], 99. | 99.99th=[64256] | | 99.99th=[54016] Type=write,Size=512B,NumJobs=1,depth=16 Type=write,Size=512B,NumJobs=1,depth=16 write: IOPS=255k, BW=125MiB/s (131MB/s)(1248MiB/10001msec) | write: IOPS=328k, BW=160MiB/s (168MB/s)(1601MiB/10001msec) slat (nsec): min=1628, max=69589, avg=2649.87, stdev=2876 | slat (nsec): min=1641, max=32941, avg=2199.97, stdev=236. clat (usec): min=23, max=182, avg=59.49, stdev= 8.44 | clat (usec): min=14, max=113, avg=46.29, stdev= 1.27 lat (usec): min=25, max=184, avg=62.20, stdev= 8.26 | lat (usec): min=16, max=115, avg=48.52, stdev= 1.28 | 1.00th=[ 45], 5.00th=[ 46], 10.00th=[ 48], 20. | | 1.00th=[43776], 5.00th=[44800], 10.00th=[45312], 20. | 30.00th=[ 58], 40.00th=[ 59], 50.00th=[ 60], 60. | | 30.00th=[45824], 40.00th=[45824], 50.00th=[46336], 60. | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 70], 95. | | 70.00th=[46336], 80.00th=[46848], 90.00th=[47360], 95. | 99.00th=[ 85], 99.50th=[ 90], 99.90th=[ 103], 99. | | 99.00th=[50432], 99.50th=[53504], 99.90th=[57600], 99. | 99.99th=[ 139] | | 99.99th=[63744] Type=write,Size=512B,NumJobs=1,depth=64 Type=write,Size=512B,NumJobs=1,depth=64 write: IOPS=262k, BW=128MiB/s (134MB/s)(1277MiB/10001msec) | write: IOPS=337k, BW=164MiB/s (172MB/s)(1644MiB/10001msec) slat (nsec): min=1626, max=45520, avg=2718.39, stdev=3134 | slat (nsec): min=1642, max=43437, avg=2146.30, stdev=234. clat (usec): min=31, max=304, avg=241.50, stdev=10.69 | clat (usec): min=16, max=235, avg=187.62, stdev= 2.46 lat (usec): min=33, max=307, avg=244.27, stdev=10.71 | lat (usec): min=18, max=237, avg=189.80, stdev= 2.48 | 1.00th=[ 221], 5.00th=[ 225], 10.00th=[ 229], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20. | 30.00th=[ 235], 40.00th=[ 237], 50.00th=[ 241], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60. | 70.00th=[ 247], 80.00th=[ 251], 90.00th=[ 258], 95. | | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 192], 95. | 99.00th=[ 265], 99.50th=[ 265], 99.90th=[ 269], 99. | | 99.00th=[ 194], 99.50th=[ 196], 99.90th=[ 200], 99. | 99.99th=[ 277] | | 99.99th=[ 204] Type=write,Size=512B,NumJobs=16,depth=1 Type=write,Size=512B,NumJobs=16,depth=1 write: IOPS=886k, BW=433MiB/s (454MB/s)(4327MiB/10002msec) | write: IOPS=914k, BW=446MiB/s (468MB/s)(4462MiB/10001msec) slat (nsec): min=1827, max=51604, avg=2633.04, stdev=591. | slat (nsec): min=1818, max=83376, avg=2566.34, stdev=824. clat (nsec): min=515, max=695999, avg=14905.52, stdev=209 | clat (nsec): min=465, max=343253, avg=14459.07, stdev=181 lat (usec): min=9, max=698, avg=17.60, stdev= 2.26 | lat (usec): min=11, max=345, avg=17.08, stdev= 2.09 | 1.00th=[12736], 5.00th=[13376], 10.00th=[13632], 20. | | 1.00th=[12224], 5.00th=[12992], 10.00th=[13248], 20. | 30.00th=[14400], 40.00th=[14528], 50.00th=[14784], 60. | | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60. | 70.00th=[15168], 80.00th=[15552], 90.00th=[16064], 95. | | 70.00th=[14528], 80.00th=[15040], 90.00th=[16320], 95. | 99.00th=[18560], 99.50th=[19840], 99.90th=[28544], 99. | | 99.00th=[21888], 99.50th=[23936], 99.90th=[32384], 99. | 99.99th=[87552] | | 99.99th=[35584] Type=write,Size=512B,NumJobs=16,depth=16 Type=write,Size=512B,NumJobs=16,depth=16 write: IOPS=4155k, BW=2029MiB/s (2127MB/s)(19.8GiB/10002mse | write: IOPS=3218k, BW=1571MiB/s (1648MB/s)(15.3GiB/10001mse slat (nsec): min=1622, max=171986, avg=2797.12, stdev=184 | slat (nsec): min=1621, max=154997, avg=3548.60, stdev=323 clat (usec): min=18, max=1041, avg=58.37, stdev=10.11 | clat (usec): min=14, max=473, avg=75.44, stdev=65.86 lat (usec): min=20, max=1043, avg=61.22, stdev=10.54 | lat (usec): min=17, max=489, avg=79.05, stdev=68.82 | 1.00th=[ 44], 5.00th=[ 45], 10.00th=[ 47], 20. | | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 46], 20. | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 60], 60. | | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 52], 60. | 70.00th=[ 65], 80.00th=[ 67], 90.00th=[ 73], 95. | | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 178], 95. | 99.00th=[ 81], 99.50th=[ 82], 99.90th=[ 86], 99. | | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99. | 99.99th=[ 293] | | 99.99th=[ 367] Type=write,Size=512B,NumJobs=16,depth=64 Type=write,Size=512B,NumJobs=16,depth=64 write: IOPS=4304k, BW=2102MiB/s (2204MB/s)(20.5GiB/10001mse | write: IOPS=3973k, BW=1940MiB/s (2034MB/s)(18.0GiB/10002mse slat (nsec): min=1621, max=167405, avg=2712.72, stdev=195 | slat (nsec): min=1627, max=171980, avg=2926.70, stdev=209 clat (usec): min=23, max=1767, avg=234.76, stdev=51.39 | clat (usec): min=10, max=1606, avg=254.29, stdev=176.36 lat (usec): min=25, max=1788, avg=237.52, stdev=52.04 | lat (usec): min=16, max=1615, avg=257.27, stdev=178.32 | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 184], 20. | | 1.00th=[ 184], 5.00th=[ 186], 10.00th=[ 188], 20. | 30.00th=[ 188], 40.00th=[ 215], 50.00th=[ 237], 60. | | 30.00th=[ 194], 40.00th=[ 202], 50.00th=[ 237], 60. | 70.00th=[ 260], 80.00th=[ 265], 90.00th=[ 310], 95. | | 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 293], 95. | 99.00th=[ 359], 99.50th=[ 404], 99.90th=[ 412], 99. | | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1467], 99. | 99.99th=[ 1565] | | 99.99th=[ 1500] Type=write,Size=512B,NumJobs=56,depth=1 Type=write,Size=512B,NumJobs=56,depth=1 write: IOPS=2325k, BW=1135MiB/s (1191MB/s)(11.1GiB/10001mse | write: IOPS=2369k, BW=1157MiB/s (1213MB/s)(11.3GiB/10002mse slat (nsec): min=1761, max=894949, avg=3151.04, stdev=221 | slat (nsec): min=1823, max=501953, avg=3567.43, stdev=201 clat (nsec): min=443, max=1452.0k, avg=20283.73, stdev=15 | clat (nsec): min=449, max=1722.5k, avg=19318.77, stdev=42 lat (usec): min=12, max=1454, avg=23.51, stdev=16.01 | lat (usec): min=11, max=1724, avg=22.98, stdev= 4.83 | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 17], 20. | | 1.00th=[12992], 5.00th=[14656], 10.00th=[15680], 20. | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17536], 40.00th=[18048], 50.00th=[18816], 60. | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95. | | 70.00th=[20352], 80.00th=[21632], 90.00th=[23424], 95. | 99.00th=[ 34], 99.50th=[ 40], 99.90th=[ 347], 99. | | 99.00th=[31872], 99.50th=[35584], 99.90th=[41728], 99. | 99.99th=[ 519] | | 99.99th=[54016] Type=write,Size=512B,NumJobs=56,depth=16 Type=write,Size=512B,NumJobs=56,depth=16 write: IOPS=7222k, BW=3526MiB/s (3698MB/s)(34.4GiB/10001mse | write: IOPS=6237k, BW=3046MiB/s (3193MB/s)(29.7GiB/10002mse slat (nsec): min=1630, max=325931, avg=4224.26, stdev=656 | slat (nsec): min=1640, max=2381.5k, avg=5462.05, stdev=67 clat (usec): min=15, max=5606, avg=119.11, stdev=48.84 | clat (usec): min=5, max=10435, avg=137.19, stdev=82.68 lat (usec): min=26, max=5610, avg=123.43, stdev=49.91 | lat (usec): min=15, max=10447, avg=142.77, stdev=85.58 | 1.00th=[ 68], 5.00th=[ 74], 10.00th=[ 78], 20. | | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 70], 20. | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 104], 60. | | 30.00th=[ 85], 40.00th=[ 96], 50.00th=[ 104], 60. | 70.00th=[ 126], 80.00th=[ 145], 90.00th=[ 176], 95. | | 70.00th=[ 149], 80.00th=[ 210], 90.00th=[ 265], 95. | 99.00th=[ 318], 99.50th=[ 351], 99.90th=[ 420], 99. | | 99.00th=[ 363], 99.50th=[ 396], 99.90th=[ 553], 99. | 99.99th=[ 603] | | 99.99th=[ 873] Type=write,Size=512B,NumJobs=56,depth=64 Type=write,Size=512B,NumJobs=56,depth=64 write: IOPS=8407k, BW=4105MiB/s (4304MB/s)(40.1GiB/10002mse | write: IOPS=7488k, BW=3656MiB/s (3834MB/s)(35.7GiB/10002mse slat (nsec): min=1642, max=22197k, avg=4988.96, stdev=171 | slat (nsec): min=1666, max=7929.3k, avg=5382.53, stdev=12 clat (usec): min=21, max=22624, avg=420.46, stdev=251.29 | clat (usec): min=14, max=12517, avg=472.19, stdev=331.30 lat (usec): min=25, max=22628, avg=425.55, stdev=253.99 | lat (usec): min=17, max=12520, avg=477.70, stdev=334.84 | 1.00th=[ 277], 5.00th=[ 293], 10.00th=[ 302], 20. | | 1.00th=[ 265], 5.00th=[ 277], 10.00th=[ 289], 20. | 30.00th=[ 322], 40.00th=[ 338], 50.00th=[ 359], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 314], 60. | 70.00th=[ 404], 80.00th=[ 441], 90.00th=[ 482], 95. | | 70.00th=[ 347], 80.00th=[ 562], 90.00th=[ 1057], 95. | 99.00th=[ 1729], 99.50th=[ 1860], 99.90th=[ 2024], 99. | | 99.00th=[ 1434], 99.50th=[ 1696], 99.90th=[ 2040], 99. | 99.99th=[ 2606] | | 99.99th=[ 2474] Type=write,Size=4K,NumJobs=1,depth=1 Type=write,Size=4K,NumJobs=1,depth=1 write: IOPS=44.4k, BW=173MiB/s (182MB/s)(1735MiB/10001msec) | write: IOPS=55.8k, BW=218MiB/s (229MB/s)(2180MiB/10000msec) slat (nsec): min=2009, max=36370, avg=3363.25, stdev=2555 | slat (nsec): min=2138, max=44613, avg=2552.61, stdev=1404 clat (nsec): min=453, max=106416, avg=18501.98, stdev=678 | clat (nsec): min=627, max=624863, avg=14910.38, stdev=517 lat (usec): min=14, max=109, avg=21.94, stdev= 7.54 | lat (usec): min=13, max=627, avg=17.50, stdev= 5.55 | 1.00th=[12864], 5.00th=[14016], 10.00th=[14144], 20. | | 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20. | 30.00th=[14272], 40.00th=[14400], 50.00th=[14656], 60. | | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60. | 70.00th=[18816], 80.00th=[21632], 90.00th=[30080], 95. | | 70.00th=[13248], 80.00th=[13632], 90.00th=[20352], 95. | 99.00th=[42240], 99.50th=[46336], 99.90th=[51456], 99. | | 99.00th=[35584], 99.50th=[37120], 99.90th=[44800], 99. | 99.99th=[64768] | | 99.99th=[53504] Type=write,Size=4K,NumJobs=1,depth=16 Type=write,Size=4K,NumJobs=1,depth=16 write: IOPS=245k, BW=956MiB/s (1002MB/s)(9556MiB/10001msec) | write: IOPS=325k, BW=1271MiB/s (1332MB/s)(12.4GiB/10001msec slat (nsec): min=1646, max=69111, avg=2966.78, stdev=2405 | slat (nsec): min=1697, max=32986, avg=2232.67, stdev=205. clat (usec): min=28, max=146, avg=61.95, stdev= 5.61 | clat (nsec): min=14620, max=89670, avg=46638.22, stdev=11 lat (usec): min=31, max=148, avg=64.97, stdev= 5.64 | lat (nsec): min=16814, max=91833, avg=48907.48, stdev=11 | 1.00th=[ 51], 5.00th=[ 56], 10.00th=[ 57], 20. | | 1.00th=[44800], 5.00th=[45312], 10.00th=[45824], 20. | 30.00th=[ 59], 40.00th=[ 60], 50.00th=[ 62], 60. | | 30.00th=[46336], 40.00th=[46336], 50.00th=[46336], 60. | 70.00th=[ 65], 80.00th=[ 66], 90.00th=[ 69], 95. | | 70.00th=[46848], 80.00th=[46848], 90.00th=[47360], 95. | 99.00th=[ 81], 99.50th=[ 83], 99.90th=[ 90], 99. | | 99.00th=[50944], 99.50th=[53504], 99.90th=[58112], 99. | 99.99th=[ 108] | | 99.99th=[63744] Type=write,Size=4K,NumJobs=1,depth=64 Type=write,Size=4K,NumJobs=1,depth=64 write: IOPS=350k, BW=1367MiB/s (1434MB/s)(13.4GiB/10001msec | write: IOPS=322k, BW=1259MiB/s (1320MB/s)(12.3GiB/10001msec slat (nsec): min=1669, max=39786, avg=2055.37, stdev=197. | slat (nsec): min=1697, max=33655, avg=2273.54, stdev=208. clat (usec): min=21, max=222, avg=180.47, stdev= 1.67 | clat (usec): min=18, max=238, avg=195.95, stdev= 2.07 lat (usec): min=23, max=224, avg=182.56, stdev= 1.69 | lat (usec): min=20, max=241, avg=198.26, stdev= 2.08 | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 180], 20. | | 1.00th=[ 192], 5.00th=[ 194], 10.00th=[ 194], 20. | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 182], 60. | | 30.00th=[ 196], 40.00th=[ 196], 50.00th=[ 196], 60. | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 184], 95. | | 70.00th=[ 198], 80.00th=[ 198], 90.00th=[ 198], 95. | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 190], 99. | | 99.00th=[ 202], 99.50th=[ 204], 99.90th=[ 208], 99. | 99.99th=[ 194] | | 99.99th=[ 212] Type=write,Size=4K,NumJobs=16,depth=1 Type=write,Size=4K,NumJobs=16,depth=1 write: IOPS=851k, BW=3325MiB/s (3486MB/s)(32.5GiB/10001msec | write: IOPS=855k, BW=3339MiB/s (3502MB/s)(32.6GiB/10001msec slat (nsec): min=1844, max=67596, avg=2580.48, stdev=494. | slat (nsec): min=1832, max=65363, avg=2579.38, stdev=826. clat (nsec): min=670, max=351867, avg=15732.23, stdev=195 | clat (nsec): min=472, max=134403, avg=15646.36, stdev=196 lat (usec): min=12, max=354, avg=18.37, stdev= 2.08 | lat (usec): min=12, max=136, avg=18.28, stdev= 2.21 | 1.00th=[12608], 5.00th=[14016], 10.00th=[14400], 20. | | 1.00th=[13120], 5.00th=[13760], 10.00th=[14016], 20. | 30.00th=[15168], 40.00th=[15296], 50.00th=[15552], 60. | | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60. | 70.00th=[16064], 80.00th=[16512], 90.00th=[17024], 95. | | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95. | 99.00th=[20096], 99.50th=[22144], 99.90th=[34048], 99. | | 99.00th=[23680], 99.50th=[25728], 99.90th=[33536], 99. | 99.99th=[89600] | | 99.99th=[37120] Type=write,Size=4K,NumJobs=16,depth=16 Type=write,Size=4K,NumJobs=16,depth=16 write: IOPS=1885k, BW=7364MiB/s (7722MB/s)(71.9GiB/10001mse | write: IOPS=1826k, BW=7133MiB/s (7480MB/s)(69.7GiB/10001mse slat (nsec): min=1629, max=135927, avg=2178.04, stdev=783 | slat (nsec): min=1639, max=113071, avg=3312.71, stdev=236 clat (usec): min=40, max=387, avg=133.25, stdev=31.90 | clat (usec): min=40, max=454, avg=136.35, stdev=35.22 lat (usec): min=44, max=391, avg=135.47, stdev=31.86 | lat (usec): min=43, max=456, avg=139.72, stdev=35.40 | 1.00th=[ 89], 5.00th=[ 99], 10.00th=[ 103], 20. | | 1.00th=[ 72], 5.00th=[ 91], 10.00th=[ 101], 20. | 30.00th=[ 116], 40.00th=[ 125], 50.00th=[ 131], 60. | | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60. | 70.00th=[ 139], 80.00th=[ 143], 90.00th=[ 176], 95. | | 70.00th=[ 143], 80.00th=[ 149], 90.00th=[ 182], 95. | 99.00th=[ 253], 99.50th=[ 269], 99.90th=[ 289], 99. | | 99.00th=[ 258], 99.50th=[ 273], 99.90th=[ 314], 99. | 99.99th=[ 310] | | 99.99th=[ 367] Type=write,Size=4K,NumJobs=16,depth=64 Type=write,Size=4K,NumJobs=16,depth=64 write: IOPS=481k, BW=1880MiB/s (1971MB/s)(23.6GiB/12877msec | write: IOPS=1875k, BW=7325MiB/s (7680MB/s)(71.5GiB/10001mse slat (nsec): min=1635, max=3356.0M, avg=3206.62, stdev=13 | slat (nsec): min=1638, max=627057, avg=3232.91, stdev=324 clat (usec): min=27, max=7636.9k, avg=1985.08, stdev=8553 | clat (usec): min=49, max=1357, avg=542.32, stdev=162.71 lat (usec): min=29, max=7636.9k, avg=1988.34, stdev=8555 | lat (usec): min=52, max=1384, avg=545.61, stdev=162.80 | 1.00th=[ 178], 5.00th=[ 180], 10.00th=[ 180 | | 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20. | 20.00th=[ 182], 30.00th=[ 192], 40.00th=[ 233 | | 30.00th=[ 453], 40.00th=[ 498], 50.00th=[ 537], 60. | 50.00th=[ 235], 60.00th=[ 239], 70.00th=[ 247 | | 70.00th=[ 635], 80.00th=[ 685], 90.00th=[ 758], 95. | 80.00th=[ 253], 90.00th=[ 262], 95.00th=[ 269 | | 99.00th=[ 922], 99.50th=[ 955], 99.90th=[ 1029], 99. | 99.00th=[ 330], 99.50th=[ 338], 99.90th=[ 506 | | 99.99th=[ 1123] | 99.95th=[ 832], 99.99th=[4328522] < Type=write,Size=4K,NumJobs=56,depth=1 Type=write,Size=4K,NumJobs=56,depth=1 write: IOPS=1456k, BW=5686MiB/s (5962MB/s)(55.5GiB/10002mse | write: IOPS=1416k, BW=5533MiB/s (5802MB/s)(54.0GiB/10002mse slat (nsec): min=1774, max=827924, avg=2804.02, stdev=137 | slat (nsec): min=1846, max=533231, avg=2881.71, stdev=111 clat (nsec): min=462, max=3982.0k, avg=35108.12, stdev=23 | clat (nsec): min=498, max=4453.2k, avg=36057.87, stdev=38 lat (usec): min=13, max=3987, avg=37.98, stdev=23.37 | lat (usec): min=15, max=4456, avg=39.01, stdev= 4.01 | 1.00th=[ 21], 5.00th=[ 27], 10.00th=[ 29], 20. | | 1.00th=[29312], 5.00th=[32128], 10.00th=[33024], 20. | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 35], 60. | | 30.00th=[34560], 40.00th=[35072], 50.00th=[35584], 60. | 70.00th=[ 36], 80.00th=[ 37], 90.00th=[ 38], 95. | | 70.00th=[37120], 80.00th=[37632], 90.00th=[38656], 95. | 99.00th=[ 52], 99.50th=[ 141], 99.90th=[ 437], 99. | | 99.00th=[49408], 99.50th=[52480], 99.90th=[56576], 99. | 99.99th=[ 603] | | 99.99th=[64768] Type=write,Size=4K,NumJobs=56,depth=16 Type=write,Size=4K,NumJobs=56,depth=16 write: IOPS=1662k, BW=6493MiB/s (6808MB/s)(75.8GiB/11961mse | write: IOPS=1616k, BW=6314MiB/s (6621MB/s)(61.7GiB/10001mse slat (nsec): min=1629, max=3178.7M, avg=3532.32, stdev=16 | slat (nsec): min=1649, max=738008, avg=3245.81, stdev=163 clat (usec): min=20, max=4303.6k, avg=500.29, stdev=35989 | clat (usec): min=44, max=5046, avg=550.36, stdev=143.07 lat (usec): min=27, max=4303.6k, avg=503.87, stdev=36029 | lat (usec): min=46, max=5049, avg=553.68, stdev=143.03 | 1.00th=[ 60], 5.00th=[ 68], 10.00th=[ 75 | | 1.00th=[ 215], 5.00th=[ 343], 10.00th=[ 404], 20. | 20.00th=[ 89], 30.00th=[ 97], 40.00th=[ 101 | | 30.00th=[ 490], 40.00th=[ 510], 50.00th=[ 529], 60. | 50.00th=[ 109], 60.00th=[ 125], 70.00th=[ 139 | | 70.00th=[ 586], 80.00th=[ 660], 90.00th=[ 758], 95. | 80.00th=[ 153], 90.00th=[ 182], 95.00th=[ 215 | | 99.00th=[ 938], 99.50th=[ 996], 99.90th=[ 1106], 99. | 99.00th=[ 297], 99.50th=[ 326], 99.90th=[ 392 | | 99.99th=[ 1221] | 99.95th=[ 429], 99.99th=[2298479] < Type=write,Size=4K,NumJobs=56,depth=64 Type=write,Size=4K,NumJobs=56,depth=64 write: IOPS=699k, BW=2730MiB/s (2862MB/s)(33.6GiB/12594msec | write: IOPS=1625k, BW=6348MiB/s (6656MB/s)(62.0GiB/10003mse slat (nsec): min=1628, max=4304.3M, avg=40740.65, stdev=1 | slat (nsec): min=1656, max=1933.3k, avg=21620.58, stdev=7 clat (usec): min=27, max=12592k, avg=4999.61, stdev=14711 | clat (usec): min=66, max=8663, avg=2182.95, stdev=879.90 lat (usec): min=43, max=12592k, avg=5040.40, stdev=14795 | lat (usec): min=69, max=8669, avg=2204.64, stdev=895.51 | 1.00th=[ 178], 5.00th=[ 180], 10.00th=[ 182 | | 1.00th=[ 693], 5.00th=[ 971], 10.00th=[ 1074], 20. | 20.00th=[ 184], 30.00th=[ 229], 40.00th=[ 245 | | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60. | 50.00th=[ 247], 60.00th=[ 251], 70.00th=[ 255 | | 70.00th=[ 2343], 80.00th=[ 2507], 90.00th=[ 3392], 95. | 80.00th=[ 260], 90.00th=[ 273], 95.00th=[ 334 | | 99.00th=[ 4883], 99.50th=[ 5211], 99.90th=[ 5800], 99. | 99.00th=[ 383], 99.50th=[ 388], 99.90th=[3003122 | | 99.99th=[ 6587] | 99.95th=[4328522], 99.99th=[5402264] < Type=write,Size=64K,NumJobs=1,depth=1 Type=write,Size=64K,NumJobs=1,depth=1 write: IOPS=23.3k, BW=1458MiB/s (1529MB/s)(14.2GiB/10001mse | write: IOPS=23.3k, BW=1454MiB/s (1525MB/s)(14.2GiB/10001mse slat (nsec): min=4814, max=41610, avg=6697.07, stdev=3203 | slat (nsec): min=4849, max=39702, avg=6880.02, stdev=3240 clat (usec): min=3, max=130, avg=35.61, stdev= 9.49 | clat (usec): min=17, max=133, avg=35.53, stdev= 9.39 lat (usec): min=33, max=136, avg=42.35, stdev=10.60 | lat (usec): min=33, max=146, avg=42.46, stdev=10.52 | 1.00th=[29312], 5.00th=[29312], 10.00th=[29312], 20. | | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20. | 30.00th=[29568], 40.00th=[29824], 50.00th=[30080], 60. | | 30.00th=[29568], 40.00th=[29824], 50.00th=[29824], 60. | 70.00th=[37120], 80.00th=[43776], 90.00th=[49920], 95. | | 70.00th=[36608], 80.00th=[43776], 90.00th=[49408], 95. | 99.00th=[69120], 99.50th=[72192], 99.90th=[86528], 99. | | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99. | 99.99th=[90624] | 99.99th=[90624] Type=write,Size=64K,NumJobs=1,depth=16 Type=write,Size=64K,NumJobs=1,depth=16 write: IOPS=140k, BW=8738MiB/s (9162MB/s)(85.3GiB/10001msec | write: IOPS=146k, BW=9109MiB/s (9552MB/s)(88.0GiB/10001msec slat (nsec): min=3279, max=79959, avg=5945.90, stdev=3397 | slat (nsec): min=3787, max=44724, avg=5539.44, stdev=622. clat (usec): min=63, max=272, avg=108.05, stdev=10.23 | clat (usec): min=76, max=228, avg=103.90, stdev=11.08 lat (usec): min=67, max=276, avg=114.06, stdev=10.46 | lat (usec): min=82, max=233, avg=109.48, stdev=11.10 | 1.00th=[ 91], 5.00th=[ 97], 10.00th=[ 99], 20. | | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20. | 30.00th=[ 102], 40.00th=[ 103], 50.00th=[ 105], 60. | | 30.00th=[ 98], 40.00th=[ 99], 50.00th=[ 101], 60. | 70.00th=[ 112], 80.00th=[ 116], 90.00th=[ 123], 95. | | 70.00th=[ 105], 80.00th=[ 109], 90.00th=[ 120], 95. | 99.00th=[ 139], 99.50th=[ 145], 99.90th=[ 169], 99. | | 99.00th=[ 141], 99.50th=[ 149], 99.90th=[ 182], 99. | 99.99th=[ 206] | | 99.99th=[ 200] Type=write,Size=64K,NumJobs=1,depth=64 Type=write,Size=64K,NumJobs=1,depth=64 write: IOPS=145k, BW=9085MiB/s (9526MB/s)(88.7GiB/10001msec | write: IOPS=139k, BW=8708MiB/s (9131MB/s)(85.0GiB/10001msec slat (usec): min=2, max=164, avg= 5.45, stdev= 3.82 | slat (nsec): min=2676, max=64177, avg=5388.79, stdev=1847 clat (usec): min=235, max=952, avg=434.38, stdev=42.96 | clat (usec): min=39, max=894, avg=453.54, stdev=56.76 lat (usec): min=239, max=957, avg=439.89, stdev=42.76 | lat (usec): min=45, max=898, avg=458.97, stdev=57.24 | 1.00th=[ 371], 5.00th=[ 383], 10.00th=[ 392], 20. | | 1.00th=[ 338], 5.00th=[ 355], 10.00th=[ 367], 20. | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60. | | 30.00th=[ 408], 40.00th=[ 457], 50.00th=[ 478], 60. | 70.00th=[ 445], 80.00th=[ 461], 90.00th=[ 494], 95. | | 70.00th=[ 494], 80.00th=[ 498], 90.00th=[ 506], 95. | 99.00th=[ 570], 99.50th=[ 586], 99.90th=[ 652], 99. | | 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 734], 99. | 99.99th=[ 881] | | 99.99th=[ 848] Type=write,Size=64K,NumJobs=16,depth=1 Type=write,Size=64K,NumJobs=16,depth=1 write: IOPS=170k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) slat (nsec): min=3811, max=82769, avg=5460.88, stdev=1987 | slat (nsec): min=4383, max=73755, avg=5607.60, stdev=1278 clat (usec): min=13, max=490, avg=87.97, stdev=11.39 | clat (usec): min=22, max=424, avg=85.24, stdev= 8.07 lat (usec): min=37, max=494, avg=93.49, stdev=11.57 | lat (usec): min=38, max=430, avg=90.90, stdev= 8.07 | 1.00th=[ 57], 5.00th=[ 72], 10.00th=[ 77], 20. | | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 79], 20. | 30.00th=[ 84], 40.00th=[ 86], 50.00th=[ 88], 60. | | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60. | 70.00th=[ 92], 80.00th=[ 96], 90.00th=[ 102], 95. | | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 98], 95. | 99.00th=[ 120], 99.50th=[ 126], 99.90th=[ 141], 99. | | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 124], 99. | 99.99th=[ 190] | | 99.99th=[ 137] Type=write,Size=64K,NumJobs=16,depth=16 Type=write,Size=64K,NumJobs=16,depth=16 write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec) | write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=3415, max=60314, avg=6662.15, stdev=1972 | slat (nsec): min=3611, max=70931, avg=6507.12, stdev=1389 clat (usec): min=94, max=3871, avg=1480.76, stdev=360.52 | clat (usec): min=109, max=3915, avg=1451.13, stdev=409.41 lat (usec): min=111, max=3880, avg=1487.48, stdev=360.80 | lat (usec): min=129, max=3920, avg=1457.68, stdev=409.50 | 1.00th=[ 1270], 5.00th=[ 1287], 10.00th=[ 1287], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 30.00th=[ 1369], 40.00th=[ 1369], 50.00th=[ 1385], 60. | | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1287], 60. | 70.00th=[ 1418], 80.00th=[ 1467], 90.00th=[ 1500], 95. | | 70.00th=[ 1352], 80.00th=[ 1369], 90.00th=[ 2540], 95. | 99.00th=[ 2802], 99.50th=[ 2802], 99.90th=[ 2835], 99. | | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99. | 99.99th=[ 2900] | | 99.99th=[ 2769] Type=write,Size=64K,NumJobs=16,depth=64 Type=write,Size=64K,NumJobs=16,depth=64 write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10005msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) slat (usec): min=2, max=5927, avg=15.49, stdev=123.06 | slat (nsec): min=3042, max=96073, avg=6866.24, stdev=2170 clat (usec): min=114, max=16669, avg=5934.82, stdev=1800. | clat (usec): min=124, max=10103, avg=5850.40, stdev=798.1 lat (usec): min=122, max=17554, avg=5950.36, stdev=1825. | lat (usec): min=131, max=10109, avg=5857.31, stdev=798.1 | 1.00th=[ 4293], 5.00th=[ 4424], 10.00th=[ 4555], 20. | | 1.00th=[ 4752], 5.00th=[ 4883], 10.00th=[ 4948], 20. | 30.00th=[ 4948], 40.00th=[ 5145], 50.00th=[ 5342], 60. | | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5669], 60. | 70.00th=[ 5932], 80.00th=[ 6390], 90.00th=[ 9110], 95. | | 70.00th=[ 6259], 80.00th=[ 6587], 90.00th=[ 7046], 95. | 99.00th=[11994], 99.50th=[12518], 99.90th=[13566], 99. | | 99.00th=[ 7767], 99.50th=[ 7898], 99.90th=[ 8455], 99. | 99.99th=[15401] | | 99.99th=[ 8848] Type=write,Size=64K,NumJobs=56,depth=1 Type=write,Size=64K,NumJobs=56,depth=1 write: IOPS=171k, BW=10.5GiB/s (11.2GB/s)(105GiB/10002msec) | write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec) slat (nsec): min=3796, max=64949, avg=5590.94, stdev=2033 | slat (nsec): min=4029, max=63780, avg=5771.34, stdev=1537 clat (usec): min=47, max=940, avg=320.48, stdev=68.32 | clat (usec): min=32, max=979, avg=314.28, stdev=57.25 lat (usec): min=53, max=947, avg=326.12, stdev=68.40 | lat (usec): min=40, max=985, avg=320.11, stdev=57.29 | 1.00th=[ 190], 5.00th=[ 202], 10.00th=[ 302], 20. | | 1.00th=[ 186], 5.00th=[ 293], 10.00th=[ 297], 20. | 30.00th=[ 306], 40.00th=[ 310], 50.00th=[ 314], 60. | | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60. | 70.00th=[ 322], 80.00th=[ 326], 90.00th=[ 334], 95. | | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95. | 99.00th=[ 635], 99.50th=[ 644], 99.90th=[ 660], 99. | | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99. | 99.99th=[ 783] | | 99.99th=[ 750] Type=write,Size=64K,NumJobs=56,depth=16 Type=write,Size=64K,NumJobs=56,depth=16 write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10005msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) slat (nsec): min=3155, max=79675, avg=7689.97, stdev=2581 | slat (nsec): min=3542, max=89310, avg=7585.26, stdev=2460 clat (usec): min=151, max=13705, avg=5207.44, stdev=985.7 | clat (usec): min=190, max=14367, avg=5116.17, stdev=1396. lat (usec): min=175, max=13712, avg=5215.19, stdev=985.8 | lat (usec): min=198, max=14375, avg=5123.82, stdev=1396. | 1.00th=[ 2573], 5.00th=[ 2606], 10.00th=[ 5014], 20. | | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2573], 20. | 30.00th=[ 5145], 40.00th=[ 5211], 50.00th=[ 5211], 60. | | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5145], 60. | 70.00th=[ 5211], 80.00th=[ 5276], 90.00th=[ 5866], 95. | | 70.00th=[ 5145], 80.00th=[ 5669], 90.00th=[ 6849], 95. | 99.00th=[ 8455], 99.50th=[ 8848], 99.90th=[10028], 99. | | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[11207], 99. | 99.99th=[11600] | | 99.99th=[12780] Type=write,Size=64K,NumJobs=56,depth=64 Type=write,Size=64K,NumJobs=56,depth=64 write: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10014msec) | write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(107GiB/10011msec) slat (usec): min=2, max=45005, avg=184.96, stdev=907.62 | slat (usec): min=3, max=32839, avg=176.71, stdev=925.53 clat (usec): min=195, max=128349, avg=20590.58, stdev=799 | clat (usec): min=189, max=132076, avg=20366.61, stdev=728 lat (usec): min=229, max=128358, avg=20775.62, stdev=814 | lat (usec): min=218, max=132086, avg=20543.39, stdev=740 | 1.00th=[ 8455], 5.00th=[ 9503], 10.00th=[10552], 20. | | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[10945], 20. | 30.00th=[19006], 40.00th=[20055], 50.00th=[20317], 60. | | 30.00th=[17957], 40.00th=[19530], 50.00th=[20317], 60. | 70.00th=[20841], 80.00th=[22152], 90.00th=[26608], 95. | | 70.00th=[21103], 80.00th=[23462], 90.00th=[28705], 95. | 99.00th=[52691], 99.50th=[58459], 99.90th=[70779], 99. | | 99.00th=[45876], 99.50th=[50594], 99.90th=[63177], 99. | 99.99th=[94897] | | 99.99th=[84411] Target with DIM Without DIM Type=read,Size=512B,NumJobs=1,depth=1 Type=read,Size=512B,NumJobs=1,depth=1 read: IOPS=57.6k, BW=28.1MiB/s (29.5MB/s)(281MiB/10001msec | read: IOPS=47.0k, BW=22.0MiB/s (24.1MB/s)(230MiB/10001msec slat (nsec): min=2050, max=44230, avg=2445.95, stdev=1144 | slat (nsec): min=1981, max=53196, avg=3048.29, stdev=2391 clat (nsec): min=838, max=98356, avg=14486.18, stdev=4396 | clat (nsec): min=442, max=107156, avg=17658.37, stdev=647 lat (usec): min=12, max=116, avg=16.97, stdev= 4.73 | lat (usec): min=13, max=109, avg=20.78, stdev= 7.18 | 1.00th=[12864], 5.00th=[12864], 10.00th=[12992], 20. | | 1.00th=[13632], 5.00th=[13760], 10.00th=[13888], 20. | 30.00th=[13120], 40.00th=[13120], 50.00th=[13120], 60. | | 30.00th=[14144], 40.00th=[14144], 50.00th=[14400], 60. | 70.00th=[13376], 80.00th=[13504], 90.00th=[15552], 95. | | 70.00th=[17024], 80.00th=[19072], 90.00th=[27776], 95. | 99.00th=[35072], 99.50th=[36096], 99.90th=[40704], 99. | | 99.00th=[40704], 99.50th=[45824], 99.90th=[51456], 99. | 99.99th=[51968] | | 99.99th=[58624] Type=read,Size=512B,NumJobs=1,depth=16 Type=read,Size=512B,NumJobs=1,depth=16 read: IOPS=348k, BW=170MiB/s (178MB/s)(1697MiB/10001msec) | read: IOPS=334k, BW=163MiB/s (171MB/s)(1629MiB/10001msec) slat (nsec): min=1594, max=45285, avg=1933.31, stdev=410. | slat (nsec): min=1564, max=28780, avg=2162.48, stdev=588. clat (usec): min=36, max=162, avg=43.80, stdev= 8.09 | clat (usec): min=17, max=118, avg=45.50, stdev= 6.30 lat (usec): min=37, max=164, avg=45.77, stdev= 8.11 | lat (usec): min=20, max=120, avg=47.70, stdev= 6.64 | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20. | | 1.00th=[41728], 5.00th=[42752], 10.00th=[42752], 20. | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 42], 60. | | 30.00th=[43264], 40.00th=[43776], 50.00th=[43776], 60. | 70.00th=[ 43], 80.00th=[ 44], 90.00th=[ 52], 95. | | 70.00th=[44288], 80.00th=[44288], 90.00th=[46336], 95. | 99.00th=[ 81], 99.50th=[ 90], 99.90th=[ 112], 99. | | 99.00th=[72192], 99.50th=[73216], 99.90th=[77312], 99. | 99.99th=[ 139] | | 99.99th=[86528] Type=read,Size=512B,NumJobs=1,depth=64 Type=read,Size=512B,NumJobs=1,depth=64 read: IOPS=373k, BW=182MiB/s (191MB/s)(1822MiB/10001msec) | read: IOPS=344k, BW=168MiB/s (176MB/s)(1682MiB/10001msec) slat (nsec): min=1581, max=43893, avg=1907.45, stdev=238. | slat (nsec): min=1607, max=30650, avg=2093.56, stdev=233. clat (usec): min=58, max=295, avg=169.31, stdev= 2.74 | clat (usec): min=19, max=238, avg=183.42, stdev= 2.22 lat (usec): min=60, max=297, avg=171.25, stdev= 2.76 | lat (usec): min=21, max=241, avg=185.55, stdev= 2.23 | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20. | | 1.00th=[ 180], 5.00th=[ 182], 10.00th=[ 182], 20. | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60. | | 30.00th=[ 182], 40.00th=[ 184], 50.00th=[ 184], 60. | 70.00th=[ 169], 80.00th=[ 172], 90.00th=[ 172], 95. | | 70.00th=[ 184], 80.00th=[ 186], 90.00th=[ 186], 95. | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 210], 99. | | 99.00th=[ 190], 99.50th=[ 192], 99.90th=[ 196], 99. | 99.99th=[ 231] | | 99.99th=[ 210] Type=read,Size=512B,NumJobs=16,depth=1 Type=read,Size=512B,NumJobs=16,depth=1 read: IOPS=924k, BW=451MiB/s (473MB/s)(4510MiB/10001msec) | read: IOPS=907k, BW=443MiB/s (465MB/s)(4431MiB/10001msec) slat (nsec): min=1865, max=66933, avg=2473.89, stdev=487. | slat (nsec): min=1925, max=69082, avg=2505.79, stdev=869. clat (nsec): min=598, max=426580, avg=14400.84, stdev=125 | clat (nsec): min=476, max=112521, avg=14672.56, stdev=182 lat (usec): min=11, max=428, avg=16.92, stdev= 1.37 | lat (usec): min=12, max=131, avg=17.23, stdev= 2.08 | 1.00th=[11840], 5.00th=[13248], 10.00th=[13504], 20. | | 1.00th=[11584], 5.00th=[13248], 10.00th=[13376], 20. | 30.00th=[13888], 40.00th=[14016], 50.00th=[14144], 60. | | 30.00th=[13888], 40.00th=[14016], 50.00th=[14272], 60. | 70.00th=[14528], 80.00th=[14912], 90.00th=[15552], 95. | | 70.00th=[14784], 80.00th=[15296], 90.00th=[16512], 95. | 99.00th=[18816], 99.50th=[20864], 99.90th=[24960], 99. | | 99.00th=[22400], 99.50th=[24448], 99.90th=[32384], 99. | 99.99th=[32128] | | 99.99th=[35584] Type=read,Size=512B,NumJobs=16,depth=16 Type=read,Size=512B,NumJobs=16,depth=16 read: IOPS=4616k, BW=2254MiB/s (2363MB/s)(22.0GiB/10001mse | read: IOPS=4055k, BW=1980MiB/s (2076MB/s)(19.3GiB/10003mse slat (nsec): min=1571, max=101196, avg=2275.70, stdev=782 | slat (nsec): min=1572, max=239521, avg=2836.25, stdev=282 clat (usec): min=24, max=492, avg=52.83, stdev= 9.68 | clat (usec): min=9, max=616, avg=59.85, stdev=55.40 lat (usec): min=26, max=523, avg=55.15, stdev= 9.91 | lat (usec): min=15, max=630, avg=62.74, stdev=57.98 | 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 43], 20. | | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 52], 60. | | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 46], 60. | 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 65], 95. | | 70.00th=[ 48], 80.00th=[ 52], 90.00th=[ 60], 95. | 99.00th=[ 79], 99.50th=[ 82], 99.90th=[ 88], 99. | | 99.00th=[ 326], 99.50th=[ 343], 99.90th=[ 367], 99. | 99.99th=[ 306] | | 99.99th=[ 519] Type=read,Size=512B,NumJobs=16,depth=64 Type=read,Size=512B,NumJobs=16,depth=64 read: IOPS=5208k, BW=2543MiB/s (2666MB/s)(24.8GiB/10002mse | read: IOPS=3451k, BW=1685MiB/s (1767MB/s)(16.5GiB/10001mse slat (nsec): min=1575, max=61538, avg=2219.34, stdev=1268 | slat (nsec): min=1569, max=204726, avg=3322.74, stdev=274 clat (usec): min=42, max=378, avg=194.04, stdev=28.59 | clat (usec): min=5, max=1543, avg=292.82, stdev=220.59 lat (usec): min=44, max=380, avg=196.30, stdev=28.94 | lat (usec): min=14, max=1561, avg=296.20, stdev=223.03 | 1.00th=[ 172], 5.00th=[ 174], 10.00th=[ 174], 20. | | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 180], 20. | 30.00th=[ 176], 40.00th=[ 178], 50.00th=[ 178], 60. | | 30.00th=[ 188], 40.00th=[ 227], 50.00th=[ 243], 60. | 70.00th=[ 208], 80.00th=[ 212], 90.00th=[ 233], 95. | | 70.00th=[ 249], 80.00th=[ 285], 90.00th=[ 685], 95. | 99.00th=[ 306], 99.50th=[ 318], 99.90th=[ 330], 99. | | 99.00th=[ 1270], 99.50th=[ 1319], 99.90th=[ 1385], 99. | 99.99th=[ 343] | | 99.99th=[ 1450] Type=read,Size=512B,NumJobs=56,depth=1 Type=read,Size=512B,NumJobs=56,depth=1 read: IOPS=2294k, BW=1120MiB/s (1175MB/s)(10.9GiB/10003mse | read: IOPS=2351k, BW=1148MiB/s (1204MB/s)(11.2GiB/10002mse slat (nsec): min=1826, max=842064, avg=3629.96, stdev=274 | slat (nsec): min=1744, max=502891, avg=3525.21, stdev=199 clat (nsec): min=468, max=3959.9k, avg=20048.74, stdev=10 | clat (nsec): min=456, max=781278, avg=19563.02, stdev=418 lat (usec): min=10, max=3962, avg=23.76, stdev=11.12 | lat (usec): min=11, max=785, avg=23.18, stdev= 4.79 | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20. | | 1.00th=[13120], 5.00th=[14912], 10.00th=[15936], 20. | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 20], 60. | | 30.00th=[17792], 40.00th=[18304], 50.00th=[19072], 60. | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 25], 95. | | 70.00th=[20608], 80.00th=[21632], 90.00th=[23680], 95. | 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 61], 99. | | 99.00th=[31872], 99.50th=[35072], 99.90th=[41728], 99. | 99.99th=[ 529] | | 99.99th=[56064] Type=read,Size=512B,NumJobs=56,depth=16 Type=read,Size=512B,NumJobs=56,depth=16 read: IOPS=8192k, BW=4000MiB/s (4194MB/s)(39.1GiB/10001mse | read: IOPS=7234k, BW=3532MiB/s (3704MB/s)(34.5GiB/10002mse slat (nsec): min=1639, max=7118.6k, avg=5019.37, stdev=74 | slat (nsec): min=1588, max=13073k, avg=5296.03, stdev=602 clat (usec): min=5, max=7261, avg=103.57, stdev=52.37 | clat (usec): min=9, max=13311, avg=117.66, stdev=77.29 lat (usec): min=34, max=7264, avg=108.69, stdev=54.81 | lat (usec): min=18, max=13322, avg=123.07, stdev=80.61 | 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 69], 20. | | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20. | 30.00th=[ 73], 40.00th=[ 74], 50.00th=[ 77], 60. | | 30.00th=[ 72], 40.00th=[ 76], 50.00th=[ 81], 60. | 70.00th=[ 94], 80.00th=[ 157], 90.00th=[ 182], 95. | | 70.00th=[ 108], 80.00th=[ 184], 90.00th=[ 247], 95. | 99.00th=[ 247], 99.50th=[ 306], 99.90th=[ 412], 99. | | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 494], 99. | 99.99th=[ 947] | | 99.99th=[ 594] Type=read,Size=512B,NumJobs=56,depth=64 Type=read,Size=512B,NumJobs=56,depth=64 read: IOPS=8312k, BW=4059MiB/s (4256MB/s)(39.6GiB/10002mse | read: IOPS=7769k, BW=3793MiB/s (3978MB/s)(37.1GiB/10002mse slat (nsec): min=1629, max=669215, avg=5032.55, stdev=769 | slat (nsec): min=1583, max=3785.4k, avg=5319.20, stdev=81 clat (usec): min=43, max=1951, avg=425.25, stdev=208.07 | clat (usec): min=2, max=5212, avg=455.02, stdev=334.71 lat (usec): min=63, max=1968, avg=430.39, stdev=210.56 | lat (usec): min=17, max=5214, avg=460.45, stdev=338.64 | 1.00th=[ 273], 5.00th=[ 281], 10.00th=[ 289], 20. | | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 281], 20. | 30.00th=[ 302], 40.00th=[ 302], 50.00th=[ 306], 60. | | 30.00th=[ 293], 40.00th=[ 297], 50.00th=[ 302], 60. | 70.00th=[ 322], 80.00th=[ 709], 90.00th=[ 750], 95. | | 70.00th=[ 310], 80.00th=[ 437], 90.00th=[ 1074], 95. | 99.00th=[ 971], 99.50th=[ 1270], 99.90th=[ 1500], 99. | | 99.00th=[ 1401], 99.50th=[ 1991], 99.90th=[ 2245], 99. | 99.99th=[ 1663] | | 99.99th=[ 2442] Type=read,Size=4K,NumJobs=1,depth=1 Type=read,Size=4K,NumJobs=1,depth=1 read: IOPS=52.3k, BW=204MiB/s (214MB/s)(2042MiB/10001msec) | read: IOPS=54.4k, BW=213MiB/s (223MB/s)(2126MiB/10001msec) slat (nsec): min=2073, max=41533, avg=2731.01, stdev=1940 | slat (nsec): min=2044, max=28781, avg=2556.03, stdev=1441 clat (nsec): min=647, max=121164, avg=15919.33, stdev=561 | clat (nsec): min=621, max=70566, avg=15359.54, stdev=5250 lat (usec): min=13, max=123, avg=18.70, stdev= 6.22 | lat (nsec): min=13302, max=73823, avg=17966.25, stdev=56 | 1.00th=[13376], 5.00th=[13504], 10.00th=[13504], 20. | | 1.00th=[13120], 5.00th=[13248], 10.00th=[13248], 20. | 30.00th=[13632], 40.00th=[13760], 50.00th=[13888], 60. | | 30.00th=[13376], 40.00th=[13376], 50.00th=[13376], 60. | 70.00th=[14144], 80.00th=[15040], 90.00th=[20096], 95. | | 70.00th=[13632], 80.00th=[14144], 90.00th=[19328], 95. | 99.00th=[36608], 99.50th=[40192], 99.90th=[50944], 99. | | 99.00th=[35584], 99.50th=[36608], 99.90th=[45312], 99. | 99.99th=[73216] | | 99.99th=[53504] Type=read,Size=4K,NumJobs=1,depth=16 Type=read,Size=4K,NumJobs=1,depth=16 read: IOPS=321k, BW=1255MiB/s (1316MB/s)(12.3GiB/10001msec | read: IOPS=334k, BW=1306MiB/s (1370MB/s)(12.8GiB/10001msec slat (nsec): min=1652, max=42407, avg=2045.49, stdev=500. | slat (nsec): min=1696, max=45538, avg=2168.76, stdev=208. clat (usec): min=34, max=181, avg=47.45, stdev=10.18 | clat (usec): min=18, max=148, avg=45.39, stdev= 1.47 lat (usec): min=36, max=183, avg=49.54, stdev=10.20 | lat (usec): min=20, max=192, avg=47.59, stdev= 1.48 | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20. | | 1.00th=[43776], 5.00th=[44288], 10.00th=[44288], 20. | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 44], 60. | | 30.00th=[44800], 40.00th=[45312], 50.00th=[45312], 60. | 70.00th=[ 45], 80.00th=[ 53], 90.00th=[ 60], 95. | | 70.00th=[45312], 80.00th=[45824], 90.00th=[45824], 95. | 99.00th=[ 92], 99.50th=[ 98], 99.90th=[ 122], 99. | | 99.00th=[49408], 99.50th=[54528], 99.90th=[64256], 99. | 99.99th=[ 147] | | 99.99th=[70144] Type=read,Size=4K,NumJobs=1,depth=64 Type=read,Size=4K,NumJobs=1,depth=64 read: IOPS=358k, BW=1398MiB/s (1466MB/s)(13.7GiB/10001msec | read: IOPS=338k, BW=1319MiB/s (1383MB/s)(12.9GiB/10001msec slat (nsec): min=1586, max=39977, avg=2013.84, stdev=228. | slat (nsec): min=1621, max=52632, avg=2145.29, stdev=209. clat (usec): min=33, max=277, avg=176.53, stdev= 3.73 | clat (usec): min=36, max=236, avg=187.02, stdev= 2.05 lat (usec): min=34, max=279, avg=178.58, stdev= 3.75 | lat (usec): min=38, max=238, avg=189.20, stdev= 2.06 | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20. | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60. | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 180], 95. | | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 190], 95. | 99.00th=[ 192], 99.50th=[ 204], 99.90th=[ 217], 99. | | 99.00th=[ 192], 99.50th=[ 196], 99.90th=[ 200], 99. | 99.99th=[ 260] | | 99.99th=[ 217] Type=read,Size=4K,NumJobs=16,depth=1 Type=read,Size=4K,NumJobs=16,depth=1 read: IOPS=876k, BW=3422MiB/s (3588MB/s)(33.4GiB/10001msec | read: IOPS=869k, BW=3396MiB/s (3561MB/s)(33.2GiB/10002msec slat (nsec): min=2018, max=54502, avg=2490.14, stdev=570. | slat (nsec): min=1890, max=70947, avg=2497.58, stdev=813. clat (nsec): min=586, max=146434, avg=15319.20, stdev=175 | clat (nsec): min=590, max=79209, avg=15456.14, stdev=1968 lat (usec): min=11, max=148, avg=17.86, stdev= 1.86 | lat (usec): min=12, max=110, avg=18.01, stdev= 2.18 | 1.00th=[12864], 5.00th=[13888], 10.00th=[14144], 20. | | 1.00th=[12352], 5.00th=[13888], 10.00th=[14144], 20. | 30.00th=[14656], 40.00th=[14784], 50.00th=[14912], 60. | | 30.00th=[14528], 40.00th=[14784], 50.00th=[15040], 60. | 70.00th=[15424], 80.00th=[15936], 90.00th=[16768], 95. | | 70.00th=[15552], 80.00th=[16192], 90.00th=[17280], 95. | 99.00th=[22400], 99.50th=[24192], 99.90th=[32640], 99. | | 99.00th=[23936], 99.50th=[25728], 99.90th=[33536], 99. | 99.99th=[72192] | | 99.99th=[36096] Type=read,Size=4K,NumJobs=16,depth=16 Type=read,Size=4K,NumJobs=16,depth=16 read: IOPS=2665k, BW=10.2GiB/s (10.9GB/s)(102GiB/10001msec | read: IOPS=1843k, BW=7201MiB/s (7551MB/s)(70.3GiB/10002mse slat (nsec): min=1576, max=160409, avg=2277.16, stdev=105 | slat (nsec): min=1582, max=104435, avg=3170.92, stdev=170 clat (usec): min=21, max=895, avg=93.43, stdev=13.26 | clat (usec): min=40, max=554, avg=135.18, stdev=32.57 lat (usec): min=25, max=897, avg=95.75, stdev=13.22 | lat (usec): min=42, max=557, avg=138.42, stdev=32.59 | 1.00th=[ 74], 5.00th=[ 78], 10.00th=[ 80], 20. | | 1.00th=[ 68], 5.00th=[ 92], 10.00th=[ 104], 20. | 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 96], 60. | | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 135], 60. | 70.00th=[ 99], 80.00th=[ 100], 90.00th=[ 102], 95. | | 70.00th=[ 141], 80.00th=[ 147], 90.00th=[ 161], 95. | 99.00th=[ 167], 99.50th=[ 174], 99.90th=[ 186], 99. | | 99.00th=[ 262], 99.50th=[ 306], 99.90th=[ 392], 99. | 99.99th=[ 258] | | 99.99th=[ 461] Type=read,Size=4K,NumJobs=16,depth=64 Type=read,Size=4K,NumJobs=16,depth=64 read: IOPS=2670k, BW=10.2GiB/s (10.9GB/s)(102GiB/10002msec | read: IOPS=1917k, BW=7487MiB/s (7850MB/s)(73.1GiB/10001mse slat (nsec): min=1559, max=332598, avg=2408.47, stdev=359 | slat (nsec): min=1566, max=653412, avg=3347.14, stdev=385 clat (usec): min=25, max=1243, avg=380.63, stdev=95.54 | clat (usec): min=43, max=1445, avg=530.39, stdev=127.84 lat (usec): min=30, max=1245, avg=383.08, stdev=96.26 | lat (usec): min=45, max=1619, avg=533.81, stdev=128.49 | 1.00th=[ 182], 5.00th=[ 247], 10.00th=[ 302], 20. | | 1.00th=[ 188], 5.00th=[ 351], 10.00th=[ 420], 20. | 30.00th=[ 347], 40.00th=[ 359], 50.00th=[ 367], 60. | | 30.00th=[ 486], 40.00th=[ 502], 50.00th=[ 519], 60. | 70.00th=[ 383], 80.00th=[ 408], 90.00th=[ 465], 95. | | 70.00th=[ 553], 80.00th=[ 578], 90.00th=[ 644], 95. | 99.00th=[ 742], 99.50th=[ 766], 99.90th=[ 816], 99. | | 99.00th=[ 1012], 99.50th=[ 1037], 99.90th=[ 1090], 99. | 99.99th=[ 873] | | 99.99th=[ 1205] Type=read,Size=4K,NumJobs=56,depth=1 Type=read,Size=4K,NumJobs=56,depth=1 read: IOPS=1594k, BW=6227MiB/s (6530MB/s)(60.8GiB/10002mse | read: IOPS=1571k, BW=6138MiB/s (6436MB/s)(59.9GiB/10001mse slat (nsec): min=1747, max=711701, avg=2984.71, stdev=222 | slat (nsec): min=1693, max=505053, avg=2771.65, stdev=110 clat (nsec): min=468, max=4376.0k, avg=31538.77, stdev=12 | clat (nsec): min=479, max=1348.1k, avg=32311.24, stdev=36 lat (usec): min=13, max=4378, avg=34.59, stdev=13.16 | lat (usec): min=15, max=1350, avg=35.15, stdev= 3.81 | 1.00th=[ 22], 5.00th=[ 26], 10.00th=[ 28], 20. | | 1.00th=[26752], 5.00th=[28800], 10.00th=[29568], 20. | 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60. | | 30.00th=[30848], 40.00th=[31360], 50.00th=[31872], 60. | 70.00th=[ 33], 80.00th=[ 34], 90.00th=[ 36], 95. | | 70.00th=[33024], 80.00th=[33536], 90.00th=[35584], 95. | 99.00th=[ 45], 99.50th=[ 50], 99.90th=[ 243], 99. | | 99.00th=[42240], 99.50th=[43776], 99.90th=[49920], 99. | 99.99th=[ 519] | | 99.99th=[59136] Type=read,Size=4K,NumJobs=56,depth=16 Type=read,Size=4K,NumJobs=56,depth=16 read: IOPS=2652k, BW=10.1GiB/s (10.9GB/s)(101GiB/10002msec | read: IOPS=2009k, BW=7846MiB/s (8227MB/s)(76.6GiB/10001mse slat (nsec): min=1571, max=937333, avg=3004.91, stdev=194 | slat (nsec): min=1566, max=379585, avg=3204.59, stdev=230 clat (usec): min=26, max=1529, avg=334.21, stdev=57.70 | clat (usec): min=44, max=1761, avg=442.22, stdev=244.27 lat (usec): min=28, max=1531, avg=337.28, stdev=57.71 | lat (usec): min=46, max=1764, avg=445.50, stdev=244.24 | 1.00th=[ 147], 5.00th=[ 210], 10.00th=[ 297], 20. | | 1.00th=[ 161], 5.00th=[ 194], 10.00th=[ 215], 20. | 30.00th=[ 322], 40.00th=[ 330], 50.00th=[ 334], 60. | | 30.00th=[ 273], 40.00th=[ 306], 50.00th=[ 351], 60. | 70.00th=[ 351], 80.00th=[ 359], 90.00th=[ 388], 95. | | 70.00th=[ 506], 80.00th=[ 644], 90.00th=[ 832], 95. | 99.00th=[ 490], 99.50th=[ 529], 99.90th=[ 709], 99. | | 99.00th=[ 1156], 99.50th=[ 1221], 99.90th=[ 1336], 99. | 99.99th=[ 1004] | | 99.99th=[ 1450] Type=read,Size=4K,NumJobs=56,depth=64 Type=read,Size=4K,NumJobs=56,depth=64 read: IOPS=2659k, BW=10.1GiB/s (10.9GB/s)(101GiB/10003msec | read: IOPS=1801k, BW=7036MiB/s (7378MB/s)(68.7GiB/10002mse slat (nsec): min=1576, max=1678.0k, avg=14535.12, stdev=5 | slat (nsec): min=1586, max=2410.2k, avg=20136.27, stdev=8 clat (usec): min=29, max=7242, avg=1332.35, stdev=730.40 | clat (usec): min=58, max=8215, avg=1968.75, stdev=947.22 lat (usec): min=32, max=7244, avg=1346.94, stdev=743.63 | lat (usec): min=60, max=8632, avg=1988.96, stdev=962.84 | 1.00th=[ 515], 5.00th=[ 619], 10.00th=[ 644], 20. | | 1.00th=[ 449], 5.00th=[ 734], 10.00th=[ 865], 20. | 30.00th=[ 693], 40.00th=[ 807], 50.00th=[ 1303], 60. | | 30.00th=[ 1418], 40.00th=[ 1729], 50.00th=[ 1909], 60. | 70.00th=[ 1418], 80.00th=[ 1975], 90.00th=[ 2442], 95. | | 70.00th=[ 2212], 80.00th=[ 2507], 90.00th=[ 3326], 95. | 99.00th=[ 3392], 99.50th=[ 3621], 99.90th=[ 4047], 99. | | 99.00th=[ 4817], 99.50th=[ 5080], 99.90th=[ 5735], 99. | 99.99th=[ 4752] | | 99.99th=[ 6587] Type=read,Size=64K,NumJobs=1,depth=1 Type=read,Size=64K,NumJobs=1,depth=1 read: IOPS=29.7k, BW=1854MiB/s (1944MB/s)(18.1GiB/10001mse | read: IOPS=29.8k, BW=1864MiB/s (1955MB/s)(18.2GiB/10001mse slat (nsec): min=4662, max=82948, avg=5935.55, stdev=2735 | slat (nsec): min=4768, max=95484, avg=6119.74, stdev=2779 clat (usec): min=3, max=136, avg=27.27, stdev= 7.95 | clat (nsec): min=1555, max=75556, avg=26881.73, stdev=781 lat (usec): min=25, max=141, avg=33.25, stdev= 9.18 | lat (usec): min=25, max=102, avg=33.05, stdev= 9.07 | 1.00th=[21888], 5.00th=[22144], 10.00th=[22144], 20. | | 1.00th=[21632], 5.00th=[21632], 10.00th=[21632], 20. | 30.00th=[22400], 40.00th=[22400], 50.00th=[22656], 60. | | 30.00th=[21888], 40.00th=[22144], 50.00th=[22144], 60. | 70.00th=[28288], 80.00th=[34560], 90.00th=[41216], 95. | | 70.00th=[28032], 80.00th=[34048], 90.00th=[40704], 95. | 99.00th=[48384], 99.50th=[50944], 99.90th=[66048], 99. | | 99.00th=[47360], 99.50th=[49920], 99.90th=[59648], 99. | 99.99th=[78336] | | 99.99th=[70144] Type=read,Size=64K,NumJobs=1,depth=16 Type=read,Size=64K,NumJobs=1,depth=16 read: IOPS=168k, BW=10.3GiB/s (11.0GB/s)(103GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10001msec) slat (nsec): min=3463, max=75651, avg=4588.13, stdev=499. | slat (nsec): min=3756, max=65673, avg=4930.28, stdev=334. clat (usec): min=67, max=271, avg=90.06, stdev= 7.31 | clat (usec): min=83, max=243, avg=87.86, stdev= 2.04 lat (usec): min=72, max=348, avg=94.68, stdev= 7.26 | lat (usec): min=87, max=310, avg=92.83, stdev= 2.05 | 1.00th=[ 81], 5.00th=[ 84], 10.00th=[ 85], 20. | | 1.00th=[ 86], 5.00th=[ 87], 10.00th=[ 87], 20. | 30.00th=[ 87], 40.00th=[ 89], 50.00th=[ 90], 60. | | 30.00th=[ 87], 40.00th=[ 88], 50.00th=[ 88], 60. | 70.00th=[ 92], 80.00th=[ 93], 90.00th=[ 96], 95. | | 70.00th=[ 88], 80.00th=[ 89], 90.00th=[ 91], 95. | 99.00th=[ 127], 99.50th=[ 135], 99.90th=[ 172], 99. | | 99.00th=[ 96], 99.50th=[ 99], 99.90th=[ 108], 99. | 99.99th=[ 194] | | 99.99th=[ 114] Type=read,Size=64K,NumJobs=1,depth=64 Type=read,Size=64K,NumJobs=1,depth=64 read: IOPS=171k, BW=10.5GiB/s (11.2GB/s)(105GiB/10001msec) | read: IOPS=168k, BW=10.2GiB/s (10.0GB/s)(102GiB/10001msec) slat (usec): min=2, max=419, avg= 3.53, stdev= 1.24 | slat (usec): min=2, max=390, avg= 4.15, stdev= 1.56 clat (usec): min=63, max=973, avg=369.73, stdev=13.21 | clat (usec): min=37, max=947, avg=376.95, stdev=15.70 lat (usec): min=69, max=1053, avg=373.30, stdev=13.09 | lat (usec): min=42, max=988, avg=381.15, stdev=16.25 | 1.00th=[ 338], 5.00th=[ 359], 10.00th=[ 359], 20. | | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 367], 20. | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 30.00th=[ 367], 40.00th=[ 367], 50.00th=[ 371], 60. | 70.00th=[ 371], 80.00th=[ 375], 90.00th=[ 379], 95. | | 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 400], 95. | 99.00th=[ 404], 99.50th=[ 416], 99.90th=[ 461], 99. | | 99.00th=[ 412], 99.50th=[ 416], 99.90th=[ 429], 99. | 99.99th=[ 652] | | 99.99th=[ 668] Type=read,Size=64K,NumJobs=16,depth=1 Type=read,Size=64K,NumJobs=16,depth=1 read: IOPS=171k, BW=10.4GiB/s (11.2GB/s)(104GiB/10001msec) | read: IOPS=172k, BW=10.5GiB/s (11.3GB/s)(105GiB/10002msec) slat (nsec): min=3795, max=63452, avg=5333.62, stdev=2920 | slat (nsec): min=3480, max=74756, avg=5273.21, stdev=2961 clat (nsec): min=626, max=175700, avg=87848.01, stdev=183 | clat (nsec): min=616, max=453786, avg=87271.90, stdev=161 lat (usec): min=26, max=180, avg=93.23, stdev=18.52 | lat (usec): min=26, max=458, avg=92.60, stdev=16.20 | 1.00th=[ 31], 5.00th=[ 48], 10.00th=[ 62], 20. | | 1.00th=[ 37], 5.00th=[ 57], 10.00th=[ 68], 20. | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 93], 60. | | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60. | 70.00th=[ 97], 80.00th=[ 100], 90.00th=[ 104], 95. | | 70.00th=[ 95], 80.00th=[ 98], 90.00th=[ 104], 95. | 99.00th=[ 125], 99.50th=[ 130], 99.90th=[ 139], 99. | | 99.00th=[ 126], 99.50th=[ 130], 99.90th=[ 143], 99. | 99.99th=[ 155] | | 99.99th=[ 161] Type=read,Size=64K,NumJobs=16,depth=16 Type=read,Size=64K,NumJobs=16,depth=16 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=3878, max=65832, avg=5321.95, stdev=1310 | slat (nsec): min=3866, max=57337, avg=5204.47, stdev=1220 clat (usec): min=104, max=3983, avg=1458.24, stdev=313.83 | clat (usec): min=116, max=3985, avg=1453.77, stdev=349.92 lat (usec): min=111, max=3988, avg=1463.62, stdev=313.72 | lat (usec): min=128, max=3990, avg=1459.03, stdev=349.93 | 1.00th=[ 1352], 5.00th=[ 1352], 10.00th=[ 1352], 20. | | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 30.00th=[ 1352], 40.00th=[ 1369], 50.00th=[ 1369], 60. | | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60. | 70.00th=[ 1385], 80.00th=[ 1450], 90.00th=[ 1467], 95. | | 70.00th=[ 1369], 80.00th=[ 1401], 90.00th=[ 1467], 95. | 99.00th=[ 2769], 99.50th=[ 2769], 99.90th=[ 2802], 99. | | 99.00th=[ 2737], 99.50th=[ 2769], 99.90th=[ 2769], 99. | 99.99th=[ 2802] | 99.99th=[ 2802] Type=read,Size=64K,NumJobs=16,depth=64 Type=read,Size=64K,NumJobs=16,depth=64 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) slat (usec): min=2, max=1287, avg=13.57, stdev=76.18 | slat (usec): min=2, max=1185, avg= 9.71, stdev=60.22 clat (usec): min=116, max=15982, avg=5828.20, stdev=1647. | clat (usec): min=116, max=16490, avg=5824.81, stdev=1345. lat (usec): min=127, max=15986, avg=5841.81, stdev=1674. | lat (usec): min=133, max=16494, avg=5834.58, stdev=1367. | 1.00th=[ 5080], 5.00th=[ 5080], 10.00th=[ 5080], 20. | | 1.00th=[ 5407], 5.00th=[ 5407], 10.00th=[ 5473], 20. | 30.00th=[ 5080], 40.00th=[ 5145], 50.00th=[ 5145], 60. | | 30.00th=[ 5473], 40.00th=[ 5473], 50.00th=[ 5473], 60. | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[10028], 95. | | 70.00th=[ 5473], 80.00th=[ 5473], 90.00th=[ 5538], 95. | 99.00th=[10814], 99.50th=[10945], 99.90th=[10945], 99. | | 99.00th=[10945], 99.50th=[10945], 99.90th=[11076], 99. | 99.99th=[11469] | | 99.99th=[11338] Type=read,Size=64K,NumJobs=56,depth=1 Type=read,Size=64K,NumJobs=56,depth=1 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (usec): min=3, max=155, avg= 4.93, stdev= 1.32 | slat (nsec): min=3801, max=63985, avg=4896.45, stdev=1489 clat (usec): min=39, max=624, avg=314.57, stdev=91.62 | clat (usec): min=29, max=550, avg=314.45, stdev=76.21 lat (usec): min=48, max=629, avg=319.55, stdev=91.63 | lat (usec): min=35, max=555, avg=319.41, stdev=76.15 | 1.00th=[ 172], 5.00th=[ 174], 10.00th=[ 176], 20. | | 1.00th=[ 174], 5.00th=[ 178], 10.00th=[ 194], 20. | 30.00th=[ 302], 40.00th=[ 302], 50.00th=[ 306], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 306], 60. | 70.00th=[ 310], 80.00th=[ 449], 90.00th=[ 457], 95. | | 70.00th=[ 310], 80.00th=[ 322], 90.00th=[ 457], 95. | 99.00th=[ 478], 99.50th=[ 482], 99.90th=[ 494], 99. | | 99.00th=[ 474], 99.50th=[ 482], 99.90th=[ 490], 99. | 99.99th=[ 594] | | 99.99th=[ 506] Type=read,Size=64K,NumJobs=56,depth=16 Type=read,Size=64K,NumJobs=56,depth=16 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10006msec) slat (usec): min=3, max=156, avg= 5.40, stdev= 2.21 | slat (usec): min=3, max=251, avg= 5.40, stdev= 2.34 clat (usec): min=186, max=13998, avg=5112.23, stdev=1678. | clat (usec): min=236, max=14486, avg=5106.35, stdev=1241. lat (usec): min=201, max=14003, avg=5117.68, stdev=1678. | lat (usec): min=265, max=14491, avg=5111.80, stdev=1241. | 1.00th=[ 2442], 5.00th=[ 2442], 10.00th=[ 2474], 20. | | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 4883], 20. | 30.00th=[ 4883], 40.00th=[ 4948], 50.00th=[ 4948], 60. | | 30.00th=[ 4948], 40.00th=[ 5080], 50.00th=[ 5080], 60. | 70.00th=[ 5014], 80.00th=[ 7373], 90.00th=[ 7439], 95. | | 70.00th=[ 5080], 80.00th=[ 5145], 90.00th=[ 7373], 95. | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[10290], 99. | | 99.00th=[ 9896], 99.50th=[ 9896], 99.90th=[ 9896], 99. | 99.99th=[10290] | | 99.99th=[10028] Type=read,Size=64K,NumJobs=56,depth=64 Type=read,Size=64K,NumJobs=56,depth=64 read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10016msec) | read: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10018msec) slat (usec): min=2, max=9793, avg=214.10, stdev=778.24 | slat (usec): min=2, max=10184, avg=189.22, stdev=637.78 clat (usec): min=174, max=61025, avg=20233.29, stdev=9310 | clat (usec): min=347, max=61952, avg=20252.15, stdev=7056 lat (usec): min=178, max=63984, avg=20447.45, stdev=9493 | lat (usec): min=351, max=65141, avg=20441.44, stdev=7203 | 1.00th=[ 9110], 5.00th=[ 9110], 10.00th=[ 9503], 20. | | 1.00th=[ 9503], 5.00th=[ 9765], 10.00th=[ 9896], 20. | 30.00th=[17957], 40.00th=[18744], 50.00th=[18744], 60. | | 30.00th=[19530], 40.00th=[19530], 50.00th=[19792], 60. | 70.00th=[23987], 80.00th=[28443], 90.00th=[36439], 95. | | 70.00th=[20317], 80.00th=[20579], 90.00th=[29492], 95. | 99.00th=[41157], 99.50th=[56886], 99.90th=[56886], 99. | | 99.00th=[41157], 99.50th=[58983], 99.90th=[59507], 99. | 99.99th=[58459] | | 99.99th=[61080] Type=write,Size=512B,NumJobs=1,depth=1 Type=write,Size=512B,NumJobs=1,depth=1 write: IOPS=57.8k, BW=28.2MiB/s (29.6MB/s)(282MiB/10001msec | write: IOPS=56.0k, BW=27.4MiB/s (28.7MB/s)(274MiB/10001msec slat (nsec): min=2145, max=77559, avg=2556.80, stdev=1310 | slat (nsec): min=2112, max=43957, avg=2561.65, stdev=1423 clat (nsec): min=626, max=100854, avg=14269.28, stdev=471 | clat (nsec): min=633, max=60031, avg=14812.68, stdev=5638 lat (usec): min=12, max=107, avg=16.87, stdev= 5.13 | lat (nsec): min=12369, max=72124, avg=17416.39, stdev=60 | 1.00th=[12480], 5.00th=[12480], 10.00th=[12480], 20. | | 1.00th=[12352], 5.00th=[12352], 10.00th=[12480], 20. | 30.00th=[12608], 40.00th=[12608], 50.00th=[12736], 60. | | 30.00th=[12480], 40.00th=[12608], 50.00th=[12608], 60. | 70.00th=[12864], 80.00th=[13248], 90.00th=[16512], 95. | | 70.00th=[12864], 80.00th=[14656], 90.00th=[20096], 95. | 99.00th=[34560], 99.50th=[35584], 99.90th=[43264], 99. | | 99.00th=[35072], 99.50th=[36608], 99.90th=[47360], 99. | 99.99th=[53504] | | 99.99th=[54016] Type=write,Size=512B,NumJobs=1,depth=16 Type=write,Size=512B,NumJobs=1,depth=16 write: IOPS=302k, BW=147MiB/s (154MB/s)(1472MiB/10001msec) | write: IOPS=328k, BW=160MiB/s (168MB/s)(1601MiB/10001msec) slat (nsec): min=1637, max=35628, avg=1993.52, stdev=591. | slat (nsec): min=1641, max=32941, avg=2199.97, stdev=236. clat (usec): min=24, max=194, avg=50.75, stdev=12.31 | clat (usec): min=14, max=113, avg=46.29, stdev= 1.27 lat (usec): min=25, max=198, avg=52.78, stdev=12.32 | lat (usec): min=16, max=115, avg=48.52, stdev= 1.28 | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20. | | 1.00th=[43776], 5.00th=[44800], 10.00th=[45312], 20. | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 47], 60. | | 30.00th=[45824], 40.00th=[45824], 50.00th=[46336], 60. | 70.00th=[ 54], 80.00th=[ 59], 90.00th=[ 70], 95. | | 70.00th=[46336], 80.00th=[46848], 90.00th=[47360], 95. | 99.00th=[ 93], 99.50th=[ 102], 99.90th=[ 127], 99. | | 99.00th=[50432], 99.50th=[53504], 99.90th=[57600], 99. | 99.99th=[ 145] | | 99.99th=[63744] Type=write,Size=512B,NumJobs=1,depth=64 Type=write,Size=512B,NumJobs=1,depth=64 write: IOPS=362k, BW=177MiB/s (185MB/s)(1769MiB/10001msec) | write: IOPS=337k, BW=164MiB/s (172MB/s)(1644MiB/10001msec) slat (nsec): min=1644, max=36941, avg=1976.26, stdev=244. | slat (nsec): min=1642, max=43437, avg=2146.30, stdev=234. clat (usec): min=50, max=219, avg=174.38, stdev= 3.50 | clat (usec): min=16, max=235, avg=187.62, stdev= 2.46 lat (usec): min=52, max=221, avg=176.39, stdev= 3.54 | lat (usec): min=18, max=237, avg=189.80, stdev= 2.48 | 1.00th=[ 172], 5.00th=[ 172], 10.00th=[ 172], 20. | | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 186], 20. | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60. | | 30.00th=[ 186], 40.00th=[ 188], 50.00th=[ 188], 60. | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 178], 95. | | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 192], 95. | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 194], 99. | | 99.00th=[ 194], 99.50th=[ 196], 99.90th=[ 200], 99. | 99.99th=[ 198] | | 99.99th=[ 204] Type=write,Size=512B,NumJobs=16,depth=1 Type=write,Size=512B,NumJobs=16,depth=1 write: IOPS=928k, BW=453MiB/s (475MB/s)(4531MiB/10001msec) | write: IOPS=914k, BW=446MiB/s (468MB/s)(4462MiB/10001msec) slat (nsec): min=2034, max=70704, avg=2534.91, stdev=709. | slat (nsec): min=1818, max=83376, avg=2566.34, stdev=824. clat (nsec): min=588, max=855972, avg=14244.47, stdev=155 | clat (nsec): min=465, max=343253, avg=14459.07, stdev=181 lat (usec): min=12, max=858, avg=16.82, stdev= 1.78 | lat (usec): min=11, max=345, avg=17.08, stdev= 2.09 | 1.00th=[12480], 5.00th=[13120], 10.00th=[13248], 20. | | 1.00th=[12224], 5.00th=[12992], 10.00th=[13248], 20. | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60. | 30.00th=[13632], 40.00th=[13760], 50.00th=[14016], 60. | 70.00th=[14272], 80.00th=[14528], 90.00th=[15296], 95. | | 70.00th=[14528], 80.00th=[15040], 90.00th=[16320], 95. | 99.00th=[20864], 99.50th=[22912], 99.90th=[30848], 99. | | 99.00th=[21888], 99.50th=[23936], 99.90th=[32384], 99. | 99.99th=[34560] | | 99.99th=[35584] Type=write,Size=512B,NumJobs=16,depth=16 Type=write,Size=512B,NumJobs=16,depth=16 write: IOPS=4532k, BW=2213MiB/s (2320MB/s)(21.6GiB/10001mse | write: IOPS=3218k, BW=1571MiB/s (1648MB/s)(15.3GiB/10001mse slat (nsec): min=1608, max=191266, avg=2313.43, stdev=780 | slat (nsec): min=1621, max=154997, avg=3548.60, stdev=323 clat (usec): min=13, max=557, avg=53.80, stdev=10.94 | clat (usec): min=14, max=473, avg=75.44, stdev=65.86 lat (usec): min=15, max=574, avg=56.16, stdev=11.19 | lat (usec): min=17, max=489, avg=79.05, stdev=68.82 | 1.00th=[ 42], 5.00th=[ 43], 10.00th=[ 44], 20. | | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 46], 20. | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60. | | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 52], 60. | 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 65], 95. | | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 178], 95. | 99.00th=[ 85], 99.50th=[ 90], 99.90th=[ 109], 99. | | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99. | 99.99th=[ 396] | | 99.99th=[ 367] Type=write,Size=512B,NumJobs=16,depth=64 Type=write,Size=512B,NumJobs=16,depth=64 write: IOPS=5003k, BW=2443MiB/s (2562MB/s)(23.9GiB/10001mse | write: IOPS=3973k, BW=1940MiB/s (2034MB/s)(18.0GiB/10002mse slat (nsec): min=1613, max=68192, avg=2314.76, stdev=1502 | slat (nsec): min=1627, max=171980, avg=2926.70, stdev=209 clat (usec): min=32, max=588, avg=201.99, stdev=31.05 | clat (usec): min=10, max=1606, avg=254.29, stdev=176.36 lat (usec): min=35, max=590, avg=204.35, stdev=31.42 | lat (usec): min=16, max=1615, avg=257.27, stdev=178.32 | 1.00th=[ 176], 5.00th=[ 178], 10.00th=[ 178], 20. | | 1.00th=[ 184], 5.00th=[ 186], 10.00th=[ 188], 20. | 30.00th=[ 182], 40.00th=[ 182], 50.00th=[ 186], 60. | | 30.00th=[ 194], 40.00th=[ 202], 50.00th=[ 237], 60. | 70.00th=[ 212], 80.00th=[ 217], 90.00th=[ 241], 95. | | 70.00th=[ 255], 80.00th=[ 260], 90.00th=[ 293], 95. | 99.00th=[ 310], 99.50th=[ 318], 99.90th=[ 330], 99. | | 99.00th=[ 1385], 99.50th=[ 1418], 99.90th=[ 1467], 99. | 99.99th=[ 351] | | 99.99th=[ 1500] Type=write,Size=512B,NumJobs=56,depth=1 Type=write,Size=512B,NumJobs=56,depth=1 write: IOPS=2299k, BW=1123MiB/s (1177MB/s)(10.0GiB/10002mse | write: IOPS=2369k, BW=1157MiB/s (1213MB/s)(11.3GiB/10002mse slat (nsec): min=1875, max=1144.8k, avg=3714.43, stdev=29 | slat (nsec): min=1823, max=501953, avg=3567.43, stdev=201 clat (nsec): min=471, max=1780.1k, avg=19859.06, stdev=11 | clat (nsec): min=449, max=1722.5k, avg=19318.77, stdev=42 lat (usec): min=10, max=1783, avg=23.66, stdev=12.06 | lat (usec): min=11, max=1724, avg=22.98, stdev= 4.83 | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20. | | 1.00th=[12992], 5.00th=[14656], 10.00th=[15680], 20. | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 19], 60. | | 30.00th=[17536], 40.00th=[18048], 50.00th=[18816], 60. | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 25], 95. | | 70.00th=[20352], 80.00th=[21632], 90.00th=[23424], 95. | 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 70], 99. | | 99.00th=[31872], 99.50th=[35584], 99.90th=[41728], 99. | 99.99th=[ 553] | | 99.99th=[54016] Type=write,Size=512B,NumJobs=56,depth=16 Type=write,Size=512B,NumJobs=56,depth=16 write: IOPS=8134k, BW=3972MiB/s (4165MB/s)(38.8GiB/10001mse | write: IOPS=6237k, BW=3046MiB/s (3193MB/s)(29.7GiB/10002mse slat (nsec): min=1697, max=8922.2k, avg=5089.14, stdev=73 | slat (nsec): min=1640, max=2381.5k, avg=5462.05, stdev=67 clat (usec): min=6, max=9049, avg=104.25, stdev=53.53 | clat (usec): min=5, max=10435, avg=137.19, stdev=82.68 lat (usec): min=33, max=9053, avg=109.44, stdev=56.00 | lat (usec): min=15, max=10447, avg=142.77, stdev=85.58 | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 70], 20. | | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 70], 20. | 30.00th=[ 74], 40.00th=[ 75], 50.00th=[ 77], 60. | | 30.00th=[ 85], 40.00th=[ 96], 50.00th=[ 104], 60. | 70.00th=[ 92], 80.00th=[ 161], 90.00th=[ 186], 95. | | 70.00th=[ 149], 80.00th=[ 210], 90.00th=[ 265], 95. | 99.00th=[ 255], 99.50th=[ 326], 99.90th=[ 424], 99. | | 99.00th=[ 363], 99.50th=[ 396], 99.90th=[ 553], 99. | 99.99th=[ 775] | | 99.99th=[ 873] Type=write,Size=512B,NumJobs=56,depth=64 Type=write,Size=512B,NumJobs=56,depth=64 write: IOPS=8214k, BW=4011MiB/s (4205MB/s)(39.2GiB/10002mse | write: IOPS=7488k, BW=3656MiB/s (3834MB/s)(35.7GiB/10002mse slat (nsec): min=1699, max=946558, avg=5094.80, stdev=734 | slat (nsec): min=1666, max=7929.3k, avg=5382.53, stdev=12 clat (nsec): min=1952, max=1945.9k, avg=430363.82, stdev= | clat (usec): min=14, max=12517, avg=472.19, stdev=331.30 lat (usec): min=43, max=1949, avg=435.56, stdev=208.15 | lat (usec): min=17, max=12520, avg=477.70, stdev=334.84 | 1.00th=[ 281], 5.00th=[ 289], 10.00th=[ 297], 20. | | 1.00th=[ 265], 5.00th=[ 277], 10.00th=[ 289], 20. | 30.00th=[ 306], 40.00th=[ 310], 50.00th=[ 310], 60. | | 30.00th=[ 302], 40.00th=[ 306], 50.00th=[ 314], 60. | 70.00th=[ 330], 80.00th=[ 709], 90.00th=[ 750], 95. | | 70.00th=[ 347], 80.00th=[ 562], 90.00th=[ 1057], 95. | 99.00th=[ 971], 99.50th=[ 1254], 99.90th=[ 1450], 99. | | 99.00th=[ 1434], 99.50th=[ 1696], 99.90th=[ 2040], 99. | 99.99th=[ 1582] | | 99.99th=[ 2474] Type=write,Size=4K,NumJobs=1,depth=1 Type=write,Size=4K,NumJobs=1,depth=1 write: IOPS=53.8k, BW=210MiB/s (220MB/s)(2100MiB/10001msec) | write: IOPS=55.8k, BW=218MiB/s (229MB/s)(2180MiB/10000msec) slat (nsec): min=2141, max=79859, avg=2640.37, stdev=1677 | slat (nsec): min=2138, max=44613, avg=2552.61, stdev=1404 clat (nsec): min=616, max=110520, avg=15483.87, stdev=567 | clat (nsec): min=627, max=624863, avg=14910.38, stdev=517 lat (usec): min=13, max=116, avg=18.16, stdev= 6.15 | lat (usec): min=13, max=627, avg=17.50, stdev= 5.55 | 1.00th=[12864], 5.00th=[12992], 10.00th=[12992], 20. | | 1.00th=[12736], 5.00th=[12864], 10.00th=[12864], 20. | 30.00th=[13120], 40.00th=[13248], 50.00th=[13248], 60. | | 30.00th=[12992], 40.00th=[12992], 50.00th=[12992], 60. | 70.00th=[13504], 80.00th=[15552], 90.00th=[20864], 95. | | 70.00th=[13248], 80.00th=[13632], 90.00th=[20352], 95. | 99.00th=[36096], 99.50th=[39168], 99.90th=[50432], 99. | | 99.00th=[35584], 99.50th=[37120], 99.90th=[44800], 99. | 99.99th=[74240] | | 99.99th=[53504] Type=write,Size=4K,NumJobs=1,depth=16 Type=write,Size=4K,NumJobs=1,depth=16 write: IOPS=280k, BW=1093MiB/s (1146MB/s)(10.7GiB/10001msec | write: IOPS=325k, BW=1271MiB/s (1332MB/s)(12.4GiB/10001msec slat (nsec): min=1746, max=32381, avg=2137.66, stdev=633. | slat (nsec): min=1697, max=32986, avg=2232.67, stdev=205. clat (usec): min=41, max=167, avg=54.70, stdev=12.58 | clat (nsec): min=14620, max=89670, avg=46638.22, stdev=11 lat (usec): min=43, max=169, avg=56.87, stdev=12.60 | lat (nsec): min=16814, max=91833, avg=48907.48, stdev=11 | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 44], 20. | | 1.00th=[44800], 5.00th=[45312], 10.00th=[45824], 20. | 30.00th=[ 45], 40.00th=[ 49], 50.00th=[ 52], 60. | | 30.00th=[46336], 40.00th=[46336], 50.00th=[46336], 60. | 70.00th=[ 59], 80.00th=[ 64], 90.00th=[ 72], 95. | | 70.00th=[46848], 80.00th=[46848], 90.00th=[47360], 95. | 99.00th=[ 96], 99.50th=[ 111], 99.90th=[ 130], 99. | | 99.00th=[50944], 99.50th=[53504], 99.90th=[58112], 99. | 99.99th=[ 147] | | 99.99th=[63744] Type=write,Size=4K,NumJobs=1,depth=64 Type=write,Size=4K,NumJobs=1,depth=64 write: IOPS=340k, BW=1329MiB/s (1394MB/s)(12.0GiB/10001msec | write: IOPS=322k, BW=1259MiB/s (1320MB/s)(12.3GiB/10001msec slat (nsec): min=1728, max=34272, avg=2133.44, stdev=216. | slat (nsec): min=1697, max=33655, avg=2273.54, stdev=208. clat (usec): min=76, max=257, avg=185.64, stdev= 1.60 | clat (usec): min=18, max=238, avg=195.95, stdev= 2.07 lat (usec): min=78, max=259, avg=187.81, stdev= 1.61 | lat (usec): min=20, max=241, avg=198.26, stdev= 2.08 | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20. | | 1.00th=[ 192], 5.00th=[ 194], 10.00th=[ 194], 20. | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60. | | 30.00th=[ 196], 40.00th=[ 196], 50.00th=[ 196], 60. | 70.00th=[ 186], 80.00th=[ 188], 90.00th=[ 188], 95. | | 70.00th=[ 198], 80.00th=[ 198], 90.00th=[ 198], 95. | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 194], 99. | | 99.00th=[ 202], 99.50th=[ 204], 99.90th=[ 208], 99. | 99.99th=[ 200] | | 99.99th=[ 212] Type=write,Size=4K,NumJobs=16,depth=1 Type=write,Size=4K,NumJobs=16,depth=1 write: IOPS=863k, BW=3371MiB/s (3535MB/s)(32.9GiB/10001msec | write: IOPS=855k, BW=3339MiB/s (3502MB/s)(32.6GiB/10001msec slat (nsec): min=1980, max=66528, avg=2566.89, stdev=734. | slat (nsec): min=1832, max=65363, avg=2579.38, stdev=826. clat (nsec): min=583, max=653122, avg=15498.34, stdev=199 | clat (nsec): min=472, max=134403, avg=15646.36, stdev=196 lat (usec): min=12, max=655, avg=18.11, stdev= 2.17 | lat (usec): min=12, max=136, avg=18.28, stdev= 2.21 | 1.00th=[12992], 5.00th=[13888], 10.00th=[14144], 20. | | 1.00th=[13120], 5.00th=[13760], 10.00th=[14016], 20. | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60. | | 30.00th=[14656], 40.00th=[14912], 50.00th=[15168], 60. | 70.00th=[15680], 80.00th=[16192], 90.00th=[17280], 95. | | 70.00th=[15936], 80.00th=[16512], 90.00th=[17536], 95. | 99.00th=[23168], 99.50th=[25216], 99.90th=[33536], 99. | | 99.00th=[23680], 99.50th=[25728], 99.90th=[33536], 99. | 99.99th=[72192] | | 99.99th=[37120] Type=write,Size=4K,NumJobs=16,depth=16 Type=write,Size=4K,NumJobs=16,depth=16 write: IOPS=2631k, BW=10.0GiB/s (10.8GB/s)(100GiB/10002msec | write: IOPS=1826k, BW=7133MiB/s (7480MB/s)(69.7GiB/10001mse slat (nsec): min=1629, max=70383, avg=2374.75, stdev=939. | slat (nsec): min=1639, max=113071, avg=3312.71, stdev=236 clat (usec): min=28, max=370, avg=94.53, stdev=12.94 | clat (usec): min=40, max=454, avg=136.35, stdev=35.22 lat (usec): min=31, max=387, avg=96.95, stdev=12.90 | lat (usec): min=43, max=456, avg=139.72, stdev=35.40 | 1.00th=[ 72], 5.00th=[ 78], 10.00th=[ 80], 20. | | 1.00th=[ 72], 5.00th=[ 91], 10.00th=[ 101], 20. | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 97], 60. | | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60. | 70.00th=[ 100], 80.00th=[ 102], 90.00th=[ 105], 95. | | 70.00th=[ 143], 80.00th=[ 149], 90.00th=[ 182], 95. | 99.00th=[ 141], 99.50th=[ 172], 99.90th=[ 190], 99. | | 99.00th=[ 258], 99.50th=[ 273], 99.90th=[ 314], 99. | 99.99th=[ 231] | | 99.99th=[ 367] Type=write,Size=4K,NumJobs=16,depth=64 Type=write,Size=4K,NumJobs=16,depth=64 write: IOPS=2615k, BW=9.97GiB/s (10.7GB/s)(99.7GiB/10001mse | write: IOPS=1875k, BW=7325MiB/s (7680MB/s)(71.5GiB/10001mse slat (nsec): min=1633, max=196327, avg=2501.76, stdev=263 | slat (nsec): min=1638, max=627057, avg=3232.91, stdev=324 clat (usec): min=18, max=1044, avg=388.76, stdev=64.99 | clat (usec): min=49, max=1357, avg=542.32, stdev=162.71 lat (usec): min=21, max=1052, avg=391.30, stdev=65.69 | lat (usec): min=52, max=1384, avg=545.61, stdev=162.80 | 1.00th=[ 318], 5.00th=[ 338], 10.00th=[ 347], 20. | | 1.00th=[ 202], 5.00th=[ 277], 10.00th=[ 330], 20. | 30.00th=[ 367], 40.00th=[ 375], 50.00th=[ 383], 60. | | 30.00th=[ 453], 40.00th=[ 498], 50.00th=[ 537], 60. | 70.00th=[ 392], 80.00th=[ 396], 90.00th=[ 412], 95. | | 70.00th=[ 635], 80.00th=[ 685], 90.00th=[ 758], 95. | 99.00th=[ 734], 99.50th=[ 750], 99.90th=[ 775], 99. | | 99.00th=[ 922], 99.50th=[ 955], 99.90th=[ 1029], 99. | 99.99th=[ 799] | | 99.99th=[ 1123] Type=write,Size=4K,NumJobs=56,depth=1 Type=write,Size=4K,NumJobs=56,depth=1 write: IOPS=1419k, BW=5545MiB/s (5814MB/s)(54.2GiB/10002mse | write: IOPS=1416k, BW=5533MiB/s (5802MB/s)(54.0GiB/10002mse slat (nsec): min=1813, max=624655, avg=3014.79, stdev=214 | slat (nsec): min=1846, max=533231, avg=2881.71, stdev=111 clat (nsec): min=498, max=1174.4k, avg=35813.70, stdev=12 | clat (nsec): min=498, max=4453.2k, avg=36057.87, stdev=38 lat (usec): min=13, max=1177, avg=38.89, stdev=12.35 | lat (usec): min=15, max=4456, avg=39.01, stdev= 4.01 | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 32], 20. | | 1.00th=[29312], 5.00th=[32128], 10.00th=[33024], 20. | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 36], 60. | | 30.00th=[34560], 40.00th=[35072], 50.00th=[35584], 60. | 70.00th=[ 37], 80.00th=[ 38], 90.00th=[ 40], 95. | | 70.00th=[37120], 80.00th=[37632], 90.00th=[38656], 95. | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 229], 99. | | 99.00th=[49408], 99.50th=[52480], 99.90th=[56576], 99. | 99.99th=[ 498] | | 99.99th=[64768] Type=write,Size=4K,NumJobs=56,depth=16 Type=write,Size=4K,NumJobs=56,depth=16 write: IOPS=2489k, BW=9724MiB/s (10.2GB/s)(94.0GiB/10002mse | write: IOPS=1616k, BW=6314MiB/s (6621MB/s)(61.7GiB/10001mse slat (nsec): min=1637, max=742083, avg=3046.89, stdev=226 | slat (nsec): min=1649, max=738008, avg=3245.81, stdev=163 clat (usec): min=39, max=1654, avg=356.23, stdev=126.26 | clat (usec): min=44, max=5046, avg=550.36, stdev=143.07 lat (usec): min=44, max=1657, avg=359.34, stdev=126.24 | lat (usec): min=46, max=5049, avg=553.68, stdev=143.03 | 1.00th=[ 217], 5.00th=[ 237], 10.00th=[ 247], 20. | | 1.00th=[ 215], 5.00th=[ 343], 10.00th=[ 404], 20. | 30.00th=[ 277], 40.00th=[ 285], 50.00th=[ 297], 60. | | 30.00th=[ 490], 40.00th=[ 510], 50.00th=[ 529], 60. | 70.00th=[ 383], 80.00th=[ 474], 90.00th=[ 562], 95. | | 70.00th=[ 586], 80.00th=[ 660], 90.00th=[ 758], 95. | 99.00th=[ 709], 99.50th=[ 742], 99.90th=[ 865], 99. | | 99.00th=[ 938], 99.50th=[ 996], 99.90th=[ 1106], 99. | 99.99th=[ 1074] | | 99.99th=[ 1221] Type=write,Size=4K,NumJobs=56,depth=64 Type=write,Size=4K,NumJobs=56,depth=64 write: IOPS=2216k, BW=8657MiB/s (9078MB/s)(84.6GiB/10003mse | write: IOPS=1625k, BW=6348MiB/s (6656MB/s)(62.0GiB/10003mse slat (nsec): min=1658, max=1799.0k, avg=16827.79, stdev=6 | slat (nsec): min=1656, max=1933.3k, avg=21620.58, stdev=7 clat (usec): min=40, max=7314, avg=1599.52, stdev=743.15 | clat (usec): min=66, max=8663, avg=2182.95, stdev=879.90 lat (usec): min=43, max=7318, avg=1616.41, stdev=754.88 | lat (usec): min=69, max=8669, avg=2204.64, stdev=895.51 | 1.00th=[ 400], 5.00th=[ 644], 10.00th=[ 734], 20. | | 1.00th=[ 693], 5.00th=[ 971], 10.00th=[ 1074], 20. | 30.00th=[ 1254], 40.00th=[ 1418], 50.00th=[ 1516], 60. | | 30.00th=[ 1942], 40.00th=[ 2114], 50.00th=[ 2212], 60. | 70.00th=[ 1778], 80.00th=[ 2057], 90.00th=[ 2638], 95. | | 70.00th=[ 2343], 80.00th=[ 2507], 90.00th=[ 3392], 95. | 99.00th=[ 3884], 99.50th=[ 4178], 99.90th=[ 4817], 99. | | 99.00th=[ 4883], 99.50th=[ 5211], 99.90th=[ 5800], 99. | 99.99th=[ 5604] | | 99.99th=[ 6587] Type=write,Size=64K,NumJobs=1,depth=1 Type=write,Size=64K,NumJobs=1,depth=1 write: IOPS=23.1k, BW=1444MiB/s (1515MB/s)(14.1GiB/10001mse | write: IOPS=23.3k, BW=1454MiB/s (1525MB/s)(14.2GiB/10001mse slat (nsec): min=4822, max=98365, avg=6876.23, stdev=3203 | slat (nsec): min=4849, max=39702, avg=6880.02, stdev=3240 clat (usec): min=3, max=121, avg=35.83, stdev= 9.35 | clat (usec): min=17, max=133, avg=35.53, stdev= 9.39 lat (usec): min=33, max=133, avg=42.75, stdev=10.50 | lat (usec): min=33, max=146, avg=42.46, stdev=10.52 | 1.00th=[29568], 5.00th=[29568], 10.00th=[29568], 20. | | 1.00th=[29312], 5.00th=[29312], 10.00th=[29568], 20. | 30.00th=[29824], 40.00th=[29824], 50.00th=[30080], 60. | | 30.00th=[29568], 40.00th=[29824], 50.00th=[29824], 60. | 70.00th=[37120], 80.00th=[44800], 90.00th=[49408], 95. | | 70.00th=[36608], 80.00th=[43776], 90.00th=[49408], 95. | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99. | | 99.00th=[68096], 99.50th=[72192], 99.90th=[85504], 99. | 99.99th=[90624] | 99.99th=[90624] Type=write,Size=64K,NumJobs=1,depth=16 Type=write,Size=64K,NumJobs=1,depth=16 write: IOPS=140k, BW=8743MiB/s (9167MB/s)(85.4GiB/10001msec | write: IOPS=146k, BW=9109MiB/s (9552MB/s)(88.0GiB/10001msec slat (nsec): min=3861, max=47468, avg=5274.03, stdev=845. | slat (nsec): min=3787, max=44724, avg=5539.44, stdev=622. clat (usec): min=82, max=297, avg=108.76, stdev=13.98 | clat (usec): min=76, max=228, avg=103.90, stdev=11.08 lat (usec): min=87, max=302, avg=114.07, stdev=13.98 | lat (usec): min=82, max=233, avg=109.48, stdev=11.10 | 1.00th=[ 92], 5.00th=[ 96], 10.00th=[ 98], 20. | | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20. | 30.00th=[ 102], 40.00th=[ 104], 50.00th=[ 105], 60. | | 30.00th=[ 98], 40.00th=[ 99], 50.00th=[ 101], 60. | 70.00th=[ 111], 80.00th=[ 114], 90.00th=[ 123], 95. | | 70.00th=[ 105], 80.00th=[ 109], 90.00th=[ 120], 95. | 99.00th=[ 167], 99.50th=[ 180], 99.90th=[ 206], 99. | | 99.00th=[ 141], 99.50th=[ 149], 99.90th=[ 182], 99. | 99.99th=[ 239] | | 99.99th=[ 200] Type=write,Size=64K,NumJobs=1,depth=64 Type=write,Size=64K,NumJobs=1,depth=64 write: IOPS=144k, BW=8977MiB/s (9413MB/s)(87.7GiB/10001msec | write: IOPS=139k, BW=8708MiB/s (9131MB/s)(85.0GiB/10001msec slat (nsec): min=2697, max=72397, avg=4695.54, stdev=1467 | slat (nsec): min=2676, max=64177, avg=5388.79, stdev=1847 clat (usec): min=125, max=987, avg=440.52, stdev=70.72 | clat (usec): min=39, max=894, avg=453.54, stdev=56.76 lat (usec): min=135, max=993, avg=445.25, stdev=70.70 | lat (usec): min=45, max=898, avg=458.97, stdev=57.24 | 1.00th=[ 330], 5.00th=[ 338], 10.00th=[ 347], 20. | | 1.00th=[ 338], 5.00th=[ 355], 10.00th=[ 367], 20. | 30.00th=[ 396], 40.00th=[ 412], 50.00th=[ 437], 60. | | 30.00th=[ 408], 40.00th=[ 457], 50.00th=[ 478], 60. | 70.00th=[ 478], 80.00th=[ 498], 90.00th=[ 529], 95. | | 70.00th=[ 494], 80.00th=[ 498], 90.00th=[ 506], 95. | 99.00th=[ 619], 99.50th=[ 668], 99.90th=[ 791], 99. | | 99.00th=[ 545], 99.50th=[ 562], 99.90th=[ 734], 99. | 99.99th=[ 889] | | 99.99th=[ 848] Type=write,Size=64K,NumJobs=16,depth=1 Type=write,Size=64K,NumJobs=16,depth=1 write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(106GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) slat (nsec): min=4083, max=63053, avg=5530.24, stdev=1175 | slat (nsec): min=4383, max=73755, avg=5607.60, stdev=1278 clat (usec): min=17, max=167, avg=85.80, stdev= 7.67 | clat (usec): min=22, max=424, avg=85.24, stdev= 8.07 lat (usec): min=37, max=172, avg=91.37, stdev= 7.76 | lat (usec): min=38, max=430, avg=90.90, stdev= 8.07 | 1.00th=[ 60], 5.00th=[ 76], 10.00th=[ 80], 20. | | 1.00th=[ 68], 5.00th=[ 76], 10.00th=[ 79], 20. | 30.00th=[ 84], 40.00th=[ 85], 50.00th=[ 86], 60. | | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 84], 60. | 70.00th=[ 88], 80.00th=[ 90], 90.00th=[ 94], 95. | | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 98], 95. | 99.00th=[ 109], 99.50th=[ 115], 99.90th=[ 128], 99. | | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 124], 99. | 99.99th=[ 143] | | 99.99th=[ 137] Type=write,Size=64K,NumJobs=16,depth=16 Type=write,Size=64K,NumJobs=16,depth=16 write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) | write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10002msec) slat (nsec): min=3477, max=59587, avg=6630.17, stdev=1304 | slat (nsec): min=3611, max=70931, avg=6507.12, stdev=1389 clat (usec): min=125, max=5561, avg=1451.88, stdev=382.58 | clat (usec): min=109, max=3915, avg=1451.13, stdev=409.41 lat (usec): min=136, max=5567, avg=1458.55, stdev=382.73 | lat (usec): min=129, max=3920, avg=1457.68, stdev=409.50 | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 1.00th=[ 1254], 5.00th=[ 1270], 10.00th=[ 1270], 20. | 30.00th=[ 1352], 40.00th=[ 1352], 50.00th=[ 1352], 60. | | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1287], 60. | 70.00th=[ 1369], 80.00th=[ 1369], 90.00th=[ 1401], 95. | | 70.00th=[ 1352], 80.00th=[ 1369], 90.00th=[ 2540], 95. | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99. | 99.00th=[ 2737], 99.50th=[ 2737], 99.90th=[ 2769], 99. | 99.99th=[ 2802] | | 99.99th=[ 2769] Type=write,Size=64K,NumJobs=16,depth=64 Type=write,Size=64K,NumJobs=16,depth=64 write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10004msec) slat (usec): min=2, max=6867, avg=10.05, stdev=84.22 | slat (nsec): min=3042, max=96073, avg=6866.24, stdev=2170 clat (usec): min=114, max=17739, avg=5829.80, stdev=1403. | clat (usec): min=124, max=10103, avg=5850.40, stdev=798.1 lat (usec): min=121, max=17748, avg=5839.90, stdev=1417. | lat (usec): min=131, max=10109, avg=5857.31, stdev=798.1 | 1.00th=[ 4228], 5.00th=[ 4490], 10.00th=[ 4621], 20. | | 1.00th=[ 4752], 5.00th=[ 4883], 10.00th=[ 4948], 20. | 30.00th=[ 4948], 40.00th=[ 5145], 50.00th=[ 5473], 60. | | 30.00th=[ 5276], 40.00th=[ 5407], 50.00th=[ 5669], 60. | 70.00th=[ 6194], 80.00th=[ 6718], 90.00th=[ 7242], 95. | | 70.00th=[ 6259], 80.00th=[ 6587], 90.00th=[ 7046], 95. | 99.00th=[11338], 99.50th=[12125], 99.90th=[14484], 99. | | 99.00th=[ 7767], 99.50th=[ 7898], 99.90th=[ 8455], 99. | 99.99th=[16188] | | 99.99th=[ 8848] Type=write,Size=64K,NumJobs=56,depth=1 Type=write,Size=64K,NumJobs=56,depth=1 write: IOPS=176k, BW=10.7GiB/s (11.5GB/s)(107GiB/10001msec) | write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10002msec) slat (nsec): min=4056, max=63403, avg=6384.23, stdev=1521 | slat (nsec): min=4029, max=63780, avg=5771.34, stdev=1537 clat (usec): min=57, max=4543, avg=312.06, stdev=48.67 | clat (usec): min=32, max=979, avg=314.28, stdev=57.25 lat (usec): min=66, max=4549, avg=318.50, stdev=48.73 | lat (usec): min=40, max=985, avg=320.11, stdev=57.29 | 1.00th=[ 188], 5.00th=[ 289], 10.00th=[ 297], 20. | | 1.00th=[ 186], 5.00th=[ 293], 10.00th=[ 297], 20. | 30.00th=[ 306], 40.00th=[ 306], 50.00th=[ 310], 60. | | 30.00th=[ 297], 40.00th=[ 302], 50.00th=[ 302], 60. | 70.00th=[ 310], 80.00th=[ 314], 90.00th=[ 318], 95. | | 70.00th=[ 310], 80.00th=[ 310], 90.00th=[ 326], 95. | 99.00th=[ 478], 99.50th=[ 611], 99.90th=[ 627], 99. | | 99.00th=[ 603], 99.50th=[ 611], 99.90th=[ 627], 99. | 99.99th=[ 750] | 99.99th=[ 750] Type=write,Size=64K,NumJobs=56,depth=16 Type=write,Size=64K,NumJobs=56,depth=16 write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) write: IOPS=175k, BW=10.7GiB/s (11.5GB/s)(107GiB/10005msec) slat (usec): min=3, max=197, avg= 7.76, stdev= 2.34 | slat (nsec): min=3542, max=89310, avg=7585.26, stdev=2460 clat (usec): min=183, max=15544, avg=5102.42, stdev=1298. | clat (usec): min=190, max=14367, avg=5116.17, stdev=1396. lat (usec): min=193, max=15551, avg=5110.24, stdev=1298. | lat (usec): min=198, max=14375, avg=5123.82, stdev=1396. | 1.00th=[ 2442], 5.00th=[ 2540], 10.00th=[ 2573], 20. | | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2573], 20. | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5080], 60. | | 30.00th=[ 5080], 40.00th=[ 5080], 50.00th=[ 5145], 60. | 70.00th=[ 5145], 80.00th=[ 5538], 90.00th=[ 6718], 95. | | 70.00th=[ 5145], 80.00th=[ 5669], 90.00th=[ 6849], 95. | 99.00th=[ 8717], 99.50th=[ 9241], 99.90th=[10552], 99. | | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[11207], 99. | 99.99th=[12125] | | 99.99th=[12780] Type=write,Size=64K,NumJobs=56,depth=64 Type=write,Size=64K,NumJobs=56,depth=64 write: IOPS=175k, BW=10.7GiB/s (11.4GB/s)(107GiB/10013msec) | write: IOPS=174k, BW=10.6GiB/s (11.4GB/s)(107GiB/10011msec) slat (usec): min=2, max=40799, avg=182.18, stdev=918.40 | slat (usec): min=3, max=32839, avg=176.71, stdev=925.53 clat (usec): min=178, max=107843, avg=20332.76, stdev=698 | clat (usec): min=189, max=132076, avg=20366.61, stdev=728 lat (usec): min=182, max=107849, avg=20515.01, stdev=709 | lat (usec): min=218, max=132086, avg=20543.39, stdev=740 | 1.00th=[ 7832], 5.00th=[ 9372], 10.00th=[11469], 20. | | 1.00th=[ 7898], 5.00th=[ 9110], 10.00th=[10945], 20. | 30.00th=[18220], 40.00th=[19530], 50.00th=[20317], 60. | | 30.00th=[17957], 40.00th=[19530], 50.00th=[20317], 60. | 70.00th=[20841], 80.00th=[22938], 90.00th=[27657], 95. | | 70.00th=[21103], 80.00th=[23462], 90.00th=[28705], 95. | 99.00th=[45876], 99.50th=[51643], 99.90th=[64226], 99. | | 99.00th=[45876], 99.50th=[50594], 99.90th=[63177], 99. | 99.99th=[84411] | 99.99th=[84411]
On Wed, Mar 20, 2019 at 11:34:27AM -0700, Sagi Grimberg wrote: > > > It doesn't sound reasonable that we have per-ULP feature (DIM is > > per-ULP) to be configured globally. Especially given the fact that > > users can find themselves running different workloads with different > > requirements on the same system. > > > > Currently each ULP has some sort of tool to configure itself and I think > > that once ULP is converted to use DIM, it should have on/off knob in the > > tool used by their users. > > Its not any different than socket options that can have a global sysctl > knob that can be overridden by individual socket consumers. Right, but there is a major difference between socket example and your proposal. Combination of socket option with general knob gives you maximum versatility in order to disable/enable/configure through program or through some sensible default. In your proposal, you will limit yourself to some system wide default, without any ability to override it specifically for your load. Thanks