| Message ID | 20241114095930.200-4-darinzon@amazon.com (mailing list archive) |
|---|---|
| State | Deferred |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | PHC support in ENA driver | expand |
On Thu, Nov 14, 2024 at 11:59:30AM +0200, David Arinzon wrote: > +**phc_skp** Number of skipped get time attempts (during block period). > +**phc_err** Number of failed get time attempts (entering into block state). Just curious... I understand that the HW can't support a very high rate of gettime calls and that the driver will throttle them. But why did you feel the need to document the throttling behavior in such a overt way? Are there user space programs out there calling gettime excessively? Thanks, Richard
On Sat, Nov 16, 2024 at 05:53:26PM -0800, Richard Cochran wrote: > On Thu, Nov 14, 2024 at 11:59:30AM +0200, David Arinzon wrote: > > > +**phc_skp** Number of skipped get time attempts (during block period). > > +**phc_err** Number of failed get time attempts (entering into block state). > > Just curious... I understand that the HW can't support a very high > rate of gettime calls and that the driver will throttle them. > > But why did you feel the need to document the throttling behavior in > such a overt way? Are there user space programs out there calling > gettime excessively? Answering my own question (maybe) I see that your PHC only supports gettime(), and so I guess you must have some atypical system setup in mind. I didn't see any comments in the cover letter or in the patch about why the PHC isn't adjustable or how offering gettime() only is useful? Thanks, Richard
> On Sat, Nov 16, 2024 at 05:53:26PM -0800, Richard Cochran wrote: > > On Thu, Nov 14, 2024 at 11:59:30AM +0200, David Arinzon wrote: > > > > > +**phc_skp** Number of skipped get time attempts (during block > period). > > > +**phc_err** Number of failed get time attempts (entering into > block state). > > > > Just curious... I understand that the HW can't support a very high > > rate of gettime calls and that the driver will throttle them. > > > > But why did you feel the need to document the throttling behavior in > > such a overt way? Are there user space programs out there calling > > gettime excessively? > > Answering my own question (maybe) > > I see that your PHC only supports gettime(), and so I guess you must have > some atypical system setup in mind. > > I didn't see any comments in the cover letter or in the patch about why the > PHC isn't adjustable or how offering gettime() only is useful? > > Thanks, > Richard Hi Richard Thank you for the queries. Our device limits the number of requests per client (VM) through throttling. To avoid reaching our device throttling limit and ensure a fail-fast mechanism, the driver preemptively applies throttling. In addition, AWS cannot be adjusted or controlled by the OS/User to correct any time deviations, instead, AWS manages the synchronization and accuracy of the clock internally and provides a pre-disciplined clock that already adheres to Coordinated Universal Time (UTC) standards. An upcoming patch, which is a continuation of https://lore.kernel.org/netdev/4c2e99b4-b19e-41f5-a048-3bcc8c33a51c@lunn.ch/T/#m58cf9b05b34046b3952ae8bf4cc7d7f2c8b011d7, will introduce the error bound in ENA PHC, providing internal AWS accuracy error measurements (in nanoseconds). Thanks, David
On Tue, Nov 19, 2024 at 08:45:52AM +0000, Arinzon, David wrote:
> Our device limits the number of requests per client (VM) through throttling.
So it sounds like this device provides time to VM guests?
If so, you might consider generating an interrupt to provide a
"virtual" PPS as some of the other VM solutions do. That way, you
avoid all of those unwanted gettime() calls.
HTH,
Richard
> On Tue, Nov 19, 2024 at 08:45:52AM +0000, Arinzon, David wrote: > > Our device limits the number of requests per client (VM) through > throttling. > > So it sounds like this device provides time to VM guests? > > If so, you might consider generating an interrupt to provide a "virtual" PPS as > some of the other VM solutions do. That way, you avoid all of those > unwanted gettime() calls. > > HTH, > Richard Hi Richard, AWS instances currently lack support for the 1PPS signal. Currently, time information can only be accessed via the GetTime call. Some instance families already provide this functionality, with the solution designed to work consistently across both bare-metal and virtualized environments. Thanks, David
diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst index 4561e8ab..12b13da0 100644 --- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst +++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst @@ -56,6 +56,7 @@ ena_netdev.[ch] Main Linux kernel driver. ena_ethtool.c ethtool callbacks. ena_xdp.[ch] XDP files ena_pci_id_tbl.h Supported device IDs. +ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more info) ================= ====================================================== Management Interface: @@ -221,6 +222,83 @@ descriptor it was received on would be recycled. When a packet smaller than RX copybreak bytes is received, it is copied into a new memory buffer and the RX descriptor is returned to HW. +.. _`PHC`: + +PTP Hardware Clock (PHC) +======================== +.. _`ptp-userspace-api`: https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-space-api +.. _`testptp`: https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests/ptp/testptp.c + +ENA Linux driver supports PTP hardware clock providing timestamp reference to achieve nanosecond resolution. + +**PHC support** + +PHC depends on the PTP module, which needs to be either loaded as a module or compiled into the kernel. + +Verify if the PTP module is present: + +.. code-block:: shell + + grep -w '^CONFIG_PTP_1588_CLOCK=[ym]' /boot/config-`uname -r` + +- If no output is provided, the ENA driver cannot be loaded with PHC support. + +- ``CONFIG_PTP_1588_CLOCK=y``: the PTP module is already compiled and loaded inside the kernel binary file. + +- ``CONFIG_PTP_1588_CLOCK=m``: the PTP module needs to be loaded prior to loading the ENA driver: + +Load PTP module: + +.. code-block:: shell + + sudo modprobe ptp + +All available PTP clock sources can be tracked here: + +.. code-block:: shell + + ls /sys/class/ptp + +PHC support and capabilities can be verified using ethtool: + +.. code-block:: shell + + ethtool -T <interface> + +**PHC timestamp** + +To retrieve PHC timestamp, use `ptp-userspace-api`_, usage example using `testptp`_: + +.. code-block:: shell + + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware Clock:/ {print $NF}') -k 1 + +PHC get time requests should be within reasonable bounds, +avoid excessive utilization to ensure optimal performance and efficiency. +The ENA device restricts the frequency of PHC get time requests to a maximum +of 125 requests per second. If this limit is surpassed, the get time request +will fail, leading to an increment in the phc_err statistic. + +**PHC statistics** + +PHC can be monitored using :code:`ethtool -S` counters: + +================= ====================================================== +**phc_cnt** Number of successful retrieved timestamps (below expire timeout). +**phc_exp** Number of expired retrieved timestamps (above expire timeout). +**phc_skp** Number of skipped get time attempts (during block period). +**phc_err** Number of failed get time attempts (entering into block state). +================= ====================================================== + +PHC timeouts: + +================= ====================================================== +**expire** Max time for a valid timestamp retrieval, passing this threshold will fail + the get time request and block new requests until block timeout. +**block** Blocking period starts once get time request expires or fails, all get time + requests during block period will be skipped. +================= ====================================================== + Statistics ==========