[RFC] directio and aio-max-nr

Commit Message

Bryn M. Reeves July 2, 2009, 5:12 p.m. UTC

The number of available aio contexts is limited by the fs.aio-max-nr
sysctl. This means it's possible for the io_setup() call that the
directio path checker uses in its libcheck_init() to fail with EAGAIN if
there are already aio-max-nr outstanding event contexts allocated.

On a system with several users of the aio interface it's possible for
this to happen, e.g. when running multipath -ll, leading to output like
this:

sdb: ownership set to 200a0d1c5f449bf01
sdb: not found in pathvec
sdb: mask = 0xc
sdb: path checker = directio (config file default)
io_setup failed
sda: ownership set to 200a0d1c5f449bf01
sda: not found in pathvec
sda: mask = 0xc
sda: path checker = directio (config file default)
io_setup failed
[...]
create: 200a0d1c5f449bf01 n/a Inventec,IX3150-FS200
[size=1.6T][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][undef]
 \_ 1:0:0:0 sdb 8:16  [undef][faulty]
\_ round-robin 0 [prio=0][undef]
 \_ 0:0:0:0 sda 8:0   [undef][faulty]

There's nothing wrong with the paths but since the checker cannot
allocate an aio context it is unable to determine the path status and
reports them as faulty.

Of course, the administrator should be monitoring the use of aio on the
system and setting fs.aio-max-nr to an appropriate value but we're
seeing quite frequent reports of this problem (particularly for Oracle
users with NetApp storage as the NetApp defaults to the directio
checker) and the use of aio by the directio checker is not currently
documented.

I think it's worth documenting this interaction in the FAQ (patch
attached). It think it may also be worth trying to make this more robust
or at least warning the administrator of the problem.

So far, the reports I've seen of this problem have affected users
running multipath -ll manually since this will create new checkers for
each path. Since multipathd starts before most aio-using applications
will start it's able to grab the contexts that it needs. In principal
though it would be possible for multipathd to run into problems if for
e.g. a user added new storage to the system post-boot.

In this situation I'm not sure if it's best to keep trying to allocate
an aio context indefinitely in the checker thread or to fail immediately
and warn the administrator of the situation?

Regards,
Bryn.
The fs.aio-max-nr parameter interacts with the directio path checker
and other users of the aio interfaces on the system. Document the
parameter and the errors seen when it is set too low in the FAQ.
---
 FAQ |   31 +++++++++++++++++++++++++++++++
 1 files changed, 31 insertions(+), 0 deletions(-)

Patch

diff --git a/FAQ b/FAQ
index 35dc2f9..5ceb32b 100644
--- a/FAQ
+++ b/FAQ
@@ -63,3 +63,34 @@  current/default config like so:
 lvm dumpconfig > /etc/lvm/lvm.conf
 
 (tip from Christophe Saout)
+
+4. I see a lot of "io_setup failed" message using the directio checker
+======================================================================
+
+The directio path checker makes use of the asynchronous I/O API (aio) provided
+by modern Linux systems. Asynchronous I/O allows an application to submit I/O
+requests asynchronously and be notified later of their completion status. To
+allow this, we must allocate an asynchronous I/O context (an object of type
+aio_context_t) and this task is handled by the io_setup system call.
+
+A system wide limit on the number of AIO contexts that may be active
+simultaneously is imposed via the aio-max-nr sysctl parameter.
+
+Once this limit has been reached further calls to io_setup will fail with the
+error number EAGAIN leading to the "io_setup failed" messages seen for e.g. when
+running "multipath -ll".
+
+To avoid this problem the number of available aio contexts should be increased
+by setting the aio-max-nr parameter. This can be set on a one-time basis via the
+/proc file system, for e.g.:
+
+        # echo 131072 > /proc/sys/fs/aio-max-nr
+
+Doubles the number of available contexts from the default value of 65536.
+
+To make this setting persistent a line may be added to /etc/sysctl.conf:
+
+        fs.aio-max-nr = 131072
+
+Consult appropriate application and operating system tuning recommendations for
+guidance on appropriate values for this parameter.