| Message ID | 1698402948-10618-1-git-send-email-shengjiu.wang@nxp.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add audio support in v4l2 framework | expand |
Hi Shengjiu, Is there a reason why this series is still marked RFC? Just wondering about that. Regards, Hans On 27/10/2023 12:35, Shengjiu Wang wrote: > Audio signal processing also has the requirement for memory to > memory similar as Video. > > This asrc memory to memory (memory ->asrc->memory) case is a non > real time use case. > > User fills the input buffer to the asrc module, after conversion, then asrc > sends back the output buffer to user. So it is not a traditional ALSA playback > and capture case. > > It is a specific use case, there is no reference in current kernel. > v4l2 memory to memory is the closed implementation, v4l2 current > support video, image, radio, tuner, touch devices, so it is not > complicated to add support for this specific audio case. > > Because we had implemented the "memory -> asrc ->i2s device-> codec" > use case in ALSA. Now the "memory->asrc->memory" needs > to reuse the code in asrc driver, so the first 3 patches is for refining > the code to make it can be shared by the "memory->asrc->memory" > driver. > > The main change is in the v4l2 side, A /dev/vl4-audioX will be created, > user applications only use the ioctl of v4l2 framework. > > Other change is to add memory to memory support for two kinds of i.MX ASRC > module. > > changes in v8: > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 > - update doc for FIXED_POINT > - address comments for imx-asrc > > changes in v7: > - add acked-by from Mark > - separate commit for fixed point, m2m audio class, audio rate controls > - use INTEGER_MENU for rate, FIXED_POINT for rate offset > - remove used fmts > - address other comments for Hans > > changes in v6: > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT > - fix warning by kernel test rebot > - remove some unused format V4L2_AUDIO_FMT_XX > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. > - rename audm2m to viaudm2m. > > changes in v5: > - remove V4L2_AUDIO_FMT_LPCM > - define audio pixel format like V4L2_AUDIO_FMT_S8... > - remove rate and format in struct v4l2_audio_format. > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls > - updata document accordingly. > > changes in v4: > - update document style > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit > > changes in v3: > - Modify documents for adding audio m2m support > - Add audio virtual m2m driver > - Defined V4L2_AUDIO_FMT_LPCM format type for audio. > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. > - with modification in v4l-utils, pass v4l2-compliance test. > > changes in v2: > - decouple the implementation in v4l2 and ALSA > - implement the memory to memory driver as a platfrom driver > and move it to driver/media > - move fsl_asrc_common.h to include/sound folder > > Shengjiu Wang (13): > ASoC: fsl_asrc: define functions for memory to memory usage > ASoC: fsl_easrc: define functions for memory to memory usage > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound > ASoC: fsl_asrc: register m2m platform device > ASoC: fsl_easrc: register m2m platform device > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag > media: v4l2: Add audio capture and output support > media: uapi: Define audio sample format fourcc type > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO > media: uapi: Add V4L2_CTRL_TYPE_FIXED_POINT > media: uapi: Add audio rate controls support > media: imx-asrc: Add memory to memory driver > media: vim2m_audio: add virtual driver for audio memory to memory > > .../userspace-api/media/v4l/buffer.rst | 6 + > .../userspace-api/media/v4l/common.rst | 1 + > .../media/v4l/dev-audio-mem2mem.rst | 71 + > .../userspace-api/media/v4l/devices.rst | 1 + > .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + > .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ > .../userspace-api/media/v4l/pixfmt.rst | 1 + > .../media/v4l/vidioc-enum-fmt.rst | 2 + > .../media/v4l/vidioc-g-ext-ctrls.rst | 17 +- > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + > .../media/v4l/vidioc-querycap.rst | 3 + > .../media/v4l/vidioc-queryctrl.rst | 9 +- > .../media/videodev2.h.rst.exceptions | 4 + > .../media/common/videobuf2/videobuf2-v4l2.c | 4 + > drivers/media/platform/nxp/Kconfig | 12 + > drivers/media/platform/nxp/Makefile | 1 + > drivers/media/platform/nxp/imx-asrc.c | 1186 +++++++++++++++++ > drivers/media/test-drivers/Kconfig | 9 + > drivers/media/test-drivers/Makefile | 1 + > drivers/media/test-drivers/vim2m_audio.c | 680 ++++++++++ > drivers/media/v4l2-core/v4l2-ctrls-api.c | 5 +- > drivers/media/v4l2-core/v4l2-ctrls-core.c | 2 + > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 16 + > drivers/media/v4l2-core/v4l2-dev.c | 17 + > drivers/media/v4l2-core/v4l2-ioctl.c | 66 + > include/media/v4l2-dev.h | 2 + > include/media/v4l2-ioctl.h | 34 + > .../fsl => include/sound}/fsl_asrc_common.h | 60 + > include/uapi/linux/v4l2-controls.h | 9 + > include/uapi/linux/videodev2.h | 42 + > sound/soc/fsl/fsl_asrc.c | 144 ++ > sound/soc/fsl/fsl_asrc.h | 4 +- > sound/soc/fsl/fsl_asrc_dma.c | 2 +- > sound/soc/fsl/fsl_easrc.c | 233 ++++ > sound/soc/fsl/fsl_easrc.h | 6 +- > 35 files changed, 2771 insertions(+), 11 deletions(-) > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst > create mode 100644 drivers/media/platform/nxp/imx-asrc.c > create mode 100644 drivers/media/test-drivers/vim2m_audio.c > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) >
On Fri, Oct 27, 2023 at 7:18 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > > Hi Shengjiu, > > Is there a reason why this series is still marked RFC? > > Just wondering about that. In the very beginning I started this series with RFC, So I still use RFC now... Best regards Wang shengjiu > > Regards, > > Hans > > On 27/10/2023 12:35, Shengjiu Wang wrote: > > Audio signal processing also has the requirement for memory to > > memory similar as Video. > > > > This asrc memory to memory (memory ->asrc->memory) case is a non > > real time use case. > > > > User fills the input buffer to the asrc module, after conversion, then asrc > > sends back the output buffer to user. So it is not a traditional ALSA playback > > and capture case. > > > > It is a specific use case, there is no reference in current kernel. > > v4l2 memory to memory is the closed implementation, v4l2 current > > support video, image, radio, tuner, touch devices, so it is not > > complicated to add support for this specific audio case. > > > > Because we had implemented the "memory -> asrc ->i2s device-> codec" > > use case in ALSA. Now the "memory->asrc->memory" needs > > to reuse the code in asrc driver, so the first 3 patches is for refining > > the code to make it can be shared by the "memory->asrc->memory" > > driver. > > > > The main change is in the v4l2 side, A /dev/vl4-audioX will be created, > > user applications only use the ioctl of v4l2 framework. > > > > Other change is to add memory to memory support for two kinds of i.MX ASRC > > module. > > > > changes in v8: > > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 > > - update doc for FIXED_POINT > > - address comments for imx-asrc > > > > changes in v7: > > - add acked-by from Mark > > - separate commit for fixed point, m2m audio class, audio rate controls > > - use INTEGER_MENU for rate, FIXED_POINT for rate offset > > - remove used fmts > > - address other comments for Hans > > > > changes in v6: > > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace > > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. > > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT > > - fix warning by kernel test rebot > > - remove some unused format V4L2_AUDIO_FMT_XX > > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. > > - rename audm2m to viaudm2m. > > > > changes in v5: > > - remove V4L2_AUDIO_FMT_LPCM > > - define audio pixel format like V4L2_AUDIO_FMT_S8... > > - remove rate and format in struct v4l2_audio_format. > > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls > > - updata document accordingly. > > > > changes in v4: > > - update document style > > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit > > > > changes in v3: > > - Modify documents for adding audio m2m support > > - Add audio virtual m2m driver > > - Defined V4L2_AUDIO_FMT_LPCM format type for audio. > > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. > > - with modification in v4l-utils, pass v4l2-compliance test. > > > > changes in v2: > > - decouple the implementation in v4l2 and ALSA > > - implement the memory to memory driver as a platfrom driver > > and move it to driver/media > > - move fsl_asrc_common.h to include/sound folder > > > > Shengjiu Wang (13): > > ASoC: fsl_asrc: define functions for memory to memory usage > > ASoC: fsl_easrc: define functions for memory to memory usage > > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound > > ASoC: fsl_asrc: register m2m platform device > > ASoC: fsl_easrc: register m2m platform device > > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag > > media: v4l2: Add audio capture and output support > > media: uapi: Define audio sample format fourcc type > > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO > > media: uapi: Add V4L2_CTRL_TYPE_FIXED_POINT > > media: uapi: Add audio rate controls support > > media: imx-asrc: Add memory to memory driver > > media: vim2m_audio: add virtual driver for audio memory to memory > > > > .../userspace-api/media/v4l/buffer.rst | 6 + > > .../userspace-api/media/v4l/common.rst | 1 + > > .../media/v4l/dev-audio-mem2mem.rst | 71 + > > .../userspace-api/media/v4l/devices.rst | 1 + > > .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + > > .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ > > .../userspace-api/media/v4l/pixfmt.rst | 1 + > > .../media/v4l/vidioc-enum-fmt.rst | 2 + > > .../media/v4l/vidioc-g-ext-ctrls.rst | 17 +- > > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + > > .../media/v4l/vidioc-querycap.rst | 3 + > > .../media/v4l/vidioc-queryctrl.rst | 9 +- > > .../media/videodev2.h.rst.exceptions | 4 + > > .../media/common/videobuf2/videobuf2-v4l2.c | 4 + > > drivers/media/platform/nxp/Kconfig | 12 + > > drivers/media/platform/nxp/Makefile | 1 + > > drivers/media/platform/nxp/imx-asrc.c | 1186 +++++++++++++++++ > > drivers/media/test-drivers/Kconfig | 9 + > > drivers/media/test-drivers/Makefile | 1 + > > drivers/media/test-drivers/vim2m_audio.c | 680 ++++++++++ > > drivers/media/v4l2-core/v4l2-ctrls-api.c | 5 +- > > drivers/media/v4l2-core/v4l2-ctrls-core.c | 2 + > > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 16 + > > drivers/media/v4l2-core/v4l2-dev.c | 17 + > > drivers/media/v4l2-core/v4l2-ioctl.c | 66 + > > include/media/v4l2-dev.h | 2 + > > include/media/v4l2-ioctl.h | 34 + > > .../fsl => include/sound}/fsl_asrc_common.h | 60 + > > include/uapi/linux/v4l2-controls.h | 9 + > > include/uapi/linux/videodev2.h | 42 + > > sound/soc/fsl/fsl_asrc.c | 144 ++ > > sound/soc/fsl/fsl_asrc.h | 4 +- > > sound/soc/fsl/fsl_asrc_dma.c | 2 +- > > sound/soc/fsl/fsl_easrc.c | 233 ++++ > > sound/soc/fsl/fsl_easrc.h | 6 +- > > 35 files changed, 2771 insertions(+), 11 deletions(-) > > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst > > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst > > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst > > create mode 100644 drivers/media/platform/nxp/imx-asrc.c > > create mode 100644 drivers/media/test-drivers/vim2m_audio.c > > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) > > >
On Mon, Oct 30, 2023 at 3:56 AM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > On Fri, Oct 27, 2023 at 7:18 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > > > > Hi Shengjiu, > > > > Is there a reason why this series is still marked RFC? > > > > Just wondering about that. > > In the very beginning I started this series with RFC, So > I still use RFC now... I think we are way past the RFC stage so if there will be another version, it is better to remove the RFC tag. RFC means that this patch series is not ready to be merged but is merely in the incipient stages of development. thanks, Daniel.
On Mon, Oct 30, 2023 at 9:56 AM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > On Fri, Oct 27, 2023 at 7:18 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > > > > Hi Shengjiu, > > > > Is there a reason why this series is still marked RFC? > > > > Just wondering about that. > > In the very beginning I started this series with RFC, So > I still use RFC now... > Should I resend patches which remove the 'RFC'? Best regards Wang shengjiu > > > > Regards, > > > > Hans > > > > On 27/10/2023 12:35, Shengjiu Wang wrote: > > > Audio signal processing also has the requirement for memory to > > > memory similar as Video. > > > > > > This asrc memory to memory (memory ->asrc->memory) case is a non > > > real time use case. > > > > > > User fills the input buffer to the asrc module, after conversion, then asrc > > > sends back the output buffer to user. So it is not a traditional ALSA playback > > > and capture case. > > > > > > It is a specific use case, there is no reference in current kernel. > > > v4l2 memory to memory is the closed implementation, v4l2 current > > > support video, image, radio, tuner, touch devices, so it is not > > > complicated to add support for this specific audio case. > > > > > > Because we had implemented the "memory -> asrc ->i2s device-> codec" > > > use case in ALSA. Now the "memory->asrc->memory" needs > > > to reuse the code in asrc driver, so the first 3 patches is for refining > > > the code to make it can be shared by the "memory->asrc->memory" > > > driver. > > > > > > The main change is in the v4l2 side, A /dev/vl4-audioX will be created, > > > user applications only use the ioctl of v4l2 framework. > > > > > > Other change is to add memory to memory support for two kinds of i.MX ASRC > > > module. > > > > > > changes in v8: > > > - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 > > > - update doc for FIXED_POINT > > > - address comments for imx-asrc > > > > > > changes in v7: > > > - add acked-by from Mark > > > - separate commit for fixed point, m2m audio class, audio rate controls > > > - use INTEGER_MENU for rate, FIXED_POINT for rate offset > > > - remove used fmts > > > - address other comments for Hans > > > > > > changes in v6: > > > - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace > > > m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. > > > - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT > > > - fix warning by kernel test rebot > > > - remove some unused format V4L2_AUDIO_FMT_XX > > > - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. > > > - rename audm2m to viaudm2m. > > > > > > changes in v5: > > > - remove V4L2_AUDIO_FMT_LPCM > > > - define audio pixel format like V4L2_AUDIO_FMT_S8... > > > - remove rate and format in struct v4l2_audio_format. > > > - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls > > > - updata document accordingly. > > > > > > changes in v4: > > > - update document style > > > - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit > > > > > > changes in v3: > > > - Modify documents for adding audio m2m support > > > - Add audio virtual m2m driver > > > - Defined V4L2_AUDIO_FMT_LPCM format type for audio. > > > - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. > > > - with modification in v4l-utils, pass v4l2-compliance test. > > > > > > changes in v2: > > > - decouple the implementation in v4l2 and ALSA > > > - implement the memory to memory driver as a platfrom driver > > > and move it to driver/media > > > - move fsl_asrc_common.h to include/sound folder > > > > > > Shengjiu Wang (13): > > > ASoC: fsl_asrc: define functions for memory to memory usage > > > ASoC: fsl_easrc: define functions for memory to memory usage > > > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound > > > ASoC: fsl_asrc: register m2m platform device > > > ASoC: fsl_easrc: register m2m platform device > > > media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag > > > media: v4l2: Add audio capture and output support > > > media: uapi: Define audio sample format fourcc type > > > media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO > > > media: uapi: Add V4L2_CTRL_TYPE_FIXED_POINT > > > media: uapi: Add audio rate controls support > > > media: imx-asrc: Add memory to memory driver > > > media: vim2m_audio: add virtual driver for audio memory to memory > > > > > > .../userspace-api/media/v4l/buffer.rst | 6 + > > > .../userspace-api/media/v4l/common.rst | 1 + > > > .../media/v4l/dev-audio-mem2mem.rst | 71 + > > > .../userspace-api/media/v4l/devices.rst | 1 + > > > .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + > > > .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ > > > .../userspace-api/media/v4l/pixfmt.rst | 1 + > > > .../media/v4l/vidioc-enum-fmt.rst | 2 + > > > .../media/v4l/vidioc-g-ext-ctrls.rst | 17 +- > > > .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + > > > .../media/v4l/vidioc-querycap.rst | 3 + > > > .../media/v4l/vidioc-queryctrl.rst | 9 +- > > > .../media/videodev2.h.rst.exceptions | 4 + > > > .../media/common/videobuf2/videobuf2-v4l2.c | 4 + > > > drivers/media/platform/nxp/Kconfig | 12 + > > > drivers/media/platform/nxp/Makefile | 1 + > > > drivers/media/platform/nxp/imx-asrc.c | 1186 +++++++++++++++++ > > > drivers/media/test-drivers/Kconfig | 9 + > > > drivers/media/test-drivers/Makefile | 1 + > > > drivers/media/test-drivers/vim2m_audio.c | 680 ++++++++++ > > > drivers/media/v4l2-core/v4l2-ctrls-api.c | 5 +- > > > drivers/media/v4l2-core/v4l2-ctrls-core.c | 2 + > > > drivers/media/v4l2-core/v4l2-ctrls-defs.c | 16 + > > > drivers/media/v4l2-core/v4l2-dev.c | 17 + > > > drivers/media/v4l2-core/v4l2-ioctl.c | 66 + > > > include/media/v4l2-dev.h | 2 + > > > include/media/v4l2-ioctl.h | 34 + > > > .../fsl => include/sound}/fsl_asrc_common.h | 60 + > > > include/uapi/linux/v4l2-controls.h | 9 + > > > include/uapi/linux/videodev2.h | 42 + > > > sound/soc/fsl/fsl_asrc.c | 144 ++ > > > sound/soc/fsl/fsl_asrc.h | 4 +- > > > sound/soc/fsl/fsl_asrc_dma.c | 2 +- > > > sound/soc/fsl/fsl_easrc.c | 233 ++++ > > > sound/soc/fsl/fsl_easrc.h | 6 +- > > > 35 files changed, 2771 insertions(+), 11 deletions(-) > > > create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst > > > create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst > > > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst > > > create mode 100644 drivers/media/platform/nxp/imx-asrc.c > > > create mode 100644 drivers/media/test-drivers/vim2m_audio.c > > > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) > > > > >
On 01/11/2023 12:05, Shengjiu Wang wrote: > On Mon, Oct 30, 2023 at 9:56 AM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: >> >> On Fri, Oct 27, 2023 at 7:18 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: >>> >>> Hi Shengjiu, >>> >>> Is there a reason why this series is still marked RFC? >>> >>> Just wondering about that. >> >> In the very beginning I started this series with RFC, So >> I still use RFC now... >> > > Should I resend patches which remove the 'RFC'? No need, but if a v9 is needed, then please remove the RFC. I haven't had time yet to do a proper review, I hope to get to it next week or the week after. Ping me if it takes longer, just to remind me of it. One problem is that besides reviewing the driver, it is also required to add support for this new V4L2 feature to the v4l2-ctl and v4l2-compliance utilities, and also regression tests for the new virtual audio driver need to be added to contrib/test/test-media. All part of git://linuxtv.org/v4l-utils.git. Ideally I would just ask you to do this work, but it might be quicker if I do it since it is a bit of a learning curve. Adding support for it to v4l2-compliance is especially important since the drivers have to pass the compliance tests before we can accept them in the media subsystem. But in this case, v4l2-compliance first needs to be taught about the new v4l-audio device type. Regards, Hans > > Best regards > Wang shengjiu > >>> >>> Regards, >>> >>> Hans >>> >>> On 27/10/2023 12:35, Shengjiu Wang wrote: >>>> Audio signal processing also has the requirement for memory to >>>> memory similar as Video. >>>> >>>> This asrc memory to memory (memory ->asrc->memory) case is a non >>>> real time use case. >>>> >>>> User fills the input buffer to the asrc module, after conversion, then asrc >>>> sends back the output buffer to user. So it is not a traditional ALSA playback >>>> and capture case. >>>> >>>> It is a specific use case, there is no reference in current kernel. >>>> v4l2 memory to memory is the closed implementation, v4l2 current >>>> support video, image, radio, tuner, touch devices, so it is not >>>> complicated to add support for this specific audio case. >>>> >>>> Because we had implemented the "memory -> asrc ->i2s device-> codec" >>>> use case in ALSA. Now the "memory->asrc->memory" needs >>>> to reuse the code in asrc driver, so the first 3 patches is for refining >>>> the code to make it can be shared by the "memory->asrc->memory" >>>> driver. >>>> >>>> The main change is in the v4l2 side, A /dev/vl4-audioX will be created, >>>> user applications only use the ioctl of v4l2 framework. >>>> >>>> Other change is to add memory to memory support for two kinds of i.MX ASRC >>>> module. >>>> >>>> changes in v8: >>>> - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 >>>> - update doc for FIXED_POINT >>>> - address comments for imx-asrc >>>> >>>> changes in v7: >>>> - add acked-by from Mark >>>> - separate commit for fixed point, m2m audio class, audio rate controls >>>> - use INTEGER_MENU for rate, FIXED_POINT for rate offset >>>> - remove used fmts >>>> - address other comments for Hans >>>> >>>> changes in v6: >>>> - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace >>>> m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. >>>> - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT >>>> - fix warning by kernel test rebot >>>> - remove some unused format V4L2_AUDIO_FMT_XX >>>> - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. >>>> - rename audm2m to viaudm2m. >>>> >>>> changes in v5: >>>> - remove V4L2_AUDIO_FMT_LPCM >>>> - define audio pixel format like V4L2_AUDIO_FMT_S8... >>>> - remove rate and format in struct v4l2_audio_format. >>>> - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls >>>> - updata document accordingly. >>>> >>>> changes in v4: >>>> - update document style >>>> - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit >>>> >>>> changes in v3: >>>> - Modify documents for adding audio m2m support >>>> - Add audio virtual m2m driver >>>> - Defined V4L2_AUDIO_FMT_LPCM format type for audio. >>>> - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. >>>> - with modification in v4l-utils, pass v4l2-compliance test. >>>> >>>> changes in v2: >>>> - decouple the implementation in v4l2 and ALSA >>>> - implement the memory to memory driver as a platfrom driver >>>> and move it to driver/media >>>> - move fsl_asrc_common.h to include/sound folder >>>> >>>> Shengjiu Wang (13): >>>> ASoC: fsl_asrc: define functions for memory to memory usage >>>> ASoC: fsl_easrc: define functions for memory to memory usage >>>> ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound >>>> ASoC: fsl_asrc: register m2m platform device >>>> ASoC: fsl_easrc: register m2m platform device >>>> media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag >>>> media: v4l2: Add audio capture and output support >>>> media: uapi: Define audio sample format fourcc type >>>> media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO >>>> media: uapi: Add V4L2_CTRL_TYPE_FIXED_POINT >>>> media: uapi: Add audio rate controls support >>>> media: imx-asrc: Add memory to memory driver >>>> media: vim2m_audio: add virtual driver for audio memory to memory >>>> >>>> .../userspace-api/media/v4l/buffer.rst | 6 + >>>> .../userspace-api/media/v4l/common.rst | 1 + >>>> .../media/v4l/dev-audio-mem2mem.rst | 71 + >>>> .../userspace-api/media/v4l/devices.rst | 1 + >>>> .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + >>>> .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ >>>> .../userspace-api/media/v4l/pixfmt.rst | 1 + >>>> .../media/v4l/vidioc-enum-fmt.rst | 2 + >>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 17 +- >>>> .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + >>>> .../media/v4l/vidioc-querycap.rst | 3 + >>>> .../media/v4l/vidioc-queryctrl.rst | 9 +- >>>> .../media/videodev2.h.rst.exceptions | 4 + >>>> .../media/common/videobuf2/videobuf2-v4l2.c | 4 + >>>> drivers/media/platform/nxp/Kconfig | 12 + >>>> drivers/media/platform/nxp/Makefile | 1 + >>>> drivers/media/platform/nxp/imx-asrc.c | 1186 +++++++++++++++++ >>>> drivers/media/test-drivers/Kconfig | 9 + >>>> drivers/media/test-drivers/Makefile | 1 + >>>> drivers/media/test-drivers/vim2m_audio.c | 680 ++++++++++ >>>> drivers/media/v4l2-core/v4l2-ctrls-api.c | 5 +- >>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 2 + >>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 16 + >>>> drivers/media/v4l2-core/v4l2-dev.c | 17 + >>>> drivers/media/v4l2-core/v4l2-ioctl.c | 66 + >>>> include/media/v4l2-dev.h | 2 + >>>> include/media/v4l2-ioctl.h | 34 + >>>> .../fsl => include/sound}/fsl_asrc_common.h | 60 + >>>> include/uapi/linux/v4l2-controls.h | 9 + >>>> include/uapi/linux/videodev2.h | 42 + >>>> sound/soc/fsl/fsl_asrc.c | 144 ++ >>>> sound/soc/fsl/fsl_asrc.h | 4 +- >>>> sound/soc/fsl/fsl_asrc_dma.c | 2 +- >>>> sound/soc/fsl/fsl_easrc.c | 233 ++++ >>>> sound/soc/fsl/fsl_easrc.h | 6 +- >>>> 35 files changed, 2771 insertions(+), 11 deletions(-) >>>> create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst >>>> create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst >>>> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst >>>> create mode 100644 drivers/media/platform/nxp/imx-asrc.c >>>> create mode 100644 drivers/media/test-drivers/vim2m_audio.c >>>> rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%) >>>> >>>
Audio signal processing also has the requirement for memory to memory similar as Video. This asrc memory to memory (memory ->asrc->memory) case is a non real time use case. User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case. It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case. Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver. The main change is in the v4l2 side, A /dev/vl4-audioX will be created, user applications only use the ioctl of v4l2 framework. Other change is to add memory to memory support for two kinds of i.MX ASRC module. changes in v8: - refine V4L2_CAP_AUDIO_M2M to be 0x00000008 - update doc for FIXED_POINT - address comments for imx-asrc changes in v7: - add acked-by from Mark - separate commit for fixed point, m2m audio class, audio rate controls - use INTEGER_MENU for rate, FIXED_POINT for rate offset - remove used fmts - address other comments for Hans changes in v6: - use m2m_prepare/m2m_unprepare/m2m_start/m2m_stop to replace m2m_start_part_one/m2m_stop_part_one, m2m_start_part_two/m2m_stop_part_two. - change V4L2_CTRL_TYPE_ASRC_RATE to V4L2_CTRL_TYPE_FIXED_POINT - fix warning by kernel test rebot - remove some unused format V4L2_AUDIO_FMT_XX - Get SNDRV_PCM_FORMAT from V4L2_AUDIO_FMT in driver. - rename audm2m to viaudm2m. changes in v5: - remove V4L2_AUDIO_FMT_LPCM - define audio pixel format like V4L2_AUDIO_FMT_S8... - remove rate and format in struct v4l2_audio_format. - Add V4L2_CID_ASRC_SOURCE_RATE and V4L2_CID_ASRC_DEST_RATE controls - updata document accordingly. changes in v4: - update document style - separate V4L2_AUDIO_FMT_LPCM and V4L2_CAP_AUDIO_M2M in separate commit changes in v3: - Modify documents for adding audio m2m support - Add audio virtual m2m driver - Defined V4L2_AUDIO_FMT_LPCM format type for audio. - Defined V4L2_CAP_AUDIO_M2M capability type for audio m2m case. - with modification in v4l-utils, pass v4l2-compliance test. changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder Shengjiu Wang (13): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag media: v4l2: Add audio capture and output support media: uapi: Define audio sample format fourcc type media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO media: uapi: Add V4L2_CTRL_TYPE_FIXED_POINT media: uapi: Add audio rate controls support media: imx-asrc: Add memory to memory driver media: vim2m_audio: add virtual driver for audio memory to memory .../userspace-api/media/v4l/buffer.rst | 6 + .../userspace-api/media/v4l/common.rst | 1 + .../media/v4l/dev-audio-mem2mem.rst | 71 + .../userspace-api/media/v4l/devices.rst | 1 + .../media/v4l/ext-ctrls-audio-m2m.rst | 41 + .../userspace-api/media/v4l/pixfmt-audio.rst | 87 ++ .../userspace-api/media/v4l/pixfmt.rst | 1 + .../media/v4l/vidioc-enum-fmt.rst | 2 + .../media/v4l/vidioc-g-ext-ctrls.rst | 17 +- .../userspace-api/media/v4l/vidioc-g-fmt.rst | 4 + .../media/v4l/vidioc-querycap.rst | 3 + .../media/v4l/vidioc-queryctrl.rst | 9 +- .../media/videodev2.h.rst.exceptions | 4 + .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/imx-asrc.c | 1186 +++++++++++++++++ drivers/media/test-drivers/Kconfig | 9 + drivers/media/test-drivers/Makefile | 1 + drivers/media/test-drivers/vim2m_audio.c | 680 ++++++++++ drivers/media/v4l2-core/v4l2-ctrls-api.c | 5 +- drivers/media/v4l2-core/v4l2-ctrls-core.c | 2 + drivers/media/v4l2-core/v4l2-ctrls-defs.c | 16 + drivers/media/v4l2-core/v4l2-dev.c | 17 + drivers/media/v4l2-core/v4l2-ioctl.c | 66 + include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 60 + include/uapi/linux/v4l2-controls.h | 9 + include/uapi/linux/videodev2.h | 42 + sound/soc/fsl/fsl_asrc.c | 144 ++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 233 ++++ sound/soc/fsl/fsl_easrc.h | 6 +- 35 files changed, 2771 insertions(+), 11 deletions(-) create mode 100644 Documentation/userspace-api/media/v4l/dev-audio-mem2mem.rst create mode 100644 Documentation/userspace-api/media/v4l/ext-ctrls-audio-m2m.rst create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-audio.rst create mode 100644 drivers/media/platform/nxp/imx-asrc.c create mode 100644 drivers/media/test-drivers/vim2m_audio.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (60%)