From: Jaroslav Kysela <perex@perex.cz>
To: Shengjiu Wang <shengjiu.wang@gmail.com>
Cc: nicoleotsuka@gmail.com, alsa-devel@alsa-project.org,
lgirdwood@gmail.com,
"Sebastian Fricke" <sebastian.fricke@collabora.com>,
Xiubo.Lee@gmail.com, "Takashi Iwai" <tiwai@suse.de>,
linux-kernel@vger.kernel.org,
"Shengjiu Wang" <shengjiu.wang@nxp.com>,
tiwai@suse.com, linux-media@vger.kernel.org, tfiga@chromium.org,
"Hans Verkuil" <hverkuil@xs4all.nl>,
linuxppc-dev@lists.ozlabs.org, "Mark Brown" <broonie@kernel.org>,
sakari.ailus@iki.fi,
"Amadeusz Sławiński" <amadeuszx.slawinski@linux.intel.com>,
"Mauro Carvalho Chehab" <mchehab@kernel.org>,
festevam@gmail.com, m.szyprowski@samsung.com
Subject: Re: [PATCH v15 00/16] Add audio support in v4l2 framework
Date: Thu, 16 May 2024 16:58:53 +0200 [thread overview]
Message-ID: <2411016f-2289-4a2b-8bf8-39ab2f9f1571@perex.cz> (raw)
In-Reply-To: <CAA+D8AOj2ZkiSg2sXfQypg-xc4f8dMykENu5GoGMx6REGu+WBQ@mail.gmail.com>
On 15. 05. 24 15:34, Shengjiu Wang wrote:
> On Wed, May 15, 2024 at 6:46 PM Jaroslav Kysela <perex@perex.cz> wrote:
>>
>> On 15. 05. 24 12:19, Takashi Iwai wrote:
>>> On Wed, 15 May 2024 11:50:52 +0200,
>>> Jaroslav Kysela wrote:
>>>>
>>>> On 15. 05. 24 11:17, Hans Verkuil wrote:
>>>>> Hi Jaroslav,
>>>>>
>>>>> On 5/13/24 13:56, Jaroslav Kysela wrote:
>>>>>> On 09. 05. 24 13:13, Jaroslav Kysela wrote:
>>>>>>> On 09. 05. 24 12:44, Shengjiu Wang wrote:
>>>>>>>>>> mem2mem is just like the decoder in the compress pipeline. which is
>>>>>>>>>> one of the components in the pipeline.
>>>>>>>>>
>>>>>>>>> I was thinking of loopback with endpoints using compress streams,
>>>>>>>>> without physical endpoint, something like:
>>>>>>>>>
>>>>>>>>> compress playback (to feed data from userspace) -> DSP (processing) ->
>>>>>>>>> compress capture (send data back to userspace)
>>>>>>>>>
>>>>>>>>> Unless I'm missing something, you should be able to process data as fast
>>>>>>>>> as you can feed it and consume it in such case.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Actually in the beginning I tried this, but it did not work well.
>>>>>>>> ALSA needs time control for playback and capture, playback and capture
>>>>>>>> needs to synchronize. Usually the playback and capture pipeline is
>>>>>>>> independent in ALSA design, but in this case, the playback and capture
>>>>>>>> should synchronize, they are not independent.
>>>>>>>
>>>>>>> The core compress API core no strict timing constraints. You can eventually0
>>>>>>> have two half-duplex compress devices, if you like to have really independent
>>>>>>> mechanism. If something is missing in API, you can extend this API (like to
>>>>>>> inform the user space that it's a producer/consumer processing without any
>>>>>>> relation to the real time). I like this idea.
>>>>>>
>>>>>> I was thinking more about this. If I am right, the mentioned use in gstreamer
>>>>>> is supposed to run the conversion (DSP) job in "one shot" (can be handled
>>>>>> using one system call like blocking ioctl). The goal is just to offload the
>>>>>> CPU work to the DSP (co-processor). If there are no requirements for the
>>>>>> queuing, we can implement this ioctl in the compress ALSA API easily using the
>>>>>> data management through the dma-buf API. We can eventually define a new
>>>>>> direction (enum snd_compr_direction) like SND_COMPRESS_CONVERT or so to allow
>>>>>> handle this new data scheme. The API may be extended later on real demand, of
>>>>>> course.
>>>>>>
>>>>>> Otherwise all pieces are already in the current ALSA compress API
>>>>>> (capabilities, params, enumeration). The realtime controls may be created
>>>>>> using ALSA control API.
>>>>>
>>>>> So does this mean that Shengjiu should attempt to use this ALSA approach first?
>>>>
>>>> I've not seen any argument to use v4l2 mem2mem buffer scheme for this
>>>> data conversion forcefully. It looks like a simple job and ALSA APIs
>>>> may be extended for this simple purpose.
>>>>
>>>> Shengjiu, what are your requirements for gstreamer support? Would be a
>>>> new blocking ioctl enough for the initial support in the compress ALSA
>>>> API?
>>>
>>> If it works with compress API, it'd be great, yeah.
>>> So, your idea is to open compress-offload devices for read and write,
>>> then and let them convert a la batch jobs without timing control?
>>>
>>> For full-duplex usages, we might need some more extensions, so that
>>> both read and write parameters can be synchronized. (So far the
>>> compress stream is a unidirectional, and the runtime buffer for a
>>> single stream.)
>>>
>>> And the buffer management is based on the fixed size fragments. I
>>> hope this doesn't matter much for the intended operation?
>>
>> It's a question, if the standard I/O is really required for this case. My
>> quick idea was to just implement a new "direction" for this job supporting
>> only one ioctl for the data processing which will execute the job in "one
>> shot" at the moment. The I/O may be handled through dma-buf API (which seems
>> to be standard nowadays for this purpose and allows future chaining).
>>
>> So something like:
>>
>> struct dsp_job {
>> int source_fd; /* dma-buf FD with source data - for dma_buf_get() */
>> int target_fd; /* dma-buf FD for target data - for dma_buf_get() */
>> ... maybe some extra data size members here ...
>> ... maybe some special parameters here ...
>> };
>>
>> #define SNDRV_COMPRESS_DSPJOB _IOWR('C', 0x60, struct dsp_job)
>>
>> This ioctl will be blocking (thus synced). My question is, if it's feasible
>> for gstreamer or not. For this particular case, if the rate conversion is
>> implemented in software, it will block the gstreamer data processing, too.
>>
>
> Thanks.
>
> I have several questions:
> 1. Compress API alway binds to a sound card. Can we avoid that?
> For ASRC, it is just one component,
Is this a real issue? Usually, I would expect a sound hardware (card) presence
when ASRC is available, or not? Eventually, a separate sound card with one
compress device may be created, too. For enumeration - the user space may just
iterate through all sound cards / compress devices to find ASRC in the system.
The devices/interfaces in the sound card are independent. Also, USB MIDI
converters offer only one serial MIDI interface for example, too.
> 2. Compress API doesn't seem to support mmap(). Is this a problem
> for sending and getting data to/from the driver?
I proposed to use dma-buf for I/O (separate source and target buffer).
> 3. How does the user get output data from ASRC after each conversion?
> it should happen every period.
target dma-buf
Jaroslav
--
Jaroslav Kysela <perex@perex.cz>
Linux Sound Maintainer; ALSA Project; Red Hat, Inc.
next prev parent reply other threads:[~2024-05-16 15:00 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-19 7:50 [PATCH v15 00/16] Add audio support in v4l2 framework Shengjiu Wang
2024-03-19 7:50 ` [PATCH v15 01/16] media: v4l2-ctrls: add support for fraction_bits Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 02/16] ASoC: fsl_asrc: define functions for memory to memory usage Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 03/16] ASoC: fsl_easrc: " Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 04/16] ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 05/16] ASoC: fsl_asrc: register m2m platform device Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 06/16] ASoC: fsl_easrc: " Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 07/16] media: uapi: Add V4L2_CAP_AUDIO_M2M capability flag Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 08/16] media: v4l2: Add audio capture and output support Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 09/16] media: uapi: Define audio sample format fourcc type Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 10/16] media: uapi: Add V4L2_CTRL_CLASS_M2M_AUDIO Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 11/16] media: uapi: Add audio rate controls support Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 12/16] media: uapi: Declare interface types for Audio Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 13/16] media: uapi: Add an entity type for audio resampler Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 14/16] media: vivid: add fixed point test controls Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 15/16] media: imx-asrc: Add memory to memory driver Shengjiu Wang
2024-03-19 7:51 ` [PATCH v15 16/16] media: vim2m-audio: add virtual driver for audio memory to memory Shengjiu Wang
[not found] ` <20240430082112.jrovosb6lgblgpfg@basti-XPS-13-9310>
2024-04-30 8:47 ` [PATCH v15 00/16] Add audio support in v4l2 framework Hans Verkuil
2024-04-30 13:52 ` Mauro Carvalho Chehab
2024-04-30 14:46 ` Mark Brown
2024-04-30 15:03 ` Jaroslav Kysela
2024-04-30 16:27 ` Mauro Carvalho Chehab
2024-05-01 1:56 ` Mark Brown
2024-05-02 7:46 ` Takashi Iwai
2024-05-02 8:59 ` Mauro Carvalho Chehab
2024-05-02 9:26 ` Mauro Carvalho Chehab
2024-05-03 1:47 ` Mark Brown
2024-05-03 8:42 ` Mauro Carvalho Chehab
2024-05-06 8:49 ` Shengjiu Wang
2024-05-06 9:42 ` Jaroslav Kysela
2024-05-08 8:00 ` Hans Verkuil
2024-05-08 8:13 ` Amadeusz Sławiński
2024-05-09 9:36 ` Shengjiu Wang
2024-05-09 9:50 ` Amadeusz Sławiński
2024-05-09 10:12 ` Shengjiu Wang
2024-05-09 10:28 ` Amadeusz Sławiński
2024-05-09 10:44 ` Shengjiu Wang
2024-05-09 11:13 ` Jaroslav Kysela
2024-05-13 11:56 ` Jaroslav Kysela
2024-05-15 9:17 ` Hans Verkuil
2024-05-15 9:50 ` Jaroslav Kysela
2024-05-15 10:19 ` Takashi Iwai
2024-05-15 10:46 ` Jaroslav Kysela
2024-05-15 13:34 ` Shengjiu Wang
2024-05-16 14:58 ` Jaroslav Kysela [this message]
2024-05-15 20:33 ` Nicolas Dufresne
2024-05-16 14:50 ` Jaroslav Kysela
2024-05-27 7:24 ` Jaroslav Kysela
2024-05-15 14:04 ` Pierre-Louis Bossart
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2411016f-2289-4a2b-8bf8-39ab2f9f1571@perex.cz \
--to=perex@perex.cz \
--cc=Xiubo.Lee@gmail.com \
--cc=alsa-devel@alsa-project.org \
--cc=amadeuszx.slawinski@linux.intel.com \
--cc=broonie@kernel.org \
--cc=festevam@gmail.com \
--cc=hverkuil@xs4all.nl \
--cc=lgirdwood@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=m.szyprowski@samsung.com \
--cc=mchehab@kernel.org \
--cc=nicoleotsuka@gmail.com \
--cc=sakari.ailus@iki.fi \
--cc=sebastian.fricke@collabora.com \
--cc=shengjiu.wang@gmail.com \
--cc=shengjiu.wang@nxp.com \
--cc=tfiga@chromium.org \
--cc=tiwai@suse.com \
--cc=tiwai@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).