6.18-stable review patch. If anyone has any objections, please let me know. ------------------ From: Junrui Luo [ Upstream commit 2532e9d2f8ccb330a9c924b1ee34c5629240d352 ] Setting the encoder's compressed CAPTURE format to a smaller resolution than the raw OUTPUT format makes the encoder write past the end of the CAPTURE buffer. For a stateful encoder the CAPTURE width and height are not client-settable; Documentation/userspace-api/media/v4l/dev-encoder.rst specifies them as "ignored (read-only)" on VIDIOC_S_FMT. vicodec only implements half of that: vidioc_s_fmt_vid_out() derives the CAPTURE coded size and sizeimage from the OUTPUT format, but S_FMT on the CAPTURE queue overwrites them. The encoder then takes its geometry from the OUTPUT queue alone, and v4l2_fwht_encode() gets no destination length. Overwrite the requested width and height with the OUTPUT queue's coded dimensions in vidioc_try_fmt_vid_cap(), making them read-only as the interface requires. Fixes: efec9c815e5d ("media: vicodec: pass on enc output format to capture side") Reported-by: Yuhao Jiang Assisted-by: Claude:claude-opus-5 Cc: stable@vger.kernel.org Signed-off-by: Junrui Luo Signed-off-by: Hans Verkuil Signed-off-by: Sasha Levin --- drivers/media/test-drivers/vicodec/vicodec-core.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/media/test-drivers/vicodec/vicodec-core.c b/drivers/media/test-drivers/vicodec/vicodec-core.c index 654da27b11406..cdc3af33b4bbe 100644 --- a/drivers/media/test-drivers/vicodec/vicodec-core.c +++ b/drivers/media/test-drivers/vicodec/vicodec-core.c @@ -893,6 +893,8 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv, struct v4l2_format *f) { struct vicodec_ctx *ctx = file2ctx(file); + struct vicodec_q_data *q_data_out = + get_q_data(ctx, V4L2_BUF_TYPE_VIDEO_OUTPUT); struct v4l2_pix_format_mplane *pix_mp; struct v4l2_pix_format *pix; @@ -903,6 +905,10 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv, pix = &f->fmt.pix; pix->pixelformat = ctx->is_enc ? V4L2_PIX_FMT_FWHT : find_fmt(f->fmt.pix.pixelformat)->id; + if (ctx->is_enc) { + pix->width = q_data_out->coded_width; + pix->height = q_data_out->coded_height; + } pix->colorspace = ctx->state.colorspace; pix->xfer_func = ctx->state.xfer_func; pix->ycbcr_enc = ctx->state.ycbcr_enc; @@ -914,6 +920,10 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv, pix_mp = &f->fmt.pix_mp; pix_mp->pixelformat = ctx->is_enc ? V4L2_PIX_FMT_FWHT : find_fmt(pix_mp->pixelformat)->id; + if (ctx->is_enc) { + pix_mp->width = q_data_out->coded_width; + pix_mp->height = q_data_out->coded_height; + } pix_mp->colorspace = ctx->state.colorspace; pix_mp->xfer_func = ctx->state.xfer_func; pix_mp->ycbcr_enc = ctx->state.ycbcr_enc; -- 2.53.0