staging: vc04_services: ISP: Add a more complex ISP processing component

Driver for the BCM2835 ISP hardware block.  This driver uses the MMAL
component to program the ISP hardware through the VC firmware.

The ISP component can produce two video stream outputs, and Bayer
image statistics. This can't be encompassed in a simple V4L2
M2M device, so create a new device that registers 4 video nodes.

This patch squashes all the development patches from the earlier
rpi-5.4.y branch into one

Signed-off-by: Naushir Patuck <[email protected]>

staging/bcm2835-isp: Add the unpacked (16bpp) raw formats

Now that the firmware supports the unpacked (16bpp) variants
of the MIPI raw formats, add the mappings.

Signed-off-by: Dave Stevenson <[email protected]>

staging/bcm2835-isp: Log the number of excess supported formats

When logging that the firmware has provided more supported formats
than we had allocated storage for, log the number allocated and
returned.

Signed-off-by: Dave Stevenson <[email protected]>

staging: vc04_services: ISP: Add colour denoise control

Add colour denoise control to the bcm2835 driver through a new v4l2
control: V4L2_CID_USER_BCM2835_ISP_CDN.

Add the accompanying MMAL configuration structure definitions as well.

Signed-off-by: Naushir Patuck <[email protected]>

bcm2835-isp: Allow formats with different colour spaces.

Each supported format now includes a mask showing the allowed colour
spaces, as well as a default colour space for when one was not
specified.

Additionally we translate the colour space to mmal format and pass it
over to the VideoCore.

Signed-off-by: David Plowman <[email protected]>

media: i2c: add ov9281 driver.

Change-Id: I7b77250bbc56d2f861450cf77271ad15f9b88ab1
Signed-off-by: Zefa Chen <[email protected]>

media: i2c: ov9281: fix mclk issue when probe multiple camera.

Takes the ov9281 part only from the Rockchip's patch.

Change-Id: I30e833baf2c1bb07d6d87ddb3b00759ab45a90e4
Signed-off-by: Zefa Chen <[email protected]>

media: i2c: ov9281: add enum_frame_interval function for iq tool 2.2 and hal3

Adds the ov9281 parts of the Rockchip patch adding enum_frame_interval to
a large number of drivers.

Change-Id: I03344cd6cf278dd7c18fce8e97479089ef185a5c
Signed-off-by: Zefa Chen <[email protected]>

media: i2c: ov9281: Fixup for recent kernel releases, and remove custom code

The Rockchip driver was based on a 4.4 kernel, and had several custom
Rockchip parts.

Update to 5.4 kernel APIs, with the relevant controls required by
libcamera, and remove custom Rockchip parts.

Signed-off-by: Dave Stevenson <[email protected]>

media: i2c: ov9281: Read chip ID via 2 reads

Vision Components have made an OV9281 module which blocks reading
back the majority of registers to comply with NDAs, and in doing
so doesn't allow auto-increment register reading as used when
reading the chip ID.

Use two reads and manually combine the results.

Signed-off-by: Dave Stevenson <[email protected]>

media: i2c: ov9281: Add support for 8 bit readout

The sensor supports 8 bit mode as well as 10bit, so add the
relevant code to allow selection of this.

Signed-off-by: Dave Stevenson <[email protected]>

media: ov9281: Add 1280x720 and 640x480 modes

Breaks out common register set and adds the different registers
for 1280x720 (cropped) and 640x480 (skipped) modes

Signed-off-by: Dave Stevenson <[email protected]>

Fixed picture line bug in all ov9281 modes

Signed-off-by: Mathias Anhalt <[email protected]>

Added hflip and vflip controls to ov9281

Signed-off-by: Mathias Anhalt <[email protected]>

media: i2c: ov9281: Remove override of subdev name

From the original Rockchip driver, the subdev was renamed
from the default to being "mov9281 <dev_name>" whereas the
default would have been "ov9281 <dev_name>".

Remove the override to drop back to the default rather than
a vendor custom string.

Signed-off-by: Dave Stevenson <[email protected]>

media: v4l2-subdev: add subdev-wide state struct

Signed-off-by: Dom Cobley <[email protected]>

media: i2c: ov9281: Add fwnode properties controls

Add call to v4l2_ctrl_new_fwnode_properties to read and
create the fwnode based controls.

Signed-off-by: Dave Stevenson <[email protected]>

media: i2c: ov9281: Sensor should report RAW color space

Tested on Raspberry Pi running libcamera.

Signed-off-by: David Plowman <[email protected]>

Partial revert "media: i2c: add ov9281 driver."

This partially reverts commit 84e98e3.

The commit had merged some changes to other drivers with adding the ov9281
driver. Only the ov9281 parts have been reverted.

staging/bcm2835-isp: Fix compiler warning

The result of dividing a u32 by a size_t is an unsigned int on arm32
and a long unsigned int on arm64. Use "%zu" (the size_t format) to
remove the build warning for 64-bit builds.

Signed-off-by: Phil Elwell <[email protected]>

staging: vc04_services: isp: Set the YUV420/YVU420 format stride to 64 bytes

The bcm2835 ISP requires the base address of all input/output planes to have 32
byte alignment. Using a Y stride of 32 bytes would not guarantee that the V
plane would fulfil this, e.g. a height of 650 lines would mean the V plane
buffer is not 32 byte aligned for YUV420 formats.

Having a Y stride of 64 bytes would ensure both U and V planes have a 32 byte
alignment, as the luma height will always be an even number of lines.

Signed-off-by: Naushir Patuck <[email protected]>

vc04_services: isp: Report input node as wanting full range RAW color space

RAW color spaces are more usually reported as having full range
quantization.

Tested using libcamera.

Signed-off-by: David Plowman <[email protected]>

drivers: bcm2835_isp: Allow multiple users for the ISP driver.

Add a second (identical) set of device nodes to allow concurrent use of the ISP
hardware by another user. This change effectively creates a second state
structure (struct bcm2835_isp_dev) to maintain independent state for the second
user. Node and media entity names are appened with the instance index
appropriately.

Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES define.

Signed-off-by: Naushir Patuck <[email protected]>

drivers: bcm2835_isp: Fix div by 0 bug.

Fix a possible division by 0 bug when setting up the mmal port for the stats
port.

Signed-off-by: Naushir Patuck <[email protected]>

staging/bcm2835-isp: Fix cleanup after init fail

bcm2835_isp_remove is called on an initialisation failure, but at that
point the drvdata hasn't been set. This causes a crash when e.g. using
the cutdown firmware (gpu_mem=16).

Move platform_set_drvdata before the instance probing loop to avoid the
problem.

See: #4774

Signed-off-by: Phil Elwell <[email protected]>

bcm2835-v4l2-isp: Add missing lock initialization

ISP device allocation is dynamic hence the locks too.
struct mutex queue_lock is not initialized which result in bug.

Fixing same by initializing it.

[   29.847138] INFO: trying to register non-static key.
[   29.847156] The code is fine but needs lockdep annotation, or maybe
[   29.847159] you didn't initialize this object before use?
[   29.847161] turning off the locking correctness validator.
[   29.847167] CPU: 1 PID: 343 Comm: v4l_id Tainted: G         C        5.15.11-rt24-v8+ #8
[   29.847187] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
[   29.847194] Call trace:
[   29.847197]  dump_backtrace+0x0/0x1b8
[   29.847227]  show_stack+0x20/0x30
[   29.847240]  dump_stack_lvl+0x8c/0xb8
[   29.847254]  dump_stack+0x18/0x34
[   29.847263]  register_lock_class+0x494/0x4a0
[   29.847278]  __lock_acquire+0x80/0x1680
[   29.847289]  lock_acquire+0x214/0x3a0
[   29.847300]  mutex_lock_nested+0x70/0xc8
[   29.847312]  _vb2_fop_release+0x3c/0xa8 [videobuf2_v4l2]
[   29.847346]  vb2_fop_release+0x34/0x60 [videobuf2_v4l2]
[   29.847367]  v4l2_release+0xc8/0x108 [videodev]
[   29.847453]  __fput+0x8c/0x258
[   29.847476]  ____fput+0x18/0x28
[   29.847487]  task_work_run+0x98/0x180
[   29.847502]  do_notify_resume+0x228/0x3f8
[   29.847515]  el0_svc+0xec/0xf0
[   29.847523]  el0t_64_sync_handler+0x90/0xb8
[   29.847531]  el0t_64_sync+0x180/0x184

Signed-off-by: Padmanabha Srinivasaiah <[email protected]>

staging: vc04_services: isp: Permit all sRGB colour spaces on ISP outputs

ISP outputs actually support all colour spaces that are fundamentally
sRGB underneath, regardless of whether an RGB or YUV output format is
actually requested.

Signed-off-by: David Plowman <[email protected]>

drivers: staging: bcm2835-isp: Do not cleanup mmal vcsm buffer on stop_streaming

On stop_streaming() the vcsm buffer handle gets released by the buffer cleanup
code.  This will subsequently cause and error if userland re-queues the same
buffer on the next start_streaming() call.

Remove this cleanup code and rely on the vb2_ops->buf_cleanup() call to do the
cleanups instead.

Signed-off-by: Naushir Patuck <[email protected]>

drivers: staging: bcm2835-isp: Clear LS table handle in the firmware

When all nodes have stopped streaming, ensure the firmware has released its
handle on the LS table dmabuf. This is done by passing a null handle in the
LS params.

Signed-off-by: Naushir Patuck <[email protected]>

drivers: staging: bcm2835-isp: Respect caller's stride value

The stride value reported for output image buffers should be at least
as large as any value that was passed in by the caller (subject to
correct alignment for the pixel format). If the value is zero (meaning
no value was passed), or is too small, the minimum acceptable value
will be substituted.

Signed-off-by: David Plowman <[email protected]>

staging: vc04_services: bcm2835-isp: Drop include Makefile directive

Drop the include directive. They can break the build, when one only
wants to build a subdirectory. Replace with "../" for the includes in
the bcm2835-isp instead.

The fix is equivalent to the four patches between 29d49a7
("staging: vc04_services: bcm2835-audio: Drop include Makefile
directive")...2529ca2 ("staging: vc04_services: interface: Drop
include Makefile directive")

Fixes: c8f89c9 ("staging: vc04_services: ISP: Add a more complex ISP processing component")
Suggested-by: Greg Kroah-Hartman <[email protected]>
Signed-off-by: Kieran Bingham <[email protected]>

staging: vc04_services: bcm2835-v4l2-isp: Register with vchiq_bus_type

Register the bcm2835-v4l2-isp driver with the vchiq_bus_type instead of
using the platform driver/device.

Signed-off-by: Kieran Bingham <[email protected]>

staging: vc04_services: bcm2835-v4l2-isp: Explicitly set DMA mask

The platform model originally handled the DMA mask. Now that
we are on the vchiq_bus we need to explicitly set this.

Signed-off-by: Kieran Bingham <[email protected]>

Author: naushir

MAINTAINERS

@@ -4435,6 +4435,15 @@ S:	Maintained
 F:	Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
 F:	drivers/media/platform/broadcom/bcm2835-unicam*
 
+BROADCOM BCM2835 ISP DRIVER
+M:	Raspberry Pi Kernel Maintenance <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	Documentation/media/uapi/v4l/pixfmt-meta-bcm2835-isp-stats.rst
+F:	Documentation/media/v4l-drivers/bcm2835-isp.rst
+F:	drivers/staging/vc04_services/bcm2835-isp
+F:	include/uapi/linux/bcm2835-isp.h
+
 BROADCOM BCM2711 HEVC DECODER
 M:	Raspberry Pi Kernel Maintenance <[email protected]>
 L:	[email protected]

drivers/media/platform/bcm2835/bcm2835-unicam.c

@@ -431,7 +431,7 @@ struct unicam_device {
 	/* ptr to  sub device */
 	struct v4l2_subdev *sensor;
 	/* Pad config for the sensor */
-	struct v4l2_subdev_pad_config *sensor_config;
+	struct v4l2_subdev_state *sensor_state;
 
 	enum v4l2_mbus_type bus_type;
 	/*
@@ -582,7 +582,7 @@ static int __subdev_get_format(struct unicam_device *dev,
 	};
 	int ret;
 
-	ret = v4l2_subdev_call(dev->sensor, pad, get_fmt, dev->sensor_config,
+	ret = v4l2_subdev_call(dev->sensor, pad, get_fmt, dev->sensor_state,
 			       &sd_fmt);
 	if (ret < 0)
 		return ret;
@@ -606,7 +606,7 @@ static int __subdev_set_format(struct unicam_device *dev,
 
 	sd_fmt.format = *fmt;
 
-	ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_config,
+	ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_state,
 			       &sd_fmt);
 	if (ret < 0)
 		return ret;
@@ -1067,7 +1067,7 @@ static int unicam_try_fmt_vid_cap(struct file *file, void *priv,
 	 */
 	mbus_fmt->field = V4L2_FIELD_NONE;
 
-	ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_config,
+	ret = v4l2_subdev_call(dev->sensor, pad, set_fmt, dev->sensor_state,
 			       &sd_fmt);
 	if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV)
 		return ret;
@@ -1089,7 +1089,7 @@ static int unicam_try_fmt_vid_cap(struct file *file, void *priv,
 			mbus_fmt->code = fmt->code;
 
 			ret = v4l2_subdev_call(dev->sensor, pad, set_fmt,
-					       dev->sensor_config, &sd_fmt);
+					       dev->sensor_state, &sd_fmt);
 			if (ret && ret != -ENOIOCTLCMD && ret != -ENODEV)
 				return ret;
 
@@ -2293,8 +2293,8 @@ static void unicam_release(struct kref *kref)
 	v4l2_ctrl_handler_free(&unicam->ctrl_handler);
 	media_device_cleanup(&unicam->mdev);
 
-	if (unicam->sensor_config)
-		v4l2_subdev_free_pad_config(unicam->sensor_config);
+	if (unicam->sensor_state)
+		__v4l2_subdev_state_free(unicam->sensor_state);
 
 	kfree(unicam);
 }
@@ -2552,12 +2552,14 @@ static void unregister_nodes(struct unicam_device *unicam)
 
 static int unicam_probe_complete(struct unicam_device *unicam)
 {
+	static struct lock_class_key key;
 	int ret;
 
 	unicam->v4l2_dev.notify = unicam_notify;
 
-	unicam->sensor_config = v4l2_subdev_alloc_pad_config(unicam->sensor);
-	if (!unicam->sensor_config)
+	unicam->sensor_state = __v4l2_subdev_state_alloc(unicam->sensor,
+							 "unicam:async->lock", &key);
+	if (!unicam->sensor_state)
 		return -ENOMEM;
 
 	unicam->sensor_embedded_data = (unicam->sensor->entity.num_pads >= 2);

drivers/staging/vc04_services/Kconfig

@@ -52,6 +52,7 @@ source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
 
 source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
 source "drivers/staging/vc04_services/bcm2835-codec/Kconfig"
+source "drivers/staging/vc04_services/bcm2835-isp/Kconfig"
 
 source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
 

drivers/staging/vc04_services/Makefile

@@ -16,4 +16,5 @@ obj-$(CONFIG_VIDEO_BCM2835)		+= bcm2835-camera/
 obj-$(CONFIG_BCM2835_VCHIQ_MMAL)	+= vchiq-mmal/
 obj-$(CONFIG_BCM_VC_SM_CMA)		+= vc-sm-cma/
 obj-$(CONFIG_VIDEO_CODEC_BCM2835)	+= bcm2835-codec/
+obj-$(CONFIG_VIDEO_ISP_BCM2835)		+= bcm2835-isp/
 

drivers/staging/vc04_services/bcm2835-isp/Kconfig

@@ -0,0 +1,14 @@
+config VIDEO_ISP_BCM2835
+	tristate "BCM2835 ISP support"
+	depends on MEDIA_SUPPORT
+	depends on VIDEO_DEV && (ARCH_BCM2835 || COMPILE_TEST)
+	depends on MEDIA_CONTROLLER
+	select BCM2835_VCHIQ_MMAL
+	select VIDEOBUF2_DMA_CONTIG
+	help
+	  This is the V4L2 driver for the Broadcom BCM2835 ISP hardware.
+	  This operates over the VCHIQ interface to a service running on
+	  VideoCore.
+
+	  To compile this driver as a module, choose M here: the module
+	  will be called bcm2835-isp.

drivers/staging/vc04_services/bcm2835-isp/Makefile

@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+bcm2835-isp-objs := bcm2835-v4l2-isp.o
+
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp.o
+
+ccflags-y += \
+	-D__VCCOREVER__=0x04000000

drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h

@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck ([email protected])
+ *
+ */
+
+#ifndef BCM2835_ISP_CTRLS
+#define BCM2835_ISP_CTRLS
+
+#include <linux/bcm2835-isp.h>
+
+struct bcm2835_isp_custom_ctrl {
+	const char *name;
+	u32 id;
+	u32 size;
+	u32 flags;
+};
+
+static const struct bcm2835_isp_custom_ctrl custom_ctrls[] = {
+	{
+		.name	= "Colour Correction Matrix",
+		.id	= V4L2_CID_USER_BCM2835_ISP_CC_MATRIX,
+		.size	= sizeof(struct bcm2835_isp_custom_ccm),
+		.flags  = 0
+	}, {
+		.name	= "Lens Shading",
+		.id	= V4L2_CID_USER_BCM2835_ISP_LENS_SHADING,
+		.size	= sizeof(struct bcm2835_isp_lens_shading),
+		.flags  = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE
+	}, {
+		.name	= "Black Level",
+		.id	= V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL,
+		.size	= sizeof(struct bcm2835_isp_black_level),
+		.flags  = 0
+	}, {
+		.name	= "Green Equalisation",
+		.id	= V4L2_CID_USER_BCM2835_ISP_GEQ,
+		.size	= sizeof(struct bcm2835_isp_geq),
+		.flags  = 0
+	}, {
+		.name	= "Gamma",
+		.id	= V4L2_CID_USER_BCM2835_ISP_GAMMA,
+		.size	= sizeof(struct bcm2835_isp_gamma),
+		.flags  = 0
+	}, {
+		.name	= "Sharpen",
+		.id	= V4L2_CID_USER_BCM2835_ISP_SHARPEN,
+		.size	= sizeof(struct bcm2835_isp_sharpen),
+		.flags  = 0
+	}, {
+		.name	= "Denoise",
+		.id	= V4L2_CID_USER_BCM2835_ISP_DENOISE,
+		.size	= sizeof(struct bcm2835_isp_denoise),
+		.flags  = 0
+	}, {
+		.name	= "Colour Denoise",
+		.id	= V4L2_CID_USER_BCM2835_ISP_CDN,
+		.size	= sizeof(struct bcm2835_isp_cdn),
+		.flags  = 0
+	}, {
+		.name	= "Defective Pixel Correction",
+		.id	= V4L2_CID_USER_BCM2835_ISP_DPC,
+		.size	= sizeof(struct bcm2835_isp_dpc),
+		.flags  = 0
+	}
+};
+
+#endif