The STM32MP25 processor provides a Digital Camera Memory Interface Pixel Processor (DCMIPP) sourced from a MIPI CSI-2 host controller.

The ConnectCore MP25 Development Kit supports a MIPI CSI camera via a connector composed of two differential data lines, differential clock, reset, and the I2C1 bus.

The BSP includes support for the Omnivision ov5640 CSI camera model.

Kernel configuration

You can manage the camera driver support and Video4Linux (V4L2) capture driver through the following kernel configuration options:

  • STM32 Digital Camera Memory Interface (DCMI) support (CONFIG_VIDEO_STM32_DCMI)

  • STM32 Digital Camera Memory Interface Pixel Processor (DCMIPP) support (CONFIG_VIDEO_STM32_DCMIPP)

  • STM32 Camera Serial Interface (CSI) support (CONFIG_VIDEO_STM32_CSI2HOST)

  • OmniVision ov5640 camera support (CONFIG_VIDEO_OV5640)

These options are enabled as modules and built-in on the default ConnectCore MP25 kernel configuration file.

Kernel driver

The drivers for the camera are located at:

File Description

drivers/media/platform/st/stm32/stm32-dcmi.c

STM32 Digital Camera Memory Interface (DCMI) driver

drivers/media/platform/st/stm32/stm32-dcmipp/

STM32 Digital Camera Memory Interface Pixel Processor (DCMIPP) driver

drivers/media/platform/st/stm32/stm32-csi2host.c

STM32 Camera Serial Interface (CSI) support

drivers/media/i2c/ov5640.c

Omnivision OV5640 sensor driver

Device tree bindings and customization

Common bindings for video receiver and transmitter interfaces are described at Documentation/devicetree/bindings/media/video-interfaces.yaml.

The device tree must contain entries for:

  • The V4L2 capture interface

  • The camera sensor

V4L2 capture interface (DCMI)

ConnectCore MP25 Development Kit device tree
&csi2host {
	status = "okay";
	ports {
		#address-cells = <1>;
		#size-cells = <0>;
		port@0 {
			reg = <0>;
			csi2host_sink: endpoint {
				remote-endpoint = <&ov5640_mipi_ep>;
				data-lanes = <0 1>;
				bus-type = <4>;
			};
		};
		port@1 {
			reg = <1>;
				csi2host_source: endpoint {
				remote-endpoint = <&dcmipp_0>;
			};
		};
	};
};

&dcmipp {
	status = "okay";
	port {
		dcmipp_0: endpoint {
			remote-endpoint = <&csi2host_source>;
			bus-type = <4>;
		};
	};
};

Camera sensor (I2C1 slave)

ConnectCore MP25 Development Kit device tree
&i2c1 {
...
	ov5640_mipi: ov5640_mipi@3c {
		compatible = "ovti,ov5640";
		reg = <0x3c>;
		clocks = <&clk_ext_camera>;
		clock-names = "xclk";
		csi_id = <0>;
		DOVDD-supply = <&reg_3v3_board>;
		reset-gpios = <&gpiog 7 (GPIO_ACTIVE_LOW | GPIO_PUSH_PULL)>; /* CSI_RESET */
		status = "okay";

		port {
			ov5640_mipi_ep: endpoint {
				remote-endpoint = <&csi2host_sink>;
				clock-lanes = <0>;
				data-lanes = <1 2>;
				link-frequencies = /bits/ 64 <594000000>;
			};
		};
	};
...
};

Using the camera

Identify the camera capture devices

Camera configuration and operation is divided between several interconnected subdevices, called entities, sharing video data.

You can use the media-ctl tool to obtain the full camera subsystem topology, a list of subdevices, the connections between them, and some additional postprocessing operations that can be performed on the fly such as cropping or downscaling:

# media-ctl -d platform:dcmipp -p
Media controller API version 6.1.28

Media device information
------------------------
driver          dcmipp
model           DCMIPP MDEV
serial
bus info        platform:dcmipp
hw revision     0x0
driver version  6.1.28

Device topology
- entity 1: dcmipp_parallel (2 pads, 3 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:RGB565_2X8_LE/640x480 field:none colorspace:rec709]
        pad1: Source
                [fmt:RGB565_2X8_LE/640x480 field:none colorspace:rec709]
                -> "dcmipp_dump_postproc":0 []
                -> "dcmipp_main_isp":0 []
                -> "dcmipp_aux_postproc":0 []
[...]
- entity 16: dcmipp_main_postproc (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:RGB888_1X24/640x480 field:none
                 crop.bounds:(0,0)/640x480
                 crop:(0,0)/640x480
                 compose:(0,0)/640x480]
                <- "dcmipp_main_isp":1 [ENABLED,IMMUTABLE]
        pad1: Source
                [fmt:RGB565_2X8_LE/640x480 field:none]
                -> "dcmipp_main_capture":0 [ENABLED,IMMUTABLE]

- entity 19: dcmipp_main_capture (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video3
        pad0: Sink
                <- "dcmipp_main_postproc":1 [ENABLED,IMMUTABLE]
[...]
- entity 68: ov5640 0-003c (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev6
        pad0: Source
                [fmt:UYVY8_2X8/640x480@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]
                -> "stm32_csi2host.48020000.csi2hos":0 [ENABLED,IMMUTABLE]

Alternatively, you can create a dot graph using the same tool to have a visual representation of the whole camera subsystem:

# media-ctl -d platform:dcmipp --print-dot > graph.dot

Use your host PC to convert the graph.dot file to a PNG image:

$ dot -Tpng -Nfontname=Roboto -Nfontsize=10 -Efontname=Roboto -Efontsize=10 graph.dot > graph.png
ConnectCore MP25 camera topology

This graph, generated by the media-ctl tool, shows the different video pipes the data can flow through from the camera sensor on the top (source) to the Linux video devices on the bottom (sink). The camera subsystem creates four video devices, each for a different purpose:

  • /dev/video2 is the capture device for the dump pipe. Through the dump pipe, you can read unprocessed video data.

  • /dev/video3 is the capture device for the main pipe. The main pipe contains most of the processing capabilities of the system, such as scaling, frame skipping, de-Bayering, image correction, color conversion, etc.

  • /dev/video4 is the capture device for the auxiliary pipe. This pipe can perform some limited processing, like scaling and color conversion.

  • /dev/video5 is the capture device for statistical data about the image.

Preview a camera image using gstreamer

To get a camera preview:

  1. Configure all the nodes in the main pipe for the desired resolution and format (in this example, raw video at 1280x720):

    # media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:SBGGR8_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:SBGGR8_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_isp':1[fmt:RGB888_1X24/1280x720 field:none]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':1[fmt:YUYV8_2X8/1280x720]"
  2. Use gstreamer to capture from videosink /dev/video3 using the selected video format and redirect it to your display:

    # gst-launch-1.0 v4l2src device=/dev/video3 ! 'video/x-raw, format=YUY2, width=1280, height=720, framerate=(fraction)30/1' ! queue ! autovideosink

Take a picture with the camera using gstreamer

To take a picture:

  1. Configure all the nodes in the main pipe for the desired resolution and format (in this example, raw video at 1280x720). Ensure the selected format is supported by the jpegenc plugin of gstreamer:

    # media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:SBGGR8_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:SBGGR8_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_isp':1[fmt:RGB888_1X24/1280x720 field:none]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':1[fmt:YUYV8_2X8/1280x720]"
  2. Use gstreamer to capture from videosink /dev/video3 using the selected video format and encode it as a JPEG picture adding jpegenc to the gstreamer pipe:

    # gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=1 ! 'video/x-raw, format=YUY2, width=1280, height=720, framerate=(fraction)30/1' ! jpegenc ! filesink location=grab-1280x720.jpeg
  3. Display the captured image:

    # weston-image grab-1280x720.jpeg

Record a video with the camera using gstreamer

To record a video:

  1. Configure the camera sensor, stm32_csi2host and dcmipp_main_isp nodes for the desired resolution and format (in this example, raw video at 1280x720).

  2. Downscale the frame size using the dcmipp_main_postproc node to avoid a penalization in frame rate produced by video processing latency (in this example, to 640x480). Ensure the selected format is supported by the avimux plugin of gstreamer:

    # media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:SBGGR8_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:SBGGR8_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_isp':1[fmt:RGB888_1X24/1280x720 field:none]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':0[compose:(0,0)/640x480]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':1[fmt:YUYV8_2X8/640x480]"
  3. Use gstreamer to capture several frames from videosink /dev/video3 using the selected video format and encode it as an AVI video adding avimux to the gstreamer pipe (in this example, capture 150 frames at 30 fps for a 5 second video):

    # gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=150 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=(fraction)30/1' ! avimux ! filesink location=output.avi
  4. Play back the recorded video:

    # gst-play-1.0 output.avi

Advanced camera configurations

List allowed configurations for each subdevice

To capture video data from the camera, you must configure every subdevice in the pipe from top to bottom to manage the video data using certain format and frame size. Some subdevices are able to convert the video format on the fly and therefore use a different format on their input than on their output. You can rely on the graph to understand which subdevice should be used for each element of the pipe, and which pad corresponds to their input or output.

For example, list the video format of the first node of the graph (at the top), /dev/v4l-subdev6, which corresponds to the OV5640 camera sensor:

# v4l2-ctl -d /dev/v4l-subdev6 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
        0x4001: MEDIA_BUS_FMT_JPEG_1X8
        0x2006: MEDIA_BUS_FMT_UYVY8_2X8
        0x200f: MEDIA_BUS_FMT_UYVY8_1X16
        0x2008: MEDIA_BUS_FMT_YUYV8_2X8
        0x2011: MEDIA_BUS_FMT_YUYV8_1X16
        0x1008: MEDIA_BUS_FMT_RGB565_2X8_LE
        0x1007: MEDIA_BUS_FMT_RGB565_2X8_BE
        0x3001: MEDIA_BUS_FMT_SBGGR8_1X8
        0x3013: MEDIA_BUS_FMT_SGBRG8_1X8
        0x3002: MEDIA_BUS_FMT_SGRBG8_1X8
        0x3014: MEDIA_BUS_FMT_SRGGB8_1X8

Select one, for example MEDIA_BUS_FMT_RGB565_2X8_LE, and list the available frame sizes for that format:

# v4l2-ctl -d /dev/v4l-subdev6 --list-subdev-framesizes pad=0,code=MEDIA_BUS_FMT_RGB565_2X8_LE
ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0)
        Size Range: 160x120 - 160x120
        Size Range: 176x144 - 176x144
        Size Range: 320x240 - 320x240
        Size Range: 640x480 - 640x480
        Size Range: 720x480 - 720x480
        Size Range: 720x576 - 720x576
        Size Range: 1024x768 - 1024x768
        Size Range: 1280x720 - 1280x720
        Size Range: 1920x1080 - 1920x1080
        Size Range: 2592x1944 - 2592x1944

You can list the available format for the rest of the subdevices using similar commands.

For example, you can see the /dev/v4l-subdev2 node, which corresponds to the main image processor, accepts many possible video formats on its input (pad 0) but is only capable of streaming on two 24-bit video raw formats on its output (pad 1):

# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
        0x1008: MEDIA_BUS_FMT_RGB565_2X8_LE
        0x2008: MEDIA_BUS_FMT_YUYV8_2X8
        0x2006: MEDIA_BUS_FMT_UYVY8_2X8
        0x2009: MEDIA_BUS_FMT_YVYU8_2X8
        0x2007: MEDIA_BUS_FMT_VYUY8_2X8
        0x2001: MEDIA_BUS_FMT_Y8_1X8
        0x3001: MEDIA_BUS_FMT_SBGGR8_1X8
        0x3013: MEDIA_BUS_FMT_SGBRG8_1X8
        0x3002: MEDIA_BUS_FMT_SGRBG8_1X8
        0x3014: MEDIA_BUS_FMT_SRGGB8_1X8
        0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
        0x300e: MEDIA_BUS_FMT_SGBRG10_1X10
        0x300a: MEDIA_BUS_FMT_SGRBG10_1X10
        0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
        0x3008: MEDIA_BUS_FMT_SBGGR12_1X12
        0x3010: MEDIA_BUS_FMT_SGBRG12_1X12
        0x3011: MEDIA_BUS_FMT_SGRBG12_1X12
        0x3012: MEDIA_BUS_FMT_SRGGB12_1X12
        0x3019: MEDIA_BUS_FMT_SBGGR14_1X14
        0x301a: MEDIA_BUS_FMT_SGBRG14_1X14
        0x301b: MEDIA_BUS_FMT_SGRBG14_1X14
        0x301c: MEDIA_BUS_FMT_SRGGB14_1X14
# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 1
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=1)
        0x100a: MEDIA_BUS_FMT_RGB888_1X24
        0x2025: MEDIA_BUS_FMT_YUV8_1X24

Set configurations for a subdevice

You can use the media-ctl tool to set configurations for any media entity in the graph:

# media-ctl -d platform:dcmipp --set-v4l2 "'<entity-name>':<pad>[<settings>]"

Configurable settings include:

  • Video format: fmt:<format-code>/<frame-size>

  • Video field: field:<field-name>

  • Video colorspace: colorspace:<colorspace-name>

  • Crop processing: crop:(<left>,<top>)/<frame-size>

  • Scale processing: compose:(<left>,<top>)/<frame-size>

For example, to configure the camera sensor (which is associated with the ov5640 0-003c media entity in the graph) to stream video with MEDIA_BUS_FMT_RGB565_2X8_LE format and 1280x720 frame size, you must set the media entity output pad (0) fmt property with the following command:

# media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:RGB565_2X8_LE/1280x720]"

If, for example, later down the pipe, you want to downscale the video to a 640x480 frame size, configure the dcmipp_main_postproc media entity input pad (0) to perform a compose operation to that size with the following command:

# media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':0[compose:(0,0)/640x480]"

Get and set additional controls

Some video subdevices have additional controls you can set.

Use the following command to get the list of available controls:

# v4l2-ctl -d <V4L2 subdevice> -L

Use the following command to set a control with a new value:

# v4l2-ctl -d <V4L2 subdevice> --set-ctrl <control-name>=<control-value>

For example, list the available controls of subdevice /dev/v4l-subdev6, which corresponds to the OV5640 camera sensor:

# v4l2-ctl -d /dev/v4l-subdev6 -L

User Controls

                       contrast 0x00980901 (int)    : min=0 max=255 step=1 default=0 value=0 flags=slider
                     saturation 0x00980902 (int)    : min=0 max=255 step=1 default=64 value=64 flags=slider
                            hue 0x00980903 (int)    : min=0 max=359 step=1 default=0 value=0 flags=slider
        white_balance_automatic 0x0098090c (bool)   : default=1 value=1 flags=update
                    red_balance 0x0098090e (int)    : min=0 max=4095 step=1 default=0 value=0 flags=inactive, slider
                   blue_balance 0x0098090f (int)    : min=0 max=4095 step=1 default=0 value=0 flags=inactive, slider
                       exposure 0x00980911 (int)    : min=0 max=65535 step=1 default=0 value=562 flags=inactive, volatile
                 gain_automatic 0x00980912 (bool)   : default=1 value=1 flags=update
                           gain 0x00980913 (int)    : min=0 max=1023 step=1 default=0 value=36 flags=inactive, volatile
                horizontal_flip 0x00980914 (bool)   : default=0 value=0
                  vertical_flip 0x00980915 (bool)   : default=0 value=0
           power_line_frequency 0x00980918 (menu)   : min=0 max=3 default=1 value=1 (50 Hz)
                                0: Disabled
                                1: 50 Hz
                                2: 60 Hz
                                3: Auto

Camera Controls

                  auto_exposure 0x009a0901 (menu)   : min=0 max=1 default=0 value=0 (Auto Mode) flags=update
                                0: Auto Mode
                                1: Manual Mode

Image Processing Controls

                 link_frequency 0x009f0901 (intmenu): min=0 max=8 default=0 value=3 (126273600 0x786c840) flags=read-only
                                0: 63136800 (0x3c36420)
                                1: 83954880 (0x5010cc0)
                                2: 92145600 (0x57e07c0)
                                3: 126273600 (0x786c840)
                                4: 167909760 (0xa021980)
                                5: 184291200 (0xafc0f80)
                                6: 191116800 (0xb643600)
                                7: 335819520 (0x14043300)
                                8: 382233600 (0x16c86c00)
                     pixel_rate 0x009f0902 (int64)  : min=0 max=2147483647 step=1 default=92145600 value=63136800 flags=read-only
                   test_pattern 0x009f0903 (menu)   : min=0 max=4 default=0 value=0 (Disabled)
                                0: Disabled
                                1: Color bars
                                2: Color bars w/ rolling bar
                                3: Color squares
                                4: Color squares w/ rolling bar

Configure, for example, an horizontal flip of the image in the camera sensor. This is performed by the camera sensor itself and does not require later postprocessing:

# v4l2-ctl -d /dev/v4l-subdev6 --set-ctrl horizontal_flip=1

List allowed formats for each video device

Run the following v4l2-ctl command to get the available video formats of the dump, main and aux video devices, which you will later use to capture the frames using v4l2-ctl or gst-launch-1.0 tools:

# v4l2-ctl -d /dev/video2 --list-formats
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'RGBP' (16-bit RGB 5-6-5)
        [1]: 'YUYV' (YUYV 4:2:2)
        [2]: 'YVYU' (YVYU 4:2:2)
        [3]: 'UYVY' (UYVY 4:2:2)
        [4]: 'VYUY' (VYUY 4:2:2)
        [5]: 'GREY' (8-bit Greyscale)
        [6]: 'BA81' (8-bit Bayer BGBG/GRGR)
        [7]: 'GBRG' (8-bit Bayer GBGB/RGRG)
        [8]: 'GRBG' (8-bit Bayer GRGR/BGBG)
        [9]: 'RGGB' (8-bit Bayer RGRG/GBGB)
        [10]: 'JPEG' (JFIF JPEG, compressed)

# v4l2-ctl -d /dev/video3 --list-formats
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'RGBP' (16-bit RGB 5-6-5)
        [1]: 'YUYV' (YUYV 4:2:2)
        [2]: 'YVYU' (YVYU 4:2:2)
        [3]: 'UYVY' (UYVY 4:2:2)
        [4]: 'VYUY' (VYUY 4:2:2)
        [5]: 'GREY' (8-bit Greyscale)
        [6]: 'RGB3' (24-bit RGB 8-8-8)
        [7]: 'BA24' (32-bit ARGB 8-8-8-8)
        [8]: 'AYUV' (32-bit AYUV 8-8-8-8)
        [9]: 'NV12' (Y/UV 4:2:0)
        [10]: 'NV21' (Y/VU 4:2:0)
        [11]: 'NV16' (Y/UV 4:2:2)
        [12]: 'NV61' (Y/VU 4:2:2)
        [13]: 'YU12' (Planar YUV 4:2:0)
        [14]: 'YV12' (Planar YVU 4:2:0)

# v4l2-ctl -d /dev/video4 --list-formats
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'RGBP' (16-bit RGB 5-6-5)
        [1]: 'YUYV' (YUYV 4:2:2)
        [2]: 'YVYU' (YVYU 4:2:2)
        [3]: 'UYVY' (UYVY 4:2:2)
        [4]: 'VYUY' (VYUY 4:2:2)
        [5]: 'GREY' (8-bit Greyscale)
        [6]: 'RGB3' (24-bit RGB 8-8-8)
        [7]: 'BA24' (32-bit ARGB 8-8-8-8)
        [8]: 'AYUV' (32-bit AYUV 8-8-8-8)

These format names are only valid to perform a capture with the v4l2-ctl tool. If you use gstreamer to perform the capture, you must translate the names to the accepted formats. In most cases, it is a direct translation. For example:

v4l2-ctl gstreamer

RGBP

RGB16

YUYV

YUY2

RGB3

RGB

Take a picture with the camera using v4l2-ctl

Once the camera is configured, use the following command to take a picture using v4l2-ctl instead of gstreamer:

# v4l2-ctl --device <video device> --set-fmt-video=width=<width>,height=<height>,pixelformat=<format> --stream-mmap --stream-count=1 --stream-to=<ouput file>

Capturing through the main pipe only supports raw video format which later has to be encoded into JPEG format by a different tool for visualization, for example, gstreamer:

# gst-launch-1.0 filesrc location=<raw file> blocksize=<raw file size> ! "video/x-raw, format=(string)<raw format>, width=(int)<width>, height=(int)<height>, framerate=(fraction)<rate>" ! jpegenc ! filesink location=<JPEG file>

However, both the OV5640 camera sensor and the dump capture device support JPEG format as seen above.

To take a JPEG picture using v4l2-ctl:

  1. Configure all the nodes in the dump pipe for JPEG format and the desired resolution (in this example, 1280x720):

    # media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:JPEG_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:JPEG_1X8/1280x720]"
    # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_dump_postproc':1[fmt:JPEG_1X8/1280x720]"
  2. Use v4l2-ctl to capture from videosink /dev/video2 using the JPEG video format:

    # v4l2-ctl --device /dev/video2 --set-fmt-video=width=1280,height=720,pixelformat=JPEG --stream-mmap --stream-count=1 --stream-to=grab-1280x720.jpeg
  3. Display the captured image:

    # weston-image grab-1280x720.jpeg