The STM32MP25 processor provides a Digital Camera Memory Interface Pixel Processor (DCMIPP) sourced from a MIPI CSI-2 host controller.
The ConnectCore MP25 Development Kit supports a MIPI CSI camera via a connector composed of two differential data lines, differential clock, reset, and the I2C1 bus.
The BSP includes support for the Omnivision ov5640 CSI camera model.
Kernel configuration
You can manage the camera driver support and Video4Linux (V4L2) capture driver through the following kernel configuration options:
-
STM32 Digital Camera Memory Interface (DCMI) support (
CONFIG_VIDEO_STM32_DCMI
) -
STM32 Digital Camera Memory Interface Pixel Processor (DCMIPP) support (
CONFIG_VIDEO_STM32_DCMIPP
) -
STM32 Camera Serial Interface (CSI) support (
CONFIG_VIDEO_STM32_CSI2HOST
) -
OmniVision ov5640 camera support (
CONFIG_VIDEO_OV5640
)
These options are enabled as modules and built-in on the default ConnectCore MP25 kernel configuration file.
Kernel driver
The drivers for the camera are located at:
File | Description |
---|---|
STM32 Digital Camera Memory Interface (DCMI) driver |
|
STM32 Digital Camera Memory Interface Pixel Processor (DCMIPP) driver |
|
STM32 Camera Serial Interface (CSI) support |
|
Omnivision OV5640 sensor driver |
Device tree bindings and customization
Common bindings for video receiver and transmitter interfaces are described at Documentation/devicetree/bindings/media/video-interfaces.yaml
.
The device tree must contain entries for:
-
The V4L2 capture interface
-
The camera sensor
V4L2 capture interface (DCMI)
&csi2host {
status = "okay";
ports {
#address-cells = <1>;
#size-cells = <0>;
port@0 {
reg = <0>;
csi2host_sink: endpoint {
remote-endpoint = <&ov5640_mipi_ep>;
data-lanes = <0 1>;
bus-type = <4>;
};
};
port@1 {
reg = <1>;
csi2host_source: endpoint {
remote-endpoint = <&dcmipp_0>;
};
};
};
};
&dcmipp {
status = "okay";
port {
dcmipp_0: endpoint {
remote-endpoint = <&csi2host_source>;
bus-type = <4>;
};
};
};
Camera sensor (I2C1 slave)
&i2c1 {
...
ov5640_mipi: ov5640_mipi@3c {
compatible = "ovti,ov5640";
reg = <0x3c>;
clocks = <&clk_ext_camera>;
clock-names = "xclk";
csi_id = <0>;
DOVDD-supply = <®_3v3_board>;
reset-gpios = <&gpiog 7 (GPIO_ACTIVE_LOW | GPIO_PUSH_PULL)>; /* CSI_RESET */
status = "okay";
port {
ov5640_mipi_ep: endpoint {
remote-endpoint = <&csi2host_sink>;
clock-lanes = <0>;
data-lanes = <1 2>;
link-frequencies = /bits/ 64 <594000000>;
};
};
};
...
};
Using the camera
Identify the camera capture devices
Camera configuration and operation is divided between several interconnected subdevices, called entities, sharing video data.
You can use the media-ctl
tool to obtain the full camera subsystem topology, a list of subdevices, the connections between them, and some additional postprocessing operations that can be performed on the fly such as cropping or downscaling:
# media-ctl -d platform:dcmipp -p
Media controller API version 6.1.28
Media device information
------------------------
driver dcmipp
model DCMIPP MDEV
serial
bus info platform:dcmipp
hw revision 0x0
driver version 6.1.28
Device topology
- entity 1: dcmipp_parallel (2 pads, 3 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
[fmt:RGB565_2X8_LE/640x480 field:none colorspace:rec709]
pad1: Source
[fmt:RGB565_2X8_LE/640x480 field:none colorspace:rec709]
-> "dcmipp_dump_postproc":0 []
-> "dcmipp_main_isp":0 []
-> "dcmipp_aux_postproc":0 []
[...]
- entity 16: dcmipp_main_postproc (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev3
pad0: Sink
[fmt:RGB888_1X24/640x480 field:none
crop.bounds:(0,0)/640x480
crop:(0,0)/640x480
compose:(0,0)/640x480]
<- "dcmipp_main_isp":1 [ENABLED,IMMUTABLE]
pad1: Source
[fmt:RGB565_2X8_LE/640x480 field:none]
-> "dcmipp_main_capture":0 [ENABLED,IMMUTABLE]
- entity 19: dcmipp_main_capture (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video3
pad0: Sink
<- "dcmipp_main_postproc":1 [ENABLED,IMMUTABLE]
[...]
- entity 68: ov5640 0-003c (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev6
pad0: Source
[fmt:UYVY8_2X8/640x480@1/30 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:full-range]
-> "stm32_csi2host.48020000.csi2hos":0 [ENABLED,IMMUTABLE]
Alternatively, you can create a dot graph using the same tool to have a visual representation of the whole camera subsystem:
# media-ctl -d platform:dcmipp --print-dot > graph.dot
Use your host PC to convert the graph.dot
file to a PNG image:
$ dot -Tpng -Nfontname=Roboto -Nfontsize=10 -Efontname=Roboto -Efontsize=10 graph.dot > graph.png
This graph, generated by the media-ctl
tool, shows the different video pipes the data can flow through from the camera sensor on the top (source) to the Linux video devices on the bottom (sink).
The camera subsystem creates four video devices, each for a different purpose:
-
/dev/video2
is the capture device for the dump pipe. Through the dump pipe, you can read unprocessed video data. -
/dev/video3
is the capture device for the main pipe. The main pipe contains most of the processing capabilities of the system, such as scaling, frame skipping, de-Bayering, image correction, color conversion, etc. -
/dev/video4
is the capture device for the auxiliary pipe. This pipe can perform some limited processing, like scaling and color conversion. -
/dev/video5
is the capture device for statistical data about the image.
Preview a camera image using gstreamer
To get a camera preview:
-
Configure all the nodes in the main pipe for the desired resolution and format (in this example, raw video at 1280x720):
# media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:SBGGR8_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:SBGGR8_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_isp':1[fmt:RGB888_1X24/1280x720 field:none]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':1[fmt:YUYV8_2X8/1280x720]"
-
Use gstreamer to capture from videosink
/dev/video3
using the selected video format and redirect it to your display:# gst-launch-1.0 v4l2src device=/dev/video3 ! 'video/x-raw, format=YUY2, width=1280, height=720, framerate=(fraction)30/1' ! queue ! autovideosink
Take a picture with the camera using gstreamer
To take a picture:
-
Configure all the nodes in the main pipe for the desired resolution and format (in this example, raw video at 1280x720). Ensure the selected format is supported by the jpegenc plugin of gstreamer:
# media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:SBGGR8_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:SBGGR8_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_isp':1[fmt:RGB888_1X24/1280x720 field:none]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':1[fmt:YUYV8_2X8/1280x720]"
-
Use gstreamer to capture from videosink
/dev/video3
using the selected video format and encode it as a JPEG picture addingjpegenc
to the gstreamer pipe:# gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=1 ! 'video/x-raw, format=YUY2, width=1280, height=720, framerate=(fraction)30/1' ! jpegenc ! filesink location=grab-1280x720.jpeg
-
Display the captured image:
# weston-image grab-1280x720.jpeg
Record a video with the camera using gstreamer
To record a video:
-
Configure the camera sensor,
stm32_csi2host
anddcmipp_main_isp
nodes for the desired resolution and format (in this example, raw video at 1280x720). -
Downscale the frame size using the
dcmipp_main_postproc
node to avoid a penalization in frame rate produced by video processing latency (in this example, to 640x480). Ensure the selected format is supported by the avimux plugin of gstreamer:# media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:SBGGR8_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:SBGGR8_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_isp':1[fmt:RGB888_1X24/1280x720 field:none]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':0[compose:(0,0)/640x480]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':1[fmt:YUYV8_2X8/640x480]"
-
Use gstreamer to capture several frames from videosink
/dev/video3
using the selected video format and encode it as an AVI video addingavimux
to the gstreamer pipe (in this example, capture 150 frames at 30 fps for a 5 second video):# gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=150 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=(fraction)30/1' ! avimux ! filesink location=output.avi
-
Play back the recorded video:
# gst-play-1.0 output.avi
Advanced camera configurations
List allowed configurations for each subdevice
To capture video data from the camera, you must configure every subdevice in the pipe from top to bottom to manage the video data using certain format and frame size. Some subdevices are able to convert the video format on the fly and therefore use a different format on their input than on their output. You can rely on the graph to understand which subdevice should be used for each element of the pipe, and which pad corresponds to their input or output.
For example, list the video format of the first node of the graph (at the top), /dev/v4l-subdev6
, which corresponds to the OV5640 camera sensor:
# v4l2-ctl -d /dev/v4l-subdev6 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
0x4001: MEDIA_BUS_FMT_JPEG_1X8
0x2006: MEDIA_BUS_FMT_UYVY8_2X8
0x200f: MEDIA_BUS_FMT_UYVY8_1X16
0x2008: MEDIA_BUS_FMT_YUYV8_2X8
0x2011: MEDIA_BUS_FMT_YUYV8_1X16
0x1008: MEDIA_BUS_FMT_RGB565_2X8_LE
0x1007: MEDIA_BUS_FMT_RGB565_2X8_BE
0x3001: MEDIA_BUS_FMT_SBGGR8_1X8
0x3013: MEDIA_BUS_FMT_SGBRG8_1X8
0x3002: MEDIA_BUS_FMT_SGRBG8_1X8
0x3014: MEDIA_BUS_FMT_SRGGB8_1X8
Select one, for example MEDIA_BUS_FMT_RGB565_2X8_LE
, and list the available frame sizes for that format:
# v4l2-ctl -d /dev/v4l-subdev6 --list-subdev-framesizes pad=0,code=MEDIA_BUS_FMT_RGB565_2X8_LE
ioctl: VIDIOC_SUBDEV_ENUM_FRAME_SIZE (pad=0)
Size Range: 160x120 - 160x120
Size Range: 176x144 - 176x144
Size Range: 320x240 - 320x240
Size Range: 640x480 - 640x480
Size Range: 720x480 - 720x480
Size Range: 720x576 - 720x576
Size Range: 1024x768 - 1024x768
Size Range: 1280x720 - 1280x720
Size Range: 1920x1080 - 1920x1080
Size Range: 2592x1944 - 2592x1944
You can list the available format for the rest of the subdevices using similar commands.
For example, you can see the /dev/v4l-subdev2
node, which corresponds to the main image processor, accepts many possible video formats on its input (pad 0) but is only capable of streaming on two 24-bit video raw formats on its output (pad 1):
# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 0
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=0)
0x1008: MEDIA_BUS_FMT_RGB565_2X8_LE
0x2008: MEDIA_BUS_FMT_YUYV8_2X8
0x2006: MEDIA_BUS_FMT_UYVY8_2X8
0x2009: MEDIA_BUS_FMT_YVYU8_2X8
0x2007: MEDIA_BUS_FMT_VYUY8_2X8
0x2001: MEDIA_BUS_FMT_Y8_1X8
0x3001: MEDIA_BUS_FMT_SBGGR8_1X8
0x3013: MEDIA_BUS_FMT_SGBRG8_1X8
0x3002: MEDIA_BUS_FMT_SGRBG8_1X8
0x3014: MEDIA_BUS_FMT_SRGGB8_1X8
0x3007: MEDIA_BUS_FMT_SBGGR10_1X10
0x300e: MEDIA_BUS_FMT_SGBRG10_1X10
0x300a: MEDIA_BUS_FMT_SGRBG10_1X10
0x300f: MEDIA_BUS_FMT_SRGGB10_1X10
0x3008: MEDIA_BUS_FMT_SBGGR12_1X12
0x3010: MEDIA_BUS_FMT_SGBRG12_1X12
0x3011: MEDIA_BUS_FMT_SGRBG12_1X12
0x3012: MEDIA_BUS_FMT_SRGGB12_1X12
0x3019: MEDIA_BUS_FMT_SBGGR14_1X14
0x301a: MEDIA_BUS_FMT_SGBRG14_1X14
0x301b: MEDIA_BUS_FMT_SGRBG14_1X14
0x301c: MEDIA_BUS_FMT_SRGGB14_1X14
# v4l2-ctl -d /dev/v4l-subdev2 --list-subdev-mbus-codes 1
ioctl: VIDIOC_SUBDEV_ENUM_MBUS_CODE (pad=1)
0x100a: MEDIA_BUS_FMT_RGB888_1X24
0x2025: MEDIA_BUS_FMT_YUV8_1X24
Set configurations for a subdevice
You can use the media-ctl
tool to set configurations for any media entity in the graph:
# media-ctl -d platform:dcmipp --set-v4l2 "'<entity-name>':<pad>[<settings>]"
Configurable settings include:
-
Video format:
fmt:<format-code>/<frame-size>
-
Video field:
field:<field-name>
-
Video colorspace:
colorspace:<colorspace-name>
-
Crop processing:
crop:(<left>,<top>)/<frame-size>
-
Scale processing:
compose:(<left>,<top>)/<frame-size>
For example, to configure the camera sensor (which is associated with the ov5640 0-003c
media entity in the graph) to stream video with MEDIA_BUS_FMT_RGB565_2X8_LE
format and 1280x720
frame size, you must set the media entity output pad (0
) fmt
property with the following command:
# media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:RGB565_2X8_LE/1280x720]"
If, for example, later down the pipe, you want to downscale the video to a 640x480
frame size, configure the dcmipp_main_postproc
media entity input pad (0
) to perform a compose
operation to that size with the following command:
# media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_main_postproc':0[compose:(0,0)/640x480]"
Get and set additional controls
Some video subdevices have additional controls you can set.
Use the following command to get the list of available controls:
# v4l2-ctl -d <V4L2 subdevice> -L
Use the following command to set a control with a new value:
# v4l2-ctl -d <V4L2 subdevice> --set-ctrl <control-name>=<control-value>
For example, list the available controls of subdevice /dev/v4l-subdev6
, which corresponds to the OV5640 camera sensor:
# v4l2-ctl -d /dev/v4l-subdev6 -L
User Controls
contrast 0x00980901 (int) : min=0 max=255 step=1 default=0 value=0 flags=slider
saturation 0x00980902 (int) : min=0 max=255 step=1 default=64 value=64 flags=slider
hue 0x00980903 (int) : min=0 max=359 step=1 default=0 value=0 flags=slider
white_balance_automatic 0x0098090c (bool) : default=1 value=1 flags=update
red_balance 0x0098090e (int) : min=0 max=4095 step=1 default=0 value=0 flags=inactive, slider
blue_balance 0x0098090f (int) : min=0 max=4095 step=1 default=0 value=0 flags=inactive, slider
exposure 0x00980911 (int) : min=0 max=65535 step=1 default=0 value=562 flags=inactive, volatile
gain_automatic 0x00980912 (bool) : default=1 value=1 flags=update
gain 0x00980913 (int) : min=0 max=1023 step=1 default=0 value=36 flags=inactive, volatile
horizontal_flip 0x00980914 (bool) : default=0 value=0
vertical_flip 0x00980915 (bool) : default=0 value=0
power_line_frequency 0x00980918 (menu) : min=0 max=3 default=1 value=1 (50 Hz)
0: Disabled
1: 50 Hz
2: 60 Hz
3: Auto
Camera Controls
auto_exposure 0x009a0901 (menu) : min=0 max=1 default=0 value=0 (Auto Mode) flags=update
0: Auto Mode
1: Manual Mode
Image Processing Controls
link_frequency 0x009f0901 (intmenu): min=0 max=8 default=0 value=3 (126273600 0x786c840) flags=read-only
0: 63136800 (0x3c36420)
1: 83954880 (0x5010cc0)
2: 92145600 (0x57e07c0)
3: 126273600 (0x786c840)
4: 167909760 (0xa021980)
5: 184291200 (0xafc0f80)
6: 191116800 (0xb643600)
7: 335819520 (0x14043300)
8: 382233600 (0x16c86c00)
pixel_rate 0x009f0902 (int64) : min=0 max=2147483647 step=1 default=92145600 value=63136800 flags=read-only
test_pattern 0x009f0903 (menu) : min=0 max=4 default=0 value=0 (Disabled)
0: Disabled
1: Color bars
2: Color bars w/ rolling bar
3: Color squares
4: Color squares w/ rolling bar
Configure, for example, an horizontal flip of the image in the camera sensor. This is performed by the camera sensor itself and does not require later postprocessing:
# v4l2-ctl -d /dev/v4l-subdev6 --set-ctrl horizontal_flip=1
List allowed formats for each video device
Run the following v4l2-ctl
command to get the available video formats of the dump, main and aux video devices, which you will later use to capture the frames using v4l2-ctl
or gst-launch-1.0
tools:
# v4l2-ctl -d /dev/video2 --list-formats
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'RGBP' (16-bit RGB 5-6-5)
[1]: 'YUYV' (YUYV 4:2:2)
[2]: 'YVYU' (YVYU 4:2:2)
[3]: 'UYVY' (UYVY 4:2:2)
[4]: 'VYUY' (VYUY 4:2:2)
[5]: 'GREY' (8-bit Greyscale)
[6]: 'BA81' (8-bit Bayer BGBG/GRGR)
[7]: 'GBRG' (8-bit Bayer GBGB/RGRG)
[8]: 'GRBG' (8-bit Bayer GRGR/BGBG)
[9]: 'RGGB' (8-bit Bayer RGRG/GBGB)
[10]: 'JPEG' (JFIF JPEG, compressed)
# v4l2-ctl -d /dev/video3 --list-formats
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'RGBP' (16-bit RGB 5-6-5)
[1]: 'YUYV' (YUYV 4:2:2)
[2]: 'YVYU' (YVYU 4:2:2)
[3]: 'UYVY' (UYVY 4:2:2)
[4]: 'VYUY' (VYUY 4:2:2)
[5]: 'GREY' (8-bit Greyscale)
[6]: 'RGB3' (24-bit RGB 8-8-8)
[7]: 'BA24' (32-bit ARGB 8-8-8-8)
[8]: 'AYUV' (32-bit AYUV 8-8-8-8)
[9]: 'NV12' (Y/UV 4:2:0)
[10]: 'NV21' (Y/VU 4:2:0)
[11]: 'NV16' (Y/UV 4:2:2)
[12]: 'NV61' (Y/VU 4:2:2)
[13]: 'YU12' (Planar YUV 4:2:0)
[14]: 'YV12' (Planar YVU 4:2:0)
# v4l2-ctl -d /dev/video4 --list-formats
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'RGBP' (16-bit RGB 5-6-5)
[1]: 'YUYV' (YUYV 4:2:2)
[2]: 'YVYU' (YVYU 4:2:2)
[3]: 'UYVY' (UYVY 4:2:2)
[4]: 'VYUY' (VYUY 4:2:2)
[5]: 'GREY' (8-bit Greyscale)
[6]: 'RGB3' (24-bit RGB 8-8-8)
[7]: 'BA24' (32-bit ARGB 8-8-8-8)
[8]: 'AYUV' (32-bit AYUV 8-8-8-8)
These format names are only valid to perform a capture with the
|
Take a picture with the camera using v4l2-ctl
Once the camera is configured, use the following command to take a picture using v4l2-ctl
instead of gstreamer
:
# v4l2-ctl --device <video device> --set-fmt-video=width=<width>,height=<height>,pixelformat=<format> --stream-mmap --stream-count=1 --stream-to=<ouput file>
Capturing through the main pipe only supports raw video format which later has to be encoded into JPEG format by a different tool for visualization, for example, gstreamer
:
# gst-launch-1.0 filesrc location=<raw file> blocksize=<raw file size> ! "video/x-raw, format=(string)<raw format>, width=(int)<width>, height=(int)<height>, framerate=(fraction)<rate>" ! jpegenc ! filesink location=<JPEG file>
However, both the OV5640 camera sensor and the dump capture device support JPEG format as seen above.
To take a JPEG picture using v4l2-ctl
:
-
Configure all the nodes in the dump pipe for JPEG format and the desired resolution (in this example, 1280x720):
# media-ctl -d platform:dcmipp --set-v4l2 "'ov5640 0-003c':0[fmt:JPEG_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'stm32_csi2host.48020000.csi2hos':1[fmt:JPEG_1X8/1280x720]" # media-ctl -d platform:dcmipp --set-v4l2 "'dcmipp_dump_postproc':1[fmt:JPEG_1X8/1280x720]"
-
Use
v4l2-ctl
to capture from videosink/dev/video2
using the JPEG video format:# v4l2-ctl --device /dev/video2 --set-fmt-video=width=1280,height=720,pixelformat=JPEG --stream-mmap --stream-count=1 --stream-to=grab-1280x720.jpeg
-
Display the captured image:
# weston-image grab-1280x720.jpeg