Add a shield for DVP FPC 24-pins mt9m114 camera modules #72435

ngphibang · 2024-05-07T21:40:58Z

This PR adds a shield for DVP FPC 24-pins mt9m114 camera modules which are upported across NXP's i.MX RT10xx series.

It is split from #69810 to ease the review process.

It already got reviewed by @danieldegrasse.

boards/shields/huatian_mt9m114/huatian_mt9m114.overlay

decsny · 2024-05-13T16:43:53Z

boards/shields/huatian_mt9m114/huatian_mt9m114.overlay

+
+	port {
+		csi_ep_in: endpoint {
+			remote-endpoint = <&mt9m114_ep_out>;


how is this building, isn't it circular?

The use of port, endpoint and remote-endpoint concepts follows what is done in Linux, see video-interfaces.yaml, but there is currently no backend implementation yet in Zephyr.
But when it is implemented, it has to be buildable

AFAIK circular phandles are not supported by zephyr DT tooling, so this overlay seems invalid for zephyr

@decsny : I think it is misunderstood here. I know that Zephyr tool will check phandle dependencies at compile time and circular phandles are not allowed. But this is not the case. These are not phandles but "endpoints" and "remote-endpoints" as you can see as well, it actually builds.

It is not me who introduced these into Zephyr. In this PR, I just moved existed things from devicetree to an overlay. Please see this commit in this same PR.

In fact, "endpoints" and "remote-endpoints" concepts are adopted into Zephyr and it is not limited to video, it is used for display too, please see @MaureenHelm 's commit message here. The video APIs by @loicpoulain also uses it.

It is a bit out of scope of this PR but "port/endpoint" (circular dependency) is a good concept that needs to be implemented in the future.

okay, I actually don't get what's going on here, I will see what is going on with these overlays because maybe my understanding is wrong, because I would have thought none of these overlays can build

Yes, the overlay actually builds and the mimxrt1064_evk.dts also builds with these nodes defined like that before my commit.

The discussion about endpoint / circular dependency is good but deserves another topic I think.

This tiny / trivial PR just moves the camera nodes which are already defined in the mimxrt1064_evk.dts to a separate overlay because, as said in the commit message:

"The mt9m114 camera module is not always attached to this board. Camera stuffs should go into a seperate overlay / shield. Also, the CSI should be enabled only if there is a camera sensor attached to it."

@decsny : If you think "port/endpoint" concept should not be in Zephyr for the time being (because it is currently not implemented). We can submit a PR to remove them all from Zephyr (video API need to be changed as well because it uses endpoint, I think ?).

BTW, this PR simply moves the existing camera nodes from mimxrt1064_evk.dts to a separate overlay ...

AFAIK circular phandles are not supported by zephyr DT tooling, so this overlay seems invalid for zephyr

This was further characterized recently:
#57708

I tried to go to the bottom of the implication of a remote-endpoint in Zephyr here:
#72311

AFAIK circular phandles are not supported by zephyr DT tooling, so this overlay seems invalid for zephyr

This was further characterized recently: #57708

I tried to go to the bottom of the implication of a remote-endpoint in Zephyr here: #72311

thanks for this information, it's exactly what I needed I think, I will look into this further soon to see what is going on here

decsny

overlay appears invalid for zephyr DT

dts/arm/nxp/nxp_rt10xx.dtsi

decsny

approving because @ngphibang accurately pointed out that the part of the PR that I have a problem with is actually only a move, not new, and it's correct these DT items should be in a shield

mmahadevan108 · 2024-05-20T21:08:09Z

@kartben, can you help review this PR.

danieldegrasse · 2024-05-21T18:17:05Z

BTW, the pinouts are common for RT10xx (1050, 1060, 1064 EVK, etc.) and the mt9m114 can be used across these NXP 10XX boards. So, I think we could always define a shield here (with nxp prefixes in the shield) ? But it seems that @kartben wants that there are no specific nxp stuffs in a shield (If I understand well) ?

Yeah, I think that's accurate. Sort of a tricky problem. Perhaps we prefix the node name with the manufacturer of the module?

kartben · 2024-05-22T14:40:39Z

@ngphibang @danieldegrasse I think what's really confusing is that the shield name, and README, refer to it very generically as a "MT9M114 Camera Module" so I am naively (and it looks, wrongly?) assuming this is some kind of generic module for the MT9M114 CMOS chip exposing an FPC connector. If it is in fact a module built for NXP boards specifically, then I would like to ask you to please clarify the naming and README so that it is clearer what actual piece of hardware this is referring to.

btw @erwango might be able to chime in here and provide some insights

ngphibang · 2024-05-22T17:31:49Z

@kartben After discussing internally and searching on the internet, it appears that this DVP (Digital Video Port, aka parallel) 24-pins FPC camera interface is generic and not specific to NXP. We have at least another ov7725 camera module that can be plugged into RT10xx boards as a replacement for the MT9M114 module.

I also found some camera modules sold on Aliexpress and other pages which have the same form factor and pin function order with ours :

https://www.supertekmodule.com/wp-content/uploads/2018/11/ST-OV7725DVP-Datasheet-V2.1.pdf
https://fr.aliexpress.com/item/1005001802973334.html?gatewayAdapt=glo2fra

So, I would rename the shield as : dvp_fpc24 with dvp_fpc24_i2c and dvp_fpc24_interface nodes inside it.

The naming could be ok now but I have a technical issue that I don't know how to resolve it. The sensor node is specific (it could be mt9m114 or ov7725, etc.), so I have to put them into boards/shields/huatian_mt9m114/boards/mimxrt1064_evk.overlay as your suggestion :

&dvp_fpc24_i2c {
	cam_module: mt9m114@48 {
		compatible = "aptina,mt9m114";
		reg = <0x48>;
	};
};

And in the dvp_fpc24 shield overlay, we have

&cam_module {
	port {
		cam_ep_out: endpoint {
			remote-endpoint = <&dfi_ep_in>;
		};
	};
};

&dvp_fpc24_interface {
	status = "okay";

	port {
		dfi_ep_in: endpoint {
			remote-endpoint = <&cam_ep_out>;
		};
	};
};

But the problem is the board overlay is parsed AFTER the shield overlay so we have an error "undefined cam_module node label" here.

Do you know if there is a solution for this ? i.e., something like forward declaration node ... ?
@danieldegrasse @erwango

Thanks

ngphibang · 2024-05-23T13:09:52Z

Reworked the PR according to the above comments. The "port / endpoint" part is common and it's better if it is put in dvp_fpc24.overlay rather than a board overlay but I don't know how resolve the issue mentionned in my previous comment.

josuah · 2024-05-23T13:24:43Z

I found another board that seems to use that pinout too but a hirose connector instead:
https://www1.futureelectronics.com/doc/IWAVE%20SYSTEMS%20TECHNOLOGIES/G15DDevelopmentkitDataSheet.pdf#page=124

2.9.4 CMOS Camera Connector
The Qseven Generic Carrier Board supports 24pin CMOS Camera connector (J51) for CMOS camera interface. This connector supports 8bit CMOS camera interface signals and connected to CSI port2 pins of Expansion connector2 through 2.8V to 3.3V Voltage translator. External reference clock for camera is provided by using on-board 26MHz Oscillator. This Camera connector (J51) is physically located at the bottom of the board as shown below.

Compatible Camera:

Part Number: CN003VEF2052

Description: VGA lens of 640x480 pixel CMOS camera based on OV7725 sensor

Manufacturer Name: Global Optics Limited

Global Optics CMOS Camera Web link: http://www.globaloptics.cn/

danieldegrasse · 2024-05-28T21:47:40Z

The naming could be ok now but I have a technical issue that I don't know how to resolve it. The sensor node is specific (it could be mt9m114 or ov7725, etc.), so I have to put them into boards/shields/huatian_mt9m114/boards/mimxrt1064_evk.overlay as your suggestion :

I'm confused why we would have a board overlay for this- it seems like the generic connector is the DVP FPC 24 pin interface, and the specific shield is the MT9M114 sensor module. Given this, I think we should define the DVP FPC 24 pin interface at the board level. This is analogous to Arduino shields, where the Arduino interface is defined at the board level (IE in the board devicetree, not as an overlay), then the shield references the nodes defined at the board level (which have standard names)

In my mind, a module with the OV7725 sensor should be a different shield definition

The mt9m114 camera module is not always attached to this board. Camera stuffs should go into a seperate overlay / shield. Also, the CSI should be enabled only if there is a camera sensor attached to it. Signed-off-by: Phi Bang Nguyen <[email protected]>

danieldegrasse · 2024-05-29T14:38:26Z

boards/shields/dvp_fpc24_mt9m114/boards/mimxrt1064_evk.overlay

Why is this board overlay needed? Couldn't this be part of the shield definition itself? I'm unclear what makes this RT1064 specific

The "source" property is defined in the DT binding of the CSI which is NXP specific. @kartben said that it must be in a board overlay because it's not generic. Other vendors can use different names.

Ah, this makes sense- in that case, no worries. We should keep this as is. May be worth considering (as a future improvement) if we can standardize the source property name, since CSI interfaces to camera sensors are somewhat generic IP. We've done this for a lot of the LCD controller timing properties, which allows us to avoid board specific overlays in many cases for this shields

danieldegrasse · 2024-05-29T14:39:35Z

dts/arm/nxp/nxp_rt10xx.dtsi

@@ -1160,3 +1160,5 @@
 	 */
 	status = "disabled";
 };
+
+dvp_fpc24_interface: &csi {};


Probably want to put this at the board level- what this line effectively says is "the CSI is broken out to a 24 pin dvp FPC interface" which is only true for the RT10xx EVK boards, not for all RT10xx SOCs

Okay ... so nxp_rt10xx is for all RT10xx SoCs not only EVK boards ... so there is not a "common" place to put common things for RT10xx (and RT11xx) EVKs / EVKB ? As we know that CSI is a common IP for those boards, it is a bit "inconvenient" to duplicate code for each board. But well, I will do as your suggestion then.

No, there isn't unfortunately- although we do duplicate a lot across our RT10xx EVKs, Zephyr does not have any infrastructure I'm aware of for sharing a DTS file describing common board properties across multiple boards

Add a shield for DVP FPC 24-pins mt9m114 camera modules. These camera modules are supported on i.MX RT10xx series, for example. Signed-off-by: Phi Bang Nguyen <[email protected]>

ngphibang · 2024-05-29T15:18:05Z

@danieldegrasse Thanks for the insight ! You are right. But it's a pity that we cannot generalize the shield for other camera modules which use the same connector inteface e.g. ov7725.
I made changes as your suggestion.
Honestly, the "shield" concept with all things about "generic / specific naming" was a bit tricky for me to understand and to do it correctly (as technically, what we need is just a reusable overlay and I didn't see much differences between different versions of the PR though). I hope it is OK this time.

dleach02 · 2024-05-31T14:41:44Z

@kartben can you return to this PR please

ngphibang requested review from mmahadevan108 and dleach02 as code owners May 7, 2024 21:40

zephyrbot added platform: NXP NXP area: Shields Shields (add-on boards) labels May 7, 2024

zephyrbot requested review from avisconti, danieldegrasse, decsny, DerekSnell, EmilioCBen, erwango, jfischer-no, kartben and yvanderv May 7, 2024 21:41

zephyrbot assigned kartben May 7, 2024

danieldegrasse reviewed May 9, 2024

View reviewed changes

boards/shields/huatian_mt9m114/huatian_mt9m114.overlay Outdated Show resolved Hide resolved

ngphibang force-pushed the add_mt9m114_shield branch 2 times, most recently from b29f2e0 to 3858f01 Compare May 12, 2024 06:35

DerekSnell removed their request for review May 13, 2024 14:06

ngphibang mentioned this pull request May 13, 2024

Add mipi csi2 rx driver #71859

Merged

decsny reviewed May 13, 2024

View reviewed changes

decsny requested changes May 13, 2024

View reviewed changes

ngphibang requested a review from decsny May 13, 2024 19:10

mmahadevan108 reviewed May 14, 2024

View reviewed changes

dts/arm/nxp/nxp_rt10xx.dtsi Outdated Show resolved Hide resolved

ngphibang force-pushed the add_mt9m114_shield branch from 3858f01 to b4ba65d Compare May 16, 2024 08:40

decsny previously approved these changes May 17, 2024

View reviewed changes

dleach02 previously approved these changes May 17, 2024

View reviewed changes

loicpoulain self-requested a review May 20, 2024 16:55

loicpoulain previously approved these changes May 20, 2024

View reviewed changes

mmahadevan108 previously approved these changes May 20, 2024

View reviewed changes

ngphibang force-pushed the add_mt9m114_shield branch from 0f05313 to ea181db Compare May 23, 2024 13:03

zephyrbot requested a review from butok May 23, 2024 13:03

ngphibang changed the title ~~Add mt9m114 shield~~ Add a shield for DVP FPC 24-pins camera connectors May 27, 2024

ngphibang changed the title ~~Add a shield for DVP FPC 24-pins camera connectors~~ Add a shield for DVP FPC 24-pins camera modules May 27, 2024

ngphibang mentioned this pull request May 27, 2024

Update and improve video capture sample #72633

Merged

ngphibang force-pushed the add_mt9m114_shield branch 2 times, most recently from c93e305 to ea02aa5 Compare May 29, 2024 14:36

danieldegrasse reviewed May 29, 2024

View reviewed changes

ngphibang changed the title ~~Add a shield for DVP FPC 24-pins camera modules~~ Add a shield for DVP FPC 24-pins mt9m114 camera modules May 29, 2024

danieldegrasse reviewed May 29, 2024

View reviewed changes

boards: shields: Add a shield for DVP FPC-24 mt9m114 camera modules

4f39b49

Add a shield for DVP FPC 24-pins mt9m114 camera modules. These camera modules are supported on i.MX RT10xx series, for example. Signed-off-by: Phi Bang Nguyen <[email protected]>

ngphibang force-pushed the add_mt9m114_shield branch from ea02aa5 to 4f39b49 Compare May 29, 2024 15:08

danieldegrasse approved these changes May 29, 2024

View reviewed changes

ngphibang requested a review from dleach02 May 30, 2024 13:59

dleach02 approved these changes May 31, 2024

View reviewed changes

loicpoulain approved these changes May 31, 2024

View reviewed changes

kartben approved these changes May 31, 2024

View reviewed changes

henrikbrixandersen merged commit c23132a into zephyrproject-rtos:main Jun 1, 2024
22 checks passed

ycsin mentioned this pull request Jun 1, 2024

samples: video: tcpserversink: fix build regression #73604

Merged

ngphibang mentioned this pull request Jun 3, 2024

Update tcpserversink sample #73665

Merged

ngphibang mentioned this pull request Feb 5, 2025

boards: shields: add suport for st_b_cams_omv_mb1683 #76143

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a shield for DVP FPC 24-pins mt9m114 camera modules #72435

Add a shield for DVP FPC 24-pins mt9m114 camera modules #72435

ngphibang commented May 7, 2024 •

edited

Loading

decsny May 13, 2024

ngphibang May 13, 2024 •

edited

Loading

decsny May 13, 2024

ngphibang May 13, 2024 •

edited

Loading

decsny May 13, 2024

ngphibang May 13, 2024 •

edited

Loading

ngphibang May 14, 2024 •

edited

Loading

josuah May 15, 2024 •

edited

Loading

decsny May 17, 2024

decsny left a comment

decsny left a comment •

edited

Loading

mmahadevan108 commented May 20, 2024

danieldegrasse commented May 21, 2024

kartben commented May 22, 2024

ngphibang commented May 22, 2024 •

edited

Loading

ngphibang commented May 23, 2024 •

edited

Loading

josuah commented May 23, 2024

danieldegrasse commented May 28, 2024

danieldegrasse May 29, 2024

ngphibang May 29, 2024 •

edited

Loading

danieldegrasse May 29, 2024

danieldegrasse May 29, 2024

ngphibang May 29, 2024 •

edited

Loading

danieldegrasse May 29, 2024

ngphibang commented May 29, 2024 •

edited

Loading

dleach02 commented May 31, 2024

Add a shield for DVP FPC 24-pins mt9m114 camera modules #72435

Add a shield for DVP FPC 24-pins mt9m114 camera modules #72435

Conversation

ngphibang commented May 7, 2024 • edited Loading

Choose a reason for hiding this comment

ngphibang May 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngphibang May 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngphibang May 13, 2024 • edited Loading

Choose a reason for hiding this comment

ngphibang May 14, 2024 • edited Loading

Choose a reason for hiding this comment

josuah May 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

decsny left a comment

Choose a reason for hiding this comment

decsny left a comment • edited Loading

Choose a reason for hiding this comment

mmahadevan108 commented May 20, 2024

danieldegrasse commented May 21, 2024

kartben commented May 22, 2024

ngphibang commented May 22, 2024 • edited Loading

ngphibang commented May 23, 2024 • edited Loading

josuah commented May 23, 2024

danieldegrasse commented May 28, 2024

Choose a reason for hiding this comment

ngphibang May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngphibang May 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ngphibang commented May 29, 2024 • edited Loading

dleach02 commented May 31, 2024

ngphibang commented May 7, 2024 •

edited

Loading

ngphibang May 13, 2024 •

edited

Loading

ngphibang May 13, 2024 •

edited

Loading

ngphibang May 13, 2024 •

edited

Loading

ngphibang May 14, 2024 •

edited

Loading

josuah May 15, 2024 •

edited

Loading

decsny left a comment •

edited

Loading

ngphibang commented May 22, 2024 •

edited

Loading

ngphibang commented May 23, 2024 •

edited

Loading

ngphibang May 29, 2024 •

edited

Loading

ngphibang May 29, 2024 •

edited

Loading

ngphibang commented May 29, 2024 •

edited

Loading