IMX477 Improvements - 6.12

[6by9]

6.12 version of #6208

Only compile tested.

[ameybarapatre]

Hi, after fixing my noob setup i can confirm its working, meanwhile are we expecting to merge this soon?

[6by9]

Rebased which will retrigger the CI builds.

My comment from #6208 (comment) that 2028x1080 and 2028x1520 modes with 10 bit readout give totally white frames needs to be resolved before contemplating merging.
Unless you're saying that you've tested those modes and that they work.

[Carnarts]

I have done some testing with this, yes the 10bit 2K modes are white frames
12 bit 2K modes seem to work ok but haven't tested those much.
The 4056 x 3040 10 & 12 bit uncropped modes at 24fps drop frames when recording in these modes, it's not my drive speed being the problem so I assume there's a bottleneck in the stream.
4056 x 2160 10 and 12 bit modes seem to work well though. Hope these can be resolved as great to see this sensor support 4K at 24fps :)

[6by9]

I've finally had a few minutes to relook at this one.

I've dropped the patch that added 10 and 12 bit readout to all modes. That can be reintroduced in due course.

What I have done is reworked setting the link frequency so that it will accept any value that can be achieved (basically a multiple of 3MHz).
The minimum line length is then computed rather than being tabulated. That means that we can run at 750MHz to remain within the specs of RP1, but the user can choose to overclock just by setting it higher (I've not tested that).

At 750MHz we get

Available cameras
-----------------
0 : imx477 [4056x3040 12-bit RGGB] (/base/axi/pcie@1000120000/rp1/i2c@80000/imx477@1a)
    Modes: 'SRGGB10_CSI2P' : 1332x990 [169.87 fps - (696, 528)/2664x1980 crop]
           'SRGGB12_CSI2P' : 2028x1080 [105.95 fps - (0, 440)/4056x2160 crop]
                             2028x1520 [75.36 fps - (0, 0)/4056x3040 crop]
                             4056x2160 [27.48 fps - (0, 440)/4056x3040 crop]
                             4056x3040 [19.53 fps - (0, 0)/4056x3040 crop]

It's not quite right as the base 450MHz drops performance on the 1332x990 mode. My 500usecs for LP<>HS is probably too generous, and I'm not accounting for it only being a 10bit mode.

Available cameras
-----------------
0 : imx477 [4056x3040 12-bit RGGB] (/base/axi/pcie@1000120000/rp1/i2c@80000/imx477@1a)
    Modes: 'SRGGB10_CSI2P' : 1332x990 [106.18 fps - (696, 528)/2664x1980 crop]
           'SRGGB12_CSI2P' : 2028x1080 [65.36 fps - (0, 440)/4056x2160 crop]
                             2028x1520 [46.49 fps - (0, 0)/4056x3040 crop]
                             4056x2160 [16.72 fps - (0, 440)/4056x3040 crop]
                             4056x3040 [11.89 fps - (0, 0)/4056x3040 crop]

I haven't seen any image corruption issues, but it hasn't had masses of testing. It would be appreciated if those interested would test it and feed back any issues.

[sandyol55]

Some testing ** (well beyond the suggested max of 750 MHz) showed only occasional frame dropping in any of the modes up to the point where I stopped at 996 Mhz. At that frequency the full frame mode was exceeding 25 fps. However somewhere between 924 and 948 MHz both of the 2x2 binned modes started outputting solid colour frames. Magenta this time rather than white as in the previous 10 bit issue.
** Can post the results if they're likely to help provide any clues, it is just a list of 'advertised' frame rates at each mode, as the link frequency is increased, versus the achieved result in that mode - which I guess is giving an indication of whether there were dropped frames or not?
Edit
PS This was with a 'stock clock' 4Gb Pi5 and with camera connected via a 150mm FFC.

[6by9]

Thanks for testing.

As this is a sensor driver, I'm not that interested in anything above 750MHz as then you're exceeding the Pi's capabilities rather than the sensor's.
Frame drops and corrupt frames are again most likely to be the Pi rather than the sensor. Ideally I'd like to get CSI2 level debug for error rates etc, but I don't believe I have an easy way to do that.

Pi5's libcamera pipeline handler already knows that RP1 CFE has a limit on pixel rate (250MPix/s IIRC), so I'm expecting that will be automatically extending the HBLANK control value once you get to a point where the link frequency can exceed the limit, but you won't get any increase in output framerate from libcamera on Pi5.

[schoolpost]

Thanks for testing.

As this is a sensor driver, I'm not that interested in anything above 750MHz as then you're exceeding the Pi's capabilities rather than the sensor's. Frame drops and corrupt frames are again most likely to be the Pi rather than the sensor. Ideally I'd like to get CSI2 level debug for error rates etc, but I don't believe I have an easy way to do that.

Pi5's libcamera pipeline handler already knows that RP1 CFE has a limit on pixel rate (250MPix/s IIRC), so I'm expecting that will be automatically extending the HBLANK control value once you get to a point where the link frequency can exceed the limit, but you won't get any increase in output framerate from libcamera on Pi5.

There should be more headroom than 250Mpix/s, ~380Mpix/s is the standard / supported pixel rate. Even at the higher link frequencies tested by @sandyol55, I never saw any frame size/frame rate combo that exceeds 300Mpix/s.
https://github.com/raspberrypi/libcamera/blob/bfd68f786964636b09f8122e6c09c230367390e7/src/ipa/rpi/controller/controller.cpp#L64

Here is 945Mhz

Mode selection for 4056:3040:12:U(24)
    SRGGB10_CSI2P,1332x990/208.594 - Score: 10552.2
    SRGGB12_CSI2P,2028x1080/131.165 - Score: 8179.84
    SRGGB12_CSI2P,2028x1520/93.301 - Score: 7096
    SRGGB12_CSI2P,4056x2160/34.303 - Score: 1963.84
    SRGGB12_CSI2P,4056x3040/24.3861 - Score: 0

So there must be something else at play that is causing those white frames?

It's just so strange that it would be isolated to the 2x2 binned modes, even at low frame rates, because the full readout/4k modes operate just fine from what I've seen; and those ones do in fact operate at pixel rates in many cases approaching 300Mpix/s, which are still well below the 380Mpix/s limit.

[Carnarts]

@schoolpost Are you managing to record ok the uncropped 4056x3040/24 modes? Curious how you're managing that if so. Testing with Cinemate, the preview looks good but as soon as I record the preview drops frames making these modes unusable for me, it isn't dropping frames on the disk but at the camera.