Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Camera support #1

Open
akobel opened this issue Feb 27, 2022 · 189 comments
Open

Camera support #1

akobel opened this issue Feb 27, 2022 · 189 comments

Comments

@akobel
Copy link
Owner

akobel commented Feb 27, 2022

Continued from linux-surface/linux-surface#91, where similar work has been done for the camera of Surface Go 2; let's hope that migrating that effort to the X1 tablet is feasible...

(more precisely, coming over from this comment)

@markum
Copy link

markum commented Feb 27, 2022

Thanks for starting this. I assume I not need to create the DSDT for the same device and can cease my efforts go get it?

@akobel
Copy link
Owner Author

akobel commented Feb 27, 2022

Correct, I think. But on Linux, getting them is fairly easy; it could be worth to double-check whether they are identical:
acpidump -b from acpica and iasl -d *.dat to disassemble.

@djrscally
Copy link

djrscally commented Feb 27, 2022

Howdy.

OK, you guys have an annoying DSDT like the Surface Go line, where most of the settings are defined by values read through OpRegions instead of hardcoded. Unfortunately that means the DSDT is not readable, and the right values need to be discovered at runtime. Fortunately, one of the other devs already wrote something to let you do that.

Can you please build and load this module, and share the output (which will be written to dmesg)?

In terms of what will need to be done, it depends on the output of that module. If it says the PMICs are discrete type we just need to make the sensor drivers work. If they're the TPS68470 type we'll need to do some reverse engineering on Windows to sniff the right voltage settings for the PMIC to make sure we don't damage your devices. I can do the sensor drivers but you guys would have to test for me. The reverse engineering you'll need to do, but I can talk you through it if you need.

@akobel
Copy link
Owner Author

akobel commented Feb 27, 2022

Thanks @djrscally already! Testing and some amount of debugging I can offer, but my expertise in this are of reverse engineering is pretty low; not sure about @markum.

For the dumps: sure thing, see https://github.com/akobel/linux-thinkpad-x1-tablet/tree/main/x1-tablet-gen2/intel_ipu_dump
Note: this is on the Arch zen kernel (near-vanilla with only few tweaks for desktop usage); I can reproduce on linux-surface if necessary.

@markum
Copy link

markum commented Feb 27, 2022

Unfortunately I am not very qualified in this field. But I can ask a friend to help me.

@djrscally
Copy link

djrscally commented Feb 28, 2022 via email

@akobel
Copy link
Owner Author

akobel commented Feb 28, 2022

This requires you to boot windows, does either of you have it set to dual boot by any chance?

Yep, can do. (Note to myself: next time before you upgrade to Win11, make sure you have a backup to revert, not just because it's so horribly slow.)

On the sensor drivers front the OV2470 is well supported already and might only need a one line change to be useable. The OV8858 has an atomisp driver which would be adaptable with some effort.

Sounds great! I would have expected problems/significant effort for the 8858; if that could be made work, even better.
For the time being, anything is an improvement, and it seems that you have the steps to initiate the cameras freshly in your head... :-)

@djrscally
Copy link

Yep, can do.

Excellent. How do you feel trying the reverse engineering steps then? It looks complicated but isn't really so bad - I can help get the DSDT entry written up and from there it's just getting the compiler installed and running a bunch of commands in the right order. Fair warning though: there's a very slight risk of messing the Windows install up a bit. I've never done it, but it is apparently a potential problem.

Sounds great! I would have expected problems/significant effort for the 8858; if that could be made work, even better.
For the time being, anything is an improvement, and it seems that you have the steps to initiate the cameras freshly in your head... :-)

Yup still actively tackling the surface ones so we're kinda used to this now.

@akobel
Copy link
Owner Author

akobel commented Mar 2, 2022

I'm struggling with the very first steps already, unfortunately.

First, on Linux, cause that's what I have currently booted: iasl is unable to process its own output. iasl dsdt.dsl (see here) gives the following output: iasl.log
iasl *.dsl doesn't help, either.

I did not investigate the error itself yet, but will try to do so. Problem report on Windows will follow after a reboot.

@akobel
Copy link
Owner Author

akobel commented Mar 2, 2022

Situation on Windows:

asl.exe /tab=DSDT produces an DSDT.ASL, but with the following error message (see asl-read-DSDT.log):

asl_ERR: UnAsmSuperName: invalid SuperName - 0x0a

asl.exe DSDT.ASL afterwards complains about some invalid names with funny characters (encoding issue?), but then seems to find an actual error: see asl-compile-DSDT.log...

@djrscally
Copy link

djrscally commented Mar 2, 2022

I'm struggling with the very first steps already, unfortunately.

First, on Linux, cause that's what I have currently booted: iasl is unable to process its own output. iasl dsdt.dsl (see here) gives the following output: iasl.log iasl *.dsl doesn't help, either.

I did not investigate the error itself yet, but will try to do so. Problem report on Windows will follow after a reboot.

I never understand how they manage to include a .dsl that won't compile. The windows tool is far worse than iasl at this; I actually make the edits in Linux, compile using iasl and then simply transfer the compiled .aml to windows to load.

Anyway; you have a ton of noise there, all that matters are the errors. It's complaining about this section:


    Name (SS1, 0x00)
    Name (SS2, 0x00)
    Name (SS3, One)
    One
    Name (SS4, One)
    One
    OperationRegion (GNVS, SystemMemory, 0xBFF4F000, 0x072C)
    Field (GNVS, AnyAcc, Lock, Preserve

Just CTRL-F for SS4 to find that bit. Delete those standalone "One" after SS3 and SS4, and subsequent to that it should compile (at least it is doing for me)

EDIT: My advice: stick to iasl for compiling the dsdt and just use asl on Windows to load it. The MS tool seems much worse than Intel's

@akobel
Copy link
Owner Author

akobel commented Mar 2, 2022

Okay, that helps. Compilation works now, trying to progress from here.
One question: the DSDT needs to be loaded on Windows; so I assume it's not persistently flashed to the device, but only temporary until next power cycle, right?

@djrscally
Copy link

djrscally commented Mar 2, 2022

Okay, that helps. Compilation works now, trying to progress from here. One question: the DSDT needs to be loaded on Windows; so I assume it's not persistently flashed to the device, but only temporary until next power cycle, right?

No that's not right; it's stored in the registry and lasts until it's replaced with a new version or you go and clear out the registry keys (which will make it restore to default). You need to increase the version number each time you load a new DSDT on Windows, or it will ignore the new file:

DefinitionBlock ("", "DSDT", 2, "LENOVO", "SKL     ", 0x00000000)

The last field in the definition block is the version number

@akobel
Copy link
Owner Author

akobel commented Mar 2, 2022

Hehe, okay, makes sense. And that's soothing, cause...

IMG_20220302_121609

Will set me back a bit, but I'll recover. :)

@djrscally
Copy link

djrscally commented Mar 2, 2022

Ah-ha! Never mind - didn't want that OS anyway; here's recovery steps: https://github.com/bentiss/SimplePeripheralBusProbe#what-if-the-dsdt-is-wrong-and-i-get-the-blue-screen-acpi_bios_error-at-boot

@akobel
Copy link
Owner Author

akobel commented Mar 2, 2022

Hm, the good news is that the recovery works like a charm. The bad news is that even without any modification, I get the same message.
That is, just removing the two One lines, incrementing the version number, iasl dsdt.dsl and installing the patched aml via asl.exe /loadtable...

@djrscally
Copy link

Hm, the good news is that the recovery works like a charm. The bad news is that even without any modification, I get the same message. That is, just removing the two One lines, incrementing the version number, iasl dsdt.dsl and installing the patched aml via asl.exe /loadtable...

Ah; that is indeed suboptimal yeah. Maybe try the whole Windows asl compiling thing after all and see if that does a better job...

The alternative is scouring the internet for leaked schematics that will give us the connections and voltage settings instead.

@akobel
Copy link
Owner Author

akobel commented Mar 2, 2022

Ah; that is indeed suboptimal yeah. Maybe try the whole Windows asl compiling thing after all and see if that does a better job...

Will do. But ...

The alternative is scouring the internet for leaked schematics that will give us the connections and voltage settings instead.

... what exactly are we after here? Just voltages, or something else, too?
For the OV8858, I found a datasheet which might help? - From page 23:

  • AVDD is 2.6~3.0V for sensor analog power (clean), 2.8V is recommended
  • DOVDD is 1.7~3.0V for sensor digital IO power (clean). 1.8V is recommended
  • DVDD should be 1.27V for internal regulator. If using external, prefer 1.11~1.30V for DVDD

A datasheet for the TPS68470 is also available here.

Unfortunately, no luck searching for OV2740 apart from a product brief yet, and this would likely be the most interesting one for a start...

@djrscally
Copy link

... what exactly are we after here? Just voltages, or something else, too? For the OV8858, I found a datasheet which might help? - From page 23:

* AVDD is 2.6~3.0V for sensor analog power (clean), 2.8V is recommended

* DOVDD is 1.7~3.0V for sensor digital IO power (clean). 1.8V is recommended

* DVDD should be 1.27V for internal regulator. If using external, prefer 1.11~1.30V for DVDD

A datasheet for the TPS68470 is also available here.

The datasheets don't help us here. The TPS68470 has 10 regulator lines, any 3 of which could be configured to supply anything from .875V to over 5V IIRC, and we have no idea which line supplies which voltage to which sensor input. Additionally there will almost certainly be some GPIO's that need to be toggled to send power on / reset signals to the sensor, and they probably come from the TPS68470 too - it has ten of those too so we need to figure out which drive which sensor.

So the datasheets are helpful (I have the TPS68470 one, the ov8858 one we'll need to get a driver running) but we need either the reverse engineering or schematics too.

Unfortunately, no luck searching for OV2740 apart from a product brief yet, and this would likely be the most interesting one for a start...

Don't need that one anyway as the driver that's already upstream looks fine to me, will need one or two very minor edits but nothing that we'd need the datasheet for.

@akobel
Copy link
Owner Author

akobel commented Mar 3, 2022

First achievements: Loading an AML compiled with iasl version 20160527 worked better, apparently. Here is a precompiled binary distribution for Windows, too.

> ./I2cTestTool.exe -list
  FriendlyName DeviceId
          I2C2 \\?\ACPI#MSFT8000#1#{a11ee3c6-8421-4202-a3e7-b91ff90188e4}\I2C2

I'd assume I2C2 is the correct bus, since the DSDT has

    Scope (\_SB.PCI0.I2C2)
    {
        Device (PMIC)
        {
            Name (_ADR, Zero)  // _ADR: Address
            Name (_HID, "INT3472")  // _HID: Hardware ID
            Name (_CID, "INT3472")  // _CID: Compatible ID
            Name (_DDN, "PMIC-CRDG2")  // _DDN: DOS Device Name
            Name (_UID, "0")  // _UID: Unique ID
            Name (_PLD, Package (0x01)  // _PLD: Physical Location of Device

and this should be the camera power management module, right?

@djrscally
Copy link

djrscally commented Mar 3, 2022

Nice one, well done. That's the PMIC yes; the entry gives its I2C address further down:

            Method (_CRS, 0, NotSerialized)  // _CRS: Current Resource Settings
            {
                Name (SBUF, ResourceTemplate ()
                {
                    I2cSerialBusV2 (0x004C, ControllerInitiated, 0x00061A80,
                        AddressingMode7Bit, "\\_SB.PCI0.I2C2",
                        0x00, ResourceConsumer, , Exclusive,
                        )
                })
                Return (SBUF) /* \_SB_.PCI0.I2C2.PMIC._CRS.SBUF */
            }

So you should be able to connect with I2CTestTool 0x4c I2C2, and then subsequently communicate by running writeread {ff} 1 which means "write the byte 0xff and then read 1 byte". The first byte you read sets the PMICs pointer to that address, 0xff being the register containing the Chip's ID. The output should say "21".

edit:

assuming the above works, best thing to do is simply dump out all the registers for the PMIC three times:

  1. On boot, with no cameras running
  2. With the front camera running
  3. With the back camera running

That combination will allow us to identify which bits are being set for each camera. The PMIC does a lot of stuff actually; the regulators of course but also provides the clock for the sensors, GPIO lines and at least on the Surface Go2 drives the privacy and IR LEDs too, so there's probably a lot to learn.

You can literally just do writeread {00} 256 and dump the whole chip in one go.

@akobel
Copy link
Owner Author

akobel commented Mar 4, 2022

Strange. On 0x4c I only get "The transfer failed for an unknown reason". On 0x4d however, I can communicate, and I see reactions depending on whether the camera is connected, see below.
Note: writeread doesn't work for me, but write followed by read does.

Dumps:

No camera:

00000000:  00  20  02  00  00  00  01  03    02  11  aa  20  01  d0  00  00
00000010:  05  00  00  00  01  08  01  08    01  08  8a  08  8a  08  c2  08
00000020:  c2  08  00  00  00  00  00  00    00  00  0a  00  00  00  00  00
00000030:  07  38  00  00  00  00  00  00    00  00  00  00  6d  13  34  34
00000040:  34  6d  0c  00  00  00  00  00    04  00  00  00  00  00  00  00
00000050:  00  00  00  00  00  00  00  00    00  00  00  00  00  00  00  00
*
000000f0:  00  00  00  00  00  00  00  00    00  00  00  00  00  00  00  21

Front camera:

00000000:  00  20 [d2] 00  00  00  01  03    02  11  aa  20  01 [d7] 00 [0a]
00000010:  05  00  00  00  01  08  01  08    01  08  8a  08  8a  08  c2  08
00000020:  c2  08  00  00  00  00  00 [70]  [c0] 00  0a  00  00  00  00  00
00000030:  07  38  00  00  00  00  00  00    00  00  00  00  6d  13  34  34
00000040:  34  6d  0c [03] 00 [01][01] 00    04  00  00  00  00  00  00  00
00000050:  00  00  00  00  00  00  00  00    00  00  00  00  00  00  00  00
*
000000f0:  00  00  00  00  00  00  00  00    00  00  00  00  00  00  00  21

Rear camera:

00000000:  00  20 [1f] 00  00  00  01  03    02  11  aa  20  01 [d7] 00 [0a]
00000010:  05  00  00  00  01  08  01  08    01  08  8a  08  8a  08  c2  08
00000020:  c2  08 [05] 00  00  00  00  00   [04] 00  0a  00  00  00  00  00
00000030:  07  38  00  00  00  00  00  00    00  00  00  00  6d  13  34  34
00000040:  34  6d  0c [03][01] 00  00 [01]  [05] 00  00  00  00  00  00  00
00000050:  00  00  00  00  00  00  00  00    00  00  00  00  00  00  00  00
*
000000f0:  00  00  00  00  00  00  00  00    00  00  00  00  00  00  00  21

Changes over camera off are marked with [ ]. (Lines with * are only zeros.)

@djrscally
Copy link

So front camera:

0x02, 0x07 and 0x09 are status and clock settings
0x27 is a status checker
0x28 set to c0 is a bit weird, it says ILED_B is the indicator LED line but also that it failed. That's inaccurate on my Go2, does the LED come on ok?
0x43 turns on the S_IO
0x45 and 0x46 turn on VAUX 1 and 2. Thsese are the voltage regulators driving that camera. 0x3d is the value for VAUX1 and looks like it's set to 1.213V and so on.

I have to stop now, but I'll decipher the rest over the weekend (unless you beat me to it) and I can make a patch adding support for the OV2740 camera for you to test.

@djrscally
Copy link

Sorry; I need the DMI vendor and product name for your device; can you give me sudo dmidecode -t1?

@akobel
Copy link
Owner Author

akobel commented Mar 5, 2022

Sure, that'd be

Handle 0x000C, DMI type 1, 27 bytes
System Information
	Manufacturer: LENOVO
	Product Name: 20JCS00C00
	Version: ThinkPad X1 Tablet Gen 2
	Serial Number: XXXXXXXX
	UUID: XXXXXXXX
	Wake-up Type: Power Switch
	SKU Number: LENOVO_MT_20JC_BU_Think_FM_ThinkPad X1 Tablet Gen 2
	Family: ThinkPad X1 Tablet Gen 2

There's also a 20JB variant (identical concerning the camera AFAIK, but without LTE or with other minor differences), but I don't have one available to confirm for sure. I don't know if that would warrant filtering for version or family instead of product name, if this is even possible at this stage.

@akobel
Copy link
Owner Author

akobel commented Mar 5, 2022

Also trying to follow, by the way. You got the meanings of the entries in the I2cTestTool dump from the register map in the TPS68470 datasheet, right? Starts to make sense to me.

0x28 set to c0 is a bit weird, it says ILED_B is the indicator LED line but also that it failed. That's inaccurate on my Go2, does the LED come on ok?

Yes, it does. Datasheet says bits 2 and 6 are set for ILED_A/B enabled (A with 16mA, B with 2mA with bits[5:4]=00), and bits 3 and 7 are unset for failure mode "open"; which apparently means "no failure"?

0x3d is the value for VAUX1 and looks like it's set to 1.213V and so on.

From what I can tell, voltages are

  • VCM: 2.8152 V
  • VAUX1: 1.2132 V
  • VAUX2: 1.8006 V
  • VIO: 1.8006 V
  • VSIO: 1.8006 V
  • ANA (VA): 2.8152 V
  • CORE (VD): 1.198 V

That'd be nearly identical to the Surface Go settings that are already in tps68470_board_data.c; I tried to add a near-verbatim copy for the X1 Tablet, but only got it to

int3472-tps68470 i2c-INT3472:05: INT3472 seems to have no dependents.

I hope I can free some more time tomorrow for a bit of investigation...

@djrscally
Copy link

djrscally commented Mar 5, 2022

Also trying to follow, by the way. You got the meanings of the entries in the I2cTestTool dump from the register map in the TPS68470 datasheet, right? Starts to make sense to me.

Exactly.

Yes, it does. Datasheet says bits 2 and 6 are set for ILED_A/B enabled (A with 16mA, B with 2mA with bits[5:4]=00), and bits 3 and 7 are unset for failure mode "open"; which apparently means "no failure"?

Yes, but 0xc0 is bits 7 and 6 set, so ILED_B enabled and failure mode shorted...which doesn't really fit the fact that it works fine. It's the same on my Go2

I hope I can free some more time tomorrow for a bit of investigation...

That's the driver for the PMIC. It's complaining that it doesn't find any ACPI devices (I.E. Linux's representation of the Device (LNK0) and Device (LNK1) from the DSDT table) declaring themselves as dependent on it in their _DEP entry, but according to your DSDT dump they both do...bit weird that one; acpi device creation should be finished by the time this driver probes so I'd expect them to be available. You're building a custom kernel are you? Have you configured that driver as built-in if so?

There might be an additional problem here in that the PMIC driver as written only expects a single sensor to be dependent on it, so it only fetches the first sensor and ignores any others. That won't work if the OV2470 turns out to be the second one. I patched that recently so it'll handle multiple consuming sensors but it's not upstream anywhere yet...if you get past the "no dependents" problem and later find the OV2470 fails probe for missing clock / GPIOs, this will be why.

@markum
Copy link

markum commented Mar 6, 2022

Sorry; I need the DMI vendor and product name for your device; can you give me sudo dmidecode -t1?

dmidecode 3.2

Getting SMBIOS data from sysfs.
SMBIOS 3.0.0 present.

Handle 0x000B, DMI type 1, 27 bytes
System Information
Manufacturer: LENOVO
Product Name: 20JBCTO1WW
Version: ThinkPad X1 Tablet Gen 2
Serial Number: XXXXXXXXX
UUID: XXXXXXXXXXXXXXX
Wake-up Type: Power Switch
SKU Number: LENOVO_MT_20JB_BU_Think_FM_ThinkPad
Family: ThinkPad

@markum
Copy link

markum commented Mar 6, 2022

Here is the list of model variants of the thinkpad x1 tablet gen 2 which should all have the same camera model:
https://psref.lenovo.com/Product/Think_Tablets/ThinkPad_X1_Tablet_2nd_Gen
I am not sure, but I believe it is the same for gen 1:
https://psref.lenovo.com/Product/Think_Tablets/ThinkPad_X1_Tablet

@akobel
Copy link
Owner Author

akobel commented May 1, 2022

Sorry for the silence; finally could manage to try a couple of things.

First, some (probably) simple patches over Dan's branch, to bring the Gen 1 and Gen 2 sensor activation on solid ground: e0daada and 88bccfe.

Second, with the minimal diff that leads to somehow useful output on my machine (sorry guys, I lost track of what the current best bet is):

@@ -222 +222 @@ static const struct ov2740_reg mode_1932x1092_regs[] = {
-       {0x3805, 0x8e},
+       {0x3805, 0x8f},

(Pinkish output, ipu3-cio2 0000:00:14.3: payload length is 2725632, received 2718144 over dmesg while running qcam)

With this, I used the ipu3-capture.sh script with a while true; do i2ctransfer -y -f 8 w3@0x10 0x50 0x40 0x82; done running in the background. This is, IIUC, expected to give the following test pattern (cropped):
img-052

Instead, I receive:
frame-000000
frame-000000.bin.gz
frame-000000.raw.gz

First observation: I assume the color coding is messed up; I had some interim solution with better alignment, but I lost it somewhere during testing, and there it looked like the BG/GR was rather interpreted as GBGR or something; I assume this is what we hope could be fixed with horizontal and vertical flipping?

Second observation: the image has size 1950x1092, while I'd expect 1920x1080 (nominal camera resolution) or perhaps 1936x1112 or 1936x1116 (physical array size according to the datasheet, depending whether you believe more the sketch of the array or the claimed 20 rows of black, which can probably be used for black level compensation or whatever), or at least 1932x1092 (description of the mode_reg line in ov2740.c).

Which brings me to my next open question: why the funny 1932x1092? Shouldn't it be 1920x1080? Or, if cropping is disabled and the entire sensor array is used, 1936x1096?

And does it even make sense that we mess around with the TIMING_HTS and TIMING_VTS values (0x380c to 0x380f), and perhaps even the OUTPUT_SIZE values? Are they overwritten in ov2740_set_format according to supported_modes?

@akobel
Copy link
Owner Author

akobel commented May 1, 2022

Ah, at least I can explain the 1950: ipu3-unpack and ipu3-capture pad to the next multiple of 50, IIUC. Cropping the black border (and cutting off the bit of noise at the bottom) makes the output 1933 pixels wide, which is sufficiently close to 1932 that I assume that 1932 pixels per row are actually read...

@akobel
Copy link
Owner Author

akobel commented May 1, 2022

58b5201 does wonders for my complexion. :)

@djrscally
Copy link

djrscally commented May 1, 2022

58b5201 does wonders for my complexion. :)

Oh good spot! I was stuck on the the width and flip thing screwing with the bayer ordering somehow (you are right that this can cause the same effect. Libcamera needs to know the sensor's orientation and have control over the flipping of the image so that it can calculate and apply a correction). Is that a mistake we introduced, or was it always there? Do you still need to tweak 0x3805 to make it work after that change?

Second observation: the image has size 1950x1092, while I'd expect 1920x1080 (nominal camera resolution) or perhaps 1936x1112 or 1936x1116 (physical array size according to the datasheet, depending whether you believe more the sketch of the array or the claimed 20 rows of black, which can probably be used for black level compensation or whatever), or at least 1932x1092 (description of the mode_reg line in ov2740.c).

Which brings me to my next open question: why the funny 1932x1092? Shouldn't it be 1920x1080? Or, if cropping is disabled and the entire sensor array is used, 1936x1096?

That's just the size that the driver had already defined when we started working on it; I don't know why they chose that size either.

Ah, at least I can explain the 1950: ipu3-unpack and ipu3-capture pad to the next multiple of 50, IIUC. Cropping the black border (and cutting off the bit of noise at the bottom) makes the output 1933 pixels wide, which is sufficiently close to 1932 that I assume that 1932 pixels per row are actually read...

Actually, setting 0x3805 = 0x8f makes the output 1933 pixels wide. The mode definition says the width should be 1932, but we're telling the sensor to output 1933. That's why it is confusing me that that change "works"...because as far as I understand things, it's wrong.

And does it even make sense that we mess around with the TIMING_HTS and TIMING_VTS values (0x380c to 0x380f), and perhaps even the OUTPUT_SIZE values? Are they overwritten in ov2740_set_format according to supported_modes?

Output size if set during ov2740_start_streaming() as part of the array of register values. HTS and VTS are controlled by the HBLANK and VBLANK controls, so will be configured to some defaults during ov2740_start_streaming() too but then can be changed at any point after that. Those will affect the framerate though, not the size of the output image.

Sorry for going quiet here the last week or so also, had a bunch of other things to work on. I think we're pretty close with this now though right? We could get it cleaned up and raise a patch series.

@martin9959
Copy link

58b5201 does wonders for my complexion. :)

Oh good spot! I was stuck on the the width and flip thing screwing with the bayer ordering somehow (you are right that this can cause the same effect. Libcamera needs to know the sensor's orientation and have control over the flipping of the image so that it can calculate and apply a correction). Is that a mistake we introduced, or was it always there? Do you still need to tweak 0x3805 to make it work after that change?

For me, tweaking 0x3805 is still needed. With your patch, the image is green instead of pink. Flipping does not work (as reported here); if i add this:

0x3810 set to 0x00
0x3811 set to 0x00
0x3812 set to 0x00
0x3813 set to 0x00
0x3821 set to 0x08

(as suggested here) to make flipping work again, vertical_flip=1 turns the image pink again, so pink and green are basically exchanged.

@akobel
Copy link
Owner Author

akobel commented May 8, 2022

58b5201 does wonders for my complexion. :)

Oh good spot! I was stuck on the the width and flip thing screwing with the bayer ordering somehow (you are right that this can cause the same effect. Libcamera needs to know the sensor's orientation and have control over the flipping of the image so that it can calculate and apply a correction). Is that a mistake we introduced, or was it always there?

It's been there ever since. Not sure which driver this was forked from; but somehow I feel that this was either meant for a platform where significant work has been applied in user space, or it wasn't really tested.

Do you still need to tweak 0x3805 to make it work after that change?

Yes. [...] So, regarding register, mode, and crop setting:

I got sick and tired of rebooting (which always takes me ages, thanks to Bitlocker being overly picky about changing default boot targets, plus disk encryption wanting a password, plus boot loader/"enter BIOS" hooks being somewhat unreliable on this machine.) So I spent some time offline with one of my crudest hacks ever, to avoid reboots: x1-tablet-gen2/0004-media-i2c-ov2740-Add-dynamic-changes-of-mode-crop-re.patch, and I share it in case it makes someone's future testing easier.

In short: this allows to you define three files, which are dynamically loaded into the driver upon powerup of the device.

/ov2740.reg has the syntax

# address length-in-bytes value, e.g.
0x3800 2 0x0000
0x3802 1 0x00
0x3803 1 0
# or
0x3804 2 1935

and accepts hex (prefix 0x) or decimal values, and ignores unparsable lines (especially ones starting with #).

/ov2740.mode and /ov2740.crop take plain decimal or hex numbers (again ignoring comments) and allow you to override

mode->width
mode->height
mode->hts
mode->vts_def
mode->vts_min

and

sel->r.top
sel->r.left
sel->r.width
sel->r.height

in that order. Ugly hack, but it does the job.

And I tested a number of settings, and eventually ended up with something vaguely usable - no errors anymore in dmesg, and it's values that seem to make some sense to me:

Reg:

# H_CROP_START
0x3800 2 0
# V_CROP_START
0x3802 2 0
# H_CROP_END
0x3804 2 1935
# V_CROP_END
0x3806 2 1095
# H_OUTPUT_SIZE
0x3808 2 1936
# V_OUTPUT_SIZE
0x380a 2 1096
# TIMING_HTS
0x380c 2 2184
# TIMING_VTS
0x380e 2 1092
# H_WIN_OFF
0x3810 2 0
# V_WIN_OFF
0x3812 2 0

Crop:

# top
8
# left
8
# width
1920
# height
1080

Mode:

# width
1936
# height
1096
# hts
1080
# vts_def
2184
# vts_min
1092
# link_freq_index
# 0

Framerate is still stuck at 15 FPS, but apart from that it looks pretty okay.

Sorry for going quiet here the last week or so also, had a bunch of other things to work on. I think we're pretty close with this now though right? We could get it cleaned up and raise a patch series.

Same here. And right, I think so, too; would be nice to at least get this to a baseline where we can suggest other interested parties to give it a shot without need to do custom patching upon the tree. I might have found a victim in real life yesterday. ;-)

@akobel
Copy link
Owner Author

akobel commented May 10, 2022

Meh. Except that it doesn't work with those values at the initial boot-up. Something about HTS and VTS is messed up, and the device doesn't probe properly. Coincidentally, the values I used in the reg and mode sections don't match - my bad, I'll investigate.

Actually using it after a successful initial detection is apparently more robust; the actual HTS/VTS setting during use and the interpretation of the blanking periods has some more involved logic than just fixed register values.

However, I've got to admit that I don't really understand what the HTS and VTS values are. My Google-Fu for "MIPI SCLK HTS VTS" claims that fps = SCLK / HTS/VTS (http://www.terasic.com.tw/wiki/D8M_resolution%26frame_rate_setting), probably rather a <= than =. But SCLK is just a clock for synchronization, not a "bandwidth" or something, right? What units/meaning do HTS and VTS have then?
My understanding was that HTS means the slice length of a row read; so, for full resolution images, 1936 + some padding for blanking, essentially as a gap between neighboring rows. And measurement would be in pixels, not bytes - then, the 2160 would sound reasonable. Similarly for VTS.
But how does this affect FPS then? If HTS and VTS are just measured in "number of pixels", that wouldn't really determine the bandwidth that is used, wouldn't it?

Finally, the driver currently has

#define OV2740_LINK_FREQ_360MHZ		360000000ULL
#define OV2740_SCLK			72000000LL
#define OV2740_MCLK			19200000

What is MCLK? And where do the 360 MHz come from? The datasheet claims 720 Mbps/lane and a 720 MHz MIPI serial clock, and I can't find any mention of 360 in the sheet...

@djrscally
Copy link

djrscally commented May 10, 2022

Meh. Except that it doesn't work with those values at the initial boot-up. Something about HTS and VTS is messed up, and the device doesn't probe properly. Coincidentally, the values I used in the reg and mode sections don't match - my bad, I'll investigate.

Ah, ok - good luck! The values you gave looked ok at a quick glance, though I confess that starting the window at 0,0 instead of in the centre causes me pain :P

However, I've got to admit that I don't really understand what the HTS and VTS values are. My Google-Fu for "MIPI SCLK HTS VTS" claims that fps = SCLK / HTS/VTS (http://www.terasic.com.tw/wiki/D8M_resolution%26frame_rate_setting), probably rather a <= than =. But SCLK is just a clock for synchronization, not a "bandwidth" or something, right? What units/meaning do HTS and VTS have then?

Columns/Lines of pixels. MCLK is the sync clock, LINK_FREQ is the "bandwidth"

My understanding was that HTS means the slice length of a row read; so, for full resolution images, 1936 + some padding for blanking, essentially as a gap between neighboring rows. And measurement would be in pixels, not bytes - then, the 2160 would sound reasonable. Similarly for VTS.

This is correct

Finally, the driver currently has

#define OV2740_LINK_FREQ_360MHZ		360000000ULL
#define OV2740_SCLK			72000000LL
#define OV2740_MCLK			19200000

What is MCLK? And where do the 360 MHz come from? The datasheet claims 720 Mbps/lane and a 720 MHz MIPI serial clock, and I can't find any mention of 360 in the sheet...

MCLK is the 19.2MHz reference clock being provided to the camera sensor by something else in the device, in your case it's part of the TPS68470 stuff we looked at ages ago. The 360 MHz is the rate of the transfer between the camera sensor and the CSI-2 receiver; the data captured by the sensor is transmitted electrically along a bus to the receiver (controlled by the ipu3-cio2 driver) which loads it into memory and send it to the image processing unit. The datasheet is probably giving the maximum rate, but my understanding is it just needs to be sufficient to carry the data that the sensor puts out, which looks something like this:

link frequency = (pixel rate * bits per sample) / nr of lanes / 2

pixel rate being the number of pixels in an image multiplied by the frame rate you need to support. Bits per sample is the number of bits of data it takes to transmit a single pixel (which is encoded in the media bus format code MEDIA_BUS_FMT_SBGGR10_1X10 - the number after the X is the bits per sample). Nr of lanes is the number of data lanes; the MIPI interface allows for up to four data lanes to run concurrently, though in your case there's only 2. Finally, divided by 2 because the format is double data rate.

So for the only mode the ov2740 has that's ((1932 * 1092 * 60) * 10) / 2 / 2 = 316461600 (assuming it is 60fps, the datasheet says it should be). The actual link frequency that gets set depends on the capabilities of the sensor's PLL, but as far as I understand things is ideally selected by the vendor to reduce EM interference with other components in the platform. In the cio2-bridge code we mimic firmware setting a link frequency for our devices:

static const struct cio2_sensor_config cio2_supported_sensors[] = {
	/* Omnivision OV5693 */
	CIO2_SENSOR_CONFIG("INT33BE", 1, 419200000),
	/* Omnivision OV8865 */
	CIO2_SENSOR_CONFIG("INT347A", 1, 360000000),
	/* Omnivision OV7251 */
	CIO2_SENSOR_CONFIG("INT347E", 1, 319200000),
	/* Omnivision OV2680 */
	CIO2_SENSOR_CONFIG("OVTI2680", 0),
};

and drivers compare the link frequency that's defined in firmware with the values that they have been built to support, and will fail out if the firmware asks for a link freq that they can't do. My experience so far is (as with the first and 3rd entry in that list above) they're setting some value plus the MCLK, so if you scraped the PLL registers whilst running windows you might find it was running at 319.2MHz there. We could add support for that frequency to the driver too.

But how does this affect FPS then? If HTS and VTS are just measured in "number of pixels", that wouldn't really determine the bandwidth that is used, wouldn't it?

VTS and HTS, minus the height and width of the image that's output, define "blanking" time which is an idle period during which no data is produced. You can increase the blanking to lower frame rate without otherwise changing the output format

@akobel
Copy link
Owner Author

akobel commented May 10, 2022

So for the only mode the ov2740 has that's ((1932 * 1092 * 60) * 10) / 2 / 2 = 316461600 (assuming it is 60fps, the datasheet says it should be). The actual link frequency that gets set depends on the capabilities of the sensor's PLL, but as far as I understand things is ideally selected by the vendor to reduce EM interference with other components in the platform.

Thanks once again for a verbose explanation, really appreciated. I'll try and digest at due time. ;-)

Just one minor note: wouldn't a factor 3 or 4 come in for RGB or BGGR? That has already puzzled me - I cannot link the sizes of the .bin and .raw output of the ipu3-capture.sh script (2735616 and 4274400 bytes, with my current settings) to any reasonable multiple of (1936 or 1920) and (1096 or 1080)... With the script saying

Video format set: IPU3_SBGGR10 (62337069) 1936x1096 field none, 1 planes:

and looking at https://www.kernel.org/doc/html/v4.15/media/uapi/v4l/subdev-formats.html#bayer-formats, I'd expect 19361096(10/8)*4 = 10368000 bytes per frame. On the other hand, the output goes on with

 * Stride 2496, buffer size 2735616

which matches the .bin size? (And 2735616 = 1096*2496; but where the heck does the 2496 come from?)

@djrscally
Copy link

Just one minor note: wouldn't a factor 3 or 4 come in for RGB or BGGR?

Er, I don't see that it would. Why do you think so?

That has already puzzled me - I cannot link the sizes of the .bin and .raw output of the ipu3-capture.sh script (2735616 and 4274400 bytes, with my current settings) to any reasonable multiple of (1936 or 1920) and (1096 or 1080)... With the script saying

Video format set: IPU3_SBGGR10 (62337069) 1936x1096 field none, 1 planes:

and looking at https://www.kernel.org/doc/html/v4.15/media/uapi/v4l/subdev-formats.html#bayer-formats, I'd expect 1936_1096_(10/8)*4 = 10368000 bytes per frame. On the other hand, the output goes on with

 * Stride 2496, buffer size 2735616

which matches the .bin size? (And 2735616 = 1096*2496; but where the heck does the 2496 come from?)

Stride is the number of bytes needed to store a full row with some configurable padding in libcamera. The format the data is output from the sensor to the CSI-2 receiver is actually a packed one - 25 pixels into each 32 bytes, with 6 bytes of padding. 1936 / 25 = 77.44, which in that format becomes (77 * 32) + 32 = 2496 bytes. The ipu3-capture script unpacks that format into one that can be converted easily into a viewable format.

@markum
Copy link

markum commented Jun 4, 2022

Is there anything new, which I can try/test? I am a little limited with what I can do, but before this thread goes dead, I will try new things out :-)

@akobel
Copy link
Owner Author

akobel commented Jun 7, 2022

Good point, got side-tracked a little bit. But my camera actually works now, as long as I use the right kernel... ;-)

@djrscally Any plans how we should proceed here? Do you want to, uhm, foreport (as opposed to backport?) your branch onto 5.18 / master before we go any further? Do you think this is in a state where you'd feel comfortable to send a patch to Hans or the ACPI maintainers?

As far as I can see, the change that is still missing upstream is 1677773acbcc6bbd5e75223db0dae6f4a8d9bdd6. Most likely, everything else (int3472 stuff, board data, multiple consumers etc., and ov2740 stuff) should be pretty much isolated from the rest of the kernel, right?

In particular, I have a working setting for the camera that I can dig out from my sources. It's not fully warning-free, but it works overall without hickups, and decodes color correctly; I assume that's a good starting point.
Next steps after that would probably be support for automatic gain, but first have something that works on upstream, I guess.

@akobel
Copy link
Owner Author

akobel commented Jun 7, 2022

Just one minor note: wouldn't a factor 3 or 4 come in for RGB or BGGR?

Er, I don't see that it would. Why do you think so?

(snip)

Stride is the number of bytes needed to store a full row with some configurable padding in libcamera. The format the data is output from the sensor to the CSI-2 receiver is actually a packed one - 25 pixels into each 32 bytes, with 6 bytes of padding. 1936 / 25 = 77.44, which in that format becomes (77 * 32) + 32 = 2496 bytes. The ipu3-capture script unpacks that format into one that can be converted easily into a viewable format.

Ah, because I was mistaken as always by being idealistic about marketing figures. I thought that 1936x1096 pixels means 1936x1096 full-color sensors, each a 2x2-array of BG/GR subpixels. But your explanation made it clear that it's actually just 1936x1096 monochromatic pixels, half of which are green and a quarter each are blue and red; and the overall color information is interpolated in the debayering stage. That plus the packed format makes sense; I should have RTFM with more attention.

@djrscally
Copy link

Good point, got side-tracked a little bit. But my camera actually works now, as long as I use the right kernel... ;-)

@djrscally Any plans how we should proceed here? Do you want to, uhm, foreport (as opposed to backport?) your branch onto 5.18 / master before we go any further? Do you think this is in a state where you'd feel comfortable to send a patch to Hans or the ACPI maintainers?

Er there'll be some cleanup to do and I think that my bare branch still doesn't actually work as I have the settings wrong...can you link the tree you've got working and I'll cross reference them and make sure I get the settings right, rebase onto media-master, tidy, and then we can send a patchset to linux-media.

As far as I can see, the change that is still missing upstream is 1677773acbcc6bbd5e75223db0dae6f4a8d9bdd6. Most likely, everything else (int3472 stuff, board data, multiple consumers etc., and ov2740 stuff) should be pretty much isolated from the rest of the kernel, right?

That whole multiple consumers series actually needs independently of this to enable the IR camera on my surface go2. I totally forgot that Rafael asked for a (fairly minor) change that I haven't done, so it's waiting for me. I'll fix that and send it. Otherwise yes everything else should be pretty standalone

In particular, I have a working setting for the camera that I can dig out from my sources. It's not fully warning-free, but it works overall without hickups, and decodes color correctly; I assume that's a good starting point. Next steps after that would probably be support for automatic gain, but first have something that works on upstream, I guess.

Automatic gain on the sensor side you mean?

akobel added a commit to akobel/linux-djrscally that referenced this issue Jun 29, 2022
Changes over Lenovo's ThinkPad X1 Tablet Gen 2:
- INT3472:05 -> INT3472:04
- INT3474:01 -> INT3474:00
as reported in
akobel/linux-thinkpad-x1-tablet#1 (comment)

Signed-off-by: Alexander Kobel <[email protected]>
@akobel
Copy link
Owner Author

akobel commented Jun 29, 2022

Good point, got side-tracked a little bit.
...again...

But my camera actually works now, as long as I use the right kernel... ;-)
@djrscally Any plans how we should proceed here? Do you want to, uhm, foreport (as opposed to backport?) your branch onto 5.18 / master before we go any further? Do you think this is in a state where you'd feel comfortable to send a patch to Hans or the ACPI maintainers?

Er there'll be some cleanup to do and I think that my bare branch still doesn't actually work as I have the settings wrong...can you link the tree you've got working and I'll cross reference them and make sure I get the settings right, rebase onto media-master, tidy, and then we can send a patchset to linux-media.

Right. akobel/linux-djrscally@9b096f4 should be reasonable.
I spent some hours to try and polish out the stuff mentioned in the commit message, but to no avail right now.

TL;DR: 30Hz at 1920x1080 works. First stream is hit-and-miss, second and subsequent streams are steady.
(It is more reliable for some reason with the hacky on-the-fly-read register setting from the latest commit in that branch, but this is obviously not meant for real.)

Edit: Not sure whether it's just coincidence, but the first attempt at streaming seems to freeze after some 20 seconds or so if I start very early after boot (E.g., to test these driver settings). It's stable if I start later, like, several minutes after booting.
I don't know whether there's something more or less unrelated - initialization of a different driver, some deferred probes still in the queue, ... - that have side effects on the camera. Couldn't find any correlation to other activities, though.

That's the only thing that really annoys me for publishing, but I couldn't find a fully solid version without freezes (or I accidentally removed it while I was looking for the right color and couldn't find it again). Perhaps someone else can test or tweak until it's reliable; everything else (gain, FPS, HDR, lens correction, different resolutions/modes) is not that crucial for very basic (webcam) use.

In particular, I have a working setting for the camera that I can dig out from my sources. It's not fully warning-free, but it works overall without hickups, and decodes color correctly; I assume that's a good starting point. Next steps after that would probably be support for automatic gain, but first have something that works on upstream, I guess.

Automatic gain on the sensor side you mean?

Yes, but I realized that there is a corresponding v4l knob, right? So I think the control is actually supported, it's just that qcam doesn't try to use it. But perhaps there's also automatic support, who knows.
In any case, image quality on Windows is clearly better. ;-)

@akobel
Copy link
Owner Author

akobel commented Oct 23, 2022

@djrscally Lost track again, sorry. Are there any news and/or plans from your side?

I personally didn't spend any time on that one since June, but if I know the route to go, I can take over a couple of steps. If I read the recent kernel logs correctly, the multi-consumer patch is in the 6.1 release candidate, right?
Should I try and port our fruits of labor once this one's out, or would now already be the right time?

@akobel
Copy link
Owner Author

akobel commented Mar 16, 2023

@djrscally Hey Dan, got back to that one again. I successfully ran a test with fairly minimal modifications on a recent 6.2.6 kernel (Arch's build).

All I had to do was to 1. apply from your djrscally/thinkpad-x1 branch

961ddfb79b92 media: ipu3-cio2: Add INT3474 to cio2-bridge
  1. apply board-data.patch, which itself is heavily inspired by your
cb8135b0bac3 platform/x86: int3472: Add board data for Thinkpad X1

and 3. copy over the results on ov2740.c that we achieved above. Those include

5961c659e632 (djrscally/thinkpad-x1) media: i2c: fixup ov2740 output sizes
14a1d7a95aa1 media: i2c: Add location and rotation controls to ov2740
a50263acc021 media: i2c: Add flip controls to ov2740
# 961ddfb79b92 media: ipu3-cio2: Add INT3474 to cio2-bridge
b7fde3eb12ce media: i2c: Implement .get_selection() for ov2740
4d1be84bb640 media: i2c: Fix array crop registers in ov2740
390020e10a8d meda: i2c: Defer probe if no endpoint in ov2740
e73686349f4c media: i2c: Add pm_runtime to ov2740
ec4639695e52 media: i2c: Acquire GPIOs in ov2740
eaccb5fb3459 media: i2c: Acquire clock in ov2740
572121656810 media: i2c: Handle regulators in ov2740
# cb8135b0bac3 platform/x86: int3472: Add board data for Thinkpad X1

plus additional changes regarding fixing the Bayer format and some mode setting adjustments.

Now, IIUC, patches towards ov2740 shall be based on the linuxtv media_tree master. There are some other mostly unrelated changes on that branch; some are cosmetic, some others small detail fixes, but all shall be kept for sure. Your patches don't cleanly apply.

I'm unsure on how to proceed. I would agree to try and rebase your patches, but first I'm not quite sure whether the patches on int3472 go to the same branch (probably not?), those on ipu3-cio2 go to the same maintainers (probably not, and there's no base tree given - is it Linus' tree then?), and whether all can be submitted at the same time.
Furthermore, since your old patches don't cleanly apply anymore on media_tree, I can't fast-forward. And I don't know whether it would be appropriate if I submit patches by you, but changed by me, as Authored-by Dan, or even Signed-off-by Dan. And whether all should be sent as one batch with the collection of all chunks, or independently.

So, long story short: would you be able/willing to revisit the patches mentioned above, authored by you, and rebase them to the proper branch, so that I can add the few things required to get some useful output? It's really "around the corner" now, I think.

@djrscally
Copy link

@akobel Hey dude - I'm sorry this one keeps slipping off my radar!

I'm unsure on how to proceed. I would agree to try and rebase your patches, but first I'm not quite sure whether the patches on int3472 go to the same branch (probably not?)

No those should be based on platform-drivers-x86/for-next and sent to the platform-driver-x86 mailing list.

those on ipu3-cio2 go to the same maintainers (probably not, and there's no base tree given - is it Linus' tree then?)

Those go to linux media (so based on media_tree next)...but I'd be the one looking at them anyway.

And I don't know whether it would be appropriate if I submit patches by you, but changed by me, as Authored-by Dan, or even Signed-off-by Dan. And whether all should be sent as one batch with the collection of all chunks, or independently.

It would depend on the scope of your changes; you can always add a Co-developed-by (which should be followed by a Signed-off-by-akobel) to have both authors on there.

And whether all should be sent as one batch with the collection of all chunks, or independently.

Batches to each mailing list; so all the ov2740 ones and the ipu3-cio2 one to linux media in one series, but the platform/x86 one separately.

So, long story short: would you be able/willing to revisit the patches mentioned above, authored by you, and rebase them to the proper branch, so that I can add the few things required to get some useful output? It's really "around the corner" now, I think.

Yes, absolutely - and I'm sorry that I didn't do it already. I'll try to get it done over the weekend.

@akobel
Copy link
Owner Author

akobel commented Mar 18, 2023

@akobel Hey dude - I'm sorry this one keeps slipping off my radar!

@djrscally No worries, thanks for getting back to us!

I'm unsure on how to proceed. I would agree to try and rebase your patches, but first I'm not quite sure whether the patches on int3472 go to the same branch (probably not?)

No those should be based on [...]

Okay, that's pretty much what I expected.

So, long story short: would you be able/willing to revisit the patches mentioned above, authored by you, and rebase them to the proper branch, so that I can add the few things required to get some useful output? It's really "around the corner" now, I think.

Yes, absolutely - and I'm sorry that I didn't do it already. I'll try to get it done over the weekend.

Cool. My main concern was to submit patches (whether mildly changed or not) without your confirmation / LGTM. After all, I wouldn't claim to have written them, but also don't want to put your name under something that I could potentially mess up. ;-) Most of the adjustments should be trivial, but there's actually a few that might be trivial to merge for you, but aren't fully obvious for me (regarding powerup/powerdown/autosuspend).

Anyway, if you submit your part, please cc me; I'll be happy to test and work on the remaining adjustments (timing/cropping) that will be cumbersome for you without access to the hardware.

@djrscally
Copy link

@akobel Hey dude - I'm sorry this one keeps slipping off my radar!

@djrscally No worries, thanks for getting back to us!

I'm unsure on how to proceed. I would agree to try and rebase your patches, but first I'm not quite sure whether the patches on int3472 go to the same branch (probably not?)

No those should be based on [...]

Okay, that's pretty much what I expected.

So, long story short: would you be able/willing to revisit the patches mentioned above, authored by you, and rebase them to the proper branch, so that I can add the few things required to get some useful output? It's really "around the corner" now, I think.

Yes, absolutely - and I'm sorry that I didn't do it already. I'll try to get it done over the weekend.

Cool. My main concern was to submit patches (whether mildly changed or not) without your confirmation / LGTM. After all, I wouldn't claim to have written them, but also don't want to put your name under something that I could potentially mess up. ;-) Most of the adjustments should be trivial, but there's actually a few that might be trivial to merge for you, but aren't fully obvious for me (regarding powerup/powerdown/autosuspend).

Yes this part is always annoying. Having to account for both acpi and devicetree firmware (which work completely differently) and for our hacks which are kinda half way between the two makes it a bit rubbish. The Intel guys have got some work going on to make all that much much easier though.

Anyway, if you submit your part, please cc me; I'll be happy to test and work on the remaining adjustments (timing/cropping) that will be cumbersome for you without access to the hardware.

So I rebased my thinkpad-x1 branch onto media master and pushed it here. It's not got the board data patch that I made since you had to change it; you should be able to drop yours in, but rebase so that it's at the front (I.E. directly after eeac8ede1755 ("media/master, media-master) Linux 6.3-rc2")) to make it easier to output the series for linux media. Do you want to add your changes on top and make sure everything's working and then we can look to post them? Separately would be fine, but you could also just send the lot together if you like and I'll pick up any review comments that people make for my patches. Up to you.

@akobel
Copy link
Owner Author

akobel commented Mar 20, 2023

So I rebased my thinkpad-x1 branch onto media master and pushed it here. It's not got the board data patch that I made since you had to change it; you should be able to drop yours in, but rebase so that it's at the front (I.E. directly after eeac8ede1755 ("media/master, media-master) Linux 6.3-rc2")) to make it easier to output the series for linux media.

Thanks, I'll have a look; tomorrow or the day after, I guess.

Do you want to add your changes on top and make sure everything's working and then we can look to post them?

Yes, I think it makes sense if I try and add my bits and pieces; IIRC, they are required to get a picture on the first attempt. If someone can test on their hardware, it'd be nice if they only have to apply one self-contained patch set.
Plus it's a sanity check that I didn't miss anything in my list of patches required.

Separately would be fine, but you could also just send the lot together if you like and I'll pick up any review comments that people make for my patches. Up to you.

That one I don't really care about. Let me first prepare the branch, then we'll see.

@akobel
Copy link
Owner Author

akobel commented May 9, 2023

Hi again,
this time the delay was entirely on my side; life and work had different plans.
In short: doesn't work as seamless as expected (in fact, doesn't work at all without modifications that I still have to figure out). But the main reason for delay is that I stumbled across a bunch of issues entirely unrelated that kept me from getting to the root.

So, I went for building the entire kernel from your branch. That didn't work out for me; my usual "keep it simple, stupid"-approach of taking the Arch build script for a kernel package (to rule out any misconfigurations on my end) gave me some (essentially unrelated) errors that I don't quite remember right now anymore, sorry.
Long story short, I'm on kernel 6.3.1-arch1 (stock Arch kernel from the repos) now and only built the modules in media/i2c (including ov2740, obviously), platform/x86/intel/int3472 and media/pci/intel/ipu3.
With that, ov2740 is detected and listed as Internal front camera (\_SB_.PCI0.LNK2) in cam -l, but trying to activate it, I get

[  903.927439] BUG: unable to handle page fault for address: ffffffff876e12c0
[  903.927444] #PF: supervisor write access in kernel mode
[  903.927446] #PF: error_code(0x0003) - permissions violation
[  903.927448] PGD 1eb425067 P4D 1eb425067 PUD 1eb426063 PMD 1e98000e1 
[  903.927452] Oops: 0003 [#1] PREEMPT SMP PTI
[  903.927456] CPU: 1 PID: 2470 Comm: qcam Tainted: G     U   C OE      6.3.1-arch1-1 #1 2f4443c3fa3529b1ac13dc02f36f7de43ade3ecd
[  903.927459] Hardware name: LENOVO 20JCS00C00/20JCS00C00, BIOS N1OET58W (1.43 ) 11/29/2021
[  903.927461] RIP: 0010:native_queued_spin_lock_slowpath+0x299/0x2e0
[  903.927467] Code: ff ff f3 90 8b 03 85 c0 74 ee eb f6 c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 05 48 63 d2 48 05 00 4d 03 00 48 03 04 d5 00 cb ac 88 <48> 89 28 8b 45 08 85 c0 75 09 f3 90 8b 45 08 85 c0 74 f7 48 8b 55
[  903.927469] RSP: 0018:ffffae7a9f3ffab0 EFLAGS: 00010286
[  903.927471] RAX: ffffffff876e12c0 RBX: ffff952d46380658 RCX: ffff952dc43fc180
[  903.927473] RDX: 000000000000118d RSI: 0000000046380650 RDI: ffff952d46380658
[  903.927474] RBP: ffff952e678b4d00 R08: 0000000000000004 R09: ffff952d46382f28
[  903.927476] R10: 0000000000000004 R11: ffff952dde896d80 R12: 0000000000080000
[  903.927477] R13: 0000000000080000 R14: ffff952d46382dd0 R15: ffff952d46382f28
[  903.927479] FS:  00007f6747fff6c0(0000) GS:ffff952e67880000(0000) knlGS:0000000000000000
[  903.927481] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  903.927483] CR2: ffffffff876e12c0 CR3: 00000001a0e54001 CR4: 00000000003706e0
[  903.927485] Call Trace:
[  903.927487]  <TASK>
[  903.927490]  _raw_spin_lock+0x29/0x30
[  903.927493]  __mutex_lock.constprop.0+0x253/0x6a0
[  903.927498]  ? kmalloc_trace+0x2a/0xa0
[  903.927504]  v4l2_subdev_link_validate+0xa0/0xc0 [videodev bc856520f610bf86bf5970a47c4f01fa754c4ac8]
[  903.927529]  __media_pipeline_start+0x4f0/0x6f0 [mc b10943db0fe5192f4364055c83bc3c263fddcc6f]
[  903.927540]  media_pipeline_start+0x30/0x50 [mc b10943db0fe5192f4364055c83bc3c263fddcc6f]
[  903.927548]  imgu_vb2_start_streaming+0x98/0x2a0 [ipu3_imgu 8876e3fdc707d3b337c966142d002748756463be]
[  903.927560]  vb2_start_streaming+0x60/0x130 [videobuf2_common 163e94dafcb1707046bfed21822648e1e844e543]
[  903.927570]  vb2_core_streamon+0xc5/0x170 [videobuf2_common 163e94dafcb1707046bfed21822648e1e844e543]
[  903.927577]  __video_do_ioctl+0x3c6/0x4a0 [videodev bc856520f610bf86bf5970a47c4f01fa754c4ac8]
[  903.927596]  video_usercopy+0x3a3/0x7e0 [videodev bc856520f610bf86bf5970a47c4f01fa754c4ac8]
[  903.927611]  ? __pfx___video_do_ioctl+0x10/0x10 [videodev bc856520f610bf86bf5970a47c4f01fa754c4ac8]
[  903.927628]  v4l2_ioctl+0x4a/0x60 [videodev bc856520f610bf86bf5970a47c4f01fa754c4ac8]
[  903.927642]  __x64_sys_ioctl+0x91/0xd0
[  903.927645]  do_syscall_64+0x5d/0x90
[  903.927649]  ? syscall_exit_to_user_mode+0x1b/0x40
[  903.927653]  ? do_syscall_64+0x6c/0x90
[  903.927657]  ? v4l2_ioctl+0x4a/0x60 [videodev bc856520f610bf86bf5970a47c4f01fa754c4ac8]
[  903.927670]  ? __x64_sys_ioctl+0xac/0xd0
[  903.927672]  ? syscall_exit_to_user_mode+0x1b/0x40
[  903.927676]  ? do_syscall_64+0x6c/0x90
[  903.927679]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[  903.927682] RIP: 0033:0x7f6771d1576f
[  903.927713] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[  903.927715] RSP: 002b:00007f6747ffdd90 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  903.927718] RAX: ffffffffffffffda RBX: 00007f673c0077b0 RCX: 00007f6771d1576f
[  903.927720] RDX: 00007f673c0079e0 RSI: 0000000040045612 RDI: 000000000000001a
[  903.927721] RBP: 00007f673c022280 R08: 00007f673c004bd0 R09: 00007f6747ffdcc0
[  903.927722] R10: 0000000000000001 R11: 0000000000000246 R12: 00007f673c01e410
[  903.927724] R13: 00007f673c022160 R14: 00007f673c022280 R15: 00007f673c01e570
[  903.927728]  </TASK>
[  903.927729] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_soc_avs snd_soc_hda_codec snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core snd_soc_sst_ipc ccm snd_soc_sst_dsp snd_soc_acpi_intel_match algif_aead snd_soc_acpi snd_hda_codec_hdmi bnep snd_soc_core des_generic libdes snd_compress ecb snd_ctl_led snd_hda_codec_realtek ac97_bus snd_hda_codec_generic snd_pcm_dmaengine algif_skcipher intel_tcc_cooling spi_pxa2xx_platform x86_pkg_temp_thermal cmac iTCO_wdt intel_powerclamp dw_dmac kvm_intel intel_pmc_bxt 8250_dw wacom iTCO_vendor_support gpio_tps68470 snd_hda_intel mei_pxp md4 kvm iwlmvm algif_hash coretemp mei_hdcp mei_wdt irqbypass btusb snd_intel_dspcfg af_alg mac80211 btrtl libarc4 rapl snd_intel_sdw_acpi btbcm snd_hda_codec btintel iwlwifi snd_hda_core intel_cstate intel_rapl_msr btmtk snd_hwdep think_lmi intel_uncore bluetooth pcspkr i2c_i801 snd_pcm firmware_attributes_class intel_wmi_thunderbolt ecdh_generic cfg80211 mei_me wmi_bmof crc16 i2c_smbus intel_lpss_pci snd_timer
[  903.927802]  intel_lpss intel_pch_thermal mei ipu3_cio2(OE) idma64 processor_thermal_device_pci_legacy hid_sensor_magn_3d ipu3_imgu(C) processor_thermal_device processor_thermal_rfim hid_sensor_als videobuf2_dma_sg hid_sensor_rotation hid_sensor_gyro_3d hid_sensor_incl_3d hid_sensor_accel_3d videobuf2_memops ov2740(OE) processor_thermal_mbox processor_thermal_rapl videobuf2_v4l2 v4l2_fwnode hid_sensor_trigger industrialio_triggered_buffer kfifo_buf intel_rapl_common thinkpad_acpi hid_sensor_iio_common intel_soc_dts_iosf industrialio intel_xhci_usb_role_switch videobuf2_common ledtrig_audio platform_profile nls_iso8859_1 v4l2_async joydev intel_skl_int3472_tps68470(OE) rfkill tps68470_regulator int3400_thermal videodev snd int3403_thermal mousedev vfat soundcore clk_tps68470 int340x_thermal_zone intel_vbtn soc_button_array intel_hid mc fat acpi_thermal_rel intel_skl_int3472_discrete(OE) sparse_keymap acpi_pad mac_hid fuse loop ip_tables x_tables xxhash_generic dm_crypt cbc encrypted_keys trusted asn1_encoder tee
[  903.927861]  hid_sensor_custom hid_sensor_hub hid_lenovo hid_multitouch intel_ishtp_hid usbhid i915 serio_raw atkbd rtsx_pci_sdmmc libps2 mmc_core vivaldi_fmap crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 aesni_intel i2c_algo_bit nvme drm_buddy ucsi_acpi typec_ucsi roles xhci_pci intel_ish_ipc crypto_simd intel_ishtp intel_gtt xhci_pci_renesas nvme_core drm_display_helper cryptd nvme_common cec ttm rtsx_pci typec video i8042 i2c_hid_acpi serio wmi i2c_hid btrfs blake2b_generic xor raid6_pq libcrc32c crc32c_generic crc32c_intel vboxnetflt(OE) vboxnetadp(OE) vboxdrv(OE) pkcs8_key_parser dm_multipath dm_mod ec_sys sg crypto_user
[  903.927903] CR2: ffffffff876e12c0
[  903.927905] ---[ end trace 0000000000000000 ]---
[  903.927907] RIP: 0010:native_queued_spin_lock_slowpath+0x299/0x2e0
[  903.927911] Code: ff ff f3 90 8b 03 85 c0 74 ee eb f6 c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 05 48 63 d2 48 05 00 4d 03 00 48 03 04 d5 00 cb ac 88 <48> 89 28 8b 45 08 85 c0 75 09 f3 90 8b 45 08 85 c0 74 f7 48 8b 55
[  903.927913] RSP: 0018:ffffae7a9f3ffab0 EFLAGS: 00010286
[  903.927915] RAX: ffffffff876e12c0 RBX: ffff952d46380658 RCX: ffff952dc43fc180
[  903.927916] RDX: 000000000000118d RSI: 0000000046380650 RDI: ffff952d46380658
[  903.927918] RBP: ffff952e678b4d00 R08: 0000000000000004 R09: ffff952d46382f28
[  903.927919] R10: 0000000000000004 R11: ffff952dde896d80 R12: 0000000000080000
[  903.927920] R13: 0000000000080000 R14: ffff952d46382dd0 R15: ffff952d46382f28
[  903.927922] FS:  00007f6747fff6c0(0000) GS:ffff952e67880000(0000) knlGS:0000000000000000
[  903.927924] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  903.927925] CR2: ffffffff876e12c0 CR3: 00000001a0e54001 CR4: 00000000003706e0
[  903.927927] note: qcam[2470] exited with irqs disabled
[  903.927967] note: qcam[2470] exited with preempt_count 2

No further investigation done yet, sorry... Try to give it some more time asap.

@djrscally
Copy link

@akobel it's a bug that has cropped up after some other patches in 6.3 - I'm actually trying to figure it out now already, but for now if you run 6.2 instead you'll avoid this error.

@akobel
Copy link
Owner Author

akobel commented May 10, 2023

Thanks for the hint; I think I had build problems on 6.2, though. That's one of the reasons for the procrastination; I'll check again.

@djrscally
Copy link

Thanks for the hint; I think I had build problems on 6.2, though. That's one of the reasons for the procrastination; I'll check again.

Well I got to the bottom of the problem, so you could also apply this fix:

---
 drivers/media/v4l2-core/v4l2-subdev.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index b10045c02f43..21c9860ce43a 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1236,6 +1236,10 @@ int v4l2_subdev_link_validate(struct media_link *link)
        struct v4l2_subdev_state *source_state, *sink_state;
        int ret;
 
+       if (!is_media_entity_v4l2_subdev(link->source->entity) ||
+           !is_media_entity_v4l2_subdev(link->sink->entity))
+               return 0;
+
        sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
        source_sd = media_entity_to_v4l2_subdev(link->source->entity);
 
-- 
2.34.1

And then it should work fine on 6.3 again

@akobel
Copy link
Owner Author

akobel commented Jul 1, 2023

Finally.
Yes, this works (and is included these days with 55f1ecb1199000932cf82e357841cc7498ac904f).

Overall set of patches is: patches-over-a18dd1c42508.tar.gz
This has the board data patch rebased at the front, and two additional patches for fixing the sensor layout and the mode settings as suggested in #1 (comment).
@djrscally Is that something we can work with, or are there changes upstream again that need to be taken into account? Sorry for the huge delay...

@djrscally
Copy link

djrscally commented Jul 3, 2023

@akobel no need to apologise! I'm just catching up on things (been at a conference and prepping for that for the last two weeks so I have a lot to get through) and then I'll rebase the set onto linux-media for-next and we can post them - there are indeed upstream changes that need to be taken into account. There's a couple of comments I'd have out the gate though that we'll need to fix:

  • Patch 1 needs your Signed-of-by. I can add that as long as you're happy.
  • Patch 13 replaces the existing mode with the new 1920x1080 mode - we can't do that, really. We'd have to support both. It just means adding a new array of register address/value pairs and an entry to supported_modes.
  • Patch 12 defines a macro called OV2740_MEDIA_BUS_FMT - that should just be dropped and left as the MEDIA_BUS_FMT_* macro instead.

@akobel
Copy link
Owner Author

akobel commented Jul 3, 2023

@akobel no need to apologise! I'm just catching up on things (been at a conference and prepping for that for the last two weeks so I have a lot to get through) and then I'll rebase the set onto linux-media for-next and we can post them - there are indeed upstream changes that need to be taken into account.

Cool. I'll have another look then, and prepare for the following in the meanwhile (but I have a busy week ahead, hence unlikely that I can do it before mid-month).

There's a couple of comments I'd have out the gate though that we'll need to fix:

  • Patch 1 needs your Signed-of-by. I can add that as long as you're happy.

Sure, no problem (can add it myself, though; will anyways post you a new patchset, I guess, with the changes below).

  • Patch 13 replaces the existing mode with the new 1920x1080 mode - we can't do that, really. We'd have to support both. It just means adding a new array of register address/value pairs and an entry to supported_modes.

Hm, okay. The problem here is that the original definition never worked for me. For admittedly unclear reasons; it's not that it was invalid per se, as far as I can tell from the datasheet.
But honestly this looks like it wasn't properly tested ever (or at least I'd be fairly surprised, looking at the bug with the color coding). If it didn't work before, is it a regression? Not sure. But there might be a previous use case, so I can keep the old mode, too.

In any case, the mode name (mode_1932x1092) would need an adjustment. Can do that.

An additional complication might be that, according to my experiments, the register settings (especially timing control) need to be set to 1936x1096 (native size), and only then the cropping comes into effect. But OV2740_ACTIVE_START_TOP/LEFT are so large that 1932x1092 cannot be achieved with that crop window. So perhaps I should try and get rid of those macros predefined, and then try to adapt ov2740_get_selection appropriately.

  • Patch 12 defines a macro called OV2740_MEDIA_BUS_FMT - that should just be dropped and left as the MEDIA_BUS_FMT_* macro instead.

Reason was that I had to replace it at several locations (easy to grep, but still); so I reckoned it'd be good to only have one central definition for that. But I can change that, of course.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants