Fails to boot

[damanuel]

We are using a habanero board with firmware version OpenWrt-8devices v3.2 r10194+769-c53f62b111. We recently experience a problem where the board does not properly boot anymore. The serial boot message looks like this:

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.171 (baker@wrt_builder) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r10947-65030d81f3)) #0 SMP Thu Apr 9 21:53:20 2020
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: 8devices Habanero DVK
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] random: get_random_bytes called from start_kernel+0x8c/0x3d0 with crng_init=0
[    0.000000] percpu: Embedded 15 pages/cpu s29388 r8192 d23860 u61440
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129536
[    0.000000] Kernel command line: rootfstype=squashfs
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 509808K/522240K available (4526K kernel code, 153K rwdata, 1224K rodata, 1024K init, 241K bss, 12432K reserved, 0K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xe0800000 - 0xff800000   ( 496 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0208000 - 0xc076bbe0   (5519 kB)
[    0.000000]       .init : 0xc0900000 - 0xc0a00000   (1024 kB)
[    0.000000]       .data : 0xc0a00000 - 0xc0a26580   ( 154 kB)
[    0.000000]        .bss : 0xc0a28000 - 0xc0a644d8   ( 242 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] arch_timer: cp15 timer(s) running at 48.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb11fd3bfb, max_idle_ns: 440795203732 ns
[    0.000007] sched_clock: 56 bits at 48MHz, resolution 20ns, wraps every 4398046511096ns
[    0.000021] Switching to timer-based delay loop, resolution 20ns
[    0.000274] Calibrating delay loop (skipped), value calculated using timer frequency.. 96.00 BogoMIPS (lpj=480000)
[    0.000294] pid_max: default: 32768 minimum: 301
[    0.000439] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000454] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.001136] CPU: Testing write buffer coherency: ok
[    0.001849] Setting up static identity map for 0x80300000 - 0x80300060
[    0.001997] Hierarchical SRCU implementation.
[    0.002679] smp: Bringing up secondary CPUs ...
[    0.005481] smp: Brought up 1 node, 4 CPUs
[    0.005501] SMP: Total of 4 processors activated (384.00 BogoMIPS).
[    0.005509] CPU: All CPU(s) started in SVC mode.
[    0.010169] VFP support v0.3: implementor 41 architecture 2 part 30 variant 7 rev 5
[    0.010317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.010340] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.010621] pinctrl core: initialized pinctrl subsystem
[    0.011507] NET: Registered protocol family 16
[    0.011790] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.012799] cpuidle: using governor ladder
[    0.012841] cpuidle: using governor menu
[    0.031445] usbcore: registered new interface driver usbfs
[    0.031508] usbcore: registered new interface driver hub
[    0.031588] usbcore: registered new device driver usb
[    0.031642] pps_core: LinuxPPS API ver. 1 registered
[    0.031652] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.031676] PTP clock support registered
[    0.032849] clocksource: Switched to clocksource arch_sys_counter
[    0.033789] NET: Registered protocol family 2
[    0.034469] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    0.034532] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
[    0.034621] TCP: Hash tables configured (established 4096 bind 4096)
[    0.034742] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.034779] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.034971] NET: Registered protocol family 1
[    0.035970] No memory allocated for crashlog
[    0.036188] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    0.039418] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.039433] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.044923] io scheduler noop registered
[    0.044940] io scheduler deadline registered (default)
[    0.046781] OF: PCI: host bridge /soc/pci@40000000 ranges:
[    0.046820] OF: PCI:    IO 0x40200000..0x402fffff -> 0x40200000
[    0.046837] OF: PCI:   MEM 0x40300000..0x40ffffff -> 0x40300000
[    1.162858] qcom-pcie 40000000.pci: phy link never came up
[    1.172890] qcom-pcie 40000000.pci: cannot initialize host
[    1.173002] qcom-pcie: probe of 40000000.pci failed with error -110
[    1.174473] bam-dma-engine 8e04000.dma: num-channels unspecified in dt
[    1.174490] bam-dma-engine 8e04000.dma: num-ees unspecified in dt
[    1.175785] tcsr 1949000.tcsr: setting wifi_glb_cfg = 41000000
[    1.175869] tcsr 194b000.tcsr: setting usb hs phy mode select = e700e7
[    1.175943] tcsr 1953000.ess_tcsr: setting ess interface select = 0
[    1.176015] tcsr 1957000.tcsr: setting wifi_noc_memtype_m0_m2 = 2222222
[    1.176688] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    1.178443] msm_serial 78af000.serial: msm_serial: detected port #0
[    1.178491] msm_serial 78af000.serial: uartclk = 1843200
[    1.178547] 78af000.serial: ttyMSM0 at MMIO 0x78af000 (irq = 29, base_baud = 115200) is a MSM
[    1.178575] msm_serial: console setup on port #0
[    1.720441] console [ttyMSM0] enabled
[    1.725407] msm_serial 78b0000.serial: msm_serial: detected port #1
[    1.728679] msm_serial 78b0000.serial: uartclk = 48000000
[    1.734794] 78b0000.serial: ttyMSM1 at MMIO 0x78b0000 (irq = 30, base_baud = 3000000) is a MSM
[    1.740536] msm_serial: driver initialized
[    1.753373] loop: module loaded
[    1.754853] nand: No NAND device found
[    1.756592] spi_qup 78b5000.spi: IN:block:16, fifo:64, OUT:block:16, fifo:64
[    1.765904] random: fast init done
[    1.769529] m25p80 spi0.0: w25q256 (32768 Kbytes)
[    1.770192] 10 fixed-partitions partitions found on MTD device spi0.0
[    1.774939] Creating 10 MTD partitions on "spi0.0":
[    1.781356] 0x000000000000-0x000000040000 : "SBL1"
[    1.786750] 0x000000040000-0x000000060000 : "MIBIB"
[    1.791452] 0x000000060000-0x0000000c0000 : "QSEE"
[    1.796291] 0x0000000c0000-0x0000000d0000 : "CDT"
[    1.801095] 0x0000000d0000-0x0000000e0000 : "DDRPARAMS"
[    1.805928] 0x0000000e0000-0x0000000f0000 : "APPSBLENV"
[    1.810895] 0x0000000f0000-0x000000170000 : "APPSBL"
[    1.816132] 0x000000170000-0x000000180000 : "ART"
[    1.821346] 0x000000180000-0x0000001c0000 : "cfg"
[    1.825942] 0x0000001c0000-0x000002000000 : "firmware"
[    1.830962] 2 fit-fw partitions found on MTD device firmware
[    1.835093] Creating 2 MTD partitions on "firmware":
[    1.840902] 0x000000000000-0x000000240000 : "kernel"
[    1.846516] 0x000000240000-0x000001e40000 : "rootfs"
[    1.851335] mtd: device 11 (rootfs) set to be root filesystem
[    1.855925] 1 squashfs-split partitions found on MTD device rootfs
[    1.861383] 0x000000670000-0x000001e40000 : "rootfs_data"
[    1.868934] libphy: ipq40xx_mdio: probed
[    1.903863] ESS reset ok!
[    1.936848] ESS reset ok!
[    2.373224] libphy: Fixed MDIO Bus: probed
[    2.472985] i2c /dev entries driver
[    2.473283] i2c_qup 78b7000.i2c: using default clock-frequency 100000
[    2.475943] i2c_qup 78b8000.i2c: using default clock-frequency 100000
[    2.514026] NET: Registered protocol family 10
[    2.515497] Segment Routing with IPv6
[    2.517432] NET: Registered protocol family 17
[    2.521137] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    2.525818] 8021q: 802.1Q VLAN Support v1.8
�[    2.553594] VFS: Mounted root (squashfs filesystem) readonly on device 31:11.
[    2.555205] Freeing unused kernel memory: 1024K
[    3.393032] init: Console is alive
[    3.393272] init: - watchdog -
[    5.928277] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    6.144725] SCSI subsystem initialized
[    6.436715] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    6.436782] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[    6.441326] xhci-hcd xhci-hcd.0.auto: hcc params 0x0228f665 hci version 0x100 quirks 0x0000000002010010
[    6.448767] xhci-hcd xhci-hcd.0.auto: irq 132, io mem 0x08a00000
[    6.458957] hub 1-0:1.0: USB hub found
[    6.464374] hub 1-0:1.0: 1 port detected
[    6.468260] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    6.471938] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
[    6.477284] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0  SuperSpeed
[    6.484963] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    6.492076] hub 2-0:1.0: USB hub found
[    6.499880] hub 2-0:1.0: 1 port detected
[    6.503696] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    6.507356] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
[    6.512904] xhci-hcd xhci-hcd.1.auto: hcc params 0x0220f665 hci version 0x100 quirks 0x0000000002010010
[    6.520250] xhci-hcd xhci-hcd.1.auto: irq 133, io mem 0x06000000
[    6.530371] hub 3-0:1.0: USB hub found
[    6.535886] hub 3-0:1.0: 1 port detected
[    6.539789] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    6.543478] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
[    6.548750] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0  SuperSpeed
[    6.556404] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    6.563414] hub 4-0:1.0: USB hub found
[    6.571256] hub 4-0:1.0: config failed, hub doesn't have any ports! (err -19)
[    6.581704] sdhci: Secure Digital Host Controller Interface driver
[    6.581984] sdhci: Copyright(c) Pierre Ossman
[    6.588823] sdhci-pltfm: SDHCI platform and OF driver helper
[    6.594360] usbcore: registered new interface driver usb-storage
[    6.600946] sdhci_msm 7824900.sdhci: Got CD GPIO
[    6.604341] sdhci_msm 7824900.sdhci: core clock boost failed
[    6.685948] mmc0: SDHCI controller on 7824900.sdhci [7824900.sdhci] using ADMA 64-bit
[    6.689784] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    6.713192] init: - preinit -
[    7.886617] random: jshn: uninitialized urandom read (4 bytes read)
[    7.923121] random: jshn: uninitialized urandom read (4 bytes read)
[    8.017890] random: jshn: uninitialized urandom read (4 bytes read)
[    8.477584] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    8.477676] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
[   11.932926] jffs2: notice: (180) jffs2_build_xattr_subsystem: complete building xattr subsystem, 22 of xdatum (2 unchecked, 20 orphan) and 43 of xref (20 dead, 0 orphan) found.
[   11.975704] mount_root: switching to jffs2 overlay
[   12.005217] overlayfs: upper fs does not support tmpfile.
[   12.021693] urandom-seed: Seeding with /etc/urandom.seed
[   12.148054] procd: - early -
[   12.148194] procd: - watchdog -
[   12.804445] procd: - watchdog -
[   12.809308] procd: - ubus -
[   12.870984] urandom_read: 5 callbacks suppressed
[   12.870994] random: ubusd: uninitialized urandom read (4 bytes read)
[   13.862271] random: ubusd: uninitialized urandom read (4 bytes read)
[   13.863327] procd: - init -
Please press Enter to activate this console.
[   14.910849] random: crng init done

Failed to execute /usr/libexec/login enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
Please press Enter to activate this console.

Failed to execute /usr/libexec/login.sh
Please press Enter to activate this console.

After that, we can't login.

Going to failsafe mode and do a firstboot fixes the system. But the error has reoccurred, requiring us to do a system recovery again.

Do you have any idea what could be causing this? Thanks a lot!