Repository navigation
[BUG][GLK]"DSP panic" when do module load/unload test #2646
Description
Describe the bug
"DSP panic" when do module load/unload test,Seems only a error message,has no imapct on function
After error occured,all the pipeline can work normally
To Reproduce
1.sudo reboot
2. cd sof-test
3.cd test-case
4.export TPLG=/lib/firmware/intel/sof-glk-da7219.tplg
5../check-kmod-load-unload.sh(https://github.com/thesofproject/sof-test/blob/master/test-case/check-kmod-load-unload.sh)
Reproduction Rate
10/10
Expected behavior
There is no DSP panic when do module load/unload
Impact
"DSP panic" showed in kernel log
Environment
* Kernel: {a272777} /sof/dev
* SOF: {9b1b615} master
2) Name of the topology file
* Topology: {sof-glk-da7219.tplg}
3) Name of the platform(s) on which the bug is observed.
* Platform: {GLK Chrome with onboard codec DA7219 in I2S mode}
dmesg_error.log
[ 92.559577] sof-audio-pci 0000:00:0e.0: periods:1
[ 92.560193] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x3030303 successful
[ 92.560221] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x3030302 successful
[ 92.560223] sof-audio-pci 0000:00:0e.0: unstall/run core: core_mask = 1
[ 92.560226] sof-audio-pci 0000:00:0e.0: DSP core(s) enabled? 1 : core_mask 1
[ 92.562330] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x40000000 successful
[ 92.562347] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x3010202 successful
[ 92.591822] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x5000001 successful
[ 92.608163] sof-audio-pci 0000:00:0e.0: error : DSP panic!
[ 92.608170] sof-audio-pci 0000:00:0e.0: unknown ROM status value 00000000
[ 92.608172] sof-audio-pci 0000:00:0e.0: error: status = 0x0dead000 panic = 0x00000220
[ 92.608192] sof-audio-pci 0000:00:0e.0: unknown ROM status value 00000000
[ 92.608815] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x5 successful
[ 92.609313] sof-audio-pci 0000:00:0e.0: FW Poll Status: reg=0x140000 successful
[ 92.609316] sof-audio-pci 0000:00:0e.0: Firmware download successful, booting...
[ 92.612338] sof-audio-pci 0000:00:0e.0: ipc rx: 0x70000000: FW_READY
[ 92.612341] sof-audio-pci 0000:00:0e.0: ipc: DSP is ready 0x70000000 offset 0x81000
[ 92.612358] sof-audio-pci 0000:00:0e.0: Firmware info: version 1:4:1-9b1b6
[ 92.612360] sof-audio-pci 0000:00:0e.0: Firmware: ABI 3:14:0 Kernel ABI 3:13:0
[ 92.612361] sof-audio-pci 0000:00:0e.0: warn: FW ABI is more recent than kernel
[ 92.612362] sof-audio-pci 0000:00:0e.0: Firmware debug build 1 on Mar 27 2020-05:17:14 - options:
GDB: disabled
lock debug: disabled
lock vdebug: disabled
[ 92.612390] sof-audio-pci 0000:00:0e.0: found ext header type 1 size 0xb8
[ 92.612403] sof-audio-pci 0000:00:0e.0: found ext header type 2 size 0x50
[ 92.612405] sof-audio-pci 0000:00:0e.0: Firmware info: used compiler XCC 12:0:8 RG-2017.8-linux� used optimization flags O2
[ 92.612420] sof-audio-pci 0000:00:0e.0: found ext header type 3 size 0x1c
[ 92.612422] sof-audio-pci 0000:00:0e.0: warning: unknown ext header type 3 size 0x1c
[ 92.612426] sof-audio-pci 0000:00:0e.0: found ext header type 4 size 0x10
[ 92.612427] sof-audio-pci 0000:00:0e.0: warning: unknown ext header type 4 size 0x10
[ 92.612457] sof-audio-pci 0000:00:0e.0: mailbox upstream 0x81000 - size 0x1000
[ 92.612459] sof-audio-pci 0000:00:0e.0: mailbox downstream 0xa0000 - size 0x2000
[ 92.612460] sof-audio-pci 0000:00:0e.0: stream region 0xc1000 - size 0x1000
[ 92.612464] sof-audio-pci 0000:00:0e.0: ipc rx done: 0x70000000: FW_READY
[ 92.612469] sof-audio-pci 0000:00:0e.0: firmware boot complete
[ 92.612475] sof-audio-pci 0000:00:0e.0: Debug PCIR: 00000000 at 00000048
No sof_errortrace
sof_trace0317.log
Activity
Liviali155 commentedon Mar 27, 2020
Changed the Firmware to 20191201(feb9e2b) also has this problem
Liviali155 commentedon Mar 27, 2020
On below platforms has no this issue
CML Chrome with onboard codec RT5682 and RT1011 in I2S mode
TGL-U RVP with onboard codec ALC711 in HDA mode
CML laptop with onboard codec ALC3204 in HDA mode
lgirdwood commentedon Mar 27, 2020
I see this twice in the logs before the FW boots. Could this be left over from a previous FW crash or @lbetlej is it a ROM error ?
xiulipan commentedon Apr 1, 2020
@lgirdwood the pattern is quite wired, I think there is a DSP panic happen before the FW load success. log from a PR CI test: https://sof-ci.01.org/sofpr/PR2680/build5647/devicetest/GLK_BOB_DA7219/check-kmod-load-unload/dmesg.txt
0x0dead000 => SOF_IPC_PANIC_MEMwenqingfu commentedon Apr 1, 2020
from CI results in a timeline, this is a regression started happening in the last two days.
lgirdwood commentedon Apr 1, 2020
@xiulipan panic does a
while(1)so should not get any further.@Liviali155 can you bisect given it's only started to happen in last 2 days.
Liviali155 commentedon Apr 2, 2020
I check the test result of CI,it should not be a regression,the result of 20200320 has DSP panic too,because the issue thesofproject/sof-test#154, the error was not found
dmesg_error.log
check-kmod-load-unload.sh.txt
cmd.txt
dmesg.txt
Liviali155 commentedon Apr 9, 2020
Tried with PR #1992 ,issue still can be reproduced
lgirdwood commentedon Apr 9, 2020
@Liviali155 can you update which PR you tested, above PR is a bug.
Please also put a space between "PR" and "#nnnn" as GH will then automatically insert a link for you.
Liviali155 commentedon Apr 9, 2020
OK,this PR thesofproject/linux#1992
Liviali155 commentedon Apr 10, 2020
Tried with PR #2751,issue still can be reproduced
tlauda commentedon Apr 10, 2020
@Liviali155 But PR #2751 has nothing to do with this bug. Why would you test with it?
20 remaining items
wenqingfu commentedon Apr 16, 2020
There was a false positive test case bug that we only recently corrected.
ranj063 commentedon Apr 16, 2020
@lgirdwood I increased the delay to 512 in lp_sram_init() and it didnt help.
keqiaozhang commentedon Apr 17, 2020
Bi-sect result points to:
commit c17de6a
Author: Marcin Rajwa marcin.rajwa@linux.intel.com
Date: Mon Apr 15 12:46:48 2019 +0200
memory: enable LPSRAM on APL
Signed-off-by: Marcin Rajwa marcin.rajwa@linux.intel.com
Kernel version for testing:thesofproject/linux@2291b8c
@mrajwa , please help to confirm if it's reasonable.
As we debugged yesterday, this issue cannot be reproduced after disabling LPSRAM check status.
Do you have any new findings after that?
mrajwa commentedon Apr 17, 2020
Yes, debugging is ongoing...in touch...
mrajwa commentedon Apr 21, 2020
@keqiaozhang, I know what is happening and how to fix it. However the rootcause is unusual bahaviour of the HW. So before I go into implemntation of the solution, I would like you to tell on how many GLK platforms this issue is observed? I want to rule out bad silicon part.
ranj063 commentedon Apr 21, 2020
@mrajwa iwe have observed this on our CI GLK systems and I have seen this on my yorp as well. So i doubt its the issue with one or two boards.
fredoh9 commentedon Apr 21, 2020
@mrajwa I found same problem in my GLK too. This was consistent problem for a while.
keqiaozhang commentedon Apr 22, 2020
@mrajwa , according to my observation, there is no GLK models that cannot reproduce this issue.
wenqingfu commentedon Apr 22, 2020
when can we have a patch? @mrajwa thanks!
mrajwa commentedon Apr 23, 2020
@wenqingfu, I have just pushed patches
Kernel -> thesofproject/linux#2044
FW -> #2837
sinahuang commentedon Apr 24, 2020
Check this issue with sof-dev (commit:8dc25) +PR thesofproject/linux#2044 and master (commit: 761bbac)+PR #2837, error can't be reproduced.
sinahuang commentedon May 13, 2020
Error can't be seen on sof-dev + master after thesofproject/linux#2044 merged, so closed it.