[Bug 1699004] [NEW] APST gets enabled against explicit kernel option

[Nils]

Public bug reported:

I have a Lenovo ThinkPad X270 with a Toshiba nvme SSD

$ sudo nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 177S10WYTAMT THNSF5256GPUK TOSHIBA 1 256.06 GB / 256.06 GB 512 B + 0 B 51025KLA

I was affected by this bug
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1678184?comments=all
before. After disabling APST by adding "nvme_core.default_ps_max_latency_us=0" in /etc/default/grub, the bug went away.

Since yesterday, however the bug has returned, with the system dying
with I/O errors after an hour or so.

I verified, that the kernel option is still being set

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.10.0-22-generic.efi.signed root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 vt.handoff=7

however it turns out that it is being ignored for some reason, and
running

$ sudo nvme get-feature -f 0x0c -H /dev/nvme0

reports that APST is enabled. I can successfully disable it manually
using

$ sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0

and the problem goes away. However, after any reboot and even after
waking the system from suspend, it is reenabled, causing the system to
crash after a short while.

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: linux-image-4.10.0-22-generic 4.10.0-22.24
ProcVersionSignature: Ubuntu 4.10.0-22.24-generic 4.10.15
Uname: Linux 4.10.0-22-generic x86_64
ApportVersion: 2.20.4-0ubuntu4.1
Architecture: amd64
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/pcmC0D0p: maeher 1669 F...m pulseaudio
/dev/snd/controlC0: maeher 1669 F.... pulseaudio
CurrentDesktop: Unity:Unity7
Date: Mon Jun 19 23:29:31 2017
InstallationDate: Installed on 2017-04-17 (64 days ago)
InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: LENOVO 20HN001RUS
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.10.0-22-generic.efi.signed root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 vt.handoff=7
RelatedPackageVersions:
linux-restricted-modules-4.10.0-22-generic N/A
linux-backports-modules-4.10.0-22-generic N/A
linux-firmware 1.164.1
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/16/2017
dmi.bios.vendor: LENOVO
dmi.bios.version: R0IET30W (1.08 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20HN001RUS
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40697 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.modalias: dmi:bvnLENOVO:bvrR0IET30W(1.08):bd01/16/2017:svnLENOVO:pn20HN001RUS:pvrThinkPadX270:rvnLENOVO:rn20HN001RUS:rvrSDK0J40697WIN:cvnLENOVO:ct10:cvrNone:
dmi.product.name: 20HN001RUS
dmi.product.version: ThinkPad X270
dmi.sys.vendor: LENOVO

** Affects: linux (Ubuntu)
Importance: Undecided
Status: New


** Tags: amd64 apport-bug zesty
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

What's the value of `cat
/sys/module/nvme_core/parameters/default_ps_max_latency_us`?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Joseph Salisbury]

This change was made by a bot.

** Changed in: linux (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Please attach the value of `sudo find /sys -name '*latency*' | grep
nvme0 | xargs cat` as well.

** Changed in: linux (Ubuntu)
Assignee: (unassigned) => Kai-Heng Feng (kaihengfeng)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

I get

$ cat /sys/module/nvme_core/parameters/default_ps_max_latency_us
0

which kind of surprises me.

$ sudo find /sys -name '*latency*' | grep nvme0 | xargs cat

does not return anything.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Can you install latest mainline kernel [1], add "nvme_core.dyndbg=+p" to
your kernel parameter, and do

$ dmesg | grep -i nvme
$ sudo nvme get-feature -f 0x0c -H /dev/nvme0

[1] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12-rc5/linux-
headers-4.12.0-041200rc5-generic_4.12.0-041200rc5.201706112031_amd64.deb
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

I installed the kernel you linked to and set the kernel parameter.
On first boot the system immediately crashed with I/O errors upon login.
After reboot I could run the commands:

$ dmesg | grep -i nvme
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 2.225100] nvme nvme0: pci function 0000:04:00.0
[ 2.441319] nvme0n1: p1 p2
[ 4.220749] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 4.411077] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro


$ sudo nvme get-feature -f 0x0c -H /dev/nvme0
get-feature:0xc (Autonomous Power State Transition), Current value:0x000001
Autonomous Power State Transition Enable (APSTE): Enabled
Auto PST Entries .................
Entry[ 0]
.................
Idle Time Prior to Transition (ITPT): 50 ms
Idle Transition Power State (ITPS): 4
.................
Entry[ 1]
.................
Idle Time Prior to Transition (ITPT): 50 ms
Idle Transition Power State (ITPS): 4
.................
Entry[ 2]
.................
Idle Time Prior to Transition (ITPT): 50 ms
Idle Transition Power State (ITPS): 4
.................
Entry[ 3]
.................
Idle Time Prior to Transition (ITPT): 10000 ms
Idle Transition Power State (ITPS): 5
.................
Entry[ 4]
.................
Idle Time Prior to Transition (ITPT): 10000 ms
Idle Transition Power State (ITPS): 5
.................
Entry[ 5]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 5
.................
Entry[ 6]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[ 7]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[ 8]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[ 9]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[10]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[11]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[12]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[13]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[14]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[15]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[16]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[17]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[18]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[19]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[20]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[21]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[22]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[23]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[24]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[25]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[26]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[27]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[28]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[29]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[30]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
Entry[31]
.................
Idle Time Prior to Transition (ITPT): 0 ms
Idle Transition Power State (ITPS): 0
.................
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

It means the quirk doesn't work:

commit be56945c4edd5a3da15f8254b68d1ddb1588d0c4
Author: Andy Lutomirski <***@kernel.org>
Date: Thu Apr 20 13:37:56 2017 -0700

nvme: Quirk APST off on "THNSF5256GPUK TOSHIBA"

Can you do
$ dmesg | grep -i apst
instead?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

$ dmesg | grep -i apst

does not return anything.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

That's weird. 4.12-rc5 should have APST message if dynamic debug is
enabled.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

It is indeed very odd.

dmesg | grep dyndbg
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7

seems to indicate, that I DID actually set the kernel parameter.
Is there some way to check whether dynamic debug is ACTUALLY enabled?

Given that the system seems to quietly ignore
"nvme_core.default_ps_max_latency_us=0", maybe it also ignores
"nvme_core.dyndbg=+p".
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

I don't know (yet) how kernel parses its cmdline parameters - can you
remove "nvme_core.default_ps_max_latency_us=0" but keep
"nvme_core.dyndbg=+p", then do the grep again?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

$ dmesg | grep -i apst

still does not return anything

$ dmesg | grep -i nvme
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 2.230941] nvme nvme0: pci function 0000:04:00.0
[ 2.446574] nvme0n1: p1 p2
[ 4.145140] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 4.342813] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro

appears to be the same (minus the
"nvme_core.default_ps_max_latency_us=0")
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

"nvme_core.default_ps_max_latency_us=0" is still in the kernel
parameter.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

Sorry, you are indeed correct, apparently I forgot to update-grub.

Nevertheless, I get the same results

$ dmesg | grep -i apst
$ dmesg | grep -i nvme
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.dyndbg=+p vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-041200rc5-generic root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.dyndbg=+p vt.handoff=7
[ 2.230470] nvme nvme0: pci function 0000:04:00.0
[ 2.446516] nvme0n1: p1 p2
[ 4.291495] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 4.494027] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Please try this one:
http://people.canonical.com/~khfeng/lp1699004/

I added several messages to debug what code path it took.

Grep apst again:
$ dmesg | grep -i apst
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

Now I get the following:

$ dmesg | grep -i apst
[ 2.232133] nvme nvme0: apsta = 0, no APST
[ 2.448134] nvme 0000:04:00.0: NVME_QUIRK_NO_APST
[ 2.448135] nvme 0000:04:00.0: disabling APST.
[ 2.448137] nvme nvme0: apsta = 0, no APST

It seems like those messages are telling me that apst should be disabled?
However, I still get

sudo nvme get-feature -f 0x0c -H /dev/nvme0
get-feature:0xc (Autonomous Power State Transition), Current value:0x000001
Autonomous Power State Transition Enable (APSTE): Enabled
Auto PST Entries .................
Entry[ 0]
.................
Idle Time Prior to Transition (ITPT): 50 ms
Idle Transition Power State (ITPS): 4
.................
Entry[ 1]
.................
Idle Time Prior to Transition (ITPT): 50 ms
Idle Transition Power State (ITPS): 4
.................
Entry[ 2]
.................
Idle Time Prior to Transition (ITPT): 50 ms
Idle Transition Power State (ITPS): 4
.................
Entry[ 3]
.................
Idle Time Prior to Transition (ITPT): 10000 ms
Idle Transition Power State (ITPS): 5
.................
Entry[ 4]
.................
Idle Time Prior to Transition (ITPT): 10000 ms
Idle Transition Power State (ITPS): 5
...
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

So APST is not enabled in NVMe driver. I have no idea who does it,
probably BIOS or NVMe firmware?

Did you upgrade them before the issue reappear?

Anyway, we probably need to explicitly disable APST in the NVMe driver.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Please try http://people.canonical.com/~khfeng/lp1699004/linux-
image-4.12.0-rc6+_4.12.0-rc6+-3_amd64.deb
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

I have not upgraded the BIOS or the firmware, no.
This is really weird.

I tried the new kernel, however my system does not boot up with it.
It also does not show me an error that i can interpret, and the boot messages do not seem to be logged anywhere. I'm assuming that the system has problems writing the log-file, but I'm not sure.

The messages on screen when it stops booting do contain a call trace
that has something to do with nvme_core, but I'm not sure what it's
about. As I said I cannot find any logs.

Best I can do right now is a picture of the call trace as it appears on
screen.

** Attachment added: "Picture of Call Trace"
Loading Image...
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Built a new kernel for testing:
http://people.canonical.com/~khfeng/lp1699004/linux-image-4.12.0-rc6+_4.12.0-rc6+-6_amd64.deb

Also, you can remove "nvme_core.default_ps_max_latency_us=0" - the model
you use is already in the quirk table. Keep "nvme_core.dyndbg=+p" in the
meantime.

Attach `dmesg | grep -i nvme` afterwards.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

With the new kernel the system boots up.
I get the following:

$ dmesg | grep -i nvme
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-rc6+ root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.dyndbg=+p vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.0-rc6+ root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.dyndbg=+p vt.handoff=7
[ 2.228574] nvme nvme0: pci function 0000:04:00.0
[ 2.444448] nvme 0000:04:00.0: NVME_QUIRK_NO_APST
[ 2.444449] nvme 0000:04:00.0: disabling APST.
[ 2.444451] nvme nvme0: Disable APST at first time
[ 2.444452] nvme nvme0: APST disabled
[ 2.446363] nvme0n1: p1 p2
[ 4.140319] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 4.358751] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro

And indeed, this time APST is disabled!

$ sudo nvme get-feature -f 0x0c -H /dev/nvme0
get-feature:0xc (Autonomous Power State Transition), Current value:00000000
Autonomous Power State Transition Enable (APSTE): Disabled

So whatever it was you changes works! I hope it will be in the standard kernel soon.
Thanks a lot for taking the time to look into this, in particular because I seem to be the only person affected by it. I appreciate it.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Your system somehow saves APST config across reboots. The current code
logic will skip configuring APST if the device is in quirk table - so
the new APST value will never get set despite
default_ps_max_latency_us=0.

I'll send a patch for this issue. Also, thanks for your testing =)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Can you do one more testing? If now you boot with Ubuntu's 4.10 kernel
with default_ps_max_latency_us=0, is APST enabled or disabled?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

In that case it's enabled again:

$ dmesg | grep -i nvme
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.10.0-24-generic.efi.signed root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.10.0-24-generic.efi.signed root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 nvme_core.dyndbg=+p vt.handoff=7
[ 2.372993] nvme nvme0: pci function 0000:04:00.0
[ 2.587597] nvme0n1: p1 p2
[ 3.343336] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)
[ 3.548268] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro

sudo nvme get-feature -f 0x0c -H /dev/nvme0
get-feature:0xc (Autonomous Power State Transition), Current value:0x000001
Autonomous Power State Transition Enable (APSTE): Enabled
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Looks like your system enables APST by default. So we can't assume APST
is disabled by default - APST needs to be set on all circumstances.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Patch sent:
https://lists.infradead.org/pipermail/linux-nvme/2017-June/011382.html
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Andy Lutomirski]

Nils, what the actual failure mode on a bad kernel? Do you have dmesg
output?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

I'm not sure what exactly you're asking. The kernel above that does not
boot? The boot seems to fail before the drive is mounted, so I have no
logfiles whatsoever. The only thing I have is a photo of the last
messages on screen.

Or are you talking about something else?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Nils,

Andy means what exactly error message when booting with kernel like
4.10.0-22. Bad means "APST is enabled all the time" bad.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

** Description changed:

+ [Impact]
+ The NVMe driver doesn't set APSTE on APST quirked devices at initialization. If the BIOS or NVMe enables APST before driver loading, APST will never be disabled, it also ignores explicit kernel option as result.
+ So the faulty NVMe may not work as intended.
+
+ [Test Case]
+ $ sudo nvme get-feature -f 0x0c -H /dev/nvme0
+ ...will show APST is "Disabled" instead of "Enabled"
+
+ [Regression Potential]
+ Very low.
+ This SRU didn't change anything really - it just explicitly set APSTE at initialization.
+
+ ---
+
I have a Lenovo ThinkPad X270 with a Toshiba nvme SSD

$ sudo nvme list
- Node SN Model Namespace Usage Format FW Rev
+ Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 177S10WYTAMT THNSF5256GPUK TOSHIBA 1 256.06 GB / 256.06 GB 512 B + 0 B 51025KLA

I was affected by this bug
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1678184?comments=all
before. After disabling APST by adding "nvme_core.default_ps_max_latency_us=0" in /etc/default/grub, the bug went away.

Since yesterday, however the bug has returned, with the system dying
with I/O errors after an hour or so.

I verified, that the kernel option is still being set

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.10.0-22-generic.efi.signed root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 vt.handoff=7

however it turns out that it is being ignored for some reason, and
running

$ sudo nvme get-feature -f 0x0c -H /dev/nvme0

reports that APST is enabled. I can successfully disable it manually
using

$ sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0

and the problem goes away. However, after any reboot and even after
waking the system from suspend, it is reenabled, causing the system to
crash after a short while.

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: linux-image-4.10.0-22-generic 4.10.0-22.24
ProcVersionSignature: Ubuntu 4.10.0-22.24-generic 4.10.15
Uname: Linux 4.10.0-22-generic x86_64
ApportVersion: 2.20.4-0ubuntu4.1
Architecture: amd64
AudioDevicesInUse:
- USER PID ACCESS COMMAND
- /dev/snd/pcmC0D0p: maeher 1669 F...m pulseaudio
- /dev/snd/controlC0: maeher 1669 F.... pulseaudio
+  USER PID ACCESS COMMAND
+  /dev/snd/pcmC0D0p: maeher 1669 F...m pulseaudio
+  /dev/snd/controlC0: maeher 1669 F.... pulseaudio
CurrentDesktop: Unity:Unity7
Date: Mon Jun 19 23:29:31 2017
InstallationDate: Installed on 2017-04-17 (64 days ago)
InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: LENOVO 20HN001RUS
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.10.0-22-generic.efi.signed root=UUID=365f1a9c-9598-4ad5-a387-d02f771767a1 ro quiet splash nvme_core.default_ps_max_latency_us=0 vt.handoff=7
RelatedPackageVersions:
- linux-restricted-modules-4.10.0-22-generic N/A
- linux-backports-modules-4.10.0-22-generic N/A
- linux-firmware 1.164.1
+  linux-restricted-modules-4.10.0-22-generic N/A
+  linux-backports-modules-4.10.0-22-generic N/A
+  linux-firmware 1.164.1
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/16/2017
dmi.bios.vendor: LENOVO
dmi.bios.version: R0IET30W (1.08 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20HN001RUS
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40697 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.modalias: dmi:bvnLENOVO:bvrR0IET30W(1.08):bd01/16/2017:svnLENOVO:pn20HN001RUS:pvrThinkPadX270:rvnLENOVO:rn20HN001RUS:rvrSDK0J40697WIN:cvnLENOVO:ct10:cvrNone:
dmi.product.name: 20HN001RUS
dmi.product.version: ThinkPad X270
dmi.sys.vendor: LENOVO
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Seth Forshee]

** Also affects: linux (Ubuntu Zesty)
Importance: Undecided
Status: New

** Also affects: linux (Ubuntu Yakkety)
Importance: Undecided
Status: New

** Changed in: linux (Ubuntu)
Status: Confirmed => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Thadeu Lima de Souza Cascardo]

** Changed in: linux (Ubuntu Zesty)
Status: New => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Thadeu Lima de Souza Cascardo]

** Changed in: linux (Ubuntu Yakkety)
Status: New => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kleber Sacilotto de Souza]

This bug is awaiting verification that the kernel in -proposed solves
the problem. Please test the kernel and update this bug with the
results. If the problem is solved, change the tag 'verification-needed-
zesty' to 'verification-done-zesty'. If the problem still exists, change
the tag 'verification-needed-zesty' to 'verification-failed-zesty'.

If verification is not done by 5 working days from today, this fix will
be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how
to enable and use -proposed. Thank you!


** Tags added: verification-needed-zesty
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Nils]

I'm sorry, for taking so long to respond.

I do not have any logs, since they cannot be written to my hard drive in
the event of failure. (The SSD is my only hard drive and it can no
longer be accessed.)

But the error messages are something like:

EXT4-fs error (device nvme0n1p2): ext4_find_entry:1463: inode #12059331:
comm NetworkManager: reading directory lblock 0

Basically the only things that change are the specific inode numbers and
what I assume is the process name.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

@Nils,

Wha you need to do is to enable -proposed repository, and verify that
the Linux kernel in -proposed actually fixes the problem.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kai-Heng Feng]

Just realized you were replying previous comment...
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Kleber Sacilotto de Souza]

Hi @Nils,

Could you please test the Zesty kernel that's currently in proposed
(4.10.0-29.33)?

Thank you.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Launchpad Bug Tracker]

This bug was fixed in the package linux - 4.10.0-30.34

---------------
linux (4.10.0-30.34) zesty; urgency=low

* CVE-2017-7533
- dentry name snapshots

linux (4.10.0-29.33) zesty; urgency=low

* linux: 4.10.0-29.33 -proposed tracker (LP: #1704961)

* Opal and POWER9 DD2 (LP: #1702159)
- powerpc/powernv: Tell OPAL about our MMU mode on POWER9
- powerpc/powernv: Fix boot on Power8 bare metal due to opal_configure_cores()

* CVE-2017-1000364
- mm/mmap.c: do not blow on PROT_NONE MAP_FIXED holes in the stack
- mm/mmap.c: expand_downwards: don't require the gap if !vm_prev

* [Xenial] nvme: Quirks for PM1725 controllers (LP: #1704435)
- nvme: Quirks for PM1725 controllers

* hns: under heavy load, NIC may fail and require reboot (LP: #1704146)
- net: hns: Bugfix for Tx timeout handling in hns driver

* New ACPI identifiers for ThunderX SMMU (LP: #1703437)
- iommu/arm-smmu: Plumb in new ACPI identifiers

* CVE-2017-7482
- rxrpc: Fix several cases where a padded len isn't checked in ticket decode

* CVE-2017-1000365
- fs/exec.c: account for argv/envp pointers

* CVE-2017-10810
- drm/virtio: don't leak bo on drm_gem_object_init failure

* Data corruption with hio driver (LP: #1701316)
- SAUCE: hio: Fix incorrect use of enum req_opf values

* arm64: fix crash reading /proc/kcore (LP: #1702749)
- fs/proc: kcore: use kcore_list type to check for vmalloc/module address
- arm64: mm: select CONFIG_ARCH_PROC_KCORE_TEXT

* cxlflash update request in the Xenial SRU stream (LP: #1702521)
- scsi: cxlflash: Refactor context reset to share reset logic
- scsi: cxlflash: Support SQ Command Mode
- scsi: cxlflash: Cleanup prints
- scsi: cxlflash: Cancel scheduled workers before stopping AFU
- scsi: cxlflash: Enable PCI device ID for future IBM CXL Flash AFU
- scsi: cxlflash: Separate RRQ processing from the RRQ interrupt handler
- scsi: cxlflash: Serialize RRQ access and support offlevel processing
- scsi: cxlflash: Implement IRQ polling for RRQ processing
- scsi: cxlflash: Update sysfs helper routines to pass config structure
- scsi: cxlflash: Support dynamic number of FC ports
- scsi: cxlflash: Remove port configuration assumptions
- scsi: cxlflash: Hide FC internals behind common access routine
- scsi: cxlflash: SISlite updates to support 4 ports
- scsi: cxlflash: Support up to 4 ports
- scsi: cxlflash: Fence EEH during probe
- scsi: cxlflash: Remove unnecessary DMA mapping
- scsi: cxlflash: Fix power-of-two validations
- scsi: cxlflash: Fix warnings/errors
- scsi: cxlflash: Improve asynchronous interrupt processing
- scsi: cxlflash: Support multiple hardware queues
- scsi: cxlflash: Add hardware queues attribute
- scsi: cxlflash: Introduce hardware queue steering
- cxl: Enable PCI device IDs for future IBM CXL adapters
- scsi: cxlflash: Select IRQ_POLL
- scsi: cxlflash: Combine the send queue locks
- scsi: cxlflash: Update cxlflash_afu_sync() to return errno
- scsi: cxlflash: Reset hardware queue context via specified register
- scsi: cxlflash: Schedule asynchronous reset of the host
- scsi: cxlflash: Handle AFU sync failures
- scsi: cxlflash: Track pending scsi commands in each hardware queue
- scsi: cxlflash: Flush pending commands in cleanup path
- scsi: cxlflash: Add scsi command abort handler
- scsi: cxlflash: Create character device to provide host management interface
- scsi: cxlflash: Separate AFU internal command handling from AFU sync
specifics
- scsi: cxlflash: Introduce host ioctl support
- scsi: cxlflash: Refactor AFU capability checking
- scsi: cxlflash: Support LUN provisioning
- scsi: cxlflash: Support AFU debug
- scsi: cxlflash: Support WS16 unmap
- scsi: cxlflash: Remove zeroing of private command data
- scsi: cxlflash: Update TMF command processing
- scsi: cxlflash: Avoid double free of character device
- scsi: cxlflash: Update send_tmf() parameters
- scsi: cxlflash: Update debug prints in reset handlers

* Ath10k to read different board data file if specify in SMBIOS (LP: #1666742)
- ath10k: search SMBIOS for OEM board file extension

* APST gets enabled against explicit kernel option (LP: #1699004)
- nvme: Display raw APST configuration via DYNAMIC_DEBUG
- nvme: Add nvme_core.force_apst to ignore the NO_APST quirk
- nvme: explicitly disable APST on quirked devices

* Quirk for non-compliant PCI bridge on HiSilicon D05 board (LP: #1698706)
- SAUCE: PCI: Support hibmc VGA cards behind a misbehaving HiSilicon bridge

* New NVLINK2 patches (LP: #1701272)
- powerpc/powernv/npu-dma: Add explicit flush when sending an ATSD
- powerpc/npu-dma: Remove spurious WARN_ON when a PCI device has no of_node

* ERAT invalidate on context switch removal (LP: #1700819)
- powerpc: Only do ERAT invalidate on radix context switch on P9 DD1

* powerpc: Invalidate ERAT on powersave wakeup for POWER9 (LP: #1700521)
- powerpc/64s: Invalidate ERAT on powersave wakeup for POWER9

* update ENA driver to 1.2.0k from net-next (LP: #1701575)
- net/ena: use napi_complete_done() return value
- net: ena: remove superfluous check in ena_remove()
- net/ena: switch to pci_alloc_irq_vectors
- net: ena: fix rare uncompleted admin command false alarm
- net: ena: fix bug that might cause hang after consecutive open/close
interface.
- net: ena: add missing return when ena_com_get_io_handlers() fails
- net: ena: fix race condition between submit and completion admin command
- net: ena: add missing unmap bars on device removal
- net: ena: fix theoretical Rx hang on low memory systems
- net: ena: disable admin msix while working in polling mode
- net: ena: bug fix in lost tx packets detection mechanism
- net: ena: update ena driver to version 1.1.7
- net: ena: change return value for unsupported features unsupported return
value
- net: ena: add hardware hints capability to the driver
- net: ena: change sizeof() argument to be the type pointer
- net: ena: add reset reason for each device FLR
- net: ena: add support for out of order rx buffers refill
- net: ena: allow the driver to work with small number of msix vectors
- net: ena: use napi_schedule_irqoff when possible
- net: ena: separate skb allocation to dedicated function
- net: ena: use lower_32_bits()/upper_32_bits() to split dma address
- net: ena: update driver's rx drop statistics
- net: ena: update ena driver to version 1.2.0

-- Thadeu Lima de Souza Cascardo <***@canonical.com> Mon, 31 Jul
2017 14:27:53 -0300

** Changed in: linux (Ubuntu Zesty)
Status: Fix Committed => Fix Released

** CVE added: https://cve.mitre.org/cgi-
bin/cvename.cgi?name=2017-1000364

** CVE added: https://cve.mitre.org/cgi-
bin/cvename.cgi?name=2017-1000365

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-10810

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-7482

** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2017-7533
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1699004

Title:
APST gets enabled against explicit kernel option

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1699004/+subscriptions

[Launchpad Bug Tracker]

This post might be inappropriate. Click to display it.