end_request: critical target error on USB HDD after kernel upgrade #703

bosscyril · 2014-10-06T16:40:12Z

After upgrade to 3.12.28 (de69b134dc6e4066fe70db29816d57895dffd9b9) my raspberry started to show kernel errors while trying to write to USB HD, causing the filesystem to go into read-only mode.

[    3.318266] usb 1-1.2: new high-speed USB device number 4 using dwc_otg
[    3.448660] usb 1-1.2: New USB device found, idVendor=1a40, idProduct=0101
[    3.457251] usb 1-1.2: New USB device strings: Mfr=0, Product=1, SerialNumber=0
[    3.483453] usb 1-1.2: Product: USB 2.0 Hub
[    3.499054] hub 1-1.2:1.0: USB hub found
[    3.504713] hub 1-1.2:1.0: 4 ports detected
[    3.798172] usb 1-1.2.1: new high-speed USB device number 5 using dwc_otg
[    3.908850] usb 1-1.2.1: New USB device found, idVendor=059f, idProduct=0651
[    3.917474] usb 1-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    3.945489] usb 1-1.2.1: Product: LaCie Hard Drive USB
[    3.959613] usb 1-1.2.1: Manufacturer: LaCie
[    3.965447] usb 1-1.2.1: SerialNumber: 10000E000CDAA0F5
[    3.987634] usb-storage 1-1.2.1:1.0: USB Mass Storage device detected
[    4.008681] scsi0 : usb-storage 1-1.2.1:1.0
[    4.293223] udevd[158]: starting version 175
[    5.009144] scsi 0:0:0:0: Direct-Access     SEAGATE  ST3250820A       3.AA PQ: 0 ANSI: 2
[    5.039377] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB)
[    5.069351] sd 0:0:0:0: [sda] Write Protect is off
[    5.075778] sd 0:0:0:0: [sda] Mode Sense: 53 00 00 08
[    5.092152] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    5.152806]  sda: sda1 sda2 sda3
[    5.171294] sd 0:0:0:0: [sda] Attached SCSI disk
[    6.479249] bcm2708-i2s bcm2708-i2s.0: Failed to create debugfs directory
[    7.541473] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   11.052194] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   11.810684] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   18.189332] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[   18.379614] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[   18.591988] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
[   21.250969] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   22.870871] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0x45E1
[   23.254135] bcm2835-cpufreq: switching to governor ondemand
[   23.254168] bcm2835-cpufreq: switching to governor ondemand
[   25.312796] ip_tables: (C) 2000-2006 Netfilter Core Team
[   25.365061] nf_conntrack version 0.5.0 (3699 buckets, 14796 max)
[   26.733315] Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SSFS
[   74.207016] end_request: critical target error, dev sda, sector 404211137
[   74.207108] Aborting journal on device sda2-8.
[   74.364615] EXT4-fs error (device sda2): ext4_journal_check_start:56: Detected aborted journal
[   74.376026] EXT4-fs (sda2): Remounting filesystem read-only
[   96.341358] EXT4-fs error (device sda2): ext4_put_super:772: Couldn't clean up the journal

Then if I try to fsck it keeps throwing errors:

[  108.403251] end_request: critical target error, dev sda, sector 0
[  108.491317] end_request: critical target error, dev sda, sector 0
[  108.495367] end_request: critical target error, dev sda, sector 0
[  161.770145] end_request: critical target error, dev sda, sector 0
[  161.772852] end_request: critical target error, dev sda, sector 0
[  161.774345] end_request: critical target error, dev sda, sector 0

The issue does not reproduce in previous commits. Last working version is bdc1ecc6c89d23d9edd0ed23d2b3c13063b3beb7.

The text was updated successfully, but these errors were encountered:

popcornmix · 2014-10-06T17:50:47Z

Are you sure it's completely stable with:
Hexxeh/rpi-firmware@bdc1ecc
and completely broken with:
Hexxeh/rpi-firmware@de69b13
(i.e. not just an intermittent problem that happened to work with one firmware and fail with the other).

Nothing changed that could explain in Pi specific code. There were the upstream commits from 3.12.28, but I'd be a little surprised if that broke this.
Does the latest firmware (which is now 3.12.29) make any difference?

@P33M any thoughts? There were a couple of changes between 3.12.27 and 3.12.28 in drivers/usb/core.

bosscyril · 2014-10-06T22:45:31Z

Yes, I've done several tests with different versions and it just starts failing with 3.12.28. Latest firmware (de69b134dc6e4066fe70db29816d57895dffd9b9) makes no difference.

Also I've connected the hard drive to my computer with no problems, and even reproduced the issue in another Pi with same SD, so I think any hardware or HDD issues are discarded...

P33M · 2014-10-08T09:06:43Z

The error is -EREMOTEIO which is not something passed back by the dwc driver (unless something further down the stack mangles it).

There have been no firmware changes or dwc driver changes which may make hunting down this regression fun.

mod186k1 · 2014-10-25T20:28:56Z

I have the same issue: "end_request: critical target error, dev sda, sector 0" only whith the newest kernel 3.12.28+ (package raspberrypi-bootloader_1.20140908-1_armhf.deb).
Instead with the kernel 3.12.26+ (package raspberrypi-bootloader_1.20140830-1_armhf.deb) and the same USB HDD I have no error at all.

I have also tried armel version but I had no luck.

tsjk · 2014-11-10T15:52:51Z

I think this is a upstream problem. I stumbled upon it on a Gentoo x86_64 system. I upgraded from 3.12.21, with which my external Maxtor OneTouch was working ok with, to 3.16.5, which made me hit this. I experimented a bit, and it seems to affect kernels released later than September or so. I found that with 3.14.14 all works ok, but in 3.14.21 there is breakage.

sgebbie · 2014-11-11T20:13:26Z

I see exactly the same issues with:

root@raspbmc:~# uname -a
Linux raspbmc 3.12.31 #2 PREEMPT Wed Oct 29 09:24:56 UTC 2014 armv6l GNU/Linux

My USB drive also happens to be Seagate.

syslog:

Nov 11 21:08:28 raspbmc kernel: end_request: critical target error, dev sda, sector 2147747896
Nov 11 21:08:28 raspbmc kernel: Aborting journal on device sda1-8.
Nov 11 21:08:32 raspbmc kernel: EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
Nov 11 21:08:32 raspbmc kernel: EXT4-fs (sda1): Remounting filesystem read-only

partitions:

root@raspbmc:/var/log# fdisk -l

Disk /dev/mmcblk0: 3974 MB, 3974103040 bytes
4 heads, 16 sectors/track, 121280 cylinders, total 7761920 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000798a3

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            4096      147455       71680    c  W95 FAT32 (LBA)
/dev/mmcblk0p2          151552     7761919     3805184   83  Linux
Note: sector size is 4096 (not 512)

Disk /dev/sda: 3000.6 GB, 3000592965632 bytes
255 heads, 63 sectors/track, 45600 cylinders, total 732566642 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0002be9f

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1             256   536871167  2147483648   83  Linux

P33M · 2014-11-12T09:14:31Z

Nov 11 21:08:28 raspbmc kernel: end_request: critical target error, dev sda, sector 2147747896

I am fairly sure your drive doesn't have MAXINT-264,928 sectors. It's only got 732,566,642.

This smells like an upstream bug.

What triggers the behaviour?

sgebbie · 2014-11-12T09:34:12Z

Yes, I was curious about that. Especially that 'fdisk -l' lists the total number of blocks as 2147483648=2^31. I, however, wasn't sure relationship between 'blocks' and 'sectors'.

In terms of triggers, in the logs I see both:

Nov  9 15:56:20 raspbmc kernel: end_request: critical target error, dev sda, sector 0

and:

Nov 10 22:34:33 raspbmc kernel: end_request: critical target error, dev sda, sector 2147747872

The 'sector 2147747872' errors always lead to:

Nov 10 22:34:33 raspbmc kernel: Aborting journal on device sda1-8.
Nov 10 22:34:35 raspbmc kernel: EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
Nov 10 22:34:35 raspbmc kernel: EXT4-fs (sda1): Remounting filesystem read-only

sgebbie · 2014-11-12T09:57:33Z

Also, I realise this is known, but looking back through the syslogs I can confirm that the issue only showed up after upgrading kernels from 3.12.21 to 3.12.31:

# grep 'Linux version' syslog
...
Oct 23 20:53:34 raspbmc kernel: Linux version 3.12.21 (root@samnazarko-compiler) (gcc version 4.7.1 20120402 (prerelease) (crosstool-NG 1.15.2) ) #2 PREEMPT Wed Jun 11 04:53:06 UTC 2014
Nov  1 12:50:11 raspbmc kernel: Linux version 3.12.21 (root@samnazarko-compiler) (gcc version 4.7.1 20120402 (prerelease) (crosstool-NG 1.15.2) ) #2 PREEMPT Wed Jun 11 04:53:06 UTC 2014
Nov  1 17:05:52 raspbmc kernel: Linux version 3.12.31 (root@raspbmcbb-legacy) (gcc version 4.7.1 20120402 (prerelease) (crosstool-NG 1.15.2) ) #2 PREEMPT Wed Oct 29 09:24:56 UTC 2014
...

And:

# grep 'Aborting journal' syslog 
Nov  1 17:16:50 raspbmc kernel: Aborting journal on device sda1-8.
Nov  2 11:57:18 raspbmc kernel: Aborting journal on device sda1-8.
...

dod38fr · 2014-11-15T17:01:50Z

For what it's worth, I have a similar issue on Debian kernel 3.16 amd64 with one PATA/HDD behind a usb controller. No issue with kernel 3.14.15

To confuse the issue more, I have another PATA/HDD behind a different usb controller which works fine with both kernels.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=763966#10 for more details

bcdonadio · 2014-11-22T20:23:29Z

I have the same issue with Pidora and Raspbian (both with the 3.12 kernel). It's an external HDD with it's own power supply, nothing else connected to the USB ports. Already measured the TP1 against TP2 with a Fluke voltimeter, and it never drops below 5.015V. Also, to be sure, I tested with both a chinese 1A power brick, and an iPad 2A charger, and it's always reproducible. So, I don't think it's a power supply issue.

Right now I'm using the following kernel with Pidora: 3.12.26-1.20140808git4ab8abb.rpfr20.armv6hl.bcm2708

Also, it only happens when there's high activity on the HDD, like copying a lot of data or streaming a movie. It never happens when the HDD is idle. The HDD works wonderfully connected with either my PC (3.17 kernel amd64) or my OpenWRT router (3.3.8 kernel mips).

It sometimes also kill the networking (since it's also hooked in the USB bus), sometimes it does not.

Here's the lsusb output regarding the USB HDD:

Bus 001 Device 013: ID 4971:ce07 SimpleTech 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x4971 SimpleTech
  idProduct          0xce07 
  bcdDevice            0.00
  iManufacturer          10 Simpletech
  iProduct               11 SimpleDrive USB 2.0
  iSerial                 3 123457789CE8
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           32
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          4 USB Mass Storage
    bmAttributes         0xc0
      Self Powered
    MaxPower                2mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              6 MSC Bulk-Only Transfer
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
Device Status:     0x0701
  Self Powered

And here's the complete relevant output of the journal:

Nov 22 17:45:08 sentinel.donadio.be sudo[472]: bcdonadio : TTY=pts/2 ; PWD=/media/appz ; USER=root ; COMMAND=/bin/rm -rf teste2/
##FILE COPY STARTS HERE##
Nov 22 17:45:28 sentinel.donadio.be sudo[484]: bcdonadio : TTY=pts/2 ; PWD=/media/appz ; USER=root ; COMMAND=/bin/cp -r teste1/ teste2
##ERRORS STARTS HERE, NOTICE THE TIMEFRAME##
Nov 22 17:46:18 sentinel.donadio.be kernel: sd 0:0:0:0: Device offlined - not ready after error recovery
Nov 22 17:46:18 sentinel.donadio.be kernel: sd 0:0:0:0: [sda] Unhandled error code
Nov 22 17:46:18 sentinel.donadio.be kernel: sd 0:0:0:0: [sda]  
Nov 22 17:46:18 sentinel.donadio.be kernel: Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
Nov 22 17:46:18 sentinel.donadio.be kernel: sd 0:0:0:0: [sda] CDB: 
Nov 22 17:46:18 sentinel.donadio.be kernel: Write(10): 2a 00 13 5e 54 b0 00 00 f0 00
Nov 22 17:46:18 sentinel.donadio.be kernel: end_request: I/O error, dev sda, sector 324949168
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403734)
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403734
Nov 22 17:46:18 sentinel.donadio.be kernel: usb 1-1.3: USB disconnect, device number 8
Nov 22 17:46:18 sentinel.donadio.be kernel: scsi 0:0:0:0: rejecting I/O to offline device
Nov 22 17:46:18 sentinel.donadio.be kernel: scsi 0:0:0:0: [sda] killing request
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403794)
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403794
Nov 22 17:46:18 sentinel.donadio.be systemd[1]: Stopping LVM2 PV scan on device 8:1...
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403795
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403796
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403797
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403798
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403799
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403800
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403801
Nov 22 17:46:18 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14403802
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403824)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403854)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403884)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403914)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403944)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403974)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404004)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404034)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404064)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404094)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404124)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404154)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404184)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404214)
Nov 22 17:46:18 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404244)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404274)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404304)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404334)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404364)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404394)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404424)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404454)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404484)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404514)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404544)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404574)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404604)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404634)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404664)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404694)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404724)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404754)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404784)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404814)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404844)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404874)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404904)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404934)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404964)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14404994)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405024)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405054)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405084)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405114)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405144)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405174)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405204)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405234)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405264)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405294)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405324)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405354)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405384)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405414)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405444)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405474)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405504)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405534)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405564)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405594)
Nov 22 17:46:19 sentinel.donadio.be pvscan[496]: Device 8:1 not found. Cleared from lvmetad cache.
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14405624)
Nov 22 17:46:19 sentinel.donadio.be systemd[1]: Stopped LVM2 PV scan on device 8:1.
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405632)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405662)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405692)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405722)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405752)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405782)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405812)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405842)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405872)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405902)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405932)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405962)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14405992)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406022)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406052)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406082)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406112)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406142)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406172)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406202)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406232)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406262)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406292)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406322)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406352)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406382)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406412)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406442)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406472)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406502)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406532)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406562)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406592)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406622)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406652)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406682)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406712)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406742)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406772)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406802)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406832)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406862)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406892)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406922)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406952)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14406982)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407012)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407042)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407072)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407102)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407132)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407162)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407192)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407222)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407252)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407282)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407312)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407342)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407372)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407402)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407432)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407462)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407492)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407522)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407552)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407582)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407612)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407642)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 41943040 size 8388608 starting block 14407672)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407680)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407710)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407740)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407770)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407800)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407830)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407860)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407890)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407920)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407950)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14407980)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408010)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408040)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408070)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408100)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408130)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408160)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408190)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408220)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408250)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408280)
Nov 22 17:46:19 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408310)
Nov 22 17:46:19 sentinel.donadio.be kernel: scsi 0:0:0:0: [sda] Unhandled error code
Nov 22 17:46:19 sentinel.donadio.be kernel: scsi 0:0:0:0: [sda]  
Nov 22 17:46:19 sentinel.donadio.be kernel: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Nov 22 17:46:19 sentinel.donadio.be kernel: scsi 0:0:0:0: [sda] CDB: 
Nov 22 17:46:19 sentinel.donadio.be kernel: Write(10): 2a 00 13 5e 55 a0 00 00 f0 00
Nov 22 17:46:20 sentinel.donadio.be kernel: end_request: I/O error, dev sda, sector 324949408
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -5 writing to inode 10223621 (offset 33554432 size 8388608 starting block 14403764)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 2830336 starting block 14408340)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 2953216 starting block 14408370)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3076096 starting block 14408400)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3198976 starting block 14408430)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3321856 starting block 14408460)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3444736 starting block 14408490)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3567616 starting block 14408520)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3690496 starting block 14408550)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3813376 starting block 14408580)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 3936256 starting block 14408610)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4059136 starting block 14408640)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4182016 starting block 14408670)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4304896 starting block 14408700)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4427776 starting block 14408730)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4550656 starting block 14408760)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4673536 starting block 14408790)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4796416 starting block 14408820)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 4919296 starting block 14408850)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5042176 starting block 14408880)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5165056 starting block 14408910)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5287936 starting block 14408940)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5410816 starting block 14408970)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5533696 starting block 14409000)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5656576 starting block 14409030)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5779456 starting block 14409060)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 5902336 starting block 14409090)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6025216 starting block 14409120)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6148096 starting block 14409150)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6270976 starting block 14409180)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6393856 starting block 14409210)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6516736 starting block 14409240)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6639616 starting block 14409270)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6762496 starting block 14409300)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 6885376 starting block 14409330)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7008256 starting block 14409360)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7131136 starting block 14409390)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7254016 starting block 14409420)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7376896 starting block 14409450)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7499776 starting block 14409480)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7622656 starting block 14409510)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7745536 starting block 14409540)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7868416 starting block 14409570)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 7991296 starting block 14409600)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 8114176 starting block 14409630)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 8237056 starting block 14409660)
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 8359936 starting block 14409690)
Nov 22 17:46:20 sentinel.donadio.be kernel: Aborting journal on device dm-1-8.
Nov 22 17:46:20 sentinel.donadio.be kernel: EXT4-fs error (device dm-1) in __ext4_new_inode:820: Readonly filesystem
Nov 22 17:46:22 sentinel.donadio.be kernel: EXT4-fs warning (device dm-1): ext4_end_bio:316: I/O error -19 writing to inode 10223621 (offset 50331648 size 8388608 starting block 14409720)
Nov 22 17:46:22 sentinel.donadio.be kernel: EXT4-fs (dm-1): ext4_writepages: jbd2_start: -11264 pages, ino 10223621; err -30
Nov 22 17:46:22 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 26247168
Nov 22 17:46:22 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:22 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 0
Nov 22 17:46:22 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:22 sentinel.donadio.be kernel: JBD2: Error -5 detected when updating journal superblock for dm-1-8.
Nov 22 17:46:22 sentinel.donadio.be kernel: EXT4-fs (dm-1): previous I/O error to superblock detected
Nov 22 17:46:22 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 0
Nov 22 17:46:22 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:22 sentinel.donadio.be kernel: EXT4-fs (dm-1): previous I/O error to superblock detected
Nov 22 17:46:22 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 0
Nov 22 17:46:22 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:22 sentinel.donadio.be kernel: EXT4-fs error (device dm-1): ext4_journal_check_start:56: Detected aborted journal
Nov 22 17:46:23 sentinel.donadio.be kernel: EXT4-fs error (device dm-1): ext4_journal_check_start:56: 
Nov 22 17:46:23 sentinel.donadio.be kernel: 
Nov 22 17:46:23 sentinel.donadio.be kernel: EXT4-fs (dm-1): Remounting filesystem read-only
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: new high-speed USB device number 9 using dwc_otg
Nov 22 17:46:23 sentinel.donadio.be kernel: Detected aborted journal
Nov 22 17:46:23 sentinel.donadio.be kernel: EXT4-fs (dm-1): Remounting filesystem read-only
Nov 22 17:46:23 sentinel.donadio.be kernel: EXT4-fs (dm-1): previous I/O error to superblock detected
Nov 22 17:46:23 sentinel.donadio.be kernel: EXT4-fs (dm-1): previous I/O error to superblock detected
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/64, error -32
Nov 22 17:46:23 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 0
Nov 22 17:46:23 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:23 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 0
Nov 22 17:46:23 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:23 sentinel.donadio.be kernel: EXT4-fs (dm-1): ext4_writepages: jbd2_start: 1024 pages, ino 10223621; err -30
Nov 22 17:46:23 sentinel.donadio.be kernel: Buffer I/O error on device dm-1, logical block 14155782
Nov 22 17:46:23 sentinel.donadio.be kernel: lost page write due to I/O error on dm-1
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/64, error -32
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: new high-speed USB device number 10 using dwc_otg
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/64, error -32
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/64, error -32
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: new high-speed USB device number 11 using dwc_otg
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/8, error -71
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/8, error -71
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: new high-speed USB device number 12 using dwc_otg
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/8, error -71
Nov 22 17:46:23 sentinel.donadio.be kernel: usb 1-1.3: device descriptor read/8, error -71
Nov 22 17:46:23 sentinel.donadio.be kernel: hub 1-1:1.0: unable to enumerate USB device on port 3

bcdonadio · 2014-11-23T00:32:49Z

I confirm the issue also happens with 3.12.23.

Latvala · 2014-12-12T19:03:17Z

Is there anything new about this? In ubuntu this have been propably fixed? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1366538

And I have the same bug with two old Lacie 250Gb usb hard drives.

popcornmix · 2014-12-12T19:47:01Z

Interesting. I could try reverting 89fb4cd

Latvala · 2014-12-13T06:21:12Z

And now I tested that 3.12.26+ works good but 3.12.34+ doesn't. I hope You can find the answer...

This reverts commit 89fb4cd. See: #703

…error See: raspberrypi/linux#703 kernel: Fix for bcm2708 frame buffer does not work without fbcon See: raspberrypi/linux#731 firmware: Improvements to logging with device tree manipulation

popcornmix · 2014-12-13T16:08:04Z

I've reverted the commit ubuntu identified.
It is in latest rpi-update firmware. Please test and report here.

Latvala · 2014-12-13T16:30:41Z

Thanks a lot!
It looks like the newest firmware 3.12.34+ works now when I ran rpi-update.
No more corruption anymore with 250gb old lacie.

bcdonadio · 2014-12-22T22:23:58Z

With 3.12.35+ I couldn't reproduce the bug anymore. I think it's fixed.

Thank you very much. ^^

popcornmix · 2014-12-22T22:59:56Z

Great.

bcdonadio · 2014-12-24T23:14:32Z

False alarm. I got it again with 3.12.35+ downloading a torrent in the HDD.

Dec 24 22:58:32 sentinel kernel: [12766.548011] usb 1-1.2: reset high-speed USB device number 7 using dwc_otg
Dec 24 22:58:32 sentinel kernel: [12766.648090] usb 1-1.2: device firmware changed
Dec 24 22:58:32 sentinel kernel: [12766.648495] usb 1-1.2: USB disconnect, device number 7
Dec 24 22:58:32 sentinel kernel: [12766.652855] sd 0:0:0:0: Device offlined - not ready after error recovery
Dec 24 22:58:32 sentinel kernel: [12766.654603] scsi 0:0:0:0: rejecting I/O to offline device
Dec 24 22:58:32 sentinel kernel: [12766.654646] scsi 0:0:0:0: [sda] killing request
Dec 24 22:58:32 sentinel kernel: [12766.654737] scsi 0:0:0:0: [sda] Unhandled error code
Dec 24 22:58:32 sentinel kernel: [12766.654758] scsi 0:0:0:0: [sda]  
Dec 24 22:58:32 sentinel kernel: [12766.654770] Result: hostbyte=0x01 driverbyte=0x00
Dec 24 22:58:32 sentinel kernel: [12766.654783] scsi 0:0:0:0: [sda] CDB: 
Dec 24 22:58:32 sentinel kernel: [12766.654794] cdb[0]=0x28: 28 00 3e 48 3e 50 00 00 f0 00
Dec 24 22:58:32 sentinel kernel: [12766.654838] end_request: I/O error, dev sda, sector 1044921936
Dec 24 22:58:32 sentinel kernel: [12766.654938] scsi 0:0:0:0: rejecting I/O to offline device
Dec 24 22:58:32 sentinel kernel: [12766.654962] scsi 0:0:0:0: [sda] killing request
Dec 24 22:58:32 sentinel kernel: [12766.655226] EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 3145951 (offset 35569664 size 3739648 starting block 36035052)
Dec 24 22:58:32 sentinel kernel: [12766.655253] Buffer I/O error on device dm-0, logical block 36035052
###LOTS OF REPETITIONS OF THE ABOVE LINE###
Dec 24 22:58:32 sentinel kernel: [12766.655479] EXT4-fs warning (device dm-0): ext4_end_bio:316: I/O error -5 writing to inode 3145951 (offset 35569664 size 3739648 starting block 36035082)
###LOTS OF REPETITIONS OF THE ABOVE LINE###
Dec 24 22:58:32 sentinel kernel: [12766.669851] scsi 0:0:0:0: [sda] Unhandled error code
Dec 24 22:58:32 sentinel kernel: [12766.669911] scsi 0:0:0:0: [sda]  
Dec 24 22:58:32 sentinel kernel: [12766.669926] Result: hostbyte=0x01 driverbyte=0x00
Dec 24 22:58:32 sentinel kernel: [12766.669941] scsi 0:0:0:0: [sda] CDB: 
Dec 24 22:58:32 sentinel kernel: [12766.669952] cdb[0]=0x28: 28 00 3e 48 3f 40 00 00 f0 00
Dec 24 22:58:32 sentinel kernel: [12766.670002] end_request: I/O error, dev sda, sector 1044922176
Dec 24 22:58:32 sentinel kernel: [12766.678030] Aborting journal on device dm-0-8.
Dec 24 22:58:32 sentinel kernel: [12766.678137] Buffer I/O error on device dm-0, logical block 13139968
Dec 24 22:58:32 sentinel kernel: [12766.678151] lost page write due to I/O error on dm-0
Dec 24 22:58:32 sentinel kernel: [12766.678169] JBD2: Error -5 detected when updating journal superblock for dm-0-8.
Dec 24 22:58:32 sentinel kernel: [12766.758222] usb 1-1.2: new high-speed USB device number 8 using dwc_otg
Dec 24 22:58:32 sentinel kernel: [12766.859582] usb 1-1.2: New USB device found, idVendor=152d, idProduct=2329
Dec 24 22:58:32 sentinel kernel: [12766.859618] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=5
Dec 24 22:58:32 sentinel kernel: [12766.859634] usb 1-1.2: Product: USB to ATA/ATAPI Bridge
Dec 24 22:58:32 sentinel kernel: [12766.859651] usb 1-1.2: Manufacturer: JMicron
Dec 24 22:58:32 sentinel kernel: [12766.859664] usb 1-1.2: SerialNumber: F6071DFFFFFF
Dec 24 22:58:32 sentinel kernel: [12766.864403] usb-storage 1-1.2:1.0: USB Mass Storage device detected
Dec 24 22:58:32 sentinel kernel: [12766.869150] usb-storage 1-1.2:1.0: Quirks match for vid 152d pid 2329: 8020
Dec 24 22:58:32 sentinel kernel: [12766.878896] scsi1 : usb-storage 1-1.2:1.0
Dec 24 22:58:33 sentinel kernel: [12767.879562] scsi 1:0:0:0: Direct-Access     Hitachi  HDT721010SLA360       PQ: 0 ANSI: 2 CCS
Dec 24 22:58:33 sentinel kernel: [12767.884680] sd 1:0:0:0: Attached scsi generic sg0 type 0
Dec 24 22:58:33 sentinel kernel: [12767.885673] sd 1:0:0:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
Dec 24 22:58:33 sentinel kernel: [12767.887953] sd 1:0:0:0: [sdb] Write Protect is off
Dec 24 22:58:33 sentinel kernel: [12767.887992] sd 1:0:0:0: [sdb] Mode Sense: 34 00 00 00
Dec 24 22:58:33 sentinel kernel: [12767.891986] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Dec 24 22:58:33 sentinel kernel: [12767.944327]  sdb: sdb1
Dec 24 22:58:33 sentinel kernel: [12767.949104] sd 1:0:0:0: [sdb] Attached SCSI disk
Dec 24 22:58:39 sentinel kernel: [12773.834966] Buffer I/O error on device dm-0, logical block 0
Dec 24 22:58:39 sentinel kernel: [12773.834997] lost page write due to I/O error on dm-0
Dec 24 22:58:39 sentinel kernel: [12773.835017] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
Dec 24 22:58:39 sentinel kernel: [12773.850031] EXT4-fs (dm-0): Remounting filesystem read-only
Dec 24 22:58:39 sentinel kernel: [12773.857155] EXT4-fs (dm-0): previous I/O error to superblock detected
Dec 24 22:58:39 sentinel kernel: [12773.857275] Buffer I/O error on device dm-0, logical block 0
Dec 24 22:58:39 sentinel kernel: [12773.857291] lost page write due to I/O error on dm-0
Dec 24 23:05:24 sentinel kernel: [13178.171985] EXT4-fs error (device dm-1): ext4_read_inode_bitmap:174: comm vim: Cannot read inode bitmap - block_group = 0, inode_bitmap = 1041
Dec 24 23:05:24 sentinel kernel: [13178.189448] Buffer I/O error on device dm-1, logical block 0
Dec 24 23:05:24 sentinel kernel: [13178.189476] lost page write due to I/O error on dm-1
Dec 24 23:05:24 sentinel kernel: [13178.189780] EXT4-fs (dm-1): previous I/O error to superblock detected
Dec 24 23:05:24 sentinel kernel: [13178.189828] Buffer I/O error on device dm-1, logical block 0
Dec 24 23:05:24 sentinel kernel: [13178.189842] lost page write due to I/O error on dm-1
Dec 24 23:05:24 sentinel kernel: [13178.189864] EXT4-fs error (device dm-1): __ext4_get_inode_loc:3941: inode #2: block 1057: comm vim: unable to read itable block
Dec 24 23:05:24 sentinel kernel: [13178.207083] EXT4-fs error (device dm-1) in ext4_reserve_inode_write:4891: IO failure
Dec 24 23:05:24 sentinel kernel: [13178.219001] EXT4-fs (dm-1): previous I/O error to superblock detected
Dec 24 23:05:24 sentinel kernel: [13178.219100] Buffer I/O error on device dm-1, logical block 0
Dec 24 23:05:24 sentinel kernel: [13178.219116] lost page write due to I/O error on dm-1
Dec 24 23:05:24 sentinel kernel: [13178.219412] EXT4-fs error (device dm-1): ext4_read_inode_bitmap:174: comm vim: Cannot read inode bitmap - block_group = 0, inode_bitmap = 1041
###LOTS OF REPETITIONS OF THE ABOVE LINES###
Dec 24 23:05:29 sentinel kernel: [13183.457760] lost page write due to I/O error on dm-1
Dec 24 23:05:29 sentinel kernel: [13183.457774] JBD2: Error -5 detected when updating journal superblock for dm-1-8.

Latvala · 2014-12-25T03:49:59Z

Strange. I have used my old lacie and now it has worked very well. Without reverting the commit from newest kernel the ext3/ext4/xfs partition ccorrupted and became read-only immediately. It's interesting that you have problems with network too...

popcornmix · 2014-12-25T14:30:08Z

@bcdonadio this doesn't sound like the same issue (No "end_request: critical target error" in log).
The reset/disconnect messages are often caused by too low a voltage from power supply.
Measuring the voltage would be useful: http://elinux.org/R-Pi_Troubleshooting#Troubleshooting_power_problems

bcdonadio · 2014-12-25T17:29:11Z

Indeed, when the issue occurs also there's a network freeze. However, I don't think it's a power source issue, since the HDD is self powered...

Anyway, I will try to use an iPad power brick and try to reproduce the behaviour. If I find it again, should I file a new bug or report it here?

(And BTW, the storytales that the microUSB connector may be causing too much of a voltage drop holds true? If so, what's the usual solution?)

popcornmix · 2014-12-28T11:44:31Z

Some self-powered disks only power the 12V and may still draw significant power from 5V supply (from Pi).
Ipad chargers tend to produce about 4.5V under load so aren't recommended.
The micro USB connector should be fine for 2A, so is unlikely to be the problem. The USB power cable is often the source of the voltage drop.

Measure the voltage to be sure:
http://elinux.org/R-Pi_Troubleshooting#Troubleshooting_power_problems
or use a recommended supply:
http://swag.raspberrypi.org/products/raspberry-pi-universal-power-supply

bosscyril · 2015-04-14T13:43:51Z

I think this issue have risen again, the only difference I see is that the log message changed from "end_request" to "blk_update_request", but behaviour seems the same:

Apr 14 13:00:39 raspberrypi kernel: [  561.226469] blk_update_request: critical target error, dev sda, sector 404211137
Apr 14 13:00:39 raspberrypi kernel: [  561.226579] Aborting journal on device sda2-8.
Apr 14 13:00:40 raspberrypi kernel: [  563.043697] EXT4-fs error (device sda2): ext4_journal_check_start:56: Detected aborted journal
Apr 14 13:00:40 raspberrypi kernel: [  563.057031] EXT4-fs (sda2): Remounting filesystem read-only
Apr 14 13:00:51 raspberrypi kernel: [  574.083665] EXT4-fs error (device sda2): ext4_put_super:789: Couldn't clean up the journal

If I try to fsck:

Apr 14 12:45:24 raspberrypi kernel: [ 1603.653290] EXT4-fs error (device sda2): ext4_put_super:789: Couldn't clean up the journal
Apr 14 12:45:30 raspberrypi kernel: [ 1610.138494] blk_update_request: critical target error, dev sda, sector 0
Apr 14 12:45:30 raspberrypi kernel: [ 1610.197294] blk_update_request: critical target error, dev sda, sector 0
Apr 14 12:45:30 raspberrypi kernel: [ 1610.203760] blk_update_request: critical target error, dev sda, sector 0
Apr 14 12:46:41 raspberrypi kernel: [ 1681.278773] blk_update_request: critical target error, dev sda, sector 0
Apr 14 12:46:41 raspberrypi kernel: [ 1681.282161] blk_update_request: critical target error, dev sda, sector 0
Apr 14 12:46:41 raspberrypi kernel: [ 1681.283998] blk_update_request: critical target error, dev sda, sector 0

Based on my tests, last working commit is Hexxeh/rpi-firmware@f74b921

Broken from Hexxeh/rpi-firmware@cad071a and onwards.

See: http://www.raspberrypi.org/forums/viewtopic.php?f=29&t=93015

…error See: raspberrypi/linux#703 kernel: Fix for bcm2708 frame buffer does not work without fbcon See: raspberrypi/linux#731 firmware: Improvements to logging with device tree manipulation

Geneve implementation changed mlx5 tc to user direct pointer to tunnel_key action's internal struct ip_tunnel_info instance. However, this leads to use-after-free error when initial filter that caused creation of new encap entry is deleted or when tunnel_key action is manually overwritten through action API. Moreover, with recent TC offloads API unlocking change struct flow_action_entry->tunnel point to temporal copy of tunnel info that is deallocated after filter is offloaded to hardware which causes bug to reproduce every time new filter is attached to existing encap entry with following KASAN bug: [ 314.885555] ================================================================== [ 314.886641] BUG: KASAN: use-after-free in memcmp+0x2c/0x60 [ 314.886864] Read of size 1 at addr ffff88886c746280 by task tc/2682 [ 314.887179] CPU: 22 PID: 2682 Comm: tc Not tainted 5.3.0-rc7+ #703 [ 314.887188] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017 [ 314.887195] Call Trace: [ 314.887215] dump_stack+0x9a/0xf0 [ 314.887236] print_address_description+0x67/0x323 [ 314.887248] ? memcmp+0x2c/0x60 [ 314.887257] ? memcmp+0x2c/0x60 [ 314.887272] __kasan_report.cold+0x1a/0x3d [ 314.887474] ? __mlx5e_tc_del_fdb_peer_flow+0x100/0x1b0 [mlx5_core] [ 314.887484] ? memcmp+0x2c/0x60 [ 314.887509] kasan_report+0xe/0x12 [ 314.887521] memcmp+0x2c/0x60 [ 314.887662] mlx5e_tc_add_fdb_flow+0x51b/0xbe0 [mlx5_core] [ 314.887838] ? mlx5e_encap_take+0x110/0x110 [mlx5_core] [ 314.887902] ? lockdep_init_map+0x87/0x2c0 [ 314.887924] ? __init_waitqueue_head+0x4f/0x60 [ 314.888062] ? mlx5e_alloc_flow.isra.0+0x18c/0x1c0 [mlx5_core] [ 314.888207] __mlx5e_add_fdb_flow+0x2d7/0x440 [mlx5_core] [ 314.888359] ? mlx5e_tc_update_neigh_used_value+0x6f0/0x6f0 [mlx5_core] [ 314.888374] ? match_held_lock+0x2e/0x240 [ 314.888537] mlx5e_configure_flower+0x830/0x16a0 [mlx5_core] [ 314.888702] ? __mlx5e_add_fdb_flow+0x440/0x440 [mlx5_core] [ 314.888713] ? down_read+0x118/0x2c0 [ 314.888728] ? down_read_killable+0x300/0x300 [ 314.888882] ? mlx5e_rep_get_ethtool_stats+0x180/0x180 [mlx5_core] [ 314.888899] tc_setup_cb_add+0x127/0x270 [ 314.888937] fl_hw_replace_filter+0x2ac/0x380 [cls_flower] [ 314.888976] ? fl_hw_destroy_filter+0x1b0/0x1b0 [cls_flower] [ 314.888990] ? fl_change+0xbcf/0x27ef [cls_flower] [ 314.889030] ? fl_change+0xa57/0x27ef [cls_flower] [ 314.889069] fl_change+0x16bd/0x27ef [cls_flower] [ 314.889135] ? __rhashtable_insert_fast.constprop.0+0xa00/0xa00 [cls_flower] [ 314.889167] ? __radix_tree_lookup+0xa4/0x130 [ 314.889200] ? fl_get+0x169/0x240 [cls_flower] [ 314.889218] ? fl_walk+0x230/0x230 [cls_flower] [ 314.889249] tc_new_tfilter+0x5e1/0xd40 [ 314.889281] ? __rhashtable_insert_fast.constprop.0+0xa00/0xa00 [cls_flower] [ 314.889309] ? tc_del_tfilter+0xa30/0xa30 [ 314.889335] ? __lock_acquire+0x5b5/0x2460 [ 314.889378] ? find_held_lock+0x85/0xa0 [ 314.889442] ? tc_del_tfilter+0xa30/0xa30 [ 314.889465] rtnetlink_rcv_msg+0x4ab/0x5f0 [ 314.889488] ? rtnl_dellink+0x490/0x490 [ 314.889518] ? lockdep_hardirqs_on+0x260/0x260 [ 314.889538] ? netlink_deliver_tap+0xab/0x5a0 [ 314.889550] ? match_held_lock+0x1b/0x240 [ 314.889575] netlink_rcv_skb+0xd0/0x200 [ 314.889588] ? rtnl_dellink+0x490/0x490 [ 314.889605] ? netlink_ack+0x440/0x440 [ 314.889635] ? netlink_deliver_tap+0x161/0x5a0 [ 314.889648] ? lock_downgrade+0x360/0x360 [ 314.889657] ? lock_acquire+0xe5/0x210 [ 314.889686] netlink_unicast+0x296/0x350 [ 314.889707] ? netlink_attachskb+0x390/0x390 [ 314.889726] ? _copy_from_iter_full+0xe0/0x3a0 [ 314.889738] ? __virt_addr_valid+0xbb/0x130 [ 314.889771] netlink_sendmsg+0x394/0x600 [ 314.889800] ? netlink_unicast+0x350/0x350 [ 314.889817] ? move_addr_to_kernel.part.0+0x90/0x90 [ 314.889852] ? netlink_unicast+0x350/0x350 [ 314.889872] sock_sendmsg+0x96/0xa0 [ 314.889891] ___sys_sendmsg+0x482/0x520 [ 314.889919] ? copy_msghdr_from_user+0x250/0x250 [ 314.889930] ? __fput+0x1fa/0x390 [ 314.889941] ? task_work_run+0xb7/0xf0 [ 314.889957] ? exit_to_usermode_loop+0x117/0x120 [ 314.889972] ? entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 314.889982] ? do_syscall_64+0x74/0xe0 [ 314.889992] ? entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 314.890012] ? mark_lock+0xac/0x9a0 [ 314.890028] ? __lock_acquire+0x5b5/0x2460 [ 314.890053] ? mark_lock+0xac/0x9a0 [ 314.890083] ? __lock_acquire+0x5b5/0x2460 [ 314.890112] ? match_held_lock+0x1b/0x240 [ 314.890144] ? __fget_light+0xa1/0xf0 [ 314.890166] ? sockfd_lookup_light+0x91/0xb0 [ 314.890187] __sys_sendmsg+0xba/0x130 [ 314.890201] ? __sys_sendmsg_sock+0xb0/0xb0 [ 314.890225] ? __blkcg_punt_bio_submit+0xd0/0xd0 [ 314.890264] ? lockdep_hardirqs_off+0xbe/0x100 [ 314.890274] ? mark_held_locks+0x24/0x90 [ 314.890286] ? do_syscall_64+0x1e/0xe0 [ 314.890308] do_syscall_64+0x74/0xe0 [ 314.890325] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 314.890336] RIP: 0033:0x7f00ca33d7b8 [ 314.890348] Code: 89 02 48 c7 c0 ff ff ff ff eb bb 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 65 8f 0c 00 8b 00 85 c0 75 17 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 48 83 ec 28 89 5 4 [ 314.890356] RSP: 002b:00007ffea2983928 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 314.890369] RAX: ffffffffffffffda RBX: 000000005d777d5b RCX: 00007f00ca33d7b8 [ 314.890377] RDX: 0000000000000000 RSI: 00007ffea2983990 RDI: 0000000000000003 [ 314.890384] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000006 [ 314.890392] R10: 0000000000404eda R11: 0000000000000246 R12: 0000000000000001 [ 314.890400] R13: 000000000047f640 R14: 00007ffea2987b58 R15: 0000000000000021 [ 314.890529] Allocated by task 2687: [ 314.890684] save_stack+0x1b/0x80 [ 314.890694] __kasan_kmalloc.constprop.0+0xc2/0xd0 [ 314.890705] __kmalloc_track_caller+0x102/0x340 [ 314.890721] kmemdup+0x1d/0x40 [ 314.890730] tc_setup_flow_action+0x731/0x2c27 [ 314.890743] fl_hw_replace_filter+0x23b/0x380 [cls_flower] [ 314.890756] fl_change+0x16bd/0x27ef [cls_flower] [ 314.890765] tc_new_tfilter+0x5e1/0xd40 [ 314.890776] rtnetlink_rcv_msg+0x4ab/0x5f0 [ 314.890786] netlink_rcv_skb+0xd0/0x200 [ 314.890796] netlink_unicast+0x296/0x350 [ 314.890805] netlink_sendmsg+0x394/0x600 [ 314.890815] sock_sendmsg+0x96/0xa0 [ 314.890825] ___sys_sendmsg+0x482/0x520 [ 314.890834] __sys_sendmsg+0xba/0x130 [ 314.890844] do_syscall_64+0x74/0xe0 [ 314.890854] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 314.890937] Freed by task 2687: [ 314.891076] save_stack+0x1b/0x80 [ 314.891086] __kasan_slab_free+0x12c/0x170 [ 314.891095] kfree+0xeb/0x2f0 [ 314.891106] tc_cleanup_flow_action+0x69/0xa0 [ 314.891119] fl_hw_replace_filter+0x2c5/0x380 [cls_flower] [ 314.891132] fl_change+0x16bd/0x27ef [cls_flower] [ 314.891140] tc_new_tfilter+0x5e1/0xd40 [ 314.891151] rtnetlink_rcv_msg+0x4ab/0x5f0 [ 314.891161] netlink_rcv_skb+0xd0/0x200 [ 314.891170] netlink_unicast+0x296/0x350 [ 314.891180] netlink_sendmsg+0x394/0x600 [ 314.891190] sock_sendmsg+0x96/0xa0 [ 314.891200] ___sys_sendmsg+0x482/0x520 [ 314.891208] __sys_sendmsg+0xba/0x130 [ 314.891218] do_syscall_64+0x74/0xe0 [ 314.891228] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 314.891315] The buggy address belongs to the object at ffff88886c746280 which belongs to the cache kmalloc-96 of size 96 [ 314.891762] The buggy address is located 0 bytes inside of 96-byte region [ffff88886c746280, ffff88886c7462e0) [ 314.892196] The buggy address belongs to the page: [ 314.892387] page:ffffea0021b1d180 refcount:1 mapcount:0 mapping:ffff88835d00ef80 index:0x0 [ 314.892398] flags: 0x57ffffc0000200(slab) [ 314.892413] raw: 0057ffffc0000200 ffffea00219e0340 0000000800000008 ffff88835d00ef80 [ 314.892423] raw: 0000000000000000 0000000080200020 00000001ffffffff 0000000000000000 [ 314.892430] page dumped because: kasan: bad access detected [ 314.892515] Memory state around the buggy address: [ 314.892707] ffff88886c746180: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc [ 314.892976] ffff88886c746200: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc [ 314.893251] >ffff88886c746280: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc [ 314.893522] ^ [ 314.893657] ffff88886c746300: fb fb fb fb fb fb fb fb fb fb fb fb fc fc fc fc [ 314.893924] ffff88886c746380: 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc [ 314.894189] ================================================================== Fix the issue by duplicating tunnel info into per-encap copy that is deallocated with encap structure. Also, duplicate tunnel info in flow parse attribute to support cases when flow might be attached asynchronously. Fixes: 1f6da30 ("net/mlx5e: Geneve, Keep tunnel info as pointer to the original struct") Signed-off-by: Vlad Buslov <[email protected]> Reviewed-by: Yevgeny Kliteynik <[email protected]> Reviewed-by: Roi Dayan <[email protected]> Signed-off-by: Saeed Mahameed <[email protected]>

popcornmix added a commit that referenced this issue Dec 13, 2014

Revert "scsi: handle flush errors properly"

5abc55a

This reverts commit 89fb4cd. See: #703

popcornmix closed this as completed Dec 22, 2014

bosscyril referenced this issue in Hexxeh/rpi-firmware Apr 14, 2015

firmware: Move to 3.18 kernel with device tree enabled

cad071a

See: http://www.raspberrypi.org/forums/viewtopic.php?f=29&t=93015

bosscyril mentioned this issue May 4, 2015

blk_update_request: critical target error on USB HDD after kernel upgrade #951

Closed

end_request: critical target error on USB HDD after kernel upgrade #703

end_request: critical target error on USB HDD after kernel upgrade #703

Comments

bosscyril commented Oct 6, 2014

popcornmix commented Oct 6, 2014

Uh oh!

bosscyril commented Oct 6, 2014

Uh oh!

P33M commented Oct 8, 2014

Uh oh!

mod186k1 commented Oct 25, 2014

Uh oh!

tsjk commented Nov 10, 2014

Uh oh!

sgebbie commented Nov 11, 2014

Uh oh!

P33M commented Nov 12, 2014

Uh oh!

sgebbie commented Nov 12, 2014

Uh oh!

sgebbie commented Nov 12, 2014

Uh oh!

dod38fr commented Nov 15, 2014

Uh oh!

bcdonadio commented Nov 22, 2014

Uh oh!

bcdonadio commented Nov 23, 2014

Uh oh!

Latvala commented Dec 12, 2014

Uh oh!

popcornmix commented Dec 12, 2014

Uh oh!

Latvala commented Dec 13, 2014

Uh oh!

popcornmix commented Dec 13, 2014

Uh oh!

Latvala commented Dec 13, 2014

Uh oh!

bcdonadio commented Dec 22, 2014

Uh oh!

popcornmix commented Dec 22, 2014

Uh oh!

bcdonadio commented Dec 24, 2014

Uh oh!

Latvala commented Dec 25, 2014

Uh oh!

popcornmix commented Dec 25, 2014

Uh oh!

bcdonadio commented Dec 25, 2014

Uh oh!

popcornmix commented Dec 28, 2014

Uh oh!

bosscyril commented Apr 14, 2015

Uh oh!