Re: Data can't be wrote to XFS RIP [<ffffffffa041a99a>] xfs_dir2_sf_get_parent_ino+0xa/0x20

From: Kuo Hugo <tonytkdk@gmail.com>
To: Brian Foster <bfoster@redhat.com>
Cc: Hugo Kuo <hugo@swiftstack.com>,
	Eric Sandeen <sandeen@sandeen.net>,
	Darrell Bishop <darrell@swiftstack.com>,
	xfs@oss.sgi.com
Subject: Re: Data can't be wrote to XFS RIP [<ffffffffa041a99a>] xfs_dir2_sf_get_parent_ino+0xa/0x20
Date: Mon, 20 Jul 2015 22:30:31 +0800	[thread overview]
Message-ID: <CA++_uhvwR1KucdHWnPzS5ysFuYyssFnUB95kS-piC_pRnq=dXw@mail.gmail.com> (raw)
In-Reply-To: <20150720114648.GB53450@bfoster.bfoster>

[-- Attachment #1.1: Type: text/plain, Size: 3100 bytes --]

Hi Brain,

>I don’t know much about the Swift bug. A BUG() or crash in the kernel is
generally always a kernel bug, regardless of what userspace is doing. It
>certainly could be that whatever userspace is doing to trigger the kernel
bug is a bug in the userspace application, but either way it shouldn’t
cause the >kernel to crash. By the same token, if Swift is updated to fix
the aforementioned bug and the kernel crash no longer reproduces, that
doesn’t >necessarily mean the kernel bug is fixed (just potentially hidden).

Understand.

[Previous Message]

The valid inode has an inode number of 13668207561.
- The fsname for this inode is "sdb."
- The inode does appear to have a non-NULL if_data:

    if_u1 = {
      if_extents = 0xffff88084feaf5c0,
      if_ext_irec = 0xffff88084feaf5c0,
      if_data = 0xffff88084feaf5c0 "\004"
    },

        find <mntpath> -inum 13668207561

Q1: Were you able to track down the directory inode mentioned in the
previous message?

Ans: Yes, it’s the directory/file as below. /srv/node/d224 is the mount
point of /dev/sdb . This is the original location of the path. This folder
includes the file 1436266052.71893.ts now. The .ts file is 0 size

[root@r2obj01 ~]# find /srv/node/d224 -inum 13668207561
/srv/node/d224/objects/45382/b32/b146865bf8034bfc42570b747c341b32

[root@r2obj01 ~]# ls -lrt
/srv/node/d224/objects/45382/b32/b146865bf8034bfc42570b747c341b32
-rw------- 1 swift swift 0 Jul 7 22:37 1436266052.71893.ts

Q2: Is it some kind of internal directory used by the application (e.g.,
perhaps related to the quarantine mechanism mentioned in the bug)?

Ans: Yes, it’s a directory which accessing by application.

 37 ffff8810718343c0 ffff88105b9d32c0 ffff8808745aa5e8 REG  [eventpoll]
 38 ffff8808713da780 ffff880010c9a900 ffff88096368a188 REG
/srv/node/d224/quarantined/objects/b146865bf8034bfc42570b747c341b32/1436266042.57775.ts
 39 ffff880871cb03c0 ffff880495a8b380 ffff8808a5e6c988 REG
/srv/node/d224/tmp/tmpSpnrHg

 40 ffff8808715b4540 ffff8804819c58c0 ffff8802381f8d88 DIR
/srv/node/d224/quarantined/objects/b146865bf8034bfc42570b747c341b32

The above operation in the swift-object-server was doing python function
call to rename the file
* /srv/node/d224/objects/45382/b32/b146865bf8034bfc42570b747c341b32/1436266042.57775.ts*
as
*/srv/node/d224/quarantined/objects/b146865bf8034bfc42570b747c341b32/1436266042.57775.ts*

os.rename(old, new)

And it crashed at this point. In the Q1, we found the inum is pointing to
the directory
/srv/node/d224/objects/45382/b32/b146865bf8034bfc42570b747c341b32 .

We found that multiple(over 10) DELETE from application against the target
file at almost same moment. The DELETE is removing the original file in the
directory and create new empty .ts file in this directory. I suspect that
multiple os.rename on the same file in that directory will cause the kernel
panic.

And the file
/srv/node/d224/quarantined/objects/b146865bf8034bfc42570b747c341b32/1436266042.57775.ts
was not created.

Regards // Hugo

[-- Attachment #1.2: Type: text/html, Size: 13843 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs