From mboxrd@z Thu Jan 1 00:00:00 1970 From: Shijith Thotton Subject: Re: vf init issue with patch igb_uio: issue FLR during open and release of device file Date: Thu, 14 Sep 2017 12:45:19 +0530 Message-ID: <20170914071518.GA24657@localhost.localdomain> References: <20170913104840.GA14229@localhost.localdomain> <95faf7af-367a-ff42-ed16-b8bf3fdea466@intel.com> <88A92D351643BA4CB23E303155170626630E6054@SHSMSX103.ccr.corp.intel.com> <50bb4358-cd0c-003f-1ae7-a953ef569807@intel.com> <20170914070053.GA23787@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: "dev@dpdk.org" , "Tan, Jianfeng" , "Hu, Xuekun" , Gregory Etelson , "Yigit, Ferruh" , "Patil, Harish" To: "Yang, Qiming" Return-path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0067.outbound.protection.outlook.com [104.47.34.67]) by dpdk.org (Postfix) with ESMTP id D0BE47CCD for ; Thu, 14 Sep 2017 09:15:39 +0200 (CEST) Content-Disposition: inline In-Reply-To: <20170914070053.GA23787@localhost.localdomain> List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, Sep 14, 2017 at 12:30:55PM +0530, Shijith Thotton wrote: > On Thu, Sep 14, 2017 at 03:16:05AM +0000, Yang, Qiming wrote: > > > -----Original Message----- > > > From: Yigit, Ferruh > > > Sent: Thursday, September 14, 2017 1:06 AM > > > To: Hu, Xuekun ; Shijith Thotton > > > ; Yang, Qiming > > > ; Gregory Etelson > > > Cc: dev@dpdk.org; Tan, Jianfeng > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > and release of device file > > > > > > On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > > > I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > > > Sent: Wednesday, September 13, 2017 7:04 PM > > > > To: Shijith Thotton ; Yang, Qiming > > > > ; Gregory Etelson > > > > Cc: dev@dpdk.org; Tan, Jianfeng > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > > > during open and release of device file > > > > > > > > On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > > >>> Hi, Shijith > > > >>> > > > >>> > > > >>> > > > >>> VF init error will happen after apply your patch, error log as below. If > > > >>> revert your commit, all things work well. And this issue is not only occur > > > >>> in i40 VF but also ixgbe. > > > > > > Hi Qiming, > > > > > > I can reproduce the issue for the case: > > > > > > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and > > > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest, > > > this works fine) > > > > > > And I confirm the function igbuio_pci_open() added with below patch is causing > > > it. > > > > > Hi, Shijith > > Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs. > > Reset was added in open to make the device ready for init. I'm not sure about > the exact cause of this issue and was unable to reproduce the same with LiquidIO > VFs. > > Reset happens during open and if VFs are not ready by the time init happens in > PMD, it can fail. Issue could be in firmware as we don't see it for all versions. > If there are no other way to fix this, I think, it is best to remove reset > during open as it breaks PMD from multiple vendors. > > Please try removing igbuio_pci_open. Sorry. Please try removing pci_reset_function from igbuio_pci_open. Thanks, Shijith > > > igbuio_pci_open() does: > > > pci_reset_function(dev); > > > pci_set_master(dev); > > > > > > Don't know yet root cause of the error, will dig more. > > > > > > >>> > > > >>> Could you help to check it soon? > > > >>> > > > >>> > > > >>> > > > >>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > > >>> > > > >>> EAL: Detected 10 lcore(s) > > > >>> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB > > > >>> > > > >>> EAL: Probing VFIO support... > > > >>> > > > >>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > > >>> unreliable clock cycles ! > > > >>> > > > >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > > >>> > > > >>> EAL: Invalid NUMA socket, default to 0 > > > >>> > > > >>> EAL: probe driver: 8086:154c net_i40e_vf > > > >>> > > > >>> i40evf_init_vf(): init_adminq failed: -53 > > > >>> > > > >>> i40evf_dev_init(): Init vf failed > > > >>> > > > >>> EAL: Requested device 0000:00:03.0 cannot be used > > > >>> > > > >>> > > > >>> > > > >>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > > >>> Author: Shijith Thotton > > > >>> Date: Fri Jul 7 16:43:51 2017 +0530 > > > >>> > > > >>> igb_uio: issue FLR during open and release of device file > > > >>> > > > >>> Set UIO info device file operations open and release. Call pci reset > > > >>> function inside open and release to clear device state at start and end. > > > >>> Copied this behaviour from vfio_pci kernel module code. With this patch, > > > >>> it is not mandatory to issue FLR by PMD's during init and close. > > > >>> > > > >>> Bus master enable and disable are added in open and release respectively > > > >>> to take care of device DMA. > > > >>> > > > >>> Signed-off-by: Shijith Thotton > > > >>> Reviewed-by: Jianfeng Tan > > > >>> Acked-by: Ferruh Yigit > > > >>> Acked-by: Gregory Etelson > > > >>> > > > >>> > > > >>> > > > >>> Best Regard, > > > >>> > > > >>> Yang Qiming > > > >>> > > > >>> > > > >> > > > >> Hi Yang Qiming, > > > >> > > > >> We didn't face this issue while preparing patch. I think, Gregory > > > >> tested it for Intel NICs. Adding more people to the loop for help. > > > >> > > > > > > > > I will try to reproduce the issue. > > > > > > > >> Shijith > > > >> > > > > > >