[11/16] fpga: intel: fme: add partial reconfiguration sub feature support

Commit Message

Wu, Hao March 30, 2017, 12:08 p.m. UTC

From: Kang Luwei <[e-mail]>

Partial Reconfiguration (PR) is the most important function for FME. It
allows reconfiguration for given Port/Accelerated Function Unit (AFU).

This patch adds support for PR sub feature. In this patch, it registers
a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
for PR operation once PR request received via ioctl. Below user space
interfaces are exposed by this sub feature.

Sysfs interface:
* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
  Read-only. Indicate the hardware interface information. Userspace
  applications need to check this interface to select correct green
  bitstream format before PR.

Ioctl interface:
* FPGA_FME_PORT_PR
  Do partial reconfiguration per information from userspace, including
  target port(AFU), buffer size and address info. It returns the PR status
  (PR error code if failed) to userspace.

Signed-off-by: Tim Whisonant <[e-mail]>
Signed-off-by: Enno Luebbers <[e-mail]>
Signed-off-by: Shiva Rao <[e-mail]>
Signed-off-by: Christopher Rauer <[e-mail]>
Signed-off-by: Alan Tull <[e-mail]>
Signed-off-by: Kang Luwei <[e-mail]>
Signed-off-by: Xiao Guangrong <[e-mail]>
Signed-off-by: Wu Hao <[e-mail]>
---
 drivers/fpga/intel/Makefile      |   2 +-
 drivers/fpga/intel/feature-dev.h |  58 ++++++
 drivers/fpga/intel/fme-main.c    |  44 ++++-
 drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
 drivers/fpga/intel/fme.h         |  32 ++++
 include/uapi/linux/intel-fpga.h  |  44 +++++
 6 files changed, 578 insertions(+), 2 deletions(-)
 create mode 100644 drivers/fpga/intel/fme-pr.c
 create mode 100644 drivers/fpga/intel/fme.h

Comments

Alan Tull March 30, 2017, 8:30 p.m. UTC | #1

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> From: Kang Luwei <[e-mail]>
>
> Partial Reconfiguration (PR) is the most important function for FME. It
> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>
> This patch adds support for PR sub feature. In this patch, it registers
> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> for PR operation once PR request received via ioctl. Below user space
> interfaces are exposed by this sub feature.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>   Read-only. Indicate the hardware interface information. Userspace
>   applications need to check this interface to select correct green
>   bitstream format before PR.
>
> Ioctl interface:
> * FPGA_FME_PORT_PR
>   Do partial reconfiguration per information from userspace, including
>   target port(AFU), buffer size and address info. It returns the PR status
>   (PR error code if failed) to userspace.
>
> Signed-off-by: Tim Whisonant <[e-mail]>
> Signed-off-by: Enno Luebbers <[e-mail]>
> Signed-off-by: Shiva Rao <[e-mail]>
> Signed-off-by: Christopher Rauer <[e-mail]>
> Signed-off-by: Alan Tull <[e-mail]>

Hi Wu Hao,

Thanks for submitting your patches.

I think there's been a misunderstanding of the meaning of
'Signed-off-by' [1].  I have not signed off on this code or had a hand
in its development.  But I'm happy to get to review it now.  It will
take a bit of time; I expect to be replying next week.

Alan Tull
[1] linux/Documentation/process/5.Posting.rst

> Signed-off-by: Kang Luwei <[e-mail]>
> Signed-off-by: Xiao Guangrong <[e-mail]>
> Signed-off-by: Wu Hao <[e-mail]>
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Xiao Guangrong March 31, 2017, 4:11 a.m. UTC | #2

On 31/03/2017 4:30 AM, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
>> From: Kang Luwei <[e-mail]>
>>
>> Partial Reconfiguration (PR) is the most important function for FME. It
>> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>>
>> This patch adds support for PR sub feature. In this patch, it registers
>> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> for PR operation once PR request received via ioctl. Below user space
>> interfaces are exposed by this sub feature.
>>
>> Sysfs interface:
>> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>>   Read-only. Indicate the hardware interface information. Userspace
>>   applications need to check this interface to select correct green
>>   bitstream format before PR.
>>
>> Ioctl interface:
>> * FPGA_FME_PORT_PR
>>   Do partial reconfiguration per information from userspace, including
>>   target port(AFU), buffer size and address info. It returns the PR status
>>   (PR error code if failed) to userspace.
>>
>> Signed-off-by: Tim Whisonant <[e-mail]>
>> Signed-off-by: Enno Luebbers <[e-mail]>
>> Signed-off-by: Shiva Rao <[e-mail]>
>> Signed-off-by: Christopher Rauer <[e-mail]>
>> Signed-off-by: Alan Tull <[e-mail]>
>
> Hi Wu Hao,
>
> Thanks for submitting your patches.
>
> I think there's been a misunderstanding of the meaning of
> 'Signed-off-by' [1].  I have not signed off on this code or had a hand
> in its development.  But I'm happy to get to review it now.  It will
> take a bit of time; I expect to be replying next week.

Hi Alan,

Sorry to confuse you, i think it's because you helped Chris a lot to
implement this interface and we'd like to include your credit as this
way. If you dislike, it will be dropped. :)

Thanks for your review in advance.


--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao March 31, 2017, 8:50 a.m. UTC | #3

On Fri, Mar 31, 2017 at 12:11:12PM +0800, Xiao Guangrong wrote:
> On 31/03/2017 4:30 AM, Alan Tull wrote:
> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> >>From: Kang Luwei <[e-mail]>
> >>
> >>Partial Reconfiguration (PR) is the most important function for FME. It
> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >>
> >>This patch adds support for PR sub feature. In this patch, it registers
> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >>for PR operation once PR request received via ioctl. Below user space
> >>interfaces are exposed by this sub feature.
> >>
> >>Sysfs interface:
> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >>  Read-only. Indicate the hardware interface information. Userspace
> >>  applications need to check this interface to select correct green
> >>  bitstream format before PR.
> >>
> >>Ioctl interface:
> >>* FPGA_FME_PORT_PR
> >>  Do partial reconfiguration per information from userspace, including
> >>  target port(AFU), buffer size and address info. It returns the PR status
> >>  (PR error code if failed) to userspace.
> >>
> >>Signed-off-by: Tim Whisonant <[e-mail]>
> >>Signed-off-by: Enno Luebbers <[e-mail]>
> >>Signed-off-by: Shiva Rao <[e-mail]>
> >>Signed-off-by: Christopher Rauer <[e-mail]>
> >>Signed-off-by: Alan Tull <[e-mail]>
> >
> >Hi Wu Hao,
> >
> >Thanks for submitting your patches.
> >
> >I think there's been a misunderstanding of the meaning of
> >'Signed-off-by' [1].  I have not signed off on this code or had a hand
> >in its development.  But I'm happy to get to review it now.  It will
> >take a bit of time; I expect to be replying next week.
> 
> Hi Alan,
> 
> Sorry to confuse you, i think it's because you helped Chris a lot to
> implement this interface and we'd like to include your credit as this
> way. If you dislike, it will be dropped. :)
> 
> Thanks for your review in advance.
>

Hi Alan,

Sorry about this, we should ask you firstly before doing it this way.
Let me know if you don't like it, I will drop it in the next version.

Many thanks for your time and review on these patches. Look forward
for your feedback and comments. :)

Thanks
Hao 
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull March 31, 2017, 7:10 p.m. UTC | #4

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> From: Kang Luwei <[e-mail]>
>
> Partial Reconfiguration (PR) is the most important function for FME. It
> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>
> This patch adds support for PR sub feature. In this patch, it registers
> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> for PR operation once PR request received via ioctl. Below user space
> interfaces are exposed by this sub feature.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>   Read-only. Indicate the hardware interface information. Userspace
>   applications need to check this interface to select correct green
>   bitstream format before PR.
>
> Ioctl interface:
> * FPGA_FME_PORT_PR
>   Do partial reconfiguration per information from userspace, including
>   target port(AFU), buffer size and address info. It returns the PR status
>   (PR error code if failed) to userspace.
>
> Signed-off-by: Tim Whisonant <[e-mail]>
> Signed-off-by: Enno Luebbers <[e-mail]>
> Signed-off-by: Shiva Rao <[e-mail]>
> Signed-off-by: Christopher Rauer <[e-mail]>
> Signed-off-by: Alan Tull <[e-mail]>
> Signed-off-by: Kang Luwei <[e-mail]>
> Signed-off-by: Xiao Guangrong <[e-mail]>
> Signed-off-by: Wu Hao <[e-mail]>
> ---
>  drivers/fpga/intel/Makefile      |   2 +-
>  drivers/fpga/intel/feature-dev.h |  58 ++++++
>  drivers/fpga/intel/fme-main.c    |  44 ++++-
>  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>  drivers/fpga/intel/fme.h         |  32 ++++
>  include/uapi/linux/intel-fpga.h  |  44 +++++
>  6 files changed, 578 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/fpga/intel/fme-pr.c
>  create mode 100644 drivers/fpga/intel/fme.h
>
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 546861d..0452cb6 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>
>  intel-fpga-pci-objs := pcie.o feature-dev.o
> -intel-fpga-fme-objs := fme-main.o
> +intel-fpga-fme-objs := fme-main.o fme-pr.o
> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> index dccc283..5a25c915 100644
> --- a/drivers/fpga/intel/feature-dev.h
> +++ b/drivers/fpga/intel/feature-dev.h
> @@ -150,8 +150,66 @@ struct feature_fme_err {
>  };
>
>  /* FME Partial Reconfiguration Sub Feature Register Set */
> +/* FME PR Control Register */
> +struct feature_fme_pr_ctl {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u8  pr_reset:1;         /* Reset PR Engine */
> +                       u8  rsvdz1:3;
> +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> +                       u8  rsvdz2:3;
> +                       u8  pr_regionid:2;      /* PR Region ID */
> +                       u8  rsvdz3:2;
> +                       u8  pr_start_req:1;     /* PR Start Request */
> +                       u8  pr_push_complete:1; /* PR Data push complete */
> +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> +                       u32 rsvdz4:17;
> +                       u32 config_data;
> +               };
> +       };
> +};
> +
> +/* FME PR Status Register */
> +struct feature_fme_pr_status {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u16 pr_credit:9;        /* Number of PR Credits */
> +                       u8  rsvdz1:7;
> +                       u8  pr_status:1;        /* PR Operation status */
> +                       u8  rsvdz2:3;
> +                       u8  pr_ctrlr_status:3;  /* Controller status */
> +                       u8  rsvdz3:1;
> +                       u8  pr_host_status:4;   /* PR Host status */
> +                       u64 rsvdz4:36;
> +               };
> +       };
> +};
> +
> +/* FME PR Data Register */
> +struct feature_fme_pr_data {
> +       union {
> +               u64 csr;
> +               struct {
> +                       /* PR data from the raw-binary file */
> +                       u32 pr_data_raw;
> +                       u32 rsvd;
> +               };
> +       };
> +};
> +
>  struct feature_fme_pr {
>         struct feature_header header;
> +       struct feature_fme_pr_ctl control;
> +       struct feature_fme_pr_status status;
> +       struct feature_fme_pr_data data;
> +       u64 error;
> +
> +       u64 rsvd[16];
> +
> +       u64 intfc_id_l;         /* PR interface Id Low */
> +       u64 intfc_id_h;         /* PR interface Id High */
>  };
>
>  /* PORT Header Register Set */
> diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> index 36d0c4c..0d9a7a6 100644
> --- a/drivers/fpga/intel/fme-main.c
> +++ b/drivers/fpga/intel/fme-main.c
> @@ -23,6 +23,7 @@
>  #include <linux/intel-fpga.h>
>
>  #include "feature-dev.h"
> +#include "fme.h"
>
>  static ssize_t ports_num_show(struct device *dev,
>                               struct device_attribute *attr, char *buf)
> @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>                 .ops = &fme_hdr_ops,
>         },
>         {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .ops = &pr_mgmt_ops,
> +       },
> +       {
>                 .ops = NULL,
>         },
>  };
> @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>         .unlocked_ioctl = fme_ioctl,
>  };
>
> +static int fme_dev_init(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> +       if (!fme)
> +               return -ENOMEM;
> +
> +       fme->pdata = pdata;
> +
> +       mutex_lock(&pdata->lock);
> +       fpga_pdata_set_private(pdata, fme);
> +       mutex_unlock(&pdata->lock);
> +       return 0;
> +}
> +
> +static void fme_dev_destroy(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       fpga_pdata_set_private(pdata, NULL);
> +       mutex_unlock(&pdata->lock);
> +
> +       devm_kfree(&pdev->dev, fme);
> +}
> +
>  static int fme_probe(struct platform_device *pdev)
>  {
>         int ret;
>
> -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       ret = fme_dev_init(pdev);
>         if (ret)
>                 goto exit;
>
> +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       if (ret)
> +               goto dev_destroy;
> +
>         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>         if (ret)
>                 goto feature_uinit;
> @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>
>  feature_uinit:
>         fpga_dev_feature_uinit(pdev);
> +dev_destroy:
> +       fme_dev_destroy(pdev);
>  exit:
>         return ret;
>  }
> @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>  {
>         fpga_dev_feature_uinit(pdev);
>         fpga_unregister_dev_ops(pdev);
> +       fme_dev_destroy(pdev);
>         return 0;
>  }
>
> diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> new file mode 100644
> index 0000000..3b44a3e
> --- /dev/null
> +++ b/drivers/fpga/intel/fme-pr.c
> @@ -0,0 +1,400 @@
> +/*
> + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <[e-mail]>
> + *   Xiao Guangrong <[e-mail]>
> + *   Joseph Grecco <[e-mail]>
> + *   Enno Luebbers <[e-mail]>
> + *   Tim Whisonant <[e-mail]>
> + *   Ananda Ravuri <[e-mail]>
> + *   Christopher Rauer <[e-mail]>
> + *   Henry Mitchel <[e-mail]>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/device.h>
> +#include <linux/vmalloc.h>
> +#include <linux/uaccess.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/intel-fpga.h>
> +
> +#include "feature-dev.h"
> +#include "fme.h"
> +
> +#define PR_WAIT_TIMEOUT   8000000
> +
> +#define PR_HOST_STATUS_IDLE    0
> +
> +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> +
> +static ssize_t interface_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       u64 intfc_id_l, intfc_id_h;
> +       struct feature_fme_pr *fme_pr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> +
> +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> +                       (unsigned long long)intfc_id_h,
> +                       (unsigned long long)intfc_id_l);
> +}
> +static DEVICE_ATTR_RO(interface_id);
> +
> +static struct attribute *pr_mgmt_attrs[] = {
> +       &dev_attr_interface_id.attr,
> +       NULL,
> +};
> +
> +struct attribute_group pr_mgmt_attr_group = {
> +       .attrs  = pr_mgmt_attrs,
> +       .name   = "pr",
> +};
> +
> +static u64
> +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> +{
> +       struct feature_fme_pr_status fme_pr_status;
> +       unsigned long err_code;
> +       u64 fme_pr_error;
> +       int i = 0;
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       if (!fme_pr_status.pr_status)
> +               return 0;
> +
> +       err_code = fme_pr_error = readq(&fme_pr->error);
> +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> +       writeq(fme_pr_error, &fme_pr->error);
> +       return fme_pr_error;
> +}
> +
> +static int fme_pr_write_init(struct fpga_manager *mgr,
> +               struct fpga_image_info *info, const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +       if (!fme_pr)
> +               return -EINVAL;
> +
> +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> +               return -EINVAL;
> +
> +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       fme_pr_ctl.pr_reset_ack = 1;
> +
> +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 0;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev,
> +               "waiting for PR resource in HW to be initialized and ready\n");
> +
> +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> +
> +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> +       pr_err_handle(pdev, fme_pr);
> +       return 0;
> +}
> +
> +static int fme_pr_write(struct fpga_manager *mgr,
> +                       const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +       struct feature_fme_pr_data fme_pr_data;
> +       int delay, pr_credit, i = 0;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_regionid = fme->port_id;
> +       fme_pr_ctl.pr_start_req = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       pr_credit = fme_pr_status.pr_credit;
> +
> +       while (count > 0) {
> +               delay = 0;
> +               while (pr_credit <= 1) {
> +                       if (delay++ > PR_WAIT_TIMEOUT) {
> +                               dev_err(&pdev->dev, "maximum try\n");
> +                               return -ETIMEDOUT;
> +                       }
> +                       udelay(1);
> +
> +                       fme_pr_status.csr = readq(&fme_pr->status);
> +                       pr_credit = fme_pr_status.pr_credit;
> +               };
> +
> +               if (count >= 4) {
> +                       fme_pr_data.rsvd = 0;
> +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> +                       writeq(fme_pr_data.csr, &fme_pr->data);
> +                       count -= 4;
> +                       pr_credit--;
> +                       i++;
> +               } else {
> +                       WARN_ON(1);
> +                       return -EINVAL;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static int fme_pr_write_complete(struct fpga_manager *mgr,
> +                       struct fpga_image_info *info)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_push_complete = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> +
> +       fme_pr_ctl.pr_start_req = 0;
> +
> +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum try.\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> +       if (fme->pr_err)
> +               return -EIO;
> +
> +       dev_dbg(&pdev->dev, "PR done successfully\n");
> +       return 0;
> +}
> +
> +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> +{
> +       return FPGA_MGR_STATE_UNKNOWN;
> +}
> +
> +static const struct fpga_manager_ops fme_pr_ops = {
> +       .write_init = fme_pr_write_init,
> +       .write = fme_pr_write,
> +       .write_complete = fme_pr_write_complete,
> +       .state = fme_pr_state,
> +};
> +
> +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> +{
> +       void __user *argp = (void __user *)arg;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +       struct fpga_manager *mgr;
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_capability fme_capability;
> +       struct fpga_image_info info;
> +       struct fpga_fme_port_pr port_pr;
> +       struct platform_device *port;
> +       unsigned long minsz;
> +       void *buf = NULL;
> +       int ret = 0;
> +
> +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> +
> +       if (copy_from_user(&port_pr, argp, minsz))
> +               return -EFAULT;
> +
> +       if (port_pr.argsz < minsz || port_pr.flags)
> +               return -EINVAL;
> +
> +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> +               return -EINVAL;
> +
> +       /* get fme header region */
> +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                       FME_FEATURE_ID_HEADER);
> +       if (WARN_ON(!fme_hdr))
> +               return -EINVAL;
> +
> +       /* check port id */
> +       fme_capability.csr = readq(&fme_hdr->capability);
> +       if (port_pr.port_id >= fme_capability.num_ports) {
> +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> +               return -EINVAL;
> +       }
> +
> +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> +                                   port_pr.buffer_size))
> +               return -EFAULT;
> +
> +       buf = vmalloc(port_pr.buffer_size);
> +       if (!buf)
> +               return -ENOMEM;
> +
> +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> +                                              port_pr.buffer_size)) {
> +               ret = -EFAULT;
> +               goto free_exit;
> +       }
> +
> +       memset(&info, 0, sizeof(struct fpga_image_info));
> +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> +
> +       mgr = fpga_mgr_get(&pdev->dev);
> +       if (IS_ERR(mgr)) {
> +               ret = PTR_ERR(mgr);
> +               goto free_exit;
> +       }
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       /* fme device has been unregistered. */
> +       if (!fme) {
> +               ret = -EINVAL;
> +               goto unlock_exit;
> +       }
> +
> +       fme->pr_err = 0;
> +       fme->port_id = port_pr.port_id;

It looks like you're using private data to communicate with the
driver, i.e. there is something you want to do with the fpga manager
framework and it doesn't have that feature.  The better way would be
for us to expand the framework so you don't need to do that.

port_id is the kind of thing that should be communicated to the driver
through fpga_image_info, so we could add that to the struct.  Should
we call it port_id?  Or is there something more generic that may be
useful in the future for other architectures?.

pr_err appears to be machine specific error codes that are
communicated outside your low level driver.  (Calling it pr_err is
extra confusing since Linux already has a commonly name function by
the same name).  The framework has state, but that's not doing what
you want here.  Maybe we could add a framework ops called status so
that status could be communicated from the low level driver. It would
be useful to abstract the machine specific state to a defined enum
that would be part of the fpga mgr framework.  I remember we had
something like that in the earliest version of fpga manager but it got
changed to state rather than status for some reason.

mgr->dev won't work for you for some of the things you are using fme->pdev for?

Alan

> +
> +       /* Find and get port device by index */
> +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> +                                        fpga_port_check_id);
> +       WARN_ON(!port);
> +
> +       /* Disable Port before PR */
> +       fpga_port_disable(port);
> +
> +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> +       port_pr.status = fme->pr_err;
> +
> +       /* Re-enable Port after PR finished */
> +       fpga_port_enable(port);
> +
> +       put_device(&port->dev);
> +
> +unlock_exit:
> +       mutex_unlock(&pdata->lock);
> +       fpga_mgr_put(mgr);
> +free_exit:
> +       vfree(buf);
> +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> +               return -EFAULT;
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_probe(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *priv;
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       priv = fpga_pdata_get_private(pdata);
> +       ret = fpga_mgr_register(&pdata->dev->dev,
> +               "Intel FPGA Manager", &fme_pr_ops, priv);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_remove(struct platform_device *pdev)
> +{
> +       fpga_mgr_unregister(&pdev->dev);
> +       return 0;
> +}
> +
> +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> +{
> +       int ret;
> +
> +       ret = fpga_fme_pr_probe(pdev);
> +       if (ret)
> +               return ret;
> +
> +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       if (ret)
> +               fpga_fme_pr_remove(pdev);
> +
> +       return ret;
> +}
> +
> +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> +{
> +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       fpga_fme_pr_remove(pdev);
> +}
> +
> +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> +       unsigned int cmd, unsigned long arg)
> +{
> +       long ret;
> +
> +       switch (cmd) {
> +       case FPGA_FME_PORT_PR:
> +               ret = fme_pr(pdev, arg);
> +               break;
> +       default:
> +               ret = -ENODEV;
> +       }
> +
> +       return ret;
> +}
> +
> +struct feature_ops pr_mgmt_ops = {
> +       .init = pr_mgmt_init,
> +       .uinit = pr_mgmt_uinit,
> +       .ioctl = fme_pr_ioctl,
> +};
> diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> new file mode 100644
> index 0000000..d6cb7ce
> --- /dev/null
> +++ b/drivers/fpga/intel/fme.h
> @@ -0,0 +1,32 @@
> +/*
> + * Header file for Intel FPGA Management Engine (FME) Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <[e-mail]>
> + *   Xiao Guangrong <[e-mail]>
> + *   Joseph Grecco <[e-mail]>
> + *   Enno Luebbers <[e-mail]>
> + *   Tim Whisonant <[e-mail]>
> + *   Ananda Ravuri <[e-mail]>
> + *   Henry Mitchel <[e-mail]>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#ifndef __INTEL_FME_H
> +#define __INTEL_FME_H
> +
> +struct fpga_fme {
> +       u8  port_id;
> +       u64 pr_err;
> +       struct feature_platform_data *pdata;
> +};
> +
> +extern struct feature_ops pr_mgmt_ops;
> +
> +#endif
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index 992e556..77658316 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -18,6 +18,8 @@
>  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>  #define _UAPI_LINUX_INTEL_FPGA_H
>
> +#include <linux/types.h>
> +
>  #define FPGA_API_VERSION 0
>
>  /*
> @@ -30,6 +32,7 @@
>  #define FPGA_MAGIC 0xB6
>
>  #define FPGA_BASE 0
> +#define FME_BASE 0x80
>
>  /**
>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> @@ -49,4 +52,45 @@
>
>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>
> +/* IOCTLs for FME file descriptor */
> +
> +/**
> + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> + *
> + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> + * provided by caller.
> + * Return: 0 on success, -errno on failure.
> + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> + * some errors during PR, under this case, the user can fetch HW error code
> + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> + * Otherwise, it is always zero.
> + */
> +
> +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> +static const char * const _name_[] = {                 \
> +       "PR operation error detected",                  \
> +       "PR CRC error detected",                        \
> +       "PR incompatiable bitstream error detected",    \
> +       "PR IP protocol error detected",                \
> +       "PR FIFO overflow error detected",              \
> +       "Reserved",                                     \
> +       "PR secure load error detected",                \
> +}
> +
> +#define PR_MAX_ERR_NUM 7
> +
> +struct fpga_fme_port_pr {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u32 port_id;
> +       __u32 buffer_size;
> +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> +       /* Output */
> +       __u64 status;           /* HW error code if ioctl returns -EIO */
> +};
> +
> +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> +
>  #endif /* _UAPI_INTEL_FPGA_H */
> --
> 2.7.4
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

kernel test robot March 31, 2017, 11:45 p.m. UTC | #5

Hi Kang,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.11-rc4 next-20170331]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Wu-Hao/Intel-FPGA-Device-Drivers/20170401-052017
config: frv-allmodconfig (attached as .config)
compiler: frv-linux-gcc (GCC) 6.2.0
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=frv 

All warnings (new ones prefixed by >>):

   drivers/fpga/intel/fme-pr.c: In function 'interface_id_show':
   drivers/fpga/intel/fme-pr.c:45:15: error: implicit declaration of function 'readq' [-Werror=implicit-function-declaration]
     intfc_id_l = readq(&fme_pr->intfc_id_l);
                  ^~~~~
   drivers/fpga/intel/fme-pr.c: In function 'pr_err_handle':
   drivers/fpga/intel/fme-pr.c:79:2: error: implicit declaration of function 'writeq' [-Werror=implicit-function-declaration]
     writeq(fme_pr_error, &fme_pr->error);
     ^~~~~~
   In file included from include/linux/uaccess.h:5:0,
                    from drivers/fpga/intel/fme-pr.c:25:
   drivers/fpga/intel/fme-pr.c: In function 'fme_pr':
>> arch/frv/include/asm/uaccess.h:63:47: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
                                                  ^
   arch/frv/include/asm/uaccess.h:61:60: note: in definition of macro '__range_ok'
    #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
                                                               ^~~~
>> drivers/fpga/intel/fme-pr.c:278:7: note: in expansion of macro 'access_ok'
     if (!access_ok(VERIFY_READ, port_pr.buffer_address,
          ^~~~~~~~~
   drivers/fpga/intel/fme-pr.c:286:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
                             ^
   cc1: some warnings being treated as errors
--
   drivers/fpga//intel/fme-pr.c: In function 'interface_id_show':
   drivers/fpga//intel/fme-pr.c:45:15: error: implicit declaration of function 'readq' [-Werror=implicit-function-declaration]
     intfc_id_l = readq(&fme_pr->intfc_id_l);
                  ^~~~~
   drivers/fpga//intel/fme-pr.c: In function 'pr_err_handle':
   drivers/fpga//intel/fme-pr.c:79:2: error: implicit declaration of function 'writeq' [-Werror=implicit-function-declaration]
     writeq(fme_pr_error, &fme_pr->error);
     ^~~~~~
   In file included from include/linux/uaccess.h:5:0,
                    from drivers/fpga//intel/fme-pr.c:25:
   drivers/fpga//intel/fme-pr.c: In function 'fme_pr':
>> arch/frv/include/asm/uaccess.h:63:47: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
    #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
                                                  ^
   arch/frv/include/asm/uaccess.h:61:60: note: in definition of macro '__range_ok'
    #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
                                                               ^~~~
   drivers/fpga//intel/fme-pr.c:278:7: note: in expansion of macro 'access_ok'
     if (!access_ok(VERIFY_READ, port_pr.buffer_address,
          ^~~~~~~~~
   drivers/fpga//intel/fme-pr.c:286:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
                             ^
   cc1: some warnings being treated as errors

vim +63 arch/frv/include/asm/uaccess.h

^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  47  	return flag;
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  48  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  49  #else
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  50  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  51  	if (addr < memory_start ||
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  52  	    addr > memory_end ||
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  53  	    size > memory_end - memory_start ||
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  54  	    addr + size > memory_end)
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  55  		return -EFAULT;
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  56  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  57  	return 0;
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  58  #endif
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  59  }
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  60  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  61  #define __range_ok(addr,size) ___range_ok((unsigned long) (addr), (unsigned long) (size))
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  62  
a8a77573c9 include/asm-frv/uaccess.h Al Viro        2006-06-23 @63  #define access_ok(type,addr,size) (__range_ok((void __user *)(addr), (size)) == 0)
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  64  #define __access_ok(addr,size) (__range_ok((addr), (size)) == 0)
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  65  
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  66  /*
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  67   * The exception table consists of pairs of addresses: the first is the
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  68   * address of an instruction that is allowed to fault, and the second is
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  69   * the address at which the program should continue.  No registers are
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  70   * modified, so it is entirely up to the continuation code to figure out
^1da177e4c include/asm-frv/uaccess.h Linus Torvalds 2005-04-16  71   * what to do.

:::::: The code at line 63 was first introduced by commit
:::::: a8a77573c9e5345bcf6a963858745cd83c923f44 [PATCH] frv: __user infrastructure

:::::: TO: Al Viro <[e-mail]>
:::::: CC: Linus Torvalds <[e-mail]>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

kernel test robot April 1, 2017, 1:12 a.m. UTC | #6

Hi Kang,

[auto build test WARNING on linus/master]
[also build test WARNING on v4.11-rc4 next-20170331]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Wu-Hao/Intel-FPGA-Device-Drivers/20170401-052017
config: sparc-allyesconfig (attached as .config)
compiler: sparc64-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=sparc 

All warnings (new ones prefixed by >>):

   drivers/fpga/intel/fme-pr.c: In function 'fme_pr':
>> drivers/fpga/intel/fme-pr.c:278:30: warning: passing argument 2 of 'access_ok' makes pointer from integer without a cast [-Wint-conversion]
     if (!access_ok(VERIFY_READ, port_pr.buffer_address,
                                 ^~~~~~~
   In file included from arch/sparc/include/asm/uaccess.h:4:0,
                    from include/linux/uaccess.h:5,
                    from drivers/fpga/intel/fme-pr.c:25:
   arch/sparc/include/asm/uaccess_64.h:80:19: note: expected 'const void *' but argument is of type '__u64 {aka long long unsigned int}'
    static inline int access_ok(int type, const void __user * addr, unsigned long size)
                      ^~~~~~~~~

vim +/access_ok +278 drivers/fpga/intel/fme-pr.c

   262		if (!IS_ALIGNED(port_pr.buffer_size, 4))
   263			return -EINVAL;
   264	
   265		/* get fme header region */
   266		fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
   267						FME_FEATURE_ID_HEADER);
   268		if (WARN_ON(!fme_hdr))
   269			return -EINVAL;
   270	
   271		/* check port id */
   272		fme_capability.csr = readq(&fme_hdr->capability);
   273		if (port_pr.port_id >= fme_capability.num_ports) {
   274			dev_dbg(&pdev->dev, "port number more than maximum\n");
   275			return -EINVAL;
   276		}
   277	
 > 278		if (!access_ok(VERIFY_READ, port_pr.buffer_address,
   279					    port_pr.buffer_size))
   280			return -EFAULT;
   281	
   282		buf = vmalloc(port_pr.buffer_size);
   283		if (!buf)
   284			return -ENOMEM;
   285	
   286		if (copy_from_user(buf, (void __user *)port_pr.buffer_address,

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

Wu, Hao April 1, 2017, 11:08 a.m. UTC | #7

On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> > From: Kang Luwei <[e-mail]>
> >
> > Partial Reconfiguration (PR) is the most important function for FME. It
> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >
> > This patch adds support for PR sub feature. In this patch, it registers
> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> > for PR operation once PR request received via ioctl. Below user space
> > interfaces are exposed by this sub feature.
> >
> > Sysfs interface:
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >   Read-only. Indicate the hardware interface information. Userspace
> >   applications need to check this interface to select correct green
> >   bitstream format before PR.
> >
> > Ioctl interface:
> > * FPGA_FME_PORT_PR
> >   Do partial reconfiguration per information from userspace, including
> >   target port(AFU), buffer size and address info. It returns the PR status
> >   (PR error code if failed) to userspace.
> >
> > Signed-off-by: Tim Whisonant <[e-mail]>
> > Signed-off-by: Enno Luebbers <[e-mail]>
> > Signed-off-by: Shiva Rao <[e-mail]>
> > Signed-off-by: Christopher Rauer <[e-mail]>
> > Signed-off-by: Alan Tull <[e-mail]>
> > Signed-off-by: Kang Luwei <[e-mail]>
> > Signed-off-by: Xiao Guangrong <[e-mail]>
> > Signed-off-by: Wu Hao <[e-mail]>
> > ---
> >  drivers/fpga/intel/Makefile      |   2 +-
> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/intel/fme.h         |  32 ++++
> >  include/uapi/linux/intel-fpga.h  |  44 +++++
> >  6 files changed, 578 insertions(+), 2 deletions(-)
> >  create mode 100644 drivers/fpga/intel/fme-pr.c
> >  create mode 100644 drivers/fpga/intel/fme.h
> >
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 546861d..0452cb6 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >
> >  intel-fpga-pci-objs := pcie.o feature-dev.o
> > -intel-fpga-fme-objs := fme-main.o
> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> > index dccc283..5a25c915 100644
> > --- a/drivers/fpga/intel/feature-dev.h
> > +++ b/drivers/fpga/intel/feature-dev.h
> > @@ -150,8 +150,66 @@ struct feature_fme_err {
> >  };
> >
> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > +/* FME PR Control Register */
> > +struct feature_fme_pr_ctl {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u8  pr_reset:1;         /* Reset PR Engine */
> > +                       u8  rsvdz1:3;
> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_regionid:2;      /* PR Region ID */
> > +                       u8  rsvdz3:2;
> > +                       u8  pr_start_req:1;     /* PR Start Request */
> > +                       u8  pr_push_complete:1; /* PR Data push complete */
> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> > +                       u32 rsvdz4:17;
> > +                       u32 config_data;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Status Register */
> > +struct feature_fme_pr_status {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u16 pr_credit:9;        /* Number of PR Credits */
> > +                       u8  rsvdz1:7;
> > +                       u8  pr_status:1;        /* PR Operation status */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
> > +                       u8  rsvdz3:1;
> > +                       u8  pr_host_status:4;   /* PR Host status */
> > +                       u64 rsvdz4:36;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Data Register */
> > +struct feature_fme_pr_data {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       /* PR data from the raw-binary file */
> > +                       u32 pr_data_raw;
> > +                       u32 rsvd;
> > +               };
> > +       };
> > +};
> > +
> >  struct feature_fme_pr {
> >         struct feature_header header;
> > +       struct feature_fme_pr_ctl control;
> > +       struct feature_fme_pr_status status;
> > +       struct feature_fme_pr_data data;
> > +       u64 error;
> > +
> > +       u64 rsvd[16];
> > +
> > +       u64 intfc_id_l;         /* PR interface Id Low */
> > +       u64 intfc_id_h;         /* PR interface Id High */
> >  };
> >
> >  /* PORT Header Register Set */
> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> > index 36d0c4c..0d9a7a6 100644
> > --- a/drivers/fpga/intel/fme-main.c
> > +++ b/drivers/fpga/intel/fme-main.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/intel-fpga.h>
> >
> >  #include "feature-dev.h"
> > +#include "fme.h"
> >
> >  static ssize_t ports_num_show(struct device *dev,
> >                               struct device_attribute *attr, char *buf)
> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >                 .ops = &fme_hdr_ops,
> >         },
> >         {
> > +               .name = FME_FEATURE_PR_MGMT,
> > +               .ops = &pr_mgmt_ops,
> > +       },
> > +       {
> >                 .ops = NULL,
> >         },
> >  };
> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >         .unlocked_ioctl = fme_ioctl,
> >  };
> >
> > +static int fme_dev_init(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> > +       if (!fme)
> > +               return -ENOMEM;
> > +
> > +       fme->pdata = pdata;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fpga_pdata_set_private(pdata, fme);
> > +       mutex_unlock(&pdata->lock);
> > +       return 0;
> > +}
> > +
> > +static void fme_dev_destroy(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       fpga_pdata_set_private(pdata, NULL);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       devm_kfree(&pdev->dev, fme);
> > +}
> > +
> >  static int fme_probe(struct platform_device *pdev)
> >  {
> >         int ret;
> >
> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       ret = fme_dev_init(pdev);
> >         if (ret)
> >                 goto exit;
> >
> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       if (ret)
> > +               goto dev_destroy;
> > +
> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >         if (ret)
> >                 goto feature_uinit;
> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >
> >  feature_uinit:
> >         fpga_dev_feature_uinit(pdev);
> > +dev_destroy:
> > +       fme_dev_destroy(pdev);
> >  exit:
> >         return ret;
> >  }
> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >  {
> >         fpga_dev_feature_uinit(pdev);
> >         fpga_unregister_dev_ops(pdev);
> > +       fme_dev_destroy(pdev);
> >         return 0;
> >  }
> >
> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> > new file mode 100644
> > index 0000000..3b44a3e
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme-pr.c
> > @@ -0,0 +1,400 @@
> > +/*
> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <[e-mail]>
> > + *   Xiao Guangrong <[e-mail]>
> > + *   Joseph Grecco <[e-mail]>
> > + *   Enno Luebbers <[e-mail]>
> > + *   Tim Whisonant <[e-mail]>
> > + *   Ananda Ravuri <[e-mail]>
> > + *   Christopher Rauer <[e-mail]>
> > + *   Henry Mitchel <[e-mail]>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/device.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/uaccess.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +#include <linux/intel-fpga.h>
> > +
> > +#include "feature-dev.h"
> > +#include "fme.h"
> > +
> > +#define PR_WAIT_TIMEOUT   8000000
> > +
> > +#define PR_HOST_STATUS_IDLE    0
> > +
> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> > +
> > +static ssize_t interface_id_show(struct device *dev,
> > +                                struct device_attribute *attr, char *buf)
> > +{
> > +       u64 intfc_id_l, intfc_id_h;
> > +       struct feature_fme_pr *fme_pr
> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> > +
> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > +                       (unsigned long long)intfc_id_h,
> > +                       (unsigned long long)intfc_id_l);
> > +}
> > +static DEVICE_ATTR_RO(interface_id);
> > +
> > +static struct attribute *pr_mgmt_attrs[] = {
> > +       &dev_attr_interface_id.attr,
> > +       NULL,
> > +};
> > +
> > +struct attribute_group pr_mgmt_attr_group = {
> > +       .attrs  = pr_mgmt_attrs,
> > +       .name   = "pr",
> > +};
> > +
> > +static u64
> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> > +{
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       unsigned long err_code;
> > +       u64 fme_pr_error;
> > +       int i = 0;
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       if (!fme_pr_status.pr_status)
> > +               return 0;
> > +
> > +       err_code = fme_pr_error = readq(&fme_pr->error);
> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> > +       writeq(fme_pr_error, &fme_pr->error);
> > +       return fme_pr_error;
> > +}
> > +
> > +static int fme_pr_write_init(struct fpga_manager *mgr,
> > +               struct fpga_image_info *info, const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +       if (!fme_pr)
> > +               return -EINVAL;
> > +
> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> > +               return -EINVAL;
> > +
> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       fme_pr_ctl.pr_reset_ack = 1;
> > +
> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 0;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev,
> > +               "waiting for PR resource in HW to be initialized and ready\n");
> > +
> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> > +
> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> > +       pr_err_handle(pdev, fme_pr);
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write(struct fpga_manager *mgr,
> > +                       const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       struct feature_fme_pr_data fme_pr_data;
> > +       int delay, pr_credit, i = 0;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_regionid = fme->port_id;
> > +       fme_pr_ctl.pr_start_req = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       pr_credit = fme_pr_status.pr_credit;
> > +
> > +       while (count > 0) {
> > +               delay = 0;
> > +               while (pr_credit <= 1) {
> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > +                               dev_err(&pdev->dev, "maximum try\n");
> > +                               return -ETIMEDOUT;
> > +                       }
> > +                       udelay(1);
> > +
> > +                       fme_pr_status.csr = readq(&fme_pr->status);
> > +                       pr_credit = fme_pr_status.pr_credit;
> > +               };
> > +
> > +               if (count >= 4) {
> > +                       fme_pr_data.rsvd = 0;
> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
> > +                       count -= 4;
> > +                       pr_credit--;
> > +                       i++;
> > +               } else {
> > +                       WARN_ON(1);
> > +                       return -EINVAL;
> > +               }
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
> > +                       struct fpga_image_info *info)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_push_complete = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> > +
> > +       fme_pr_ctl.pr_start_req = 0;
> > +
> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum try.\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> > +       if (fme->pr_err)
> > +               return -EIO;
> > +
> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
> > +       return 0;
> > +}
> > +
> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> > +{
> > +       return FPGA_MGR_STATE_UNKNOWN;
> > +}
> > +
> > +static const struct fpga_manager_ops fme_pr_ops = {
> > +       .write_init = fme_pr_write_init,
> > +       .write = fme_pr_write,
> > +       .write_complete = fme_pr_write_complete,
> > +       .state = fme_pr_state,
> > +};
> > +
> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> > +{
> > +       void __user *argp = (void __user *)arg;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +       struct fpga_manager *mgr;
> > +       struct feature_fme_header *fme_hdr;
> > +       struct feature_fme_capability fme_capability;
> > +       struct fpga_image_info info;
> > +       struct fpga_fme_port_pr port_pr;
> > +       struct platform_device *port;
> > +       unsigned long minsz;
> > +       void *buf = NULL;
> > +       int ret = 0;
> > +
> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> > +
> > +       if (copy_from_user(&port_pr, argp, minsz))
> > +               return -EFAULT;
> > +
> > +       if (port_pr.argsz < minsz || port_pr.flags)
> > +               return -EINVAL;
> > +
> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> > +               return -EINVAL;
> > +
> > +       /* get fme header region */
> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                       FME_FEATURE_ID_HEADER);
> > +       if (WARN_ON(!fme_hdr))
> > +               return -EINVAL;
> > +
> > +       /* check port id */
> > +       fme_capability.csr = readq(&fme_hdr->capability);
> > +       if (port_pr.port_id >= fme_capability.num_ports) {
> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> > +                                   port_pr.buffer_size))
> > +               return -EFAULT;
> > +
> > +       buf = vmalloc(port_pr.buffer_size);
> > +       if (!buf)
> > +               return -ENOMEM;
> > +
> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> > +                                              port_pr.buffer_size)) {
> > +               ret = -EFAULT;
> > +               goto free_exit;
> > +       }
> > +
> > +       memset(&info, 0, sizeof(struct fpga_image_info));
> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> > +
> > +       mgr = fpga_mgr_get(&pdev->dev);
> > +       if (IS_ERR(mgr)) {
> > +               ret = PTR_ERR(mgr);
> > +               goto free_exit;
> > +       }
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       /* fme device has been unregistered. */
> > +       if (!fme) {
> > +               ret = -EINVAL;
> > +               goto unlock_exit;
> > +       }
> > +
> > +       fme->pr_err = 0;
> > +       fme->port_id = port_pr.port_id;
> 
> It looks like you're using private data to communicate with the
> driver, i.e. there is something you want to do with the fpga manager
> framework and it doesn't have that feature.  The better way would be
> for us to expand the framework so you don't need to do that.
> 
> port_id is the kind of thing that should be communicated to the driver
> through fpga_image_info, so we could add that to the struct.  Should
> we call it port_id?  Or is there something more generic that may be
> useful in the future for other architectures?.

Hi Alan

Thanks for your feedback. :)

As you know, each Intel FPGA device may have more than one accelerator,
and all accelerators share the same fpga_manager (only one FME module).
port_id = 0 means the first accelerator on this fpga devices. So it's
needed by the fpga_manager to distinguish one accelerator from others
for partial reconfiguration.

Adding a member like a 'id' to fpga_image_info definitely works for us
in this case. We can add it this way, but I feel 'id' here seems not
related to fpga image, but characteristic of fpga region. It may be a
little confusing. One rough idea is that keep this info under fpga region
(maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
then fpga_manager knows the target region for partial reconfiguration.
If consider pr sysfs interface support under fpga-region in the future,
then we don't need to create a new 'id' sysfs file, as fpga-region itself
could provide this kind of info. But I'm not sure if this is the right
direction.

> 
> pr_err appears to be machine specific error codes that are
> communicated outside your low level driver.  (Calling it pr_err is
> extra confusing since Linux already has a commonly name function by
> the same name).  The framework has state, but that's not doing what
> you want here.  Maybe we could add a framework ops called status so
> that status could be communicated from the low level driver. It would
> be useful to abstract the machine specific state to a defined enum
> that would be part of the fpga mgr framework.  I remember we had
> something like that in the earliest version of fpga manager but it got
> changed to state rather than status for some reason.
> 

Adding a status function and a 'status' member to fpga manager sounds good.
But I'm not sure how to abstract these error codes, currently what we have
are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
"PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
fine to let each driver to define how to use that 'status' for machine
specific status?

> mgr->dev won't work for you for some of the things you are using fme->pdev for?
>

Does 'fme->pdev' mean the platform device for fme? I think we use platform
device to represent the FME module, and FME module may have multiple sub
features, driver discovers these sub features via the 'Device Feature List'
in the PCIe Device Bar. PR is only one of the sub features under FME module,
if any FME module doesn't have PR sub feature, fpga-manager will not be 
created at all. But this will not impact other sub features, e.g thermal
management, error reporting and etc, to create their own interfaces under
the platform device.

Thanks
Hao
 
> Alan
> 
> > +
> > +       /* Find and get port device by index */
> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> > +                                        fpga_port_check_id);
> > +       WARN_ON(!port);
> > +
> > +       /* Disable Port before PR */
> > +       fpga_port_disable(port);
> > +
> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> > +       port_pr.status = fme->pr_err;
> > +
> > +       /* Re-enable Port after PR finished */
> > +       fpga_port_enable(port);
> > +
> > +       put_device(&port->dev);
> > +
> > +unlock_exit:
> > +       mutex_unlock(&pdata->lock);
> > +       fpga_mgr_put(mgr);
> > +free_exit:
> > +       vfree(buf);
> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> > +               return -EFAULT;
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *priv;
> > +       int ret;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       priv = fpga_pdata_get_private(pdata);
> > +       ret = fpga_mgr_register(&pdata->dev->dev,
> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
> > +{
> > +       fpga_mgr_unregister(&pdev->dev);
> > +       return 0;
> > +}
> > +
> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       int ret;
> > +
> > +       ret = fpga_fme_pr_probe(pdev);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       if (ret)
> > +               fpga_fme_pr_remove(pdev);
> > +
> > +       return ret;
> > +}
> > +
> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       fpga_fme_pr_remove(pdev);
> > +}
> > +
> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> > +       unsigned int cmd, unsigned long arg)
> > +{
> > +       long ret;
> > +
> > +       switch (cmd) {
> > +       case FPGA_FME_PORT_PR:
> > +               ret = fme_pr(pdev, arg);
> > +               break;
> > +       default:
> > +               ret = -ENODEV;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +struct feature_ops pr_mgmt_ops = {
> > +       .init = pr_mgmt_init,
> > +       .uinit = pr_mgmt_uinit,
> > +       .ioctl = fme_pr_ioctl,
> > +};
> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> > new file mode 100644
> > index 0000000..d6cb7ce
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme.h
> > @@ -0,0 +1,32 @@
> > +/*
> > + * Header file for Intel FPGA Management Engine (FME) Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <[e-mail]>
> > + *   Xiao Guangrong <[e-mail]>
> > + *   Joseph Grecco <[e-mail]>
> > + *   Enno Luebbers <[e-mail]>
> > + *   Tim Whisonant <[e-mail]>
> > + *   Ananda Ravuri <[e-mail]>
> > + *   Henry Mitchel <[e-mail]>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#ifndef __INTEL_FME_H
> > +#define __INTEL_FME_H
> > +
> > +struct fpga_fme {
> > +       u8  port_id;
> > +       u64 pr_err;
> > +       struct feature_platform_data *pdata;
> > +};
> > +
> > +extern struct feature_ops pr_mgmt_ops;
> > +
> > +#endif
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index 992e556..77658316 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -18,6 +18,8 @@
> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
> >  #define _UAPI_LINUX_INTEL_FPGA_H
> >
> > +#include <linux/types.h>
> > +
> >  #define FPGA_API_VERSION 0
> >
> >  /*
> > @@ -30,6 +32,7 @@
> >  #define FPGA_MAGIC 0xB6
> >
> >  #define FPGA_BASE 0
> > +#define FME_BASE 0x80
> >
> >  /**
> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> > @@ -49,4 +52,45 @@
> >
> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >
> > +/* IOCTLs for FME file descriptor */
> > +
> > +/**
> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> > + *
> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> > + * provided by caller.
> > + * Return: 0 on success, -errno on failure.
> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> > + * some errors during PR, under this case, the user can fetch HW error code
> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> > + * Otherwise, it is always zero.
> > + */
> > +
> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> > +static const char * const _name_[] = {                 \
> > +       "PR operation error detected",                  \
> > +       "PR CRC error detected",                        \
> > +       "PR incompatiable bitstream error detected",    \
> > +       "PR IP protocol error detected",                \
> > +       "PR FIFO overflow error detected",              \
> > +       "Reserved",                                     \
> > +       "PR secure load error detected",                \
> > +}
> > +
> > +#define PR_MAX_ERR_NUM 7
> > +
> > +struct fpga_fme_port_pr {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u32 port_id;
> > +       __u32 buffer_size;
> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> > +       /* Output */
> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
> > +};
> > +
> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> > +
> >  #endif /* _UAPI_INTEL_FPGA_H */
> > --
> > 2.7.4
> >
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 3, 2017, 4:30 p.m. UTC | #8

On Sat, Apr 1, 2017 at 6:08 AM, Wu Hao <[e-mail]> wrote:
> On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
>> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
>> > From: Kang Luwei <[e-mail]>
>> >
>> > Partial Reconfiguration (PR) is the most important function for FME. It
>> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>> >
>> > This patch adds support for PR sub feature. In this patch, it registers
>> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> > for PR operation once PR request received via ioctl. Below user space
>> > interfaces are exposed by this sub feature.
>> >
>> > Sysfs interface:
>> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>> >   Read-only. Indicate the hardware interface information. Userspace
>> >   applications need to check this interface to select correct green
>> >   bitstream format before PR.
>> >
>> > Ioctl interface:
>> > * FPGA_FME_PORT_PR
>> >   Do partial reconfiguration per information from userspace, including
>> >   target port(AFU), buffer size and address info. It returns the PR status
>> >   (PR error code if failed) to userspace.
>> >
>> > Signed-off-by: Tim Whisonant <[e-mail]>
>> > Signed-off-by: Enno Luebbers <[e-mail]>
>> > Signed-off-by: Shiva Rao <[e-mail]>
>> > Signed-off-by: Christopher Rauer <[e-mail]>
>> > Signed-off-by: Alan Tull <[e-mail]>
>> > Signed-off-by: Kang Luwei <[e-mail]>
>> > Signed-off-by: Xiao Guangrong <[e-mail]>
>> > Signed-off-by: Wu Hao <[e-mail]>
>> > ---
>> >  drivers/fpga/intel/Makefile      |   2 +-
>> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
>> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
>> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>> >  drivers/fpga/intel/fme.h         |  32 ++++
>> >  include/uapi/linux/intel-fpga.h  |  44 +++++
>> >  6 files changed, 578 insertions(+), 2 deletions(-)
>> >  create mode 100644 drivers/fpga/intel/fme-pr.c
>> >  create mode 100644 drivers/fpga/intel/fme.h
>> >
>> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> > index 546861d..0452cb6 100644
>> > --- a/drivers/fpga/intel/Makefile
>> > +++ b/drivers/fpga/intel/Makefile
>> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>> >
>> >  intel-fpga-pci-objs := pcie.o feature-dev.o
>> > -intel-fpga-fme-objs := fme-main.o
>> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
>> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
>> > index dccc283..5a25c915 100644
>> > --- a/drivers/fpga/intel/feature-dev.h
>> > +++ b/drivers/fpga/intel/feature-dev.h
>> > @@ -150,8 +150,66 @@ struct feature_fme_err {
>> >  };
>> >
>> >  /* FME Partial Reconfiguration Sub Feature Register Set */
>> > +/* FME PR Control Register */
>> > +struct feature_fme_pr_ctl {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u8  pr_reset:1;         /* Reset PR Engine */
>> > +                       u8  rsvdz1:3;
>> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
>> > +                       u8  rsvdz2:3;
>> > +                       u8  pr_regionid:2;      /* PR Region ID */
>> > +                       u8  rsvdz3:2;
>> > +                       u8  pr_start_req:1;     /* PR Start Request */
>> > +                       u8  pr_push_complete:1; /* PR Data push complete */
>> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
>> > +                       u32 rsvdz4:17;
>> > +                       u32 config_data;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* FME PR Status Register */
>> > +struct feature_fme_pr_status {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       u16 pr_credit:9;        /* Number of PR Credits */
>> > +                       u8  rsvdz1:7;
>> > +                       u8  pr_status:1;        /* PR Operation status */
>> > +                       u8  rsvdz2:3;
>> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
>> > +                       u8  rsvdz3:1;
>> > +                       u8  pr_host_status:4;   /* PR Host status */
>> > +                       u64 rsvdz4:36;
>> > +               };
>> > +       };
>> > +};
>> > +
>> > +/* FME PR Data Register */
>> > +struct feature_fme_pr_data {
>> > +       union {
>> > +               u64 csr;
>> > +               struct {
>> > +                       /* PR data from the raw-binary file */
>> > +                       u32 pr_data_raw;
>> > +                       u32 rsvd;
>> > +               };
>> > +       };
>> > +};
>> > +
>> >  struct feature_fme_pr {
>> >         struct feature_header header;
>> > +       struct feature_fme_pr_ctl control;
>> > +       struct feature_fme_pr_status status;
>> > +       struct feature_fme_pr_data data;
>> > +       u64 error;
>> > +
>> > +       u64 rsvd[16];
>> > +
>> > +       u64 intfc_id_l;         /* PR interface Id Low */
>> > +       u64 intfc_id_h;         /* PR interface Id High */
>> >  };
>> >
>> >  /* PORT Header Register Set */
>> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
>> > index 36d0c4c..0d9a7a6 100644
>> > --- a/drivers/fpga/intel/fme-main.c
>> > +++ b/drivers/fpga/intel/fme-main.c
>> > @@ -23,6 +23,7 @@
>> >  #include <linux/intel-fpga.h>
>> >
>> >  #include "feature-dev.h"
>> > +#include "fme.h"
>> >
>> >  static ssize_t ports_num_show(struct device *dev,
>> >                               struct device_attribute *attr, char *buf)
>> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>> >                 .ops = &fme_hdr_ops,
>> >         },
>> >         {
>> > +               .name = FME_FEATURE_PR_MGMT,
>> > +               .ops = &pr_mgmt_ops,
>> > +       },
>> > +       {
>> >                 .ops = NULL,
>> >         },
>> >  };
>> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>> >         .unlocked_ioctl = fme_ioctl,
>> >  };
>> >
>> > +static int fme_dev_init(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *fme;
>> > +
>> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
>> > +       if (!fme)
>> > +               return -ENOMEM;
>> > +
>> > +       fme->pdata = pdata;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       fpga_pdata_set_private(pdata, fme);
>> > +       mutex_unlock(&pdata->lock);
>> > +       return 0;
>> > +}
>> > +
>> > +static void fme_dev_destroy(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *fme;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       fme = fpga_pdata_get_private(pdata);
>> > +       fpga_pdata_set_private(pdata, NULL);
>> > +       mutex_unlock(&pdata->lock);
>> > +
>> > +       devm_kfree(&pdev->dev, fme);
>> > +}
>> > +
>> >  static int fme_probe(struct platform_device *pdev)
>> >  {
>> >         int ret;
>> >
>> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> > +       ret = fme_dev_init(pdev);
>> >         if (ret)
>> >                 goto exit;
>> >
>> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> > +       if (ret)
>> > +               goto dev_destroy;
>> > +
>> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>> >         if (ret)
>> >                 goto feature_uinit;
>> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>> >
>> >  feature_uinit:
>> >         fpga_dev_feature_uinit(pdev);
>> > +dev_destroy:
>> > +       fme_dev_destroy(pdev);
>> >  exit:
>> >         return ret;
>> >  }
>> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>> >  {
>> >         fpga_dev_feature_uinit(pdev);
>> >         fpga_unregister_dev_ops(pdev);
>> > +       fme_dev_destroy(pdev);
>> >         return 0;
>> >  }
>> >
>> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
>> > new file mode 100644
>> > index 0000000..3b44a3e
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel/fme-pr.c
>> > @@ -0,0 +1,400 @@
>> > +/*
>> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <[e-mail]>
>> > + *   Xiao Guangrong <[e-mail]>
>> > + *   Joseph Grecco <[e-mail]>
>> > + *   Enno Luebbers <[e-mail]>
>> > + *   Tim Whisonant <[e-mail]>
>> > + *   Ananda Ravuri <[e-mail]>
>> > + *   Christopher Rauer <[e-mail]>
>> > + *   Henry Mitchel <[e-mail]>
>> > + *
>> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> > + * redistributing this file, you may do so under either license. See the
>> > + * LICENSE.BSD file under this directory for the BSD license and see
>> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> > + */
>> > +
>> > +#include <linux/types.h>
>> > +#include <linux/device.h>
>> > +#include <linux/vmalloc.h>
>> > +#include <linux/uaccess.h>
>> > +#include <linux/fpga/fpga-mgr.h>
>> > +#include <linux/intel-fpga.h>
>> > +
>> > +#include "feature-dev.h"
>> > +#include "fme.h"
>> > +
>> > +#define PR_WAIT_TIMEOUT   8000000
>> > +
>> > +#define PR_HOST_STATUS_IDLE    0
>> > +
>> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
>> > +
>> > +static ssize_t interface_id_show(struct device *dev,
>> > +                                struct device_attribute *attr, char *buf)
>> > +{
>> > +       u64 intfc_id_l, intfc_id_h;
>> > +       struct feature_fme_pr *fme_pr
>> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
>> > +
>> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
>> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
>> > +
>> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> > +                       (unsigned long long)intfc_id_h,
>> > +                       (unsigned long long)intfc_id_l);
>> > +}
>> > +static DEVICE_ATTR_RO(interface_id);
>> > +
>> > +static struct attribute *pr_mgmt_attrs[] = {
>> > +       &dev_attr_interface_id.attr,
>> > +       NULL,
>> > +};
>> > +
>> > +struct attribute_group pr_mgmt_attr_group = {
>> > +       .attrs  = pr_mgmt_attrs,
>> > +       .name   = "pr",
>> > +};
>> > +
>> > +static u64
>> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
>> > +{
>> > +       struct feature_fme_pr_status fme_pr_status;
>> > +       unsigned long err_code;
>> > +       u64 fme_pr_error;
>> > +       int i = 0;
>> > +
>> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> > +       if (!fme_pr_status.pr_status)
>> > +               return 0;
>> > +
>> > +       err_code = fme_pr_error = readq(&fme_pr->error);
>> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
>> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
>> > +       writeq(fme_pr_error, &fme_pr->error);
>> > +       return fme_pr_error;
>> > +}
>> > +
>> > +static int fme_pr_write_init(struct fpga_manager *mgr,
>> > +               struct fpga_image_info *info, const char *buf, size_t count)
>> > +{
>> > +       struct fpga_fme *fme = mgr->priv;
>> > +       struct platform_device *pdev;
>> > +       struct feature_fme_pr *fme_pr;
>> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> > +       struct feature_fme_pr_status fme_pr_status;
>> > +
>> > +       pdev = fme->pdata->dev;
>> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                               FME_FEATURE_ID_PR_MGMT);
>> > +       if (!fme_pr)
>> > +               return -EINVAL;
>> > +
>> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
>> > +               return -EINVAL;
>> > +
>> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_reset = 1;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       fme_pr_ctl.pr_reset_ack = 1;
>> > +
>> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
>> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_reset = 0;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       dev_dbg(&pdev->dev,
>> > +               "waiting for PR resource in HW to be initialized and ready\n");
>> > +
>> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> > +
>> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
>> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
>> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
>> > +       pr_err_handle(pdev, fme_pr);
>> > +       return 0;
>> > +}
>> > +
>> > +static int fme_pr_write(struct fpga_manager *mgr,
>> > +                       const char *buf, size_t count)
>> > +{
>> > +       struct fpga_fme *fme = mgr->priv;
>> > +       struct platform_device *pdev;
>> > +       struct feature_fme_pr *fme_pr;
>> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> > +       struct feature_fme_pr_status fme_pr_status;
>> > +       struct feature_fme_pr_data fme_pr_data;
>> > +       int delay, pr_credit, i = 0;
>> > +
>> > +       pdev = fme->pdata->dev;
>> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                               FME_FEATURE_ID_PR_MGMT);
>> > +
>> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_regionid = fme->port_id;
>> > +       fme_pr_ctl.pr_start_req = 1;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
>> > +
>> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> > +       pr_credit = fme_pr_status.pr_credit;
>> > +
>> > +       while (count > 0) {
>> > +               delay = 0;
>> > +               while (pr_credit <= 1) {
>> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> > +                               dev_err(&pdev->dev, "maximum try\n");
>> > +                               return -ETIMEDOUT;
>> > +                       }
>> > +                       udelay(1);
>> > +
>> > +                       fme_pr_status.csr = readq(&fme_pr->status);
>> > +                       pr_credit = fme_pr_status.pr_credit;
>> > +               };
>> > +
>> > +               if (count >= 4) {
>> > +                       fme_pr_data.rsvd = 0;
>> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
>> > +                       count -= 4;
>> > +                       pr_credit--;
>> > +                       i++;
>> > +               } else {
>> > +                       WARN_ON(1);
>> > +                       return -EINVAL;
>> > +               }
>> > +       }
>> > +
>> > +       return 0;
>> > +}
>> > +
>> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
>> > +                       struct fpga_image_info *info)
>> > +{
>> > +       struct fpga_fme *fme = mgr->priv;
>> > +       struct platform_device *pdev;
>> > +       struct feature_fme_pr *fme_pr;
>> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> > +
>> > +       pdev = fme->pdata->dev;
>> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                               FME_FEATURE_ID_PR_MGMT);
>> > +
>> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> > +       fme_pr_ctl.pr_push_complete = 1;
>> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> > +
>> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
>> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
>> > +
>> > +       fme_pr_ctl.pr_start_req = 0;
>> > +
>> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
>> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> > +               dev_err(&pdev->dev, "maximum try.\n");
>> > +               return -ETIMEDOUT;
>> > +       }
>> > +
>> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
>> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
>> > +       if (fme->pr_err)
>> > +               return -EIO;
>> > +
>> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
>> > +       return 0;
>> > +}
>> > +
>> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
>> > +{
>> > +       return FPGA_MGR_STATE_UNKNOWN;
>> > +}
>> > +
>> > +static const struct fpga_manager_ops fme_pr_ops = {
>> > +       .write_init = fme_pr_write_init,
>> > +       .write = fme_pr_write,
>> > +       .write_complete = fme_pr_write_complete,
>> > +       .state = fme_pr_state,
>> > +};
>> > +
>> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
>> > +{
>> > +       void __user *argp = (void __user *)arg;
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *fme;
>> > +       struct fpga_manager *mgr;
>> > +       struct feature_fme_header *fme_hdr;
>> > +       struct feature_fme_capability fme_capability;
>> > +       struct fpga_image_info info;
>> > +       struct fpga_fme_port_pr port_pr;
>> > +       struct platform_device *port;
>> > +       unsigned long minsz;
>> > +       void *buf = NULL;
>> > +       int ret = 0;
>> > +
>> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
>> > +
>> > +       if (copy_from_user(&port_pr, argp, minsz))
>> > +               return -EFAULT;
>> > +
>> > +       if (port_pr.argsz < minsz || port_pr.flags)
>> > +               return -EINVAL;
>> > +
>> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
>> > +               return -EINVAL;
>> > +
>> > +       /* get fme header region */
>> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> > +                                       FME_FEATURE_ID_HEADER);
>> > +       if (WARN_ON(!fme_hdr))
>> > +               return -EINVAL;
>> > +
>> > +       /* check port id */
>> > +       fme_capability.csr = readq(&fme_hdr->capability);
>> > +       if (port_pr.port_id >= fme_capability.num_ports) {
>> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
>> > +               return -EINVAL;
>> > +       }
>> > +
>> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
>> > +                                   port_pr.buffer_size))
>> > +               return -EFAULT;
>> > +
>> > +       buf = vmalloc(port_pr.buffer_size);
>> > +       if (!buf)
>> > +               return -ENOMEM;
>> > +
>> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
>> > +                                              port_pr.buffer_size)) {
>> > +               ret = -EFAULT;
>> > +               goto free_exit;
>> > +       }
>> > +
>> > +       memset(&info, 0, sizeof(struct fpga_image_info));
>> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
>> > +
>> > +       mgr = fpga_mgr_get(&pdev->dev);
>> > +       if (IS_ERR(mgr)) {
>> > +               ret = PTR_ERR(mgr);
>> > +               goto free_exit;
>> > +       }
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       fme = fpga_pdata_get_private(pdata);
>> > +       /* fme device has been unregistered. */
>> > +       if (!fme) {
>> > +               ret = -EINVAL;
>> > +               goto unlock_exit;
>> > +       }
>> > +
>> > +       fme->pr_err = 0;
>> > +       fme->port_id = port_pr.port_id;
>>
>> It looks like you're using private data to communicate with the
>> driver, i.e. there is something you want to do with the fpga manager
>> framework and it doesn't have that feature.  The better way would be
>> for us to expand the framework so you don't need to do that.
>>
>> port_id is the kind of thing that should be communicated to the driver
>> through fpga_image_info, so we could add that to the struct.  Should
>> we call it port_id?

Let's call it region_id.

>> Or is there something more generic that may be
>> useful in the future for other architectures?.
>
> Hi Alan
>
> Thanks for your feedback. :)
>
> As you know, each Intel FPGA device may have more than one accelerator,
> and all accelerators share the same fpga_manager (only one FME module).
> port_id = 0 means the first accelerator on this fpga devices. So it's
> needed by the fpga_manager to distinguish one accelerator from others
> for partial reconfiguration.
>
> Adding a member like a 'id' to fpga_image_info definitely works for us
> in this case. We can add it this way, but I feel 'id' here seems not
> related to fpga image, but characteristic of fpga region.

The  fpga_image_info struct started life as just image specific info,
but I want it to go in the direction of including parameters needed to
program it this specific time. Otherwise we are stuck having to keep
adding parameters as our use of FPGA develops.  It probably could be
documented better as 'information needed to program a FPGA image'
rather than strictly 'information about this particular FPGA image'.
My patch "fpga-mgr: pass parameters for loading fpga in image info"
goes in this direction by having the buf, firmware name, or sg list
passed in the info for the added fpga_mgr_load() function.  Actually I
should probably simplify the API and get rid of fpga_mgr_buf_load,
fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
use fpga_mgr_load (passing all parameters in fpga_image_info).

> It may be a
> little confusing. One rough idea is that keep this info under fpga region
> (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,

Yes, keep this info in fpga-region.  When the region wants to program
using fpga-mgr, add the region id to fpga_image_info.  I propose
calling it region_id.

> then fpga_manager knows the target region for partial reconfiguration.
> If consider pr sysfs interface support under fpga-region in the future,
> then we don't need to create a new 'id' sysfs file, as fpga-region itself
> could provide this kind of info. But I'm not sure if this is the right
> direction.
>
>>
>> pr_err appears to be machine specific error codes that are
>> communicated outside your low level driver.  (Calling it pr_err is
>> extra confusing since Linux already has a commonly name function by
>> the same name).  The framework has state, but that's not doing what
>> you want here.  Maybe we could add a framework ops called status so
>> that status could be communicated from the low level driver. It would
>> be useful to abstract the machine specific state to a defined enum
>> that would be part of the fpga mgr framework.  I remember we had
>> something like that in the earliest version of fpga manager but it got
>> changed to state rather than status for some reason.
>>
>
> Adding a status function and a 'status' member to fpga manager sounds good.
> But I'm not sure how to abstract these error codes, currently what we have
> are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
> "PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
> fine to let each driver to define how to use that 'status' for machine
> specific status?

I looked at the list of errors in include/uapi/linux/intel-fpga.h.
They all seem pretty generic to me except I am not clear what "secure
load error" or "IP protocol error" mean and whether other
architectures would have them.  But certainly things like crc error,
incompatible bitstream, fifo overflow are generic.  So let's see if we
can define all these in a way that's generic and just pass up a error
number.  That way upper layers can know how to deal with them
possibly.  I would take the word "PR" off these since the error set
applies whether someone is doing full reconfig or partial reconfig.

>
>> mgr->dev won't work for you for some of the things you are using fme->pdev for?
>>
>
> Does 'fme->pdev' mean the platform device for fme? I think we use platform
> device to represent the FME module, and FME module may have multiple sub
> features, driver discovers these sub features via the 'Device Feature List'
> in the PCIe Device Bar. PR is only one of the sub features under FME module,
> if any FME module doesn't have PR sub feature, fpga-manager will not be
> created at all. But this will not impact other sub features, e.g thermal
> management, error reporting and etc, to create their own interfaces under
> the platform device.

If we need it for private data, that's may be actually OK.   Just
wondering whether the priv was needed.   I think that's the one
element in the private data struct that was used privately instead of
being used to pass info outside the framework.

>
> Thanks
> Hao
>
>> Alan
>>
>> > +
>> > +       /* Find and get port device by index */
>> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
>> > +                                        fpga_port_check_id);
>> > +       WARN_ON(!port);
>> > +
>> > +       /* Disable Port before PR */
>> > +       fpga_port_disable(port);
>> > +
>> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
>> > +       port_pr.status = fme->pr_err;
>> > +
>> > +       /* Re-enable Port after PR finished */
>> > +       fpga_port_enable(port);
>> > +
>> > +       put_device(&port->dev);
>> > +
>> > +unlock_exit:
>> > +       mutex_unlock(&pdata->lock);
>> > +       fpga_mgr_put(mgr);
>> > +free_exit:
>> > +       vfree(buf);
>> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
>> > +               return -EFAULT;
>> > +       return ret;
>> > +}
>> > +
>> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
>> > +{
>> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> > +       struct fpga_fme *priv;
>> > +       int ret;
>> > +
>> > +       mutex_lock(&pdata->lock);
>> > +       priv = fpga_pdata_get_private(pdata);
>> > +       ret = fpga_mgr_register(&pdata->dev->dev,
>> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
>> > +       mutex_unlock(&pdata->lock);
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
>> > +{
>> > +       fpga_mgr_unregister(&pdev->dev);
>> > +       return 0;
>> > +}
>> > +
>> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
>> > +{
>> > +       int ret;
>> > +
>> > +       ret = fpga_fme_pr_probe(pdev);
>> > +       if (ret)
>> > +               return ret;
>> > +
>> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> > +       if (ret)
>> > +               fpga_fme_pr_remove(pdev);
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
>> > +{
>> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> > +       fpga_fme_pr_remove(pdev);
>> > +}
>> > +
>> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
>> > +       unsigned int cmd, unsigned long arg)
>> > +{
>> > +       long ret;
>> > +
>> > +       switch (cmd) {
>> > +       case FPGA_FME_PORT_PR:
>> > +               ret = fme_pr(pdev, arg);
>> > +               break;
>> > +       default:
>> > +               ret = -ENODEV;
>> > +       }
>> > +
>> > +       return ret;
>> > +}
>> > +
>> > +struct feature_ops pr_mgmt_ops = {
>> > +       .init = pr_mgmt_init,
>> > +       .uinit = pr_mgmt_uinit,
>> > +       .ioctl = fme_pr_ioctl,
>> > +};
>> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
>> > new file mode 100644
>> > index 0000000..d6cb7ce
>> > --- /dev/null
>> > +++ b/drivers/fpga/intel/fme.h
>> > @@ -0,0 +1,32 @@
>> > +/*
>> > + * Header file for Intel FPGA Management Engine (FME) Driver
>> > + *
>> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> > + *
>> > + * Authors:
>> > + *   Kang Luwei <[e-mail]>
>> > + *   Xiao Guangrong <[e-mail]>
>> > + *   Joseph Grecco <[e-mail]>
>> > + *   Enno Luebbers <[e-mail]>
>> > + *   Tim Whisonant <[e-mail]>
>> > + *   Ananda Ravuri <[e-mail]>
>> > + *   Henry Mitchel <[e-mail]>
>> > + *
>> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> > + * redistributing this file, you may do so under either license. See the
>> > + * LICENSE.BSD file under this directory for the BSD license and see
>> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> > + */
>> > +
>> > +#ifndef __INTEL_FME_H
>> > +#define __INTEL_FME_H
>> > +
>> > +struct fpga_fme {
>> > +       u8  port_id;
>> > +       u64 pr_err;
>> > +       struct feature_platform_data *pdata;
>> > +};
>> > +
>> > +extern struct feature_ops pr_mgmt_ops;
>> > +
>> > +#endif
>> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
>> > index 992e556..77658316 100644
>> > --- a/include/uapi/linux/intel-fpga.h
>> > +++ b/include/uapi/linux/intel-fpga.h
>> > @@ -18,6 +18,8 @@
>> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>> >  #define _UAPI_LINUX_INTEL_FPGA_H
>> >
>> > +#include <linux/types.h>
>> > +
>> >  #define FPGA_API_VERSION 0
>> >
>> >  /*
>> > @@ -30,6 +32,7 @@
>> >  #define FPGA_MAGIC 0xB6
>> >
>> >  #define FPGA_BASE 0
>> > +#define FME_BASE 0x80
>> >
>> >  /**
>> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>> > @@ -49,4 +52,45 @@
>> >
>> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>> >
>> > +/* IOCTLs for FME file descriptor */
>> > +
>> > +/**
>> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
>> > + *
>> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
>> > + * provided by caller.
>> > + * Return: 0 on success, -errno on failure.
>> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
>> > + * some errors during PR, under this case, the user can fetch HW error code
>> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
>> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
>> > + * Otherwise, it is always zero.
>> > + */
>> > +
>> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
>> > +static const char * const _name_[] = {                 \
>> > +       "PR operation error detected",                  \
>> > +       "PR CRC error detected",                        \
>> > +       "PR incompatiable bitstream error detected",    \
>> > +       "PR IP protocol error detected",                \
>> > +       "PR FIFO overflow error detected",              \
>> > +       "Reserved",                                     \
>> > +       "PR secure load error detected",                \
>> > +}
>> > +
>> > +#define PR_MAX_ERR_NUM 7
>> > +
>> > +struct fpga_fme_port_pr {
>> > +       /* Input */
>> > +       __u32 argsz;            /* Structure length */
>> > +       __u32 flags;            /* Zero for now */
>> > +       __u32 port_id;
>> > +       __u32 buffer_size;
>> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
>> > +       /* Output */
>> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
>> > +};
>> > +
>> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
>> > +
>> >  #endif /* _UAPI_INTEL_FPGA_H */
>> > --
>> > 2.7.4
>> >
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 3, 2017, 8:26 p.m. UTC | #9

On Fri, Mar 31, 2017 at 3:50 AM, Wu Hao <[e-mail]> wrote:
> On Fri, Mar 31, 2017 at 12:11:12PM +0800, Xiao Guangrong wrote:
>> On 31/03/2017 4:30 AM, Alan Tull wrote:
>> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
>> >>From: Kang Luwei <[e-mail]>
>> >>
>> >>Partial Reconfiguration (PR) is the most important function for FME. It
>> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>> >>
>> >>This patch adds support for PR sub feature. In this patch, it registers
>> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> >>for PR operation once PR request received via ioctl. Below user space
>> >>interfaces are exposed by this sub feature.
>> >>
>> >>Sysfs interface:
>> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>> >>  Read-only. Indicate the hardware interface information. Userspace
>> >>  applications need to check this interface to select correct green
>> >>  bitstream format before PR.
>> >>
>> >>Ioctl interface:
>> >>* FPGA_FME_PORT_PR
>> >>  Do partial reconfiguration per information from userspace, including
>> >>  target port(AFU), buffer size and address info. It returns the PR status
>> >>  (PR error code if failed) to userspace.
>> >>
>> >>Signed-off-by: Tim Whisonant <[e-mail]>
>> >>Signed-off-by: Enno Luebbers <[e-mail]>
>> >>Signed-off-by: Shiva Rao <[e-mail]>
>> >>Signed-off-by: Christopher Rauer <[e-mail]>
>> >>Signed-off-by: Alan Tull <[e-mail]>
>> >
>> >Hi Wu Hao,
>> >
>> >Thanks for submitting your patches.
>> >
>> >I think there's been a misunderstanding of the meaning of
>> >'Signed-off-by' [1].  I have not signed off on this code or had a hand
>> >in its development.  But I'm happy to get to review it now.  It will
>> >take a bit of time; I expect to be replying next week.
>>
>> Hi Alan,
>>
>> Sorry to confuse you, i think it's because you helped Chris a lot to
>> implement this interface and we'd like to include your credit as this
>> way. If you dislike, it will be dropped. :)
>>
>> Thanks for your review in advance.
>>
>
> Hi Alan,
>
> Sorry about this, we should ask you firstly before doing it this way.
> Let me know if you don't like it, I will drop it in the next version.

Yes please drop the signed-off-by: me in the next version.  Also, you
don't need to cc my Intel email address.

Alan

>
> Many thanks for your time and review on these patches. Look forward
> for your feedback and comments. :)
>
> Thanks
> Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 3, 2017, 9:24 p.m. UTC | #10

On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> From: Kang Luwei <[e-mail]>
>
> Partial Reconfiguration (PR) is the most important function for FME. It
> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>
> This patch adds support for PR sub feature. In this patch, it registers
> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> for PR operation once PR request received via ioctl.

The code that invokes fpga_mgr_buf_load should be a different layer.

> Below user space
> interfaces are exposed by this sub feature.
>
> Sysfs interface:
> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>   Read-only. Indicate the hardware interface information. Userspace
>   applications need to check this interface to select correct green
>   bitstream format before PR.
>
> Ioctl interface:
> * FPGA_FME_PORT_PR
>   Do partial reconfiguration per information from userspace, including
>   target port(AFU), buffer size and address info. It returns the PR status
>   (PR error code if failed) to userspace.
>
> Signed-off-by: Tim Whisonant <[e-mail]>
> Signed-off-by: Enno Luebbers <[e-mail]>
> Signed-off-by: Shiva Rao <[e-mail]>
> Signed-off-by: Christopher Rauer <[e-mail]>
> Signed-off-by: Alan Tull <[e-mail]>
> Signed-off-by: Kang Luwei <[e-mail]>
> Signed-off-by: Xiao Guangrong <[e-mail]>
> Signed-off-by: Wu Hao <[e-mail]>
> ---
>  drivers/fpga/intel/Makefile      |   2 +-
>  drivers/fpga/intel/feature-dev.h |  58 ++++++
>  drivers/fpga/intel/fme-main.c    |  44 ++++-
>  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>  drivers/fpga/intel/fme.h         |  32 ++++
>  include/uapi/linux/intel-fpga.h  |  44 +++++
>  6 files changed, 578 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/fpga/intel/fme-pr.c
>  create mode 100644 drivers/fpga/intel/fme.h
>
> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> index 546861d..0452cb6 100644
> --- a/drivers/fpga/intel/Makefile
> +++ b/drivers/fpga/intel/Makefile
> @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>
>  intel-fpga-pci-objs := pcie.o feature-dev.o
> -intel-fpga-fme-objs := fme-main.o
> +intel-fpga-fme-objs := fme-main.o fme-pr.o
> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> index dccc283..5a25c915 100644
> --- a/drivers/fpga/intel/feature-dev.h
> +++ b/drivers/fpga/intel/feature-dev.h
> @@ -150,8 +150,66 @@ struct feature_fme_err {
>  };
>
>  /* FME Partial Reconfiguration Sub Feature Register Set */
> +/* FME PR Control Register */
> +struct feature_fme_pr_ctl {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u8  pr_reset:1;         /* Reset PR Engine */
> +                       u8  rsvdz1:3;
> +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> +                       u8  rsvdz2:3;
> +                       u8  pr_regionid:2;      /* PR Region ID */
> +                       u8  rsvdz3:2;
> +                       u8  pr_start_req:1;     /* PR Start Request */
> +                       u8  pr_push_complete:1; /* PR Data push complete */
> +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> +                       u32 rsvdz4:17;
> +                       u32 config_data;
> +               };
> +       };
> +};
> +
> +/* FME PR Status Register */
> +struct feature_fme_pr_status {
> +       union {
> +               u64 csr;
> +               struct {
> +                       u16 pr_credit:9;        /* Number of PR Credits */
> +                       u8  rsvdz1:7;
> +                       u8  pr_status:1;        /* PR Operation status */
> +                       u8  rsvdz2:3;
> +                       u8  pr_ctrlr_status:3;  /* Controller status */
> +                       u8  rsvdz3:1;
> +                       u8  pr_host_status:4;   /* PR Host status */
> +                       u64 rsvdz4:36;
> +               };
> +       };
> +};
> +
> +/* FME PR Data Register */
> +struct feature_fme_pr_data {
> +       union {
> +               u64 csr;
> +               struct {
> +                       /* PR data from the raw-binary file */
> +                       u32 pr_data_raw;
> +                       u32 rsvd;
> +               };
> +       };
> +};
> +
>  struct feature_fme_pr {
>         struct feature_header header;
> +       struct feature_fme_pr_ctl control;
> +       struct feature_fme_pr_status status;
> +       struct feature_fme_pr_data data;
> +       u64 error;
> +
> +       u64 rsvd[16];
> +
> +       u64 intfc_id_l;         /* PR interface Id Low */
> +       u64 intfc_id_h;         /* PR interface Id High */
>  };
>
>  /* PORT Header Register Set */
> diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> index 36d0c4c..0d9a7a6 100644
> --- a/drivers/fpga/intel/fme-main.c
> +++ b/drivers/fpga/intel/fme-main.c
> @@ -23,6 +23,7 @@
>  #include <linux/intel-fpga.h>
>
>  #include "feature-dev.h"
> +#include "fme.h"
>
>  static ssize_t ports_num_show(struct device *dev,
>                               struct device_attribute *attr, char *buf)
> @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>                 .ops = &fme_hdr_ops,
>         },
>         {
> +               .name = FME_FEATURE_PR_MGMT,
> +               .ops = &pr_mgmt_ops,
> +       },
> +       {
>                 .ops = NULL,
>         },
>  };
> @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>         .unlocked_ioctl = fme_ioctl,
>  };
>
> +static int fme_dev_init(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> +       if (!fme)
> +               return -ENOMEM;
> +
> +       fme->pdata = pdata;
> +
> +       mutex_lock(&pdata->lock);
> +       fpga_pdata_set_private(pdata, fme);
> +       mutex_unlock(&pdata->lock);
> +       return 0;
> +}
> +
> +static void fme_dev_destroy(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       fpga_pdata_set_private(pdata, NULL);
> +       mutex_unlock(&pdata->lock);
> +
> +       devm_kfree(&pdev->dev, fme);
> +}
> +
>  static int fme_probe(struct platform_device *pdev)
>  {
>         int ret;
>
> -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       ret = fme_dev_init(pdev);
>         if (ret)
>                 goto exit;
>
> +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> +       if (ret)
> +               goto dev_destroy;
> +
>         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>         if (ret)
>                 goto feature_uinit;
> @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>
>  feature_uinit:
>         fpga_dev_feature_uinit(pdev);
> +dev_destroy:
> +       fme_dev_destroy(pdev);
>  exit:
>         return ret;
>  }
> @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>  {
>         fpga_dev_feature_uinit(pdev);
>         fpga_unregister_dev_ops(pdev);
> +       fme_dev_destroy(pdev);
>         return 0;
>  }
>
> diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> new file mode 100644
> index 0000000..3b44a3e
> --- /dev/null
> +++ b/drivers/fpga/intel/fme-pr.c
> @@ -0,0 +1,400 @@
> +/*
> + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <[e-mail]>
> + *   Xiao Guangrong <[e-mail]>
> + *   Joseph Grecco <[e-mail]>
> + *   Enno Luebbers <[e-mail]>
> + *   Tim Whisonant <[e-mail]>
> + *   Ananda Ravuri <[e-mail]>
> + *   Christopher Rauer <[e-mail]>
> + *   Henry Mitchel <[e-mail]>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#include <linux/types.h>
> +#include <linux/device.h>
> +#include <linux/vmalloc.h>
> +#include <linux/uaccess.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/intel-fpga.h>
> +
> +#include "feature-dev.h"
> +#include "fme.h"
> +
> +#define PR_WAIT_TIMEOUT   8000000
> +
> +#define PR_HOST_STATUS_IDLE    0
> +
> +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> +
> +static ssize_t interface_id_show(struct device *dev,
> +                                struct device_attribute *attr, char *buf)
> +{
> +       u64 intfc_id_l, intfc_id_h;
> +       struct feature_fme_pr *fme_pr
> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> +
> +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> +
> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> +                       (unsigned long long)intfc_id_h,
> +                       (unsigned long long)intfc_id_l);
> +}
> +static DEVICE_ATTR_RO(interface_id);
> +
> +static struct attribute *pr_mgmt_attrs[] = {
> +       &dev_attr_interface_id.attr,
> +       NULL,
> +};
> +
> +struct attribute_group pr_mgmt_attr_group = {
> +       .attrs  = pr_mgmt_attrs,
> +       .name   = "pr",
> +};
> +
> +static u64
> +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> +{
> +       struct feature_fme_pr_status fme_pr_status;
> +       unsigned long err_code;
> +       u64 fme_pr_error;
> +       int i = 0;
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       if (!fme_pr_status.pr_status)
> +               return 0;
> +
> +       err_code = fme_pr_error = readq(&fme_pr->error);
> +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> +       writeq(fme_pr_error, &fme_pr->error);
> +       return fme_pr_error;
> +}
> +
> +static int fme_pr_write_init(struct fpga_manager *mgr,
> +               struct fpga_image_info *info, const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +       if (!fme_pr)
> +               return -EINVAL;
> +
> +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> +               return -EINVAL;

flags is bitmapped so please do:

if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))

> +
> +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       fme_pr_ctl.pr_reset_ack = 1;
> +
> +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_reset = 0;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev,
> +               "waiting for PR resource in HW to be initialized and ready\n");
> +
> +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> +
> +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum PR timeout\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> +       pr_err_handle(pdev, fme_pr);
> +       return 0;
> +}
> +
> +static int fme_pr_write(struct fpga_manager *mgr,
> +                       const char *buf, size_t count)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +       struct feature_fme_pr_status fme_pr_status;
> +       struct feature_fme_pr_data fme_pr_data;
> +       int delay, pr_credit, i = 0;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_regionid = fme->port_id;
> +       fme_pr_ctl.pr_start_req = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> +
> +       fme_pr_status.csr = readq(&fme_pr->status);
> +       pr_credit = fme_pr_status.pr_credit;
> +
> +       while (count > 0) {
> +               delay = 0;
> +               while (pr_credit <= 1) {
> +                       if (delay++ > PR_WAIT_TIMEOUT) {
> +                               dev_err(&pdev->dev, "maximum try\n");
> +                               return -ETIMEDOUT;
> +                       }
> +                       udelay(1);
> +
> +                       fme_pr_status.csr = readq(&fme_pr->status);
> +                       pr_credit = fme_pr_status.pr_credit;
> +               };
> +
> +               if (count >= 4) {
> +                       fme_pr_data.rsvd = 0;
> +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> +                       writeq(fme_pr_data.csr, &fme_pr->data);
> +                       count -= 4;
> +                       pr_credit--;
> +                       i++;
> +               } else {
> +                       WARN_ON(1);
> +                       return -EINVAL;
> +               }
> +       }
> +
> +       return 0;
> +}
> +
> +static int fme_pr_write_complete(struct fpga_manager *mgr,
> +                       struct fpga_image_info *info)
> +{
> +       struct fpga_fme *fme = mgr->priv;
> +       struct platform_device *pdev;
> +       struct feature_fme_pr *fme_pr;
> +       struct feature_fme_pr_ctl fme_pr_ctl;
> +
> +       pdev = fme->pdata->dev;
> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> +                               FME_FEATURE_ID_PR_MGMT);
> +
> +       fme_pr_ctl.csr = readq(&fme_pr->control);
> +       fme_pr_ctl.pr_push_complete = 1;
> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> +
> +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> +
> +       fme_pr_ctl.pr_start_req = 0;
> +
> +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> +               dev_err(&pdev->dev, "maximum try.\n");
> +               return -ETIMEDOUT;
> +       }
> +
> +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> +       if (fme->pr_err)
> +               return -EIO;
> +
> +       dev_dbg(&pdev->dev, "PR done successfully\n");
> +       return 0;
> +}
> +
> +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> +{
> +       return FPGA_MGR_STATE_UNKNOWN;
> +}
> +
> +static const struct fpga_manager_ops fme_pr_ops = {
> +       .write_init = fme_pr_write_init,
> +       .write = fme_pr_write,
> +       .write_complete = fme_pr_write_complete,
> +       .state = fme_pr_state,
> +};
> +

The following fme_pr() function shouldn't be in a fpga-mgr driver.  It
is calling
the framework that this driver is registered with.
A fpga-mgr low level driver is supposed to be a very low level driver.
This function is controlling the port and calling fpga-mgr layer to do the
programming.  There should be a layer above the fpga-mgr and fpga-bridge
that coordinates these things.  I.e. can you use my proposed fpga-region code?

> +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> +{
> +       void __user *argp = (void __user *)arg;
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *fme;
> +       struct fpga_manager *mgr;
> +       struct feature_fme_header *fme_hdr;
> +       struct feature_fme_capability fme_capability;
> +       struct fpga_image_info info;
> +       struct fpga_fme_port_pr port_pr;
> +       struct platform_device *port;
> +       unsigned long minsz;
> +       void *buf = NULL;
> +       int ret = 0;
> +
> +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> +
> +       if (copy_from_user(&port_pr, argp, minsz))
> +               return -EFAULT;
> +
> +       if (port_pr.argsz < minsz || port_pr.flags)
> +               return -EINVAL;
> +
> +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> +               return -EINVAL;
> +
> +       /* get fme header region */
> +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> +                                       FME_FEATURE_ID_HEADER);
> +       if (WARN_ON(!fme_hdr))
> +               return -EINVAL;
> +
> +       /* check port id */
> +       fme_capability.csr = readq(&fme_hdr->capability);
> +       if (port_pr.port_id >= fme_capability.num_ports) {
> +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> +               return -EINVAL;
> +       }
> +
> +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> +                                   port_pr.buffer_size))
> +               return -EFAULT;
> +
> +       buf = vmalloc(port_pr.buffer_size);
> +       if (!buf)
> +               return -ENOMEM;
> +
> +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> +                                              port_pr.buffer_size)) {
> +               ret = -EFAULT;
> +               goto free_exit;
> +       }
> +
> +       memset(&info, 0, sizeof(struct fpga_image_info));
> +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> +
> +       mgr = fpga_mgr_get(&pdev->dev);
> +       if (IS_ERR(mgr)) {
> +               ret = PTR_ERR(mgr);
> +               goto free_exit;
> +       }
> +
> +       mutex_lock(&pdata->lock);
> +       fme = fpga_pdata_get_private(pdata);
> +       /* fme device has been unregistered. */
> +       if (!fme) {
> +               ret = -EINVAL;
> +               goto unlock_exit;
> +       }
> +
> +       fme->pr_err = 0;
> +       fme->port_id = port_pr.port_id;
> +
> +       /* Find and get port device by index */
> +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> +                                        fpga_port_check_id);
> +       WARN_ON(!port);
> +
> +       /* Disable Port before PR */
> +       fpga_port_disable(port);
> +
> +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> +       port_pr.status = fme->pr_err;
> +
> +       /* Re-enable Port after PR finished */
> +       fpga_port_enable(port);
> +
> +       put_device(&port->dev);
> +
> +unlock_exit:
> +       mutex_unlock(&pdata->lock);
> +       fpga_mgr_put(mgr);
> +free_exit:
> +       vfree(buf);
> +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> +               return -EFAULT;
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_probe(struct platform_device *pdev)
> +{
> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> +       struct fpga_fme *priv;
> +       int ret;
> +
> +       mutex_lock(&pdata->lock);
> +       priv = fpga_pdata_get_private(pdata);
> +       ret = fpga_mgr_register(&pdata->dev->dev,
> +               "Intel FPGA Manager", &fme_pr_ops, priv);
> +       mutex_unlock(&pdata->lock);
> +
> +       return ret;
> +}
> +
> +static int fpga_fme_pr_remove(struct platform_device *pdev)
> +{
> +       fpga_mgr_unregister(&pdev->dev);
> +       return 0;
> +}
> +
> +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> +{
> +       int ret;
> +
> +       ret = fpga_fme_pr_probe(pdev);
> +       if (ret)
> +               return ret;
> +
> +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       if (ret)
> +               fpga_fme_pr_remove(pdev);
> +
> +       return ret;
> +}
> +
> +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> +{
> +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> +       fpga_fme_pr_remove(pdev);
> +}
> +
> +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> +       unsigned int cmd, unsigned long arg)
> +{
> +       long ret;
> +
> +       switch (cmd) {
> +       case FPGA_FME_PORT_PR:
> +               ret = fme_pr(pdev, arg);
> +               break;
> +       default:
> +               ret = -ENODEV;
> +       }
> +
> +       return ret;
> +}
> +
> +struct feature_ops pr_mgmt_ops = {
> +       .init = pr_mgmt_init,
> +       .uinit = pr_mgmt_uinit,
> +       .ioctl = fme_pr_ioctl,
> +};
> diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> new file mode 100644
> index 0000000..d6cb7ce
> --- /dev/null
> +++ b/drivers/fpga/intel/fme.h
> @@ -0,0 +1,32 @@
> +/*
> + * Header file for Intel FPGA Management Engine (FME) Driver
> + *
> + * Copyright (C) 2017 Intel Corporation, Inc.
> + *
> + * Authors:
> + *   Kang Luwei <[e-mail]>
> + *   Xiao Guangrong <[e-mail]>
> + *   Joseph Grecco <[e-mail]>
> + *   Enno Luebbers <[e-mail]>
> + *   Tim Whisonant <[e-mail]>
> + *   Ananda Ravuri <[e-mail]>
> + *   Henry Mitchel <[e-mail]>
> + *
> + * This work is licensed under a dual BSD/GPLv2 license. When using or
> + * redistributing this file, you may do so under either license. See the
> + * LICENSE.BSD file under this directory for the BSD license and see
> + * the COPYING file in the top-level directory for the GPLv2 license.
> + */
> +
> +#ifndef __INTEL_FME_H
> +#define __INTEL_FME_H
> +
> +struct fpga_fme {
> +       u8  port_id;
> +       u64 pr_err;
> +       struct feature_platform_data *pdata;
> +};
> +
> +extern struct feature_ops pr_mgmt_ops;
> +
> +#endif
> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> index 992e556..77658316 100644
> --- a/include/uapi/linux/intel-fpga.h
> +++ b/include/uapi/linux/intel-fpga.h
> @@ -18,6 +18,8 @@
>  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>  #define _UAPI_LINUX_INTEL_FPGA_H
>
> +#include <linux/types.h>
> +
>  #define FPGA_API_VERSION 0
>
>  /*
> @@ -30,6 +32,7 @@
>  #define FPGA_MAGIC 0xB6
>
>  #define FPGA_BASE 0
> +#define FME_BASE 0x80
>
>  /**
>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> @@ -49,4 +52,45 @@
>
>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>
> +/* IOCTLs for FME file descriptor */
> +
> +/**
> + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> + *
> + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> + * provided by caller.
> + * Return: 0 on success, -errno on failure.
> + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> + * some errors during PR, under this case, the user can fetch HW error code
> + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> + * Otherwise, it is always zero.
> + */
> +
> +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> +static const char * const _name_[] = {                 \
> +       "PR operation error detected",                  \
> +       "PR CRC error detected",                        \
> +       "PR incompatiable bitstream error detected",    \
> +       "PR IP protocol error detected",                \
> +       "PR FIFO overflow error detected",              \
> +       "Reserved",                                     \
> +       "PR secure load error detected",                \
> +}
> +
> +#define PR_MAX_ERR_NUM 7
> +
> +struct fpga_fme_port_pr {
> +       /* Input */
> +       __u32 argsz;            /* Structure length */
> +       __u32 flags;            /* Zero for now */
> +       __u32 port_id;
> +       __u32 buffer_size;
> +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> +       /* Output */
> +       __u64 status;           /* HW error code if ioctl returns -EIO */
> +};
> +
> +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> +
>  #endif /* _UAPI_INTEL_FPGA_H */
> --
> 2.7.4
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Matthew Gerlach April 3, 2017, 10:49 p.m. UTC | #11

On Mon, 3 Apr 2017, Alan Tull wrote:

> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
>> From: Kang Luwei <[e-mail]>
>>
>> Partial Reconfiguration (PR) is the most important function for FME. It
>> allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>>
>> This patch adds support for PR sub feature. In this patch, it registers
>> a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> for PR operation once PR request received via ioctl.
>
> The code that invokes fpga_mgr_buf_load should be a different layer.
>
>> Below user space
>> interfaces are exposed by this sub feature.
>>
>> Sysfs interface:
>> * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>>   Read-only. Indicate the hardware interface information. Userspace
>>   applications need to check this interface to select correct green
>>   bitstream format before PR.
>>
>> Ioctl interface:
>> * FPGA_FME_PORT_PR
>>   Do partial reconfiguration per information from userspace, including
>>   target port(AFU), buffer size and address info. It returns the PR status
>>   (PR error code if failed) to userspace.
>>
>> Signed-off-by: Tim Whisonant <[e-mail]>
>> Signed-off-by: Enno Luebbers <[e-mail]>
>> Signed-off-by: Shiva Rao <[e-mail]>
>> Signed-off-by: Christopher Rauer <[e-mail]>
>> Signed-off-by: Alan Tull <[e-mail]>
>> Signed-off-by: Kang Luwei <[e-mail]>
>> Signed-off-by: Xiao Guangrong <[e-mail]>
>> Signed-off-by: Wu Hao <[e-mail]>
>> ---
>>  drivers/fpga/intel/Makefile      |   2 +-
>>  drivers/fpga/intel/feature-dev.h |  58 ++++++
>>  drivers/fpga/intel/fme-main.c    |  44 ++++-
>>  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>>  drivers/fpga/intel/fme.h         |  32 ++++
>>  include/uapi/linux/intel-fpga.h  |  44 +++++
>>  6 files changed, 578 insertions(+), 2 deletions(-)
>>  create mode 100644 drivers/fpga/intel/fme-pr.c
>>  create mode 100644 drivers/fpga/intel/fme.h
>>
>> diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> index 546861d..0452cb6 100644
>> --- a/drivers/fpga/intel/Makefile
>> +++ b/drivers/fpga/intel/Makefile
>> @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>>  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>>
>>  intel-fpga-pci-objs := pcie.o feature-dev.o
>> -intel-fpga-fme-objs := fme-main.o
>> +intel-fpga-fme-objs := fme-main.o fme-pr.o
>> diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
>> index dccc283..5a25c915 100644
>> --- a/drivers/fpga/intel/feature-dev.h
>> +++ b/drivers/fpga/intel/feature-dev.h
>> @@ -150,8 +150,66 @@ struct feature_fme_err {
>>  };
>>
>>  /* FME Partial Reconfiguration Sub Feature Register Set */
>> +/* FME PR Control Register */
>> +struct feature_fme_pr_ctl {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u8  pr_reset:1;         /* Reset PR Engine */
>> +                       u8  rsvdz1:3;
>> +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
>> +                       u8  rsvdz2:3;
>> +                       u8  pr_regionid:2;      /* PR Region ID */
>> +                       u8  rsvdz3:2;
>> +                       u8  pr_start_req:1;     /* PR Start Request */
>> +                       u8  pr_push_complete:1; /* PR Data push complete */
>> +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
>> +                       u32 rsvdz4:17;
>> +                       u32 config_data;
>> +               };
>> +       };
>> +};
>> +
>> +/* FME PR Status Register */
>> +struct feature_fme_pr_status {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       u16 pr_credit:9;        /* Number of PR Credits */
>> +                       u8  rsvdz1:7;
>> +                       u8  pr_status:1;        /* PR Operation status */
>> +                       u8  rsvdz2:3;
>> +                       u8  pr_ctrlr_status:3;  /* Controller status */
>> +                       u8  rsvdz3:1;
>> +                       u8  pr_host_status:4;   /* PR Host status */
>> +                       u64 rsvdz4:36;
>> +               };
>> +       };
>> +};
>> +
>> +/* FME PR Data Register */
>> +struct feature_fme_pr_data {
>> +       union {
>> +               u64 csr;
>> +               struct {
>> +                       /* PR data from the raw-binary file */
>> +                       u32 pr_data_raw;
>> +                       u32 rsvd;
>> +               };
>> +       };
>> +};
>> +
>>  struct feature_fme_pr {
>>         struct feature_header header;
>> +       struct feature_fme_pr_ctl control;
>> +       struct feature_fme_pr_status status;
>> +       struct feature_fme_pr_data data;
>> +       u64 error;
>> +
>> +       u64 rsvd[16];
>> +
>> +       u64 intfc_id_l;         /* PR interface Id Low */
>> +       u64 intfc_id_h;         /* PR interface Id High */
>>  };
>>
>>  /* PORT Header Register Set */
>> diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
>> index 36d0c4c..0d9a7a6 100644
>> --- a/drivers/fpga/intel/fme-main.c
>> +++ b/drivers/fpga/intel/fme-main.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/intel-fpga.h>
>>
>>  #include "feature-dev.h"
>> +#include "fme.h"
>>
>>  static ssize_t ports_num_show(struct device *dev,
>>                               struct device_attribute *attr, char *buf)
>> @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>>                 .ops = &fme_hdr_ops,
>>         },
>>         {
>> +               .name = FME_FEATURE_PR_MGMT,
>> +               .ops = &pr_mgmt_ops,
>> +       },
>> +       {
>>                 .ops = NULL,
>>         },
>>  };
>> @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>>         .unlocked_ioctl = fme_ioctl,
>>  };
>>
>> +static int fme_dev_init(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *fme;
>> +
>> +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
>> +       if (!fme)
>> +               return -ENOMEM;
>> +
>> +       fme->pdata = pdata;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       fpga_pdata_set_private(pdata, fme);
>> +       mutex_unlock(&pdata->lock);
>> +       return 0;
>> +}
>> +
>> +static void fme_dev_destroy(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *fme;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       fme = fpga_pdata_get_private(pdata);
>> +       fpga_pdata_set_private(pdata, NULL);
>> +       mutex_unlock(&pdata->lock);
>> +
>> +       devm_kfree(&pdev->dev, fme);
>> +}
>> +
>>  static int fme_probe(struct platform_device *pdev)
>>  {
>>         int ret;
>>
>> -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> +       ret = fme_dev_init(pdev);
>>         if (ret)
>>                 goto exit;
>>
>> +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> +       if (ret)
>> +               goto dev_destroy;
>> +
>>         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>>         if (ret)
>>                 goto feature_uinit;
>> @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>>
>>  feature_uinit:
>>         fpga_dev_feature_uinit(pdev);
>> +dev_destroy:
>> +       fme_dev_destroy(pdev);
>>  exit:
>>         return ret;
>>  }
>> @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>>  {
>>         fpga_dev_feature_uinit(pdev);
>>         fpga_unregister_dev_ops(pdev);
>> +       fme_dev_destroy(pdev);
>>         return 0;
>>  }
>>
>> diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
>> new file mode 100644
>> index 0000000..3b44a3e
>> --- /dev/null
>> +++ b/drivers/fpga/intel/fme-pr.c
>> @@ -0,0 +1,400 @@
>> +/*
>> + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <[e-mail]>
>> + *   Xiao Guangrong <[e-mail]>
>> + *   Joseph Grecco <[e-mail]>
>> + *   Enno Luebbers <[e-mail]>
>> + *   Tim Whisonant <[e-mail]>
>> + *   Ananda Ravuri <[e-mail]>
>> + *   Christopher Rauer <[e-mail]>
>> + *   Henry Mitchel <[e-mail]>
>> + *
>> + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> + * redistributing this file, you may do so under either license. See the
>> + * LICENSE.BSD file under this directory for the BSD license and see
>> + * the COPYING file in the top-level directory for the GPLv2 license.
>> + */
>> +
>> +#include <linux/types.h>
>> +#include <linux/device.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/uaccess.h>
>> +#include <linux/fpga/fpga-mgr.h>
>> +#include <linux/intel-fpga.h>
>> +
>> +#include "feature-dev.h"
>> +#include "fme.h"
>> +
>> +#define PR_WAIT_TIMEOUT   8000000
>> +
>> +#define PR_HOST_STATUS_IDLE    0
>> +
>> +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
>> +
>> +static ssize_t interface_id_show(struct device *dev,
>> +                                struct device_attribute *attr, char *buf)
>> +{
>> +       u64 intfc_id_l, intfc_id_h;
>> +       struct feature_fme_pr *fme_pr
>> +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
>> +
>> +       intfc_id_l = readq(&fme_pr->intfc_id_l);
>> +       intfc_id_h = readq(&fme_pr->intfc_id_h);
>> +
>> +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> +                       (unsigned long long)intfc_id_h,
>> +                       (unsigned long long)intfc_id_l);
>> +}
>> +static DEVICE_ATTR_RO(interface_id);
>> +
>> +static struct attribute *pr_mgmt_attrs[] = {
>> +       &dev_attr_interface_id.attr,
>> +       NULL,
>> +};
>> +
>> +struct attribute_group pr_mgmt_attr_group = {
>> +       .attrs  = pr_mgmt_attrs,
>> +       .name   = "pr",
>> +};
>> +
>> +static u64
>> +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
>> +{
>> +       struct feature_fme_pr_status fme_pr_status;
>> +       unsigned long err_code;
>> +       u64 fme_pr_error;
>> +       int i = 0;
>> +
>> +       fme_pr_status.csr = readq(&fme_pr->status);
>> +       if (!fme_pr_status.pr_status)
>> +               return 0;
>> +
>> +       err_code = fme_pr_error = readq(&fme_pr->error);
>> +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
>> +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
>> +       writeq(fme_pr_error, &fme_pr->error);
>> +       return fme_pr_error;
>> +}
>> +
>> +static int fme_pr_write_init(struct fpga_manager *mgr,
>> +               struct fpga_image_info *info, const char *buf, size_t count)
>> +{
>> +       struct fpga_fme *fme = mgr->priv;
>> +       struct platform_device *pdev;
>> +       struct feature_fme_pr *fme_pr;
>> +       struct feature_fme_pr_ctl fme_pr_ctl;
>> +       struct feature_fme_pr_status fme_pr_status;
>> +
>> +       pdev = fme->pdata->dev;
>> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                               FME_FEATURE_ID_PR_MGMT);
>> +       if (!fme_pr)
>> +               return -EINVAL;
>> +
>> +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
>> +               return -EINVAL;
>
> flags is bitmapped so please do:
>
> if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))
>
>> +
>> +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_reset = 1;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       fme_pr_ctl.pr_reset_ack = 1;
>> +
>> +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
>> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_reset = 0;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       dev_dbg(&pdev->dev,
>> +               "waiting for PR resource in HW to be initialized and ready\n");
>> +
>> +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> +
>> +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
>> +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
>> +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
>> +       pr_err_handle(pdev, fme_pr);
>> +       return 0;
>> +}
>> +
>> +static int fme_pr_write(struct fpga_manager *mgr,
>> +                       const char *buf, size_t count)
>> +{
>> +       struct fpga_fme *fme = mgr->priv;
>> +       struct platform_device *pdev;
>> +       struct feature_fme_pr *fme_pr;
>> +       struct feature_fme_pr_ctl fme_pr_ctl;
>> +       struct feature_fme_pr_status fme_pr_status;
>> +       struct feature_fme_pr_data fme_pr_data;
>> +       int delay, pr_credit, i = 0;
>> +
>> +       pdev = fme->pdata->dev;
>> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                               FME_FEATURE_ID_PR_MGMT);
>> +
>> +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_regionid = fme->port_id;
>> +       fme_pr_ctl.pr_start_req = 1;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
>> +
>> +       fme_pr_status.csr = readq(&fme_pr->status);
>> +       pr_credit = fme_pr_status.pr_credit;
>> +
>> +       while (count > 0) {
>> +               delay = 0;
>> +               while (pr_credit <= 1) {
>> +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> +                               dev_err(&pdev->dev, "maximum try\n");
>> +                               return -ETIMEDOUT;
>> +                       }
>> +                       udelay(1);
>> +
>> +                       fme_pr_status.csr = readq(&fme_pr->status);
>> +                       pr_credit = fme_pr_status.pr_credit;
>> +               };
>> +
>> +               if (count >= 4) {
>> +                       fme_pr_data.rsvd = 0;
>> +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> +                       writeq(fme_pr_data.csr, &fme_pr->data);
>> +                       count -= 4;
>> +                       pr_credit--;
>> +                       i++;
>> +               } else {
>> +                       WARN_ON(1);
>> +                       return -EINVAL;
>> +               }
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>> +static int fme_pr_write_complete(struct fpga_manager *mgr,
>> +                       struct fpga_image_info *info)
>> +{
>> +       struct fpga_fme *fme = mgr->priv;
>> +       struct platform_device *pdev;
>> +       struct feature_fme_pr *fme_pr;
>> +       struct feature_fme_pr_ctl fme_pr_ctl;
>> +
>> +       pdev = fme->pdata->dev;
>> +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                               FME_FEATURE_ID_PR_MGMT);
>> +
>> +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> +       fme_pr_ctl.pr_push_complete = 1;
>> +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> +
>> +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
>> +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
>> +
>> +       fme_pr_ctl.pr_start_req = 0;
>> +
>> +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
>> +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> +               dev_err(&pdev->dev, "maximum try.\n");
>> +               return -ETIMEDOUT;
>> +       }
>> +
>> +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
>> +       fme->pr_err = pr_err_handle(pdev, fme_pr);
>> +       if (fme->pr_err)
>> +               return -EIO;
>> +
>> +       dev_dbg(&pdev->dev, "PR done successfully\n");
>> +       return 0;
>> +}
>> +
>> +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
>> +{
>> +       return FPGA_MGR_STATE_UNKNOWN;
>> +}

The functions that implement the fme_pr_ops are really a low level fpga
manager driver for the Altera PR IP component.  A standalone version of
such a driver has been reviewed and Acked.  See the links below.
Could this file use those functions and remove this code?

http://marc.info/?l=linux-kernel&m=149019678925564&w=2
http://marc.info/?l=linux-kernel&m=149019654225457&w=2
http://marc.info/?l=linux-kernel&m=149019598025274&w=2
http://marc.info/?l=linux-kernel&m=149013051007149&w=2

>> +
>> +static const struct fpga_manager_ops fme_pr_ops = {
>> +       .write_init = fme_pr_write_init,
>> +       .write = fme_pr_write,
>> +       .write_complete = fme_pr_write_complete,
>> +       .state = fme_pr_state,
>> +};
>> +
>
> The following fme_pr() function shouldn't be in a fpga-mgr driver.  It
> is calling
> the framework that this driver is registered with.
> A fpga-mgr low level driver is supposed to be a very low level driver.
> This function is controlling the port and calling fpga-mgr layer to do the
> programming.  There should be a layer above the fpga-mgr and fpga-bridge
> that coordinates these things.  I.e. can you use my proposed fpga-region code?
>
>> +static int fme_pr(struct platform_device *pdev, unsigned long arg)
>> +{
>> +       void __user *argp = (void __user *)arg;
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *fme;
>> +       struct fpga_manager *mgr;
>> +       struct feature_fme_header *fme_hdr;
>> +       struct feature_fme_capability fme_capability;
>> +       struct fpga_image_info info;
>> +       struct fpga_fme_port_pr port_pr;
>> +       struct platform_device *port;
>> +       unsigned long minsz;
>> +       void *buf = NULL;
>> +       int ret = 0;
>> +
>> +       minsz = offsetofend(struct fpga_fme_port_pr, status);
>> +
>> +       if (copy_from_user(&port_pr, argp, minsz))
>> +               return -EFAULT;
>> +
>> +       if (port_pr.argsz < minsz || port_pr.flags)
>> +               return -EINVAL;
>> +
>> +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
>> +               return -EINVAL;
>> +
>> +       /* get fme header region */
>> +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> +                                       FME_FEATURE_ID_HEADER);
>> +       if (WARN_ON(!fme_hdr))
>> +               return -EINVAL;
>> +
>> +       /* check port id */
>> +       fme_capability.csr = readq(&fme_hdr->capability);
>> +       if (port_pr.port_id >= fme_capability.num_ports) {
>> +               dev_dbg(&pdev->dev, "port number more than maximum\n");
>> +               return -EINVAL;
>> +       }
>> +
>> +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
>> +                                   port_pr.buffer_size))
>> +               return -EFAULT;
>> +
>> +       buf = vmalloc(port_pr.buffer_size);
>> +       if (!buf)
>> +               return -ENOMEM;
>> +
>> +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
>> +                                              port_pr.buffer_size)) {
>> +               ret = -EFAULT;
>> +               goto free_exit;
>> +       }
>> +
>> +       memset(&info, 0, sizeof(struct fpga_image_info));
>> +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
>> +
>> +       mgr = fpga_mgr_get(&pdev->dev);
>> +       if (IS_ERR(mgr)) {
>> +               ret = PTR_ERR(mgr);
>> +               goto free_exit;
>> +       }
>> +
>> +       mutex_lock(&pdata->lock);
>> +       fme = fpga_pdata_get_private(pdata);
>> +       /* fme device has been unregistered. */
>> +       if (!fme) {
>> +               ret = -EINVAL;
>> +               goto unlock_exit;
>> +       }
>> +
>> +       fme->pr_err = 0;
>> +       fme->port_id = port_pr.port_id;
>> +
>> +       /* Find and get port device by index */
>> +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
>> +                                        fpga_port_check_id);
>> +       WARN_ON(!port);
>> +
>> +       /* Disable Port before PR */
>> +       fpga_port_disable(port);
>> +
>> +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
>> +       port_pr.status = fme->pr_err;
>> +
>> +       /* Re-enable Port after PR finished */
>> +       fpga_port_enable(port);
>> +
>> +       put_device(&port->dev);
>> +
>> +unlock_exit:
>> +       mutex_unlock(&pdata->lock);
>> +       fpga_mgr_put(mgr);
>> +free_exit:
>> +       vfree(buf);
>> +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
>> +               return -EFAULT;
>> +       return ret;
>> +}
>> +
>> +static int fpga_fme_pr_probe(struct platform_device *pdev)
>> +{
>> +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> +       struct fpga_fme *priv;
>> +       int ret;
>> +
>> +       mutex_lock(&pdata->lock);
>> +       priv = fpga_pdata_get_private(pdata);
>> +       ret = fpga_mgr_register(&pdata->dev->dev,
>> +               "Intel FPGA Manager", &fme_pr_ops, priv);

The call to fpga_mgr_register() above would change to a call to
alt_pr_register() if the standalone version of the PR IP driver were used.

>> +       mutex_unlock(&pdata->lock);
>> +
>> +       return ret;
>> +}
>> +
>> +static int fpga_fme_pr_remove(struct platform_device *pdev)
>> +{
>> +       fpga_mgr_unregister(&pdev->dev);

This call to fpga_mgr_unregister() would change to a call to
alt_pr_unregister() if the standalone version of the PR IP driver were
used.

>> +       return 0;
>> +}
>> +
>> +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
>> +{
>> +       int ret;
>> +
>> +       ret = fpga_fme_pr_probe(pdev);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> +       if (ret)
>> +               fpga_fme_pr_remove(pdev);
>> +
>> +       return ret;
>> +}
>> +
>> +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
>> +{
>> +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
>> +       fpga_fme_pr_remove(pdev);
>> +}
>> +
>> +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
>> +       unsigned int cmd, unsigned long arg)
>> +{
>> +       long ret;
>> +
>> +       switch (cmd) {
>> +       case FPGA_FME_PORT_PR:
>> +               ret = fme_pr(pdev, arg);
>> +               break;
>> +       default:
>> +               ret = -ENODEV;
>> +       }
>> +
>> +       return ret;
>> +}
>> +
>> +struct feature_ops pr_mgmt_ops = {
>> +       .init = pr_mgmt_init,
>> +       .uinit = pr_mgmt_uinit,
>> +       .ioctl = fme_pr_ioctl,
>> +};
>> diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
>> new file mode 100644
>> index 0000000..d6cb7ce
>> --- /dev/null
>> +++ b/drivers/fpga/intel/fme.h
>> @@ -0,0 +1,32 @@
>> +/*
>> + * Header file for Intel FPGA Management Engine (FME) Driver
>> + *
>> + * Copyright (C) 2017 Intel Corporation, Inc.
>> + *
>> + * Authors:
>> + *   Kang Luwei <[e-mail]>
>> + *   Xiao Guangrong <[e-mail]>
>> + *   Joseph Grecco <[e-mail]>
>> + *   Enno Luebbers <[e-mail]>
>> + *   Tim Whisonant <[e-mail]>
>> + *   Ananda Ravuri <[e-mail]>
>> + *   Henry Mitchel <[e-mail]>
>> + *
>> + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> + * redistributing this file, you may do so under either license. See the
>> + * LICENSE.BSD file under this directory for the BSD license and see
>> + * the COPYING file in the top-level directory for the GPLv2 license.
>> + */
>> +
>> +#ifndef __INTEL_FME_H
>> +#define __INTEL_FME_H
>> +
>> +struct fpga_fme {
>> +       u8  port_id;
>> +       u64 pr_err;
>> +       struct feature_platform_data *pdata;
>> +};
>> +
>> +extern struct feature_ops pr_mgmt_ops;
>> +
>> +#endif
>> diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
>> index 992e556..77658316 100644
>> --- a/include/uapi/linux/intel-fpga.h
>> +++ b/include/uapi/linux/intel-fpga.h
>> @@ -18,6 +18,8 @@
>>  #ifndef _UAPI_LINUX_INTEL_FPGA_H
>>  #define _UAPI_LINUX_INTEL_FPGA_H
>>
>> +#include <linux/types.h>
>> +
>>  #define FPGA_API_VERSION 0
>>
>>  /*
>> @@ -30,6 +32,7 @@
>>  #define FPGA_MAGIC 0xB6
>>
>>  #define FPGA_BASE 0
>> +#define FME_BASE 0x80
>>
>>  /**
>>   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
>> @@ -49,4 +52,45 @@
>>
>>  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
>>
>> +/* IOCTLs for FME file descriptor */
>> +
>> +/**
>> + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
>> + *
>> + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
>> + * provided by caller.
>> + * Return: 0 on success, -errno on failure.
>> + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
>> + * some errors during PR, under this case, the user can fetch HW error code
>> + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
>> + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
>> + * Otherwise, it is always zero.
>> + */
>> +
>> +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
>> +static const char * const _name_[] = {                 \
>> +       "PR operation error detected",                  \
>> +       "PR CRC error detected",                        \
>> +       "PR incompatiable bitstream error detected",    \
>> +       "PR IP protocol error detected",                \
>> +       "PR FIFO overflow error detected",              \
>> +       "Reserved",                                     \
>> +       "PR secure load error detected",                \
>> +}
>> +
>> +#define PR_MAX_ERR_NUM 7
>> +
>> +struct fpga_fme_port_pr {
>> +       /* Input */
>> +       __u32 argsz;            /* Structure length */
>> +       __u32 flags;            /* Zero for now */
>> +       __u32 port_id;
>> +       __u32 buffer_size;
>> +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
>> +       /* Output */
>> +       __u64 status;           /* HW error code if ioctl returns -EIO */
>> +};
>> +
>> +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
>> +
>>  #endif /* _UAPI_INTEL_FPGA_H */
>> --
>> 2.7.4
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 4, 2017, 5:25 a.m. UTC | #12

On Mon, Apr 03, 2017 at 03:26:14PM -0500, Alan Tull wrote:
> On Fri, Mar 31, 2017 at 3:50 AM, Wu Hao <[e-mail]> wrote:
> > On Fri, Mar 31, 2017 at 12:11:12PM +0800, Xiao Guangrong wrote:
> >> On 31/03/2017 4:30 AM, Alan Tull wrote:
> >> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> >> >>From: Kang Luwei <[e-mail]>
> >> >>
> >> >>Partial Reconfiguration (PR) is the most important function for FME. It
> >> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >> >>
> >> >>This patch adds support for PR sub feature. In this patch, it registers
> >> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >> >>for PR operation once PR request received via ioctl. Below user space
> >> >>interfaces are exposed by this sub feature.
> >> >>
> >> >>Sysfs interface:
> >> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >> >>  Read-only. Indicate the hardware interface information. Userspace
> >> >>  applications need to check this interface to select correct green
> >> >>  bitstream format before PR.
> >> >>
> >> >>Ioctl interface:
> >> >>* FPGA_FME_PORT_PR
> >> >>  Do partial reconfiguration per information from userspace, including
> >> >>  target port(AFU), buffer size and address info. It returns the PR status
> >> >>  (PR error code if failed) to userspace.
> >> >>
> >> >>Signed-off-by: Tim Whisonant <[e-mail]>
> >> >>Signed-off-by: Enno Luebbers <[e-mail]>
> >> >>Signed-off-by: Shiva Rao <[e-mail]>
> >> >>Signed-off-by: Christopher Rauer <[e-mail]>
> >> >>Signed-off-by: Alan Tull <[e-mail]>
> >> >
> >> >Hi Wu Hao,
> >> >
> >> >Thanks for submitting your patches.
> >> >
> >> >I think there's been a misunderstanding of the meaning of
> >> >'Signed-off-by' [1].  I have not signed off on this code or had a hand
> >> >in its development.  But I'm happy to get to review it now.  It will
> >> >take a bit of time; I expect to be replying next week.
> >>
> >> Hi Alan,
> >>
> >> Sorry to confuse you, i think it's because you helped Chris a lot to
> >> implement this interface and we'd like to include your credit as this
> >> way. If you dislike, it will be dropped. :)
> >>
> >> Thanks for your review in advance.
> >>
> >
> > Hi Alan,
> >
> > Sorry about this, we should ask you firstly before doing it this way.
> > Let me know if you don't like it, I will drop it in the next version.
> 
> Yes please drop the signed-off-by: me in the next version.  Also, you
> don't need to cc my Intel email address.
> 

Sure, will do.

Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 4, 2017, 6:05 a.m. UTC | #13

On Mon, Apr 03, 2017 at 11:30:55AM -0500, Alan Tull wrote:
> On Sat, Apr 1, 2017 at 6:08 AM, Wu Hao <[e-mail]> wrote:
> > On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
> >> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> >> > From: Kang Luwei <[e-mail]>
> >> >
> >> > Partial Reconfiguration (PR) is the most important function for FME. It
> >> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >> >
> >> > This patch adds support for PR sub feature. In this patch, it registers
> >> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >> > for PR operation once PR request received via ioctl. Below user space
> >> > interfaces are exposed by this sub feature.
> >> >
> >> > Sysfs interface:
> >> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >> >   Read-only. Indicate the hardware interface information. Userspace
> >> >   applications need to check this interface to select correct green
> >> >   bitstream format before PR.
> >> >
> >> > Ioctl interface:
> >> > * FPGA_FME_PORT_PR
> >> >   Do partial reconfiguration per information from userspace, including
> >> >   target port(AFU), buffer size and address info. It returns the PR status
> >> >   (PR error code if failed) to userspace.
> >> >
> >> > Signed-off-by: Tim Whisonant <[e-mail]>
> >> > Signed-off-by: Enno Luebbers <[e-mail]>
> >> > Signed-off-by: Shiva Rao <[e-mail]>
> >> > Signed-off-by: Christopher Rauer <[e-mail]>
> >> > Signed-off-by: Alan Tull <[e-mail]>
> >> > Signed-off-by: Kang Luwei <[e-mail]>
> >> > Signed-off-by: Xiao Guangrong <[e-mail]>
> >> > Signed-off-by: Wu Hao <[e-mail]>
> >> > ---
> >> >  drivers/fpga/intel/Makefile      |   2 +-
> >> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
> >> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
> >> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >> >  drivers/fpga/intel/fme.h         |  32 ++++
> >> >  include/uapi/linux/intel-fpga.h  |  44 +++++
> >> >  6 files changed, 578 insertions(+), 2 deletions(-)
> >> >  create mode 100644 drivers/fpga/intel/fme-pr.c
> >> >  create mode 100644 drivers/fpga/intel/fme.h
> >> >
> >> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> >> > index 546861d..0452cb6 100644
> >> > --- a/drivers/fpga/intel/Makefile
> >> > +++ b/drivers/fpga/intel/Makefile
> >> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >> >
> >> >  intel-fpga-pci-objs := pcie.o feature-dev.o
> >> > -intel-fpga-fme-objs := fme-main.o
> >> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
> >> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> >> > index dccc283..5a25c915 100644
> >> > --- a/drivers/fpga/intel/feature-dev.h
> >> > +++ b/drivers/fpga/intel/feature-dev.h
> >> > @@ -150,8 +150,66 @@ struct feature_fme_err {
> >> >  };
> >> >
> >> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> >> > +/* FME PR Control Register */
> >> > +struct feature_fme_pr_ctl {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u8  pr_reset:1;         /* Reset PR Engine */
> >> > +                       u8  rsvdz1:3;
> >> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> >> > +                       u8  rsvdz2:3;
> >> > +                       u8  pr_regionid:2;      /* PR Region ID */
> >> > +                       u8  rsvdz3:2;
> >> > +                       u8  pr_start_req:1;     /* PR Start Request */
> >> > +                       u8  pr_push_complete:1; /* PR Data push complete */
> >> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> >> > +                       u32 rsvdz4:17;
> >> > +                       u32 config_data;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* FME PR Status Register */
> >> > +struct feature_fme_pr_status {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       u16 pr_credit:9;        /* Number of PR Credits */
> >> > +                       u8  rsvdz1:7;
> >> > +                       u8  pr_status:1;        /* PR Operation status */
> >> > +                       u8  rsvdz2:3;
> >> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
> >> > +                       u8  rsvdz3:1;
> >> > +                       u8  pr_host_status:4;   /* PR Host status */
> >> > +                       u64 rsvdz4:36;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> > +/* FME PR Data Register */
> >> > +struct feature_fme_pr_data {
> >> > +       union {
> >> > +               u64 csr;
> >> > +               struct {
> >> > +                       /* PR data from the raw-binary file */
> >> > +                       u32 pr_data_raw;
> >> > +                       u32 rsvd;
> >> > +               };
> >> > +       };
> >> > +};
> >> > +
> >> >  struct feature_fme_pr {
> >> >         struct feature_header header;
> >> > +       struct feature_fme_pr_ctl control;
> >> > +       struct feature_fme_pr_status status;
> >> > +       struct feature_fme_pr_data data;
> >> > +       u64 error;
> >> > +
> >> > +       u64 rsvd[16];
> >> > +
> >> > +       u64 intfc_id_l;         /* PR interface Id Low */
> >> > +       u64 intfc_id_h;         /* PR interface Id High */
> >> >  };
> >> >
> >> >  /* PORT Header Register Set */
> >> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> >> > index 36d0c4c..0d9a7a6 100644
> >> > --- a/drivers/fpga/intel/fme-main.c
> >> > +++ b/drivers/fpga/intel/fme-main.c
> >> > @@ -23,6 +23,7 @@
> >> >  #include <linux/intel-fpga.h>
> >> >
> >> >  #include "feature-dev.h"
> >> > +#include "fme.h"
> >> >
> >> >  static ssize_t ports_num_show(struct device *dev,
> >> >                               struct device_attribute *attr, char *buf)
> >> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >> >                 .ops = &fme_hdr_ops,
> >> >         },
> >> >         {
> >> > +               .name = FME_FEATURE_PR_MGMT,
> >> > +               .ops = &pr_mgmt_ops,
> >> > +       },
> >> > +       {
> >> >                 .ops = NULL,
> >> >         },
> >> >  };
> >> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >> >         .unlocked_ioctl = fme_ioctl,
> >> >  };
> >> >
> >> > +static int fme_dev_init(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *fme;
> >> > +
> >> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> >> > +       if (!fme)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       fme->pdata = pdata;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       fpga_pdata_set_private(pdata, fme);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static void fme_dev_destroy(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *fme;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       fme = fpga_pdata_get_private(pdata);
> >> > +       fpga_pdata_set_private(pdata, NULL);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +
> >> > +       devm_kfree(&pdev->dev, fme);
> >> > +}
> >> > +
> >> >  static int fme_probe(struct platform_device *pdev)
> >> >  {
> >> >         int ret;
> >> >
> >> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >> > +       ret = fme_dev_init(pdev);
> >> >         if (ret)
> >> >                 goto exit;
> >> >
> >> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >> > +       if (ret)
> >> > +               goto dev_destroy;
> >> > +
> >> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >> >         if (ret)
> >> >                 goto feature_uinit;
> >> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >> >
> >> >  feature_uinit:
> >> >         fpga_dev_feature_uinit(pdev);
> >> > +dev_destroy:
> >> > +       fme_dev_destroy(pdev);
> >> >  exit:
> >> >         return ret;
> >> >  }
> >> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >> >  {
> >> >         fpga_dev_feature_uinit(pdev);
> >> >         fpga_unregister_dev_ops(pdev);
> >> > +       fme_dev_destroy(pdev);
> >> >         return 0;
> >> >  }
> >> >
> >> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> >> > new file mode 100644
> >> > index 0000000..3b44a3e
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel/fme-pr.c
> >> > @@ -0,0 +1,400 @@
> >> > +/*
> >> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <[e-mail]>
> >> > + *   Xiao Guangrong <[e-mail]>
> >> > + *   Joseph Grecco <[e-mail]>
> >> > + *   Enno Luebbers <[e-mail]>
> >> > + *   Tim Whisonant <[e-mail]>
> >> > + *   Ananda Ravuri <[e-mail]>
> >> > + *   Christopher Rauer <[e-mail]>
> >> > + *   Henry Mitchel <[e-mail]>
> >> > + *
> >> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> >> > + * redistributing this file, you may do so under either license. See the
> >> > + * LICENSE.BSD file under this directory for the BSD license and see
> >> > + * the COPYING file in the top-level directory for the GPLv2 license.
> >> > + */
> >> > +
> >> > +#include <linux/types.h>
> >> > +#include <linux/device.h>
> >> > +#include <linux/vmalloc.h>
> >> > +#include <linux/uaccess.h>
> >> > +#include <linux/fpga/fpga-mgr.h>
> >> > +#include <linux/intel-fpga.h>
> >> > +
> >> > +#include "feature-dev.h"
> >> > +#include "fme.h"
> >> > +
> >> > +#define PR_WAIT_TIMEOUT   8000000
> >> > +
> >> > +#define PR_HOST_STATUS_IDLE    0
> >> > +
> >> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> >> > +
> >> > +static ssize_t interface_id_show(struct device *dev,
> >> > +                                struct device_attribute *attr, char *buf)
> >> > +{
> >> > +       u64 intfc_id_l, intfc_id_h;
> >> > +       struct feature_fme_pr *fme_pr
> >> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> >> > +
> >> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> >> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> >> > +
> >> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> >> > +                       (unsigned long long)intfc_id_h,
> >> > +                       (unsigned long long)intfc_id_l);
> >> > +}
> >> > +static DEVICE_ATTR_RO(interface_id);
> >> > +
> >> > +static struct attribute *pr_mgmt_attrs[] = {
> >> > +       &dev_attr_interface_id.attr,
> >> > +       NULL,
> >> > +};
> >> > +
> >> > +struct attribute_group pr_mgmt_attr_group = {
> >> > +       .attrs  = pr_mgmt_attrs,
> >> > +       .name   = "pr",
> >> > +};
> >> > +
> >> > +static u64
> >> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> >> > +{
> >> > +       struct feature_fme_pr_status fme_pr_status;
> >> > +       unsigned long err_code;
> >> > +       u64 fme_pr_error;
> >> > +       int i = 0;
> >> > +
> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
> >> > +       if (!fme_pr_status.pr_status)
> >> > +               return 0;
> >> > +
> >> > +       err_code = fme_pr_error = readq(&fme_pr->error);
> >> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> >> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> >> > +       writeq(fme_pr_error, &fme_pr->error);
> >> > +       return fme_pr_error;
> >> > +}
> >> > +
> >> > +static int fme_pr_write_init(struct fpga_manager *mgr,
> >> > +               struct fpga_image_info *info, const char *buf, size_t count)
> >> > +{
> >> > +       struct fpga_fme *fme = mgr->priv;
> >> > +       struct platform_device *pdev;
> >> > +       struct feature_fme_pr *fme_pr;
> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +       struct feature_fme_pr_status fme_pr_status;
> >> > +
> >> > +       pdev = fme->pdata->dev;
> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                               FME_FEATURE_ID_PR_MGMT);
> >> > +       if (!fme_pr)
> >> > +               return -EINVAL;
> >> > +
> >> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> >> > +               return -EINVAL;
> >> > +
> >> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_reset = 1;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       fme_pr_ctl.pr_reset_ack = 1;
> >> > +
> >> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_reset = 0;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       dev_dbg(&pdev->dev,
> >> > +               "waiting for PR resource in HW to be initialized and ready\n");
> >> > +
> >> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> >> > +
> >> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> >> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> >> > +       pr_err_handle(pdev, fme_pr);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int fme_pr_write(struct fpga_manager *mgr,
> >> > +                       const char *buf, size_t count)
> >> > +{
> >> > +       struct fpga_fme *fme = mgr->priv;
> >> > +       struct platform_device *pdev;
> >> > +       struct feature_fme_pr *fme_pr;
> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +       struct feature_fme_pr_status fme_pr_status;
> >> > +       struct feature_fme_pr_data fme_pr_data;
> >> > +       int delay, pr_credit, i = 0;
> >> > +
> >> > +       pdev = fme->pdata->dev;
> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                               FME_FEATURE_ID_PR_MGMT);
> >> > +
> >> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_regionid = fme->port_id;
> >> > +       fme_pr_ctl.pr_start_req = 1;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> >> > +
> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
> >> > +       pr_credit = fme_pr_status.pr_credit;
> >> > +
> >> > +       while (count > 0) {
> >> > +               delay = 0;
> >> > +               while (pr_credit <= 1) {
> >> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> >> > +                               dev_err(&pdev->dev, "maximum try\n");
> >> > +                               return -ETIMEDOUT;
> >> > +                       }
> >> > +                       udelay(1);
> >> > +
> >> > +                       fme_pr_status.csr = readq(&fme_pr->status);
> >> > +                       pr_credit = fme_pr_status.pr_credit;
> >> > +               };
> >> > +
> >> > +               if (count >= 4) {
> >> > +                       fme_pr_data.rsvd = 0;
> >> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> >> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
> >> > +                       count -= 4;
> >> > +                       pr_credit--;
> >> > +                       i++;
> >> > +               } else {
> >> > +                       WARN_ON(1);
> >> > +                       return -EINVAL;
> >> > +               }
> >> > +       }
> >> > +
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
> >> > +                       struct fpga_image_info *info)
> >> > +{
> >> > +       struct fpga_fme *fme = mgr->priv;
> >> > +       struct platform_device *pdev;
> >> > +       struct feature_fme_pr *fme_pr;
> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> >> > +
> >> > +       pdev = fme->pdata->dev;
> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                               FME_FEATURE_ID_PR_MGMT);
> >> > +
> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> >> > +       fme_pr_ctl.pr_push_complete = 1;
> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >> > +
> >> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> >> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> >> > +
> >> > +       fme_pr_ctl.pr_start_req = 0;
> >> > +
> >> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >> > +               dev_err(&pdev->dev, "maximum try.\n");
> >> > +               return -ETIMEDOUT;
> >> > +       }
> >> > +
> >> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> >> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> >> > +       if (fme->pr_err)
> >> > +               return -EIO;
> >> > +
> >> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> >> > +{
> >> > +       return FPGA_MGR_STATE_UNKNOWN;
> >> > +}
> >> > +
> >> > +static const struct fpga_manager_ops fme_pr_ops = {
> >> > +       .write_init = fme_pr_write_init,
> >> > +       .write = fme_pr_write,
> >> > +       .write_complete = fme_pr_write_complete,
> >> > +       .state = fme_pr_state,
> >> > +};
> >> > +
> >> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> >> > +{
> >> > +       void __user *argp = (void __user *)arg;
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *fme;
> >> > +       struct fpga_manager *mgr;
> >> > +       struct feature_fme_header *fme_hdr;
> >> > +       struct feature_fme_capability fme_capability;
> >> > +       struct fpga_image_info info;
> >> > +       struct fpga_fme_port_pr port_pr;
> >> > +       struct platform_device *port;
> >> > +       unsigned long minsz;
> >> > +       void *buf = NULL;
> >> > +       int ret = 0;
> >> > +
> >> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> >> > +
> >> > +       if (copy_from_user(&port_pr, argp, minsz))
> >> > +               return -EFAULT;
> >> > +
> >> > +       if (port_pr.argsz < minsz || port_pr.flags)
> >> > +               return -EINVAL;
> >> > +
> >> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> >> > +               return -EINVAL;
> >> > +
> >> > +       /* get fme header region */
> >> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> >> > +                                       FME_FEATURE_ID_HEADER);
> >> > +       if (WARN_ON(!fme_hdr))
> >> > +               return -EINVAL;
> >> > +
> >> > +       /* check port id */
> >> > +       fme_capability.csr = readq(&fme_hdr->capability);
> >> > +       if (port_pr.port_id >= fme_capability.num_ports) {
> >> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> >> > +               return -EINVAL;
> >> > +       }
> >> > +
> >> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> >> > +                                   port_pr.buffer_size))
> >> > +               return -EFAULT;
> >> > +
> >> > +       buf = vmalloc(port_pr.buffer_size);
> >> > +       if (!buf)
> >> > +               return -ENOMEM;
> >> > +
> >> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> >> > +                                              port_pr.buffer_size)) {
> >> > +               ret = -EFAULT;
> >> > +               goto free_exit;
> >> > +       }
> >> > +
> >> > +       memset(&info, 0, sizeof(struct fpga_image_info));
> >> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> >> > +
> >> > +       mgr = fpga_mgr_get(&pdev->dev);
> >> > +       if (IS_ERR(mgr)) {
> >> > +               ret = PTR_ERR(mgr);
> >> > +               goto free_exit;
> >> > +       }
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       fme = fpga_pdata_get_private(pdata);
> >> > +       /* fme device has been unregistered. */
> >> > +       if (!fme) {
> >> > +               ret = -EINVAL;
> >> > +               goto unlock_exit;
> >> > +       }
> >> > +
> >> > +       fme->pr_err = 0;
> >> > +       fme->port_id = port_pr.port_id;
> >>
> >> It looks like you're using private data to communicate with the
> >> driver, i.e. there is something you want to do with the fpga manager
> >> framework and it doesn't have that feature.  The better way would be
> >> for us to expand the framework so you don't need to do that.
> >>
> >> port_id is the kind of thing that should be communicated to the driver
> >> through fpga_image_info, so we could add that to the struct.  Should
> >> we call it port_id?
> 
> Let's call it region_id.
> 
> >> Or is there something more generic that may be
> >> useful in the future for other architectures?.
> >
> > Hi Alan
> >
> > Thanks for your feedback. :)
> >
> > As you know, each Intel FPGA device may have more than one accelerator,
> > and all accelerators share the same fpga_manager (only one FME module).
> > port_id = 0 means the first accelerator on this fpga devices. So it's
> > needed by the fpga_manager to distinguish one accelerator from others
> > for partial reconfiguration.
> >
> > Adding a member like a 'id' to fpga_image_info definitely works for us
> > in this case. We can add it this way, but I feel 'id' here seems not
> > related to fpga image, but characteristic of fpga region.
> 
> The  fpga_image_info struct started life as just image specific info,
> but I want it to go in the direction of including parameters needed to
> program it this specific time. Otherwise we are stuck having to keep
> adding parameters as our use of FPGA develops.  It probably could be
> documented better as 'information needed to program a FPGA image'
> rather than strictly 'information about this particular FPGA image'.
> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> goes in this direction by having the buf, firmware name, or sg list
> passed in the info for the added fpga_mgr_load() function.  Actually I
> should probably simplify the API and get rid of fpga_mgr_buf_load,
> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> use fpga_mgr_load (passing all parameters in fpga_image_info).
>

Make sense.
 
> > It may be a
> > little confusing. One rough idea is that keep this info under fpga region
> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> 
> Yes, keep this info in fpga-region.  When the region wants to program
> using fpga-mgr, add the region id to fpga_image_info.  I propose
> calling it region_id.

Hm.. Do we need a function which moves info from region to image info?

Another idea is, add a priv to fpga_image_info, and use a common function
to pass the fpga_region's priv to fpga_image_info's priv before PR.
fpga-mgr then knows fpga_region priv info from the fpga_image_info.

> 
> > then fpga_manager knows the target region for partial reconfiguration.
> > If consider pr sysfs interface support under fpga-region in the future,
> > then we don't need to create a new 'id' sysfs file, as fpga-region itself
> > could provide this kind of info. But I'm not sure if this is the right
> > direction.
> >
> >>
> >> pr_err appears to be machine specific error codes that are
> >> communicated outside your low level driver.  (Calling it pr_err is
> >> extra confusing since Linux already has a commonly name function by
> >> the same name).  The framework has state, but that's not doing what
> >> you want here.  Maybe we could add a framework ops called status so
> >> that status could be communicated from the low level driver. It would
> >> be useful to abstract the machine specific state to a defined enum
> >> that would be part of the fpga mgr framework.  I remember we had
> >> something like that in the earliest version of fpga manager but it got
> >> changed to state rather than status for some reason.
> >>
> >
> > Adding a status function and a 'status' member to fpga manager sounds good.
> > But I'm not sure how to abstract these error codes, currently what we have
> > are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
> > "PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
> > fine to let each driver to define how to use that 'status' for machine
> > specific status?
> 
> I looked at the list of errors in include/uapi/linux/intel-fpga.h.
> They all seem pretty generic to me except I am not clear what "secure
> load error" or "IP protocol error" mean and whether other
> architectures would have them.  But certainly things like crc error,
> incompatible bitstream, fifo overflow are generic.  So let's see if we
> can define all these in a way that's generic and just pass up a error
> number.  That way upper layers can know how to deal with them
> possibly.  I would take the word "PR" off these since the error set
> applies whether someone is doing full reconfig or partial reconfig.

Sure, good to me, we can make it this way and see.
Thanks for the suggestions. :)

Hao

> 
> >
> >> mgr->dev won't work for you for some of the things you are using fme->pdev for?
> >>
> >
> > Does 'fme->pdev' mean the platform device for fme? I think we use platform
> > device to represent the FME module, and FME module may have multiple sub
> > features, driver discovers these sub features via the 'Device Feature List'
> > in the PCIe Device Bar. PR is only one of the sub features under FME module,
> > if any FME module doesn't have PR sub feature, fpga-manager will not be
> > created at all. But this will not impact other sub features, e.g thermal
> > management, error reporting and etc, to create their own interfaces under
> > the platform device.
> 
> If we need it for private data, that's may be actually OK.   Just
> wondering whether the priv was needed.   I think that's the one
> element in the private data struct that was used privately instead of
> being used to pass info outside the framework.
> 
> >
> > Thanks
> > Hao
> >
> >> Alan
> >>
> >> > +
> >> > +       /* Find and get port device by index */
> >> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> >> > +                                        fpga_port_check_id);
> >> > +       WARN_ON(!port);
> >> > +
> >> > +       /* Disable Port before PR */
> >> > +       fpga_port_disable(port);
> >> > +
> >> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> >> > +       port_pr.status = fme->pr_err;
> >> > +
> >> > +       /* Re-enable Port after PR finished */
> >> > +       fpga_port_enable(port);
> >> > +
> >> > +       put_device(&port->dev);
> >> > +
> >> > +unlock_exit:
> >> > +       mutex_unlock(&pdata->lock);
> >> > +       fpga_mgr_put(mgr);
> >> > +free_exit:
> >> > +       vfree(buf);
> >> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> >> > +               return -EFAULT;
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
> >> > +{
> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >> > +       struct fpga_fme *priv;
> >> > +       int ret;
> >> > +
> >> > +       mutex_lock(&pdata->lock);
> >> > +       priv = fpga_pdata_get_private(pdata);
> >> > +       ret = fpga_mgr_register(&pdata->dev->dev,
> >> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
> >> > +       mutex_unlock(&pdata->lock);
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
> >> > +{
> >> > +       fpga_mgr_unregister(&pdev->dev);
> >> > +       return 0;
> >> > +}
> >> > +
> >> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> >> > +{
> >> > +       int ret;
> >> > +
> >> > +       ret = fpga_fme_pr_probe(pdev);
> >> > +       if (ret)
> >> > +               return ret;
> >> > +
> >> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> >> > +       if (ret)
> >> > +               fpga_fme_pr_remove(pdev);
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> >> > +{
> >> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> >> > +       fpga_fme_pr_remove(pdev);
> >> > +}
> >> > +
> >> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> >> > +       unsigned int cmd, unsigned long arg)
> >> > +{
> >> > +       long ret;
> >> > +
> >> > +       switch (cmd) {
> >> > +       case FPGA_FME_PORT_PR:
> >> > +               ret = fme_pr(pdev, arg);
> >> > +               break;
> >> > +       default:
> >> > +               ret = -ENODEV;
> >> > +       }
> >> > +
> >> > +       return ret;
> >> > +}
> >> > +
> >> > +struct feature_ops pr_mgmt_ops = {
> >> > +       .init = pr_mgmt_init,
> >> > +       .uinit = pr_mgmt_uinit,
> >> > +       .ioctl = fme_pr_ioctl,
> >> > +};
> >> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> >> > new file mode 100644
> >> > index 0000000..d6cb7ce
> >> > --- /dev/null
> >> > +++ b/drivers/fpga/intel/fme.h
> >> > @@ -0,0 +1,32 @@
> >> > +/*
> >> > + * Header file for Intel FPGA Management Engine (FME) Driver
> >> > + *
> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
> >> > + *
> >> > + * Authors:
> >> > + *   Kang Luwei <[e-mail]>
> >> > + *   Xiao Guangrong <[e-mail]>
> >> > + *   Joseph Grecco <[e-mail]>
> >> > + *   Enno Luebbers <[e-mail]>
> >> > + *   Tim Whisonant <[e-mail]>
> >> > + *   Ananda Ravuri <[e-mail]>
> >> > + *   Henry Mitchel <[e-mail]>
> >> > + *
> >> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> >> > + * redistributing this file, you may do so under either license. See the
> >> > + * LICENSE.BSD file under this directory for the BSD license and see
> >> > + * the COPYING file in the top-level directory for the GPLv2 license.
> >> > + */
> >> > +
> >> > +#ifndef __INTEL_FME_H
> >> > +#define __INTEL_FME_H
> >> > +
> >> > +struct fpga_fme {
> >> > +       u8  port_id;
> >> > +       u64 pr_err;
> >> > +       struct feature_platform_data *pdata;
> >> > +};
> >> > +
> >> > +extern struct feature_ops pr_mgmt_ops;
> >> > +
> >> > +#endif
> >> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> >> > index 992e556..77658316 100644
> >> > --- a/include/uapi/linux/intel-fpga.h
> >> > +++ b/include/uapi/linux/intel-fpga.h
> >> > @@ -18,6 +18,8 @@
> >> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
> >> >  #define _UAPI_LINUX_INTEL_FPGA_H
> >> >
> >> > +#include <linux/types.h>
> >> > +
> >> >  #define FPGA_API_VERSION 0
> >> >
> >> >  /*
> >> > @@ -30,6 +32,7 @@
> >> >  #define FPGA_MAGIC 0xB6
> >> >
> >> >  #define FPGA_BASE 0
> >> > +#define FME_BASE 0x80
> >> >
> >> >  /**
> >> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> >> > @@ -49,4 +52,45 @@
> >> >
> >> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >> >
> >> > +/* IOCTLs for FME file descriptor */
> >> > +
> >> > +/**
> >> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> >> > + *
> >> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> >> > + * provided by caller.
> >> > + * Return: 0 on success, -errno on failure.
> >> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> >> > + * some errors during PR, under this case, the user can fetch HW error code
> >> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> >> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> >> > + * Otherwise, it is always zero.
> >> > + */
> >> > +
> >> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> >> > +static const char * const _name_[] = {                 \
> >> > +       "PR operation error detected",                  \
> >> > +       "PR CRC error detected",                        \
> >> > +       "PR incompatiable bitstream error detected",    \
> >> > +       "PR IP protocol error detected",                \
> >> > +       "PR FIFO overflow error detected",              \
> >> > +       "Reserved",                                     \
> >> > +       "PR secure load error detected",                \
> >> > +}
> >> > +
> >> > +#define PR_MAX_ERR_NUM 7
> >> > +
> >> > +struct fpga_fme_port_pr {
> >> > +       /* Input */
> >> > +       __u32 argsz;            /* Structure length */
> >> > +       __u32 flags;            /* Zero for now */
> >> > +       __u32 port_id;
> >> > +       __u32 buffer_size;
> >> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> >> > +       /* Output */
> >> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
> >> > +};
> >> > +
> >> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> >> > +
> >> >  #endif /* _UAPI_INTEL_FPGA_H */
> >> > --
> >> > 2.7.4
> >> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 4, 2017, 6:28 a.m. UTC | #14

On Mon, Apr 03, 2017 at 04:24:13PM -0500, Alan Tull wrote:
> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> > From: Kang Luwei <[e-mail]>
> >
> > Partial Reconfiguration (PR) is the most important function for FME. It
> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >
> > This patch adds support for PR sub feature. In this patch, it registers
> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> > for PR operation once PR request received via ioctl.
> 
> The code that invokes fpga_mgr_buf_load should be a different layer.

Please see my feedback below.

> 
> > Below user space
> > interfaces are exposed by this sub feature.
> >
> > Sysfs interface:
> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >   Read-only. Indicate the hardware interface information. Userspace
> >   applications need to check this interface to select correct green
> >   bitstream format before PR.
> >
> > Ioctl interface:
> > * FPGA_FME_PORT_PR
> >   Do partial reconfiguration per information from userspace, including
> >   target port(AFU), buffer size and address info. It returns the PR status
> >   (PR error code if failed) to userspace.
> >
> > Signed-off-by: Tim Whisonant <[e-mail]>
> > Signed-off-by: Enno Luebbers <[e-mail]>
> > Signed-off-by: Shiva Rao <[e-mail]>
> > Signed-off-by: Christopher Rauer <[e-mail]>
> > Signed-off-by: Alan Tull <[e-mail]>
> > Signed-off-by: Kang Luwei <[e-mail]>
> > Signed-off-by: Xiao Guangrong <[e-mail]>
> > Signed-off-by: Wu Hao <[e-mail]>
> > ---
> >  drivers/fpga/intel/Makefile      |   2 +-
> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >  drivers/fpga/intel/fme.h         |  32 ++++
> >  include/uapi/linux/intel-fpga.h  |  44 +++++
> >  6 files changed, 578 insertions(+), 2 deletions(-)
> >  create mode 100644 drivers/fpga/intel/fme-pr.c
> >  create mode 100644 drivers/fpga/intel/fme.h
> >
> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> > index 546861d..0452cb6 100644
> > --- a/drivers/fpga/intel/Makefile
> > +++ b/drivers/fpga/intel/Makefile
> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >
> >  intel-fpga-pci-objs := pcie.o feature-dev.o
> > -intel-fpga-fme-objs := fme-main.o
> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> > index dccc283..5a25c915 100644
> > --- a/drivers/fpga/intel/feature-dev.h
> > +++ b/drivers/fpga/intel/feature-dev.h
> > @@ -150,8 +150,66 @@ struct feature_fme_err {
> >  };
> >
> >  /* FME Partial Reconfiguration Sub Feature Register Set */
> > +/* FME PR Control Register */
> > +struct feature_fme_pr_ctl {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u8  pr_reset:1;         /* Reset PR Engine */
> > +                       u8  rsvdz1:3;
> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_regionid:2;      /* PR Region ID */
> > +                       u8  rsvdz3:2;
> > +                       u8  pr_start_req:1;     /* PR Start Request */
> > +                       u8  pr_push_complete:1; /* PR Data push complete */
> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> > +                       u32 rsvdz4:17;
> > +                       u32 config_data;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Status Register */
> > +struct feature_fme_pr_status {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       u16 pr_credit:9;        /* Number of PR Credits */
> > +                       u8  rsvdz1:7;
> > +                       u8  pr_status:1;        /* PR Operation status */
> > +                       u8  rsvdz2:3;
> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
> > +                       u8  rsvdz3:1;
> > +                       u8  pr_host_status:4;   /* PR Host status */
> > +                       u64 rsvdz4:36;
> > +               };
> > +       };
> > +};
> > +
> > +/* FME PR Data Register */
> > +struct feature_fme_pr_data {
> > +       union {
> > +               u64 csr;
> > +               struct {
> > +                       /* PR data from the raw-binary file */
> > +                       u32 pr_data_raw;
> > +                       u32 rsvd;
> > +               };
> > +       };
> > +};
> > +
> >  struct feature_fme_pr {
> >         struct feature_header header;
> > +       struct feature_fme_pr_ctl control;
> > +       struct feature_fme_pr_status status;
> > +       struct feature_fme_pr_data data;
> > +       u64 error;
> > +
> > +       u64 rsvd[16];
> > +
> > +       u64 intfc_id_l;         /* PR interface Id Low */
> > +       u64 intfc_id_h;         /* PR interface Id High */
> >  };
> >
> >  /* PORT Header Register Set */
> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> > index 36d0c4c..0d9a7a6 100644
> > --- a/drivers/fpga/intel/fme-main.c
> > +++ b/drivers/fpga/intel/fme-main.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/intel-fpga.h>
> >
> >  #include "feature-dev.h"
> > +#include "fme.h"
> >
> >  static ssize_t ports_num_show(struct device *dev,
> >                               struct device_attribute *attr, char *buf)
> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >                 .ops = &fme_hdr_ops,
> >         },
> >         {
> > +               .name = FME_FEATURE_PR_MGMT,
> > +               .ops = &pr_mgmt_ops,
> > +       },
> > +       {
> >                 .ops = NULL,
> >         },
> >  };
> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >         .unlocked_ioctl = fme_ioctl,
> >  };
> >
> > +static int fme_dev_init(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> > +       if (!fme)
> > +               return -ENOMEM;
> > +
> > +       fme->pdata = pdata;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fpga_pdata_set_private(pdata, fme);
> > +       mutex_unlock(&pdata->lock);
> > +       return 0;
> > +}
> > +
> > +static void fme_dev_destroy(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       fpga_pdata_set_private(pdata, NULL);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       devm_kfree(&pdev->dev, fme);
> > +}
> > +
> >  static int fme_probe(struct platform_device *pdev)
> >  {
> >         int ret;
> >
> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       ret = fme_dev_init(pdev);
> >         if (ret)
> >                 goto exit;
> >
> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> > +       if (ret)
> > +               goto dev_destroy;
> > +
> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >         if (ret)
> >                 goto feature_uinit;
> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >
> >  feature_uinit:
> >         fpga_dev_feature_uinit(pdev);
> > +dev_destroy:
> > +       fme_dev_destroy(pdev);
> >  exit:
> >         return ret;
> >  }
> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >  {
> >         fpga_dev_feature_uinit(pdev);
> >         fpga_unregister_dev_ops(pdev);
> > +       fme_dev_destroy(pdev);
> >         return 0;
> >  }
> >
> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> > new file mode 100644
> > index 0000000..3b44a3e
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme-pr.c
> > @@ -0,0 +1,400 @@
> > +/*
> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <[e-mail]>
> > + *   Xiao Guangrong <[e-mail]>
> > + *   Joseph Grecco <[e-mail]>
> > + *   Enno Luebbers <[e-mail]>
> > + *   Tim Whisonant <[e-mail]>
> > + *   Ananda Ravuri <[e-mail]>
> > + *   Christopher Rauer <[e-mail]>
> > + *   Henry Mitchel <[e-mail]>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#include <linux/types.h>
> > +#include <linux/device.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/uaccess.h>
> > +#include <linux/fpga/fpga-mgr.h>
> > +#include <linux/intel-fpga.h>
> > +
> > +#include "feature-dev.h"
> > +#include "fme.h"
> > +
> > +#define PR_WAIT_TIMEOUT   8000000
> > +
> > +#define PR_HOST_STATUS_IDLE    0
> > +
> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> > +
> > +static ssize_t interface_id_show(struct device *dev,
> > +                                struct device_attribute *attr, char *buf)
> > +{
> > +       u64 intfc_id_l, intfc_id_h;
> > +       struct feature_fme_pr *fme_pr
> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> > +
> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
> > +
> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> > +                       (unsigned long long)intfc_id_h,
> > +                       (unsigned long long)intfc_id_l);
> > +}
> > +static DEVICE_ATTR_RO(interface_id);
> > +
> > +static struct attribute *pr_mgmt_attrs[] = {
> > +       &dev_attr_interface_id.attr,
> > +       NULL,
> > +};
> > +
> > +struct attribute_group pr_mgmt_attr_group = {
> > +       .attrs  = pr_mgmt_attrs,
> > +       .name   = "pr",
> > +};
> > +
> > +static u64
> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> > +{
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       unsigned long err_code;
> > +       u64 fme_pr_error;
> > +       int i = 0;
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       if (!fme_pr_status.pr_status)
> > +               return 0;
> > +
> > +       err_code = fme_pr_error = readq(&fme_pr->error);
> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> > +       writeq(fme_pr_error, &fme_pr->error);
> > +       return fme_pr_error;
> > +}
> > +
> > +static int fme_pr_write_init(struct fpga_manager *mgr,
> > +               struct fpga_image_info *info, const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +       if (!fme_pr)
> > +               return -EINVAL;
> > +
> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> > +               return -EINVAL;
> 
> flags is bitmapped so please do:
> 
> if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))

Thanks for the review.

Will fix this in the next version.

> 
> > +
> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       fme_pr_ctl.pr_reset_ack = 1;
> > +
> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_reset = 0;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev,
> > +               "waiting for PR resource in HW to be initialized and ready\n");
> > +
> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> > +
> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> > +       pr_err_handle(pdev, fme_pr);
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write(struct fpga_manager *mgr,
> > +                       const char *buf, size_t count)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +       struct feature_fme_pr_status fme_pr_status;
> > +       struct feature_fme_pr_data fme_pr_data;
> > +       int delay, pr_credit, i = 0;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_regionid = fme->port_id;
> > +       fme_pr_ctl.pr_start_req = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> > +
> > +       fme_pr_status.csr = readq(&fme_pr->status);
> > +       pr_credit = fme_pr_status.pr_credit;
> > +
> > +       while (count > 0) {
> > +               delay = 0;
> > +               while (pr_credit <= 1) {
> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
> > +                               dev_err(&pdev->dev, "maximum try\n");
> > +                               return -ETIMEDOUT;
> > +                       }
> > +                       udelay(1);
> > +
> > +                       fme_pr_status.csr = readq(&fme_pr->status);
> > +                       pr_credit = fme_pr_status.pr_credit;
> > +               };
> > +
> > +               if (count >= 4) {
> > +                       fme_pr_data.rsvd = 0;
> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
> > +                       count -= 4;
> > +                       pr_credit--;
> > +                       i++;
> > +               } else {
> > +                       WARN_ON(1);
> > +                       return -EINVAL;
> > +               }
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
> > +                       struct fpga_image_info *info)
> > +{
> > +       struct fpga_fme *fme = mgr->priv;
> > +       struct platform_device *pdev;
> > +       struct feature_fme_pr *fme_pr;
> > +       struct feature_fme_pr_ctl fme_pr_ctl;
> > +
> > +       pdev = fme->pdata->dev;
> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                               FME_FEATURE_ID_PR_MGMT);
> > +
> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
> > +       fme_pr_ctl.pr_push_complete = 1;
> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
> > +
> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> > +
> > +       fme_pr_ctl.pr_start_req = 0;
> > +
> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> > +               dev_err(&pdev->dev, "maximum try.\n");
> > +               return -ETIMEDOUT;
> > +       }
> > +
> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
> > +       if (fme->pr_err)
> > +               return -EIO;
> > +
> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
> > +       return 0;
> > +}
> > +
> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> > +{
> > +       return FPGA_MGR_STATE_UNKNOWN;
> > +}
> > +
> > +static const struct fpga_manager_ops fme_pr_ops = {
> > +       .write_init = fme_pr_write_init,
> > +       .write = fme_pr_write,
> > +       .write_complete = fme_pr_write_complete,
> > +       .state = fme_pr_state,
> > +};
> > +
> 
> The following fme_pr() function shouldn't be in a fpga-mgr driver.  It
> is calling
> the framework that this driver is registered with.
> A fpga-mgr low level driver is supposed to be a very low level driver.
> This function is controlling the port and calling fpga-mgr layer to do the
> programming.  There should be a layer above the fpga-mgr and fpga-bridge
> that coordinates these things.  I.e. can you use my proposed fpga-region code?
> 

Hm.. as mentioned in the intel-fpga.txt documentation 'open discussion'.
Current FME doesn't use fpga-region as related patches are not merged.
But we will make the related changes to let FME create all fpga-regions,
bridges and managers for PR function on given Intel FPGA device per
suggestion.

Please notice below fme_pr is still needed to provide ioctl interface to
enduser for PR, it should invoke fpga_region_program_fpga or equivalent
for PR instead of fpga_mgr_buf_load then.
And if in the future, the sysfs for PR is ready, then fme_pr may not be
needed, as user can do PR via fpga-region sysfs interface.

Thanks
Hao

> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
> > +{
> > +       void __user *argp = (void __user *)arg;
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *fme;
> > +       struct fpga_manager *mgr;
> > +       struct feature_fme_header *fme_hdr;
> > +       struct feature_fme_capability fme_capability;
> > +       struct fpga_image_info info;
> > +       struct fpga_fme_port_pr port_pr;
> > +       struct platform_device *port;
> > +       unsigned long minsz;
> > +       void *buf = NULL;
> > +       int ret = 0;
> > +
> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
> > +
> > +       if (copy_from_user(&port_pr, argp, minsz))
> > +               return -EFAULT;
> > +
> > +       if (port_pr.argsz < minsz || port_pr.flags)
> > +               return -EINVAL;
> > +
> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
> > +               return -EINVAL;
> > +
> > +       /* get fme header region */
> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
> > +                                       FME_FEATURE_ID_HEADER);
> > +       if (WARN_ON(!fme_hdr))
> > +               return -EINVAL;
> > +
> > +       /* check port id */
> > +       fme_capability.csr = readq(&fme_hdr->capability);
> > +       if (port_pr.port_id >= fme_capability.num_ports) {
> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
> > +               return -EINVAL;
> > +       }
> > +
> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
> > +                                   port_pr.buffer_size))
> > +               return -EFAULT;
> > +
> > +       buf = vmalloc(port_pr.buffer_size);
> > +       if (!buf)
> > +               return -ENOMEM;
> > +
> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
> > +                                              port_pr.buffer_size)) {
> > +               ret = -EFAULT;
> > +               goto free_exit;
> > +       }
> > +
> > +       memset(&info, 0, sizeof(struct fpga_image_info));
> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
> > +
> > +       mgr = fpga_mgr_get(&pdev->dev);
> > +       if (IS_ERR(mgr)) {
> > +               ret = PTR_ERR(mgr);
> > +               goto free_exit;
> > +       }
> > +
> > +       mutex_lock(&pdata->lock);
> > +       fme = fpga_pdata_get_private(pdata);
> > +       /* fme device has been unregistered. */
> > +       if (!fme) {
> > +               ret = -EINVAL;
> > +               goto unlock_exit;
> > +       }
> > +
> > +       fme->pr_err = 0;
> > +       fme->port_id = port_pr.port_id;
> > +
> > +       /* Find and get port device by index */
> > +       port = pdata->fpga_for_each_port(pdev, &fme->port_id,
> > +                                        fpga_port_check_id);
> > +       WARN_ON(!port);
> > +
> > +       /* Disable Port before PR */
> > +       fpga_port_disable(port);
> > +
> > +       ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
> > +       port_pr.status = fme->pr_err;
> > +
> > +       /* Re-enable Port after PR finished */
> > +       fpga_port_enable(port);
> > +
> > +       put_device(&port->dev);
> > +
> > +unlock_exit:
> > +       mutex_unlock(&pdata->lock);
> > +       fpga_mgr_put(mgr);
> > +free_exit:
> > +       vfree(buf);
> > +       if (copy_to_user((void __user *)arg, &port_pr, minsz))
> > +               return -EFAULT;
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_probe(struct platform_device *pdev)
> > +{
> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> > +       struct fpga_fme *priv;
> > +       int ret;
> > +
> > +       mutex_lock(&pdata->lock);
> > +       priv = fpga_pdata_get_private(pdata);
> > +       ret = fpga_mgr_register(&pdata->dev->dev,
> > +               "Intel FPGA Manager", &fme_pr_ops, priv);
> > +       mutex_unlock(&pdata->lock);
> > +
> > +       return ret;
> > +}
> > +
> > +static int fpga_fme_pr_remove(struct platform_device *pdev)
> > +{
> > +       fpga_mgr_unregister(&pdev->dev);
> > +       return 0;
> > +}
> > +
> > +static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       int ret;
> > +
> > +       ret = fpga_fme_pr_probe(pdev);
> > +       if (ret)
> > +               return ret;
> > +
> > +       ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       if (ret)
> > +               fpga_fme_pr_remove(pdev);
> > +
> > +       return ret;
> > +}
> > +
> > +static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
> > +{
> > +       sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
> > +       fpga_fme_pr_remove(pdev);
> > +}
> > +
> > +static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
> > +       unsigned int cmd, unsigned long arg)
> > +{
> > +       long ret;
> > +
> > +       switch (cmd) {
> > +       case FPGA_FME_PORT_PR:
> > +               ret = fme_pr(pdev, arg);
> > +               break;
> > +       default:
> > +               ret = -ENODEV;
> > +       }
> > +
> > +       return ret;
> > +}
> > +
> > +struct feature_ops pr_mgmt_ops = {
> > +       .init = pr_mgmt_init,
> > +       .uinit = pr_mgmt_uinit,
> > +       .ioctl = fme_pr_ioctl,
> > +};
> > diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
> > new file mode 100644
> > index 0000000..d6cb7ce
> > --- /dev/null
> > +++ b/drivers/fpga/intel/fme.h
> > @@ -0,0 +1,32 @@
> > +/*
> > + * Header file for Intel FPGA Management Engine (FME) Driver
> > + *
> > + * Copyright (C) 2017 Intel Corporation, Inc.
> > + *
> > + * Authors:
> > + *   Kang Luwei <[e-mail]>
> > + *   Xiao Guangrong <[e-mail]>
> > + *   Joseph Grecco <[e-mail]>
> > + *   Enno Luebbers <[e-mail]>
> > + *   Tim Whisonant <[e-mail]>
> > + *   Ananda Ravuri <[e-mail]>
> > + *   Henry Mitchel <[e-mail]>
> > + *
> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
> > + * redistributing this file, you may do so under either license. See the
> > + * LICENSE.BSD file under this directory for the BSD license and see
> > + * the COPYING file in the top-level directory for the GPLv2 license.
> > + */
> > +
> > +#ifndef __INTEL_FME_H
> > +#define __INTEL_FME_H
> > +
> > +struct fpga_fme {
> > +       u8  port_id;
> > +       u64 pr_err;
> > +       struct feature_platform_data *pdata;
> > +};
> > +
> > +extern struct feature_ops pr_mgmt_ops;
> > +
> > +#endif
> > diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
> > index 992e556..77658316 100644
> > --- a/include/uapi/linux/intel-fpga.h
> > +++ b/include/uapi/linux/intel-fpga.h
> > @@ -18,6 +18,8 @@
> >  #ifndef _UAPI_LINUX_INTEL_FPGA_H
> >  #define _UAPI_LINUX_INTEL_FPGA_H
> >
> > +#include <linux/types.h>
> > +
> >  #define FPGA_API_VERSION 0
> >
> >  /*
> > @@ -30,6 +32,7 @@
> >  #define FPGA_MAGIC 0xB6
> >
> >  #define FPGA_BASE 0
> > +#define FME_BASE 0x80
> >
> >  /**
> >   * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
> > @@ -49,4 +52,45 @@
> >
> >  #define FPGA_CHECK_EXTENSION   _IO(FPGA_MAGIC, FPGA_BASE + 1)
> >
> > +/* IOCTLs for FME file descriptor */
> > +
> > +/**
> > + * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
> > + *
> > + * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
> > + * provided by caller.
> > + * Return: 0 on success, -errno on failure.
> > + * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
> > + * some errors during PR, under this case, the user can fetch HW error code
> > + * from fpga_fme_port_pr.status. Each bit on the error code is used as the
> > + * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
> > + * Otherwise, it is always zero.
> > + */
> > +
> > +#define DEFINE_FPGA_PR_ERR_MSG(_name_)                 \
> > +static const char * const _name_[] = {                 \
> > +       "PR operation error detected",                  \
> > +       "PR CRC error detected",                        \
> > +       "PR incompatiable bitstream error detected",    \
> > +       "PR IP protocol error detected",                \
> > +       "PR FIFO overflow error detected",              \
> > +       "Reserved",                                     \
> > +       "PR secure load error detected",                \
> > +}
> > +
> > +#define PR_MAX_ERR_NUM 7
> > +
> > +struct fpga_fme_port_pr {
> > +       /* Input */
> > +       __u32 argsz;            /* Structure length */
> > +       __u32 flags;            /* Zero for now */
> > +       __u32 port_id;
> > +       __u32 buffer_size;
> > +       __u64 buffer_address;   /* Userspace address to the buffer for PR */
> > +       /* Output */
> > +       __u64 status;           /* HW error code if ioctl returns -EIO */
> > +};
> > +
> > +#define FPGA_FME_PORT_PR       _IO(FPGA_MAGIC, FME_BASE + 0)
> > +
> >  #endif /* _UAPI_INTEL_FPGA_H */
> > --
> > 2.7.4
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 4, 2017, 6:48 a.m. UTC | #15

On Mon, Apr 03, 2017 at 03:49:41PM -0700, [e-mail] wrote:
> 
> 
> On Mon, 3 Apr 2017, Alan Tull wrote:
> 
> >On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
> >>From: Kang Luwei <[e-mail]>
> >>
> >>Partial Reconfiguration (PR) is the most important function for FME. It
> >>allows reconfiguration for given Port/Accelerated Function Unit (AFU).
> >>
> >>This patch adds support for PR sub feature. In this patch, it registers
> >>a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
> >>for PR operation once PR request received via ioctl.
> >
> >The code that invokes fpga_mgr_buf_load should be a different layer.
> >
> >>Below user space
> >>interfaces are exposed by this sub feature.
> >>
> >>Sysfs interface:
> >>* /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
> >>  Read-only. Indicate the hardware interface information. Userspace
> >>  applications need to check this interface to select correct green
> >>  bitstream format before PR.
> >>
> >>Ioctl interface:
> >>* FPGA_FME_PORT_PR
> >>  Do partial reconfiguration per information from userspace, including
> >>  target port(AFU), buffer size and address info. It returns the PR status
> >>  (PR error code if failed) to userspace.
> >>
> >>Signed-off-by: Tim Whisonant <[e-mail]>
> >>Signed-off-by: Enno Luebbers <[e-mail]>
> >>Signed-off-by: Shiva Rao <[e-mail]>
> >>Signed-off-by: Christopher Rauer <[e-mail]>
> >>Signed-off-by: Alan Tull <[e-mail]>
> >>Signed-off-by: Kang Luwei <[e-mail]>
> >>Signed-off-by: Xiao Guangrong <[e-mail]>
> >>Signed-off-by: Wu Hao <[e-mail]>
> >>---
> >> drivers/fpga/intel/Makefile      |   2 +-
> >> drivers/fpga/intel/feature-dev.h |  58 ++++++
> >> drivers/fpga/intel/fme-main.c    |  44 ++++-
> >> drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
> >> drivers/fpga/intel/fme.h         |  32 ++++
> >> include/uapi/linux/intel-fpga.h  |  44 +++++
> >> 6 files changed, 578 insertions(+), 2 deletions(-)
> >> create mode 100644 drivers/fpga/intel/fme-pr.c
> >> create mode 100644 drivers/fpga/intel/fme.h
> >>
> >>diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
> >>index 546861d..0452cb6 100644
> >>--- a/drivers/fpga/intel/Makefile
> >>+++ b/drivers/fpga/intel/Makefile
> >>@@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
> >> obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
> >>
> >> intel-fpga-pci-objs := pcie.o feature-dev.o
> >>-intel-fpga-fme-objs := fme-main.o
> >>+intel-fpga-fme-objs := fme-main.o fme-pr.o
> >>diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
> >>index dccc283..5a25c915 100644
> >>--- a/drivers/fpga/intel/feature-dev.h
> >>+++ b/drivers/fpga/intel/feature-dev.h
> >>@@ -150,8 +150,66 @@ struct feature_fme_err {
> >> };
> >>
> >> /* FME Partial Reconfiguration Sub Feature Register Set */
> >>+/* FME PR Control Register */
> >>+struct feature_fme_pr_ctl {
> >>+       union {
> >>+               u64 csr;
> >>+               struct {
> >>+                       u8  pr_reset:1;         /* Reset PR Engine */
> >>+                       u8  rsvdz1:3;
> >>+                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
> >>+                       u8  rsvdz2:3;
> >>+                       u8  pr_regionid:2;      /* PR Region ID */
> >>+                       u8  rsvdz3:2;
> >>+                       u8  pr_start_req:1;     /* PR Start Request */
> >>+                       u8  pr_push_complete:1; /* PR Data push complete */
> >>+                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
> >>+                       u32 rsvdz4:17;
> >>+                       u32 config_data;
> >>+               };
> >>+       };
> >>+};
> >>+
> >>+/* FME PR Status Register */
> >>+struct feature_fme_pr_status {
> >>+       union {
> >>+               u64 csr;
> >>+               struct {
> >>+                       u16 pr_credit:9;        /* Number of PR Credits */
> >>+                       u8  rsvdz1:7;
> >>+                       u8  pr_status:1;        /* PR Operation status */
> >>+                       u8  rsvdz2:3;
> >>+                       u8  pr_ctrlr_status:3;  /* Controller status */
> >>+                       u8  rsvdz3:1;
> >>+                       u8  pr_host_status:4;   /* PR Host status */
> >>+                       u64 rsvdz4:36;
> >>+               };
> >>+       };
> >>+};
> >>+
> >>+/* FME PR Data Register */
> >>+struct feature_fme_pr_data {
> >>+       union {
> >>+               u64 csr;
> >>+               struct {
> >>+                       /* PR data from the raw-binary file */
> >>+                       u32 pr_data_raw;
> >>+                       u32 rsvd;
> >>+               };
> >>+       };
> >>+};
> >>+
> >> struct feature_fme_pr {
> >>        struct feature_header header;
> >>+       struct feature_fme_pr_ctl control;
> >>+       struct feature_fme_pr_status status;
> >>+       struct feature_fme_pr_data data;
> >>+       u64 error;
> >>+
> >>+       u64 rsvd[16];
> >>+
> >>+       u64 intfc_id_l;         /* PR interface Id Low */
> >>+       u64 intfc_id_h;         /* PR interface Id High */
> >> };
> >>
> >> /* PORT Header Register Set */
> >>diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
> >>index 36d0c4c..0d9a7a6 100644
> >>--- a/drivers/fpga/intel/fme-main.c
> >>+++ b/drivers/fpga/intel/fme-main.c
> >>@@ -23,6 +23,7 @@
> >> #include <linux/intel-fpga.h>
> >>
> >> #include "feature-dev.h"
> >>+#include "fme.h"
> >>
> >> static ssize_t ports_num_show(struct device *dev,
> >>                              struct device_attribute *attr, char *buf)
> >>@@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
> >>                .ops = &fme_hdr_ops,
> >>        },
> >>        {
> >>+               .name = FME_FEATURE_PR_MGMT,
> >>+               .ops = &pr_mgmt_ops,
> >>+       },
> >>+       {
> >>                .ops = NULL,
> >>        },
> >> };
> >>@@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
> >>        .unlocked_ioctl = fme_ioctl,
> >> };
> >>
> >>+static int fme_dev_init(struct platform_device *pdev)
> >>+{
> >>+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >>+       struct fpga_fme *fme;
> >>+
> >>+       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
> >>+       if (!fme)
> >>+               return -ENOMEM;
> >>+
> >>+       fme->pdata = pdata;
> >>+
> >>+       mutex_lock(&pdata->lock);
> >>+       fpga_pdata_set_private(pdata, fme);
> >>+       mutex_unlock(&pdata->lock);
> >>+       return 0;
> >>+}
> >>+
> >>+static void fme_dev_destroy(struct platform_device *pdev)
> >>+{
> >>+       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
> >>+       struct fpga_fme *fme;
> >>+
> >>+       mutex_lock(&pdata->lock);
> >>+       fme = fpga_pdata_get_private(pdata);
> >>+       fpga_pdata_set_private(pdata, NULL);
> >>+       mutex_unlock(&pdata->lock);
> >>+
> >>+       devm_kfree(&pdev->dev, fme);
> >>+}
> >>+
> >> static int fme_probe(struct platform_device *pdev)
> >> {
> >>        int ret;
> >>
> >>-       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >>+       ret = fme_dev_init(pdev);
> >>        if (ret)
> >>                goto exit;
> >>
> >>+       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
> >>+       if (ret)
> >>+               goto dev_destroy;
> >>+
> >>        ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
> >>        if (ret)
> >>                goto feature_uinit;
> >>@@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
> >>
> >> feature_uinit:
> >>        fpga_dev_feature_uinit(pdev);
> >>+dev_destroy:
> >>+       fme_dev_destroy(pdev);
> >> exit:
> >>        return ret;
> >> }
> >>@@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
> >> {
> >>        fpga_dev_feature_uinit(pdev);
> >>        fpga_unregister_dev_ops(pdev);
> >>+       fme_dev_destroy(pdev);
> >>        return 0;
> >> }
> >>
> >>diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
> >>new file mode 100644
> >>index 0000000..3b44a3e
> >>--- /dev/null
> >>+++ b/drivers/fpga/intel/fme-pr.c
> >>@@ -0,0 +1,400 @@
> >>+/*
> >>+ * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
> >>+ *
> >>+ * Copyright (C) 2017 Intel Corporation, Inc.
> >>+ *
> >>+ * Authors:
> >>+ *   Kang Luwei <[e-mail]>
> >>+ *   Xiao Guangrong <[e-mail]>
> >>+ *   Joseph Grecco <[e-mail]>
> >>+ *   Enno Luebbers <[e-mail]>
> >>+ *   Tim Whisonant <[e-mail]>
> >>+ *   Ananda Ravuri <[e-mail]>
> >>+ *   Christopher Rauer <[e-mail]>
> >>+ *   Henry Mitchel <[e-mail]>
> >>+ *
> >>+ * This work is licensed under a dual BSD/GPLv2 license. When using or
> >>+ * redistributing this file, you may do so under either license. See the
> >>+ * LICENSE.BSD file under this directory for the BSD license and see
> >>+ * the COPYING file in the top-level directory for the GPLv2 license.
> >>+ */
> >>+
> >>+#include <linux/types.h>
> >>+#include <linux/device.h>
> >>+#include <linux/vmalloc.h>
> >>+#include <linux/uaccess.h>
> >>+#include <linux/fpga/fpga-mgr.h>
> >>+#include <linux/intel-fpga.h>
> >>+
> >>+#include "feature-dev.h"
> >>+#include "fme.h"
> >>+
> >>+#define PR_WAIT_TIMEOUT   8000000
> >>+
> >>+#define PR_HOST_STATUS_IDLE    0
> >>+
> >>+DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
> >>+
> >>+static ssize_t interface_id_show(struct device *dev,
> >>+                                struct device_attribute *attr, char *buf)
> >>+{
> >>+       u64 intfc_id_l, intfc_id_h;
> >>+       struct feature_fme_pr *fme_pr
> >>+               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
> >>+
> >>+       intfc_id_l = readq(&fme_pr->intfc_id_l);
> >>+       intfc_id_h = readq(&fme_pr->intfc_id_h);
> >>+
> >>+       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
> >>+                       (unsigned long long)intfc_id_h,
> >>+                       (unsigned long long)intfc_id_l);
> >>+}
> >>+static DEVICE_ATTR_RO(interface_id);
> >>+
> >>+static struct attribute *pr_mgmt_attrs[] = {
> >>+       &dev_attr_interface_id.attr,
> >>+       NULL,
> >>+};
> >>+
> >>+struct attribute_group pr_mgmt_attr_group = {
> >>+       .attrs  = pr_mgmt_attrs,
> >>+       .name   = "pr",
> >>+};
> >>+
> >>+static u64
> >>+pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
> >>+{
> >>+       struct feature_fme_pr_status fme_pr_status;
> >>+       unsigned long err_code;
> >>+       u64 fme_pr_error;
> >>+       int i = 0;
> >>+
> >>+       fme_pr_status.csr = readq(&fme_pr->status);
> >>+       if (!fme_pr_status.pr_status)
> >>+               return 0;
> >>+
> >>+       err_code = fme_pr_error = readq(&fme_pr->error);
> >>+       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
> >>+               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
> >>+       writeq(fme_pr_error, &fme_pr->error);
> >>+       return fme_pr_error;
> >>+}
> >>+
> >>+static int fme_pr_write_init(struct fpga_manager *mgr,
> >>+               struct fpga_image_info *info, const char *buf, size_t count)
> >>+{
> >>+       struct fpga_fme *fme = mgr->priv;
> >>+       struct platform_device *pdev;
> >>+       struct feature_fme_pr *fme_pr;
> >>+       struct feature_fme_pr_ctl fme_pr_ctl;
> >>+       struct feature_fme_pr_status fme_pr_status;
> >>+
> >>+       pdev = fme->pdata->dev;
> >>+       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >>+                               FME_FEATURE_ID_PR_MGMT);
> >>+       if (!fme_pr)
> >>+               return -EINVAL;
> >>+
> >>+       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
> >>+               return -EINVAL;
> >
> >flags is bitmapped so please do:
> >
> >if (WARN_ON(info->flags & FPGA_MGR_PARTIAL_RECONFIG))
> >
> >>+
> >>+       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_reset = 1;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       fme_pr_ctl.pr_reset_ack = 1;
> >>+
> >>+       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
> >>+                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >>+               dev_err(&pdev->dev, "maximum PR timeout\n");
> >>+               return -ETIMEDOUT;
> >>+       }
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_reset = 0;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       dev_dbg(&pdev->dev,
> >>+               "waiting for PR resource in HW to be initialized and ready\n");
> >>+
> >>+       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
> >>+
> >>+       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
> >>+                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
> >>+               dev_err(&pdev->dev, "maximum PR timeout\n");
> >>+               return -ETIMEDOUT;
> >>+       }
> >>+
> >>+       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
> >>+       pr_err_handle(pdev, fme_pr);
> >>+       return 0;
> >>+}
> >>+
> >>+static int fme_pr_write(struct fpga_manager *mgr,
> >>+                       const char *buf, size_t count)
> >>+{
> >>+       struct fpga_fme *fme = mgr->priv;
> >>+       struct platform_device *pdev;
> >>+       struct feature_fme_pr *fme_pr;
> >>+       struct feature_fme_pr_ctl fme_pr_ctl;
> >>+       struct feature_fme_pr_status fme_pr_status;
> >>+       struct feature_fme_pr_data fme_pr_data;
> >>+       int delay, pr_credit, i = 0;
> >>+
> >>+       pdev = fme->pdata->dev;
> >>+       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >>+                               FME_FEATURE_ID_PR_MGMT);
> >>+
> >>+       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_regionid = fme->port_id;
> >>+       fme_pr_ctl.pr_start_req = 1;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
> >>+
> >>+       fme_pr_status.csr = readq(&fme_pr->status);
> >>+       pr_credit = fme_pr_status.pr_credit;
> >>+
> >>+       while (count > 0) {
> >>+               delay = 0;
> >>+               while (pr_credit <= 1) {
> >>+                       if (delay++ > PR_WAIT_TIMEOUT) {
> >>+                               dev_err(&pdev->dev, "maximum try\n");
> >>+                               return -ETIMEDOUT;
> >>+                       }
> >>+                       udelay(1);
> >>+
> >>+                       fme_pr_status.csr = readq(&fme_pr->status);
> >>+                       pr_credit = fme_pr_status.pr_credit;
> >>+               };
> >>+
> >>+               if (count >= 4) {
> >>+                       fme_pr_data.rsvd = 0;
> >>+                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
> >>+                       writeq(fme_pr_data.csr, &fme_pr->data);
> >>+                       count -= 4;
> >>+                       pr_credit--;
> >>+                       i++;
> >>+               } else {
> >>+                       WARN_ON(1);
> >>+                       return -EINVAL;
> >>+               }
> >>+       }
> >>+
> >>+       return 0;
> >>+}
> >>+
> >>+static int fme_pr_write_complete(struct fpga_manager *mgr,
> >>+                       struct fpga_image_info *info)
> >>+{
> >>+       struct fpga_fme *fme = mgr->priv;
> >>+       struct platform_device *pdev;
> >>+       struct feature_fme_pr *fme_pr;
> >>+       struct feature_fme_pr_ctl fme_pr_ctl;
> >>+
> >>+       pdev = fme->pdata->dev;
> >>+       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
> >>+                               FME_FEATURE_ID_PR_MGMT);
> >>+
> >>+       fme_pr_ctl.csr = readq(&fme_pr->control);
> >>+       fme_pr_ctl.pr_push_complete = 1;
> >>+       writeq(fme_pr_ctl.csr, &fme_pr->control);
> >>+
> >>+       dev_dbg(&pdev->dev, "green bitstream push complete\n");
> >>+       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
> >>+
> >>+       fme_pr_ctl.pr_start_req = 0;
> >>+
> >>+       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
> >>+                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
> >>+               dev_err(&pdev->dev, "maximum try.\n");
> >>+               return -ETIMEDOUT;
> >>+       }
> >>+
> >>+       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
> >>+       fme->pr_err = pr_err_handle(pdev, fme_pr);
> >>+       if (fme->pr_err)
> >>+               return -EIO;
> >>+
> >>+       dev_dbg(&pdev->dev, "PR done successfully\n");
> >>+       return 0;
> >>+}
> >>+
> >>+static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
> >>+{
> >>+       return FPGA_MGR_STATE_UNKNOWN;
> >>+}
> 
> The functions that implement the fme_pr_ops are really a low level fpga
> manager driver for the Altera PR IP component.  A standalone version of
> such a driver has been reviewed and Acked.  See the links below.
> Could this file use those functions and remove this code?
> 
> http://marc.info/?l=linux-kernel&m=149019678925564&w=2
> http://marc.info/?l=linux-kernel&m=149019654225457&w=2
> http://marc.info/?l=linux-kernel&m=149019598025274&w=2
> http://marc.info/?l=linux-kernel&m=149013051007149&w=2
> 

Thanks for the info, I just checked these code quickly, but it seems the
register definitions are totally different, so I feel that we may not be
able to reuse this driver.

Thanks
Hao

--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 4, 2017, 10:37 p.m. UTC | #16

On Tue, Apr 4, 2017 at 1:05 AM, Wu Hao <[e-mail]> wrote:
> On Mon, Apr 03, 2017 at 11:30:55AM -0500, Alan Tull wrote:
>> On Sat, Apr 1, 2017 at 6:08 AM, Wu Hao <[e-mail]> wrote:
>> > On Fri, Mar 31, 2017 at 02:10:12PM -0500, Alan Tull wrote:
>> >> On Thu, Mar 30, 2017 at 7:08 AM, Wu Hao <[e-mail]> wrote:
>> >> > From: Kang Luwei <[e-mail]>
>> >> >
>> >> > Partial Reconfiguration (PR) is the most important function for FME. It
>> >> > allows reconfiguration for given Port/Accelerated Function Unit (AFU).
>> >> >
>> >> > This patch adds support for PR sub feature. In this patch, it registers
>> >> > a fpga_mgr and implements fpga_manager_ops, and invoke fpga_mgr_buf_load
>> >> > for PR operation once PR request received via ioctl. Below user space
>> >> > interfaces are exposed by this sub feature.
>> >> >
>> >> > Sysfs interface:
>> >> > * /sys/class/fpga/<fpga.x>/<intel-fpga-fme.x>/interface_id
>> >> >   Read-only. Indicate the hardware interface information. Userspace
>> >> >   applications need to check this interface to select correct green
>> >> >   bitstream format before PR.
>> >> >
>> >> > Ioctl interface:
>> >> > * FPGA_FME_PORT_PR
>> >> >   Do partial reconfiguration per information from userspace, including
>> >> >   target port(AFU), buffer size and address info. It returns the PR status
>> >> >   (PR error code if failed) to userspace.
>> >> >
>> >> > Signed-off-by: Tim Whisonant <[e-mail]>
>> >> > Signed-off-by: Enno Luebbers <[e-mail]>
>> >> > Signed-off-by: Shiva Rao <[e-mail]>
>> >> > Signed-off-by: Christopher Rauer <[e-mail]>
>> >> > Signed-off-by: Alan Tull <[e-mail]>
>> >> > Signed-off-by: Kang Luwei <[e-mail]>
>> >> > Signed-off-by: Xiao Guangrong <[e-mail]>
>> >> > Signed-off-by: Wu Hao <[e-mail]>
>> >> > ---
>> >> >  drivers/fpga/intel/Makefile      |   2 +-
>> >> >  drivers/fpga/intel/feature-dev.h |  58 ++++++
>> >> >  drivers/fpga/intel/fme-main.c    |  44 ++++-
>> >> >  drivers/fpga/intel/fme-pr.c      | 400 +++++++++++++++++++++++++++++++++++++++
>> >> >  drivers/fpga/intel/fme.h         |  32 ++++
>> >> >  include/uapi/linux/intel-fpga.h  |  44 +++++
>> >> >  6 files changed, 578 insertions(+), 2 deletions(-)
>> >> >  create mode 100644 drivers/fpga/intel/fme-pr.c
>> >> >  create mode 100644 drivers/fpga/intel/fme.h
>> >> >
>> >> > diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
>> >> > index 546861d..0452cb6 100644
>> >> > --- a/drivers/fpga/intel/Makefile
>> >> > +++ b/drivers/fpga/intel/Makefile
>> >> > @@ -2,4 +2,4 @@ obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
>> >> >  obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
>> >> >
>> >> >  intel-fpga-pci-objs := pcie.o feature-dev.o
>> >> > -intel-fpga-fme-objs := fme-main.o
>> >> > +intel-fpga-fme-objs := fme-main.o fme-pr.o
>> >> > diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
>> >> > index dccc283..5a25c915 100644
>> >> > --- a/drivers/fpga/intel/feature-dev.h
>> >> > +++ b/drivers/fpga/intel/feature-dev.h
>> >> > @@ -150,8 +150,66 @@ struct feature_fme_err {
>> >> >  };
>> >> >
>> >> >  /* FME Partial Reconfiguration Sub Feature Register Set */
>> >> > +/* FME PR Control Register */
>> >> > +struct feature_fme_pr_ctl {
>> >> > +       union {
>> >> > +               u64 csr;
>> >> > +               struct {
>> >> > +                       u8  pr_reset:1;         /* Reset PR Engine */
>> >> > +                       u8  rsvdz1:3;
>> >> > +                       u8  pr_reset_ack:1;     /* Reset PR Engine Ack */
>> >> > +                       u8  rsvdz2:3;
>> >> > +                       u8  pr_regionid:2;      /* PR Region ID */
>> >> > +                       u8  rsvdz3:2;
>> >> > +                       u8  pr_start_req:1;     /* PR Start Request */
>> >> > +                       u8  pr_push_complete:1; /* PR Data push complete */
>> >> > +                       u8  pr_kind:1;          /* Load Customer or Intel GBS */
>> >> > +                       u32 rsvdz4:17;
>> >> > +                       u32 config_data;
>> >> > +               };
>> >> > +       };
>> >> > +};
>> >> > +
>> >> > +/* FME PR Status Register */
>> >> > +struct feature_fme_pr_status {
>> >> > +       union {
>> >> > +               u64 csr;
>> >> > +               struct {
>> >> > +                       u16 pr_credit:9;        /* Number of PR Credits */
>> >> > +                       u8  rsvdz1:7;
>> >> > +                       u8  pr_status:1;        /* PR Operation status */
>> >> > +                       u8  rsvdz2:3;
>> >> > +                       u8  pr_ctrlr_status:3;  /* Controller status */
>> >> > +                       u8  rsvdz3:1;
>> >> > +                       u8  pr_host_status:4;   /* PR Host status */
>> >> > +                       u64 rsvdz4:36;
>> >> > +               };
>> >> > +       };
>> >> > +};
>> >> > +
>> >> > +/* FME PR Data Register */
>> >> > +struct feature_fme_pr_data {
>> >> > +       union {
>> >> > +               u64 csr;
>> >> > +               struct {
>> >> > +                       /* PR data from the raw-binary file */
>> >> > +                       u32 pr_data_raw;
>> >> > +                       u32 rsvd;
>> >> > +               };
>> >> > +       };
>> >> > +};
>> >> > +
>> >> >  struct feature_fme_pr {
>> >> >         struct feature_header header;
>> >> > +       struct feature_fme_pr_ctl control;
>> >> > +       struct feature_fme_pr_status status;
>> >> > +       struct feature_fme_pr_data data;
>> >> > +       u64 error;
>> >> > +
>> >> > +       u64 rsvd[16];
>> >> > +
>> >> > +       u64 intfc_id_l;         /* PR interface Id Low */
>> >> > +       u64 intfc_id_h;         /* PR interface Id High */
>> >> >  };
>> >> >
>> >> >  /* PORT Header Register Set */
>> >> > diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
>> >> > index 36d0c4c..0d9a7a6 100644
>> >> > --- a/drivers/fpga/intel/fme-main.c
>> >> > +++ b/drivers/fpga/intel/fme-main.c
>> >> > @@ -23,6 +23,7 @@
>> >> >  #include <linux/intel-fpga.h>
>> >> >
>> >> >  #include "feature-dev.h"
>> >> > +#include "fme.h"
>> >> >
>> >> >  static ssize_t ports_num_show(struct device *dev,
>> >> >                               struct device_attribute *attr, char *buf)
>> >> > @@ -101,6 +102,10 @@ static struct feature_driver fme_feature_drvs[] = {
>> >> >                 .ops = &fme_hdr_ops,
>> >> >         },
>> >> >         {
>> >> > +               .name = FME_FEATURE_PR_MGMT,
>> >> > +               .ops = &pr_mgmt_ops,
>> >> > +       },
>> >> > +       {
>> >> >                 .ops = NULL,
>> >> >         },
>> >> >  };
>> >> > @@ -182,14 +187,48 @@ static const struct file_operations fme_fops = {
>> >> >         .unlocked_ioctl = fme_ioctl,
>> >> >  };
>> >> >
>> >> > +static int fme_dev_init(struct platform_device *pdev)
>> >> > +{
>> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> >> > +       struct fpga_fme *fme;
>> >> > +
>> >> > +       fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
>> >> > +       if (!fme)
>> >> > +               return -ENOMEM;
>> >> > +
>> >> > +       fme->pdata = pdata;
>> >> > +
>> >> > +       mutex_lock(&pdata->lock);
>> >> > +       fpga_pdata_set_private(pdata, fme);
>> >> > +       mutex_unlock(&pdata->lock);
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static void fme_dev_destroy(struct platform_device *pdev)
>> >> > +{
>> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> >> > +       struct fpga_fme *fme;
>> >> > +
>> >> > +       mutex_lock(&pdata->lock);
>> >> > +       fme = fpga_pdata_get_private(pdata);
>> >> > +       fpga_pdata_set_private(pdata, NULL);
>> >> > +       mutex_unlock(&pdata->lock);
>> >> > +
>> >> > +       devm_kfree(&pdev->dev, fme);
>> >> > +}
>> >> > +
>> >> >  static int fme_probe(struct platform_device *pdev)
>> >> >  {
>> >> >         int ret;
>> >> >
>> >> > -       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> >> > +       ret = fme_dev_init(pdev);
>> >> >         if (ret)
>> >> >                 goto exit;
>> >> >
>> >> > +       ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
>> >> > +       if (ret)
>> >> > +               goto dev_destroy;
>> >> > +
>> >> >         ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
>> >> >         if (ret)
>> >> >                 goto feature_uinit;
>> >> > @@ -198,6 +237,8 @@ static int fme_probe(struct platform_device *pdev)
>> >> >
>> >> >  feature_uinit:
>> >> >         fpga_dev_feature_uinit(pdev);
>> >> > +dev_destroy:
>> >> > +       fme_dev_destroy(pdev);
>> >> >  exit:
>> >> >         return ret;
>> >> >  }
>> >> > @@ -206,6 +247,7 @@ static int fme_remove(struct platform_device *pdev)
>> >> >  {
>> >> >         fpga_dev_feature_uinit(pdev);
>> >> >         fpga_unregister_dev_ops(pdev);
>> >> > +       fme_dev_destroy(pdev);
>> >> >         return 0;
>> >> >  }
>> >> >
>> >> > diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
>> >> > new file mode 100644
>> >> > index 0000000..3b44a3e
>> >> > --- /dev/null
>> >> > +++ b/drivers/fpga/intel/fme-pr.c
>> >> > @@ -0,0 +1,400 @@
>> >> > +/*
>> >> > + * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
>> >> > + *
>> >> > + * Copyright (C) 2017 Intel Corporation, Inc.
>> >> > + *
>> >> > + * Authors:
>> >> > + *   Kang Luwei <[e-mail]>
>> >> > + *   Xiao Guangrong <[e-mail]>
>> >> > + *   Joseph Grecco <[e-mail]>
>> >> > + *   Enno Luebbers <[e-mail]>
>> >> > + *   Tim Whisonant <[e-mail]>
>> >> > + *   Ananda Ravuri <[e-mail]>
>> >> > + *   Christopher Rauer <[e-mail]>
>> >> > + *   Henry Mitchel <[e-mail]>
>> >> > + *
>> >> > + * This work is licensed under a dual BSD/GPLv2 license. When using or
>> >> > + * redistributing this file, you may do so under either license. See the
>> >> > + * LICENSE.BSD file under this directory for the BSD license and see
>> >> > + * the COPYING file in the top-level directory for the GPLv2 license.
>> >> > + */
>> >> > +
>> >> > +#include <linux/types.h>
>> >> > +#include <linux/device.h>
>> >> > +#include <linux/vmalloc.h>
>> >> > +#include <linux/uaccess.h>
>> >> > +#include <linux/fpga/fpga-mgr.h>
>> >> > +#include <linux/intel-fpga.h>
>> >> > +
>> >> > +#include "feature-dev.h"
>> >> > +#include "fme.h"
>> >> > +
>> >> > +#define PR_WAIT_TIMEOUT   8000000
>> >> > +
>> >> > +#define PR_HOST_STATUS_IDLE    0
>> >> > +
>> >> > +DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
>> >> > +
>> >> > +static ssize_t interface_id_show(struct device *dev,
>> >> > +                                struct device_attribute *attr, char *buf)
>> >> > +{
>> >> > +       u64 intfc_id_l, intfc_id_h;
>> >> > +       struct feature_fme_pr *fme_pr
>> >> > +               = get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
>> >> > +
>> >> > +       intfc_id_l = readq(&fme_pr->intfc_id_l);
>> >> > +       intfc_id_h = readq(&fme_pr->intfc_id_h);
>> >> > +
>> >> > +       return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
>> >> > +                       (unsigned long long)intfc_id_h,
>> >> > +                       (unsigned long long)intfc_id_l);
>> >> > +}
>> >> > +static DEVICE_ATTR_RO(interface_id);
>> >> > +
>> >> > +static struct attribute *pr_mgmt_attrs[] = {
>> >> > +       &dev_attr_interface_id.attr,
>> >> > +       NULL,
>> >> > +};
>> >> > +
>> >> > +struct attribute_group pr_mgmt_attr_group = {
>> >> > +       .attrs  = pr_mgmt_attrs,
>> >> > +       .name   = "pr",
>> >> > +};
>> >> > +
>> >> > +static u64
>> >> > +pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
>> >> > +{
>> >> > +       struct feature_fme_pr_status fme_pr_status;
>> >> > +       unsigned long err_code;
>> >> > +       u64 fme_pr_error;
>> >> > +       int i = 0;
>> >> > +
>> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> >> > +       if (!fme_pr_status.pr_status)
>> >> > +               return 0;
>> >> > +
>> >> > +       err_code = fme_pr_error = readq(&fme_pr->error);
>> >> > +       for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
>> >> > +               dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
>> >> > +       writeq(fme_pr_error, &fme_pr->error);
>> >> > +       return fme_pr_error;
>> >> > +}
>> >> > +
>> >> > +static int fme_pr_write_init(struct fpga_manager *mgr,
>> >> > +               struct fpga_image_info *info, const char *buf, size_t count)
>> >> > +{
>> >> > +       struct fpga_fme *fme = mgr->priv;
>> >> > +       struct platform_device *pdev;
>> >> > +       struct feature_fme_pr *fme_pr;
>> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> >> > +       struct feature_fme_pr_status fme_pr_status;
>> >> > +
>> >> > +       pdev = fme->pdata->dev;
>> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                               FME_FEATURE_ID_PR_MGMT);
>> >> > +       if (!fme_pr)
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_reset = 1;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       fme_pr_ctl.pr_reset_ack = 1;
>> >> > +
>> >> > +       if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
>> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> >> > +               return -ETIMEDOUT;
>> >> > +       }
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_reset = 0;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev,
>> >> > +               "waiting for PR resource in HW to be initialized and ready\n");
>> >> > +
>> >> > +       fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
>> >> > +
>> >> > +       if (fpga_wait_register_field(pr_host_status, fme_pr_status,
>> >> > +                                    &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
>> >> > +               dev_err(&pdev->dev, "maximum PR timeout\n");
>> >> > +               return -ETIMEDOUT;
>> >> > +       }
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "check if have any previous PR error\n");
>> >> > +       pr_err_handle(pdev, fme_pr);
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static int fme_pr_write(struct fpga_manager *mgr,
>> >> > +                       const char *buf, size_t count)
>> >> > +{
>> >> > +       struct fpga_fme *fme = mgr->priv;
>> >> > +       struct platform_device *pdev;
>> >> > +       struct feature_fme_pr *fme_pr;
>> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> >> > +       struct feature_fme_pr_status fme_pr_status;
>> >> > +       struct feature_fme_pr_data fme_pr_data;
>> >> > +       int delay, pr_credit, i = 0;
>> >> > +
>> >> > +       pdev = fme->pdata->dev;
>> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                               FME_FEATURE_ID_PR_MGMT);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "set PR port ID and start request\n");
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_regionid = fme->port_id;
>> >> > +       fme_pr_ctl.pr_start_req = 1;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
>> >> > +
>> >> > +       fme_pr_status.csr = readq(&fme_pr->status);
>> >> > +       pr_credit = fme_pr_status.pr_credit;
>> >> > +
>> >> > +       while (count > 0) {
>> >> > +               delay = 0;
>> >> > +               while (pr_credit <= 1) {
>> >> > +                       if (delay++ > PR_WAIT_TIMEOUT) {
>> >> > +                               dev_err(&pdev->dev, "maximum try\n");
>> >> > +                               return -ETIMEDOUT;
>> >> > +                       }
>> >> > +                       udelay(1);
>> >> > +
>> >> > +                       fme_pr_status.csr = readq(&fme_pr->status);
>> >> > +                       pr_credit = fme_pr_status.pr_credit;
>> >> > +               };
>> >> > +
>> >> > +               if (count >= 4) {
>> >> > +                       fme_pr_data.rsvd = 0;
>> >> > +                       fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
>> >> > +                       writeq(fme_pr_data.csr, &fme_pr->data);
>> >> > +                       count -= 4;
>> >> > +                       pr_credit--;
>> >> > +                       i++;
>> >> > +               } else {
>> >> > +                       WARN_ON(1);
>> >> > +                       return -EINVAL;
>> >> > +               }
>> >> > +       }
>> >> > +
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static int fme_pr_write_complete(struct fpga_manager *mgr,
>> >> > +                       struct fpga_image_info *info)
>> >> > +{
>> >> > +       struct fpga_fme *fme = mgr->priv;
>> >> > +       struct platform_device *pdev;
>> >> > +       struct feature_fme_pr *fme_pr;
>> >> > +       struct feature_fme_pr_ctl fme_pr_ctl;
>> >> > +
>> >> > +       pdev = fme->pdata->dev;
>> >> > +       fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                               FME_FEATURE_ID_PR_MGMT);
>> >> > +
>> >> > +       fme_pr_ctl.csr = readq(&fme_pr->control);
>> >> > +       fme_pr_ctl.pr_push_complete = 1;
>> >> > +       writeq(fme_pr_ctl.csr, &fme_pr->control);
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "green bitstream push complete\n");
>> >> > +       dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
>> >> > +
>> >> > +       fme_pr_ctl.pr_start_req = 0;
>> >> > +
>> >> > +       if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
>> >> > +                                    &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
>> >> > +               dev_err(&pdev->dev, "maximum try.\n");
>> >> > +               return -ETIMEDOUT;
>> >> > +       }
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
>> >> > +       fme->pr_err = pr_err_handle(pdev, fme_pr);
>> >> > +       if (fme->pr_err)
>> >> > +               return -EIO;
>> >> > +
>> >> > +       dev_dbg(&pdev->dev, "PR done successfully\n");
>> >> > +       return 0;
>> >> > +}
>> >> > +
>> >> > +static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
>> >> > +{
>> >> > +       return FPGA_MGR_STATE_UNKNOWN;
>> >> > +}
>> >> > +
>> >> > +static const struct fpga_manager_ops fme_pr_ops = {
>> >> > +       .write_init = fme_pr_write_init,
>> >> > +       .write = fme_pr_write,
>> >> > +       .write_complete = fme_pr_write_complete,
>> >> > +       .state = fme_pr_state,
>> >> > +};
>> >> > +
>> >> > +static int fme_pr(struct platform_device *pdev, unsigned long arg)
>> >> > +{
>> >> > +       void __user *argp = (void __user *)arg;
>> >> > +       struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
>> >> > +       struct fpga_fme *fme;
>> >> > +       struct fpga_manager *mgr;
>> >> > +       struct feature_fme_header *fme_hdr;
>> >> > +       struct feature_fme_capability fme_capability;
>> >> > +       struct fpga_image_info info;
>> >> > +       struct fpga_fme_port_pr port_pr;
>> >> > +       struct platform_device *port;
>> >> > +       unsigned long minsz;
>> >> > +       void *buf = NULL;
>> >> > +       int ret = 0;
>> >> > +
>> >> > +       minsz = offsetofend(struct fpga_fme_port_pr, status);
>> >> > +
>> >> > +       if (copy_from_user(&port_pr, argp, minsz))
>> >> > +               return -EFAULT;
>> >> > +
>> >> > +       if (port_pr.argsz < minsz || port_pr.flags)
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       if (!IS_ALIGNED(port_pr.buffer_size, 4))
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       /* get fme header region */
>> >> > +       fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
>> >> > +                                       FME_FEATURE_ID_HEADER);
>> >> > +       if (WARN_ON(!fme_hdr))
>> >> > +               return -EINVAL;
>> >> > +
>> >> > +       /* check port id */
>> >> > +       fme_capability.csr = readq(&fme_hdr->capability);
>> >> > +       if (port_pr.port_id >= fme_capability.num_ports) {
>> >> > +               dev_dbg(&pdev->dev, "port number more than maximum\n");
>> >> > +               return -EINVAL;
>> >> > +       }
>> >> > +
>> >> > +       if (!access_ok(VERIFY_READ, port_pr.buffer_address,
>> >> > +                                   port_pr.buffer_size))
>> >> > +               return -EFAULT;
>> >> > +
>> >> > +       buf = vmalloc(port_pr.buffer_size);
>> >> > +       if (!buf)
>> >> > +               return -ENOMEM;
>> >> > +
>> >> > +       if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
>> >> > +                                              port_pr.buffer_size)) {
>> >> > +               ret = -EFAULT;
>> >> > +               goto free_exit;
>> >> > +       }
>> >> > +
>> >> > +       memset(&info, 0, sizeof(struct fpga_image_info));
>> >> > +       info.flags = FPGA_MGR_PARTIAL_RECONFIG;
>> >> > +
>> >> > +       mgr = fpga_mgr_get(&pdev->dev);
>> >> > +       if (IS_ERR(mgr)) {
>> >> > +               ret = PTR_ERR(mgr);
>> >> > +               goto free_exit;
>> >> > +       }
>> >> > +
>> >> > +       mutex_lock(&pdata->lock);
>> >> > +       fme = fpga_pdata_get_private(pdata);
>> >> > +       /* fme device has been unregistered. */
>> >> > +       if (!fme) {
>> >> > +               ret = -EINVAL;
>> >> > +               goto unlock_exit;
>> >> > +       }
>> >> > +
>> >> > +       fme->pr_err = 0;
>> >> > +       fme->port_id = port_pr.port_id;
>> >>
>> >> It looks like you're using private data to communicate with the
>> >> driver, i.e. there is something you want to do with the fpga manager
>> >> framework and it doesn't have that feature.  The better way would be
>> >> for us to expand the framework so you don't need to do that.
>> >>
>> >> port_id is the kind of thing that should be communicated to the driver
>> >> through fpga_image_info, so we could add that to the struct.  Should
>> >> we call it port_id?
>>
>> Let's call it region_id.
>>
>> >> Or is there something more generic that may be
>> >> useful in the future for other architectures?.
>> >
>> > Hi Alan
>> >
>> > Thanks for your feedback. :)
>> >
>> > As you know, each Intel FPGA device may have more than one accelerator,
>> > and all accelerators share the same fpga_manager (only one FME module).
>> > port_id = 0 means the first accelerator on this fpga devices. So it's
>> > needed by the fpga_manager to distinguish one accelerator from others
>> > for partial reconfiguration.
>> >
>> > Adding a member like a 'id' to fpga_image_info definitely works for us
>> > in this case. We can add it this way, but I feel 'id' here seems not
>> > related to fpga image, but characteristic of fpga region.
>>
>> The  fpga_image_info struct started life as just image specific info,
>> but I want it to go in the direction of including parameters needed to
>> program it this specific time. Otherwise we are stuck having to keep
>> adding parameters as our use of FPGA develops.  It probably could be
>> documented better as 'information needed to program a FPGA image'
>> rather than strictly 'information about this particular FPGA image'.
>> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>> goes in this direction by having the buf, firmware name, or sg list
>> passed in the info for the added fpga_mgr_load() function.  Actually I
>> should probably simplify the API and get rid of fpga_mgr_buf_load,
>> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>> use fpga_mgr_load (passing all parameters in fpga_image_info).
>>
>
> Make sense.
>
>> > It may be a
>> > little confusing. One rough idea is that keep this info under fpga region
>> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>>
>> Yes, keep this info in fpga-region.  When the region wants to program
>> using fpga-mgr, add the region id to fpga_image_info.  I propose
>> calling it region_id.
>
> Hm.. Do we need a function which moves info from region to image info?

No, just code that sets that variable in the struct before calling the
fpga_region_program_fpga function.

>
> Another idea is, add a priv to fpga_image_info, and use a common function
> to pass the fpga_region's priv to fpga_image_info's priv before PR.
> fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>

Adding priv would make the interface for fpga-mgr non-uniform.  The point
of having a fpga-mgr framework is that there
is the potential of the upper layers working for different FPGA devices.
If the interface for each FPGA device were different, that would then
be broken.

>>
>> > then fpga_manager knows the target region for partial reconfiguration.
>> > If consider pr sysfs interface support under fpga-region in the future,
>> > then we don't need to create a new 'id' sysfs file, as fpga-region itself
>> > could provide this kind of info. But I'm not sure if this is the right
>> > direction.
>> >
>> >>
>> >> pr_err appears to be machine specific error codes that are
>> >> communicated outside your low level driver.  (Calling it pr_err is
>> >> extra confusing since Linux already has a commonly name function by
>> >> the same name).  The framework has state, but that's not doing what
>> >> you want here.  Maybe we could add a framework ops called status so
>> >> that status could be communicated from the low level driver. It would
>> >> be useful to abstract the machine specific state to a defined enum
>> >> that would be part of the fpga mgr framework.  I remember we had
>> >> something like that in the earliest version of fpga manager but it got
>> >> changed to state rather than status for some reason.
>> >>
>> >
>> > Adding a status function and a 'status' member to fpga manager sounds good.
>> > But I'm not sure how to abstract these error codes, currently what we have
>> > are all PR error codes for Intel FPGA device, e.g "PR FIFO overflow error",
>> > "PR secure load error" and etc (see include/uapi/linux/intel-fpga.h). Is it
>> > fine to let each driver to define how to use that 'status' for machine
>> > specific status?
>>
>> I looked at the list of errors in include/uapi/linux/intel-fpga.h.
>> They all seem pretty generic to me except I am not clear what "secure
>> load error" or "IP protocol error" mean and whether other
>> architectures would have them.  But certainly things like crc error,
>> incompatible bitstream, fifo overflow are generic.  So let's see if we
>> can define all these in a way that's generic and just pass up a error
>> number.  That way upper layers can know how to deal with them
>> possibly.  I would take the word "PR" off these since the error set
>> applies whether someone is doing full reconfig or partial reconfig.
>
> Sure, good to me, we can make it this way and see.
> Thanks for the suggestions. :)
>
> Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 5, 2017, 11:40 a.m. UTC | #17

PiA+PiBUaGUgIGZwZ2FfaW1hZ2VfaW5mbyBzdHJ1Y3Qgc3RhcnRlZCBsaWZlIGFzIGp1c3QgaW1h
Z2Ugc3BlY2lmaWMgaW5mbywNCj4gPj4gYnV0IEkgd2FudCBpdCB0byBnbyBpbiB0aGUgZGlyZWN0
aW9uIG9mIGluY2x1ZGluZyBwYXJhbWV0ZXJzIG5lZWRlZCB0bw0KPiA+PiBwcm9ncmFtIGl0IHRo
aXMgc3BlY2lmaWMgdGltZS4gT3RoZXJ3aXNlIHdlIGFyZSBzdHVjayBoYXZpbmcgdG8ga2VlcA0K
PiA+PiBhZGRpbmcgcGFyYW1ldGVycyBhcyBvdXIgdXNlIG9mIEZQR0EgZGV2ZWxvcHMuICBJdCBw
cm9iYWJseSBjb3VsZCBiZQ0KPiA+PiBkb2N1bWVudGVkIGJldHRlciBhcyAnaW5mb3JtYXRpb24g
bmVlZGVkIHRvIHByb2dyYW0gYSBGUEdBIGltYWdlJw0KPiA+PiByYXRoZXIgdGhhbiBzdHJpY3Rs
eSAnaW5mb3JtYXRpb24gYWJvdXQgdGhpcyBwYXJ0aWN1bGFyIEZQR0EgaW1hZ2UnLg0KPiA+PiBN
eSBwYXRjaCAiZnBnYS1tZ3I6IHBhc3MgcGFyYW1ldGVycyBmb3IgbG9hZGluZyBmcGdhIGluIGlt
YWdlIGluZm8iDQo+ID4+IGdvZXMgaW4gdGhpcyBkaXJlY3Rpb24gYnkgaGF2aW5nIHRoZSBidWYs
IGZpcm13YXJlIG5hbWUsIG9yIHNnIGxpc3QNCj4gPj4gcGFzc2VkIGluIHRoZSBpbmZvIGZvciB0
aGUgYWRkZWQgZnBnYV9tZ3JfbG9hZCgpIGZ1bmN0aW9uLiAgQWN0dWFsbHkgSQ0KPiA+PiBzaG91
bGQgcHJvYmFibHkgc2ltcGxpZnkgdGhlIEFQSSBhbmQgZ2V0IHJpZCBvZiBmcGdhX21ncl9idWZf
bG9hZCwNCj4gPj4gZnBnYV9tZ3JfYnVmX2xvYWRfc2csIGFuZCBmcGdhX21ncl9maXJtd2FyZV9s
b2FkIGFuZCByZXF1aXJlIHBlb3BsZSB0bw0KPiA+PiB1c2UgZnBnYV9tZ3JfbG9hZCAocGFzc2lu
ZyBhbGwgcGFyYW1ldGVycyBpbiBmcGdhX2ltYWdlX2luZm8pLg0KPiA+Pg0KPiA+DQo+ID4gTWFr
ZSBzZW5zZS4NCj4gPg0KPiA+PiA+IEl0IG1heSBiZSBhDQo+ID4+ID4gbGl0dGxlIGNvbmZ1c2lu
Zy4gT25lIHJvdWdoIGlkZWEgaXMgdGhhdCBrZWVwIHRoaXMgaW5mbyB1bmRlciBmcGdhIHJlZ2lv
bg0KPiA+PiA+IChtYXliZSBpdHMgcHJpdmF0ZSBkYXRhKSwgYW5kIHBhc3MgdGhlIGZwZ2EtcmVn
aW9uIHRvIGZwZ2FfbWdyX2J1Zl9sb2FkLA0KPiA+Pg0KPiA+PiBZZXMsIGtlZXAgdGhpcyBpbmZv
IGluIGZwZ2EtcmVnaW9uLiAgV2hlbiB0aGUgcmVnaW9uIHdhbnRzIHRvIHByb2dyYW0NCj4gPj4g
dXNpbmcgZnBnYS1tZ3IsIGFkZCB0aGUgcmVnaW9uIGlkIHRvIGZwZ2FfaW1hZ2VfaW5mby4gIEkg
cHJvcG9zZQ0KPiA+PiBjYWxsaW5nIGl0IHJlZ2lvbl9pZC4NCj4gPg0KPiA+IEhtLi4gRG8gd2Ug
bmVlZCBhIGZ1bmN0aW9uIHdoaWNoIG1vdmVzIGluZm8gZnJvbSByZWdpb24gdG8gaW1hZ2UgaW5m
bz8NCj4gDQo+IE5vLCBqdXN0IGNvZGUgdGhhdCBzZXRzIHRoYXQgdmFyaWFibGUgaW4gdGhlIHN0
cnVjdCBiZWZvcmUgY2FsbGluZyB0aGUNCj4gZnBnYV9yZWdpb25fcHJvZ3JhbV9mcGdhIGZ1bmN0
aW9uLg0KPg0KPiA+DQo+ID4gQW5vdGhlciBpZGVhIGlzLCBhZGQgYSBwcml2IHRvIGZwZ2FfaW1h
Z2VfaW5mbywgYW5kIHVzZSBhIGNvbW1vbiBmdW5jdGlvbg0KPiA+IHRvIHBhc3MgdGhlIGZwZ2Ff
cmVnaW9uJ3MgcHJpdiB0byBmcGdhX2ltYWdlX2luZm8ncyBwcml2IGJlZm9yZSBQUi4NCj4gPiBm
cGdhLW1nciB0aGVuIGtub3dzIGZwZ2FfcmVnaW9uIHByaXYgaW5mbyBmcm9tIHRoZSBmcGdhX2lt
YWdlX2luZm8uDQo+ID4NCj4gDQo+IEFkZGluZyBwcml2IHdvdWxkIG1ha2UgdGhlIGludGVyZmFj
ZSBmb3IgZnBnYS1tZ3Igbm9uLXVuaWZvcm0uICBUaGUgcG9pbnQNCj4gb2YgaGF2aW5nIGEgZnBn
YS1tZ3IgZnJhbWV3b3JrIGlzIHRoYXQgdGhlcmUNCj4gaXMgdGhlIHBvdGVudGlhbCBvZiB0aGUg
dXBwZXIgbGF5ZXJzIHdvcmtpbmcgZm9yIGRpZmZlcmVudCBGUEdBIGRldmljZXMuDQo+IElmIHRo
ZSBpbnRlcmZhY2UgZm9yIGVhY2ggRlBHQSBkZXZpY2Ugd2VyZSBkaWZmZXJlbnQsIHRoYXQgd291
bGQgdGhlbg0KPiBiZSBicm9rZW4uDQo+IA0KDQpJIG1lYW4gZHJpdmVycyBjYW4gcmVnaXN0ZXIg
dGhlaXIgb3duIGZwZ2EtbWdyIG9wcywgYW5kIGhhbmRsZSBwcml2IG9mDQpmcGdhX2ltYWdlX2lu
Zm8gaW4gZHJpdmVyIHNwZWNpZmljIHdheSBmb3IgcHIgKGUuZyB3cml0ZV9pbml0IGZ1bmN0aW9u
KS4NCldlIGRvbid0IG5lZWQgdG8gY2hhbmdlIHRoZSBhbnkgdXBwZXIgbGF5ZXIgaW50ZXJmYWNl
cy4NCg0KSWYgeW91IHByZWZlciB0aGUgcmVnaW9uX2lkIGZvciBmcGdhX2ltYWdlX2luZm8sIHdl
IGNhbiBnbyB3aXRoIHJlZ2lvbl9pZA0KZm9yIHN1cmUuIDogKQ0KDQpUaGFua3MNCkhhbw0K
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 5, 2017, 3:26 p.m. UTC | #18

On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <[e-mail]> wrote:
>> >> The  fpga_image_info struct started life as just image specific info,
>> >> but I want it to go in the direction of including parameters needed to
>> >> program it this specific time. Otherwise we are stuck having to keep
>> >> adding parameters as our use of FPGA develops.  It probably could be
>> >> documented better as 'information needed to program a FPGA image'
>> >> rather than strictly 'information about this particular FPGA image'.
>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>> >> goes in this direction by having the buf, firmware name, or sg list
>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
>> >>
>> >
>> > Make sense.
>> >
>> >> > It may be a
>> >> > little confusing. One rough idea is that keep this info under fpga region
>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>> >>
>> >> Yes, keep this info in fpga-region.  When the region wants to program
>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
>> >> calling it region_id.
>> >
>> > Hm.. Do we need a function which moves info from region to image info?
>>
>> No, just code that sets that variable in the struct before calling the
>> fpga_region_program_fpga function.
>>
>> >
>> > Another idea is, add a priv to fpga_image_info, and use a common function
>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>> >
>>
>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
>> of having a fpga-mgr framework is that there
>> is the potential of the upper layers working for different FPGA devices.
>> If the interface for each FPGA device were different, that would then
>> be broken.
>>
>
> I mean drivers can register their own fpga-mgr ops, and handle priv of
> fpga_image_info in driver specific way for pr (e.g write_init function).
> We don't need to change the any upper layer interfaces.

I'm trying to avoid driver specific ways of doing things.  Think of this
all as a set of blocks and we want to be able to switch out individual
blocks in the future.  It's future-proofing and also making code more
generally usable.

fpga_mgr_info is part of the interface for calls to fpga-mgr to do
reprogramming.  My patchset will push it further in that direction
as pointers to the image are added to fpga_mgr_info.

Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
It's better to add a region_id variable to fpga_mgr_info that may not be used by
all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
any other needed fpga_mgr_info variables and return error if the params
look invalid.  And ignore any it doesn't need.

I don't think priv belongs in fpga_image_info.  priv tends to be information
for a specific instance of a driver that can have several instances.  priv
usually stores a driver's memory mappings, interrupts, etc for that instance.
It's called private info as it is info that other blocks don't need to know
and don't get to look at.  It's private.  So priv as in interface strikes me as
not private any more and is sort of a red flag.

Alan

>
> If you prefer the region_id for fpga_image_info, we can go with region_id
> for sure. : )
>
> Thanks
> Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 5, 2017, 3:39 p.m. UTC | #19

On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <[e-mail]> wrote:
> On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <[e-mail]> wrote:
>>> >> The  fpga_image_info struct started life as just image specific info,
>>> >> but I want it to go in the direction of including parameters needed to
>>> >> program it this specific time. Otherwise we are stuck having to keep
>>> >> adding parameters as our use of FPGA develops.  It probably could be
>>> >> documented better as 'information needed to program a FPGA image'
>>> >> rather than strictly 'information about this particular FPGA image'.
>>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>>> >> goes in this direction by having the buf, firmware name, or sg list
>>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
>>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
>>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
>>> >>
>>> >
>>> > Make sense.
>>> >
>>> >> > It may be a
>>> >> > little confusing. One rough idea is that keep this info under fpga region
>>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>>> >>
>>> >> Yes, keep this info in fpga-region.  When the region wants to program
>>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
>>> >> calling it region_id.
>>> >
>>> > Hm.. Do we need a function which moves info from region to image info?
>>>
>>> No, just code that sets that variable in the struct before calling the
>>> fpga_region_program_fpga function.
>>>
>>> >
>>> > Another idea is, add a priv to fpga_image_info, and use a common function
>>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
>>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>>> >
>>>
>>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
>>> of having a fpga-mgr framework is that there
>>> is the potential of the upper layers working for different FPGA devices.
>>> If the interface for each FPGA device were different, that would then
>>> be broken.
>>>
>>
>> I mean drivers can register their own fpga-mgr ops, and handle priv of
>> fpga_image_info in driver specific way for pr (e.g write_init function).
>> We don't need to change the any upper layer interfaces.
>
> I'm trying to avoid driver specific ways of doing things.  Think of this
> all as a set of blocks and we want to be able to switch out individual
> blocks in the future.  It's future-proofing and also making code more
> generally usable.
>
> fpga_mgr_info is part of the interface for calls to fpga-mgr to do
> reprogramming.  My patchset will push it further in that direction
> as pointers to the image are added to fpga_mgr_info.
>
> Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
> It's better to add a region_id variable to fpga_mgr_info that may not be used by
> all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
> fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
> any other needed fpga_mgr_info variables and return error if the params
> look invalid.  And ignore any it doesn't need.
>
> I don't think priv belongs in fpga_image_info.  priv tends to be information
> for a specific instance of a driver that can have several instances.  priv
> usually stores a driver's memory mappings, interrupts, etc for that instance.
> It's called private info as it is info that other blocks don't need to know
> and don't get to look at.  It's private.  So priv as in interface strikes me as
> not private any more and is sort of a red flag.
>
> Alan

Besides that, I see that region_id is needed both for the fme_pr and
for determining which port to disable/enable.  So adding it to the
fpga_image_info would allow the fpga-region to figure out which
bridge to control as well as pass it to the mgr.

>
>>
>> If you prefer the region_id for fpga_image_info, we can go with region_id
>> for sure. : )
>>
>> Thanks
>> Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 6, 2017, 10:57 a.m. UTC | #20

On Wed, Apr 05, 2017 at 10:39:05AM -0500, Alan Tull wrote:
> On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <[e-mail]> wrote:
> > On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <[e-mail]> wrote:
> >>> >> The  fpga_image_info struct started life as just image specific info,
> >>> >> but I want it to go in the direction of including parameters needed to
> >>> >> program it this specific time. Otherwise we are stuck having to keep
> >>> >> adding parameters as our use of FPGA develops.  It probably could be
> >>> >> documented better as 'information needed to program a FPGA image'
> >>> >> rather than strictly 'information about this particular FPGA image'.
> >>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> >>> >> goes in this direction by having the buf, firmware name, or sg list
> >>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
> >>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
> >>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> >>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
> >>> >>
> >>> >
> >>> > Make sense.
> >>> >
> >>> >> > It may be a
> >>> >> > little confusing. One rough idea is that keep this info under fpga region
> >>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> >>> >>
> >>> >> Yes, keep this info in fpga-region.  When the region wants to program
> >>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
> >>> >> calling it region_id.
> >>> >
> >>> > Hm.. Do we need a function which moves info from region to image info?
> >>>
> >>> No, just code that sets that variable in the struct before calling the
> >>> fpga_region_program_fpga function.
> >>>
> >>> >
> >>> > Another idea is, add a priv to fpga_image_info, and use a common function
> >>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
> >>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
> >>> >
> >>>
> >>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
> >>> of having a fpga-mgr framework is that there
> >>> is the potential of the upper layers working for different FPGA devices.
> >>> If the interface for each FPGA device were different, that would then
> >>> be broken.
> >>>
> >>
> >> I mean drivers can register their own fpga-mgr ops, and handle priv of
> >> fpga_image_info in driver specific way for pr (e.g write_init function).
> >> We don't need to change the any upper layer interfaces.
> >
> > I'm trying to avoid driver specific ways of doing things.  Think of this
> > all as a set of blocks and we want to be able to switch out individual
> > blocks in the future.  It's future-proofing and also making code more
> > generally usable.
> >
> > fpga_mgr_info is part of the interface for calls to fpga-mgr to do
> > reprogramming.  My patchset will push it further in that direction
> > as pointers to the image are added to fpga_mgr_info.
> >
> > Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
> > It's better to add a region_id variable to fpga_mgr_info that may not be used by
> > all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
> > fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
> > any other needed fpga_mgr_info variables and return error if the params
> > look invalid.  And ignore any it doesn't need.
> >
> > I don't think priv belongs in fpga_image_info.  priv tends to be information
> > for a specific instance of a driver that can have several instances.  priv
> > usually stores a driver's memory mappings, interrupts, etc for that instance.
> > It's called private info as it is info that other blocks don't need to know
> > and don't get to look at.  It's private.  So priv as in interface strikes me as
> > not private any more and is sort of a red flag.

Thanks a lot for the details. It's more clear to me now. :)

> >
> > Alan
> 
> Besides that, I see that region_id is needed both for the fme_pr and
> for determining which port to disable/enable.  So adding it to the
> fpga_image_info would allow the fpga-region to figure out which
> bridge to control as well as pass it to the mgr.
> 

Do we need to add an 'id' to fpga-bridge too?

Per my understanding this FME driver should create fpga-region for each
accelerator, and make sure each fpga-region has correct region_id. When
this fpga-region wants to be programmed by fpga-mgr, then it add this
info to fpga_image_info, and fpga-mgr knows which region to program via
this fpga_image_info.

For each fpga-region, FME driver needs to create one fpga-bridge and link
it to region's bridge_list. When fpga_region_program_fpga is invoked for
PR. This fpga-bridge could be disabled before PR and re-enabled after PR
automatically. If fpga-bridge contains this 'id' information, then driver
knows which port to enable/disable to implement of the enable_set function
under this fpga-bridge.

Thanks
Hao

> >
> >>
> >> If you prefer the region_id for fpga_image_info, we can go with region_id
> >> for sure. : )
> >>
> >> Thanks
> >> Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alan Tull April 6, 2017, 7:27 p.m. UTC | #21

On Thu, Apr 6, 2017 at 5:57 AM, Wu Hao <[e-mail]> wrote:
> On Wed, Apr 05, 2017 at 10:39:05AM -0500, Alan Tull wrote:
>> On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <[e-mail]> wrote:
>> > On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <[e-mail]> wrote:
>> >>> >> The  fpga_image_info struct started life as just image specific info,
>> >>> >> but I want it to go in the direction of including parameters needed to
>> >>> >> program it this specific time. Otherwise we are stuck having to keep
>> >>> >> adding parameters as our use of FPGA develops.  It probably could be
>> >>> >> documented better as 'information needed to program a FPGA image'
>> >>> >> rather than strictly 'information about this particular FPGA image'.
>> >>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
>> >>> >> goes in this direction by having the buf, firmware name, or sg list
>> >>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
>> >>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
>> >>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
>> >>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
>> >>> >>
>> >>> >
>> >>> > Make sense.
>> >>> >
>> >>> >> > It may be a
>> >>> >> > little confusing. One rough idea is that keep this info under fpga region
>> >>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
>> >>> >>
>> >>> >> Yes, keep this info in fpga-region.  When the region wants to program
>> >>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
>> >>> >> calling it region_id.
>> >>> >
>> >>> > Hm.. Do we need a function which moves info from region to image info?
>> >>>
>> >>> No, just code that sets that variable in the struct before calling the
>> >>> fpga_region_program_fpga function.
>> >>>
>> >>> >
>> >>> > Another idea is, add a priv to fpga_image_info, and use a common function
>> >>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
>> >>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
>> >>> >
>> >>>
>> >>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
>> >>> of having a fpga-mgr framework is that there
>> >>> is the potential of the upper layers working for different FPGA devices.
>> >>> If the interface for each FPGA device were different, that would then
>> >>> be broken.
>> >>>
>> >>
>> >> I mean drivers can register their own fpga-mgr ops, and handle priv of
>> >> fpga_image_info in driver specific way for pr (e.g write_init function).
>> >> We don't need to change the any upper layer interfaces.
>> >
>> > I'm trying to avoid driver specific ways of doing things.  Think of this
>> > all as a set of blocks and we want to be able to switch out individual
>> > blocks in the future.  It's future-proofing and also making code more
>> > generally usable.
>> >
>> > fpga_mgr_info is part of the interface for calls to fpga-mgr to do
>> > reprogramming.  My patchset will push it further in that direction
>> > as pointers to the image are added to fpga_mgr_info.
>> >
>> > Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
>> > It's better to add a region_id variable to fpga_mgr_info that may not be used by
>> > all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
>> > fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
>> > any other needed fpga_mgr_info variables and return error if the params
>> > look invalid.  And ignore any it doesn't need.
>> >
>> > I don't think priv belongs in fpga_image_info.  priv tends to be information
>> > for a specific instance of a driver that can have several instances.  priv
>> > usually stores a driver's memory mappings, interrupts, etc for that instance.
>> > It's called private info as it is info that other blocks don't need to know
>> > and don't get to look at.  It's private.  So priv as in interface strikes me as
>> > not private any more and is sort of a red flag.
>
> Thanks a lot for the details. It's more clear to me now. :)
>
>> >
>> > Alan
>>
>> Besides that, I see that region_id is needed both for the fme_pr and
>> for determining which port to disable/enable.  So adding it to the
>> fpga_image_info would allow the fpga-region to figure out which
>> bridge to control as well as pass it to the mgr.
>>
>
> Do we need to add an 'id' to fpga-bridge too?

You don't have to, but you could.  See below.

>
> Per my understanding this FME driver should create fpga-region for each
> accelerator, and make sure each fpga-region has correct region_id. When
> this fpga-region wants to be programmed by fpga-mgr, then it add this
> info to fpga_image_info, and fpga-mgr knows which region to program via
> this fpga_image_info.

The fpga_image_info is passed both to fpga-mgr and fpga-bridge.  So
both will get the region_id through the info.

>
> For each fpga-region, FME driver needs to create one fpga-bridge and link
> it to region's bridge_list. When fpga_region_program_fpga is invoked for
> PR. This fpga-bridge could be disabled before PR and re-enabled after PR
> automatically. If fpga-bridge contains this 'id' information, then driver
> knows which port to enable/disable to implement of the enable_set function
> under this fpga-bridge.

Could implement it either way: the bridge could either use the
bridge_id that has been passed to it in the info or code that creates
the bridge could create private data and pass it to
fpga_bridge_register().  But the bridge will have the id because it
will be passed in the info, so it's not really needed.

Alan

>
> Thanks
> Hao
>
>> >
>> >>
>> >> If you prefer the region_id for fpga_image_info, we can go with region_id
>> >> for sure. : )
>> >>
>> >> Thanks
>> >> Hao
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Wu, Hao April 7, 2017, 5:56 a.m. UTC | #22

On Thu, Apr 06, 2017 at 02:27:42PM -0500, Alan Tull wrote:
> On Thu, Apr 6, 2017 at 5:57 AM, Wu Hao <[e-mail]> wrote:
> > On Wed, Apr 05, 2017 at 10:39:05AM -0500, Alan Tull wrote:
> >> On Wed, Apr 5, 2017 at 10:26 AM, Alan Tull <[e-mail]> wrote:
> >> > On Wed, Apr 5, 2017 at 6:40 AM, Wu, Hao <[e-mail]> wrote:
> >> >>> >> The  fpga_image_info struct started life as just image specific info,
> >> >>> >> but I want it to go in the direction of including parameters needed to
> >> >>> >> program it this specific time. Otherwise we are stuck having to keep
> >> >>> >> adding parameters as our use of FPGA develops.  It probably could be
> >> >>> >> documented better as 'information needed to program a FPGA image'
> >> >>> >> rather than strictly 'information about this particular FPGA image'.
> >> >>> >> My patch "fpga-mgr: pass parameters for loading fpga in image info"
> >> >>> >> goes in this direction by having the buf, firmware name, or sg list
> >> >>> >> passed in the info for the added fpga_mgr_load() function.  Actually I
> >> >>> >> should probably simplify the API and get rid of fpga_mgr_buf_load,
> >> >>> >> fpga_mgr_buf_load_sg, and fpga_mgr_firmware_load and require people to
> >> >>> >> use fpga_mgr_load (passing all parameters in fpga_image_info).
> >> >>> >>
> >> >>> >
> >> >>> > Make sense.
> >> >>> >
> >> >>> >> > It may be a
> >> >>> >> > little confusing. One rough idea is that keep this info under fpga region
> >> >>> >> > (maybe its private data), and pass the fpga-region to fpga_mgr_buf_load,
> >> >>> >>
> >> >>> >> Yes, keep this info in fpga-region.  When the region wants to program
> >> >>> >> using fpga-mgr, add the region id to fpga_image_info.  I propose
> >> >>> >> calling it region_id.
> >> >>> >
> >> >>> > Hm.. Do we need a function which moves info from region to image info?
> >> >>>
> >> >>> No, just code that sets that variable in the struct before calling the
> >> >>> fpga_region_program_fpga function.
> >> >>>
> >> >>> >
> >> >>> > Another idea is, add a priv to fpga_image_info, and use a common function
> >> >>> > to pass the fpga_region's priv to fpga_image_info's priv before PR.
> >> >>> > fpga-mgr then knows fpga_region priv info from the fpga_image_info.
> >> >>> >
> >> >>>
> >> >>> Adding priv would make the interface for fpga-mgr non-uniform.  The point
> >> >>> of having a fpga-mgr framework is that there
> >> >>> is the potential of the upper layers working for different FPGA devices.
> >> >>> If the interface for each FPGA device were different, that would then
> >> >>> be broken.
> >> >>>
> >> >>
> >> >> I mean drivers can register their own fpga-mgr ops, and handle priv of
> >> >> fpga_image_info in driver specific way for pr (e.g write_init function).
> >> >> We don't need to change the any upper layer interfaces.
> >> >
> >> > I'm trying to avoid driver specific ways of doing things.  Think of this
> >> > all as a set of blocks and we want to be able to switch out individual
> >> > blocks in the future.  It's future-proofing and also making code more
> >> > generally usable.
> >> >
> >> > fpga_mgr_info is part of the interface for calls to fpga-mgr to do
> >> > reprogramming.  My patchset will push it further in that direction
> >> > as pointers to the image are added to fpga_mgr_info.
> >> >
> >> > Adding 'priv' to fpga_mgr_info makes that interface specific to a this driver.
> >> > It's better to add a region_id variable to fpga_mgr_info that may not be used by
> >> > all fpga-mgr drivers.  The current model is that a fpga-mgr driver checks
> >> > fpga_mgr_info flags to see if its correct.  The fpga-mgr driver can check
> >> > any other needed fpga_mgr_info variables and return error if the params
> >> > look invalid.  And ignore any it doesn't need.
> >> >
> >> > I don't think priv belongs in fpga_image_info.  priv tends to be information
> >> > for a specific instance of a driver that can have several instances.  priv
> >> > usually stores a driver's memory mappings, interrupts, etc for that instance.
> >> > It's called private info as it is info that other blocks don't need to know
> >> > and don't get to look at.  It's private.  So priv as in interface strikes me as
> >> > not private any more and is sort of a red flag.
> >
> > Thanks a lot for the details. It's more clear to me now. :)
> >
> >> >
> >> > Alan
> >>
> >> Besides that, I see that region_id is needed both for the fme_pr and
> >> for determining which port to disable/enable.  So adding it to the
> >> fpga_image_info would allow the fpga-region to figure out which
> >> bridge to control as well as pass it to the mgr.
> >>
> >
> > Do we need to add an 'id' to fpga-bridge too?
> 
> You don't have to, but you could.  See below.
> 
> >
> > Per my understanding this FME driver should create fpga-region for each
> > accelerator, and make sure each fpga-region has correct region_id. When
> > this fpga-region wants to be programmed by fpga-mgr, then it add this
> > info to fpga_image_info, and fpga-mgr knows which region to program via
> > this fpga_image_info.
> 
> The fpga_image_info is passed both to fpga-mgr and fpga-bridge.  So
> both will get the region_id through the info.
> 
> >
> > For each fpga-region, FME driver needs to create one fpga-bridge and link
> > it to region's bridge_list. When fpga_region_program_fpga is invoked for
> > PR. This fpga-bridge could be disabled before PR and re-enabled after PR
> > automatically. If fpga-bridge contains this 'id' information, then driver
> > knows which port to enable/disable to implement of the enable_set function
> > under this fpga-bridge.
> 
> Could implement it either way: the bridge could either use the
> bridge_id that has been passed to it in the info or code that creates
> the bridge could create private data and pass it to
> fpga_bridge_register().  But the bridge will have the id because it
> will be passed in the info, so it's not really needed.

Thanks for the suggestion. :)

The reason I was considering adding id to fpga_bridge is, in the case
driver creates all bridges (and regions/manager) in the initialization
code, but at that time, nobody knows the actual fpga_image_info.

In order to pass fpga_image_info to the bridges of give region during PR
(e.g in fpga_region_program_fpga function), then driver must implement
region->get_bridges function, but I feel we may not need get_bridges
function for this case, as all bridges have already been linked to
fpga-region during initialization.

So I prefer the second method that keep this id in the private data. :)

Thanks
Hao

> 
> Alan
> 
> >
> > Thanks
> > Hao
> >
> >> >
> >> >>
> >> >> If you prefer the region_id for fpga_image_info, we can go with region_id
> >> >> for sure. : )
> >> >>
> >> >> Thanks
> >> >> Hao
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-fpga" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/drivers/fpga/intel/Makefile b/drivers/fpga/intel/Makefile
index 546861d..0452cb6 100644
--- a/drivers/fpga/intel/Makefile
+++ b/drivers/fpga/intel/Makefile
@@ -2,4 +2,4 @@  obj-$(CONFIG_INTEL_FPGA_PCI) += intel-fpga-pci.o
 obj-$(CONFIG_INTEL_FPGA_FME) += intel-fpga-fme.o
 
 intel-fpga-pci-objs := pcie.o feature-dev.o
-intel-fpga-fme-objs := fme-main.o
+intel-fpga-fme-objs := fme-main.o fme-pr.o
diff --git a/drivers/fpga/intel/feature-dev.h b/drivers/fpga/intel/feature-dev.h
index dccc283..5a25c915 100644
--- a/drivers/fpga/intel/feature-dev.h
+++ b/drivers/fpga/intel/feature-dev.h
@@ -150,8 +150,66 @@  struct feature_fme_err {
 };
 
 /* FME Partial Reconfiguration Sub Feature Register Set */
+/* FME PR Control Register */
+struct feature_fme_pr_ctl {
+	union {
+		u64 csr;
+		struct {
+			u8  pr_reset:1;		/* Reset PR Engine */
+			u8  rsvdz1:3;
+			u8  pr_reset_ack:1;	/* Reset PR Engine Ack */
+			u8  rsvdz2:3;
+			u8  pr_regionid:2;	/* PR Region ID */
+			u8  rsvdz3:2;
+			u8  pr_start_req:1;	/* PR Start Request */
+			u8  pr_push_complete:1;	/* PR Data push complete */
+			u8  pr_kind:1;		/* Load Customer or Intel GBS */
+			u32 rsvdz4:17;
+			u32 config_data;
+		};
+	};
+};
+
+/* FME PR Status Register */
+struct feature_fme_pr_status {
+	union {
+		u64 csr;
+		struct {
+			u16 pr_credit:9;	/* Number of PR Credits */
+			u8  rsvdz1:7;
+			u8  pr_status:1;	/* PR Operation status */
+			u8  rsvdz2:3;
+			u8  pr_ctrlr_status:3;	/* Controller status */
+			u8  rsvdz3:1;
+			u8  pr_host_status:4;	/* PR Host status */
+			u64 rsvdz4:36;
+		};
+	};
+};
+
+/* FME PR Data Register */
+struct feature_fme_pr_data {
+	union {
+		u64 csr;
+		struct {
+			/* PR data from the raw-binary file */
+			u32 pr_data_raw;
+			u32 rsvd;
+		};
+	};
+};
+
 struct feature_fme_pr {
 	struct feature_header header;
+	struct feature_fme_pr_ctl control;
+	struct feature_fme_pr_status status;
+	struct feature_fme_pr_data data;
+	u64 error;
+
+	u64 rsvd[16];
+
+	u64 intfc_id_l;		/* PR interface Id Low */
+	u64 intfc_id_h;		/* PR interface Id High */
 };
 
 /* PORT Header Register Set */
diff --git a/drivers/fpga/intel/fme-main.c b/drivers/fpga/intel/fme-main.c
index 36d0c4c..0d9a7a6 100644
--- a/drivers/fpga/intel/fme-main.c
+++ b/drivers/fpga/intel/fme-main.c
@@ -23,6 +23,7 @@ 
 #include <linux/intel-fpga.h>
 
 #include "feature-dev.h"
+#include "fme.h"
 
 static ssize_t ports_num_show(struct device *dev,
 			      struct device_attribute *attr, char *buf)
@@ -101,6 +102,10 @@  static struct feature_driver fme_feature_drvs[] = {
 		.ops = &fme_hdr_ops,
 	},
 	{
+		.name = FME_FEATURE_PR_MGMT,
+		.ops = &pr_mgmt_ops,
+	},
+	{
 		.ops = NULL,
 	},
 };
@@ -182,14 +187,48 @@  static const struct file_operations fme_fops = {
 	.unlocked_ioctl = fme_ioctl,
 };
 
+static int fme_dev_init(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	fme = devm_kzalloc(&pdev->dev, sizeof(*fme), GFP_KERNEL);
+	if (!fme)
+		return -ENOMEM;
+
+	fme->pdata = pdata;
+
+	mutex_lock(&pdata->lock);
+	fpga_pdata_set_private(pdata, fme);
+	mutex_unlock(&pdata->lock);
+	return 0;
+}
+
+static void fme_dev_destroy(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	fpga_pdata_set_private(pdata, NULL);
+	mutex_unlock(&pdata->lock);
+
+	devm_kfree(&pdev->dev, fme);
+}
+
 static int fme_probe(struct platform_device *pdev)
 {
 	int ret;
 
-	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	ret = fme_dev_init(pdev);
 	if (ret)
 		goto exit;
 
+	ret = fpga_dev_feature_init(pdev, fme_feature_drvs);
+	if (ret)
+		goto dev_destroy;
+
 	ret = fpga_register_dev_ops(pdev, &fme_fops, THIS_MODULE);
 	if (ret)
 		goto feature_uinit;
@@ -198,6 +237,8 @@  static int fme_probe(struct platform_device *pdev)
 
 feature_uinit:
 	fpga_dev_feature_uinit(pdev);
+dev_destroy:
+	fme_dev_destroy(pdev);
 exit:
 	return ret;
 }
@@ -206,6 +247,7 @@  static int fme_remove(struct platform_device *pdev)
 {
 	fpga_dev_feature_uinit(pdev);
 	fpga_unregister_dev_ops(pdev);
+	fme_dev_destroy(pdev);
 	return 0;
 }
 
diff --git a/drivers/fpga/intel/fme-pr.c b/drivers/fpga/intel/fme-pr.c
new file mode 100644
index 0000000..3b44a3e
--- /dev/null
+++ b/drivers/fpga/intel/fme-pr.c
@@ -0,0 +1,400 @@ 
+/*
+ * Driver for Intel FPGA Management Engine (FME) Partial Reconfiguration
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <[e-mail]>
+ *   Xiao Guangrong <[e-mail]>
+ *   Joseph Grecco <[e-mail]>
+ *   Enno Luebbers <[e-mail]>
+ *   Tim Whisonant <[e-mail]>
+ *   Ananda Ravuri <[e-mail]>
+ *   Christopher Rauer <[e-mail]>
+ *   Henry Mitchel <[e-mail]>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uaccess.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/intel-fpga.h>
+
+#include "feature-dev.h"
+#include "fme.h"
+
+#define PR_WAIT_TIMEOUT   8000000
+
+#define PR_HOST_STATUS_IDLE	0
+
+DEFINE_FPGA_PR_ERR_MSG(pr_err_msg);
+
+static ssize_t interface_id_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	u64 intfc_id_l, intfc_id_h;
+	struct feature_fme_pr *fme_pr
+		= get_feature_ioaddr_by_index(dev, FME_FEATURE_ID_PR_MGMT);
+
+	intfc_id_l = readq(&fme_pr->intfc_id_l);
+	intfc_id_h = readq(&fme_pr->intfc_id_h);
+
+	return scnprintf(buf, PAGE_SIZE, "%016llx%016llx\n",
+			(unsigned long long)intfc_id_h,
+			(unsigned long long)intfc_id_l);
+}
+static DEVICE_ATTR_RO(interface_id);
+
+static struct attribute *pr_mgmt_attrs[] = {
+	&dev_attr_interface_id.attr,
+	NULL,
+};
+
+struct attribute_group pr_mgmt_attr_group = {
+	.attrs	= pr_mgmt_attrs,
+	.name	= "pr",
+};
+
+static u64
+pr_err_handle(struct platform_device *pdev, struct feature_fme_pr *fme_pr)
+{
+	struct feature_fme_pr_status fme_pr_status;
+	unsigned long err_code;
+	u64 fme_pr_error;
+	int i = 0;
+
+	fme_pr_status.csr = readq(&fme_pr->status);
+	if (!fme_pr_status.pr_status)
+		return 0;
+
+	err_code = fme_pr_error = readq(&fme_pr->error);
+	for_each_set_bit(i, &err_code, PR_MAX_ERR_NUM)
+		dev_dbg(&pdev->dev, "%s\n", pr_err_msg[i]);
+	writeq(fme_pr_error, &fme_pr->error);
+	return fme_pr_error;
+}
+
+static int fme_pr_write_init(struct fpga_manager *mgr,
+		struct fpga_image_info *info, const char *buf, size_t count)
+{
+	struct fpga_fme *fme = mgr->priv;
+	struct platform_device *pdev;
+	struct feature_fme_pr *fme_pr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_status fme_pr_status;
+
+	pdev = fme->pdata->dev;
+	fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+				FME_FEATURE_ID_PR_MGMT);
+	if (!fme_pr)
+		return -EINVAL;
+
+	if (WARN_ON(info->flags != FPGA_MGR_PARTIAL_RECONFIG))
+		return -EINVAL;
+
+	dev_dbg(&pdev->dev, "resetting PR before initiated PR\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_reset = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	fme_pr_ctl.pr_reset_ack = 1;
+
+	if (fpga_wait_register_field(pr_reset_ack, fme_pr_ctl,
+				     &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
+		dev_err(&pdev->dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_reset = 0;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(&pdev->dev,
+		"waiting for PR resource in HW to be initialized and ready\n");
+
+	fme_pr_status.pr_host_status = PR_HOST_STATUS_IDLE;
+
+	if (fpga_wait_register_field(pr_host_status, fme_pr_status,
+				     &fme_pr->status, PR_WAIT_TIMEOUT, 1)) {
+		dev_err(&pdev->dev, "maximum PR timeout\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(&pdev->dev, "check if have any previous PR error\n");
+	pr_err_handle(pdev, fme_pr);
+	return 0;
+}
+
+static int fme_pr_write(struct fpga_manager *mgr,
+			const char *buf, size_t count)
+{
+	struct fpga_fme *fme = mgr->priv;
+	struct platform_device *pdev;
+	struct feature_fme_pr *fme_pr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+	struct feature_fme_pr_status fme_pr_status;
+	struct feature_fme_pr_data fme_pr_data;
+	int delay, pr_credit, i = 0;
+
+	pdev = fme->pdata->dev;
+	fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+				FME_FEATURE_ID_PR_MGMT);
+
+	dev_dbg(&pdev->dev, "set PR port ID and start request\n");
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_regionid = fme->port_id;
+	fme_pr_ctl.pr_start_req = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(&pdev->dev, "pushing data from bitstream to HW\n");
+
+	fme_pr_status.csr = readq(&fme_pr->status);
+	pr_credit = fme_pr_status.pr_credit;
+
+	while (count > 0) {
+		delay = 0;
+		while (pr_credit <= 1) {
+			if (delay++ > PR_WAIT_TIMEOUT) {
+				dev_err(&pdev->dev, "maximum try\n");
+				return -ETIMEDOUT;
+			}
+			udelay(1);
+
+			fme_pr_status.csr = readq(&fme_pr->status);
+			pr_credit = fme_pr_status.pr_credit;
+		};
+
+		if (count >= 4) {
+			fme_pr_data.rsvd = 0;
+			fme_pr_data.pr_data_raw = *(((u32 *)buf) + i);
+			writeq(fme_pr_data.csr, &fme_pr->data);
+			count -= 4;
+			pr_credit--;
+			i++;
+		} else {
+			WARN_ON(1);
+			return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+
+static int fme_pr_write_complete(struct fpga_manager *mgr,
+			struct fpga_image_info *info)
+{
+	struct fpga_fme *fme = mgr->priv;
+	struct platform_device *pdev;
+	struct feature_fme_pr *fme_pr;
+	struct feature_fme_pr_ctl fme_pr_ctl;
+
+	pdev = fme->pdata->dev;
+	fme_pr = get_feature_ioaddr_by_index(&pdev->dev,
+				FME_FEATURE_ID_PR_MGMT);
+
+	fme_pr_ctl.csr = readq(&fme_pr->control);
+	fme_pr_ctl.pr_push_complete = 1;
+	writeq(fme_pr_ctl.csr, &fme_pr->control);
+
+	dev_dbg(&pdev->dev, "green bitstream push complete\n");
+	dev_dbg(&pdev->dev, "waiting for HW to release PR resource\n");
+
+	fme_pr_ctl.pr_start_req = 0;
+
+	if (fpga_wait_register_field(pr_start_req, fme_pr_ctl,
+				     &fme_pr->control, PR_WAIT_TIMEOUT, 1)) {
+		dev_err(&pdev->dev, "maximum try.\n");
+		return -ETIMEDOUT;
+	}
+
+	dev_dbg(&pdev->dev, "PR operation complete, checking status\n");
+	fme->pr_err = pr_err_handle(pdev, fme_pr);
+	if (fme->pr_err)
+		return -EIO;
+
+	dev_dbg(&pdev->dev, "PR done successfully\n");
+	return 0;
+}
+
+static enum fpga_mgr_states fme_pr_state(struct fpga_manager *mgr)
+{
+	return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static const struct fpga_manager_ops fme_pr_ops = {
+	.write_init = fme_pr_write_init,
+	.write = fme_pr_write,
+	.write_complete = fme_pr_write_complete,
+	.state = fme_pr_state,
+};
+
+static int fme_pr(struct platform_device *pdev, unsigned long arg)
+{
+	void __user *argp = (void __user *)arg;
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *fme;
+	struct fpga_manager *mgr;
+	struct feature_fme_header *fme_hdr;
+	struct feature_fme_capability fme_capability;
+	struct fpga_image_info info;
+	struct fpga_fme_port_pr port_pr;
+	struct platform_device *port;
+	unsigned long minsz;
+	void *buf = NULL;
+	int ret = 0;
+
+	minsz = offsetofend(struct fpga_fme_port_pr, status);
+
+	if (copy_from_user(&port_pr, argp, minsz))
+		return -EFAULT;
+
+	if (port_pr.argsz < minsz || port_pr.flags)
+		return -EINVAL;
+
+	if (!IS_ALIGNED(port_pr.buffer_size, 4))
+		return -EINVAL;
+
+	/* get fme header region */
+	fme_hdr = get_feature_ioaddr_by_index(&pdev->dev,
+					FME_FEATURE_ID_HEADER);
+	if (WARN_ON(!fme_hdr))
+		return -EINVAL;
+
+	/* check port id */
+	fme_capability.csr = readq(&fme_hdr->capability);
+	if (port_pr.port_id >= fme_capability.num_ports) {
+		dev_dbg(&pdev->dev, "port number more than maximum\n");
+		return -EINVAL;
+	}
+
+	if (!access_ok(VERIFY_READ, port_pr.buffer_address,
+				    port_pr.buffer_size))
+		return -EFAULT;
+
+	buf = vmalloc(port_pr.buffer_size);
+	if (!buf)
+		return -ENOMEM;
+
+	if (copy_from_user(buf, (void __user *)port_pr.buffer_address,
+					       port_pr.buffer_size)) {
+		ret = -EFAULT;
+		goto free_exit;
+	}
+
+	memset(&info, 0, sizeof(struct fpga_image_info));
+	info.flags = FPGA_MGR_PARTIAL_RECONFIG;
+
+	mgr = fpga_mgr_get(&pdev->dev);
+	if (IS_ERR(mgr)) {
+		ret = PTR_ERR(mgr);
+		goto free_exit;
+	}
+
+	mutex_lock(&pdata->lock);
+	fme = fpga_pdata_get_private(pdata);
+	/* fme device has been unregistered. */
+	if (!fme) {
+		ret = -EINVAL;
+		goto unlock_exit;
+	}
+
+	fme->pr_err = 0;
+	fme->port_id = port_pr.port_id;
+
+	/* Find and get port device by index */
+	port = pdata->fpga_for_each_port(pdev, &fme->port_id,
+					 fpga_port_check_id);
+	WARN_ON(!port);
+
+	/* Disable Port before PR */
+	fpga_port_disable(port);
+
+	ret = fpga_mgr_buf_load(mgr, &info, buf, port_pr.buffer_size);
+	port_pr.status = fme->pr_err;
+
+	/* Re-enable Port after PR finished */
+	fpga_port_enable(port);
+
+	put_device(&port->dev);
+
+unlock_exit:
+	mutex_unlock(&pdata->lock);
+	fpga_mgr_put(mgr);
+free_exit:
+	vfree(buf);
+	if (copy_to_user((void __user *)arg, &port_pr, minsz))
+		return -EFAULT;
+	return ret;
+}
+
+static int fpga_fme_pr_probe(struct platform_device *pdev)
+{
+	struct feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
+	struct fpga_fme *priv;
+	int ret;
+
+	mutex_lock(&pdata->lock);
+	priv = fpga_pdata_get_private(pdata);
+	ret = fpga_mgr_register(&pdata->dev->dev,
+		"Intel FPGA Manager", &fme_pr_ops, priv);
+	mutex_unlock(&pdata->lock);
+
+	return ret;
+}
+
+static int fpga_fme_pr_remove(struct platform_device *pdev)
+{
+	fpga_mgr_unregister(&pdev->dev);
+	return 0;
+}
+
+static int pr_mgmt_init(struct platform_device *pdev, struct feature *feature)
+{
+	int ret;
+
+	ret = fpga_fme_pr_probe(pdev);
+	if (ret)
+		return ret;
+
+	ret = sysfs_create_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
+	if (ret)
+		fpga_fme_pr_remove(pdev);
+
+	return ret;
+}
+
+static void pr_mgmt_uinit(struct platform_device *pdev, struct feature *feature)
+{
+	sysfs_remove_group(&pdev->dev.kobj, &pr_mgmt_attr_group);
+	fpga_fme_pr_remove(pdev);
+}
+
+static long fme_pr_ioctl(struct platform_device *pdev, struct feature *feature,
+	unsigned int cmd, unsigned long arg)
+{
+	long ret;
+
+	switch (cmd) {
+	case FPGA_FME_PORT_PR:
+		ret = fme_pr(pdev, arg);
+		break;
+	default:
+		ret = -ENODEV;
+	}
+
+	return ret;
+}
+
+struct feature_ops pr_mgmt_ops = {
+	.init = pr_mgmt_init,
+	.uinit = pr_mgmt_uinit,
+	.ioctl = fme_pr_ioctl,
+};
diff --git a/drivers/fpga/intel/fme.h b/drivers/fpga/intel/fme.h
new file mode 100644
index 0000000..d6cb7ce
--- /dev/null
+++ b/drivers/fpga/intel/fme.h
@@ -0,0 +1,32 @@ 
+/*
+ * Header file for Intel FPGA Management Engine (FME) Driver
+ *
+ * Copyright (C) 2017 Intel Corporation, Inc.
+ *
+ * Authors:
+ *   Kang Luwei <[e-mail]>
+ *   Xiao Guangrong <[e-mail]>
+ *   Joseph Grecco <[e-mail]>
+ *   Enno Luebbers <[e-mail]>
+ *   Tim Whisonant <[e-mail]>
+ *   Ananda Ravuri <[e-mail]>
+ *   Henry Mitchel <[e-mail]>
+ *
+ * This work is licensed under a dual BSD/GPLv2 license. When using or
+ * redistributing this file, you may do so under either license. See the
+ * LICENSE.BSD file under this directory for the BSD license and see
+ * the COPYING file in the top-level directory for the GPLv2 license.
+ */
+
+#ifndef __INTEL_FME_H
+#define __INTEL_FME_H
+
+struct fpga_fme {
+	u8  port_id;
+	u64 pr_err;
+	struct feature_platform_data *pdata;
+};
+
+extern struct feature_ops pr_mgmt_ops;
+
+#endif
diff --git a/include/uapi/linux/intel-fpga.h b/include/uapi/linux/intel-fpga.h
index 992e556..77658316 100644
--- a/include/uapi/linux/intel-fpga.h
+++ b/include/uapi/linux/intel-fpga.h
@@ -18,6 +18,8 @@ 
 #ifndef _UAPI_LINUX_INTEL_FPGA_H
 #define _UAPI_LINUX_INTEL_FPGA_H
 
+#include <linux/types.h>
+
 #define FPGA_API_VERSION 0
 
 /*
@@ -30,6 +32,7 @@ 
 #define FPGA_MAGIC 0xB6
 
 #define FPGA_BASE 0
+#define FME_BASE 0x80
 
 /**
  * FPGA_GET_API_VERSION - _IO(FPGA_MAGIC, FPGA_BASE + 0)
@@ -49,4 +52,45 @@ 
 
 #define FPGA_CHECK_EXTENSION	_IO(FPGA_MAGIC, FPGA_BASE + 1)
 
+/* IOCTLs for FME file descriptor */
+
+/**
+ * FPGA_FME_PORT_PR - _IOWR(FPGA_MAGIC, FME_BASE + 0, struct fpga_fme_port_pr)
+ *
+ * Driver does Partial Reconfiguration based on Port ID and Buffer (Image)
+ * provided by caller.
+ * Return: 0 on success, -errno on failure.
+ * If FPGA_FME_PORT_PR returns -EIO, that indicates the HW has detected
+ * some errors during PR, under this case, the user can fetch HW error code
+ * from fpga_fme_port_pr.status. Each bit on the error code is used as the
+ * index for the array created by DEFINE_FPGA_PR_ERR_MSG().
+ * Otherwise, it is always zero.
+ */
+
+#define DEFINE_FPGA_PR_ERR_MSG(_name_)			\
+static const char * const _name_[] = {			\
+	"PR operation error detected",			\
+	"PR CRC error detected",			\
+	"PR incompatiable bitstream error detected",	\
+	"PR IP protocol error detected",		\
+	"PR FIFO overflow error detected",		\
+	"Reserved",					\
+	"PR secure load error detected",		\
+}
+
+#define PR_MAX_ERR_NUM	7
+
+struct fpga_fme_port_pr {
+	/* Input */
+	__u32 argsz;		/* Structure length */
+	__u32 flags;		/* Zero for now */
+	__u32 port_id;
+	__u32 buffer_size;
+	__u64 buffer_address;	/* Userspace address to the buffer for PR */
+	/* Output */
+	__u64 status;		/* HW error code if ioctl returns -EIO */
+};
+
+#define FPGA_FME_PORT_PR	_IO(FPGA_MAGIC, FME_BASE + 0)
+
 #endif /* _UAPI_INTEL_FPGA_H */