diff mbox

[v14,3/3] iommu: add rmrr Xen command line option for extra rmrrs

Message ID 1485195639-1996-4-git-send-email-venu.busireddy@oracle.com (mailing list archive)
State New, archived
Headers show

Commit Message

Venu Busireddy Jan. 23, 2017, 6:20 p.m. UTC
On some platforms firmware fails to specify RMRR regions in ACPI tables and
thus those regions will not be mapped in dom0 or guests and may cause IO
Page Faults and prevent dom0 from booting if "iommu=dom0-strict" option is
specified on the Xen command line.

New Xen command line option rmrr allows to specify such devices and
memory regions. These regions are added to the list of RMRR defined in ACPI
if the device is present in system. As a result, additional RMRRs will be
mapped 1:1 in dom0 with correct permissions.

The above mentioned problems were discovered during the PVH work with
ThinkCentre M and Dell 5600T. No official documentation was found so far
in regards to what devices and why cause this. Experiments show that
ThinkCentre M USB devices with enabled debug port generate DMA read
transactions to the regions of memory marked reserved in host e820 map.

For Dell 5600T the device and faulting addresses are not found yet.
For detailed history of the discussion please check following threads:
    http://lists.Xen.org/archives/html/xen-devel/2015-02/msg01724.html
    http://lists.Xen.org/archives/html/xen-devel/2015-01/msg02513.html

Format for rmrr Xen command line option:
    rmrr=start<-end>=[s1]bdf1[,[s1]bdf2[,...]];start<-end>=[s2]bdf1[,[s2]bdf2[,...]]
    For example, for Lenovo ThinkCentre M, use:
        rmrr=0xd5d45=0:0:1d.0;0xd5d46=0:0:1a.0
    If grub2 used and multiple ranges are specified, ';' should be
    quoted/escaped, refer to grub2 manual for more information.

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: Venu Busireddy <venu.busireddy@oracle.com>
Acked-by: Kevin Tian <kevin.tian@intel.com>
---
 docs/misc/xen-command-line.markdown |  13 +++
 xen/drivers/passthrough/vtd/dmar.c  | 203 +++++++++++++++++++++++++++++++++++-
 2 files changed, 215 insertions(+), 1 deletion(-)

Changes in v14:
 - Implement Jan's feedback.
   Rename some #defines and take care of boundary conditions.

Comments

Jan Beulich Jan. 24, 2017, 8:46 a.m. UTC | #1
>>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
> +        overlap = false;
> +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
> +        {
> +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
> +                 rmrru->base_address <= pfn_to_paddr(end) )

So this now looks correct as long as rmrru->base_address is
page aligned (as required by the spec), which should be good
enough for now (considering that we make this assumption
elsewhere). Nevertheless it would have been nice if you had,
following the subsequent discussion with Elena, accounted for
spec violations here.

> +        rmrr->segment = seg;
> +        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
> +        /* Align the end_address to the end of the page */
> +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) | ~PAGE_MASK_4K;

Hmm, Ive just checked - in my reply to Elena I had intentionally used
PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
uses PAGE_SHIFT?

With this corrected (which can be done upon commit, but I'd first
like to understand your reasoning):
Reviewed-by: Jan Beulich <jbeulich@suse.com>

And despite Kevin's ack being present here, in light of what I've
said on the other patch - Kevin, please confirm your ack.

Jan
Venu Busireddy Jan. 24, 2017, 3:35 p.m. UTC | #2
On Tue, Jan 24, 2017 at 01:46:44AM -0700, Jan Beulich wrote:
> >>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
> > +        overlap = false;
> > +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
> > +        {
> > +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
> > +                 rmrru->base_address <= pfn_to_paddr(end) )
> 
> So this now looks correct as long as rmrru->base_address is
> page aligned (as required by the spec), which should be good
> enough for now (considering that we make this assumption
> elsewhere). Nevertheless it would have been nice if you had,
> following the subsequent discussion with Elena, accounted for
> spec violations here.
> 
> > +        rmrr->segment = seg;
> > +        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
> > +        /* Align the end_address to the end of the page */
> > +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) | ~PAGE_MASK_4K;
> 
> Hmm, Ive just checked - in my reply to Elena I had intentionally used
> PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
> What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
> uses PAGE_SHIFT?

Elena suggested to use PAGE_MASK_4K because the functions in
drivers/passthrough/vtd/iommu.c (including rmrr_identity_mapping())
use the _4K. With the current assumptions, both will work.

If you would like me to change this to PAGE_MASK, I will do so before
committing. Please let me know.

> With this corrected (which can be done upon commit, but I'd first
> like to understand your reasoning):
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
> 
> And despite Kevin's ack being present here, in light of what I've
> said on the other patch - Kevin, please confirm your ack.
> 
> Jan
> 

Regards,

Venu
Jan Beulich Jan. 24, 2017, 3:52 p.m. UTC | #3
>>> On 24.01.17 at 16:35, <venu.busireddy@oracle.com> wrote:
> On Tue, Jan 24, 2017 at 01:46:44AM -0700, Jan Beulich wrote:
>> >>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
>> > +        overlap = false;
>> > +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
>> > +        {
>> > +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
>> > +                 rmrru->base_address <= pfn_to_paddr(end) )
>> 
>> So this now looks correct as long as rmrru->base_address is
>> page aligned (as required by the spec), which should be good
>> enough for now (considering that we make this assumption
>> elsewhere). Nevertheless it would have been nice if you had,
>> following the subsequent discussion with Elena, accounted for
>> spec violations here.
>> 
>> > +        rmrr->segment = seg;
>> > +        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
>> > +        /* Align the end_address to the end of the page */
>> > +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) | 
> ~PAGE_MASK_4K;
>> 
>> Hmm, Ive just checked - in my reply to Elena I had intentionally used
>> PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
>> What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
>> uses PAGE_SHIFT?
> 
> Elena suggested to use PAGE_MASK_4K because the functions in
> drivers/passthrough/vtd/iommu.c (including rmrr_identity_mapping())
> use the _4K. With the current assumptions, both will work.

Granted this is somewhat of a mess, but I'd prefer if at least within
a single statement things would be consistent in which page size is
being meant.

> If you would like me to change this to PAGE_MASK, I will do so before
> committing. Please let me know.

As said, I don't see a need for you to re-submit, unless there are
other issues in need of taking care of.

Jan
Venu Busireddy Jan. 24, 2017, 3:58 p.m. UTC | #4
On Tue, Jan 24, 2017 at 08:52:50AM -0700, Jan Beulich wrote:
> >>> On 24.01.17 at 16:35, <venu.busireddy@oracle.com> wrote:
> > On Tue, Jan 24, 2017 at 01:46:44AM -0700, Jan Beulich wrote:
> >> >>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
> >> > +        overlap = false;
> >> > +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
> >> > +        {
> >> > +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
> >> > +                 rmrru->base_address <= pfn_to_paddr(end) )
> >> 
> >> So this now looks correct as long as rmrru->base_address is
> >> page aligned (as required by the spec), which should be good
> >> enough for now (considering that we make this assumption
> >> elsewhere). Nevertheless it would have been nice if you had,
> >> following the subsequent discussion with Elena, accounted for
> >> spec violations here.
> >> 
> >> > +        rmrr->segment = seg;
> >> > +        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
> >> > +        /* Align the end_address to the end of the page */
> >> > +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) | 
> > ~PAGE_MASK_4K;
> >> 
> >> Hmm, Ive just checked - in my reply to Elena I had intentionally used
> >> PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
> >> What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
> >> uses PAGE_SHIFT?
> > 
> > Elena suggested to use PAGE_MASK_4K because the functions in
> > drivers/passthrough/vtd/iommu.c (including rmrr_identity_mapping())
> > use the _4K. With the current assumptions, both will work.
> 
> Granted this is somewhat of a mess, but I'd prefer if at least within
> a single statement things would be consistent in which page size is
> being meant.
> 
> > If you would like me to change this to PAGE_MASK, I will do so before
> > committing. Please let me know.
> 
> As said, I don't see a need for you to re-submit, unless there are
> other issues in need of taking care of.

Sure. I will change it to PAGE_MASK, but I will not resubmit the
patches. I will change it and commit it. Of course, assuming that there
are no other changes suggested by others.

Do I understand that I have your Ack?

> 
> Jan
> 

Venu
Jan Beulich Jan. 24, 2017, 4:08 p.m. UTC | #5
>>> On 24.01.17 at 16:58, <venu.busireddy@oracle.com> wrote:
> On Tue, Jan 24, 2017 at 08:52:50AM -0700, Jan Beulich wrote:
>> >>> On 24.01.17 at 16:35, <venu.busireddy@oracle.com> wrote:
>> > On Tue, Jan 24, 2017 at 01:46:44AM -0700, Jan Beulich wrote:
>> >> >>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
>> >> > +        overlap = false;
>> >> > +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
>> >> > +        {
>> >> > +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
>> >> > +                 rmrru->base_address <= pfn_to_paddr(end) )
>> >> 
>> >> So this now looks correct as long as rmrru->base_address is
>> >> page aligned (as required by the spec), which should be good
>> >> enough for now (considering that we make this assumption
>> >> elsewhere). Nevertheless it would have been nice if you had,
>> >> following the subsequent discussion with Elena, accounted for
>> >> spec violations here.
>> >> 
>> >> > +        rmrr->segment = seg;
>> >> > +        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
>> >> > +        /* Align the end_address to the end of the page */
>> >> > +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) | 
>> > ~PAGE_MASK_4K;
>> >> 
>> >> Hmm, Ive just checked - in my reply to Elena I had intentionally used
>> >> PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
>> >> What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
>> >> uses PAGE_SHIFT?
>> > 
>> > Elena suggested to use PAGE_MASK_4K because the functions in
>> > drivers/passthrough/vtd/iommu.c (including rmrr_identity_mapping())
>> > use the _4K. With the current assumptions, both will work.
>> 
>> Granted this is somewhat of a mess, but I'd prefer if at least within
>> a single statement things would be consistent in which page size is
>> being meant.
>> 
>> > If you would like me to change this to PAGE_MASK, I will do so before
>> > committing. Please let me know.
>> 
>> As said, I don't see a need for you to re-submit, unless there are
>> other issues in need of taking care of.
> 
> Sure. I will change it to PAGE_MASK, but I will not resubmit the
> patches. I will change it and commit it. Of course, assuming that there
> are no other changes suggested by others.

I don't follow this repeated mentioning of "commit" by you. I don't
think you can commit to our staging tree. And actually committing
the series depends on - as also stated before - Kevin confirming
his acks, which you had wrongly left in place.

> Do I understand that I have your Ack?

Well, I had given you my (conditional) R-b in the (my) morning.

Jan
Venu Busireddy Jan. 24, 2017, 8:30 p.m. UTC | #6
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, January 24, 2017 10:08 AM
> To: Venu Busireddy
> Cc: Feng Wu; Kevin Tian; xen-devel@lists.xen.org; Elena Ufimtseva; Konrad
> Rzeszutek Wilk
> Subject: Re: [PATCH v14 3/3] iommu: add rmrr Xen command line option for
> extra rmrrs
> 
> >>> On 24.01.17 at 16:58, <venu.busireddy@oracle.com> wrote:
> > On Tue, Jan 24, 2017 at 08:52:50AM -0700, Jan Beulich wrote:
> >> >>> On 24.01.17 at 16:35, <venu.busireddy@oracle.com> wrote:
> >> > On Tue, Jan 24, 2017 at 01:46:44AM -0700, Jan Beulich wrote:
> >> >> >>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
> >> >> > +        overlap = false;
> >> >> > +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
> >> >> > +        {
> >> >> > +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
> >> >> > +                 rmrru->base_address <= pfn_to_paddr(end) )
> >> >>
> >> >> So this now looks correct as long as rmrru->base_address is
> >> >> page aligned (as required by the spec), which should be good
> >> >> enough for now (considering that we make this assumption
> >> >> elsewhere). Nevertheless it would have been nice if you had,
> >> >> following the subsequent discussion with Elena, accounted for
> >> >> spec violations here.
> >> >>
> >> >> > +        rmrr->segment = seg;
> >> >> > +        rmrr->base_address =
> pfn_to_paddr(user_rmrrs[i].base_pfn);
> >> >> > +        /* Align the end_address to the end of the page */
> >> >> > +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) |
> >> > ~PAGE_MASK_4K;
> >> >>
> >> >> Hmm, Ive just checked - in my reply to Elena I had intentionally
> used
> >> >> PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
> >> >> What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
> >> >> uses PAGE_SHIFT?
> >> >
> >> > Elena suggested to use PAGE_MASK_4K because the functions in
> >> > drivers/passthrough/vtd/iommu.c (including rmrr_identity_mapping())
> >> > use the _4K. With the current assumptions, both will work.
> >>
> >> Granted this is somewhat of a mess, but I'd prefer if at least within
> >> a single statement things would be consistent in which page size is
> >> being meant.
> >>
> >> > If you would like me to change this to PAGE_MASK, I will do so before
> >> > committing. Please let me know.
> >>
> >> As said, I don't see a need for you to re-submit, unless there are
> >> other issues in need of taking care of.
> >
> > Sure. I will change it to PAGE_MASK, but I will not resubmit the
> > patches. I will change it and commit it. Of course, assuming that there
> > are no other changes suggested by others.
> 
> I don't follow this repeated mentioning of "commit" by you. I don't
> think you can commit to our staging tree. And actually committing
> the series depends on - as also stated before - Kevin confirming
> his acks, which you had wrongly left in place.

Sorry Jan! I was confused. I thought that once I have the Ack's from the Maintainers, I can push the changes myself. Hence I was saying commit - meaning, commit to my local tree, with the intent to push them upstream once Ack'ed. I am told that I cannot push the changes.

Therefore, once we have Kevin's Ack, please change PAGE_MASK_4K to PAGE_MASK and commit.

Thanks,

Venu

> 
> > Do I understand that I have your Ack?
> 
> Well, I had given you my (conditional) R-b in the (my) morning.
> 
> Jan
>
Tian, Kevin Jan. 25, 2017, 2:28 a.m. UTC | #7
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, January 24, 2017 4:47 PM
> 
> >>> On 23.01.17 at 19:20, <venu.busireddy@oracle.com> wrote:
> > +        overlap = false;
> > +        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
> > +        {
> > +            if ( pfn_to_paddr(base) <= rmrru->end_address &&
> > +                 rmrru->base_address <= pfn_to_paddr(end) )
> 
> So this now looks correct as long as rmrru->base_address is
> page aligned (as required by the spec), which should be good
> enough for now (considering that we make this assumption
> elsewhere). Nevertheless it would have been nice if you had,
> following the subsequent discussion with Elena, accounted for
> spec violations here.
> 
> > +        rmrr->segment = seg;
> > +        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
> > +        /* Align the end_address to the end of the page */
> > +        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) |
> ~PAGE_MASK_4K;
> 
> Hmm, Ive just checked - in my reply to Elena I had intentionally used
> PAGE_MASK here (and I recall correcting it from PAGE_MASK_4K).
> What has led you to use PAGE_MASK_4K here, when pfn_to_paddr()
> uses PAGE_SHIFT?
> 
> With this corrected (which can be done upon commit, but I'd first
> like to understand your reasoning):
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
> 
> And despite Kevin's ack being present here, in light of what I've
> said on the other patch - Kevin, please confirm your ack.
> 

Sure, here is my confirm:

Acked-by: Kevin Tian <kevin.tian@intel.com>

btw good to catch it right before my vacation to not block check-in. :-)

Thanks
Kevin
diff mbox

Patch

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 0138978..a11fdf9 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1377,6 +1377,19 @@  Specify the host reboot method.
 'efi' instructs Xen to reboot using the EFI reboot call (in EFI mode by
  default it will use that method first).
 
+### rmrr
+> '= start<-end>=[s1]bdf1[,[s1]bdf2[,...]];start<-end>=[s2]bdf1[,[s2]bdf2[,...]]
+
+Define RMRR units that are missing from ACPI table along with device they
+belong to and use them for 1:1 mapping. End addresses can be omitted and one
+page will be mapped. The ranges are inclusive when start and end are specified.
+If segment of the first device is not specified, segment zero will be used.
+If other segments are not specified, first device segment will be used.
+If a segment is specified for other than the first device and it does not match
+the one specified for the first one, an error will be reported.
+Note: grub2 requires to escape or use quotations if special characters are used,
+namely ';', refer to the grub2 documentation if multiple ranges are specified.
+
 ### ro-hpet
 > `= <boolean>`
 
diff --git a/xen/drivers/passthrough/vtd/dmar.c b/xen/drivers/passthrough/vtd/dmar.c
index 3c7c9b2..7c12b17 100644
--- a/xen/drivers/passthrough/vtd/dmar.c
+++ b/xen/drivers/passthrough/vtd/dmar.c
@@ -859,6 +859,134 @@  out:
     return ret;
 }
 
+#define MAX_USER_RMRR_PAGES 16
+#define MAX_USER_RMRR 10
+
+/* RMRR units derived from command line rmrr option. */
+#define MAX_USER_RMRR_DEV 20
+struct user_rmrr {
+    struct list_head list;
+    unsigned long base_pfn, end_pfn;
+    unsigned int dev_count;
+    u32 sbdf[MAX_USER_RMRR_DEV];
+};
+
+static __initdata unsigned int nr_rmrr;
+static struct __initdata user_rmrr user_rmrrs[MAX_USER_RMRR];
+
+/* Macro for RMRR inclusive range formatting. */
+#define ERMRRU_FMT "[%lx-%lx]"
+#define ERMRRU_ARG(eru) eru.base_pfn, eru.end_pfn
+
+static int __init add_user_rmrr(void)
+{
+    struct acpi_rmrr_unit *rmrr, *rmrru;
+    unsigned int idx, seg, i;
+    unsigned long base, end;
+    bool overlap;
+
+    for ( i = 0; i < nr_rmrr; i++ )
+    {
+        base = user_rmrrs[i].base_pfn;
+        end = user_rmrrs[i].end_pfn;
+
+        if ( base > end )
+        {
+            printk(XENLOG_ERR VTDPREFIX
+                   "Invalid RMRR Range "ERMRRU_FMT"\n",
+                   ERMRRU_ARG(user_rmrrs[i]));
+            continue;
+        }
+
+        if ( (end - base) >= MAX_USER_RMRR_PAGES )
+        {
+            printk(XENLOG_ERR VTDPREFIX
+                   "RMRR range "ERMRRU_FMT" exceeds "\
+                   __stringify(MAX_USER_RMRR_PAGES)" pages\n",
+                   ERMRRU_ARG(user_rmrrs[i]));
+            continue;
+        }
+
+        overlap = false;
+        list_for_each_entry(rmrru, &acpi_rmrr_units, list)
+        {
+            if ( pfn_to_paddr(base) <= rmrru->end_address &&
+                 rmrru->base_address <= pfn_to_paddr(end) )
+            {
+                printk(XENLOG_ERR VTDPREFIX
+                       "Overlapping RMRRs: "ERMRRU_FMT" and [%lx-%lx]\n",
+                       ERMRRU_ARG(user_rmrrs[i]),
+                       paddr_to_pfn(rmrru->base_address),
+                       paddr_to_pfn(rmrru->end_address));
+                overlap = true;
+                break;
+            }
+        }
+        /* Don't add overlapping RMRR. */
+        if ( overlap )
+            continue;
+
+        do
+        {
+            if ( !mfn_valid(base) )
+            {
+                printk(XENLOG_ERR VTDPREFIX
+                       "Invalid pfn in RMRR range "ERMRRU_FMT"\n",
+                       ERMRRU_ARG(user_rmrrs[i]));
+                break;
+            }
+        } while ( base++ < end );
+
+        /* Invalid pfn in range as the loop ended before end_pfn was reached. */
+        if ( base <= end )
+            continue;
+
+        rmrr = xzalloc(struct acpi_rmrr_unit);
+        if ( !rmrr )
+            return -ENOMEM;
+
+        rmrr->scope.devices = xmalloc_array(u16, user_rmrrs[i].dev_count);
+        if ( !rmrr->scope.devices )
+        {
+            xfree(rmrr);
+            return -ENOMEM;
+        }
+
+        seg = 0;
+        for ( idx = 0; idx < user_rmrrs[i].dev_count; idx++ )
+        {
+            rmrr->scope.devices[idx] = user_rmrrs[i].sbdf[idx];
+            seg |= PCI_SEG(user_rmrrs[i].sbdf[idx]);
+        }
+        if ( seg != PCI_SEG(user_rmrrs[i].sbdf[0]) )
+        {
+            printk(XENLOG_ERR VTDPREFIX
+                   "Segments are not equal for RMRR range "ERMRRU_FMT"\n",
+                   ERMRRU_ARG(user_rmrrs[i]));
+            scope_devices_free(&rmrr->scope);
+            xfree(rmrr);
+            continue;
+        }
+
+        rmrr->segment = seg;
+        rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn);
+        /* Align the end_address to the end of the page */
+        rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn) | ~PAGE_MASK_4K;
+        rmrr->scope.devices_cnt = user_rmrrs[i].dev_count;
+
+        if ( register_one_rmrr(rmrr) )
+        {
+            printk(XENLOG_ERR VTDPREFIX
+                   "Could not register RMMR range "ERMRRU_FMT"\n",
+                   ERMRRU_ARG(user_rmrrs[i]));
+            scope_devices_free(&rmrr->scope);
+            xfree(rmrr);
+        }
+    }
+
+    return 0;
+}
+
 #include <asm/tboot.h>
 /* ACPI tables may not be DMA protected by tboot, so use DMAR copy */
 /* SINIT saved in SinitMleData in TXT heap (which is DMA protected) */
@@ -868,6 +996,7 @@  int __init acpi_dmar_init(void)
 {
     acpi_physical_address dmar_addr;
     acpi_native_uint dmar_len;
+    int ret;
 
     if ( ACPI_SUCCESS(acpi_get_table_phys(ACPI_SIG_DMAR, 0,
                                           &dmar_addr, &dmar_len)) )
@@ -878,7 +1007,12 @@  int __init acpi_dmar_init(void)
         dmar_table = __va(dmar_addr);
     }
 
-    return parse_dmar_table(acpi_parse_dmar);
+    ret = parse_dmar_table(acpi_parse_dmar);
+
+    if ( !ret )
+        return add_user_rmrr();
+
+    return ret;
 }
 
 void acpi_dmar_reinstate(void)
@@ -937,3 +1071,70 @@  int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
 
     return 0;
 }
+
+/*
+ * Parse rmrr Xen command line options and add parsed devices and regions into
+ * acpi_rmrr_unit list to mapped as RMRRs parsed from ACPI.
+ * Format:
+ * rmrr=start<-end>=[s1]bdf1[,[s1]bdf2[,...]];start<-end>=[s2]bdf1[,[s2]bdf2[,...]]
+ * If the segment of the first device is not specified,
+ * segment zero will be used.
+ * If other segments are not specified, first device segment will be used.
+ * If a segment is specified for other than the first device, and it does not
+ * match the one specified for the first one, an error will be reported.
+ */
+static void __init parse_rmrr_param(const char *str)
+{
+    const char *s = str, *cur, *stmp;
+    unsigned int seg, bus, dev, func, dev_count;
+    unsigned long start, end;
+
+    do {
+        start = simple_strtoul(cur = s, &s, 0);
+        if ( cur == s )
+            break;
+
+        if ( *s == '-' )
+        {
+            end = simple_strtoul(cur = s + 1, &s, 0);
+            if ( cur == s )
+                break;
+        }
+        else
+            end = start;
+
+        user_rmrrs[nr_rmrr].base_pfn = start;
+        user_rmrrs[nr_rmrr].end_pfn = end;
+
+        if ( *s != '=' )
+            continue;
+
+        do {
+            bool def_seg = false;
+
+            stmp = parse_pci_seg(s + 1, &seg, &bus, &dev, &func, &def_seg);
+            if ( !stmp )
+                break;
+
+            /*
+             * Not specified.
+             * Segment will be replaced with one from first device.
+             */
+            if ( user_rmrrs[nr_rmrr].dev_count && def_seg )
+                seg = PCI_SEG(user_rmrrs[nr_rmrr].sbdf[0]);
+
+            /* Keep sbdf's even if they differ and later report an error. */
+            dev_count = user_rmrrs[nr_rmrr].dev_count;
+            user_rmrrs[nr_rmrr].sbdf[dev_count] = PCI_SBDF(seg, bus, dev, func);
+
+            user_rmrrs[nr_rmrr].dev_count++;
+            s = stmp;
+        } while ( *s == ',' &&
+                  user_rmrrs[nr_rmrr].dev_count < MAX_USER_RMRR_DEV );
+
+        if ( user_rmrrs[nr_rmrr].dev_count )
+            nr_rmrr++;
+
+    } while ( *s++ == ';' && nr_rmrr < MAX_USER_RMRR );
+}
+custom_param("rmrr", parse_rmrr_param);