* Userspace pages in UC mode
@ 2015-09-11 14:07 Sabela Ramos Garea
2015-09-11 14:59 ` Sabela Ramos Garea
0 siblings, 1 reply; 5+ messages in thread
From: Sabela Ramos Garea @ 2015-09-11 14:07 UTC (permalink / raw)
To: kernelnewbies
Dear all,
For research purposes I need some userspace memory pages to be in
uncacheable mode. I am using two different Intel architectures (Sandy
Bridge and Haswell) and two different kernels (2.6.32-358 and
3.19.0-28).
The non-temporal stores from Intel assembly are not a valid solution
so I am programming a kernel module that gets a set of pages from user
space reserved with posix_memalign (get_user_pages) and then sets them
as uncacheable (I have tried set_pages_uc and set_pages_array_uc).
When I use one page, the access times are not very coherent and with
more than one page the module crashes (in both architectures and both
kernels).
I wonder if I am using the correct approach or if I have to use kernel
space pages in order to work with uncacheable memory. Or if I have to
remap the memory. Just in case it makes it clearer, I am attaching the
relevant lines of a kernel module function that should set the pages
as uncacheable. (This function is the .write of a misc device; count
is treated as the number of pages).
Best and Thanks,
Sabela.
struct page *pages; //defined outside in order to be able to set them
to WB in the release function.
int numpages;
static ssize_t setup_memory(struct file *filp, const char __user *buf,
size_t count, loff_t * ppos)
{
int res;
struct vm_area_struct *vmas;
numpages = count/4096;
down_read(¤t->mm->mmap_sem);
res = get_user_pages(current, current->mm,
(unsigned long) buf,
numpages, /* Number of pages */
0, /* Do want to write into it */
1, /* do force */
&pages,
&vmas);
up_read(¤t->mm->mmap_sem);
numpages=res;
if (res > 0) {
set_pages_uc(pages, numpages); /* Uncached */
printk("Write: %d pages set as uncacheable\n",numpages);
}
else{
pr_err("Couldn't get pages to set them as UC :(\n");
return -EAGAIN;
}
}
^ permalink raw reply [flat|nested] 5+ messages in thread
* Userspace pages in UC mode
2015-09-11 14:07 Userspace pages in UC mode Sabela Ramos Garea
@ 2015-09-11 14:59 ` Sabela Ramos Garea
2015-09-11 18:10 ` David Matlack
2015-09-12 1:12 ` Pranay Srivastava
0 siblings, 2 replies; 5+ messages in thread
From: Sabela Ramos Garea @ 2015-09-11 14:59 UTC (permalink / raw)
To: kernelnewbies
Sorry, little mistake copypasting and cleaning. The pages and vma
structs should look like that:
struct page *pages --> struct page *pages[MAX_PAGES];
struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
Where MAX_PAGES is defined to 5.
Sabela.
2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
> Dear all,
>
> For research purposes I need some userspace memory pages to be in
> uncacheable mode. I am using two different Intel architectures (Sandy
> Bridge and Haswell) and two different kernels (2.6.32-358 and
> 3.19.0-28).
>
> The non-temporal stores from Intel assembly are not a valid solution
> so I am programming a kernel module that gets a set of pages from user
> space reserved with posix_memalign (get_user_pages) and then sets them
> as uncacheable (I have tried set_pages_uc and set_pages_array_uc).
> When I use one page, the access times are not very coherent and with
> more than one page the module crashes (in both architectures and both
> kernels).
>
> I wonder if I am using the correct approach or if I have to use kernel
> space pages in order to work with uncacheable memory. Or if I have to
> remap the memory. Just in case it makes it clearer, I am attaching the
> relevant lines of a kernel module function that should set the pages
> as uncacheable. (This function is the .write of a misc device; count
> is treated as the number of pages).
>
> Best and Thanks,
>
> Sabela.
>
> struct page *pages; //defined outside in order to be able to set them
> to WB in the release function.
> int numpages;
>
> static ssize_t setup_memory(struct file *filp, const char __user *buf,
> size_t count, loff_t * ppos)
> {
> int res;
> struct vm_area_struct *vmas;
>
> numpages = count/4096;
>
> down_read(¤t->mm->mmap_sem);
> res = get_user_pages(current, current->mm,
> (unsigned long) buf,
> numpages, /* Number of pages */
> 0, /* Do want to write into it */
> 1, /* do force */
> &pages,
> &vmas);
> up_read(¤t->mm->mmap_sem);
>
> numpages=res;
>
> if (res > 0) {
> set_pages_uc(pages, numpages); /* Uncached */
> printk("Write: %d pages set as uncacheable\n",numpages);
> }
> else{
> pr_err("Couldn't get pages to set them as UC :(\n");
> return -EAGAIN;
> }
> }
^ permalink raw reply [flat|nested] 5+ messages in thread
* Userspace pages in UC mode
2015-09-11 14:59 ` Sabela Ramos Garea
@ 2015-09-11 18:10 ` David Matlack
2015-09-12 1:12 ` Pranay Srivastava
1 sibling, 0 replies; 5+ messages in thread
From: David Matlack @ 2015-09-11 18:10 UTC (permalink / raw)
To: kernelnewbies
On Fri, Sep 11, 2015 at 7:59 AM, Sabela Ramos Garea
<sabelaraga@gmail.com> wrote:
> Sorry, little mistake copypasting and cleaning. The pages and vma
> structs should look like that:
>
> struct page *pages --> struct page *pages[MAX_PAGES];
> struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
>
> Where MAX_PAGES is defined to 5.
>
> Sabela.
>
> 2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
>> Dear all,
>>
>> For research purposes I need some userspace memory pages to be in
>> uncacheable mode. I am using two different Intel architectures (Sandy
>> Bridge and Haswell) and two different kernels (2.6.32-358 and
>> 3.19.0-28).
>>
>> The non-temporal stores from Intel assembly are not a valid solution
>> so I am programming a kernel module that gets a set of pages from user
>> space reserved with posix_memalign (get_user_pages) and then sets them
>> as uncacheable (I have tried set_pages_uc and set_pages_array_uc).
>> When I use one page, the access times are not very coherent and with
>> more than one page the module crashes (in both architectures and both
>> kernels).
set_pages_uc might just set the kernel virtual address mappings for
those pages as uncacheable via the Page Attribute Table, but you are
accessing from userspace virtual addresses. If you have the userspace
virtual address of the pages you can manually twiddle the PAT bits of
the page tables in your module to make the pages UC.
>>
>> I wonder if I am using the correct approach or if I have to use kernel
>> space pages in order to work with uncacheable memory. Or if I have to
>> remap the memory. Just in case it makes it clearer, I am attaching the
>> relevant lines of a kernel module function that should set the pages
>> as uncacheable. (This function is the .write of a misc device; count
>> is treated as the number of pages).
>>
>> Best and Thanks,
>>
>> Sabela.
>>
>> struct page *pages; //defined outside in order to be able to set them
>> to WB in the release function.
>> int numpages;
>>
>> static ssize_t setup_memory(struct file *filp, const char __user *buf,
>> size_t count, loff_t * ppos)
>> {
>> int res;
>> struct vm_area_struct *vmas;
>>
>> numpages = count/4096;
>>
>> down_read(¤t->mm->mmap_sem);
>> res = get_user_pages(current, current->mm,
>> (unsigned long) buf,
>> numpages, /* Number of pages */
>> 0, /* Do want to write into it */
>> 1, /* do force */
>> &pages,
>> &vmas);
>> up_read(¤t->mm->mmap_sem);
>>
>> numpages=res;
>>
>> if (res > 0) {
>> set_pages_uc(pages, numpages); /* Uncached */
>> printk("Write: %d pages set as uncacheable\n",numpages);
>> }
>> else{
>> pr_err("Couldn't get pages to set them as UC :(\n");
>> return -EAGAIN;
>> }
>> }
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
^ permalink raw reply [flat|nested] 5+ messages in thread
* Userspace pages in UC mode
2015-09-11 14:59 ` Sabela Ramos Garea
2015-09-11 18:10 ` David Matlack
@ 2015-09-12 1:12 ` Pranay Srivastava
2015-09-14 13:37 ` Sabela Ramos Garea
1 sibling, 1 reply; 5+ messages in thread
From: Pranay Srivastava @ 2015-09-12 1:12 UTC (permalink / raw)
To: kernelnewbies
Hi Sabela,
On Fri, Sep 11, 2015 at 8:29 PM, Sabela Ramos Garea
<sabelaraga@gmail.com> wrote:
> Sorry, little mistake copypasting and cleaning. The pages and vma
> structs should look like that:
>
> struct page *pages --> struct page *pages[MAX_PAGES];
> struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
>
> Where MAX_PAGES is defined to 5.
>
> Sabela.
>
> 2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
>> Dear all,
>>
>> For research purposes I need some userspace memory pages to be in
>> uncacheable mode. I am using two different Intel architectures (Sandy
>> Bridge and Haswell) and two different kernels (2.6.32-358 and
>> 3.19.0-28).
>>
>> The non-temporal stores from Intel assembly are not a valid solution
>> so I am programming a kernel module that gets a set of pages from user
>> space reserved with posix_memalign (get_user_pages) and then sets them
>> as uncacheable (I have tried set_pages_uc and set_pages_array_uc).
>> When I use one page, the access times are not very coherent and with
>> more than one page the module crashes (in both architectures and both
>> kernels).
>>
>> I wonder if I am using the correct approach or if I have to use kernel
>> space pages in order to work with uncacheable memory. Or if I have to
>> remap the memory. Just in case it makes it clearer, I am attaching the
>> relevant lines of a kernel module function that should set the pages
>> as uncacheable. (This function is the .write of a misc device; count
>> is treated as the number of pages).
>>
>> Best and Thanks,
>>
>> Sabela.
>>
>> struct page *pages; //defined outside in order to be able to set them
>> to WB in the release function.
>> int numpages;
>>
>> static ssize_t setup_memory(struct file *filp, const char __user *buf,
>> size_t count, loff_t * ppos)
>> {
>> int res;
>> struct vm_area_struct *vmas;
>>
shouldn't this be rounded this up?
>> numpages = count/4096;
>>
>> down_read(¤t->mm->mmap_sem);
>> res = get_user_pages(current, current->mm,
>> (unsigned long) buf,
>> numpages, /* Number of pages */
>> 0, /* Do want to write into it */
>> 1, /* do force */
>> &pages,
>> &vmas);
>> up_read(¤t->mm->mmap_sem);
>>
>> numpages=res;
>>
>> if (res > 0) {
>> set_pages_uc(pages, numpages); /* Uncached */
what about high-mem pages. set_memory_uc does __pa, so perhaps that's
the reason for your kernel oops?
>> printk("Write: %d pages set as uncacheable\n",numpages);
>> }
>> else{
>> pr_err("Couldn't get pages to set them as UC :(\n");
>> return -EAGAIN;
>> }
>> }
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
--
---P.K.S
^ permalink raw reply [flat|nested] 5+ messages in thread
* Userspace pages in UC mode
2015-09-12 1:12 ` Pranay Srivastava
@ 2015-09-14 13:37 ` Sabela Ramos Garea
0 siblings, 0 replies; 5+ messages in thread
From: Sabela Ramos Garea @ 2015-09-14 13:37 UTC (permalink / raw)
To: kernelnewbies
Hi Pranay,
2015-09-12 3:12 GMT+02:00 Pranay Srivastava <pranjas@gmail.com>:
> Hi Sabela,
>
> On Fri, Sep 11, 2015 at 8:29 PM, Sabela Ramos Garea
> <sabelaraga@gmail.com> wrote:
>> Sorry, little mistake copypasting and cleaning. The pages and vma
>> structs should look like that:
>>
>> struct page *pages --> struct page *pages[MAX_PAGES];
>> struct vma_area_struct *vma --> struct vma_area_struct *vma[MAX_PAGES];
>>
>> Where MAX_PAGES is defined to 5.
>>
>> Sabela.
>>
>> 2015-09-11 16:07 GMT+02:00 Sabela Ramos Garea <sabelaraga@gmail.com>:
>>> Dear all,
>>>
>>> For research purposes I need some userspace memory pages to be in
>>> uncacheable mode. I am using two different Intel architectures (Sandy
>>> Bridge and Haswell) and two different kernels (2.6.32-358 and
>>> 3.19.0-28).
>>>
>>> The non-temporal stores from Intel assembly are not a valid solution
>>> so I am programming a kernel module that gets a set of pages from user
>>> space reserved with posix_memalign (get_user_pages) and then sets them
>>> as uncacheable (I have tried set_pages_uc and set_pages_array_uc).
>>> When I use one page, the access times are not very coherent and with
>>> more than one page the module crashes (in both architectures and both
>>> kernels).
>>>
>>> I wonder if I am using the correct approach or if I have to use kernel
>>> space pages in order to work with uncacheable memory. Or if I have to
>>> remap the memory. Just in case it makes it clearer, I am attaching the
>>> relevant lines of a kernel module function that should set the pages
>>> as uncacheable. (This function is the .write of a misc device; count
>>> is treated as the number of pages).
>>>
>>> Best and Thanks,
>>>
>>> Sabela.
>>>
>>> struct page *pages; //defined outside in order to be able to set them
>>> to WB in the release function.
>>> int numpages;
>>>
>>> static ssize_t setup_memory(struct file *filp, const char __user *buf,
>>> size_t count, loff_t * ppos)
>>> {
>>> int res;
>>> struct vm_area_struct *vmas;
>>>
> shouldn't this be rounded this up?
>>> numpages = count/4096;
>>>
For the current tests I am assuming that count is multiple of 4096 and
the user *buf is aligned. Anyway, isn't it safer if I just round down
so I don't mess with addresses outside the range of pages that have to
be set as uncached?
>>> down_read(¤t->mm->mmap_sem);
>>> res = get_user_pages(current, current->mm,
>>> (unsigned long) buf,
>>> numpages, /* Number of pages */
>>> 0, /* Do want to write into it */
>>> 1, /* do force */
>>> &pages,
>>> &vmas);
>>> up_read(¤t->mm->mmap_sem);
>>>
>>> numpages=res;
>>>
>>> if (res > 0) {
>>> set_pages_uc(pages, numpages); /* Uncached */
>
> what about high-mem pages. set_memory_uc does __pa, so perhaps that's
> the reason for your kernel oops?
>
I have used kmap to map the user addresses in kernel space as follows:
if (res > 0) {
for(i=0; i<res; i++){
kaddress = kmap(pages[i]);
set_memory_uc(kaddress,1);//userspace
addresses doesn't have to be contiguous...
}
//set_pages_array_uc(pages, count); /* Uncached */
printk("Write: %d pages set as uncacheable\n",numpages);
}
But the effect in the test code (user space) that tries to measure
cached vs. uncached accesses obtains lower latency for uncached pages.
Accesses are performed and measure like that:
CL_1 = (int *) buffer;
CL_2 = (int *) (buffer+CACHELINE);
//flush caches
//get timestamp
for(j=0;j<10;j++){
CL_2 = (int *) (buffer+CACHELINE);
for (i=1; i<naccesses; i++){
*CL_1 = *CL_2+i;
*CL_2 = *CL_1+i;
CL_2 = (int *)((char *)CL_2+CACHELINE);
}
}
//get timestamp
I've tried to do it within the kernel space but the results are similar.
Thanks,
Sabela.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-09-14 13:37 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-09-11 14:07 Userspace pages in UC mode Sabela Ramos Garea
2015-09-11 14:59 ` Sabela Ramos Garea
2015-09-11 18:10 ` David Matlack
2015-09-12 1:12 ` Pranay Srivastava
2015-09-14 13:37 ` Sabela Ramos Garea
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.