Re: [Gems-users] Miss penalty query

[Dan Gibson <degibson@xxxxxxxx>]

Lyon,

Here are my initial thoughts:

I'm going to assume the following psuedocode for the swap operation:

memcpy(temp,a,sizeof(a))

memcpy(a,b,sizeof(a))

memcpy(b,temp,sizeof(a))

Where a, b, and temp are sized differently depending on the experiment. In particular, I'm going to assume you are using 32-bit load and store operations, and fully complete each step of the above before proceeding to the next. 

Contrast with:

for all bytes i in a:

  temp[i] = a[i]

  a[i] = b[i]

  b[i] = temp[i]

... which of course is quite a different access pattern.

Based on your description, its not possible to know whether and when source/temp/dest occupy different cache lines. For small sizes (e.g. 4 bytes) it seems like they could all live on the same cache line-- in which case I would expect the 'swap' time to be ~ 1 miss. However, in four byte installments, we're talking about 6 operations (3 load, 3 store), each of which has a service time in the L1D (you don't mention your L1D hit latency). Overheads like the L1D tend to become larger as there are more L1D hits (for instance, from increased spatial locality). This will have the effect of increasing (ruby_cycles)/(user_misses), as more time is spent servicing hits. 

Lastly, if your impelmentation of the swap changes the size of the instruction footprint based on the size of the swapped elements, e.g. through loop unrolling, you could see additional latency early in the operation from instruction cache misses.

Regards,

Dan

On Tue, Jul 7, 2009 at 1:51 PM, Lyon Shinseky <lyon.shinseky@xxxxxxxxx> wrote:

Dan,

The bytes are contiguous on a page. The cache line is of 64 bytes. The idea is to measure the raw swap time of x bytes with the bytes not being in the caches. The swap operation is like reading bytes from source and writing to temp placeholder from where it goes to the destination, while the destination bytes go to the source. The initialization phase includes 3 mallocs (this part is not timed by ruby) and assigning values to the source,temp and destination bytes. This is then followed by a ruby magic call and the swap operation starts off till it ends. I am assuming that all this would've removed other effects. The page size is 8192 bytes on the simulated OS and i am assuming that after a malloc and initialization phase , i won't be seeing any unwanted faults during the swap phase which is being timed by ruby.

Thanks

Lyon

For the purposes of this discussion, how do you define a 'swap' operation? Are the operands on the same cache line, or different cache lines? Is the working area warm as far as the page tables are concerned?

Regards,

Dan

On Tue, Jul 7, 2009 at 1:31 PM, Lyon Shinseky <lyon.shinseky@xxxxxxxxx> wrote:

Hi,

I'm measuring the miss penalty of swapping 4 bytes, 16 bytes, 64 bytes, 256 bytes, 1024 bytes and 8192 bytes on ruby. I have a code which simply swaps these bytes to measure the ruby cycles taken when this data is not in ruby cache. With 4 bytes, the cycles/(user misses) is 347 cycles which is close to my memory latency (300 cycles). With 16 bytes , this goes upto 392 cycles. With 256 bytes, this goes upto 677 cycles and with 8192 bytes ,  this is 1200 cycles. I'm unable to reason about the penalty increasing as I increase the  number of bytes to swap. My expectation is that it should remain around 350 cycles . . . any thoughts ?

Lyon

_______________________________________________
Gems-users mailing list
Gems-users@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/gems-users
Use Google to search the GEMS Users mailing list by adding "site:https://lists.cs.wisc.edu/archive/gems-users/" to your search.

--
http://www.cs.wisc.edu/~gibson [esc]:wq!

_______________________________________________
Gems-users mailing list
Gems-users@xxxxxxxxxxx


https://lists.cs.wisc.edu/mailman/listinfo/gems-users
Use Google to search the GEMS Users mailing list by adding "site:https://lists.cs.wisc.edu/archive/gems-users/" to your search.

_______________________________________________
Gems-users mailing list
Gems-users@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/gems-users
Use Google to search the GEMS Users mailing list by adding "site:https://lists.cs.wisc.edu/archive/gems-users/" to your search.

--
http://www.cs.wisc.edu/~gibson [esc]:wq!