So I purchased some standard Crucial DDR3-1333  PC3-10600.

If you go to crucial’s website and you enter in a P55 motherboard, you will get this suggested memory:

CT2KIT25664BA1339

I ordered 2 4GB kits, each with with 2 2GB dimms.

The dimms all looked like this:

here is what CPUZ saw from the SPD of the dimm:

So, everything seemed fine until I started to get random reboots every couple days.

I tried every possible combination of the dimms, and I always got the same result; hard-reboots.

I felt the odds of all 4 dimms being faulty is roughly zero, so I decided it must be the motherboard, until I started reading these comments on newegg about my memory.  There are/was a very high percentage of people claiming the memory did not work on their P55 motherboard.

Then I started to see newegg stamp certain memory P55 Core i5/i7 compatible, which didn’t seem to make sense, because any memory that adheres to JEDEC spec for PC3-10600 should be compatible.¿?

I found Intel’s own Validation Results for DDR3-1333 on P55 here.

It lists Crucial’s CT25664BA1339, but specifically the CT25664BA1339.16FF.

Looking closer at my memory I found it to be CT25664BA1339.M16SFD.

for reference this dimm uses micron chips with markings: 9ND22 D9JNM

Crucial said the last string of this part number is for internal use only, and refers to which chip/set is being used on the dimm.  The crucial representative said it should make no difference with respect to P55 compatibility, and was perplexed why Intel would specifically approve the 16FF.  In fact as a consumer you cannot even see what the internal chipset number is before you buy online, but you can call and ask to order a specific one over the phone.

I did just that, and ordered a set of CT25664BA1339.16FF over the phone from crucial, they looked like this:

Notice the Micron label on the left?  I assume this means the dimm itself was manufactured by Micron, and resold by crucial.

(my other CT25664BA1339.M16SFD dimms don’t have a micron sticker.)

The CT25664BA1339.16FF dimm uses different micron chips marked: 9NF22 D9KPT

So the Crucial CT25664BA1339.16FF is in fact a Micron MT16JTF25664AZ-1G4F1 dimm.

CPU-Z’s information from the SPD agrees:

It is no surprise that both the Crucial CT25664BA1339.16FF and the Micron MT16JTF25664AZ-1G4F1 passed Intel’s validation tests, as they are the exact same dimm:

These new Micron dimms seem to run fine in my system without any problems.

I still don’t know why the P55 chipset/CPUs have problems with the non-16FF crucial dimms.  So I by no means really understand the root of the issue, but I thought others might find this interesting. 

The memory Crucial will send you for a P55 system may not be the memory you want. 

It also seems that many DDR3 kits that seem fine otherwise, do not work right with the P55.

Software>

Proof of concept www.wolframalpha.com shows a taste of the future of querying data.

Windows 7 which is a refined build of Windows Vista is the best Desktop OS.

Google Chrome 3 is by far the best consumer web browser.

Google Voice provides 1 number with voice, voicemail, and SMS capability.  Those can be forwarded to a mobile phone if desired.  There is a Google Voice application for android which lists text messages and voicemails for you to read/listen/delete/reply-to individually (it transcribes voicemails so you can read them before you choose to listen).  Using this application, SMS’s and voicemails are transmitted over DATA, so they do not count as a text messages in your plan.  The whole system has a web interface for you to send and view text messages, voicemails and call history.  It will also allow you to keep in touch with SMS/voicemail while you travel outside of your cell phone service coverage.

Storage>

Seagate has 500GB platters which it uses in products like:

• 3.5” 7200rpm 1TB @$90
• 3.5” 5900rpm 2TB @$200

Intel has launched their 2nd Generation MLC Solid State Disk using 34nm flash (as opposed to 50nm)

• 2.5” 80GB @$230
• 2.5” 160GB @$450

These new Intel drives support something called TRIM, which is also supported in Windows 7.  Without Trim, when a file is deleted, the TOC is updated to consider that area on the drive empty, the data isn’t actually wiped.  This is fine for rotating discs because the drive will simply over-write those bits when a new request is received.  On current SSDs there is a problem, because blocks must actually be erased before a write, so when a request to write data on the previously used space, it must first erase the area, then write, and this costs time.  TRIM tags the areas of the drive where the data was deleted, and the drive will (on its own time) erase those blocks of bits.  This means in some cases a block won’t need to be erased prior to write, since the drive actually erased it on its own time.

This is good, but you should know that if there is live data in the vicinity of a write, the SSD must still read the entire block into memory, erase the block, add the new data to the data in memory, then write the entire block.  The push should be for reducing the block sizes, to reduce the time-tax in random writes.  Intel AFAIK has the smallest block size around.  I’ll review this new Intel SSD soon.

Memory>

DDR3 has arrived at reasonable price points.

• Crucial 4GB (2x2GB) DDR3-1333 kit @$65
• Crucial 4GB (2x2GB) DDR2-800   kit @$53

what DDR3 desperately needs is higher density, supposedly reasonably priced 4GB dimms will arrive in the coming months with Samsung chips.

CPU/Platforms>

We are in a weird space now.

Last year Intel launched its revolutionary 45nm Nehalem CPU which features an integrated memory controller.  These chips launched with a very high-end and expensive x58 platform.  Due to the price of the platform and (at the time) expensive DDR3 memory, this platform never really hit the average or value conscious consumer.

Because of this, most computers sold are using a relatively old Penryn CPUs built on 45nm technology without an integrated memory controller.  These CPUs have been for sale since January 2008!  And the Penryn was just a die-shrink of the Conroe which was famously launched in the summer of 2006.  Great for 2006, but it’s been a long time.

In the coming weeks Intel will launch consumer-oriented Nehalem-based CPUs (Lynnfield).  They will be slightly improved, slightly cheapened versions of the current Core i7.  The new platform is the Intel P55 with a new socket (LGA1156) supporting the double channel memory controller (as opposed to 3x in the current i7).

These new chips are still 45nm.

That’s all well and good for high end consumer PCs and big and clunky laptops, but for thin-light mobile notebooks it doesn’t do that much, these are still 45nm, and they aren’t even attempting a Low or Ultra-Low Voltage variant.  And the normal voltage new chip will still require 3 separate chips to complete the platform (including video = GPU).

What’s interesting is around January intel is releasing the 32nm Family of processors with GPU (graphics processor) integrated in the CPU package! 

This reduces the number of major chips needed to 2 for the entire platform, and these chips will be lower power due to the die shrinks.

It’s called Westmere, and while desktop and clunky laptops will also get their own models, they don’t need it nearly as much.

here are some Westmere Desktop options available in January:

GPU performance on Westmere will be roughly double that of the modern x4500HD, it also ads some video decoding features.

Here’s the picture for the Ultra-Low-Voltage space (think Macbook /Air and smaller) dual cores. (~10W for CPU duties, ~8W for GPU)

Exciting times after Christmas for thin and light laptops like the Acer Timeline 13.3”, Dell adamo, MacBook Air, ThinkPad x301, etc.

So in the thin notebook segment there will be a massive improvement on the platform in about 6 months, what about Netbooks with the ATOM?

Currently the atoms are a bit too slow to be useful in a mini-notebook and use a bit too much power for small gadgets.

In the coming few months there will be new Atom processors, still using 45nm tech, but will have an Integrated Memory Controller, and an Integrated Graphics Processor, both on-die!

Drastic decrease in power usage for the platform, down to 2 chips instead of 3… should allow for fan-less designs.  The atom finally becomes modernized!  May be still slower than desired.  In 2010 we should see 32nm Atoms.

So huge improvements in the netbook space to come real soon!

I havn’t talked about AMD, currently they are not competitive in the low-power mobile space (in fact they don’t have any 45nm parts, so they can’t compete on energy efficiency).  But they have migrated to 45nm on the desktop and probably have the best integrated platform out there.  Speaking of the 785g chipset paired with a 45nm AMD CPU, if you need an inexpensive and quick desktop today, buy this.

AMD has this corner dominated until Intel releases Westmere after Christmas.

Mobile Phones:

currently only 3 worth mentioning:

• iPhone 3GS
• Palm Pre
• HTC/T-mobile G1 and myTouch 3G

The next 6 months will prove interesting as around 6 more Google Android mobile phones will be released on T-mobile, AT&T and Sprint.

So I wanted to see how fast they are in the immensely popular javascript:

<–relative speed. (higher is better)—>

OS: windows 7 64-bit b7048
RAM: 4GB DDR2-800
CPU: intel 2-core 3Ghz Penryn w/6MB cache
HDD: Samsung F1 7200rpm
Chipset: intel G45

v8 is http://v8.googlecode.com/svn/data/benchmarks/v3/run.html

sunspider is http://www2.webkit.org/perf/sunspider-0.9/sunspider.html

dromaeo JS is http://dromaeo.com/?dromaeo

dromaeo DOM is http://dromaeo.com/?dom|jslib|cssquery

(ie and safari 3 were not able to complete the dromaeo DOM test suite w/default settings)

note:

the 64-bit firefox I tested is not official; http://www.mozilla-x86-64.com/

google chrome 2.0.269.0 is a dev/beta build.

opera, IE, chrome, firefox, k-meleon, safari are all 32-bit unless otherwise noted.

looks like intel is implying they have a drive cluster size an order of magnitude smaller than current mainstream MLC drives.

This (obviously?) yields much better random small write performance, and according to this info, Intel did it w/o sacrificing STR…

I stole the photos from this anandtech piece.

After my 2 previous SSD investigations, I can’t wait to get my hands on these.

Let’s take a deep look at storage, and while we are there we can stop and see if the if Intel’s ubiquitous ich9r is any good at raid.

The Performance analysis of storage is generally partitioned in two, Sequential R/W transfer rates STR, and Random R/W performance usually measured in I/Os per second or a Latency measurement.

STR>

Sequential Transfer Rate (STR) is the data throughput a storage device can achieve without having to seek to various parts of the disc.

Hard disc primer for STR:

SCSI, SAS, ATA-66/100/133, and SATA1/2 all had impressive throughput rates for their time, but the interface was never the bottleneck.

The discs themselves have sustained transfer rates (STR) limited by:

1. Linear Speed L of the Head which is a function of:

• Rotational Velocity V (revolutions per time) on most hard drives this is not dynamic.
• Location of the data r (radial distance of the head where the data is being operated on).

so L=V*2*Pi*r , in a normal 7200 revolution per minute, 3.5″ hard disc (radius near 1.5″) the Linear Velocity L = 7200*2*Pi*1.5 = ~67858 inches per minute at the outer edge, less as you get closer to the center.

this Linear Velocity reduces in a linear fashion as you approach the spindle of the disc.

2. Linear data density d, (bits per inch), which is usually proportional to the square root of the areal density (higher density means the head can traverse and r/w more sectors for a given linear velocity)

so

Sequential Throughput ~ L * d ~ 2 * Pi * V * r * d

so there you have it, in a given disc, sequential data throughput is linearly related to distance from center.

this is why 3.5″ drives are generally faster in sequential operations since much of the data is a lot further away from the center than any 2.5″ disc.

————————————————————————–

It may not be called a disc problem, but effective disc large-file transfer rate can be throttled if the data has to be fragmented on various spots on the disc, since that requires head seeks for something that could be a sequential operation. Seeks cost time and transfer no data.

Carefully chosen modern 7200rpm SATA2 high-areal density discs like the this or this can perform sustained sequential reads or writes close to 1Gb per second at the outer edge. The discs I have been messing with ( Western Digital WD6400AAKS ) have sequential performance around 900Mb/sec near the outer edge, so they will bottleneck large 1,000Mb/sec Ethernet transfers, but only a bit.

these graphs are decreasing because the program calls 100% the innermost, and 0% the outermost of the disc… also you may notice the graphs are not linear as I suggested, this is because the horizontal axis is “%” which is % of data, not % of radial distance.

I will not bore anyone with the math/logic to understand why this makes sense, but it does… and graphing it like HD-Tune did here should theoretically yield a quadratic, which it seems to by the pictures.

————————————————————————–

Latency>

The other major performance specification of any storage system is the latency, or time required to begin a read or write at a random location on the disc.

In a traditional rotating disc drive, the latency is not so straightforward to predict based on the geometry.

Clearly the latency cannot be predicted simply by the location of the data, one must also know where the head will be before needing to access the data.  But lets talk of average random access times…

Certainly faster rotation will minimize rotational delays, while higher areal density can effectively minimize real world latency if it means you can cram more data at the outer edge, minimizing or eliminating the need for the head to go towards the spindle… but on a disc full of data, the argument is irrelevant.

Let me save a more thorough analysis of seek times for another day, and lets think of it as a latency (penalty) for each read/write that is not adjacent to the previous i/o.

————————————————————————–

question: does the revision and/or firmware of a disc drive matter?

answer: I like to use graphs to communicate, so here goes:

(these are all WD6400AAKS Western Digital 640GB SE16 drives, but different versions)

So, clearly the revision/version of a specific disc drive matters significantly. The 00A7B0 was quicker in latency, but slower in STR than the other two.  The 65A7B0 was the best in STR.

ok, everything so far was for a specific type of storage device consisting of 1 head and 1 rotating disc (yes there are multiple platters , each face getting a head, but they do not seek independently, so effectively we can consider it as 1 face of 1 platter with 1 head.)

Redundant Array of Inexpensive Discs?>

Although the actual words which make the acronym are not all that applicable today (since many raid volumes lack redundancy and few are cheap), RAID has taken off as an admirably simple solution to achieve what many situations require:

Primer:

Why would anyone want to use raid?

+a given hard disk around a 5% chance of failing per year. (this depends on age and temperature and utilization) so It’s nice if a failure does not result in data loss or even any downtime at all.

+having a larger pool, rather than several smaller pools to store data is preferable as it eases file management. (no shuffling around data to various discs to make room)

+the hope that n discs could be n times faster than 1…

RAID 1

If your only concern is maintaining data integrity and availability in the event of a disc failure, RAID 1 “mirroring” is the obvious solution. The idea that all writes are done to 2 discs, instead of one. If one disc fails, it simply reads from the remaining one, since all the data on both drives is identical.

• write throughput depends on implementation, but could be slower than a single disc
• write latency should be a bit worse, since it must be written on both drives, the write is not complete until both seek, so its the slower of the two seeks, every time.
• read latency could be slightly better than a single disc, as a smart controller could only request the data from the disc who’s head is closer to the data, or request it from both, and take the one who gives it first.
• read speed throughput depends on implementation, but could be faster than a single disc, as the controller could have each disc read a different half of the file, halving the time it takes to read the whole file… this is not generally implemented.
• capacity is halved, as the 2 discs have identical data, usable storage is only the data capacity of one disc, so it costs 1/2 capacity for the redundancy of 2-disc raid 1.
• If each disc has a f % annual failure rate, the raid1 array will have a 100*( (f/100)^2 ) % failure rate. (so if the disc failure rate is 5.0%, the raid1 volume failure rate will be 0.25% )

RAID 0

If you want a larger pool of data then a single drive can provide, or you want faster file reads and writes, and you aren’t worried about disc failure than you RAID 0 is the solution.

• read and write throughput should linearly improve with the number of drives in the array.
• Latency should be a bit worse than a single disc, as each disc needs to seek to the file, so the longest any of the heads have to seek is your access time for that read or write action.

• volume capacity is simply the number of discs times the capacity of each disc (if they are not the same size, then the capacity of the smallest disc times the number of discs), no or very little wasted space.

• there is absolutely no data redundancy, in fact if each drive has a f% chance to die in a year, then a n-disc raid0 volume has a 100*(1-(1-f/100)^n)% chance of failing resulting in data loss of the entire volume. (so a 4-disc array in raid0 where each drive has a 5% chance of failing annually means the raid0 volume has a 18.55% chance of failing each year.)

So, raid1 gives you redundancy but no speed gain, while eating up 1/2 of your disc space, and raid0 can yield a lot of streaming speed gains, but will actually make your reliability much worse.

RAID1+0 or RAID 10

This is a even-number-of-discs >= 4 solution where the controller uses raid1 for each pair of discs, and RAID 0 to stripe the pairs together into a larger, faster volume.

6-disc raid 10 :

                       RAID 0
       .-----------------------------------.
       |                 |                 |
     RAID 1            RAID 1            RAID 1
   .--------.        .--------.        .--------.
   |        |        |        |        |        |
120 GB   120 GB   120 GB   120 GB   120 GB   120 GB
  A1       A1       A2       A2       A3       A3
  A4       A4       A5       A5       A6       A6
  A7       A7       A8       A8       A9       A9
  A10      A10      A11      A11      A12      A12

if there are n-discs in the array:

• Read throughput should be about n/2 times single disc speeds.
• Read latency could be close to a single disc, as raid0′s latency is the worst of all member volumes, but here member volumes are raid1, where each could take only the fastest seek of its member discs.

• Write throughput could be close to n/2 times single disc speeds, but is dependant on the implementation as each write must duplicated on both drives of each of the mirrored volumes.
• Write latency should be in the same ballpark as a single disc, but should be worse as both raid1 and raid0 must complete the seek on all member drives for the operation to complete.

• total volume size of RAID10 arrays are n/2 times the single disc capacity, so the cost of redundancy here is 50% of the disc space.

• If the disc failure rate is f%, then the raid10 volume failure rate will be 100*(1-(f/100)^2)^(n/2) %. If each drive has a 5.0% annual failure rate, a 4-disc RAID10 array will have a failure rate of 0.499% , while a 6-disc RAID10 array will have a failure rate of 0.748%

So raid10 might seem like a decent balance of performance and reliability, but it is at the cost of 50% of the hard drives involved.

RAID 5

RAID5 arrays are of 3 or more discs where the data is striped across all discs, but for each stripe of data, a parity bit is written on one of the discs, so if one drive fails, the data can be recreated or recovered for each stripe.

• Read throughput should be n-1 times as fast as a single disc.
• Read latency should be slightly worse than a single disc, as all but one of the discs must complete the seek, for the operation to complete.

• Write throughput could be n-1 times as fast as a single disc, but there are some important notes on write performance. While data is being written, the controller must calculate the parity for each stripe, this has the potential to slow down writing significantly, while some may call this parity writing a latency, but since it is a constant tax on all writes, big or small, I’ll consider to effectively throttle write throughput. Cache/buffer can buy time for the system to calculate this parity.
• Write latency in raid 5 is inherently troublesome.  not only does the system need to calculate parity for each stripe that is written, and write the corresponding parity bit, but if the data being written is smaller than the stripe itself, the controller must first read the stripe, then modify the data (adding / changing bits as needed), and then calculate parity on the entire stripe, and write it entirely.  I will refer to this as the partial-stripe-write issue/penalty.

• Total usable volume size of a RAID 5 array is about (n-1) * smallest-disc-size, so a 4 disc array will have a usable storage area of 3*disc-size.

• a n-disc RAID 5 volume failure rate will be the odds that 2 (or more) discs fail in the array.  If each disc has a failure rate of f%, the RAID 5 volume failure rate is 100*[1- ((1-f/100)^n * (f/100)^0 + nC1*(1-f/100)^(n-1) * (f/100)^1)] %.  (for a 4-disc array where the Annual Failure Rate per-disc is 5%, the failure rate of the RAID 5 volume is 1.40%, 3.2% for a 6-disc array)  In production environments the effective failure rate will be closer to zero for the volume since a failed disc will be replaced in a matter of hours or days, the array will be rebuilt and redundancy restored.

.

RAID 5 clearly looks like the best and most scalable raid technology listed above, providing streaming speed boosts and protection from disc failure, while only “wasting” 1 disc of the array for parity…. but do we need an expensive or obscure RAID controller to get decent write performance in the real world?

enter the Intel ICH9r.

this is Intel’s current raid-enabled southbridge, paired with most 3-series northbridges and many motherboard manufactures have paired it with the x48 northbridge.

Obviously the southbridge does many things, but here let’s look at it solely as a raid controller.

During the write of a raid 5 stripe, a parity bit must be calculated from the data to be written, before the write occurs.  Some high-end RAID controllers perform this calculation with the aid of an XOR processor.  Intel’s ICH9r does not have an XOR processor, it leverages the CPU to do the calculation.  In the past this may be an enormous concern, today we have dual and quad core CPUs on the cheap, yielding an excessive amount of unused processing power in most small business machines.

Another trick expensive RAID controllers use is onboard cache, for aiding in small writes, this is critically important because the system must calculate the parity of a stripe before it is written, so to have a buffer to buy time w/o holding up other IOs is key.  Intel’s ICH9r has no cache, but it can simply take some of the very cheap and abundant system memory, It may be more now days, but back in the ich7r days it allocated 4MB of system memory on boot for caching raid arrays.  The is enough to handle small random writes gracefully, dealing with the partial-stripe-write issue.

So with that out of the way, the ich9r is a hardware raid controller in that, the raid arrays are transparent to the OS, but it does utilize the system resources to do the XOR for the parity calculation and volume write caching.

That’s all well and good, but text is cheap, lets look at graphs:

Read Throughput:

Read Latency:

Write Throughput:

Here we see the ICH9R doing a pretty good job in Raid10 and  Raid5.  In raid10 performance is almost double a single disc, which is what you would expect/hope for.  In raid5 you might hope for tipple the performance of a single drive, but it looks like we will have to settle for double.  Clearly if you are not going to enable disc and raid cache, you should not use raid5 for anything where write performance matters.

Write Latency:

Here we see impressive number from the ICH9R in Raid10, latency is significantly less than a single disc if disc cache is enabled, I’m not sure why this is, but possibly a larger pool of cache from both volumes in the stripe.  Raid5 is, as expected worse than a single disc in write latency, with disc and raid cache enabled, the latency is comparable, yet still worse than a single disc.

is it really this simple? 

(Can the performance of these drives really be characterized by only these metrics?)

for a single disc:

Overall for a single disc, with or without cache the 2 metric model seems to work pretty well, although around 1MiB request sizes we see actual performance fail to match predicted, especially for writes.

What about for Raid1:

It’s almost as if it only does striped reads when RAID cache is off and the read data request is larger than 64MiB…  strange.

What about writes?:

Here we see a discrepancy across the board at 1MiB transfers…Clearly HDD cache on, RAID cache off is ideal for Raid 1 on the Intel ich9r.

What about Raid10:

Let’s see what’s going on at small write request sizes:

Here we see the 1st order latency/str model really failing to model the mid-size transfer random write performance of the drive…

Although it doesn’t have the highest write STR or latency, it seems RAID cache should be disabled, with hdd cache on because the read STR is so much larger.  Intel’s raid cache seems to have adverse affect in many scenarios.

And finally, Raid 5:

Raid5 random read performance is decently approximated by the 1st order latency/str model, although there are significantly under-performing real results in the mid-range as seen above from 1MiB to 64MiB data request size.

what about writes:

at least in the mid range this simple model of latency/str seems to fail to really characterize performance here… Let’s look at the log scale:

Cacheless and disc cache only raid5 volume configurations seem to act as expected by the simplistic 1st order latency/STR model…

But that is becuase the model doesn’t really deal with the cache.  Cache can greatly improve random write effective latency, since the system does not have to wait for the controller to calculate the parity bit and then physically write the entire stripe to the disc… eventually as the transfer request size is large, this cache becomes irrelevant.

I am not one to be content to leave things w/o a full understanding, so lets dig deeper:

OK, so we know w/o cache the raid5 volume has a latency of 62.5ms on average..

that includes a seek and a partial-stripe-write penalty…

we know from a single cacheless disc, the seek time is about 17.5ms,

so the partial-stripe-write penalty = 62.5-17.5 = 45ms

As the write request gets large relative to stripe size (here is 64KiB), the write will probably incurr 1 seek and a partial-write penalty at the start, and a partial-write-penalty at the end of the write for a total of 1 seek and 2 partial-stripe-write penalties. This means the effective random write latency is 17.5+45+45 = 107.5ms, knowing this I can go back and re-calculate the STR on buffered writes, for example with HDD&RAID cache enabled the 4-disc r5 volume got 0.31 iops in random writes of 512MiB, so

.1075 + 512/STR = 1/0.31 ==> STR = 164.2MiB/sec write with disc and ich9r cache enabled.

ok so lets model a volume with 107.5ms random write latency and 164.2MiB/sec write STR:

This predicts it dead on after 1MiB data request sizes…

Now this makes some sense… with small data sizes, the caching can buffer the writes, buying some time for the system to read the stripe, make the write in the buffer, then calculate the parity bit for that modified stripe without holding up the next operation….(the data is written to the storage device, even though its actually not written to the physical platters yet).

The combined cache does seem to adequately buffer the sequential writes, by giving the system time to calculate the parity bit before its actually written to the platters, it is not 3x the STR of a single disc, as it theoretically could be, but its still admirable.

note* performance was a bit better with OS caching,, (the “advanced performance” option under the volume properties in the device manager), and while I don’t feel scared when using the disc or raid controller’s cache… I think the slight gains (~5% write STR) isn’t worth the bother of having yet another place data is written to without actually going to the disc.

Brief Cache Talk:

I focused on various caching schemes throughout the article, and for good reason, it has massive effects on real-world performance.

there is something worth mentioning here:

In OS level caching, writes can be buffered in the system memory, it’s possible something could interfere before this data makes it to the hard disc.  While this is probably not a big concern, ive seen little-to-zero advantages in my testing in windows, in fact sometimes the overhead.

Raid Controller cache, for many fancy controllers are on the controller card itself, so its a bit independent of the OS, which is good.. but if power goes out this data is not written to the discs and power is lost.  High end controllers have cache batteries which are meant to keep that data alive until its written to the disc, in the event where power is lost.  This intel controller dedicates some of the system ram to itself on boot, so its fairly isolated from the system.  The battery backup i think is a like using a hammer to fix a computer… Yes it will be able to put the cache on teh disc, but who’s to say the cache doesn’t have partial information of files, resulting in corruption…  

Hard Disc Cache is basically a bit of ram on the drive itself, it is used for buffering writes and buffering IOs to allow for re-ordering (NCQ), anything in this buffer will not be saved to disc in the event of power-loss to the system.

Disabling cache altogether is also not a solution, if the head of the disc if half way through a write operation, or a file write, partial writes will occur, possibly resulting in corruption/unusable data.

Bottom line if you care about integrity of your data, your system simply can not lose power, spend the money on redundant PSU’s and UPS’s rather than the (imho) useless cache battery system.

Conclusions>

well, this was more a lesson in storage than anything else, but lets see:

The ICH9R does a very admiral job, we aren’t talking about breaking any records here but performs respectively as long as you configure the caching properly.  It also has price (nearly free) going for it as well as an industry giant supporting it… not to mention its ginormous userbase and forward/backward compatibility of raid volumes.

In summery, raid on the ICH9R:

There is more to do, but my time is finite.

to do: present my data and analyze how allowing multiple random IOs to accumulate before a read/write affects performance.

to do: in the future compare the ICH9r with a fancy/expensive raid controller

to do: confirm that Intel’s new ICH10R performance almost identical with the ICH9R

to do:  RAID-Z  <– this is big, but seemingly not ready for prime-time.

to do: clean up this article and fix some slight imperfections here and there.

Read Performance:

Here we see read performance being fantastic, yet not as good as the OCZ Core series I reviewed the other day.

so lets consider the following:

random_read_latency+transfer_request_size/STR_read = seconds_per_io

so lets consider 512Byte and 700MiB transfers to calculate random_read_latency and STR_read.

700MiB:

random-read_latency+700*1024KiB/STR_read = seconds_per_io = 1/.174214 = 5.74

512Byte:

random_read_latency+0.5KiB/STR_read = seconds_per_io=1/ios_per_second = 1/ 2170.88

now subtract the 2nd equation from the 1st and the result:

(700*1024 – .5)/STR_read = 5.74- 1/2170.88 = 5.73960617

716799.5/5.73960617 = STR_read= 124886.53 KiB/sec

=>The average sequential read throughput is  121.96 MiB/sec

let’s plug that back into the 512Byte equation:

random_read_latency + .5KiB/(124886.53KiB/sec) = 1/2170.88

random_read_latency = .000456639 seconds

=> The average random read latency is 0.457ms

For a sanity check:

Here we see the random read latency and STR I calculated to characterize the performance very well… but around 512KiB request sizes, the performance is slightly lower than expected.

Overall I can say with confidence that the drives read specs are

122MiB/sec average read STR

0.457ms average random read latency

write performance:

Write performance is very similar to the OCZ Core SSD, although the STR is clearly a bit less on the RiDATA.

since random_write_latency + request_size/STR_write = 1/iops

[random_write_latency+ 700MiB/STR_write = 1/.097486]

- [random_write_latency+ .5KiB/STR_write = 1/4.04]

—————————————————————————-

(700*1024-1/2)/STR_write = 1/.097486 – 1/4.04 = 10.257883 – .24752475 = 10.01035824

STR_write = 71605.78 KiB/sec

=> average writing sequential transfer rate is 69.93 MiB/sec

so, random_write_latency + .5KiB/71605.78 = 1/4.04

random_write_latency = .2475 seconds

=> average random write latency is 247.5 ms

sanity check:

So, while in the ballpark, this fixed random write latency model fails to characterize actual performance with enough precision.  As we did with the OCZ, lets assume that often on writes significant in size relative to the drive’s cluster size (minimum read-modify-erase-write block size), the write gets hit with 2 penalties, at the beginning and the end of the write, where both the start and stop points do not lie on the edge of one the drive’s clusters.

So this makes sense, writes over 2MiB in size tend to incur 2 write penalties a significant percent of the time.

Here we see as write size goes up, the odds of getting 2 (erase-block) penalties increases to nearly 100%.

In Conclusion:

The RiDATA Ultra-S Plus NSSD-S25-64-CO4MM-PM is great for reads, amazing for tiny random reads, but mediocre for writes, and amazingly terrible for small random writes.  This unit is on par with the OCZ Core SSD, at roughly the same price.

Read into it:

OCZ Core 64GB SSD	WD640AAKS (2-platter) hdd-cache enabled
Notice above that the data placement on the disc essentially does not matter, other than being noisy, its fairly flat.In the IO meter test the performance does not improve much as you allow multiple outstanding IOs, this makes sense when you are dealing with almost negligible access times.	Notice above the quadratically decreasing transfer rates, this is because of the linear relationship of speed and distance of the head from the center of the disc.. (the graph is non linear because data density per radial distance increases as you get further from the spindle (longer tracks).)In the IO meter test Notice that the drive benefits heavily with the use of Outstanding IOs, this allows the drive to read in such a way that is not as random, optimizing the order of the reads, among other things.

Here you see both drives approach their Average Read Throughput (as measured by HDTune) as the request size gets large enough (minimizing frequency of seeks). This is exactly what we would expect, and the rotating hard disc needs an enormous data size (>256MB) to make seeks insignificant, while the OCZ SSD only needs 8MB, impressive!

I know HDTune gives us an average seek time, but lets just say we didn’t trust HDTune, or we wanted more precision than the 1 significant figure it gives us on the SSD.

Let’s take random read throughput for a very large data size as an excellent measure of average read STR.

Now let’s take that as the average STR, and use it to measure average random seek times:

note that our average random read seeks and throughputs nearly matched what HDTune found.

Just for kicks, lets use our calculated numbers for avg STR and random read seek times, to see how well the predict performance:

Looks good, lets look at it on a log scale:

So on the read side of things, everything now makes sense, and this is a very impressive drive, what’s next?

.

.

‘write it down’, or, ‘the good, the bad, and the ugly’:

.

.

OCZ Core 64GB SSD	WD640AAKS (2-platter) hdd-cache enabled
Something seems fishy here with HDTune, the graph just doesn’t look right.the iometer results are pitiful, maxes out at 4 IOs per second, LOL, WTF?does enabling write-back cache in windows help?: No it doesn’t.	these normal or boring results come at a relief after seeing the OCZ write results. The HDTune graph is again as expected.It’s interesting in the IOmeter results is that the rotating disc does not seem to benefit from outstanding IOs for writes, as it did with reads, this is probably because the write-cache is doing its job very well, essentially yielding the same benefits as accumulating multiple outstanding IOs can yield. Interesting that performance with any number of outstanding IOs less than(or equal to) 256 with cache enabled is roughly equal to the performance with 256IOs with cache disabled. well 256 chunks of 64KB is 16MB, exactly the size of the cache on this disc… So the disc cache yields similar benefits of having 16MB of outstanding IOs… this is to be expected, but nice to see.

Let’s see the random write throughput as data request size gets very large, should approach the average write STR as reported by HDTune:

Now it’s the totally reversed situation, the SSD needs a much larger data size before it gets good random write data throughput.

While the situation with the WD6400AAKS seems pretty normal, the SSD’s throughput blows away HDTune’s measurement of average STR as the transfer request size gets over 64MB. Something is definitely fishy.

HDTune was fairly accurate with read STR and read access time, lets see what happens when we trust HDTune’s numbers to predict random write performance:

… and here in log/log:

Here we see HDTune’s numbers predict the rotating disc results reasonably well, but totally fail to explain the SSD’s performance in random writes.

Let’s calculate random write seeks as we did before with reads:

Now let’s take that as the average STR, and use it to measure average random seek times:

OCZ Core’s result seems unbelievably bad, while the spinning discs number seems ok, if not a bit high after disabling write cache, which makes sense.

I understand you probably would never want to disable write cache, but the result with it disabled is important for understanding performance, and thus being able to predict performance across other data use scenarios.

Now, lets see if these numbers explain the strange reality of performance with the SSD:

That’s pretty close, but can’t help but see the SSD predictions are still off a bit.

Note the 245ms random write seek time, while useful and, well, easy to understand, is not exactly how it works, after-all SSDs don’t have heads to seek.

If my understanding is correct, this SSD makes small writes by first finding a block or “cluster”, then reading in the entire cluster, then making changes to the buffered data from the cluster, and then writing the modified data as an entire cluster to the SSD.

So on tiny writes, the drive needs to read the entire cluster into memory, make the change, then write it to the flash, we have been referring to this wasted time as a seek, because we are familiar with it as a way to understand hard drive performance… but maybe we need to think a bit further:

on writes tiny compared to the cluster size, 1 entire cluster will be read into memory, the data manipulated, then written back… but as the write size gets significant relative to cluster size, there’s a good chance you will have to read/write 2 clusters, even if the data is smaller than a cluster… if you start to write 1KiB at the last bit of a cluster, you will write 512Bytes on one cluster, and 512Bytes on the next cluster.. basically paying 2 penalties of reading/writing data that you don’t care about.

As the data size gets very large relative to cluster size, you must read/write all clusters, but the first and last cluster are likely partially data you don’t care about, and hence you will pay a tax up to two times the cluster size.

Here we see it does make sense after all, since a write can pay up to 2 full “seek times” to make the transaction.

Random Write seeks/access time/penalty/tax options (for this drive 1 penalty is 245.7ms):

1. for writes very small compared to the cluster size, almost always 1 full penalty is paid, the time-cost to read and write the entire cluster.
2. for writes sizes that are an exact multiple of the cluster size, it is possible to pay 0 time in penalty, assuming the write starts at the exact beginning of a cluster. (highly unlikely), but this is essentially what happens in the middle of large writes, no tax is paid on all clusters other than the first and last in each sequential write.
3. for significantly large writes (compared to cluster size) time-cost is 0-2 penalties, the penalty will probably be 1 partial at the start of the write, and 1 partial at the end, because its unlikely the start or end will be exactly the start or end of the corresponding clusters, those partials could be good or bad, depending on how much of the start and end clusters is your data, or existing data that you don’t care about.
4. For a horribly fragmented drive, where free space is sparse, the penalties for a large write could be as numerous as the number of clusters on the disc! This situation is highly avoidable, and very unlikely.

Theoretically all writes have a 0-2 penalty tax, but its highly unlikely a tiny write will be written just at the end of the cluster, requiring a second cluster to finish the write, and on very large clusters you are likely paying 2 partial penalties, but if it’s very large, the time becomes insignificant. For this SSD, we see that around an 8MiB request size, writes pay a tax larger than 1 penalty cost:

Here we see that certain data sizes pay a higher average cost to make a random write, particularly 4MiB to 256MiB.

A quick and dirty estimate of the OCZ Core cluster size:

1 full time-penalty-cost of a random write = cluster-size/avg-write-throughput + cluster-size/avg-read-throughput

for this drive:

245.7ms = cluster-size/72.768 + cluster-size/125

>>>>>>

cluster-size = 11.3 MiB, this should over-estimate the actual size..

and again, I am just doing this to get a ball-park figure…

maybe it’s 4MiB, maybe its 8MiB don’t know for sure.

Normal rotating disc drives have a “cluster-size” of 512bytes, (yes bytes).

.

.

.

.

.

The above analysis while interesting, is probably unnecessary for getting a reasonably clear idea of the performance of this SSD.

.

.

.

.

So, know what you are getting into here…

This OCZ MLC Core 64GB SSD is a fantastic drive for properly configured desktop users, read-heavy servers, and it’s even ok for large writes, but the drive is horrendously terrible for small random writes… here are the specs I claim are accurate:

for reference, the specs I claim for the 7200 rpm rotating disc drive I used for comparison:

If you buy this drive for desktop use in windows vista, make sure to use a page file on a different disc, or disable it altogether, not a bad solution since ram is so cheap these days…

Also, disable ‘System Protection’ and any other service that will write to the disc a lot.

Many things can prevent this from happening:

1. Flow control on the transmitting node.

If a Gigabit Ethernet equipped server has a 100Mb/sec client in an active data transfer, this “feature” seems to throttle transfers to faster (Gb Ethernet) clients while the transfer with the slower client is active. link

Some drivers, including Intel’s, let you turn off flow control in the device manager.

2. SAMBA/CIFS/SMB/OS inefficiency.

SMB/CIFS is very convenient, as its fully integrated into Windows, but is not the lightest or fastest method of transferring data. You can use FTP or many other methods to transfer data quickly, but most of them are not nearly as convenient as SMB. SMB2.0, which Microsoft introduced in Vista and Windows Server 2008 is much better:

(note arrows indicate direction of transfer, the first computer listed is the initiator. “2008×64<-box” means that the windows server 2008 x64 computer initiated a copy of a file from the “box” computer to itself. )

This improved throughput with SMB2.0/vista comes at a cost of CPU Utilization:

If you want high-performance, use SMB2.0, but don’t expect miracles for free.

————————————————————————————————————————–

On the topic of OS liability in network throughput, there are two registry changes needed for Vista SP1 and Server 2008:

in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Multimedia\SystemProfile

change NetworkThrottlingIndex to 0xffffffff

and change SystemResponsiveness to 0×64.

This essentially disables the Windows MultiMedia Class Scheduler Service’s ability to prioritize multimedia playback at the cost of networking performance.

3. Encryption Tax

This may have more negative effect on delay then bandwidth, since there must be processing on each end of the transfer, but nevertheless, VPN data overhead can be significant >10% (often depending on packet size distribution as some costs are fixed per-packet).

4. Gigabit Ethernet over the PCI Bus.

It is true that the 32-bit 33Mhz PCI bus has a theoretical peak bandwidth over 1Gb/sec ( 100/3 Mhz * 32 bits = ~1.067 Gb/sec )

While this exceeds 1Gb/sec that Gigabit Ethernet needs, the PCI bus has overhead, is not full duplex, and is sharing that bandwidth with multiple devices on the bus. The effective bandwidth of a regular pci bus is around 600Mb/sec half-duplex, some chipsets are better than others for actual pci bandwidth. PCIe 1x has 4Gb/sec total bandwidth per device (2Gb/sec full-duplex), so PCIe Gb ethernet interfaces have no issues with the bus as a bottleneck)

5. Hard Disc stuff.

SCSI, SAS, ATA-66/100/133, and SATA1/2 all had impressive throughput rates for their time, but the interface was never the bottleneck.

The discs themselves have sustained transfer rates (STR) limited by:

1. Linear Speed L of the Head which is a function of:

• Rotational Velocity V (revolutions per time) on most hard drives this is not dynamic.
• Location of the data r (radial distance of the head where the data is being operated on).

so L=V*2*Pi*r , in a normal 7200 revolution per minute, 3.5″ hard disc the Linear Velocity L = 7200*2*Pi*3.5/2 = ~79168 inches per minute at the outer edge, less as you get closer to the center.

this Linear Velocity reduces in a linear fashion as you approach the spindle of the disc.

2. Linear data density d, (bits per inch), which is usually proportional to the square root of the areal density (higher density means the head can traverse and r/w more sectors for a given linear velocity)

so

Sequential Throughput ~ L * d ~ 2 * Pi * V * r * d

so there you have it, in a given disc, sequential data throughput is linearly related to distance from center.

with the same rotational velocity, a 3.5″ disc will have an outer edge 40% further from the center, which means 40% faster then the outer edge of a similarly dense 2.5″ platter.

————————————————————————–

It may not be called a disc problem, but effective disc large-file transfer rate can be throttled if the data has to be fragmented on various spots on the disc, since that requires head seeks for something that could be a sequential operation. Seeks cost time and transfer no data.

Carefully chosen modern 7200rpm SATA2 high-areal density discs like the this or this can perform sustained sequential reads or writes close to 1Gb per second at the outer edge. The discs I have been messing with ( Western Digital WD6400AAKS ) have sequential performance around 900Mb/sec near the outer edge, so they will bottleneck large 1,000Mb/sec Ethernet transfers, but only a bit.

these graphs are decreasing because the program calls 100% the innermost, and 0% the outermost of the disc… also you may notice the graphs are not linear as I suggested, this is because the horizontal axis is “%” which is % of data, not % of radial distance.

I will not bore anyone with the math/logic to understand why this makes sense, but it does… and graphing it like HD-Tune did here should theoretically yield a quadratic, which it seems to by the pictures.

————————————————————————–

For my testing, I used 2 WD WD640AAKSs in Raid0:

If the data was placed at the inner edge, where throughput is only 800Mb/sec that could obviously throttle any Gigabit Ethernet transfers relying on that.

In this case I am not worried about data location because my drives are empty and windows/ntfs is fairly smart about data placement, here is where my 10GB test file was placed:

Exactly where I would want it, at the outer edge. I can rely on Windows to do this consistently because the disc is otherwise empty.

So with this scenario, I am confident the discs will not bottleneck my transfers.

6. Framing overhead

Standard Ethernet frames are 12,000 bits each, in a bus capable of doing 1,000,000,000 bits each second this can be significant framing overhead.

72,000 bit “Jumbo” frames are small enough to be effectively checked for integrity by Ethernet’s 32-bit CRC, while reducing the overhead per bit ratio for huge (compared to frame size) data transfers.

note each end of a network and all nodes in the path which are traversed must support these “jumbo” frames in order for the transaction to use them.

further reading on jumbo frames: http://sd.wareonearth.com/~phil/jumbo.html , http://docs.hp.com/en/783/jumbo_final.pdf

well, I wanted to explore this in the real world of windows on platforms with abundant processing power:

Jumbo Frames make a big difference here, but arguably more important is the observation that the transfer is a lot faster if the current holder of the file initiates the transfer, rather than the recipient.

Here is where we see the other benefit of jumbo frames, CPU utilization is less than half on the recipient of the transfer, regardless of who initiates it.

—————————————————————————————————————————–

2008×64 is a q9450 12MB 45nm quad core @ 333×8=2.66Ghz Core 2 Quad “Yorkfield” CPU

with 8GB of ram

Windows server 2008 x64

Intel 82566DC Gigabit Ethernet on PCIe w/9.11.5.7 drivers from Microsoft on 6/21/2006

DG33TL motherboard

dedicated 2xWD640AAKS in RAID0 on ich9r for network share w/ntfs cluster size of 64kB and hdd&volume cache enabled.

—————————————————————————————————————————–

Both Computers are directly connected to a Trendnet TEG-S8/A switch.

—————————————————————————————————————————–

“box” is an e6300 2MB 65nm dual core @ 333×7 = 2.33Ghz Core 2 Duo

with 4GB of ram

vista x64 sp1

Atheros L1 Gigabit Ethernet on PCIe w/2.4.7.13 drivers from Atheros on 4/28/2008

Asus G35 motherboard

dedicated 2xWD640AAKS in RAID0 on ich9r for network share w/ntfs cluster size of 64kB and hdd&volume cache enabled.

—————————————————————————————————————————–

Why does it matter which side initiates?

Before I put this to rest, Let’s examine what goes on during the transfer, to see why it is faster when the sender initiates:

well lets consider the transfer from box to 2008×64

when it’s initiated by the sender, this happens:

	sender and initiator	recipient
Network Throughput
Memory Usage

This seems pretty normal, let’s look again at the difference in transfer rates depending on the initiator.

Now lets look and see why it’s slower when the recipient initiates:

	sender	recipient and initiator
Network
Memory

Whatever exactly is going on in the sending computer when a transfer is initiated by a recipient, it’s clearly using as much RAM as it can find, until there’s none left. When it runs out of RAM, the transfer goes on w
ithout degradation of performance, which leaves me at a loss to why it needed all that ram so badly to begin with.

Let’s look at CPU use:

Although it uses more CPU cycles on the sending node, it’s clearly better on modern multi-core CPUs with SMB2.0 as-is to initiate a transfer with the sender of a file as throughput is significantly better, and it doesn’t pillage the node of its RAM.

—————————————————————————————————————————————

Conclusion:

• High Performance is possible in Windows Shares with SMB2.0
• Jumbo Frames are good
• Sending node should initiate large transfers
• Must be careful to rule out disc bottlenecks, especially important to consider data location on the drive.

* 45nm intel quad and dual core CPUs

* 65nm AMD quad and tri-core cpus with TLB-bug fixed

*2 new integrated chipsets for AMD cpus (the AMD 780g and the nvidia 8200), with great media playback capabilities

Intel is dominating the middle and high-end, while low-power (45W dual (4850e) and 65W quad (9100e)) AMD cpus paird with the new integrated chipsets are very attractive media-centric platforms.

Coming Soon:

Intel’s G45 chipset which finally brings excellent video playback, decent 3d performance, and official 16GB ddr2 support to intel CPUs.

new 45nm notebook CPUs from intel are coming imminently, lowering power and heat with small laptops. Also new notebook chipsets similar to the G45 are arriving, along with a new wireless card with wimax.

Coming not-so-soon:

Intel’s first CPU with an integrated memory controller: (no more talking to ram via a northbridge)

for those interested in performance, It’s exciting.  This true multi-core chip will run in a similar thermal envelope as the current penryns.

more on nehalem

and here it is, the macbook air:

it seems everyone thinks its a new gift from god, after the iphone…

but lets actually take a step back and notice there are non-apple products out there…

issues with the air:

uses old 65nm cpu… what about penryn?

ram is almost free, yet its impossible to upgrade the air to 4GB

only 1 usb, no gigabit ethernet, no sd-card slot…

1.8″ IDE hard disk, so not possible to install a 7200rpm drive.

thin and light, sure… but a big footprint footprint, could have less unused space around the screen and keyboard.

A technically bigger laptop like the R500 will actually seem to be smaller in person since it’s more compact.

question: does it use UEFI 2.0 so one could install vista64SP1? other macbooks do not.

I think I’ll wait for penryn sff, and frankly the macbook air seemed large in person… thin yes, but has a large footprint, a 13.3″ display with a significant border around the screen.

I might go for another 12.1 like the toshiba R500, but again, I’ll wait for penryn.