Benchmark Tests - Continued 

Here we proceed to actually testing in an XP SP2 boot environment on both drives. Though we here have little use for Futuremark's PCMark2005 as a real-world benchmark, we decided to use it in testing here. Rather than focus on the overall PCMark scores, we drilled through the individual test results and sniped particular performance tests that either deviated significantly, or directly related in some fashion to disk subsystem performance.

As you can see in the accompanying graph, PCMark guages the Supertalent drive rather poorly in XP Startup speeds and in it's HDD General Usage pattern test. In it's somewhat more real world comparisons of File Compression, File Encryption, and Virus Scan tests (all of which work the disk subsystem to varying degrees) the performance gap narrows significantly. Here, the Hitachi drive only decisively wins in the Virus Scan test. The results leave us even less impressed than usual with PCMark2005 as a valid real world benchmark. The reasons will become more apparent later on.

IOMeter, originally a storage subsystem testing tool of Intel's, is now an open source project. IOMeter is a purely synthetic benchmarking system with a vast array of testing capabilities. So vast in fact, that it's use as a canned benchmark is all but useless. There are simply too many possible ways to use IOMeter for it to be any sort of guidepost beyond it's ability to examine specific and finite scenarios. It's because of this, that we decided to use IOMeter ironically enough. We wanted to use it to explore what relation file sizes and random access workload would have to performance when comparing magnetic and solid state storage. Since the way mechanical hard drives and solid state drives function are so diametrically different from one another, we used IOMeter to try to relate some of these differences.

We ran two basic workload scenario's on the drives...

Scenario #1

• Transfer Request Size - 1 Megabyte.
• Percentage Random/Sequential Distribution to 50% Random/Sequential.
• Change Percentage Read/Write Distribution to 50% Read 50% Write.
• Number of Worker Tasks - 8

Scenario #2

• Transfer Request Size - 32KB.
• Percentage Random/Sequential Distribution to 50% Random/Sequential.
• Change Percentage Read/Write Distribution to 50% Read 50% Write.
• Number of Worker Tasks - 8

As you can see, the only deviance from the scenario's is the Transfer Request Size. The Supertalent drive obviously has much slower sustained and burst transfer rates than the Hitachi drive, and it's write speed is even slower. By utilizing a much smaller Transfer Request Size, we wanted to see if eliminating the read/write bottleneck would allow the Supertalent drive to leverage it's much faster 1ms random seek times against the Hitachi drive.

In the first scenario, it's quite clear that when dealing with larger read/writes the Supertalent drives transfer rate restrictions are hurting it's performance. Still, it's not to the level you'd expect if you merely looked at published specifications. At the other extreme, when dealing with tiny read/writes the situation reverses significantly, with the higher random access seek time clearly hurting the Hitachi drive when asked to so many much smaller random seeks.

Keep in mind that this is a deliberately weighted set of scenarios, and neither test should be an indication of real world use or performance. What it does show however is that the actual types of transfers requested and the size of those transfers weighs heavily on how either drive type will perform. An apples-to-apples comparison of specification sheets alone just cannot show what's going on between these two very different drive topologies. As we proceed with some more "real world" examples of the drive comparisons, you'll perhaps begin to understand why we bothered with this particular test.

We came up with three arbitrary real world scenario's to test the drives. The first two involve games, and timing the load times of particular levels. The last involved stopwatch testing Windows XP and measuring startup, shutdown, resume, and suspend times, and copying files to and from the drives using Explorer.

The first game scenario involved Half Life 2 - Episode 1. We measured level load times with a stopwatch of the first levels in the "Undue Alarm" and "Direct Intervention" sections of the game. As is borne out with the graph below, the Supertalent drive ended up marginally faster than the Hitachi, despite having a much slower sustained and burst transfer rate.

The next scenario involved loading the demo version of the Doom3 engine based game, Prey. We followed a similar testing strategy, testing level load times at the beginning of the games "Last Call" and "Escape Velocity" levels. Again, we see the Supertalent drive marginally faster than the Hitachi drive. It's clear here that the distribution of many many small files (as is typical in game level design and packaging) being loaded for a given game level, causes the Hitachi drives random access speed to become the most telling factor in it's performance, despite it having a much higher theoretical throughput level than the Supertalent drive.

The next battery of tests involved stopwatch timing XP during four scenarios. Startup, Shutdown, Suspend, and Resume. As you can see from the graph, these tests largely favor the Supertalent drive, with the exception of the Windows XP Startup test. Still, the deviance between the two drives in this particular instance isn't statistically significant. The interesting aspect of the Startup test is how different an actual timed Startup was from PCMark2005's transfer rate estimate. PCMark2005 takes another one on the chin.

The last test involved doing a set of simple file copies, from within Explorer. We copied a digital camera image store folder consisting of 113 files comprising 247MB from an external Seagate FreeAgent 320 USB drive to both drives, and then did the reverse, copying the same folder back to the root of the Seagate FreeAgent drive. Here we see the Supertalent drives biggest weakness coming in to play in the Copy-To test. With the drive sporting write speeds nearly ½ of it's read speeds, it should come as no shock that the Hitachi drive walks away from it here being almost 11 seconds faster. The Copy-From test is less conclusive but still gives a couple of seconds advantage to the Hitachi drive.

We'll wrap things up here with some general comments on perception from day to day use. We spent a good solid day with each drive install, and our impression of the Supertalent drive in daily use is quite favorable. It's rather eerie using a system with an utterly noiseless drive, and you don't realize just how many audible cues you get from using a standard hard drive in day to day use. Some applications, such as using Adobe Acrobat 8 to open and view large PDF files, or using Microsoft Office 2003 (to write this article actually) would display an ever so slight "hesitation" during initial load on the Supertalent drive, though this may be a more subjective interpretation of things owing to the lack of the aforementioned audible cues. Game level loads felt "smoother" somehow, and here again we feel after thinking about this for a while that this might also relate to the lack of obvious audible cue's from the Supertalent drive. The average hard drive audibly makes you keenly aware of when it's working it's tail off, and the lack of that experience with the Supertalent drive is probably the decisive factor in this interpretation.

With this, we think it's about time to wrap things up...