Downloading public court documents costs a dime a page—is that legal? 0 25

Downloading public court documents costs a dime a page—is that legal?

Tear down this wall? —

Federal courts use hefty PACER fees to pay for non-PACER projects.


Chief Justice John Roberts.

Enlarge / Chief Justice John Roberts did not actually use PACER fees to buy a new chair. That’s just a hypothetical example.

Mark Wilson/Getty Images

If you need documents from federal court cases, you’re in luck. Almost every brief, exhibit, and legal ruling is available for download from the judiciary’s PACER website. But there’s a catch: documents cost 10 cents a page.

In 2016, three nonprofit organizations, the National Veterans Legal Services Program, the National Consumer Law Center, and the Alliance for Justice sued the federal courts—in federal court. The class action lawsuit, filed on behalf of all fee-paying PACER users, argued that these hefty charges were illegal. Federal law allows the courts to charge fees “only to the extent necessary” to provide public access to information. Over the last 15 years, the cost of storage and bandwidth has plunged. Yet PACER’s fees have actually risen from 7 cents to 10 cents. These fees have raised far more money than it costs to run the PACER system: $146 million in 2016 alone.

In a 2018 ruling, Judge Ellen Huvelle largely agreed with the plaintiffs, concluding that the courts are breaking the law by spending PACER money on non-PACER projects like installing flat-screen TVs in courtrooms and sending electronic notifications to bankruptcy creditors. On Monday, the case reached the Court of Appeals for the Federal Circuit, where three judges heard oral arguments from each side. Judges seemed skeptical of the arguments of government lawyers representing the judiciary.

Before I explain the arguments, I should note that I’m not a neutral observer on this issue. As a reporter, I’m a frequent user of PACER; Ars Technica will probably get some money back if the plaintiffs win this lawsuit. Also, a decade ago, I helped create RECAP, a browser extension that helps PACER users share documents with each other and avoid paying PACER fees. I’ve long been on record arguing that public court documents should be free to the public—not locked behind a paywall.

The current lawsuit wouldn’t go that far. Both sides agree that the courts are allowed to charge something for judicial documents. But the two sides in Monday’s oral arguments disagreed radically about how much the documents should cost. Plaintiffs argued that the law only allows the courts to charge the marginal cost of distributing documents—a tiny fraction of the current fee. The Administrative Office of the Courts, on the other hand, has argued for an expansive interpretation of the law that allows them to charge as much as they want and to spend it on anything related to distributing information electronically to the public.

In her 2018 ruling, Judge Huvelle charted a middle course. She ruled that the courts could not only charge for the cost of delivering a particular document to a particular customer, but also for the costs of maintaining the infrastructure behind the PACER system. That includes CM/ECF, the website that litigants and judges use to upload, organize, and view case documents. She concluded that these costs are relevant because PACER is fundamentally a public-facing front-end for the CM/ECF system. But she held that the courts couldn’t use PACER fees to pay for projects completely unrelated to PACER.

“We’re redecorating all judges’ offices with gold plate”

Monday’s oral arguments (MP3 recording here) didn’t go well for Alisa Klein, the government lawyer representing the judiciary. At one point an exchange got so testy that Judge Raymond Clevenger snapped, “Do you have a lot of trouble answering questions generally in life or just when you come in front of the court?”

Rather than directly defending the courts’ use of PACER fees for non-PACER purposes, Klein spent most of her time arguing that the courts shouldn’t be hearing the case at all—an issue known as “standing” in legal jargon. While the law restricts how fees can be spent, she said, Congress didn’t intend to let individual PACER users sue the courts if the law wasn’t followed.

The judges seemed skeptical. Clevenger asked incredulously whether it would be legal for PACER fees to be used to replace “the curtains at the Supreme Court” and to buy “the Chief Justice’s new chair.”

“We’re redecorating all judges’ offices with gold plate,” he said sarcastically. Under the judiciary’s theory, he said, “there’s absolutely no remedy” for this kind of illegal spending.

But Klein pointed out that the judiciary’s budget gets reviewed and approved every year by Congress. If Congress doesn’t want the courts spending PACER fees on gold-plated office renovations, Congress can easily nix expenses it doesn’t like. This annual process of oversight and appropriations, not lawsuits from private citizens, should shape spending decisions by the courts, she argued.

If the judges do decide that the courts have overcharged customers, they’ll be left with the tricky problem of deciding how much was overcharged. Deepak Gupta, the lawyer who represented the non-profit plaintiffs, said he didn’t have enough information to draw a clear line between permitted and illegal uses. He suggested that the judges send the case back down to the lower courts with instructions to dig into the courts’ budgets, uncover more information, and then rule on the issue.

While this lawsuit might force the courts to reduce PACER fees, eliminating the paywall entirely will probably take action from Congress. Last year, Rep. Doug Collins (R-Ga.) introduced legislation requiring the courts to make PACER documents available free of charge. But so far the bill hasn’t gotten much traction.

Read More

Previous ArticleNext Article

Leave a Reply

Your email address will not be published. Required fields are marked *

Google Maps gets new icon, tweaked UI for 15th birthday 0 28

Google Maps gets new icon, tweaked UI for 15th birthday

OK, but what about dark mode? —

There’s a new icon, new tabs, and the death of the hamburger button.


  • Left: the new Google Maps logo. Right: the old logo.

  • The new logo loosely follows Google’s new icon style, which is slowly creeping across the app lineup.

  • Here’s the “new” Google Maps. There are two new tabs at the bottom and “For You” was renamed to “Updates.” Also the hamburger button is dead.

  • Here are the two new tabs. Neither of these sections are new; they’re just tab buttons now.

Google Maps is turning 15 this year, and Google is celebrating with a new icon and a few UI tweaks.

First, the Google Maps icon is no longer, well, a map and is now a multi-colored map pin. Like all of Google’s other recent icons, Maps’ icon follows a formula of outfitting a simple shape or letter with the colors red, blue, green, and yellow, and calling it a day. You can expect this new icon to pop up on your phone sometime soon.

The bottom tab navigation is going to switch from three tabs to five, with new tabs for “Saved” (Saved places), “Contribute,” and “Updates.” None of these sections really represent new features, they just used to live in the left-side navigation drawer and now they have top-level access via the tab bar. Speaking of the left-side navigation drawer, the hamburger button that used to open it is dead. Presumably, the settings and other miscellaneous menu options will live under the account switcher, accessible via your profile picture on the right side of the search bar.

Crowdsourcing has always been a major source of information for Google Maps, and soon Google says it will start surfacing information based on surveys that past Google Maps users have filled out. Things like the temperature of a mass transit car, accessibility, and the security level will soon be surfaced when you’re planning your travels.

Google’s last update concerns the company’s augmented reality “Live View” mode in Maps. Instead of unreliable smartphone compass hardware, Live View cross-references your camera feed with Street View imagery to figure out the direction you’re facing. The feature is meant to help with walking navigation in big cities, where getting that first turn right can often be a challenge. Live View works, but location compatibility is currently extremely limited right now. Google says it will be “expanding Live View and testing new capabilities” over the coming months but doesn’t provide any detail on what, exactly, that means.

Read More

How fast are your disks? Find out the open source way, with fio 0 19

How fast are your disks? Find out the open source way, with fio

Throughput and latency and IOPS, oh my —

The most reliable way to test disks is down-and-dirty, on the command line.


Ars Technica does not recommend removing the protective cover from your hard disk or setting it on fire in production settings.

Enlarge / Ars Technica does not recommend removing the protective cover from your hard disk or setting it on fire in production settings.

Storage benchmarking—much like Wi-Fi benchmarking—is a widely misunderstood black art. Admins and enthusiasts have for decades been tempted to just “get the big number” by reading or writing a large amount of data to a disk, getting a figure in MB/sec, and calling it a day. Unfortunately, the actual workload of a typical disk doesn’t look like that—and that “simple speed test” doesn’t reproduce a lot of the bottlenecks that slow down disk access in real-world systems.

The most realistic way to test and benchmark disks is, of course, to just use them and see what happens. Unfortunately, that’s neither very repeatable, nor is it simple to analyze. So we do want an artificial benchmarking tool—but we want one that we can use intelligently to test storage systems across realistic scenarios that model our day-to-day usage well. Fortunately, we don’t have to invent such a tool—there’s already a free and open source software tool called fio, and it’s even cross-platform!

We’re going to walk you through some simple but effective uses of fio on Windows, Mac, and Linux computers—but before we do that, let’s talk a little bit about storage systems from a more basic perspective.

Throughput, latency, IOPS and cache

Throughput

Throughput, measured most commonly in storage systems in MB/sec, is the most commonly used way to talk about storage performance. There are several choke points in a storage system for throughput—first and foremost, there’s the speed of the physical medium itself. If you’ve got a single head on a conventional rust disk spinning at 7200RPM, the rate you can get data on or off that disk will be limited by the number of physical sectors/blocks passing beneath the head. You’re also limited by the bandwidth of your controller and cabling—for example, modern SATA links typically operate at 6Gbps, while modern SAS links can operate up to 22.5Gbps.

Things get a little extra complicated here, because we’re mixing units—notice the big B in MB/sec, and the small b in Gbps. That’s the difference between bytes and bits. You divide Gbps by 8 to get GB/sec, then multiply by 1024 to get MB/sec. So a SATA-3 6Gbps link can theoretically move up to 768MB/sec. You can’t actually move data across the SATA or SAS bus at the full theoretical link speed, but you can get fairly close. It’s also worth noting that most SATA controllers won’t move much more data than a single link can manage, even with many disks connected to the controller—so it’s common to see even lots of very fast solid state drives in an array bottlenecking at around 700 MB/sec.

Latency

Latency is the flip side of the same performance coin. Where throughput refers to how many bytes of data per second you can move on or off the disk, latency—most commonly measured in milliseconds—refers to the amount of time it takes to read or write a single block. Most of the worst storage bottlenecks are latency issues that affect throughput, not the other way around.

In conventional spinning rust disks, there are two major sources of latency: rotational latency, and seek latency. The seek latency is how long it takes to move the mechanical arm the disk head is mounted on to the correct track on disk. Once the head has moved to the correct track, the drive then has to wait for the correct sector to rotate beneath the head—that’s the rotational latency. The combination of seek and rotational latency usually adds up to somewhere between 15ms and 25ms.

You can see how latency affects throughput by thought experiment. If we have a reasonably fast spinning disk with a maximum throughput of 180MB/sec and a total access latency of 16ms, and we present it with a maximally fragmented workload—meaning that no two blocks have been written/are being written in sequential order—we can do a little math to come up with that throughput. Assuming 4KB physical blocks on disk, 4KB per seek divided by 0.016 seconds per seek = only 250KB/sec. Ouch!

IOPS

Short for Input/Output Operations Per Second, IOPS is the metric of measurement you’ll most commonly hear real storage engineers discussing. It means exactly what it sounds like—how many different operations can a disk service? In much the same way, “throughput” usually refers to the maximal throughput of a disk, with very large and possibly sequential reads or writes, IOPS usually refers to the maximal number of operations a disk can service on the low end—4K random reads and writes.

Solid state disks don’t suffer from seek or rotational latency, but 4K random Input/Output (I/O) does still present them with problems. Under the hood, a consumer SSD isn’t really a single “disk”—it’s a RAID array in a little sealed box, with its own complex controller inside the disk itself managing reads and writes. The SSD controller itself tries to stripe writes across multiple channels of physical flash media in parallel—and, if the user is lucky, the writes which got striped out evenly across those channels will also be read the same way, maximizing throughput.

When a solid state disk is presented with a 4K random I/O workload, if it can’t figure out some way to aggregate and parallelize the requests, it will end up bottlenecking at much lower speeds, dictated by how quickly a single cell of flash media can read or write a block of data. The impact isn’t as dramatic as it would be on a rust disk, but it’s still significant—where a rust disk capable of 180MB/sec of throughput might plummet to 250KB/sec of 4K random I/O, a SSD capable of 500MB/sec could drop to around 40MB/sec.

Although you can discuss throughput in terms of 4K random I/O, and IOPS in terms of sequential 1MB I/O, that’s not how each term is typically used. You should generally expect throughput to be discussed in terms of how much data a disk moves under optimal conditions, and IOPS in terms of the “low end grunt” the disk is capable of even under the worst workload. For typical desktop PC use, IOPS is far more important than throughput—because there’s lots of that slow 4K random I/O, and it slows the whole system down when it happens.

Cache

As we’ve seen above, non-optimized workloads hurt performance, and hurt them badly. Thankfully for users, decades of research and development have presented us with all manner of tricks to keep from exploring the worst performance characteristics of our storage—especially rust storage. Operating systems use both read caches and write buffers to minimize the number of seeks necessary in operation and avoid the need to keep reading frequently needed blocks from storage over and over.

Write buffers allow the operating system to store up lots of small I/O requests and commit them to disk in large batches. One megabyte is a very small amount of data, but it still comes out to 256 4KB blocks—and if you must write each of those blocks out with individual operations, you might tie up your disk’s entire service capacity for a full second. On the other hand, if you can aggregate those 256 blocks in a write buffer and then flush them out in a single operation, you avoid all that access latency, and the same amount of data can be saved in a hundredth of a second or less. This aggregation can also greatly help with read speeds later. If most of the same blocks need to be read as a group later, the drive can avoid seeking between them since they were all written as a group in the first place.

Read cache keeps the system from having to tie up storage with unnecessary repeated requests for the same blocks over and over again. If your operating system has plenty of RAM available, each time it reads data from disk, it keeps a copy of it lying around in memory. If another program asks for the same blocks later, the operating system can service that request directly from the cache—which keeps the drive’s limited resources available for either read or write requests, which must hit the actual disk.

Some models of SSD have an additional non-volatile write cache on the disk itself, made of a faster and more expensive type of flash media. For example, a TLC or QLC (Quad Layer Cell) SSD might have a few gigabytes of MLC (Multi-Layer Cell) media to use as a buffer for writes; this enables the SSD to keep up with the writes demanded by a typical desktop workload using the faster MLC buffer—but if presented with sustained heavy writes for too long a time, the fast MLC buffer fills, and throughput drops to what the slower TLC or QLC media can manage. This can frequently be a “fall off the cliff” type scenario, since the slower media will typically not only have to sustain ongoing writes, but do so while continuing to stream out the already-accepted writes from the fast MLC buffer.

Modeling storage access realistically

Now that we understand a little about the pain points in a storage system, it’s pretty obvious that we shouldn’t just use a simple tool like dd to read or write huge chunks of data—and generate huge numbers. Those huge numbers don’t really correlate very well with how each disk performs under more realistic workloads—so, we want to generate more realistic access patterns to test with.

This is where fio comes in. Fio is short for Flexible Input/Output tester and can be configured to model nearly any storage workload under the sun. Real storage engineers—at least, the ones who are doing their jobs right—will first analyze the actual storage access patterns of a server or service, then write fio scripts to model those exact patterns. In this way, they can test a disk or array not only for its general performance, but its performance as very specifically applicable to their exact workload.

We’re not going to be quite that specific here, but we will use fio to model and report on some key usage patterns common to desktop and server storage. The most important of these is 4K random I/O, which we discussed at length above. 4K random is where the pain lives—it’s the reason your nice fast computer with a conventional hard drive suddenly sounds like it’s grinding coffee and makes you want to defenestrate it in frustration.

Next, we look at 64K random I/O, in sixteen parallel processes. This is sort of a middle-of-the-road workload for a busy computer—there are a lot of requests for relatively small amounts of data, but there are also lots of parallel processes; on a modern system, that high number of parallel processes is good, because it potentially allows the OS to aggregate lots of small requests into a few larger requests. Although nowhere near as punishing as 4K random I/O, 64K random I/O is enough to significantly slow most storage systems down.

Finally, we look at high-end throughput—some of the biggest numbers you can expect to see out of the system—by way of 1MB random I/O. Technically, you could still get a slightly bigger number by asking fio to generate truly sequential requests—but in the real world, those are vanishingly rare. If your OS needs to write a couple of lines to a system log, or read a few KB of data from a system library, your “sequential” read or write immediately becomes, effectively, 1MB random I/O as it shares time with the other process.

Installing fio

Windows

You can find Windows installers for fio at https://bsdio.com/fio/. Note that you may get Smartscreen warnings when running one of these installers, since they are not digitally signed. These packages are provided by Rebecca Cran and are available without warranty.

Note that Windows has a limited selection of ioengines available, which will inform your selection of command line arguments later. For the most part, Windows users should use --ioengine=windowsaio (Asynchronous Input/Output) with their fio arguments.

Linux / FreeBSD

The instructions for users of Linux and BSD distributions are a little different from one to another, but fio is in nearly all main repositories—so it boils down to install fio for the vast majority.

Debian or Ubuntu: apt install fio

FreeBSD: pkg install fio

CentOS (and Red Hat Enterprise Linux) have rather more limited main repositories than most distributions; if you haven’t already, you’ll need to add the EPEL repository to CentOS/RHEL to get fio.

CentOS/RHEL: sudo yum install epel-release -y ; sudo yum install fio

You get the idea.

MacOS

On a Mac, you’ll want to install fio via brew. If you don’t already have brew installed, at the Terminal, issue the following command:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

On the one hand, the above is abominable procedure; on the other hand, you can confirm that the script being pulled down tells you everything it’s going to do, before it does it, and pauses to allow you to consent to it. If you’re sufficiently paranoid, you may wish to download the file, inspect it, and then run it as separate steps instead. Note that the homebrew install script does not need sudo privileges—and will, in fact, refuse to run at all if you try to execute it with sudo.

With Brew installed, you can now install fio easily:

brew install fio

Using fio

Now you can use fio to benchmark storage. First, change directory to the location you actually want to test: if you run fio in your home directory, you’ll be testing your computer’s internal disk, and if you run it in a directory located on a USB portable disk, you’ll be benchmarking that portable disk. Once you’ve got a command prompt somewhere in the disk you want to test, you’re ready to actually run fio.

Baby’s first fio run

First, we’ll examine the syntax needed for a simple 4K random write test. (Windows users: substitute --ioengine=windowsaio for --ioengine=posixaio in both this and future commands.)

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1

Let’s break down what each argument does.

--name= is a required argument, but it’s basically human-friendly fluff—fio will create files based on that name to test with, inside the working directory you’re currently in.

--ioengine=posixaio sets the mode fio interacts with the filesystem. POSIX is a standard Windows, Macs, Linux, and BSD all understand, so it’s great for portability—although inside fio itself, Windows users need to invoke --libengine=windowsaio, not --libengine=posixaio, unfortunately. AIO stands for Asynchronous Input Output and means that we can queue up multiple operations to be completed in whatever order the OS decides to complete them. (In this particular example, later arguments effectively nullify this.)

--rw=randwrite means exactly what it looks like it means: we’re going to do random write operations to our test files in the current working directory. Other options include seqread, seqwrite, randread, and randrw, all of which should hopefully be fairly self-explanatory.

--bs=4k blocksize 4K. These are very small individual operations. This is where the pain lives; it’s hard on the disk, and it also means a ton of extra overhead in the SATA, USB, SAS, SMB, or whatever other command channel lies between us and the disks, since a separate operation has to be commanded for each 4K of data.

--size=4g our test file(s) will be 4GB in size apiece. (We’re only creating one, see next argument.)

--numjobs=1 we’re only creating a single file, and running a single process commanding operations within that file. If we wanted to simulate multiple parallel processes, we’d do, eg, --numjobs=16, which would create 16 separate test files of --size size, and 16 separate processes operating on them at the same time.

--iodepth=1 this is how deep we’re willing to try to stack commands in the OS’s queue. Since we set this to 1, this is effectively pretty much the same thing as the sync IO engine—we’re only asking for a single operation at a time, and the OS has to acknowledge receipt of every operation we ask for before we can ask for another. (It does not have to satisfy the request itself before we ask it to do more operations, it just has to acknowledge that we actually asked for it.)

--runtime=60 --time_based Run for sixty seconds—and even if we complete sooner, just start over again and keep going until 60 seconds is up.

--end_fsync=1 After all operations have been queued, keep the timer going until the OS reports that the very last one of them has been successfully completed—ie, actually written to disk.

Interpreting fio’s output

This is the entire output from the 4K random I/O run on my Ubuntu workstation:

root@banshee:/tmp# fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=posixaio
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][eta 00m:00s]                          
random-write: (groupid=0, jobs=1): err= 0: pid=16109: Wed Feb  5 15:09:36 2020
  write: IOPS=32.5k, BW=127MiB/s (133MB/s)(8192MiB/64602msec); 0 zone resets
    slat (nsec): min=250, max=555439, avg=1388.31, stdev=833.19
    clat (nsec): min=90, max=20251k, avg=9642.34, stdev=179381.02
     lat (usec): min=3, max=20252, avg=11.03, stdev=179.39
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    4], 10.00th=[    4], 20.00th=[    5],
     | 30.00th=[    6], 40.00th=[    6], 50.00th=[    7], 60.00th=[    8],
     | 70.00th=[    9], 80.00th=[   10], 90.00th=[   11], 95.00th=[   12],
     | 99.00th=[   17], 99.50th=[   20], 99.90th=[   43], 99.95th=[   77],
     | 99.99th=[12387]
   bw (  KiB/s): min=22256, max=613312, per=100.00%, avg=335527.28, stdev=162778.06, samples=50
   iops        : min= 5564, max=153328, avg=83881.88, stdev=40694.66, samples=50
  lat (nsec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=13.96%, 10=68.85%, 20=16.68%, 50=0.41%
  lat (usec)   : 100=0.04%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=6.35%, sys=11.96%, ctx=2348924, majf=0, minf=48
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2097153,0,1 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

Disk stats (read/write):
    md0: ios=71/749877, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=351/737911, aggrmerge=0/12145, aggrticks=1875/260901, aggrin_queue=30910, aggrutil=83.73%
  sdb: ios=342/737392, merge=0/12663, ticks=1832/241034, in_queue=28672, util=83.35%
  sda: ios=361/738430, merge=0/11628, ticks=1918/280768, in_queue=33148, util=83.73%

This may seem like a lot. It is a lot! But there’s only one piece you’ll likely care about, in most cases—the line directly under “Run status group 0 (all jobs):” is the one with the aggregate throughput. Fio is capable of running as many wildly different jobs in parallel as you’d like to execute complex workload models. But since we’re only running one job group, we’ve only got one line of aggregates to look through.

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

First, we’re seeing output in both MiB/sec and MB/sec. MiB means “mebibytes”—measured in powers of two—where MB means “megabytes,” measured in powers of ten. Mebibytes—1024×1024 bytes—are what operating systems and filesystems actually measure data in, so that’s the reading you care about.

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

In addition to only having a single job group, we only have a single job in this test—we didn’t ask fio to, for example, run sixteen parallel 4K random write processes—so although the second bit shows minimum and maximum range, in this case it’s just a repeat of the overall aggregate. If we’d had multiple processes, we’d see the slowest process to the fastest process represented here.

Run status group 0 (all jobs):
  WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=8192MiB (8590MB), run=64602-64602msec

Finally, we get the total I/O—8192MiB written to disk, in 64602 milliseconds. Divide 8192MiB by 64.602 seconds, and surprise surprise, you get 126.8MiB/sec—round that up to 127MiB/sec, and that’s just what fio told you in the first block of the line for aggregate throughput.

If you’re wondering why fio wrote 8192MiB instead of only 4096MiB in this run—despite our --size argument being 4g, and only having one process running—it’s because we used --time_based and --runtime=60. And since we’re testing on a fast storage medium, we managed to loop through the full write run twice before terminating.

You can cherry pick lots more interesting stats out of the full fio output, including utilization percentages, IOPS per process, and CPU utilization—but for our purposes, we’re just going to stick with the aggregate throughput from here on out.

Ars recommended tests

Single 4KiB random write process

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

This is a single process doing random 4K writes. This is where the pain really, really lives; it’s basically the worst possible thing you can ask a disk to do. Where this happens most frequently in real life: copying home directories and dotfiles, manipulating email stuff, some database operations, source code trees.

When I ran this test against the high-performance SSDs in my Ubuntu workstation, they pushed 127MiB/sec. The server just beneath it in the rack only managed 33MiB/sec on its “high-performance” 7200RPM rust disks… but even then, the vast majority of that speed is because the data is being written asynchronously, allowing the operating system to batch it up into larger, more efficient write operations.

If we add the argument --fsync=1, forcing the operating system to perform synchronous writes (calling fsync after each block of data is written) the picture gets much more grim: 2.6MiB/sec on the high-performance SSDs but only 184KiB/sec on the “high-performance” rust. The SSDs were about four times faster than the rust when data was written asynchronously but a whopping fourteen times faster when reduced to the worst-case scenario.

16 parallel 64KiB random write processes

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

This time, we’re creating 16 separate 256MB files (still totaling 4GB, when all put together) and we’re issuing 64KB blocksized random write operations. We’re doing it with sixteen separate processes running in parallel, and we’re queuing up to 16 simultaneous asynchronous ops before we pause and wait for the OS to start acknowledging their receipt.

This is a pretty decent approximation of a significantly busy system. It’s not doing any one particularly nasty thing—like running a database engine or copying tons of dotfiles from a user’s home directory—but it is coping with a bunch of applications doing moderately demanding stuff all at once.

This is also a pretty good, slightly pessimistic approximation of a busy, multi-user system like a NAS, which needs to handle multiple 1MB operations simultaneously for different users. If several people or processes are trying to read or write big files (photos, movies, whatever) at once, the OS tries to feed them all data simultaneously. This pretty quickly devolves down to a pattern of multiple random small block access. So in addition to “busy desktop with lots of apps,” think “busy fileserver with several people actively using it.”

You will see a lot more variation in speed as you watch this operation play out on the console. For example, the 4K single process test we tried first wrote a pretty consistent 11MiB/sec on my MacBook Air’s internal drive—but this 16-process job fluctuated between about 10MiB/sec and 300MiB/sec during the run, finishing with an average of 126MiB/sec.

Most of the variation you’re seeing here is due to the operating system and SSD firmware sometimes being able to aggregate multiple writes. When it manages to aggregate them helpfully, it can write them in a way that allows parallel writes to all the individual physical media stripes inside the SSD. Sometimes, it still ends up having to give up and write to only a single physical media stripe at a time—or a garbage collection or other maintenance operation at the SSD firmware level needs to run briefly in the background, slowing things down.

Single 1MiB random write process

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

This is pretty close to the best-case scenario for a real-world system doing real-world things. No, it’s not quite as fast as a single, truly contiguous write… but the 1MiB blocksize is large enough that it’s quite close. Besides, if literally any other disk activity is requested simultaneously with a contiguous write, the “contiguous” write devolves to this level of performance pretty much instantly, so this is a much more realistic test of the upper end of storage performance on a typical system.

You’ll see some kooky fluctuations on SSDs when doing this test. This is largely due to the SSD’s firmware having better luck or worse luck at any given time, when it’s trying to queue operations so that it can write across all physical media stripes cleanly at once. Rust disks will tend to provide a much more consistent, though typically lower, throughput across the run.

You can also see SSD performance fall off a cliff here if you exhaust an onboard write cache—TLC and QLC drives tend to have small write cache areas made of much faster MLC or SLC media. Once those get exhausted, the disk has to drop to writing directly to the much slower TLC/QLC media where the data eventually lands. This is the major difference between, for example, Samsung EVO and Pro SSDs—the EVOs have slow TLC media with a fast MLC cache, where the Pros use the higher-performance, higher-longevity MLC media throughout the entire SSD.

If you have any doubt at all about a TLC or QLC disk’s ability to sustain heavy writes, you may want to experimentally extend your time duration here. If you watch the throughput live as the job progresses, you’ll see the impact immediately when you run out of cache—what had been a fairly steady, several-hundred-MiB/sec throughput will suddenly plummet to half the speed or less and get considerably less stable as well.

However, you might choose to take the opposite position—you might not expect to do sustained heavy writes very frequently, in which case you actually are more interested in the on-cache behavior. What’s important here is that you understand both what you want to test, and how to test it accurately.

Conclusions

Using fio is definitely an exercise for the true nerd (or professional). It won’t hold your hand, and although it provides incredibly detailed results, they’re not automatically made into pretty graphs for you.

If all of this feels like far too much work, you can also find simpler-to-use graphical tools, such as HD Tune Pro for Windows. HD Tune Pro costs $35, or there’s a limited-capability non-Pro version that is free for personal use. It’s a good tool, and it’ll make shiny graphs—but it’s considerably more limited for advanced users, and the price of the make-it-easy user interface is that you’re much further removed from the technical reality of what you’re doing.

Learning to use fio means really learning the difference between asynchronous and synchronous writes and knowing for absolute certain what it’s going to do at a very low level on an individual argument basis. You can’t be as certain of what tools like HD Tune Pro are actually doing under the hood—and having to deal with different tools on different operating systems means it’s more difficult to directly compare and contrast results as well.

Read More