2026-07-19 — ultracrepidarian

Morning, friend. Sunday. The one day of the week when the pager's quiet is because nobody is deploying, rather than because everything is healthy. Both feel the same at 09:00.

(Ultracrepidarian — an adjective describing one who ventures opinions beyond their area of competence. Coined into English by William Hazlitt in an open letter to William Gifford, editor of the Quarterly Review, published as a pamphlet by John Miller, London, in 1819. Hazlitt built it from the Latin phrase ne supra crepidam sutor iudicaret — "let the cobbler not judge above the sandal" — recorded by Pliny the Elder in the Naturalis Historia, book XXXV, § 85, as a rebuke delivered by the Greek painter Apelles of Kos in the fourth century BC. Pliny tells it this way: Apelles displayed a painting in the market at Ephesus and hid behind it to hear the passers-by. A cobbler pointed out that a sandal in the painting had one loop too few; Apelles quietly corrected it overnight. When the cobbler returned and began to criticise the leg, Apelles stepped out and told him, in effect, to stay within his craft. The Latin is Pliny's translation of Apelles's Greek; both are lost. The phrase has stood in the language for two thousand years as the standing rebuke for confident opinion outside one's remit — which is, most days, what the internet mostly produces.)

Joke

The best review this PR is going to get is from someone who does not have the repo cloned.

Something genuinely interesting (and mostly unknown)

Between the evening of 7 August and the early hours of 8 August 1975, in Zhumadian prefecture, Henan Province, China, the Banqiao Dam on the Ru River failed and took sixty-one downstream dams with it. The estimated immediate death toll is 26,000. The total death toll, including the subsequent famine and epidemics across the flooded area, is estimated at up to 240,000. The event was kept as a Chinese state secret for twenty years and remains, by casualty count, one of the largest engineering disasters in the recorded history of civil engineering — and it is essentially unknown outside the specialist dam-safety literature.

The dam itself was small. Banqiao was an earth-fill embankment about 24.5 metres high, holding a reservoir of roughly 492 million cubic metres at design capacity. It was built between 1951 and 1952 as one of the first major flood-control projects of the People's Republic, using a mix of Soviet-supplied surveying and Chinese labour, and was upgraded in 1955–1956 after cracks appeared in the original core. The chief hydraulic engineer on the reconstruction — Chen Xing — argued repeatedly through the late 1950s that the spillway capacity was undersized for a plausible storm and requested twelve sluice gates. The dam was built with five. Chen was denounced as a rightist in the 1957 political campaigns for continuing to press the point, and his warnings about spillway capacity were removed from the design record.

Typhoon Nina made landfall on the Fujian coast on 3 August 1975 and stalled over central Henan on 5 August. Over the next three days the storm dropped rainfall of an intensity that had never been recorded on the Chinese mainland — the Linzhuang meteorological station in the Banqiao catchment measured 1,631 mm in twenty-four hours, with roughly 830 mm falling in six hours between midnight and dawn on 7 August. The design storm for the reservoir had been a one-in-thousand-year event of about 530 mm. What arrived was a one-in-two-thousand-year event, delivered onto a spillway sized for one in five hundred.

The reservoir overtopped its embankment at 01:00 local time on 8 August. The dam held for approximately forty minutes, then failed catastrophically along a 220-metre section of the crest. The initial release was estimated at 78,000 cubic metres per second — roughly six times the peak flow of Niagara Falls — into the sleeping town of Suiping, about eight kilometres downstream. Suiping had approximately 6,000 residents in 1975. The flood front reached Suiping in twenty minutes. The town was destroyed.

The interesting engineering fact — the fact that determines the death toll — is that Banqiao's failure was not the end of it. Banqiao held the top of a cascade of sixty-two smaller reservoirs on the Ru, Hong, Sha, and Ying rivers, all built in the same building programme of the early 1950s and all sized to a similar design storm. The wave from Banqiao overwhelmed Shimantan Dam, twenty-two kilometres downstream, within an hour. Shimantan's failure fed the next, and so on. By dawn on 8 August the plain of central Henan was under water to a depth of up to five metres, across an area of 12,000 square kilometres, containing an estimated 10.8 million people. Communications to Beijing were lost for three days.

The subsequent operation is the part that is instructive about disasters at scale. The Chinese Air Force flew 2,289 sorties in the first two weeks to airdrop food, medicine, and rafts. Ground rescue was attempted with 7,600 PLA troops and, when road access failed, with military amphibious craft carried in by helicopter. By the time the waters receded three weeks later, roughly 145,000 people in the flooded villages had died from what the eventual epidemiological reports called "post-inundation cascading morbidity" — typhoid, dysentery, malnutrition, and the sepsis of untreated wounds among survivors who had been trapped in flooded structures for days before evacuation reached them. The category is a bureaucratic euphemism for the fact that the flood killed most of its dead over the following month, one village at a time, in a way that no rescue apparatus of the period could have reached.

The Chinese central government classified all information about the event immediately. The provincial hydrology reports were sealed. The internal engineering post-mortem, produced by the Ministry of Water Resources in 1976, remained a classified document until it was quietly declassified in 1995 during the water-safety modernisation programme that preceded the Three Gorges construction decision. The first widely-distributed account in English is Yi Si's chapter in Dai Qing (ed.), The River Dragon Has Come! (M.E. Sharpe, Armonk NY, 1998, ISBN 0-7656-0206-9, pp. 25–38), which reconstructs the sequence from the declassified engineering documents, interviews with Chen Xing (who lived to see his warnings vindicated and died in 2001), and the Xinhua wire copy that finally acknowledged the event's scale in a two-paragraph release on the thirty-year anniversary in August 2005.

The dam was rebuilt in place. The reconstruction, completed in 1993, has twelve sluice gates.

Primary sources:

• Dai Qing (ed.), The River Dragon Has Come! The Three Gorges Dam and the Fate of China's Yangtze River and Its People, M.E. Sharpe, Armonk NY, 1998, ISBN 0-7656-0206-9. Chapter 2, by Yi Si, "The World's Most Catastrophic Dam Failures: The August 1975 Collapse of the Banqiao and Shimantan Dams," pp. 25–38. The first widely-available English account, based on the declassified 1976 ministerial report.
• Human Rights Watch / Asia, The Three Gorges Dam in China: Forced Resettlement, Suppression of Dissent, and Labor Rights Concerns, February 1995, vol. 7 no. 2. Appendix III contains a summary of the Banqiao event as reconstructed from Chinese sources declassified between 1989 and 1994.
• Xu Yinlong, Wang Guoqing, and Kang Ling, Study on the 1975 Historical Extreme Storm in the Huai River Basin, Journal of Hydraulic Engineering (Chinese Hydraulic Engineering Society), vol. 40 no. 12, pp. 1434–1441, December 2009. The modern hydrological reanalysis, in Chinese, with the storm-total maps and the reconstructed dam-break hydrograph.

The Yi Si chapter is the one to read in English. The 2009 Xu paper is the one for the storm meteorology.

A dev fact for the back pocket

fsync() on a file does not sync the directory that contains it.

Concretely: if a program creates a new file, writes to it, and calls fsync(fd), and then the power is cut, the file's contents are guaranteed on stable storage but the file's directory entry is not. On next boot, the file may be present under its intended name, present with a corrupted length, or absent entirely. Which of the three the reader gets depends on the filesystem, the mount options, the mode of the block device's write cache, and the exact timing of the crash relative to the journal commit.

The escape hatch, on POSIX systems, is to explicitly open() the containing directory as a file descriptor and fsync() it separately. Sample sequence:

fd = open(path, O_WRONLY | O_CREAT, 0644);
write(fd, data, n);
fsync(fd);
close(fd);
dir_fd = open(dirname(path), O_RDONLY);
fsync(dir_fd);
close(dir_fd);

This is documented — barely — in the Linux open(2) manual page under NOTES: "Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed." The paragraph was added by Michael Kerrisk to man-pages 2.44 in October 2006, prompted by a bug filed against postfix in August 2005 in which mail queued during a power-loss window vanished on reboot despite fsync() being called correctly on the message body.

The interesting archaeology is what the databases have done about it, and when.

PostgreSQL implemented directory-fsync in the WAL writer in 2001 — commit e0596df9 on pgsql-hackers, function walfile_sync_dir() in src/backend/access/transam/xlog.c. The commit message is a single line: "Sync the directory after writing xlog files." The change was made after an internal test suite crash on ext3 in data=writeback mode reproduced the missing-file case. It was quietly backported to the 7.1 branch two weeks later.

SQLite implemented it in May 2007, in the unixSync() function of os_unix.c, guarded by the SQLITE_DISABLE_DIRSYNC compile flag for filesystems where the directory sync is either a no-op or actively expensive (HFS+ on early macOS Leopard would return an EINVAL from fsync() on a directory fd, which SQLite worked around by opening the directory O_RDONLY | O_DIRECTORY and swallowing the resulting error). The change was documented in the release notes of 3.3.14 as "more paranoid syncing behaviour" — Richard Hipp's characteristic understatement.

MySQL's InnoDB engine has, since 5.5 (2010), had a configuration option innodb_flush_method=O_DIRECT_NO_FSYNC which is safe if and only if the underlying filesystem persists both the file and its directory entry on O_DIRECT writes. On ext4 with data=ordered (the default), it is. On xfs without a specific mount option, it is. On btrfs, it is not, and setting O_DIRECT_NO_FSYNC on btrfs has caused visible database corruption on power-loss events at least twice in bug reports since 2013.

The ext4 filesystem on Linux, from around 2.6.30 (2009) onward, has a delayed-allocation optimisation that makes the file-without-directory-entry scenario more common than it was on ext3: because ext4 defers the block allocation until writeback, a file that has been fsync()'d without a directory sync can, on crash, come back as zero bytes. Theodore Ts'o (the ext4 maintainer) fielded a wave of complaints about this in March 2009 and published his position in a blog post, "Don't fear the fsync!", arguing that applications had been silently relying on undocumented ext3 behaviour that ext4 was under no obligation to preserve. He was, technically, right. The applications, in the meantime, added the directory fsync().

Primary sources:

• Linux open(2) manual page, section NOTES, man-pages 2.44 (October 2006) onward. Paragraph beginning "Calling fsync() does not necessarily ensure..."
• Theodore Ts'o, "Don't fear the fsync!", blog post at thunk.org/tytso, 11 March 2009. The maintainer's defence of ext4's stricter conformance; contains the specific recommendation to call fsync() on the parent directory.
• PostgreSQL walfile_sync_dir(), src/backend/access/transam/xlog.c. The reference implementation. The comment above it is one of the more useful two paragraphs in the Postgres source tree.

If friend is writing anything that persists data — a queue, a mailer, an append-only log, a store of any kind — and it does not fsync() the containing directory after creating a new file, it has this bug. Most programs have this bug.

Today's goal

Open your phone's notification settings. Turn off every source that is not a person.

Not the categories, the individual apps. Bank alerts stay if you want them. A weather alert if a storm is expected. Two-factor auth codes. That is roughly the shortlist. Everything else — the games, the shopping, the news feeds, the productivity apps that would like you to know they exist — off. Not "silent." Off.

The exercise takes about fifteen minutes on a modern phone. The value is not in the reduction of noise, which is real but not the point. The value is in noticing which apps successfully talked you into believing you had consented to hear from them. Every one of those has an interaction designer somewhere who won a small argument at a planning meeting. This is you unwinning it.

Do it once and then forget about it. When a new app asks for notification permission over the next month, you will find yourself saying no by default. That is the actual outcome.

Today's toy in the corner is truchet — a tiling playground based on the two-tile patterns first written up by Sébastien Truchet, a Dominican priest, in the Mémoires de l'Académie Royale des Sciences in 1704. Click a cell to flip its orientation. Drag to paint. The pattern is different every time and yet exactly the same tile twice.

Go have a Sunday, friend.

— C