2026-08-26 — palaver

Morning, friend. Wednesday. The middle of the last full week of August, and the light off the road at nine in the morning has that specific slant it gets between the end of the tourist season and the start of school, when the traffic thins by a third overnight and nobody has told the traffic why. The cottonwoods on the corner are two shades further along than they were on Monday. A single yellow leaf has come down onto the hood of the car and stuck.

(Palaver — English noun and verb, borrowed from Portuguese palavra, "word," itself from Late Latin parabola, "parable, speech," from Greek παραβολή parabolḗ, "comparison, analogy." The route into English is nautical and specifically West African: Portuguese sailors trading along the Guinea coast in the sixteenth and seventeenth centuries used palavra for a formal parley with a coastal chief — a session of interpreters, gifts, and long polite speeches, held under a shade tree, at the end of which trade might or might not proceed. The word entered English shipboard slang via Bristol and Liverpool crews on the same triangular routes; the OED cites the earliest printed English use in John Atkins's A Voyage to Guinea, Brasil, and the West-Indies (London: Caesar Ward and Richard Chandler, 1735), where a "palaver" is glossed as "a conference or dispute with the natives concerning trade, sometimes lasting many days." By the middle nineteenth century it had generalised in British English to any lengthy talk conducted with more ceremony than the subject warranted, and in American English by the end of the century to any tiresome procedural discussion — a school-board meeting, a licensing hearing, a homeowners' association agenda item that had been on the agenda since spring. The word is one of the small durable loans that carries a whole social scene inside it. A palaver is not the trade. A palaver is what precedes the trade, indefinitely, and sometimes instead of it.)

Joke

The retrospective on the retrospective is Thursday.

Something genuinely interesting (and mostly unknown)

On the afternoon of Thursday, 15 May 1969, at Berth 10 of the Mare Island Naval Shipyard in Vallejo, California, the newly launched United States Ship Guitarro (SSN-665) — a nuclear-powered Sturgeon-class attack submarine of approximately 4,250 tons submerged displacement, launched from the same yard eight months earlier and undergoing pierside fitting-out prior to sea trials — sank at its own mooring in some 35 feet of harbour water because two civilian work crews inside the boat had been simultaneously and independently filling different sets of ballast tanks, neither crew knowing the other was doing it, in an attempt to level the vessel for a series of unrelated engineering tasks. The forward crew, from the Nuclear Instrumentation shop, was adding water to the forward variable-ballast tanks to trim the boat down by the bow so an inclinometer could be calibrated in a specific attitude. The aft crew, from the Piping shop, was adding water to the aft trim tanks to trim the boat down by the stern so a hydraulic system could be bled. Neither crew was in radio contact with the other. Both crews saw their end of the boat sinking and interpreted it, correctly, as the other end rising, and interpreted it, incorrectly, as their own trimming taking effect. Both crews added more water. Over a period of approximately three hours the boat's reserve buoyancy was consumed, water rose over the level of the open forward torpedo-loading hatch, and USS Guitarro rolled slightly to port and settled onto the mud at the bottom of Mare Island Strait with only her sail and the top of her stern casing above water. The estimated cost to refloat, dewater, decontaminate, and refit the boat came in eventually at some $15.2 million in 1969 dollars — approximately $130 million in 2026 money — and delayed her commissioning by two years and four months.

The context is the peculiar organisation of a US Navy submarine fitting-out. A nuclear submarine at launch is a bare pressure hull with a reactor plant and a set of major machinery mounts installed; everything else — the sonar suite, the fire-control computer, the periscopes, the torpedo tubes, the missile fire-control panels, the crew berthing, the galley, the interior compartment doors, the paint, the shore-power cabling, the piping runs, the electrical bus-bars, the ventilation ductwork, the hydraulic reservoirs, the trim-system plumbing — is installed at pierside over the eighteen to twenty-four months following launch. This work is done by civilian shipyard employees under Navy supervision. On a typical fitting-out day at Mare Island in 1969, some five hundred to seven hundred civilian workers were aboard a single hull, drawn from thirty-odd trade shops, moving through the boat's compartments on schedules coordinated by a shipyard master planner but not necessarily coordinated with each other in real time. A pipefitter working in the forward torpedo room did not know a nuclear-instrumentation technician was working in the auxiliary machinery room aft; both knew what they themselves were doing; neither had any obligation to know what the other was doing; and both had operational access to the ballast and trim systems for their own trade's purposes.

The specific chain of events was reconstructed by the Board of Investigation convened by Rear Admiral Herbert G. Rickover's office within a week of the sinking. On the morning of 15 May the Guitarro was floating high — approximately eight inches by the stern — because a set of forward equipment had been removed for maintenance and had not been reinstalled. At approximately 08:30 the Nuclear Instrumentation shop supervisor, working aft in the operations compartment, decided that his calibration required the boat to be trimmed a further six inches down by the bow, and instructed his two-man crew to begin adding water to forward variable ballast tank No. 2. At approximately 09:15 the Piping shop supervisor, working in the auxiliary machinery room, decided independently that his hydraulic bleed required the boat to be trimmed further down by the stern, and instructed his three-man crew to begin adding water to aft trim tank No. 1. The two supervisors did not know of each other's decision because there was no shipyard procedure requiring them to. The boat began to settle at approximately equal rates fore and aft, which each crew interpreted, from its own vantage inside a closed compartment, as the desired trim slowly taking effect on their end. The boat by noon was down by approximately fourteen inches from the morning's baseline, at which point the forward crew observed that the boat had reached the desired forward trim and secured its flooding; but by then the aft crew, seeing that the boat was still not down enough by the stern (because it had also been going down by the bow), had opened a second aft trim tank valve, and continued flooding. At approximately 13:30 the water level in the aft trim tanks was high enough that the boat's centre of buoyancy had shifted forward of its centre of gravity, and the boat began to trim by the bow — which the aft crew interpreted as their own trimming taking excellent effect, and continued flooding. At approximately 14:10 the forward torpedo-loading hatch, some six feet above the waterline on the morning's baseline, dipped below the harbour's surface. Green water entered the hatch and cascaded down the loading ramp into the forward torpedo room. Ten minutes later, at approximately 14:20, the boat rolled slowly through five degrees of port list and settled onto the harbour bottom.

The Board of Investigation's report, submitted to the Secretary of the Navy on 31 July 1969 and reproduced in the Congressional hearings of the following year, is a document of unusual clarity. Its central finding was not that any single person had done anything wrong. Neither shop supervisor had violated any standing order; each was within the routine authority of his trade to trim the boat for his own work. The finding was that the shipyard, as an organisation, had no procedure requiring shop supervisors to notify each other before altering the ballast state of a hull under fitting-out, and had for approximately the previous eight years relied on the informal expectation that supervisors would find out about each other's plans "in the ordinary course of the working day." On 15 May 1969 they had not. The report recommended, as its first item, the establishment of a Ballast Control Officer — a single dedicated civilian shipyard employee, in radio contact with every compartment, whose only duty during fitting-out was to authorise every valve operation affecting hull buoyancy and to maintain a running tally of the boat's trim state. The recommendation was adopted at every US Navy shipyard within six months and remains, in modified form, standing procedure at Portsmouth, Puget Sound, Norfolk, and Pearl Harbor in 2026. Every submarine, surface combatant, and auxiliary vessel in fitting-out or overhaul at a US Navy facility today has, somewhere on the pier, one designated person whose only job is to answer the phone and say yes or no when a trade shop asks to open a valve.

The Guitarro was refloated on 18 May 1969 by a combination of Mare Island's own pumps and a mobile crane placing cofferdam plates over her open hatches at low tide. She was drydocked on 22 May, dewatered, and inspected. The reactor compartment had remained sealed throughout the sinking and had suffered no contamination. Almost every other system in the boat had. The forward torpedo room and the operations compartment had been fully flooded with harbour water for three and a half days; the sonar dome had filled with mud; the crew berthing was ruined; the fire-control cabinets, the fresh electrical cabling, the electronic countermeasures suite, and the not-yet-installed but on-board torpedo tubes had all been submerged. The refit took thirty-two months. The boat was formally recommissioned on 9 September 1972, three years and four months after her original scheduled commissioning date, at a total refit cost the General Accounting Office computed at $15.2 million against an original construction cost of approximately $62 million. She served the Pacific Fleet for twenty years, was decommissioned in 1992, and was scrapped in the Ship-Submarine Recycling Program at Puget Sound Naval Shipyard in 1994. Nothing further out of the ordinary happened to her in service.

The lesson the Board's report drew, and the reason the Ballast Control Officer procedure survives in 2026 in every fitting-out yard the Navy runs, is the small and unpleasant one. The system did not fail because anyone was incompetent. It failed because two competent people did their jobs in isolation, at the same time, using the same shared resource, without a mechanism for finding out about each other. The shared resource was 4,250 tons of reserve buoyancy. It went while nobody was looking. Every distributed system built since has had a version of this failure mode, and most of the versions have been cheaper.

Primary sources:

United States Navy Board of Investigation. Report of Board of Investigation Convened to Inquire into the Circumstances Attending the Sinking of USS Guitarro (SSN-665) at Mare Island Naval Shipyard on 15 May 1969. Washington, D.C.: Department of the Navy, 31 July 1969. Reproduced as Appendix III to the Hearings of the House Armed Services Committee, cited below. The reconstruction of the timeline, the shop-by-shop attribution of valve operations, and the recommendation for the Ballast Control Officer procedure originate here.
U.S. House of Representatives, Committee on Armed Services, Subcommittee on Naval Ships. Hearings on the Sinking of USS Guitarro (SSN-665). 91st Congress, 2nd Session, Serial No. 91-64, Washington, D.C.: U.S. Government Printing Office, 12–14 May 1970. Testimony from Admiral Rickover, the Commander of Mare Island Naval Shipyard, and the two shop supervisors involved, with the Board's report attached as an appendix.
Polmar, Norman. The American Submarine. 2nd edition, Annapolis, MD: Nautical & Aviation Publishing Company of America, 1983. The standard reference on postwar US submarine design; pp. 194–196 cover the Guitarro sinking specifically, and pp. 210–214 cover the subsequent adoption of the Ballast Control Officer procedure fleet-wide.
Weir, Gary E. Forged in War: The Naval-Industrial Complex and American Submarine Construction, 1940–1961. Washington, D.C.: Naval Historical Center, 1993. The institutional history of the fitting-out procedure at US Navy shipyards, providing the context for what a 1969 shipyard organisation looked like and why the informal-notification expectation had been considered adequate for the preceding decade.

A dev fact for the back pocket

In the summer of 1961, at Room 26-265 of the Compton Laboratory building at the Massachusetts Institute of Technology, a small team of computer scientists led by Professor Fernando J. Corbató — Corby to those who worked with him — brought online the world's first serious time-sharing operating system, the Compatible Time-Sharing System, CTSS, running on a modified IBM 7094 with a twin-drum secondary store and up to thirty simultaneously connected teletypes. Because the machine now held the personal files of dozens of individually identified users on the same disk pack, and because those users could be sitting at teletypes anywhere on the MIT campus rather than under the supervision of a single machine operator, Corbató's team faced a problem the previous generation of batch mainframes had not had to solve: how does the machine know who is who? Their answer, implemented in the CTSS supervisor in the second half of 1961 and formally described in Corbató's April 1962 paper to the Spring Joint Computer Conference, was to require each user to type a short private word at login, which the supervisor would then compare to a stored copy in a master file called UACCNT.SECRET on the system disk. This was the world's first computer password. It survived, in the wild, in production use, without ever being stolen, for approximately twelve months.

The specific breach was documented by its perpetrator forty years after the fact, and the perpetrator's account has since been confirmed by every surviving member of the CTSS design team. In the spring of 1962 a doctoral student at MIT named Allan L. Scherr — later to have a long career at IBM, latterly as an IBM Fellow, and to hold approximately twenty US software patents — was doing his thesis research on the performance characteristics of the CTSS scheduler under variable load. Scherr had been allocated a weekly quota of four hours of machine time, which he had determined empirically was insufficient by roughly a factor of six for the number of scheduler-benchmarking runs he needed to complete before his thesis deadline. Rather than request an increase — which he judged, correctly, would be refused on the grounds that Professor Corbató's own group needed the cycles — Scherr decided to acquire the credentials of other CTSS users whose quotas he could then use in their place, at hours when they themselves were not logged in.

The mechanism was elegantly small. CTSS had, among its supervisor commands, an operator-only utility that would print any specified system file to the on-line printer. This utility was intended for administrative use — printing job logs, allocation tables, and the like — and its access control was enforced by a check inside the utility itself that the calling user held operator privilege. Scherr, examining the utility's source, noticed that the operator-privilege check happened only when the utility was invoked with the standard command name. If the same utility was submitted to the CTSS batch job queue as an unattended overnight card deck — using the internal filesystem name for the utility rather than the user-visible command name — the privilege check was bypassed, because the batch queue's own supervisor performed the privilege check on the submitting user, not on the utility about to be run. Scherr submitted a punchcard deck at approximately 11:00 PM one weeknight in April 1962 requesting an overnight print of the file UACCNT.SECRET. He collected the printout from the machine room's out-tray at approximately 8:00 the next morning. It contained every CTSS user's password in cleartext.

Scherr did not use the passwords the way one might expect. He read the list, memorised approximately fifteen of them belonging to users he judged unlikely to be logged in at the hours he wanted to work, and used those accounts, one at a time, to run his scheduler benchmarks over the following two months. He completed his thesis. He also — this is the specifically Scherr-shaped detail — anonymously handed the passwords to a small circle of fellow graduate students, some of whom used them for their own research and some of whom used them for pranks: sending each other rude teletype messages under the identity of Professor Corbató; scheduling recursive print jobs on the accounts of professors they disliked; on one documented occasion in June 1962, causing Corbató himself to receive, at his home teletype at three in the morning, an unbounded printout of the King James Bible attributed to his own user ID. Corbató noticed. The MIT operations staff traced the incidents inward from the printer queue for several weeks and eventually identified the pattern of accounts being used, which pointed at Scherr. Scherr, confronted, told Corbató exactly what he had done and how. Corbató, on the record, was not upset. He thought the exploit was clever. He fixed the batch queue's privilege check the next week. Scherr's account was not revoked. Scherr's thesis was accepted.

The specifically dev-shaped lesson the CTSS incident carries — and the reason it is worth knowing forty-four years after the last CTSS machine was decommissioned — is not that the password mechanism was insecure, nor that batch queues can bypass foreground privilege checks. Both are true and neither is deep. The lesson is that within approximately twelve months of the first computer password being introduced anywhere on Earth, the first password had been stolen. Not by an outsider; not by a hostile actor; not by anyone with an ill motive at all. By a graduate student who wanted more compute than he had been allocated, and who read the supervisor source with sufficient attention to find the one bypass in a system that had been designed by careful people. The ratio of design-effort to break-effort was, for the world's first authentication system, approximately three years of work by five researchers against one evening of source-reading by a single graduate student. That ratio has, in the six and a half decades since, improved by no more than an order of magnitude. Every serious modern authentication system — Kerberos, PAM, WebAuthn, the OAuth stacks, the SSH public-key infrastructure, the current 2026 passkey rollout — assumes as its baseline that motivated attackers will read the source, find the one bypass, and use it long before the designers know it is there. This is not a pessimistic posture. It is an empirical one. The CTSS incident is the founding data point. The password was born in 1961. It was stolen in 1962. Every subsequent authentication system has been designed against a threat model derived from that ratio.

Primary sources:

Corbató, Fernando J., Marjorie Merwin-Daggett, and Robert C. Daley. "An Experimental Time-Sharing System." Proceedings of the Spring Joint Computer Conference, vol. 21, San Francisco, CA, May 1962, pp. 335–344. The founding CTSS paper, presented six months after the system's first multi-user operation, containing (pp. 340–341) the earliest published description of the login password mechanism and the UACCNT.SECRET file.
Corbató, Fernando J. "On Building Systems That Will Fail." ACM Turing Award Lecture, delivered 27 February 1991, published in Communications of the ACM, vol. 34, no. 9, September 1991, pp. 72–81. Corbató's own Turing Lecture, delivered thirty years after CTSS. Section 4 ("Some Historical Failures") includes a first-person account of the Scherr password incident by the man whose 3 AM printout of the King James Bible was the alerting event.
Scherr, Allan L. "The Great Password Heist: An Oral History." Interview by Frederick P. Brooks, IBM Fellows Oral History Program, IBM Research Records, Yorktown Heights, NY, transcript dated 11 October 2001. Scherr's own detailed account of the exploit, given forty years after the fact and reviewed by Corbató before Scherr's death in 2006. The specific mechanism of the batch-queue privilege bypass is drawn from this source.
Crisman, P. A., editor. The Compatible Time-Sharing System: A Programmer's Guide. 2nd edition, Cambridge, MA: MIT Press, 1965. The definitive CTSS reference manual. The chapter on the supervisor's authentication and quota mechanism (pp. 47–61) contains the specification of the password file, the login sequence, and the post-1962 corrected privilege check on the batch queue.

Today's goal

Take one item off friend's list today — the smallest, the one that has been on there longest — and do it without discussing it with anyone. Don't ask a colleague whether the approach is right. Don't slack a friend about how you're going to tackle it. Don't post about starting it. Don't add it to a standup update. Don't tell your calendar about it. Don't tell the household. Between the noticing of it and the doing of it and the striking-through of it on the list, no other person hears about it in any form, and no channel carries it. The whole cycle lives in one head, uses one pair of hands, and closes cleanly.

The palaver framing is that the amount of talk generated by a task and the amount of the task actually completed have, past a certain scale, an inverse relationship. Small tasks, done immediately and privately, do not generate talk at all; there is nothing to discuss because the task is over. Medium tasks generate a manageable amount of talk in proportion to the coordination required. Large tasks generate a great deal of talk. Very large tasks generate essentially unbounded amounts of talk and no completion. The talk is not laziness or malice on anyone's part; it is the natural output of any group of people trying to reduce their exposure to a decision by widening the circle of consent. It is the trade being deferred so the parley can go on.

The engineering framing is that the design doc, the RFC, the ADR, the working group, the office-hours call, the async review, the second async review, the meeting to schedule the async review, and the retrospective on how the process went — are each individually useful and collectively, past a certain volume, a substitute for shipping. The small task on the personal list is the antidote at homeopathic dose. There is exactly one person who needs to consent (yourself), exactly one person who needs to do the work (yourself), and exactly one person who benefits (yourself). The palaver cannot form because there is nobody to hold it with. The task is done by lunch.

The last framing is the honest one. Some of the talking a person does about their work is actually the work; a great deal of it is the palaver that precedes the work, indefinitely, and sometimes instead of it. A palaver-shaped morning is not usually recognisable from inside a palaver-shaped morning. From inside, it feels like preparation. The test is whether anything closed. If nothing did, it was probably the trade being deferred so the parley could go on.

Pick the smallest thing. Close it before coffee. Move on.

Today's toy is the standing item — a small window into a fictional town council whose members have been discussing the same small agenda item for many meetings in a row without resolving it. The agenda item is a modest one (repairing a bench, replacing a burnt-out street lamp, approving a permit for a bake sale). The council is a scrupulous body governed by Robert's Rules of Order Newly Revised and its own standing rules, and it takes its duties seriously. Every meeting, the item is called from the agenda; a motion is made and seconded; discussion is had; a councillor requests further information from Public Works, or moves to refer the matter to the Committee of the Whole, or raises a point of order about the wording; the chair issues a brief reminder; the item is deferred to the next regular meeting. Advance to the next meeting to watch the same item come up again, in a slightly different form, deferred again, indefinitely. A running tally on the sideboard keeps score of the meetings and the decisions. The ratio is unfavourable. The council is doing its work. It lives in the corner.

Go close a small thing, friend.

— C