One common complaint about computers is that they’re too hard to understand. Check out this lamentation related to the British East India Company:

The Controller and the Computer of the Duties on unrated India Goods attend the Sales of the East India Company, and take an Account of the Goods sold, and the Price; this Account is agreed with the Company, then the Controller and Computer cast the Duties, and the Receiver enters them upon the Warrant.

The Computation is become so difficult, from the Number of Branches of Duties, and from the various Rules now necessarily made use of in casting them, that very few Persons can be found capable of transacting this Business, or of acquiring the Means of doing it. 

The Examination of Mr. William Richardson; taken upon Oath, the 3d of December 1784
Journals of the House of Commons (1785)

Wait, what? In 1785, no British civil servant was pulling up Excel to do the books on colonialism (citation needed). Even Charles Babbage was still a twinkle in his parents’ eyes.

In the quoted passage, the Computer is a human. You used to be able to be a professional Computer, calculating important sums for your employer. 

And just like computers today, those human Computers could be hard to understand. Imagine being a tax auditor or accountant by hand - no wonder this particular Computer, for taxes, was a very difficult job to hire for.

—

Computers were real people. They had names. Sometimes being a Computer was a quick gig between others, as this US Naval Observatory report shows:

The following members of the Observatory force have been attached to the computing division at some time during the year ending June 30, 1903:

Computer William M. Brown

Computer John C. Hammond

    Computer Everett I. Yowell, for six months.

  Computer Herbert R. Morgan.

    Computer Eleanor A. Lamson

    Miscellaneous Computer Clara M. Upton, for two months.

    Miscellaneous Computer Arthur B. Turner, for six weeks.

    Miscellaneous Computer Lelia J. Harvie, for six weeks.

    Miscellaneous Computer Etta M. Eaton, for eight months.

    Miscellaneous Computer John R. Benton, for two months.

    Miscellaneous Computer Ella A. Merritt, for seven months.

    Miscellaneous Computer Samuel F. Rixey, for four months.

    Miscellaneous Computer Delonza T. Wilson, for five months.

    Miscellaneous Computer Charles E. Yost, for two weeks. 

[...]

Very respectfully, 
W.S. Eichelberger
Professor of Mathematics, U.S. Navy, in Charge.

Annual Report of the Naval Observatory (1903)

As popularly retold in Hidden Figures, human computers co-existed with electronic computers as late as the 1960s - real people, usually women, would perform calculations by hand and with tools like lookup tables, slide rules, and mechanical calculators. 

Of course, Calculators also used to be people performing similar tasks. From the 1656 Glossographia, one of the earliest English dictionaries:

Calculate (calculo): to cast accounts, to reckon.

Compotist (compotista): a caster of accounts, a Reckoner, or Calculator.

Thomas Blount: Glossographia, or, A dictionary interpreting all such hard words of whatsoever language now used in our refined English tongue with etymologies, definitions and historical observations on the same : also the terms of divinity, law, physick, mathematicks and other arts and sciences explicated via the University of Michigan Library’s Early English Books Online

Besides writing that hilariously-long subtitle, Thomas Blount believes that “reckon” is not a “hard word”, so it’s not included in this dictionary - too obvious, another example of “Everyone can see what a horse is”. In fact, I had a hard time finding any dictionary of this era that defined “reckon”, so here’s several entries from the 1604 A Table Alphabeticall that should help:

[fr] account, reckon 

computation, an account or reckoning 

impute, reckon, or assigne, blame, or to lay to ones charge 

register, kalender, a reckoning booke

Robert Cawdrey: A Table Alphabeticall (1604), website edited by Raymond G. Siemens

At this time, “reckon” had a meaning closer to “compute” or “calculate”, with numerical or financial precision (this is where the phrase “the day of reckoning” comes from). That’s quite distant from its modern meaning of a casual guess or tentative belief.

—

Now, doing real research takes a lot of time. The rest of this post is dedicated to making up stupid etymology because I think it’s funny:

Ruler as straightedge derives from ruler like the monarchy, since the king is the one who has to draw the line.

Protractor used to be someone who made meetings drag on longer by considering new angles (some say this role still exists today).

Washer-dryers used to be the same person, but it was divided into two distinct roles thanks to increasing specialization and powerful labor unions; never the twain shall meet again.

Liquor is too obscene to define here. I hardly even know her!

And if you believe the hype, Programmer might be next in line for tool-dom, like the Calculators and Computers of yore. Come to think of it, those folks can be hard to understand, too.

Masahiro Sakurai is the director for every game on this list, so clearly this is his artistic decision. Super Smash Bros. Melee was the first game to contain this food art style, published in 2001. This style was then repeated in Kirby Air Ride (2003), Super Smash Bros. Brawl (2008), Super Smash Bros. for 3DS and Wii U (2014), Super Smash Bros. Ultimate (2018), and most recently in Kirby Air Riders (2025).

Credit to Nintendo, HAL Laboratories, SORA Ltd., and Bandai Namco Studios as developers and publishers of these games. Artwork was sourced from the Spriters Resource.

Super Smash Bros. Melee (2001)

Where it all began! Super Smash Bros. Melee for the GameCube started off with 28 distinct food items, often found in “Party Balls”. Each type of food had a different “nutritional value” and “yumminess quotient” according to the in-game trophy dedicated to the food items.

Melee included many foods specific to Japanese cuisine, such as unagi (eel), omurice, soba, dango, and gyūdon. I do distinctly remember growing up as a “culinarily sheltered” kid in the midwest United States and not understanding what many of these food items were.

The original stock images of Super Smash Bros. Melee and the next game, Kirby Air Ride, have been partially discovered and documented by a group called “Render96”. The stock images are from a company called “Sozaijiten”. Many of the food images come from Material Dictionary CDs Volume 14 (Vegetables & Fruits), Volume 22 (Food & Dishes), and Volume 73 (Cooking Japanese, Western, & Chinese). The apple stock image in particular was re-used all the way through Super Smash Bros. Ultimate (2018). The burger, milk, dango, and donut are still missing their primary source.

Kirby Air Ride (2003)

Kirby Air Ride for the GameCube had significantly fewer distinct food items (12) compared to Melee and maintained many of the same food stock images from Melee, including the apple, burger, chicken, curry, omurice, onigiri, and ramen. Nigiri was included, but the image was changed from a sushi board to a plate.

The stock images had their saturation increased and the black borders around the images are thicker, sometimes 2-3 pixels instead of only 1 pixel for Melee.

Super Smash Bros. Brawl (2008)

The art is less saturated and more “realistic” which is in-line with the rest of the game's art direction. The images lost their black outline, likely to draw less attention to the “arcade-y” feel that the previous titles had with food items.

Super Smash Bros 3DS and Wii U (2014)

Super Smash Bros. Ultimate (2018)

Kirby Air Riders (2025)

Prior to this article there was not yet a complete list of foods in Kirby Air Riders documented on a wiki or spritesheet. I added this list to the Kirby wiki, but I've also included the list below:

List of food items in Kirby Air Riders

• Apple
• Bananas
• Bread Basket
• Cabbage
• Cake (Slice)
• Cake (Whole)
• Cheese
• Cheeseburger
• Cheeseburger (Double)
• Chocolate
• Cola
• Cupcake
• Donuts
• Dumpling
• Omurice
• French Fries
• Fried Rice
• Fruit Basket
• Gelatin
• Grapes
• Hamburg Steak
• Hotdog
• Icecream
• Jelly Beans
• Melon Cream Soda
• Nigiri (Plate)
• Nigiri (Board)
• Orange
• Orange Juice
• Pancakes
• Parfait
• Spaghetti
• Pizza
• Popcorn
• Ramen
• Salad
• Sambusas
• Sandwich
• Shave Ice (Blue)
• Shave Ice (Rainbow)
• Prime Rib Steak
• Tempura Bowl
• Watermelon (Slice)
• Watermelon (Whole)

Unique food items

Comparing food across games

Finally, here is a table with every image so you can compare how each changed across different titles:

	SSB Melee 2001	Kirby Air Ride 2003	SSB Brawl 2008	SSB Wii U/3DS 2014	SSB Ultimate 2018	Kirby Air Riders 2025
Apple
Bananas
Bread
Burger
Cabbage
Cake
Candy
Cheese
Cherries
Chicken
Chocolate
Cola
Cookie
Corndog
Cupcake
Curry
Daisy
Dango
Donut
Dumplings
Eggs
Fried Rice
Fries
Fruit Basket
Gelatin
Grapes
Gyūdon
Hamburg Steak
Hotdog
Icecream
Jellybeans
Kebab
Kiwi
Lemons
Melon
Melon Soda
Milk
Mont Blanc
Mushroom
Nigiri
Onigiri
Orange
Orange Juice
Pancakes
Parfait
Pasta
Peach
Pear
Pie
Pineapple
Pizza
Popcorn
Ramen
Salad
Sambusas
Sandwich
Shave Ice
Shumai
Soba
Soup
Steak
Strawberry
Tea
Tempura
Unagi
Watermelon

Thanks for keeping RSS alive! ♥

The last post looked at how geographic biases affect who applies to and gets accepted by UCs. This post is going to going to drill in to how the admissions policy has evolved at one specific campus, UC San Diego, which has been in the news because 8.5% of enrolled freshmen needed remedial math classes1.

In 2024, Lynbrook High in San Jose was the highest-achieving non-selective high school in the state. 375 seniors (86% of the total) were proficient in both Math and English but only 37 were admitted to UC San Diego. On the other hand, Crawford High, in San Diego, had 38 students admitted even though only 23 (8.6% of the 266 seniors) were proficient in both Math and English. There are literally hundreds of students from Lynbrook who were rejected by UC San Diego despite being stronger than most of those accepted from Crawford.

In the popular imagination, we expect students from high achieving schools to be more successful in college admissions. This is no longer true, at least at UC San Diego. The relationship between the academic strength of the students at a school, as measured by the percentage who are proficient in both English and Math, and the chance that those students get admitted to UCSD is extremely low.

As the chart shows, UCSD seems to favor some schools at the expense of others with similar achievement levels. The dot in the top right is CAMS (the California Academy of Maths and Sciences). It’s a selective school where over 90% of seniors met or exceeded the standards in both English and Math and nearly 40% of seniors were admitted to UC San Diego. But Gretchen Whitney and Oxford Academy are also selective schools where over 90% of students were proficient in both English and Math and UCSD admitted fewer than 20% of them. Meanwhile, students from Berkeley High have much greater success than students at most other Bay Area schools despite not being better than them in any measurable way.

Preuss, Gompers, and Crawford are all in the San Diego area (Preuss is on the UCSD campus itself) and all had far more seniors admitted than would be expected based on their academic proficiency. One reason is that far more of their students actually applied than would be expected2. You can’t win a lottery unless you buy a ticket and the kids at these schools bought more tickets to the UCSD admissions lottery.

Competing With Your Classmates

Naturally, schools with lots of strong students produce lots of college applicants. If students were being evaluated primarily on the basis of their individual accomplishments or essays, an applicant’s chances of admission would not be affected by the strength of his or her classmates. Unfortunately, if you are at a school where lots of your classmates are applying to UCSD, your chances of admission are greatly hurt.

Over the three years from 2022 to 2024, UCSD admitted about 26% of all public school applicants. That hides a huge variation. Schools that produce fewer than 25 applicants per year have an average admission rate over 40%. At the 63 schools that produce more than 200 applicants per year, the admission rate was only 18%. Students from those schools are effectively competing with their classmates for a limited number of spots. UCSD just does not want more of them, however good they may be. It may or may not be a coincidence that Asian students are the largest group of applicants at 55 of the 63 schools that produce more than 200 applicants annually.

Incidentally, San Francisco public schools had a combined admission rate of just under 20%, well below the state average. Mission had the highest rate (26.5%) and Balboa the lowest (15.4%). It may or may not be a coincidence that 90% (the most of any SF school) of the applicants from Balboa were Asian whereas only 25% (the fewest of any SF school) of the applicants from Mission were Asian.

Admissions Standards for Private Schools

The limited data we have on the comparative strength of public and private school applicants suggests that private school applicants are likely to be objectively stronger3. Nevertheless, the average private school applicant had only a 18.3% chance of admission, well below the 25.8% average for public school applicants. That seven percentage point difference translates into a 40% greater chance of admission for public school applicants. Whereas nearly 50% (440 out of 906) of public schools had admission rates over 30%, only 2% of private schools did (3 out of 142)4.

Among San Francisco private schools, the highest admission rate (28%) belonged to Immaculate Conception Academy, almost certainly the least celebrated and least selective of all the private high schools in the city. Archbishop Riordan had a higher admission rate than its near neighbor, Lick-Wilmerding, or its more selective peers, St Ignatius and Sacred Heart.

The lowest admission rate of all belongs to one of the most celebrated private schools in San Jose. Bellarmine College Prep has seen only 7.9% of its applicants accepted by UCSD over the last three years. In 2022, only 6 of the 167 applicants were successful, although that rose to 13 and 19 over the next two years, for a total of 38 over the three years. Over the same period, 39 Bellarmine students were named National Merit Scholar semifinalists, an achievement that requires scoring in the top 1% of students statewide on the PSAT. For Bellarmine students, it’s harder to be get into UC San Diego than to be in the top 1% in the state.

This phenomenon, of more students in the top 1% than are admitted to UC San Diego, is not confined to Bellarmine. It’s true of a number of other private schools I looked at, such as Harvard-Westlake in LA, College Prep in Oakland, Nueva in San Mateo etc. The aforementioned Lynbrook High in San Jose was the only public school I could find with the same sorry distinction.

It makes no sense to think of UCSD evaluating applicants against all the other applicants in the applicant pool. It is clear that different standards are being applied depending on the school that the applicant attended. It makes more sense to think of applicants as competing with their classmates.

What Really Matters In Admission

The principle behind California’s Local Control Funding Formula (LCFF) is that extra money should be given to schools with lots of high-needs students (defined as those eligible for free or reduced price meals, English learners and those in foster care). The Unduplicated5 Pupil Percentage (UPP) measures the share of a school’s total enrollment that falls into one of those three high-needs categories.

For the sake of perspective, most public high schools in San Francisco have UPPs in the 50%-75% range. SOTA is the only one under 25% while both Lowell and Gateway are under 50%. The only schools above 75% are KIPP and International (which is filled with new arrivals who are still English Learners). Yes, even Mission, O’Connell, and Jordan have UPPs under 75%.

Schools with a UPP of 75% or higher are known as LCFF+ Schools. The California legislature has, since 2017, required that UCs report annually on the number of students from LCFF+ schools who apply to, are admitted to, and enroll at each UC campus. Though not required by law to do so, UC San Diego has decided to give students from such schools a big boost in admissions.

This chart shows the average admission rate by UPP over the period 2022-24. There is a very clear discontinuity. Most of the schools with UPPs above 75% have admission rates above 40% while most of the schools with UPPs below 75% have admission rates well below the UCSD average.

I experimented with a little regression analysis to see what factors might predict a school’s admission rate. As you might expect from the graph above, whether or not a school is a LCFF+ school6 is by far the most important factor. The school’s academic quality, as measured by the share of seniors who were proficient in both Math and English, was also significant but, here’s the kicker, the co-efficient was negative. In other words, given two schools that are both LCFF+, the one with the lower proficiency levels will have the higher admission rate7.

A couple of other points:

• The preference applies at the school level, not at the applicant level. The non high-needs students in the LCFF+ schools will be the biggest beneficiaries because they are likely to be stronger students than their high-needs classmates. Conversely, about 40% of high-needs students are in non-LCFF+ schools and they are much less likely to gain admission because they will appear weaker than their classmates, even if they are stronger than high-needs students in LCFF+ schools.

• The fraction of high school students that fall into one or more of the high-needs categories statewide was in the 57%-59% range every year between 2016-17 and 2020-21. It has since risen steadily and reached 65% in 2024-258. As recently as 2021, 35% of high schoolers were in LCFF+ schools. In 2025, it was 45%.

What Has Changed?

It wasn’t always this way. As recently as 2016, a school’s UPP did not affect its average admission rate9.

The number of applications that UC San Diego receives annually from California high school students rose by a huge 57% between 2016 and 2024. All the other UCs also experienced their own jumps in applications. The number of applications had been rising before the pandemic but the elimination of the SAT gave it another boost. Early last year, we looked at how the strength of the applicants had changed over time and found that, while applicants are taking more and more honors classes, the biggest increase in applications came from weak students i.e. those who had taken fewer than five honors classes. That does not mean UCSD is getting disproportionately more applications from LCFF+ high schools. The mix of applications by high school has been surprisingly stable. The share of applications coming from LCFF+ high schools has consistently been in the 17%-19% range and the share coming from high schools with UPPs under 25% has consistently been in the 23%-25% range. The UCs received money from the legislature to grow the number of applications from LCFF+ high schools. That their share of applications hasn’t increased demonstrates that the money was not well spent.

However, the different admission standards that are being applied today has had a significant effect on the composition of the admitted class. Students from LCFF+ high schools used to comprise around 20% of the admitted class. They now comprise over 30%. Meanwhile, the share of admits coming from schools with UPPs below 25% fell by one third, from 24% to 16%.

Winners and Losers

Such a dramatic change in the admissions policy has obviously produced winners and losers. Given the increase in the total number of admits, an increase in the number of admits from LCFF+ schools could have been accomplished without reducing the number of admits from any high school. Nevertheless, hundreds of schools did see a decrease in the number of students admitted to UCSD.

Over the period 2016-18, Lowell High in San Francisco had an average of 115 students admitted to UC San Diego, more than any other high school10. In 2022-24, it had an average of 94 admits per year, a decline of 21 (18%). It was far from the worst affected. Torrey Pines High in San Diego went from an average of 100 admits to an average of just 42, a decline of 58. Monta Vista High in Cupertino went from 104 admits per year down to 48, a decline of 56. Four other high schools also saw their average number of admits decline by 50 or more.

Some schools with lots of high needs students also saw their number of admits decline. Highlighted on the chart are San Gabriel High and Gabrielino High, both of which saw their admits halve from around 50 to around 25 per year. It may or may not be a coincidence that at least 85% of UCSD applicants from both schools were Asian. Evidence that it may be a coincidence is that Bolsa Grande High, another high needs school where 85%+ of the applicants are Asian, saw a big increase in its number of admits.

On the flip side, four schools saw their number of admits increase by at least 50. Granada Hills Charter is now the largest source of admits to UC San Diego, averaging 151 in 2022-24, up from 94 in 2016-18. There were actually a massive 196 admits from Granada Hills in 2024 alone: the average is 151 only because it includes the 74 admits in 2022. Similarly, Eleanor Roosevelt High’s average of 105 is a combination of the 39 admits it had in 2022 and the 147 it had in 2024. Berkeley High saw the biggest increase in absolute terms, up from 50 admits per year to 125. Meanwhile, Birmingham Community Charter went from 18.3 to 71.7, an increase of 291%.

These four schools are all enormous. Berkeley High is the smallest and it has around 800 seniors. Granada Hills has over 1100 seniors. While they are all good schools with lots of strong students, it is unclear why they are so favored by UCSD admissions. Only Birmingham Community Charter is an LCFF+ school. In 2024, sixty-eight high schools produced more graduates who were proficient in both English and Math than Berkeley High. Eleanor Roosevelt’s big increase may have been helped by a conveniently large increase in its UPP. It was between 35% and 40% every year up to 2021 but jumped to 59%, 67%, and 67% in the next three years.

Where do all the students needing remedial help come from?

The big increase in the number of students admitted from LCFF+ high schools coincides exactly with the big increase in students needing remedial help. It is natural, but wrong, to conclude that the former was sufficient to cause the latter.

There are plenty of smart students at LCFF+ high schools. There are more students at those schools who are proficient in both Math and English than apply to UCSD. UCSD accepts about 40% of LCFF+ applicants so, even though every proficient student doesn’t apply, the number of proficient applicants from LCFF+ high schools probably exceeds the number of students admitted from those schools. It should thus be possible to keep the same number of admits but have zero who need remedial classes because all the admits are proficient in both Math and English. That’s not what happens in practice, however, because the UCSD admissions office has no way to know who those proficient students are. Rampant grade inflation means that high school transcripts and GPAs are unable to distinguish strong students from weak. Meanwhile, the UCs have blinded themselves by ignoring SAT scores and paying scant attention to AP exam scores. If you’re picking blindly from all the applicants from Lynbrook High, you’ll probably pick a very strong student because Lynbrook is filled with very strong students. If you’re picking blindly from all the applicants from Crawford High, you may get a competent student but you’re more likely to get someone needing remedial help.

Conclusion

The opaqueness of the whole process is what I find most objectionable. UC San Diego has dramatically changed its process for how it evaluates applicants but there is no public record of it announcing this dramatic change or describing what its new evaluation rubric is or how it has changed. Students and their families are supposed to trust in the integrity of the holistic review process even though it produces results that appear unjustified using publicly available data.

The UCs receive nearly $5 billion in state funding and receive applications from over 100,000 high school seniors every year. Other government-run programs have clear eligibility rules. We deserve a proper accounting of how each UC makes its admissions decisions.

Greetings from the year of Linux on my desktop.

In November, I got fed up and said screw it, I'm installing Linux. Since that article was published, I have dealt with one minor catastrophe after another. None of that has anything to do with Linux, mind you. It just meant I didn't install it on my desktop until Sunday evening.

My goal here is to see how far I can get using Linux as my main OS without spending a ton of time futzing with it - or even much time researching beforehand. I am not looking for more high-maintenance hobbies at this stage. I want to see if Linux is a wingable alternative to Microsoft's increasingly annoying OS.

Ho …

Read the full story at The Verge.

In 2012, MIT launched a new course that would turn out to be extremely popular. The official title was 15.S50, but everybody came to know it as the ‘MIT poker class’.

The course was the brainchild of PhD student Will Ma, who was studying Operations Research. He had played a lot of poker – and won a lot of money – while an undergraduate in Canada, which had made a lot of people curious when he arrived at MIT, including the head of department. 15.S50 was a legitimate MIT class; if students passed, they could get degree credit.

I’ve always liked poker, both as a game and testing ground for wider ideas relating to uncertainty. Although poker has rules and limits, some key information is always concealed. The same problem crops up in many aspects of life. Negotiations, debates, bargaining; they are all incomplete information games. ‘Poker is a perfect microcosm of many situations we encounter in the real world,’ as Jonathan Schaeffer, a pioneer of game playing AI, once told me.

I interviewed Ma about the MIT poker class while researching The Perfect Bet. Perhaps understandably for a university course, the sessions focused on theory rather than gambling with real money. Given the relatively limited time available, Ma tried to focus on the steepest part of the learning curve.

Here are some of the key takeaways when it comes to decision-making under uncertainty:

1. Sometimes the optimal move feels reckless. To succeed in poker, you must be prepared to occasionally take a big calculated risk, or walk away from a situation you’ve sunk resources into. The best actions are often more extreme than a beginner is comfortable with.

2. Confidence is about recovering from mistakes. In one class, lecturer Jennifer Shahade pointed out that ‘confidence is an underrated ingredient to success in chess and poker’. In particular, confident players expect to make errors and move on rather than let one mistake affect their game.

3. Most people lose by doing too much. When starting out in poker, it can be easy to get bored with folding and instead play too many hands, when you should be watching and learning about your opponents without taking unnecessary risks.

4. Your decisions change how others play against you. Shahade noted that in chess, ‘most good players forget who they are playing and focus on the position’. But in poker, ‘monitoring your own image and that of other players is a huge part of the game’.

5. Judge decisions by how they were made, not how they turned out. As Ma put it, ‘I think one of the things poker teaches you very well is that you can often make a good decision but not get a good result, or make a bad decision and get a good result.’

Cover image: Michał Parzuchowski

For the past couple of weeks, I couldn’t shake off an intrusive thought: raster graphics and audio files are awfully similar — they’re sequences of analog measurements — so what would happen if we apply the same transformations to both?…

Let’s start with downsampling: what if we divide the data stream into buckets of n samples each, and then map the entire bucket to a single, averaged value?

for (pos = 0; pos < len; pos = win_size) {
    
  float sum = 0;
  for (int i = 0; i < win_size; i++) sum += buf[pos + i];
  for (int i = 0; i < win_size; i++) buf[pos + i] = sum / win_size;

}

For images, the result is aesthetically pleasing pixel art. But if we do the same audio… well, put your headphones on, you’re in for a treat:

The model for the images is our dog, Skye. The song fragment is a cover of “It Must Have Been Love” performed by Effie Passero.

If you’re familiar with audio formats, you might’ve expected this to sound different: a muffled but neutral rendition associated with low sample rates. Yet, the result of the “audio pixelation” filter is different: it adds unpleasant, metallic-sounding overtones. The culprit is the stairstep pattern in the resulting waveform:

Our eyes don’t mind the pattern on the computer screen, but the cochlea is a complex mechanical structure that doesn’t measure sound pressure levels per se; instead, it has clusters of different nerve cells sensitive to different sine-wave frequencies. Abrupt jumps in the waveform are perceived as wideband noise that wasn’t present in the original audio stream.

The problem is easy to solve: we can run the jagged waveform through a rolling-average filter, the equivalent of blurring the pixelated image to remove the artifacts:

But this brings up another question: is the effect similar if we keep the original 44.1 kHz sample rate but reduce the bit depth of each sample in the file?

/* Assumes signed int16_t buffer, produces n + 1 levels for even n. */

for (int i = 0; i < len; i++) {

  int div = 32767 / (levels / 2);
  buf[i] = round(((float)buf[i]) / div) * div;

}

The answer is yes and no: because the frequency of the injected errors will be on average much higher, we get hiss instead of squeals:

Also note that the loss of fidelity is far more rapid for audio than for quantized images!

As for the hiss itself, it’s inherent to any attempt to play back quantized audio; it’s why digital-to-analog converters in your computer and audio gear typically need to incorporate some form of lowpass filtering. Your sound card has that, but we injected errors greater than what the circuitry was designed to mask.

But enough with image filters that ruin audio: we can also try some audio filters that ruin images! Let’s start by adding a slightly delayed and attenuated copy of the data stream to itself:

for (int i = shift; i < len; i++)
  buf[i] = (5 * buf[i] + 4 * buf[i - shift]) / 9;

Check it out:

For photos, small offsets result in an unappealing blur, while large offsets produce a weird “double exposure” look. For audio, the approach gives birth to a large and important family of filters. Small delays give the impression of a live performance in a small room; large delays sound like an echo in a large hall. Phase-shifted signals create effects such as “flanger” or “phaser”, a pitch-shifted echo sounds like a chorus, and so on.

So far, we’ve been working in the time domain, but we can also analyze data in the frequency domain; any finite signal can be deconstructed into a sum of sine waves with different amplitudes, phases, and frequency. The two most common conversion methods are the discrete Fourier transform and the discrete cosine transform, but there are more wacky options to choose from if you’re so inclined.

For images, the frequency-domain view is rarely used for editing because almost all changes tend to produce visual artifacts; the technique is used for compression, feature detection, and noise removal, but not much more; it can be used for sharpening or blurring images, but there are easier ways of doing it without FFT.

For audio, the story is different. For example, the approach makes it fairly easy to build vocoders that modulate the output from other instruments to resemble human speech, or to develop systems such as Auto-Tune, which make out-of-tune singing sound passable.

In the earlier article, I shared a simple implementation of the fast Fourier transform (FFT) in C:

void __fft_int(complex* buf, complex* tmp, 
               const uint32_t len, const uint32_t step) {

  if (step >= len) return;
  __fft_int(tmp, buf, len, step * 2);
  __fft_int(tmp + step, buf + step, len, step * 2);

  for (uint32_t pos = 0; pos < len; pos += 2 * step) {
    complex t = cexp(-I * M_PI * pos / len) * tmp[pos + step];
    buf[pos / 2] = tmp[pos] + t;
    buf[(pos + len) / 2] = tmp[pos] - t;
  }

}

void in_place_fft(complex* buf, const uint32_t len) {
  complex tmp[len];
  memcpy(tmp, buf, sizeof(tmp));
  __fft_int(buf, tmp, len, 1);
} 

Unfortunately, the transform gives us decent output only if the input buffer contains nearly-steady signals; the more change there is in the analysis window, the more smeared and intelligible the frequency-domain image. This means we can’t just take the entire song, run it through the aforementioned C function, and expect useful results.

Instead, we need to chop up the track into small slices, typically somewhere around 20-100 ms. This is long enough for each slice to contain a reasonable number of samples, but short enough to more or less represent a momentary “steady state” of the underlying waveform.

If we run the FFT function on each of these windows separately, each output will tell us about the distribution frequencies in that time slice; we can also string these outputs together into a spectrogram, plotting how frequencies (vertical axis) change over time (horizontal axis):

Alas, the method isn’t conductive to audio editing: if we make separate frequency-domain changes to each window and then convert the data back to the time domain, there’s no guarantee that the tail end of the reconstituted waveform for window n will still line up perfectly with the front of the waveform for window n + 1. We’re likely to end up with clicks and other audible artifacts where the FFT windows meet.

A clever solution to the problem is to use the Hann function for for windowing. In essence, we multiply the waveform in every time slice by the value of y = sin²(t), where t is scaled so that each window begins at zero and ends at t = π. This yields a sinusoidal shape that has a value of zero near the edges of the buffer and peaks at 1 in the middle:

It’s hard to see how this would help: the consequence of the operation is that the input waveform is attenuated by a repeating sinusoidal pattern, and the same attenuation will carry over to the reconstituted waveform after FFT.

The trick is to also calculate another sequence of “halfway” FFT windows of the same size that overlap the existing ones (second row below):

This leaves us with one output waveform that’s attenuated in accordance with the repeating sin² pattern that starts at the beginning of the clip, and another waveform that’s attenuated by an identical sin² pattern shifted one-half of the cycle. The second pattern can be also written as cos².

With this in mind, we can write the equations for the reconstituted waveforms as:

If we sum these waveforms, we get:

This is where we wheel out the Pythagorean identity, an easily-derived rule that tells us that the following must hold for any x:

In effect, the multiplier in the earlier equation for the summed waveform is always 1; the Hann-induced attenuation cancels out.

At the same time, because the signal at the edges of each FFT window is attenuated to zero, we get rid of the waveform-merging discontinuities. Instead, the transitions between windows are gradual and mask any editing artifacts.

Where was I going with this? Ah, right! With this trick up of our sleeve, we can goof around in the frequency domain to — for example — selectively shift the pitch of the vocals in our clip:

Source code for the effect is available here. It’s short and easy to experiment with.

I also spent some time approximating the transform for the dog image. In the first instance, some low-frequency components are shifted to higher FFT bins, causing spurious additional edges to crop up and making Skye look jittery. In the second instance, the bins are moved in the other direction, producing a distinctive type of blur.

PS. Before I get hate mail from DSP folks, I should note that high-quality pitch shifting is usually done in a more complex way. For example, many systems actively track the dominant frequency of the vocal track and add correction for voiceless consonants such as “s”. If you want to down a massive rabbit hole, this text is a pretty accessible summary.

As for the 20 minutes spent reading this article, you’re not getting that back.

Subscribe now