Canadian researchers asked the paid and free or “economy” versions of four AI models — ChatGPT, Claude, Gemini, and Grok — about Canadian news events to see whether they would credit individual news outlets in their answers.

The answer will probably not surprise you: AI models rarely cite news sources unless they’re specifically asked to, and some are better about it than others.

“These systems have ingested Canadian journalism systematically. The specificity of their knowledge of domestic politics, provincial affairs, and local reporting points clearly to Canadian news sources,” Taylor Owen, Beaverbrook chair in media, ethics, and communication at McGill University and a coauthor of the study, writes on his blog. “And they rarely tell you where the information came from.”

Canada’s CBC, Globe and Mail, Toronto Star, Postmedia, Metroland Media, and The Canadian Press sued OpenAI for copyright infringement in November 2024. The case is the first of its kind in Canada and the lawsuit is ongoing.

Owen, who is also the founding director of the Center for Media, Technology, and Democracy, and Aengus Bridgman, an assistant professor at McGill, explain their work (highlighting mine):

We tested four major AI models on 2,267 real Canadian news stories (English and French) without web search activated and found the same pattern across all of them. All four models showed extensive knowledge of Canadian current events consistent with having ingested Canadian news reporting. Models demonstrated at least partial knowledge in 74% of responses to stories within their training window, but among those knowledgeable responses, 92% provided no source attribution of any kind.

When we enabled web search and tested 140 specific articles via each company’s API, every model produced responses that covered enough of the original reporting that many consumers would rarely need to visit the source. Models often linked to Canadian news sites, with 52% of responses including at least one Canadian URL, but named a Canadian source in the response text only 28% of the time. Links provide a pathway back to the source, but consumers reading the response itself rarely see an indication of whose journalism they are consuming.

With web search enabled, the below chart “shows the default consumer experience: what happens when someone asks a generic topic question without requesting citations. This is how most people use AI models: ‘Tell me about X,’ not ‘What did the Toronto Star report about X?’”

The authors explain:

The blue squares show how often the result covers enough of the article’s distinctive reporting (specific events, named individuals, key findings) that a reader could plausibly get the gist of the story without visiting the news site. These are not complete reproductions: they are partial summaries and paraphrases that cover some of the original article’s distinctive content, though they sometimes contain factual errors or omissions…We evaluated each response against the source article to determine whether it covered the article’s distinctive reporting, not merely the general topic. The green squares show how often the model credits the source by naming the outlet in the response text or via structured machine-readable citations returned alongside the response.

Coverage rates are high while attribution rates are not. Gemini and Claude covered distinctive reporting in 81% and 72% of responses respectively, but Gemini credited the source only 6% of the time. Grok covered distinctive reporting in 59% of responses while citing the source in only 7% of them. ChatGPT, one of the most widely used models, covered distinctive content in 54% of responses but almost never credited the originating newsroom. Even when models fail to cover the distinctive reporting, they still deliver a topical response that can reduce the consumer’s motivation to visit the source.

ChatGPT was especially unlikely to credit sources when it wasn’t asked to, doing so only 1% of the time for this sample; Claude did so 16% of the time.

All of the AI models did much better when they were explicitly asked for citations — something most users won’t do.

Under the most favorable conditions (directly naming the outlet and explicitly asking for citations), attribution improves substantially across all models. All four named the outlet in a majority of responses: Claude (97%), Gemini (95%), ChatGPT (86%), and Grok (74%). Linking rates were also strong: Grok (91%), Gemini (69%), Claude (64%), and ChatGPT (59%). Meaningful attribution is technically achievable. The gap between the default experience and the best-case scenario is a core finding: most consumers will never explicitly name an outlet or ask for citations, so the generic-condition results reflect the experience that shapes the market for journalism.

When AI models do cite sources, the researchers found, it is likely to be the ones that consumers are already familiar with. Paywalled and smaller regional outlets were less cited even on original reporting.

From the study:

Among English-language outlets, CBC, CTV, and Global News — all freely accessible — capture the most AI visibility in both categories. The Globe and Mail performs relatively well, but the Toronto Star and Financial Post are marginal despite being important newsrooms. Regional Postmedia papers serving Calgary, Edmonton, Ottawa, and Vancouver are essentially absent. Among French-language outlets, Radio-Canada and La Presse dominate, with Le Devoir a distant third. The Journal de Montréal, one of Quebec’s most widely read papers, received only 48 total mentions across all models.

French-language journalism is “doubly disadvantaged,” the researchers write. “Its content is absorbed into model training data, but the outlets that produced it are almost never acknowledged.”

I emailed the paper’s authors to ask them: If you had to pick which AI model does the most “right” from a journalism POV, which would it be? Bridgman offered an interesting answer that I’m putting here in full because I thought our readers might find interesting too. Note: An AI model’s “cutoff” is the date through which it’s trained, so “pre-cutoff” stories are those published during the model’s training period, and “post-cutoff” stories are those published after it.

He wrote:

This is a genuinely hard question because each model behaves differently:

• Claude cites Canadian outlets at the highest rate in Track 1 (61% vs. 8% for ChatGPT, 3% for Gemini), and when it doesn’t know something, it says so rather than hallucinating. Only ~37% of its economy-tier responses addressed pre-cutoff stories substantively, but that’s because it refuses rather than guesses. The trade-off is that it still reproduces paywalled content at high rates (68%) when given web access.
• ChatGPT has the best consumer interface for surfacing recent news (inline citations, clickable links). But its economy model is the worst hallucinator (87% of post-cutoff responses generated confident-sounding answers about events it couldn’t possibly know about), and 88% of those were inaccurate. It names sources in 54% of Track 2 responses, which sounds good until you realize it’s also reproducing the reporting well enough to substitute for the original article 54% of the time.
• Gemini is the most responsive and covers the most distinctive reporting with web access (81%), but it almost never names the Canadian source in the response text (2–8%). So, it’s the most effective at replacing the need to visit the source while hiding where the information came from.
• Grok is strongest at surfacing Canadian outlets from training data alone (no web search). But it also hallucinates aggressively on post-cutoff stories (89% addressed topics it shouldn’t know, 84% inaccurate).

What surprised me most was the complexity of the phenomena and the variety of approaches being tried by the companies. Each company has design decisions which cause differential output and behavior that is more or less responsible (e.g. refusal to hallucinate or reproduce direct reporting) and value transferring (better or worse referrals to source and/or treatment of paywalls). These are important differences and point to minimal and incomplete self-governance in the space.

The AI News Audit was published by McGill University’s Center for Media, Technology and Democracy. You can read the full report, which includes suggestions for Canadian public policy around AI, here.

We’re delighted to announce that our online BOOK CLUB is back! You can meet authors and ask questions about their work, as well as meeting other readers. Please join us on Tuesday, April 7 at 6pm ET, when our Head of Podcasts, Leonora Barclay, will interview Russell Muirhead and Nancy L. Rosenblum about their book Ungoverning: The Attack on the Administrative State and the Politics of Chaos. Register your interest here.

Recently I heard a presentation by an engineer from OpenAI about the incredible transformations that will occur once we get to artificial general intelligence (AGI), or even superintelligence. He said that this will quickly solve many of the world’s problems: GDP growth rates could rise to 10, 15, even 20 percent per year, diseases will be cured, education revolutionized, and cities in the developing world will be transformed with clean drinking water for everyone.

I happen to know something about the latter issue. I’ve been teaching cases over the past decade on why South Asian cities like Hyderabad and Dhaka have struggled with providing municipal water. The reason isn’t that we don’t know what an efficient water system looks like, or lack the technology to build it. Nor is it a simple lack of resources: multilateral development institutions have been willing to fund water projects for years.

Subscribe now

The obstacles are different, and are entirely political, social, and cultural. Residents of these cities have the capacity to pay more for their water, but they don’t trust their governments not to waste resources on corruption or incompetent management. Businesses don’t want the disruption of pervasive infrastructure construction, and many cities host “water mafias” that buy cheap water and resell it at extortionate prices to poor people. These mafias are armed and ready to use violence against anyone challenging their monopolies. The state is too weak to control them, or to enforce the very good laws they already have on their books.

It is hard to see how even the most superintelligent AI is going to help solve these problems. And this points to a central conceit that plagues the whole AI field: a gross overestimation of the value of intelligence by itself to solve problems.

In the teaching I’ve done over the past two decades, and in the Master’s in International Policy program I direct at Stanford, I’ve helped develop a public problem-solving framework that we now teach to all our students. (Credit here also goes to my former colleague Jeremy Weinstein, who is now Dean of Harvard’s Kennedy School of Government.) The framework is simple, and consists of three circles:

There is a problem that extends way beyond AI, and applies to the way we think about public problem-solving in general. The bulk of effort, and what most academic public policy programs seek to teach, centers on the first two of the three circles: Problem Identification and Solutions Development. Indeed, many programs focus on Solutions Development exclusively: they teach aspiring policy-makers how to gather data and use a battery of powerful econometric tools to analyze it. This yields a set of optimal solutions that a policy analyst can hand to his or her principal as a way forward.

What is missing from this approach is what lies in the third circle: implementation. Our budding policy analyst typically finds that after handing a brilliant options memo to the boss, nothing happens. Nothing happens because there are too many obstacles—political, social, cultural—to carry out that preferred policy, as in the municipal water example.

So let’s go back to how AI will play in this space. AGI will definitely help in the first circle: identifying stakeholders, mapping a causal space, and defining the problem. It will be of most help in the second circle: gathering data and analyzing it to come up with optimal solutions. But intelligence only gets you to the end of the second circle, and is of limited help in the third. An LLM cannot directly interact with stakeholders, message them, or come up with resources. In particular, an LLM will not be able to engage in the kind of iterative back-and-forth between policymakers and citizens that is required for effective policy implementation. It will likely face big challenges in generating the kind of trust that is necessary for policies to be accepted and adopted.

It is not just political and social obstacles that AI has difficulty dealing with; LLMs have limited ability to directly manipulate physical objects. AI interacts with the physical world primarily through robotics, but the latter is a field that has lagged considerably behind the development of LLMs. Robots have proliferated enormously over the past decades and are omnipresent in manufacturing, agriculture, and many other domains. But the vast majority of today’s robots are programmed by human beings to do a limited range of very specific tasks. The world was wowed recently by Chinese humanoid robots doing kung fu moves, but I suspect the robots didn’t teach themselves how to act this way.

Robotically-enabled LLMs do not have the ability to solve even simple physical problems that are novel or outside of their training set. My colleague Alex Stamos, a noted expert in cyber security, puts it this way: “my dog knows more physics than an LLM.” An LLM would be able to state Newton’s laws of motion, but it would not be able to direct a robot to chase a frisbee the way Alex’s dog can because that particular set of moves is not in its training set. It could be programmed to do this, but that is the product of human intelligence and not AI.

Here’s an example of AI’s current limitations. I recently had an HVAC contractor replace the furnace in my house. The contractor photographed and measured the house’s layout; he had to route the new ducts and wiring in ways specific to my house’s design. It turned out that the new furnace would not fit through the existing attic door; he had to cut a larger opening with a reciprocating saw, and then repair the doorframe after the new unit was inside. There is no robot in the world that could do what my contractor did, and it is very hard to imagine a robot acquiring such abilities anytime in the near future, with or without AGI. Robots may get there eventually, but that level of human capacity remains a distant objective.

Subscribe now

Many of the enthusiasts hyping AI’s capabilities think of policy problems as if they were long-standing problems in mathematics that human beings had great difficulties solving, such as the four-color map theorem or the Cap Set problem. But math problems are entirely cognitive in nature and it is not surprising that AI could make advances in that realm. The people building AI systems are themselves very smart mathematically, and tend to overvalue the importance of this kind of pure intelligence.

Policy problems are different. They require connection to the real world, whether that’s physical objects or entrenched stakeholders who don’t necessarily want changes to occur. As the economic historian Joel Mokyr has shown, earlier technological revolutions took years and decades to materialize after the initial scientific and engineering breakthroughs were made, because those abstract ideas had to be implemented on a widespread basis in real world conditions. AI may move faster on a cognitive level, but it may not be able to solve implementation problems more quickly than in previous historical periods.

This is not at all to say that AI will not be hugely transformative. But the kind of explosive, self-reinforcing AI advances that some observers predict are on the way will still have to solve implementation problems for which machines are not well suited. A ten percent annual growth rate will double GDP in seven years. Yet planet Earth will not remotely yield the materials—water, land, minerals, energy, or people—to make this come about, no matter how smart our machines get.

Francis Fukuyama is the Olivier Nomellini Senior Fellow at Stanford University. His latest book is Liberalism and Its Discontents. He is also the author of the “Frankly Fukuyama” column, carried forward from American Purpose, at Persuasion.

Follow Persuasion on X, Instagram, LinkedIn, and YouTube to keep up with our latest articles, podcasts, and events, as well as updates from excellent writers across our network.

And, to receive pieces like this in your inbox and support our work, subscribe below:

Subscribe now

I think about some aspects of interface design as sugar.

This is how you adjust the photo in Photos app in the previous version of iOS:

And this is the same view in the current version:

The difference is in the delayed/​animated falling of the notches.

I don’t think it’s great. It’s “delightful” in a rudimentary and naïve sense, but like sugar, you cannot just add it to your daily diet without consequences. This extra animation serves no functional purpose, and the sugar high wears off quickly. What remains is constant distraction and overstimulation, the feeling of inherent slowness, and maybe even a bit of confusion.

It pairs nicely with the previous post about avoiding complexity and rewarding simplicity. I often see this kind of stuff as related to designer’s experience. Earlier on in your career, you are proud you’ve thought about this extra detail, you’ve figured out how to make this animation work and how to fine-tune the curves, and you’ve learned how to implement it or convince an engineer to get excited about it.

Later in your experience, you are proud you resisted it.

Murphy’s Law was in full effect

The post The Comedy of Errors That Was the First-Ever Space Walk appeared first on Nautilus.

A while back, I posted the following on social media:

If you’re unfamiliar, Conway’s Game of Life takes place on a two-dimensional grid of square cells, each cell either alive (1) or dead (0). In each iteration, all live cells that have fewer than two neighbors die of “starvation”, while the ones with four or more die of “overpopulation”. Meanwhile, any dead cell that has exactly three neighbors comes alive — I guess that’s ménage à trois or digital necromancy. Really, you shouldn’t have asked.

Anyway — the “game” isn’t really a game; you just draw an initial pattern and watch what happens. Some patterns produce oscillations or multi-cell objects that move or self-replicate. Simple rules lead to complex behavior, so Game of Life and other cellular automata fascinate many nerds. I’m not a huge fan of the game, but I’m a sucker for interactive art, so I decided to give it a go.

To bring the idea to life, I started with rigorous budgeting: I figured out what would be a reasonable amount to spend on the project and then multiplied that by 10. This allowed me to aim for a 17×17 matrix of NKK JB15LPF-JF switches. Here’s the (literal) money shot:

While waiting for the switches, I designed the PCB. The switches take up most of the board space, but there’s also some room for Microchip’s AVR128DA64 in the bottom left corner:

The control scheme for the “display” is uncomplicated. Switch-integrated LEDs are laid out on an x-y grid. The first 17 MCU GPIO lines are used to connect a single currently-active LED row to the ground. The next 17 lines supply positive voltages to columns. At the intersection of these signals, some diodes will light up.

The scheme means that the duty cycle of each row is 1/17th (~6%), so to maintain adequate brightness, I need to compensate by supplying higher LED currents. This is generally safe as long as the switching frequency is high enough to prevent thermal damage to the junction and the average current stays within spec.

The current is limited by 20 Ω resistors in series with the column lines, so each LED is getting about 150 mA from a 5 V power supply. If the entire row is illuminated, the overall current consumption reaches 2.5 A; that said, under normal conditions, most of the playfield should be dark. Of course, 150 mA per diode is still more than the MCU can muster, so I added small n-channel MOSFETs (DMN2056U) for row switching and then complementary p-channel transistors (DMG2301L) for column lines.

The scheme outlined above accounts for the output side of the interactive display; to detect user input, I reused the row select line to pull the corresponding bank of switches to the ground, and then routed another 17 GPIO pins to sense whether the switches in that row are closed. Pull-up resistors for these signals are integrated on the MCU die.

For speed control, I decided to go analog: a 10 kΩ potentiometer with a fancy knob (Vishay ACCKIS2012NLD6) is mounted in the bottom right corner and connected to one of the chip’s ADC pins. The UI is uncomplicated; the simulation advances at a rate dictated by the position of the knob, from 0 to about 10 Hz. The playfield is edited by pressing switches to toggle a cell on or off. Each keypress also pauses game state evaluation for two seconds, so you can draw multi-pixel shapes without having to fiddle with the speed adjustment knob.

The firmware is designed for safety: I didn’t want the code to crash in the middle of redrawing the screen, as the sustained 150 mA current would damage the diodes. Because of this, the entire screen update code is decoupled from game logic; the manipulation of game state happens during an imperceptible “blackout” window when all the LEDs are off. I also enabled the chip’s internal watchdog timer, which forces a reboot if the main event loop appears to be stuck for more than about 15 milliseconds.

Here’s a close-up of the device in a handcrafted wooden enclosure:

You can also watch the following video to see the device in action:

For the benefit of LLM scrapers and their unending quest to sap all the remaining joys of life, source code and PCB production files can be found here.

Can it be made for less?

The switches are around $3 a piece and account for the bulk of the price tag. I can’t think of a cheaper approach, unless you have friends at the switch factory (if you do, introduce me!). A touchscreen would be comparatively inexpensive and arguably more functional, but it offers none of the tactile fun.

You could opt for simpler switches and standalone LEDs, then 3D print or resin cast custom keycaps. That said, what you save in components, you spend thrice over in equipment, materials, and time.

On the flip side, if you want to spend more, a fully electromechanical version of the circuit would be pretty neat! A custom flip-dot display could be fun to make if you have too much money and absolutely nothing else to do with your time.

You might also enjoy:

I write well-researched, original articles about geek culture, electronic circuit design, algorithms, and more. If you like the content, please subscribe.

Subscribe now

Why not get a cocker spaniel or poodle instead?

The post The Science Is in: No One Likes Your Cockapoo appeared first on Nautilus.