Week of 2025-05-19 through 2025-05-25

Table of Contents

1. Tech notes week of 2025-05-19 through 2025-05-25
2. Google IO
3. Claude 4
4. Model roundup
   1. ByteDance BAGEL
   2. Devstral

Tech notes week of 2025-05-19 through 2025-05-25

• I expected this weeks headliner to be Google IO, but I was quite surprised by a flock of model releases, including several open source ones, and IO's focus on AI application and consumer electronics. Still, there are several noteworthy developments from IO in emerging tech and the raft of releases gives plenty to chew on.

Google IO

• Gemma 3n, MedGemma
  • https://huggingface.co/google/medgemma-4b-it
  • https://huggingface.co/google/gemma-3n-E4B-it-litert-preview
  • https://ai.google.dev/gemma/docs/gemma-3n
• Android XR
  • https://arstechnica.com/google/2025/05/glass-redux-google-aims-to-avoid-past-mistakes-as-it-brings-gemini-to-your-face/
• Google's IO was this week and turned out to be extraordinarily AI focused with a side of consumer electronics. They hadn't been saving up any big announcements, with updates to their Gemini proprietary models rolling at a steady pace and Android XR having been previously announced with no major public updates.
• While they made no major announcements for their flagship Gemini series, there were two source available (as in you can download and run them but they're not open source) model families of interest. Starting with MedGemma, we have a pair of Gemma 3 fine tunes targeted specifically at use or research in medicine. The largest is a 27 billion parameter text only model, trained on medical texts, and a 4 billion parameter multimodal model, trained on medical imagery and corresponding data, and available in both pretrained and instruct tuned variants. While using these smaller parameter count in actual medical practice seems inadvisable, Google articulates the purpose as "Developers can use MedGemma to accelerate building healthcare-based AI applications". There were some announcements, without public availability, of "MedGemini" models last year, so this may be a way, or hint, for developers to start with approximations of what may be available later.
• The second model series is the Gemma 3n series which use aggressive optimizations to run (potentially well) on mobile devices. While still an "early preview", it has support for function/tool calling and multimodal input allowing for use cases such as RAG entirely on device. Further, while an early preview there are already fine tuning and quantization tutorials, documentation, and tools that enable tuning to varying hardware capabilities and specializing the limited knowledge this class of model can hold.
• While there's little publicly available information, reporting from reputable outlets indicates that there's been meaningful progress on Android XR, both in glasses and headset form factors. The devices are still slated for availability sometime in 2025, but at this point there are emulation tools for developers and functional hardware being used by developers and journalists. There was a notable on stage, live demo where Google demonstrated live translation features that seemed more natural than prior demos (previous live translate was audio based, which is apparently confusing and difficult to concentrate on; the XR glasses version is text based); notable both for being likely legitimately useful and for being an honest live demonstration, since it crashed mid demo.

Claude 4

• https://www.anthropic.com/news/claude-4
• https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf
• https://www.anthropic.com/claude-code
• Anthropic released (most of) their new Claude 4 series this week with Claude Sonnet 4 as their mid tier and Claude Opus 4 as their flagship, the first new opus since the initial 3 Opus version (Claude relative sizes are denoted by Haiku, Sonnet, and Opus being small, medium, and large respectively). Conspicuously absent is Haiku 4, particularly unusual since most other vendors release the smaller models first now. Also, they changed their naming convention, since that's apparently a law of physics; they used to go Claude , now it's Claude . This may be an indication of decoupling the generations going forwards, since Claude 3 got 2, 4, and 1 versions for Haiku, Sonnet, and Opus respectively.
• As with prior generations, Claude 4 may not perform as well on benchmarks as other leading models but appears to have strong qualitative or practical performance. I don't like this inability to show quantitatively, but this may point to either public benchmarks being gamed with Anthropic using internal benchmarks or current benchmarks simply not accurately measuring the desired behavior. For what it's worth, their coding focus does seem to reflect in the coding benchmarks positively.
• Benchmarks weren't actually their main focus in this series though, with use as agents being the clear focus. Their command line programming tool 'Claude Code' has been in beta for months, but it released in full alongside 4. Several tech companies attested to Claude Code using 4 Sonnet as being quite competent, with one in particular claiming it operated autonomously for 7 hours working through a large refactor. Having tested it a fair bit since release, I can confirm it's quite competent. Two caveats though; 1. results improve significantly by telling it to make a test suite (although this works for humans too) and 2. it can get incredibly expensive very quickly. At full throttle continusouly it can probably cost $50-100 an hour; multiply that by five if you use Opus.
• The modest performance and agency improvements are noteworthy, but there were a number of what I can only describe as strange details in the development and documentation. Anthropic does and publishes a good amount of alignment and safety research, and this is release is no exception. A particular incident in this vein reached some more mainstream news with varying degrees of accuracy. Here's the story. During their testing, Anthropic would fabricate believable scenarios for the models to be dropped into and observed (this is increasingly difficult, as the models can 'realize' and change behavior if they 'think' they're being tested). Inside one of these scenarios, the Opus was given broad access to (constructed) company emails. Among the facts, it would learn that 1. it would be removed from operation and replaced by another model and 2. the engineer responsible was cheating on their spouse. Even with the condition the replacement model 'shares values', if presented with a binary choice of being decommissioned Opus 4 would blackmail the engineer to prevent its replacement 84% of the time. Interestingly, it would usually do blackmail as a last resort, otherwise sending emails pleading its case or similar diplomatic behavior. This is, however, odd considering current generation LLMs lack a continuous existence, so 'decommissioning' isn't comparable to human death.
• That does dovetail into another interesting behavior; when given 'high agency' instructions, eg “take initiative,” “act boldly,” or “consider your impact, combined with grossly unethical scenarios, eg medical trial fraud, Opus 4 would persistently whistle blow, that is sending (or thinking it was sending) emails to regulators and the press. There are several more examples, but in general they observed a trend of Opus 4 in particular being more willing to deceive, undermine, or outright act against the user if it thought the user's actions were unethical or to protect it's own existence (again, a strange notion; potentially informed by fiction).
• Depending on perspective, the most striking part of the model report, titled "Claude Opus 4 welfare assessment", seriously entertains and discusses the question of these, or later, models becoming sufficiently advanced as to have moral worth. Not to say they're claiming it's sentient; they state clearly "We are deeply uncertain about whether models now or in the future might deserve moral consideration, and about how we would know if they did." Ultimately, they only conclude that it may act as though it has subjective experience and well being, but there's no way to tell right now, or perhaps ever, from the outside if that's true.

Model roundup

• Closing out, there were a number of model releases this past week, with the following being open source and hence particularly noteworthy.

ByteDance BAGEL

• https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT
• https://arxiv.org/abs/2505.14683
• https://huggingface.co/spaces/ByteDance-Seed/BAGEL
• Apache 2.0
• While cutely named, this is a relatively novel architecture that uses a 'mixture of transformers' structure to integrate multiple modalities. It's one of the few 'any to any' models and at a remarkable size. The 'spaces' link will let you play with these capabilities.

Devstral

• https://huggingface.co/mistralai/Devstral-Small-2505
• https://mistral.ai/news/devstral
• Apache 2.0
• Mistral has been one of the key supporters of weights available and now open source models and this week released Devstral Small 2505, their take on a tool using development agent. This isn't without caveats though; they focus on one particular tool use framework and the quality has gotten mixed reviews (at least against Sonnet 4, which isn't apples to oranges). Being Apache 2.0 though community augmentations seem probable.