Summary: A new study captured 34,000 smartphone screenshots from 17 Android users over one week, using a multimodal LLM to analyze passive screen data. The findings reveal that non-intentional social media use, algorithmically recommended content, and comments are the biggest drivers of digital regret.
Ten years ago, most of us unlocked our phones with a clear goal. Check a text. Send an email. Today, you probably pick up your phone and end up somewhere completely different, unsure of how you got there. A new study set out to capture exactly that feeling, not by asking people about it, but by watching their screens.
How Do You Actually Study Phone Regret?
Most research on digital wellbeing relies on self-reporting. You get a survey asking how much time you spent on Instagram yesterday. The problem? People are terrible at estimating their own screen time. We forget. We round down. We misremember what we were even looking at.
This study, led by Longjie Guo and colleagues at the University of Washington, tried something different. They installed software on 17 Android phones that took a screenshot every five seconds, passively, for one full week. That produced roughly 34,000 screenshots. Then, instead of having humans manually code each image, they fed the screenshots into a multimodal large language model capable of understanding both the text and visual layout on screen.
The researchers did not rely on screenshots alone. They triangulated the visual data with experience sampling, surveys, and interviews. That combination matters. The screenshots show what happened. The other methods help explain how people felt about it.
What the Screenshots Actually Showed
The findings paint a familiar but frustrating picture. Non-intentional use ranked as especially regrettable. This means the moments people felt worst about were not the ones they planned. They were the drift moments, the accidental scrolls, the 'I just opened my phone for one thing' detours.
The Regret Triggers
Two specific content types stood out as top regret drivers: algorithmically recommended content and comments. Think about that for a second. The stuff you did not choose to see, and the stuff other people said, are what leave you feeling worst.
There was also a clear pattern around intention deviation. Participants frequently started with the goal of direct communication, like texting a friend, but ended up browsing social media instead. That shift from purposeful to passive slightly increased their regret. You open Messages, see a notification from another app, tap it, and twenty minutes later you have forgotten who you were trying to text.
Why This Methodology Matters More Than You Think
Here is what makes this study genuinely interesting, and it is not just the findings. It is the approach. Passive five-second screenshots paired with multimodal LLM analysis could fundamentally change how researchers study digital behavior. Traditional methods like screen time APIs only tell you which app was open. They cannot tell you whether you were reading a friend's post or doomscrolling a recommendation feed. Screenshots can.
The full paper is slated for CHI '25, the Conference on Human Factors in Computing Systems, scheduled for late April through early May 2025 in Yokohama, Japan. Right now, the preprint is publicly available on arXiv, though detailed results, specific platform breakdowns, and statistical figures may still be limited in the public version. What we have is a methodology preview with early findings, not a complete picture.
That caveat matters. We cannot say which app generates the most regret, or what specific UI elements function as traps, because those details are not fully available in the public text. We should not pretend otherwise.
Still, the direction is clear. If this method scales, it could give researchers an unprecedented window into the gap between what we intend to do on our phones and what we actually do. The question is not really whether we regret our screen time. Most of us already know the answer. The question is whether understanding the exact mechanics of that regret might finally help us, and the companies designing these feeds, do something about it. Next time you catch yourself mid-doomscroll, ask yourself: did I choose to be here, or did my phone decide for me?
Comments