Thesis topic: A Capability Benchmark for On-Device AI Models

Are you a final-year master's student looking for a thesis project that blends real-world impact with cutting-edge tech?

At Framna, we are passionate about creating digital products that shape markets and make everyday life better - and we want you to be part of it.

Every year, we welcome a group of master's students into our Stockholm studio to spend their final semester with us. You will sit alongside our developers, designers, testers, analysts, and product delivery leads, get a place to sit in our cozy office, and join our social and learning activities - from lunch-and-learns to our weekly Friday demos and AWs. It is a chance to do your thesis in a collaborative setting, surrounded by people who are happy to share what they know.

Background
AI is moving onto your device. On-device models can run without an internet connection, and keep data private. But their capabilities are still poorly understood compared to their cloud-based counterparts.

For product teams building mobile apps, the key question is not just how good these models are in isolation, but what they are actually useful for. When is an on-device model good enough and where does it fall short?

In this thesis project, you will design and run a structured benchmark study of on-device AI models across realistic tasks, producing a capability map that product teams can use to make informed decisions about when to run AI locally, in the cloud, or as a combination of both.

What you will do
Some potential tasks within this thesis include:

• Designing a benchmark covering realistic tasks such as summarisation, classification, intent detection, structured extraction, image understanding, OCR, and visual search
• Evaluating on-device language models and vision or multimodal models against a cloud-based frontier model as a quality reference point
• Assessing each model across dimensions including accuracy, task completion rate, latency, memory usage, battery impact, hallucination rate, and robustness to messy input
• Including product-facing metrics such as user-perceived quality, privacy sensitivity, and failure detectability
• Exploring deployment stacks such as Apple's Foundation Models framework or ExecuTorch, and building on existing benchmarks such as MobileAIBench and PalmBench
• Producing a categorization of tasks where on-device models perform well, poorly, or inconsistently, and a decision framework for choosing between local, cloud-based, and hybrid AI

Who you are
This thesis may suit you if you:

• Are interested in mobile development, edge AI or mobile computing
• Are curious about how AI capabilities translate into real product decisions
• Enjoy designing experiments and building evaluation frameworks
• Plan to start your master's thesis project in January 2027
• Have the opportunity to work from our Stockholm studio

About Framna
Framna means 'to bring forward' in Old Norse. And that is exactly what we do. We partner with industry leaders (and those about to be) to create excellent digital products that define markets, reshape industries, and deliver meaningful impact.

Born from the union of digital agencies Bontouch, Move, and Shape, Framna emerged from a strong, product-led culture. Today, over 600 of us collaborate across ten studios in Denmark, the Netherlands, Poland, Sweden, Switzerland, and the US.

Every day, millions of users around the world rely on the products we help shape. From seamless payments to daily commutes, from better health management to smarter shopping. Our work supports more than €150 million in daily transactions and consistently earns an average App Store rating of 4.5.

Together with some of the world's most ambitious brands, including Essity, SJ, Swish, and SEB, we craft products that lead markets and push boundaries. Because when products win, businesses win. We call it Win by product.

Our hiring process

• Psychometric assessment - personality and logic test
• Interview with our recruiter
• Case assignment
• Case interview

Do you want to meet the team in person? Join us on September 17th for our Master's Thesis Evening! You will hear more about our upcoming thesis topics, meet the team behind our projects, and get a feel for how we work at the intersection of creativity and technology. Sign up here!