Thesis: Can AI Agents Replace the Tester? Comparing LLM Browser Agents and Scripted End-to-End Tests

High Level Description
End-to-end (E2E) tests verify that a web application works the way a user experiences it. They are valuable but expensive: scripted tests (e.g., Playwright, Cypress) take time to write, break when the user interface changes, and only find the bugs someone thought of in advance. A new alternative is LLM-based agents that control a real browser. They read the screen, click, type and judge whether the application behaves correctly, much like a human exploratory tester. This thesis will compare AI-driven exploratory testing with traditional scripted E2E tests on open-source web applications with deliberately planted bugs, to find out which approach finds more bugs, how reliable, and at what cost.

Cost is a central question: AI agents are cheap to set up but every test run consumes paid model usage, while scripted tests are expensive to write and maintain but cheap to run.

Project Description
The work will involve setting up one or two open-source web applications as test subjects, preferably from a regulated domain where workflows, user roles and data integrity matter.

To measure how well each testing approach finds bugs, the bugs must be known in advance. A catalogue of 30-50 realistic faults will therefore be built and injected into the applications in a controlled, repeatable way. The faults will cover:

Functional logic - e.g., wrong calculations or incorrect state changes.

Input validation - invalid data is accepted, or valid data is rejected.

Authorization - users can see or change data they should not have access to.

Data integrity - data is silently lost, or audit trail entries are missing.

UI & usability - hidden controls, misleading labels or unreachable steps.

The following testing approaches will then be compared:

Scripted baseline - A scripted end-to-end test suite, written within a fixed time budget.

Free exploration - The agent only gets the URL and login credentials.

Persona-driven exploration - The agent acts as a role (e.g. technician or administrator) with goals to reach.

Spec-guided exploration - The agent also gets user documentation or requirements, so it knows what should happen.

Agent-authored scripts - The agent explores the application and then writes scripted regression tests from what it found.

The AI agents will be built by connecting large language models (LLMs) to a real web browser, so they can navigate, click, type and read the screen like a human tester. At least two different LLMs will be compared, including both a large, more capable model and a smaller, cheaper one, to see how model choice affects bug detection and cost.

Cost will be tracked for every approach throughout the work. For the agents this includes model usage (tokens and API cost), run time and the number of repeated runs needed for reliable results. For scripted tests it includes the time to write the tests, the time to update them after UI changes, and the compute time to run them. Human time spent reviewing agent reports and sorting out false alarms will also be counted.


Who are we looking for?

• Bachelor/Master of Science in Computer Science, Software Engineering, or related fields.
  
• Graduation in June 2027
  
• Fluency in Swedish, both written and spoken
  
• Fluency in English, both written and spoken
  
  

Purpose
In this thesis investigate these questions:

• Bug detection - How many planted bugs does each approach find, per fault category? Are some categories systematically missed?
  

• False alarms - How often do agents report issues that are not real bugs, and how much human effort does it take to sort them out?
  
  
• Stability & reproducibility - How much do results vary across repeated runs, and can a reported bug be reliably reproduced from the agent's report?
  

• Resilience to change - After realistic UI changes (renamed buttons, moved menus, new layouts), which approaches still work, and what does maintenance cost?
  

• Cost per run & per bug - What does a single test run cost for each approach, and what is the cost per bug found? How much can be saved with smaller models, shorter sessions or limiting exploration, and how much bug detection is lost?
  

• Total cost over time - Compare the total cost of each approach over a realistic period, including setup, running, maintenance and human review. When does each approach break even, and how does the cost change if tests run on every code change rather than nightly or before each release?
  
  
  An Exciting Journey with Knightec Group
  Semcon and Knightec have joined forces as Knightec Group. Together, we are Northern Europe's leading strategic partner in product and digital service development. With a unique combination of cross-functional expertise and a holistic business understanding, we help our clients realize their strategies - from idea to complete solution.
  

Practical Information
This is a thesis position, located at our office in Sundsvall. Start date January or March 2027.

Please submit your application as soon as possible, but no later than 2026-11-30. If you have any questions, you are welcome to contact johanna.edstrom@knightec.se. Note that due to GDPR, we only accept applications through our careers page.