IMDb Review Summaries

Overview

Role: UX lead for the reviews widget redesign and theme experience. Quality oversight for LLM output. Design of the reg-gating experience for featured reviews.
Timeline: 2025 (redesign, launch) → 2026 (reg-gating, ongoing iteration)
Team: Sarah Emerson (PM). Cross-functional collaboration with Amazon's AGI team, IMDb Foundation and Convex engineering, research, and content operations.
Platforms: iOS, Android, responsive web
Outcome: Launched to 1,565 titles in June 2025, expanded to 1,600 by September. Automated pipeline reached 25K titles by April 2026. Engagement with themes grew +46% month-over-month; customer contributions of new reviews rose +0.66% versus control. 80% of research participants rated the experience "very satisfied."

Every title on IMDb has an average rating. What it doesn't have is a way to quickly know why fans gave it that rating, or what they specifically loved or hated about it. The rating tells you the verdict. The reviews tell you the reason. But there are often hundreds of reviews, and no one has time to read them all.

IMDb Review Summaries closes that gap: a generative AI-powered summary and set of thematic breakdowns that lets customers understand fan sentiment at a glance, then dive deeper into the aspects they care about.

The Problem

Fans use reviews at two moments:

• Before watching: Is this worth my time? Will I like it? Is it appropriate for who I'm watching with?

• After watching: Did other people see what I saw? What am I missing? Where do I stand relative to the crowd?

At both moments, customers are evaluating a title along two dimensions:

• Quality signals (do fans generally like this? are opinions strong or split?)

• Context (does this fit my mood, my situation, my preferences?)

And they need two levels of depth:

• At-a-glance: a fast read that answers "should I care?"

• Deeper dive: the ability to explore why a title works or doesn't for specific aspects (pacing, acting, ending, etc.)

The old Reviews Widget served none of this well. It sat far down the title page. It showed a single featured review. To learn anything meaningful, customers had to navigate to the reviews subpage and scroll through hundreds of individual reviews looking for signal in the noise.

I mapped the problem as an evaluation matrix. Customers wanted the top-right quadrant: at-a-glance signals of context, and deeper dives into quality. The old experience delivered neither.

The Design Thesis

The redesign had to accomplish four things simultaneously:

1. Move reviews higher on the page so they're actually discoverable

2. Show sentiment at a glance without requiring anyone to read paragraphs

3. Let customers drill into specific aspects they care about (acting, story, ending)

4. Preserve, and ideally amplify, the value of individual customer voices so we're deepening engagement, not replacing it

The last point mattered most. IMDb's competitive advantage in an AI-dominated world is human voices. A GenAI summary that flattened reviews into a single machine-generated paragraph would be a betrayal of the platform. A GenAI summary that made human reviews easier to find and act on would deepen the moat.

The bet: successful implementations of GenAI will deepen, not replace, engagement.

Making Room: The "Move & Improve" Experiments

Before I could add GenAI content to the Reviews Widget, I needed the widget to be somewhere customers would actually see. In early 2025 I designed a two-treatment experiment across iOS and web to test moving the widget higher on the title page and improving the featured-review presentation. The experiment did not include the GenAI summary yet. It was infrastructure work, clearing the space where the summary would eventually live.

iOS results (5-week weblab, Mar to Apr 2025):

• Reviews page views +10.21% (+9.2MM annualized, p=0.0)

• Reviews page visits +3.78% (+128K annualized, p=0.0)

• No harm to review submissions

• Treatment 2 (widget moved below Cast + clickable ratings histogram) chose over Treatment 1 on strength of engagement lift

Web results (5-week weblab, Mar to Apr 2025):

• Reviews page views +287.3% (p=0.0)

• Total page views held flat in Treatment 2 (Treatment 1 saw -1.7%, which killed it)

The durable learning: The Cast Widget is the "scroll ceiling" on IMDb title pages. Once customers reach it, most stop scrolling. Anything placed below the Cast Widget will have significantly lower visibility, which has direct implications for both product feature placement and display ad viewability. This finding shaped years of subsequent decisions about title page layout, including where GenAI summaries could and couldn't live.

The move made the redesign possible. Everything that followed sat above the scroll ceiling.

The Redesigned Reviews Widget

With the widget relocated, I redesigned it to solve the two-dimensional evaluation problem (quality + context, at-a-glance + deep-dive). Four components:

1. Ratings histogram. A visual distribution of ratings tells a story before any words are read. Do fans agree? Is the crowd divided? The shape communicates instantly. Each bar is also clickable, filtering the reviews subpage to that rating. One glance for context, one tap for depth.

2. Review summary. A short GenAI-generated paragraph capturing the general sentiment across the most helpful reviews. Deliberately brief. The summary is scaffolding, not a replacement for reading a real review.

3. Themes. Up to 10 tagged themes across positive, mixed, and negative sentiment. Tapping a theme opens a theme summary experience with deeper context. This is where customers can pick their concern (Was the ending disappointing? Is the pacing slow?) and get an answer without wading through hundreds of reviews.

4. Featured reviews. A shoveler of 5 individual human reviews reflecting the title's rating distribution. This is the "voice of the community" layer. GenAI helps you decide whether to read reviews. The featured reviews are the reviews themselves.

The order matters. Histogram first (instant), then summary (5-second read), then themes (structured drill-down), then featured reviews (voices). Each layer down the widget offers a deeper investment for a deeper return.

Theme Sentiment Design

The theme system needed to communicate sentiment across three states (positive, negative, mixed) using color and iconography that would remain legible and scalable across:

• Small mobile screens

• Titles with all three sentiments displayed

• Titles with only one or two sentiments displayed

• Localization (color-only signals fail cross-culturally)

I explored many combinations of icons, badges, and typographic treatments before landing on the final set. The solution: + (green) for positive, – (red) for negative, and a neutral bullseye (gray) for mixed. Simple enough to render at any size. Legible at a glance. Distinguishable without relying on color alone.

The "mixed" state is the weakest of the three. A bullseye signals "neutral" or "the center" more accurately than it signals "hotly debated," which is what mixed sentiment usually means at IMDb. I flagged this concern during the design phase, and post-launch research later confirmed it: customers struggled to interpret what the neutral icon meant. The mixed indicator should communicate the heat of the debate, not just the absence of consensus. That work is now queued for a future iteration.

Building with LLMs

We partnered with Amazon's Artificial General Intelligence (AGI) team to develop the pipeline. For each title, the LLM ingested 250 of the most helpful user reviews and produced three outputs:

1. A short user review summary

2. 10 review themes with sentiments (drawn from a defined vocabulary)

3. 10 theme summaries (deeper text for each theme)

To constrain the LLM's output space, I helped define a taxonomy of 165 possible themes across 10 focus areas (acting, cinematography, story, ending, characters, and so on). We refined it by removing themes that were too similar (like "character depth" and "character development"), themes that were inherently one-sentiment ("trauma" always reads negative), and themes too abstract to be actionable. Each theme received a definition to give the LLM shared context for accurate attribution.

The prompt itself carried IMDb-specific guardrails: no spoilers, follow IMDb content guidelines, hold to a target length, keep the tone neutral rather than editorial.

Launch

The MVP launched in June 2025 to 1,565 titles worldwide, expanding to 1,600 titles by September. The pipeline was rebuilt in Phase 2 (April 2026) with automated quality validation, extending coverage to 25,000 titles.

Engagement metrics (from launch to Q3 2025):

• Engagement with themes: +46% month-over-month, outpacing page visits to titles with summaries (+24% m/m)

• Customer-contributed reviews: +0.66% versus control (validating the thesis that GenAI would deepen, not replace, human contribution)

• Coverage: 1,565 → 1,600 → 25,000 titles

What Research Revealed

We ran a mixed-method post-launch study with 45 IMDb customers who self-reported regularly consuming user reviews. The results validated the core bet, surfaced a design insight I hadn't anticipated, and identified three real usability issues.

The core bet worked

• 80% of participants rated the experience "very satisfied" (top-2 box on a 7-point scale)

• 91% recognized the summary and themes were AI-generated. Transparency was preserved without labeling.

• 60% felt neutral about AI being used in this context. As one participant put it: "I do not feel any differently." The AI didn't feel intrusive, it just felt useful. That neutrality is the win, not enthusiasm.

Customers described the summary and themes as helping them "quickly take in a lot of information" while retaining the ability to "explore deeper details":

"It makes it easy to quickly glance and understand the opinions of the content by the majority, but also it's easy to delve deeper or look at opinions on specific aspects if desired."

"It helped me not waste time scrolling through all the reviews and gave me the general consensus of even the reviews that may be at the bottom because they don't have as many likes."

Pre-watch and post-watch are different products

The most valuable finding was behavioral: the same widget serves two distinct customer moments, and customers use its components differently in each.

• Pre-watch: The summary is the primary tool. Customers use it to quickly understand why a title has its average rating and decide whether to invest their time. They rarely engage with the theme summaries at this stage. Their goal is a fast yes-or-no.

• Post-watch: The themes and theme summaries are the primary tools. Customers use them to compare their sentiment against the crowd, understand shared reactions, and dive into specific aspects they want to think more about. Their goal is reflection and connection, not decision-making.

This has implications for future iteration: the summary should be positioned prominently for pre-watch moments (title page, browse surfaces), while themes should be surfaced as filters on the reviews subpage for post-watch deep-dives. The single widget can serve both moments, but the placement of its parts across surfaces should be tuned to when customers are most likely to want them.

Impact on review contributors

A specific concern with GenAI summaries was that they would demotivate the humans who write reviews. If AI can synthesize opinions, why bother contributing?

The data on previous review writers (n=28):

• 64% felt neutral about the AI summary's impact on their motivation

• 25% felt more motivated (viewing the summary as increased visibility for their reviews)

• 11% felt less motivated (concerns about AI using their content without explicit consent)

The +0.66% contribution lift versus control confirms the aggregate signal: contributors are not being pushed away. But the 11% who feel less motivated is a real signal worth designing around. Consent and attribution around how contributions are used by AI matter, and future iterations should make that relationship transparent.

Usability issues to fix

Three usability problems came out of the study, each with a clear next step:

1. Low discoverability. The summary and themes only appear in the Reviews Widget on the title page. Customers who navigate via quick links miss them entirely. Fix: surface the summary and themes in the reviews subpage and consider promoting them from other title-page ingresses.

2. Neutral theme icon confusion. The icon representing "mixed" sentiment (a bullseye) was hard to interpret. Customers weren't sure whether it meant "the fandom is split" or "no strong opinion." I had flagged this concern in the original design phase. Research confirmed it as a real usability barrier and moved it up the priority list.

3. Linguistic mismatch. Fans describe titles in casual, descriptive language ("the ending was rushed," "the pacing dragged"). The 165-theme taxonomy uses more abstract labels ("Pacing," "Ending"). Customers had to translate. Future iteration: expand theme labels to match how fans actually talk, or allow multiple surface labels to point at the same underlying theme.

Lessons Learned

Working closely with an LLM through launch surfaced four principles I now apply to every GenAI design conversation at IMDb:

1. Be discerning. Not all content benefits from a summary. We removed summaries from title types where a summary is unexpected or unhelpful, including standup comedy, news, and documentaries. A summary of a stand-up special is either a plot summary (breaks the format) or a critique of the comedian (breaks the community norm). Sometimes the right answer is not to summarize.

2. Context is king. LLMs need domain context to produce useful output. At IMDb, that means defining terms the model might otherwise get wrong: what "adaptation" means in a franchise context, what qualifies as a "spoiler," what "helpful" means in a review. When in doubt, provide definitions for everything.

3. Disjointed prompts create disjointed experiences. Our MLP generated the summary, themes, and theme summaries with separate prompts. This produced three consistent bugs: themes mentioned in the summary sometimes didn't appear in the theme list; observations repeated across outputs; and we couldn't extract the original review text to display alongside each theme summary. The learning: for coupled outputs, use a single prompt or a coordinated pipeline that shares state.

4. Automate QA before you scale. For the first 1,600 titles, a small team of us manually reviewed every summary for spelling, grammar, facts, bias, spoilers, and prompt compliance. It didn't scale, but it taught us exactly what an automated quality pipeline needed to check. No GenAI feature is shippable to a broad customer base without an automated way to validate the output against the same criteria a human would.

We rebuilt the Phase 2 pipeline around these lessons: themes first (extractive, evidence-based), then summaries derived from themes, with LLM-as-judge quality validation at every step.

Featured Reviews and Reg-Gating (2026)

By early 2026, a new problem surfaced: the review experience was gated behind sign-in for unauthenticated customers, which was frustrating a real segment of the audience (43% of iOS App Store reviews cited it) while also protecting the GenAI content from scraping.

I contributed UX recommendations to the reg-gating conversation:

1. Prioritize quality and helpfulness over rating-matching. The original selection logic tried to match the aggregate title rating, which suppressed the most extreme opinions that fans actually seek out. Better: surface the most helpful reviews with a balanced positive/negative mix.

2. Consider a simpler alternative. Rather than 5 featured reviews aligned to the rating distribution, show 2 reviews side by side: the most helpful positive and the most helpful negative. It's a compact, opinionated presentation that respects the customer's time.

The decision (approved by Jessica Scheibach, VP): allow 5 featured reviews for unauthenticated customers on both web and apps, while keeping the AI-generated summary fully gated. This threaded the needle: relieved the customer frustration by giving unauth users real value, kept the higher-cost AI content protected from scraping, and improved the underlying selection logic as an invisible quality upgrade.

The rollout is in progress. Web dialed to 50%, apps are following. The reg-gating conversation continues, but the framing shifted from "gate everything" to "gate strategically based on the cost and value of the content."

Reflection

Three things I'll carry forward from this project.

First: GenAI is a lens, not a replacement. The strongest signal from the launch data isn't that the summaries drove engagement. It's that customers who saw the summaries contributed more reviews, not fewer, and that 60% of them felt neutral about AI being in the mix. A well-designed GenAI feature can invite customers deeper into human content without feeling intrusive or performative. That's the design bet we made explicit, and the data supported it. It's now the default assumption I bring to every GenAI conversation at IMDb.

Second: constrain the model, then trust it. The most important design decisions on this project were the ones that shaped the LLM's output space: which titles get summaries, which themes are allowed, what definitions we hand the model, what the output structure looks like. Prompts are UX. Taxonomies are UX. Quality validation is UX. The visual design of the widget mattered, but the invisible design of the pipeline mattered more. The best GenAI features I've seen are the ones where a designer was involved in shaping the model's constraints, not just its output surface.

Third: one widget can serve two moments, but its parts want different placements. The pre-watch and post-watch findings changed how I think about the widget's future. The summary is a decision tool. The themes are a reflection tool. Both belong on the title page, but the themes also want to live on the reviews subpage as filters. Same content, different moments, different surfaces. This is a pattern I expect to repeat across every GenAI summary product IMDb builds: understand when customers reach for each component, and give it prominence at that moment.