In this article
We designed this benchmark to answer a deployment question: which AI models can turn a meeting transcript into a useful, structured summary at practical speed? We compared 13 open models on a GPU, GPT-5.6 Luna, and GPT-5.4 across content quality, valid response structure, response time, throughput, GPU memory, and GPU power.
Main result: quality and speed
Quality uses the same 71 meetings with complete scores for every GPU model. P50 speed uses successful generation calls, with per-model denominators from 114 to 230 responses. Valid-response counts use all 250 attempts per model.
Gemma 4 12B scored 82.92 for overall quality and returned the required structure in 228 of 250 original attempts, or 91.2%. Its median successful generation call took 14.86 seconds.
Gemma 4 31B scored 82.65 with a 19.41-second median. Our resampling analysis placed the difference between the two models from -2.81 to 3.41 points. The quality evidence does not distinguish them at our three-point practical threshold.
Gemma 4 26B scored 80.84 and reached a 4.45-second median. Among the four GPU models that scored above 80, it had the shortest P50 and P95 generation times. Qwen3.6 27B scored 80.22 but took 24.29 seconds at the median.
GPT-5.6 Luna scored 81.89 and ranked third for quality. It returned 207 valid original responses, and its observed API calls took 14.85 seconds at P50 and 22.03 seconds at P95. We show that timing later as an observed API measurement, not as a hardware-controlled GPU result. GPT-5.4 scored 72.18 and ranked tenth; its records contain no comparable response-time or original-structure data.
The chart places all 15 comparison groups in score order. The GPU ranking uses the same 71 meetings with complete scores for every GPU model. Luna contributes 74 valid quality judgments from the 75-meeting sample, and GPT-5.4 contributes 75. Publisher marks identify model families and do not indicate endorsement.
What we evaluated
Every model received the same transcript and instructions. Each response needed four outputs:
Title: a specific name for the meeting.
Summary: a factual account of the important discussion, decisions, outcomes, and context, limited to 260 words.
Topics: organized sections with clear titles, useful detail, and related decisions or actions.
Action items: work that participants committed to complete, including the correct task, owner, due date when available, and current status.
We measured content quality separately from response structure, instruction compliance, generation time, throughput, GPU memory, and GPU power.
The 13 open models ran in the same GPU environment. We generated GPT-5.6 Luna responses for the same task and scored GPT-5.4 summary records with the same quality criteria. GPT-5.6 Sol served only as the evaluator and did not enter the comparison.
How the quality score works
GPT-5.6 Sol checked one response at a time against its source transcript without seeing competing outputs.
Rating scale: A rating of 0 means missing or fundamentally incorrect, 2 means useful with a substantial problem, and 4 means fully supported with no material problem. We multiplied ratings by 25 to report scores from 0 to 100.
Summary: Factual accuracy, important details, decisions, and clarity.
Action items: Transcript support for the commitment, task, owner, due date, and current status.
Topics: Factual accuracy, coverage, organization, repetition, and correct placement of decisions and actions.
Title: A concise, specific title that represents the meeting’s main subject. We report title separately.
Overall quality: Summary contributes 30%, action items 45%, and topics 25%. Action items receive the largest weight because an incorrect task or owner can cause direct product harm.
The overall quality score combines summary, action-item, and topic scores. Title quality, response structure, instruction compliance, and response time remain separate measurements.
These automated scores support consistent model comparison; we did not calibrate the judge against human ratings in this evaluation.
Dataset and scored-sample coverage
The full dataset contains 250 meetings. It spans three transcript-length groups and three action-density groups:
Transcript length: 85 meetings had fewer than 7,000 tokens, 90 had 7,000–12,000 tokens, and 75 exceeded 12,000 tokens.
Action density: 120 meetings had no reference action items, 90 had one to three, and 40 had at least four.
The main quality sample contains 75 meetings: 57 below 7,000 tokens and 18 from 7,000 through 12,000. The sample includes 20 meetings without actions and 55 action-positive meetings. We evaluated transcripts above 12,000 tokens in the separate long-input check later in this article.
Each bar uses its stated denominator. Length colors stay consistent across the full dataset and main sample. The full action view uses three count bins, while the main sample uses a separate binary action-presence split.
Four malformed evaluator responses reduced the shared GPU comparison from 75 to 71 meetings.
Expanding the quality sample from 50 to 75 meetings changed the GPU ranking very little: two adjacent pairs exchanged positions, and the largest score change was 2.20 points. We had not defined this stopping rule in advance, so treat this as descriptive evidence.
The dataset also contains 365 reference actions across 130 action-positive meetings. Detailed reference topics appear in 235 meetings. We used these fields to balance the sample, but the judge scored model responses against transcript evidence rather than treating the references as correct answers.
What the component scores show
GPT-5.6 Luna recorded the highest summary score at 88.8 and the highest title score at 89.9. Among the GPU-hosted models, Gemma 4 31B led summary at 84.3 and topics at 86.5. Gemma 4 12B scored 82.7 for action items and 88.4 for title quality.
GLM 4.7 Flash scored 81.5 for action items. GPT-OSS 20B scored 84.5 for title quality, but title does not enter the overall score. Qwen3.6 35B scored 76.5 for summary, 73.2 for actions, and 82.4 for title; its 29.2 topic score reduced its overall result to 63.21.
Darker cells indicate higher scores. Overall quality combines summary, action items, and topics using the stated weights; the title column remains separate.
Runtime and valid structure
P50 latency is the median time among successful generation calls. P95 marks the time in which 95% of successful calls completed. These measurements cover the recorded generation call, including prompt processing and request overhead. They do not measure concurrent capacity or full application latency.
We call an original response valid when it contains the required title, summary, action-item, and topic structure on the first attempt, without repair or retry. The rate divides valid original responses by all 250 attempts, including attempts that returned no model response.
The chart includes Luna’s observed API response time for context. The Luna request path differs from the GPU execution path, so its timing does not represent a controlled hardware comparison. GPT-5.4 does not appear because its records contain no comparable timing or original-structure measurement.
Among models above 80 quality, Gemma 4 26B had the shortest tail time at 6.51 seconds P95. Gemma 4 12B reached 20.48 seconds, Gemma 4 31B reached 28.02 seconds, and Qwen3.6 27B reached 33.22 seconds. No model led quality, response time, and valid-response rate together.
GPU operating measurements
We ran the 13 GPU-hosted candidates on one NVIDIA RTX PRO 6000 Blackwell Max-Q GPU with 96 GB of memory. Three metrics describe this test setup:
Effective output throughput: median output tokens divided by full generation-call time. This includes prompt processing and request overhead, so it does not represent decoding-only speed.
Observed peak VRAM: the 95th percentile of each response’s highest recorded GPU-memory use. This reflects the tested serving configuration, not the minimum hardware requirement.
Average GPU power: the median average GPU draw during a call. It does not measure total-system power or energy per summary.
Gemma 4 26B recorded 123.8 output tokens per second, 62.8 GiB observed peak VRAM, and 214 W average GPU power. Gemma 4 12B recorded 39.2 tokens per second, 71.3 GiB, and 289 W. Gemma 4 31B recorded 25.1 tokens per second, 66.6 GiB, and 292 W. Qwen3.6 27B recorded 20.0 tokens per second, 76.0 GiB, and 294 W.
GPT-OSS 20B reached the highest effective output throughput at 134.4 tokens per second and the lowest observed peak VRAM at 12.9 GiB. Its overall quality score reached 68.40, and only 25.6% of all attempts returned a valid original response. These measurements describe one model and server configuration; they do not provide a capacity or cost estimate.
Columns group P50 effective output throughput into below 50, 50–100, and above 100 tokens per second. Rows group P95 observed peak VRAM into below 48, 48–72, and above 72 GiB. Chip color shows median average GPU power: 230 W or below, 231–270 W, or above 270 W. Outlines mark quality above 80. The bins describe the tested serving configurations; GPU power measures draw, not energy per summary.
Format and instruction diagnostics
GPT-OSS 20B produced 153 responses whose only structural error was a missing final closing tag. Qwen3.5 9B and Qwen3.6 27B each produced 114 of these closure-only failures.
Qwen3.6 35B produced 133 responses with invalid topic JSON and 14 responses with combined topic and action JSON defects. A closing-tag correction would not resolve those errors.
Structure validity also does not guarantee instruction compliance. Granite 4.1 8B returned valid structure for 196 of all 250 attempts. Only 33.3% of its raw titles stayed below 50 characters, 80.8% of its summaries stayed within 260 words, and 22.3% of its topic details stayed within the requested range.
A missing final tag is a narrow structural defect. Invalid topic JSON changes the response data and needs a different recovery policy. Both counts describe original responses before any repair.
Long-input diagnostic
We scored five separate meetings with transcripts above 12,000 tokens. Twelve GPU-hosted candidates, GPT-5.6 Luna, and GPT-5.4 entered this check. Phi-4 did not enter because its tested generation setup excluded these inputs.
GPT-5.6 Luna scored 81.91, Gemma 4 31B scored 80.76, Qwen3.6 27B scored 77.46, and Gemma 4 12B scored 75.35. GPT-5.4 scored 61.11.
This check uses different meetings from the main sample. Compare models within the long-input chart; do not read the score differences as paired changes from the main result.
What these results do not measure
Generation latency and GPU measurements reflect one serving environment. They do not measure concurrent-request capacity, total-system energy, or performance in another configuration.
The generation records contain no cost data, so we cannot compare cost.
GPT-5.6 Luna’s observed API timing does not match the GPU execution path. GPT-5.4 has no comparable response-time or original-structure record.
Original-response structure validity and content quality measure different properties. A structurally valid response can still miss an instruction, and a malformed response can still contain useful content.
Reproducibility and privacy
We fixed the task instructions, model revisions, generation settings, score weights, evaluator instructions, output format, and meeting samples before reporting these results. We preserved each original model response before checking its structure and report original-response validity separately from quality.
The evaluation records every candidate, denominator, score, and uncertainty range. Raw transcripts, meeting identifiers, participant information, channel information, and evaluator comments remain private.
