In this article
We designed this benchmark to answer a deployment question: which AI models can turn a meeting transcript into a useful, structured summary at practical speed? We compared 13 open models on a GPU, GPT-5.6 Luna, and GPT-5.4 across content quality, valid response structure, response time, throughput, GPU memory, and GPU power.
Main result: quality and speed
Quality uses the same 71 meetings with complete scores for every GPU model. P50 speed uses successful generation calls, with per-model denominators from 114 to 230 responses. Valid-response counts use all 250 attempts per model.
Gemma 4 12B scored 82.92 for overall quality and returned the required structure in 228 of 250 original attempts, or 91.2%. Its median successful generation call took 14.86 seconds.
Gemma 4 31B scored 82.65 with a 19.41-second median. Our resampling analysis placed the difference between the two models from -2.81 to 3.41 points. The quality evidence does not distinguish them at our three-point practical threshold.
Gemma 4 26B scored 80.84 and reached a 4.45-second median. Among the four GPU models that scored above 80, it had the shortest P50 and P95 generation times. Qwen3.6 27B scored 80.22 but took 24.29 seconds at the median.
GPT-5.6 Luna scored 81.89 and ranked third for quality. It returned 207 valid original responses, and its observed API calls took 14.85 seconds at P50 and 22.03 seconds at P95. We show that timing later as an observed API measurement, not as a hardware-controlled GPU result. GPT-5.4 scored 72.18 and ranked tenth; its records contain no comparable response-time or original-structure data.
The chart places all 15 comparison groups in score order. The GPU ranking uses the same 71 meetings with complete scores for every GPU model. Luna contributes 74 valid quality judgments from the 75-meeting sample, and GPT-5.4 contributes 75. Publisher marks identify model families and do not indicate endorsement.
Model-by-model results
The numbered sections follow the quality-chart order. GPU quality scores use the same 71 meetings; Luna and GPT-5.4 use the separately stated judgment counts. P50 measures successful generation calls, while valid original responses use all 250 attempts. Small score gaps are not proof of a meaningful difference, and API timing is not a controlled GPU comparison.
1. Gemma 4 12B
Overall quality: 82.92/100. P50 generation time: 14.86 seconds. Valid original responses: 228/250.
This member of Google’s Gemma 4 family had the highest overall quality point estimate in the recorded comparison. Its action-item score was 82.7, and title quality was 88.4; the title score is reported separately and does not raise the weighted overall result. The comparison with Gemma 4 31B did not separate them at the three-point practical threshold, so the first-place label is not evidence of a decisive quality advantage.
The combination of content quality and 228 valid originals makes it a useful starting candidate for this meeting workflow. There were still 22 attempts without a valid original response, so an application needs validation and a recovery path. Its observed 71.3 GiB peak-memory statistic describes the serving configuration, not the minimum memory needed to load a 12B model. Reproduce the complete runtime before sizing hardware or estimating capacity.
2. Gemma 4 31B
Overall quality: 82.65/100. P50 generation time: 19.41 seconds. Valid original responses: 202/250.
The 31B Gemma 4 model led the GPU group in summary quality at 84.3 and topic quality at 86.5. Those component results may matter when readers need a well-organized account of discussion, but action items carry 45% of the overall score. Its overall result remained close to the 12B model, with overlapping uncertainty; increasing model size did not establish a clear overall-quality win here.
It also scored 80.76 in the five-meeting long-input diagnostic. That is useful follow-up evidence, not a paired improvement from the main sample, because the meetings differ. The tested configuration took longer at the median than Gemma 4 12B and produced 202 valid originals. Evaluate whether its summary and topic behavior helps your own transcripts enough to justify that observed runtime and recovery burden; this study did not measure deployment cost.
3. GPT-5.6 Luna
Overall quality: 81.89/100. P50 generation time: 14.85 seconds. Valid original responses: 207/250.
GPT-5.6 Luna was the measured API candidate alongside the GPU-hosted group. It scored 88.8 for summary and 89.9 for title quality. The overall 81.89 result uses 74 valid judgments from the 75-meeting sample, while the shared GPU comparison uses 71. The chart provides context across those groups; it is not a perfectly matched 15-model quality comparison on identical valid judgment sets.
Its 14.85-second median and 22.03-second P95 describe the observed API request path. They do not show that the service matches a particular GPU’s speed or concurrent capacity. The 207 valid originals also leave format recovery as a product requirement. Luna’s 81.91 long-input score is based on only five separate meetings. Treat these findings as reasons for a workload-specific trial, with independent human checks because this evaluation used an automated judge without human calibration.
4. Gemma 4 26B
Overall quality: 80.84/100. P50 generation time: 4.45 seconds. Valid original responses: 196/250.
Among the four GPU models scoring above 80, Gemma 4 26B had the shortest P50 and P95 generation times; P95 was 6.51 seconds. Google describes the 26B variant as a mixture-of-experts model , which helps explain why the parameter label alone is not a reliable ordering of runtime. The observed speed still belongs to this benchmark’s serving setup, rather than every implementation of the model.
For a product that generates notes immediately after a meeting, the 4.45-second median makes this configuration worth investigating. However, 196 of 250 attempts produced valid original structure, so fast successful calls do not establish an equally fast usable result after retries or repair. Its measured 62.8 GiB memory statistic also cannot be replaced with the number of active parameters. Evaluate time to an accepted summary, including format validation, on the intended hardware.
5. Qwen3.6 27B
Overall quality: 80.22/100. P50 generation time: 24.29 seconds. Valid original responses: 116/250.
Qwen3.6 27B reached an overall score above 80, but fewer than half of all attempts returned the required original structure. The run recorded 114 responses whose only structural defect was a missing final closing tag. That makes the distinction between content quality and packaging concrete: potentially useful meeting content can still fail the parser that supplies titles, topics, and actions to the product.
A narrowly defined closing-tag repair is a reasonable experiment, but its effect was not measured as a new success rate here. The original 116/250 count must remain visible. This was also a relatively slow successful-call path at 24.29 seconds median and 33.22 seconds P95. Before selecting it for interactive use, measure repaired-output validity, content preservation, and time to a usable response; a high content score alone does not settle those questions.
6. GLM 4.7 Flash
Overall quality: 77.97/100. P50 generation time: 7.35 seconds. Valid original responses: 219/250.
GLM 4.7 Flash scored 81.5 for action items, compared with 77.97 overall. That component matters because the benchmark gives actions the largest weight: a summary can read well while assigning work to the wrong person or inventing a deadline. The action score is still an automated rubric result, not an 81.5% probability that any individual action is correct.
Its 219 valid originals and 7.35-second median make it a useful candidate for a workflow centered on extracting commitments. Review actual tasks, owners, dates, and statuses against transcript evidence, particularly where the meeting contains suggestions rather than firm commitments. Also inspect summary and topic coverage instead of selecting solely from the strongest component. The measured runtime and structural result describe this configuration; a production acceptance test should include the same four required outputs.
7. Qwen3 Next 80B
Overall quality: 74.43/100. P50 generation time: 6.84 seconds. Valid original responses: 224/250.
The Qwen3-Next family includes distinct checkpoints, so the 80B label should not be treated as a complete deployment specification. In this recorded configuration, 224 of 250 attempts produced valid original structure and successful calls took 6.84 seconds at the median. Those observations are useful for evaluating response handling, but the overall quality point estimate remained below the leading Gemma configurations.
For a system that already hosts this model, the results suggest checking content quality before undertaking a migration solely for speed. Read whether topics preserve important decisions and whether action owners are supported by the transcript. A high valid-structure count makes outputs easier to process; it does not show that their contents are correct. The benchmark did not test concurrent traffic, so neither the 80B name nor the single-call timing establishes how many simultaneous meetings a server can handle.
8. Mistral Small 4
Overall quality: 73.96/100. P50 generation time: 7.51 seconds. Valid original responses: 160/250.
Mistral Small 4 produced an overall quality point estimate close to Qwen3 Next 80B in this task. That small numerical gap should not be treated as a demonstrated practical difference. The more visible operational contrast is the 160 valid originals: 90 attempts did not yield a response in the required original structure, even though the median successful generation call was 7.51 seconds.
These denominators answer different questions. A successful call can finish quickly while the application still needs to recover an invalid or missing response. Measure the complete path from transcript submission to accepted title, summary, topics, and actions before choosing the model for an end-user latency target. This study did not publish a repair experiment or cost comparison for Mistral, so those missing results cannot be inferred from its position in the quality chart.
9. Phi-4 14B
Overall quality: 72.33/100. P50 generation time: 12.34 seconds. Valid original responses: 44/250.
Phi-4 returned the required original structure in only 44 of all 250 attempts. Its 72.33 content score therefore needs to be read alongside the generation setup: transcripts above 12,000 tokens were excluded, and the model did not enter the separate long-input diagnostic. The all-attempt validity count includes missing responses; it does not mean that every unsuccessful attempt produced malformed text.
For an application processing short meetings only, this aggregate does not directly establish the valid-response rate for that narrower workload. A new test should use its actual length distribution and report attempted, generated, and structurally valid counts separately. Do not reinterpret the benchmark’s 12,000-token exclusion as the model’s universal context limit. The result identifies limitations of the tested pipeline, and those must be resolved before treating its median successful-call timing as a deployable service level.
10. GPT-5.4
Overall quality: 72.18/100. Comparable timing and original-structure counts: not available.
GPT-5.4 is included as a quality comparison using existing summary records and the same scoring criteria. It contributes 75 judgments, whereas the shared GPU comparison contains 71 complete judgments. Its position is therefore a descriptive result for the available records, not evidence from a fully matched timing-and-reliability experiment across all 15 groups.
The absence of comparable response-time and original-structure measurements is consequential: this article cannot use GPT-5.4 to compare end-to-end responsiveness, retries, or cost per accepted summary. Those fields are unknown, not zero. The five-meeting long-input diagnostic scored 61.11, but that small, separate sample should not be interpreted as a universal inability to handle long meetings. Anyone considering this model should collect the missing operational measurements under the same prompt and output requirements as the alternatives.
11. Qwen3.5 9B
Overall quality: 70.77/100. P50 generation time: 9.29 seconds. Valid original responses: 114/250.
Qwen3.5 9B combined a 70.77 quality point estimate with 114 valid originals. Separately, another 114 responses had a missing final closing tag as their only structural error. Those two equal counts refer to different groups and must not be combined into a reported repaired success rate. No such post-repair result was measured in this comparison.
This pattern makes output termination a focused engineering question. A strict repair could be tested for that specific defect, followed by parsing and a check that no content changed. It would not raise the measured content score or establish that all remaining failures are repairable. For a smaller-model deployment, compare time to a validated, useful summary after any recovery with the 9.29-second successful-call median, and retain the original-response result so the cost of recovery stays visible.
12. GPT-OSS 20B
Overall quality: 68.40/100. P50 generation time: 9.30 seconds. Valid original responses: 64/250.
GPT-OSS 20B is an open-weight model, distinct from the hosted GPT comparison groups. Its configuration had the highest measured effective output throughput at 134.4 tokens per second and the lowest observed peak-memory statistic at 12.9 GiB. Those figures are attractive operational observations, but throughput includes request and prompt-processing time, and the memory result is not a universal minimum hardware requirement.
Only 64 of 250 attempts returned valid original structure. The run also recorded 153 closure-only failures, suggesting a specific format-recovery experiment rather than a basis for silently replacing the original count. Its title quality reached 84.5, yet title does not enter the 68.40 overall score. For this use case, producing tokens efficiently and writing a good title did not establish strong summaries, topics, and actions. Evaluate content and accepted-response throughput together before drawing a deployment conclusion.
13. Granite 4.1 8B
Overall quality: 68.16/100. P50 generation time: 10.42 seconds. Valid original responses: 196/250.
Granite 4.1 8B illustrates a third layer beyond content quality and parseable structure: instruction compliance. Although 196 attempts returned valid original structure, only 33.3% of raw titles stayed below the requested 50-character limit. Summaries met the 260-word limit in 80.8% of cases, while only 22.3% of topic details stayed within the requested range. These are separate reported compliance measurements, not extra factors in the overall quality score.
For a product with fixed card sizes or concise meeting notes, these misses can create editing work even when every field parses correctly. A shorter prompt or output constraint would need a new evaluation; trimming text after generation can remove important meaning. Check title length, summary length, and topic detail explicitly in addition to the schema. The 10.42-second median does not include the human effort that might be needed to make an overly long result useful.
14. Qwen3.6 35B
Overall quality: 63.21/100. P50 generation time: 4.27 seconds. Valid original responses: 70/250.
Qwen3.6 35B was fast in this configuration, but its 29.2 topic score substantially reduced the weighted overall result. Summary quality reached 76.5 and action-item quality 73.2; the model’s behavior was therefore uneven across the requested outputs. Separately, the run recorded 133 responses with invalid topic JSON and 14 with combined topic and action JSON defects. Content scores and parser errors are distinct diagnostics.
This is a different recovery problem from a missing final tag. Adding a tag cannot make invalid JSON valid, and reconstructing malformed fields may change their meaning. A production trial would need a stricter output strategy or a validated regeneration path, with latency and quality measured again. The 4.27-second median covers successful generation calls; it is insufficient evidence for a fast, reliable meeting-summary experience when only 70 original attempts met the required structure.
15. Nemotron 3 Nano
Overall quality: 62.95/100. P50 generation time: 2.16 seconds. Valid original responses: 141/250.
Nemotron 3 Nano had the shortest median successful generation time in the displayed GPU comparison. It also had the lowest overall quality point estimate in this task, and 141 of 250 attempts returned valid original structure. Together, these results show why a low generation time is not sufficient to choose a model for notes that readers will rely on for decisions and commitments.
The useful next question is whether its output can meet a narrowly defined quality threshold on the intended meetings. For a draft-assistance workflow, evaluate the corrections a person must make, including omitted decisions and unsupported actions. For automatically distributed notes, evaluate factual acceptance before optimizing speed. This benchmark did not test such a narrower workflow, and its 2.16-second median does not include human review, recovery, or the surrounding application pipeline.
What we evaluated
Every model received the same transcript and instructions. Each response needed four outputs:
Title: a specific name for the meeting.
Summary: a factual account of the important discussion, decisions, outcomes, and context, limited to 260 words.
Topics: organized sections with clear titles, useful detail, and related decisions or actions.
Action items: work that participants committed to complete, including the correct task, owner, due date when available, and current status.
We measured content quality separately from response structure, instruction compliance, generation time, throughput, GPU memory, and GPU power.
The 13 open models ran in the same GPU environment. We generated GPT-5.6 Luna responses for the same task and scored GPT-5.4 summary records with the same quality criteria. GPT-5.6 Sol served only as the evaluator and did not enter the comparison.
How the quality score works
GPT-5.6 Sol checked one response at a time against its source transcript without seeing competing outputs.
Rating scale: A rating of 0 means missing or fundamentally incorrect, 2 means useful with a substantial problem, and 4 means fully supported with no material problem. We multiplied ratings by 25 to report scores from 0 to 100.
Summary: Factual accuracy, important details, decisions, and clarity.
Action items: Transcript support for the commitment, task, owner, due date, and current status.
Topics: Factual accuracy, coverage, organization, repetition, and correct placement of decisions and actions.
Title: A concise, specific title that represents the meeting’s main subject. We report title separately.
Overall quality: Summary contributes 30%, action items 45%, and topics 25%. Action items receive the largest weight because an incorrect task or owner can cause direct product harm.
The overall quality score combines summary, action-item, and topic scores. Title quality, response structure, instruction compliance, and response time remain separate measurements.
These automated scores support consistent model comparison; we did not calibrate the judge against human ratings in this evaluation.
Dataset and scored-sample coverage
The full dataset contains 250 meetings. It spans three transcript-length groups and three action-density groups:
Transcript length: 85 meetings had fewer than 7,000 tokens, 90 had 7,000–12,000 tokens, and 75 exceeded 12,000 tokens.
Action density: 120 meetings had no reference action items, 90 had one to three, and 40 had at least four.
The main quality sample contains 75 meetings: 57 below 7,000 tokens and 18 from 7,000 through 12,000. The sample includes 20 meetings without actions and 55 action-positive meetings. We evaluated transcripts above 12,000 tokens in the separate long-input check later in this article.
Each bar uses its stated denominator. Length colors stay consistent across the full dataset and main sample. The full action view uses three count bins, while the main sample uses a separate binary action-presence split.
Four malformed evaluator responses reduced the shared GPU comparison from 75 to 71 meetings.
Expanding the quality sample from 50 to 75 meetings changed the GPU ranking very little: two adjacent pairs exchanged positions, and the largest score change was 2.20 points. We had not defined this stopping rule in advance, so treat this as descriptive evidence.
The dataset also contains 365 reference actions across 130 action-positive meetings. Detailed reference topics appear in 235 meetings. We used these fields to balance the sample, but the judge scored model responses against transcript evidence rather than treating the references as correct answers.
What the component scores show
GPT-5.6 Luna recorded the highest summary score at 88.8 and the highest title score at 89.9. Among the GPU-hosted models, Gemma 4 31B led summary at 84.3 and topics at 86.5. Gemma 4 12B scored 82.7 for action items and 88.4 for title quality.
GLM 4.7 Flash scored 81.5 for action items. GPT-OSS 20B scored 84.5 for title quality, but title does not enter the overall score. Qwen3.6 35B scored 76.5 for summary, 73.2 for actions, and 82.4 for title; its 29.2 topic score reduced its overall result to 63.21.
Darker cells indicate higher scores. Overall quality combines summary, action items, and topics using the stated weights; the title column remains separate.
Runtime and valid structure
P50 latency is the median time among successful generation calls. P95 marks the time in which 95% of successful calls completed. These measurements cover the recorded generation call, including prompt processing and request overhead. They do not measure concurrent capacity or full application latency.
We call an original response valid when it contains the required title, summary, action-item, and topic structure on the first attempt, without repair or retry. The rate divides valid original responses by all 250 attempts, including attempts that returned no model response.
The chart includes Luna’s observed API response time for context. The Luna request path differs from the GPU execution path, so its timing does not represent a controlled hardware comparison. GPT-5.4 does not appear because its records contain no comparable timing or original-structure measurement.
Among models above 80 quality, Gemma 4 26B had the shortest tail time at 6.51 seconds P95. Gemma 4 12B reached 20.48 seconds, Gemma 4 31B reached 28.02 seconds, and Qwen3.6 27B reached 33.22 seconds. No model led quality, response time, and valid-response rate together.
GPU operating measurements
We ran the 13 GPU-hosted candidates on one NVIDIA RTX PRO 6000 Blackwell Max-Q GPU with 96 GB of memory. Three metrics describe this test setup:
Effective output throughput: median output tokens divided by full generation-call time. This includes prompt processing and request overhead, so it does not represent decoding-only speed.
Observed peak VRAM: the 95th percentile of each response’s highest recorded GPU-memory use. This reflects the tested serving configuration, not the minimum hardware requirement.
Average GPU power: the median average GPU draw during a call. It does not measure total-system power or energy per summary.
Gemma 4 26B recorded 123.8 output tokens per second, 62.8 GiB observed peak VRAM, and 214 W average GPU power. Gemma 4 12B recorded 39.2 tokens per second, 71.3 GiB, and 289 W. Gemma 4 31B recorded 25.1 tokens per second, 66.6 GiB, and 292 W. Qwen3.6 27B recorded 20.0 tokens per second, 76.0 GiB, and 294 W.
GPT-OSS 20B reached the highest effective output throughput at 134.4 tokens per second and the lowest observed peak VRAM at 12.9 GiB. Its overall quality score reached 68.40, and only 25.6% of all attempts returned a valid original response. These measurements describe one model and server configuration; they do not provide a capacity or cost estimate.
Columns group P50 effective output throughput into below 50, 50–100, and above 100 tokens per second. Rows group P95 observed peak VRAM into below 48, 48–72, and above 72 GiB. Chip color shows median average GPU power: 230 W or below, 231–270 W, or above 270 W. Outlines mark quality above 80. The bins describe the tested serving configurations; GPU power measures draw, not energy per summary.
Format and instruction diagnostics
GPT-OSS 20B produced 153 responses whose only structural error was a missing final closing tag. Qwen3.5 9B and Qwen3.6 27B each produced 114 of these closure-only failures.
Qwen3.6 35B produced 133 responses with invalid topic JSON and 14 responses with combined topic and action JSON defects. A closing-tag correction would not resolve those errors.
Structure validity also does not guarantee instruction compliance. Granite 4.1 8B returned valid structure for 196 of all 250 attempts. Only 33.3% of its raw titles stayed below 50 characters, 80.8% of its summaries stayed within 260 words, and 22.3% of its topic details stayed within the requested range.
A missing final tag is a narrow structural defect. Invalid topic JSON changes the response data and needs a different recovery policy. Both counts describe original responses before any repair.
Long-input diagnostic
We scored five separate meetings with transcripts above 12,000 tokens. Twelve GPU-hosted candidates, GPT-5.6 Luna, and GPT-5.4 entered this check. Phi-4 did not enter because its tested generation setup excluded these inputs.
GPT-5.6 Luna scored 81.91, Gemma 4 31B scored 80.76, Qwen3.6 27B scored 77.46, and Gemma 4 12B scored 75.35. GPT-5.4 scored 61.11.
This check uses different meetings from the main sample. Compare models within the long-input chart; do not read the score differences as paired changes from the main result.
What these results do not measure
Generation latency and GPU measurements reflect one serving environment. They do not measure concurrent-request capacity, total-system energy, or performance in another configuration.
The generation records contain no cost data, so we cannot compare cost.
GPT-5.6 Luna’s observed API timing does not match the GPU execution path. GPT-5.4 has no comparable response-time or original-structure record.
Original-response structure validity and content quality measure different properties. A structurally valid response can still miss an instruction, and a malformed response can still contain useful content.
Reproducibility and privacy
We fixed the task instructions, model revisions, generation settings, score weights, evaluator instructions, output format, and meeting samples before reporting these results. We preserved each original model response before checking its structure and report original-response validity separately from quality.
The evaluation records every candidate, denominator, score, and uncertainty range. Raw transcripts, meeting identifiers, participant information, channel information, and evaluator comments remain private.
