<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Gemini 1.5 Pro Announced | Kris Yotam</title>
<link rel="stylesheet" href="../../c/site.css">
<meta name="description" content="Google’s next-gen Gemini 1.5 Pro uses Mixture-of-Experts to match Ultra performance at lower compute, with a breakthrough 1M token context window.">
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<div class="mode-container">
<input type="radio" name="theme" id="theme-auto" checked>
<label for="theme-auto" class="auto" title="Auto theme"></label>
<input type="radio" name="theme" id="theme-light">
<label for="theme-light" class="light" title="Light theme"></label>
<input type="radio" name="theme" id="theme-dark">
<label for="theme-dark" class="dark" title="Dark theme"></label>
</div>
<div class="top-bar" role="navigation">
<a href="../../index.html" title="Main page"><img src="../../c/sigma.svg" alt="Main page"></a>
<a href="../../about.html" title="About"><img src="../../c/news.svg" alt="About"></a>
<a href="#" title="Code"><img src="../../c/code.svg" alt="Code"></a>
<a href="../../pics.html" title="Photography"><img src="../../c/camera.svg" alt="Photography"></a>
<a href="#" title="Pictures"><img src="../../c/picture.svg" alt="Pictures"></a>
<a href="../../vids.html" title="Videos"><img src="../../c/video.svg" alt="Videos"></a>
<a href="#" title="Donations"><img src="../../c/beer.svg" alt="Donations"></a>
<a href="#" title="Social"><img src="../../c/elephant.svg" alt="Social"></a>
</div>
<h1>Gemini 1.5 Pro Announced</h1>
<pre class="stat"> File: <a href="../../news.html">/n/kris/news</a>/gemini-1.5-pro-announced
Size: 255 words
Access: (0644/-rw-r--r--) Uid: (1000/kris) Gid: (100/users)
Modify: 2024-02-18
Status: <span class="status-growing">growing</span> confidence: certain importance: 8
Tags: google gemini language-models
</pre>
<p class="preview">Google’s next-gen Gemini 1.5 Pro uses Mixture-of-Experts to match Ultra performance at lower compute, with a breakthrough 1M token context window.</p>
<div class="prose">
<p>Gemini 1.5 Pro <a href="https://blog.google/innovation-and-ai/products/google-gemini-next-generation-model-february-2024/#sundar-note">released</a> this week with the promise.</p>
<blockquote>
"The model delivers dramatically enhanced performance, with a breakthrough in long-context understanding across modalities."<br>
</blockquote>
<p>It was designed to be a mid-size multi modal model that matches the performance of 1.0 Ultra (their largest model) while simultaneously managing to use less compute than the prized heifer. <strong>1.5</strong> uses a transformer, and mixture of experts architecture. MoE allows the model to be split into smaller "expert" narrow llms rather than the traditional monolith neural net. Meaning for any given input, only relevant expert pathways active, leading to more effective training and inference.</p>
<p>The defining feature of <strong>1.5 Pro</strong> is still it's context window however.</p>
<video src="https://krisyotam.com/cdn/pics/collections/krisyotam/content/news/gemini-1.5-pro-context-comparison.mp4" autoPlay loop muted playsInline />
<table>
<tr><td>Metric</td><td>Value</td></tr>
<tr><td>Standard context window</td><td><strong>128,000 tokens</strong></td></tr>
<tr><td>Max context window (preview)</td><td><strong>1 million tokens</strong></td></tr>
<tr><td>Tested in research up to</td><td><strong>10 million tokens</strong></td></tr>
</table>
<p>A context window of 1 million tokens is equivalent to 1 hour of video, 11 hours of audio, >30K lines of code, >700K words.</p>
<p>The defining feature of 1.5 Pro is its context window:</p>
<table>
<tr><td>Metric</td><td>Value</td></tr>
<tr><td>Standard context window</td><td><strong>128,000 tokens</strong></td></tr>
<tr><td>Max context window (preview)</td><td><strong>1 million tokens</strong></td></tr>
<tr><td>Tested in research up to</td><td><strong>10 million tokens</strong></td></tr>
</table>
<p>What 1 million tokens can hold:</p>
<ul>
<li>1 hour of video</li>
<li>11 hours of audio</li>
<li>30,000+ lines of code</li>
<li>700,000+ words</li>
</ul>
</div>
</body>
</html>