arstechnica.com Open in urlscan Pro
3.23.193.196  Public Scan

Form analysis
1 forms found in the DOM

GET /search/

<form action="/search/" method="GET" id="search_form">
  <input type="hidden" name="ie" value="UTF-8">
  <input type="text" name="q" id="hdr_search_input" value="" aria-label="Search..." placeholder="Search...">
</form>

Text Content

Skip to main content
 * Biz & IT
 * Tech
 * Science
 * Policy
 * Cars
 * Gaming & Culture
 * Store
 * Forums

Subscribe

Close


NAVIGATE

 * Store
 * Subscribe
 * Videos
 * Features
 * Reviews

 * RSS Feeds
 * Mobile Site

 * About Ars
 * Staff Directory
 * Contact Us

 * Advertise with Ars
 * Reprints


FILTER BY TOPIC

 * Biz & IT
 * Tech
 * Science
 * Policy
 * Cars
 * Gaming & Culture
 * Store
 * Forums


SETTINGS

Front page layout


Grid


List


Site theme

light

dark

Sign in

SLEEPER CELL —


ONCE “TOO SCARY” TO RELEASE, GPT-2 GETS SQUEEZED INTO AN EXCEL SPREADSHEET


OPENAI'S GPT-2 RUNNING LOCALLY IN MICROSOFT EXCEL TEACHES THE BASICS OF HOW LLMS
WORK.

Benj Edwards - 3/15/2024, 9:56 PM

Enlarge
Getty Images

READER COMMENTS

61

It seems like AI large language models (LLMs) are everywhere these days due to
the rise of ChatGPT. Now, a software developer named Ishan Anand has managed to
cram a precursor to ChatGPT called GPT-2—originally released in 2019 after some
trepidation from OpenAI—into a working Microsoft Excel spreadsheet. It's freely
available and is designed to educate people about how LLMs work.

"By using a spreadsheet anyone (even non-developers) can explore and play
directly with how a 'real' transformer works under the hood with minimal
abstractions to get in the way," writes Anand on the official website for the
sheet, which he calls "Spreadsheets-are-all-you-need." It's a nod to the 2017
research paper "Attention is All You Need" that first described the Transformer
architecture that has been foundational to how LLMs work.


FURTHER READING

Researchers, scared by their own work, hold back “deepfakes for text” AI

Anand packed GPT-2 into an XLSB Microsoft Excel binary file format, and it
requires the latest version of Excel to run (but won't work on the web version).
It's completely local and doesn't do any API calls to cloud AI services.

Even though the spreadsheet contains a complete AI language model, you can't
chat with it like ChatGPT. Instead, users input words in other cells and see the
predictive results displayed in different cells almost instantly. Recall that
language models like GPT-2 were designed to do next-token prediction, which
means they try to complete an input (called a prompt, which is encoded into
chunks called tokens) with the most likely text. The prediction could be the
continuation of a sentence or any other text-based task, such as software code.
Different sheets in Anand's Excel file allow users to get a sense of what is
going on under the hood while these predictions are taking place.

Advertisement


Spreadsheets-are-all-you-need only supports 10 tokens of input. That's tiny
compared to the 128,000-token context window of GPT-4 Turbo, but it's enough to
demonstrate some basic principles of how LLMs work, which Anand has detailed in
a series of free tutorial videos he has uploaded to YouTube.


A video of Iman Anand demonstrating "Spreadsheets-are-all-you-need" in a YouTube
tutorial.

In an interview with Ars Technica, Anand says he started the project so he could
satisfy his own curiosity and understand the Transformer in detail. "Modern AI
is so different from the AI I learned when I was getting my CS degree that I
felt I needed to go back to the fundamentals to truly have a mental model for
how it worked."

He says he was originally going to recreate GPT-2 in JavaScript, but he loves
spreadsheets—he calls himself "a spreadsheet addict." He pulled inspiration from
data scientist Jeremy Howard's fast.ai and former OpenAI engineer Andrej
Karpathy's AI tutorials on YouTube.

"I walked away from Karpathy's videos realizing GPT is mostly just a big
computational graph (like a spreadsheet)," he says, "And [I] loved how Jeremy
often uses spreadsheets in his course to make the material more approachable.
After watching those two, it suddenly clicked that it might be possible to do
the whole GPT-2 model in a spreadsheet."

We asked: Did he have any difficulty implementing a LLM in a spreadsheet? "The
actual algorithm for GPT2 is mostly a lot of math operations, which is perfect
for a spreadsheet," he says. "In fact, the hardest piece is where the words are
converted into numbers (a process called tokenization) because it's text
processing and the only part that isn't math. It would have been easier to do
that part in a traditional programming language than in a spreadsheet."

Advertisement


When Anand needed assistance, he naturally got a little help from GPT-2's
descendant: "Notably ChatGPT itself was very helpful in the process in terms of
helping me solve thorny issues I would come across or understanding various
stages of the algorithm, but it would also hallucinate so I had to double-check
it a lot."


GPT-2 RIDES AGAIN

This whole feat is possible because OpenAI released the neural network weights
and source code for GPT-2 in November 2019. It's particularly interesting to see
that particular model baked into an educational spreadsheet, because when it was
announced in February 2019, OpenAI was afraid to release it—the company saw the
potential that GPT-2 might be "used to generate deceptive, biased, or abusive
language at scale."

Still, the company released the full GPT-2 model (including weights files needed
to run it locally) in November 2019, but the company's next major model, GPT-3,
which launched in 2020, has not received an open-weights release. A variation of
GPT-3 later formed the basis for the initial version of ChatGPT, launched in
2022.


A video of Anand demonstrating "Spreadsheets-are-all-you-need" at AI Tinkerers
Seattle, October 2023.

Anand's spreadsheet implementation runs "GPT-2 Small," which, unlike the full
1.5-billion-parameter version of GPT-2, clocks in at 124 million parameters
(parameters are numerical values in AI models that store patterns learned from
training data). Compared to the 175 billion parameters in GPT-3 (and even larger
models), it probably would not qualify as a "large" language model if released
today. But in 2019, GPT-2 was considered state of the art.


FURTHER READING

ChatGPT is one year old. Here’s how it changed the tech world.

You can download the GPT-2-infused spreadsheet on GitHub, though be aware that
it's about 1.2GB. Because of its complexity, Anand said it can frequently lock
up or crash Excel, especially on a Mac; he recommends running the sheet on
Windows. "It is highly recommended to use the manual calculation mode in Excel
and the Windows version of Excel (either on a Windows directory or via Parallels
on a Mac)," he writes on his website.

And before you ask, Google Sheets is currently out of the question: "This
project actually started on Google Sheets, but the full 124M model was too big
and switched to Excel," Anand writes. "I’m still exploring ways to make this
work in Google Sheets, but it is unlikely to fit into a single file as it can
with Excel."


ARS VIDEO


HOW LIGHTING DESIGN IN THE CALLISTO PROTOCOL ELEVATES THE HORROR




READER COMMENTS

61
Benj Edwards Benj Edwards is an AI and Machine Learning Reporter for Ars
Technica. In his free time, he writes and records music, collects vintage
computers, and enjoys nature. He lives in Raleigh, NC.

Advertisement



PROMOTED COMMENTS

Galeran
Just in case you're curious about the resource consumption. On my Windows 11
system, the spreadsheet opened in "Microsoft® Excel® for Microsoft 365 MSO
(Version 2402 Build 16.0.17328.20124) 64-bit" idles at about 2GB RAM, 8-9GB
(10-11GB peaks) while calculating (seems to utilize 4 cores fully). It was
feeling like it was taking maybe half a minute to compute the 11th token
(i9-14900k), but I didn't actually time it.

I started with "Water" and " is" as my first two tokens, expecting it to suggest
something obvious like "wet" as the next token. When it didn't, I copied its
suggested tokens one at a time to get "Water is a great way to get a little more
energy". Hmm. I suppose 10-token hallucinations would be fairly mild.
March 15, 2024 at 11:26 pm



CHANNEL ARS TECHNICA

UNSOLVED MYSTERIES OF QUANTUM LEAP WITH DONALD P. BELLISARIO

Today "Quantum Leap" series creator Donald P. Bellisario joins Ars Technica to
answer once and for all the lingering questions we have about his enduringly
popular show. Was Dr. Sam Beckett really leaping between all those time periods
and people or did he simply imagine it all? What do people in the waiting room
do while Sam is in their bodies? What happens to Sam's loyal ally Al? 30 years
following the series finale, answers to these mysteries and more await.

 * UNSOLVED MYSTERIES OF QUANTUM LEAP WITH DONALD P. BELLISARIO

 * UNSOLVED MYSTERIES OF WARHAMMER 40K WITH AUTHOR DAN ABNETT

 * SITREP: F-16 REPLACEMENT SEARCH A SIGNAL OF F-35 FAIL?

 * SITREP: BOEING 707

 * STEVE BURKE OF GAMERSNEXUS REACTS TO THEIR TOP 1000 COMMENTS ON YOUTUBE

 * MODERN VINTAGE GAMER REACTS TO HIS TOP 1000 COMMENTS ON YOUTUBE

 * HOW THE NES CONQUERED A SKEPTICAL AMERICA IN 1985

 * SCOTT MANLEY REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * HOW HORROR WORKS IN AMNESIA: REBIRTH, SOMA AND AMNESIA: THE DARK DESCENT

 * LGR'S CLINT BASINGER REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * THE F-35'S NEXT TECH UPGRADE

 * HOW ONE GAMEPLAY DECISION CHANGED DIABLO FOREVER

 * UNSOLVED MORTAL KOMBAT MYSTERIES WITH DOMINIC CIANCIOLO FROM NETHERREALM
   STUDIOS

 * US NAVY GETS AN ITALIAN ACCENT

 * HOW AMAZON’S “UNDONE” ANIMATES DREAMS WITH ROTOSCOPING AND OIL PAINTS

 * FIGHTER PILOT BREAKS DOWN EVERY BUTTON IN AN F-15 COCKPIT

 * HOW NBA JAM BECAME A BILLION-DOLLAR SLAM DUNK

 * LINUS "TECH TIPS" SEBASTIAN REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * HOW ALAN WAKE WAS REBUILT 3 YEARS INTO DEVELOPMENT

 * HOW PRINCE OF PERSIA DEFEATED APPLE II'S MEMORY LIMITATIONS

 * HOW CRASH BANDICOOT HACKED THE ORIGINAL PLAYSTATION

 * MYST: THE CHALLENGES OF CD-ROM | WAR STORIES

 * MARKIPLIER REACTS TO HIS TOP 1000 YOUTUBE COMMENTS

 * HOW MIND CONTROL SAVED ODDWORLD: ABE'S ODDYSEE

 * BIOWARE ANSWERS UNSOLVED MYSTERIES OF THE MASS EFFECT UNIVERSE

 * CIVILIZATION: IT'S GOOD TO TAKE TURNS | WAR STORIES

 * SITREP: DOD RESETS BALLISTIC MISSILE INTERCEPTOR PROGRAM

 * WARFRAME'S REBECCA FORD REVIEWS YOUR CHARACTERS

 * SUBNAUTICA: A WORLD WITHOUT GUNS | WAR STORIES

 * HOW SLAY THE SPIRE’S ORIGINAL INTERFACE ALMOST KILLED THE GAME | WAR STORIES

 * AMNESIA: THE DARK DESCENT - THE HORROR FACADE | WAR STORIES

 * COMMAND & CONQUER: TIBERIAN SUN | WAR STORIES

 * BLADE RUNNER: SKINJOBS, VOXELS, AND FUTURE NOIR | WAR STORIES

 * DEAD SPACE: THE DRAG TENTACLE | WAR STORIES

 * TEACH THE CONTROVERSY: FLAT EARTHERS

 * DELTA V: THE BURGEONING WORLD OF SMALL ROCKETS, PAUL ALLEN'S HUGE PLANE, AND
   SPACEX GETS A CRUCIAL GREEN-LIGHT

 * CHRIS HADFIELD EXPLAINS HIS 'SPACE ODDITY' VIDEO

 * THE GREATEST LEAP, EPISODE 1: RISK

 * ULTIMA ONLINE: THE VIRTUAL ECOLOGY | WAR STORIES

More videos
← Previous story Next story →


RELATED STORIES




TODAY ON ARS

 * Store
 * Subscribe
 * About Us
 * RSS Feeds
 * View Mobile Site

 * Contact Us
 * Staff
 * Advertise with us
 * Reprints


NEWSLETTER SIGNUP

Join the Ars Orbital Transmission mailing list to get weekly updates delivered
to your inbox. Sign me up →



CNMN Collection
WIRED Media Group
© 2024 Condé Nast. All rights reserved. Use of and/or registration on any
portion of this site constitutes acceptance of our User Agreement (updated
1/1/20) and Privacy Policy and Cookie Statement (updated 1/1/20) and Ars
Technica Addendum (effective 8/21/2018). Ars may earn compensation on sales from
links on this site. Read our affiliate link policy.
Your California Privacy Rights | Manage Preferences
The material on this site may not be reproduced, distributed, transmitted,
cached or otherwise used, except with the prior written permission of Condé
Nast.
Ad Choices





WE CARE ABOUT YOUR PRIVACY

We and our 164 partners store and/or access information on a device, such as
unique IDs in cookies to process personal data. You may accept or manage your
choices by clicking below or at any time in the privacy policy page. These
choices will be signaled to our partners and will not affect browsing data.More
information about your privacy


WE AND OUR PARTNERS PROCESS DATA TO PROVIDE:

Use precise geolocation data. Actively scan device characteristics for
identification. Store and/or access information on a device. Personalised
advertising and content, advertising and content measurement, audience research
and services development. List of Partners (vendors)

I Accept
Show Purposes