Time to Guess! (+ Reflections on the Results)

When playing the guessing game, please remember to keep things fair and do not attempt to find the original, "uncensored" text on the internet. You should also generally avoid googling for anything that would give away the answer. If you coincidentally already know the original blog post (and therefore have had the answer "spoiled"), please abstain from the game by entering "I already know the original blog post" in the answer box.

One exception to the "no googling" rule: if you think you know what specific NLP algorithm the blog post was about, but you just don't know what year that algorithm was invented, you are allowed to look that up. For example, if you already think the blog post is about Markov Chains, you are allowed to search for something like "what year was the Markov Chain invented".

Make Your Guess!

The Answer Revealed!

And the answer is...the blog post was written in 2015!

The blog post was specifically about a model called Recurrent Neural Networks, or RNNs. We won't learn about RNNs specifically in this class, but (when applied to text, as done in this blog post) they belong to a broad family of models known as language models, which we will be spending quite a bit of time on.

For those who are curious, the original blog post was "The Unreasonable Effectiveness of Recurrent Neural Networks" by Andrej Karpathy.

Reflection

Now that the answer has been revealed, take a moment to reflect on the following questions:

One major thing that stands out to us (the instructors) about this blog post is how many features it has in common with a lot of modern writing about LLMs, despite being over 10 years old (and predating the release of ChatGPT by about 7 years). Seriously, just consider the following parallels:

• Right off the bat, Karpathy introduces RNNs as being "magical" and frames the blog post as "sharing...that magic with you". Not only is this kind of "hype" language still common in modern discourse about LLMs and NLP, the specific lens of such models being "magic" remains a fairly widespread one.
• The exact choice of examples Karpathy uses to illustrate the magic is, from top to bottom: imitating the style of known authors (Paul Graham, Shakespeare), writing mathematical proofs, writing code, and inventing novel words (baby names). Other than the baby names, these are exactly the same set of capabilities that modern LLMs are often advertised with. Of course, there is still a massive quality difference in the generated output, and getting RNNs to do these tasks was more technically involved than just writing a prompt. But the point is that the set of capabilities that today's authors tend to find impressive in LLMs is largely the same as the set of capabilities that Karpathy found impressive in RNNs.
• Karpathy frequently and consistently uses anthropomorphic/personifying language when discussing the RNNs. He talks about temperature settings making the RNN "more confident", describes the model as "trying" to generate LaTeX diagrams and "deciding" sample a new file, and perhaps most interestingly, uses the word "hallucinate" to describe the model generating a nonexistent Yahoo URL (again, somewhere around 7 years before "hallucination" became a broadly applied term for similar behavior from models like ChatGPT). To be clear, we aren't bringing this up to make a value judgment—humans have a natural tendency to anthropomorphize, and it may well be that from a writing perspective, this kind of personifying language is legitimately the most clear and readable. We simply wish to, again, point out the parallel between how RNNs are described here and how LLMs are described today.

While many of the course goals laid out in the syllabus are technical in nature, we will point out that one of the goals is for you to be able to "Critique the assumptions and approximations that are made about language when developing NLP problems, algorithms, datasets, and evaluations". And this, of course, connects to the HMC mission statement and its emphasis on training you to be leaders in your field who are aware of your impact on society.

Put another way: presumably, most of you are taking this class because you're interested in doing NLP, and some subset of you may walk away from this class deciding to pursue a career that involves developing or using NLP systems. If and when this happens, you will inevitably be put in the position of having to evaluate new NLP systems. You may even get asked for your opinion as a subject-matter expert! In that context, it's important to be able to take a critical eye towards claims that are being made about the system, and sort out the hype from the facts.

And notice that taking a critical eye and sorting out the hype is not the same thing as just "being negative about everything"! Recognizing that RNNs got hyped similarly to LLMs doesn't change the fact that modern LLMs really do have much better text generation quality, do have a wider range of capabilities, and are more user-friendly. The point of taking a critical eye is not to deny the usefulness of LLMs, or whatever new NLP system you're evaluating in the future. It's to try your best to detach yourself from your initial emotional reaction ("Wow, it can generate realistic-looking Shakespeare, we're literally living in the future!") and to focus instead on the facts: what can (and can't) this system do, and how does it connect to the historical context of NLP?

We'll aim to continually practice and refine those skills throughout the semester as you gain more and more historical context about past and present NLP algorithms...and we hope you look forward to that journey just as much as we do!

(When logged in, completion status appears here.)

Previous Page Next Page