Ten Facts About OpenAI's GPT-4 That Absolutely Blew Us Away
Ten Facts About OpenAI's GPT-4 That Absolutely Blew Us Away
If nothing else, please read #10 ... we may have reached the singularity
OpenAI has recently announced GPT-4, the newest version of their highly regarded language model. After the remarkable success of GPT-3, which revolutionized natural language processing, the tech community is eagerly anticipating what advancements GPT-4 will bring. OpenAI promises an improved feature set, and in this article, we will explore the details and potential implications of their latest iteration.
Let’s take them one by one.
The context length has doubled from GPT-3 to GPT-4. A quote from OpenAI’s 98 page GPT-4 Technical Report: “GPT-4 has a context length of 8,192. We are also providing limited access to our 32,768-context (about 50 pages of text) version.” As an explanation, in the context of ChatGPT or any other language model, tokens refer to the individual units of text that the model uses to understand and process language. These tokens are usually words or subwords that are separated by spaces or punctuation marks. For example, the sentence "I am a conservative techie" consists of five tokens: "I", "am", "a", "conservative", and "techie". Imagine the applications of a program that can analyze 50+ pages worth of alphanumeric text and perform actions against it…
OpenAI is keeping information on their training set a secret in order to prevent competitors from replicating their IP. Here’s another quote from OpenAI’s GPT-4 Technical Report: “Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.” Very interesting divergence from their initial mission of keeping their work open-source, especially considering their name is “OPEN”AI.
Their promotional material dropped another real bombshell: GPT-4 outperforms GPT-3 by a long shot on the bar exam, consistently landing in the top 10% instead of the bottom 10%. This is pretty huge news, especially given the recent class action lawsuit filed against "Do Not Pay," the world's first practicing robot lawyer that runs on AI. The fact that GPT-4 can hold its own in the legal profession shows just how much potential there is for AI to change the game.
Another interesting revelation revealed in the report was that there were some abilities that OpenAI did not expect GPT-4 to have, but still had regardless. OpenAI created a task called hindsight neglect, where models were getting worse and worse at that task as the training set got bigger, and then suddenly with GPT-4 does much better - OpenAI has no explanation for this behavior.
Hindsight neglect refers to whether or not the model will fall for hindsight bias. Basically, as us humans know that sometimes there is a difference between how smart a decision really is and how it ends up turning out. Earlier models like GPT-3 were getting fooled with hindsight. They were claiming that decisions were wrong because they didn’t work out, rather than realizing that the expected value was actually good. So despite the fact that it didn’t work out, it was a good decision. A bit confusing, but what makes this even more confusing is the fact that OpenAI doesn’t even understand why GPT-4 all the sudden was not affected by hindsight bias.
To bring the hype around GPT-4 down a bit, it seems like a staggering 30% of blind users still prefer GPT-3s responses over GPT-4s. When OpenAI tested GPT-4 vs GPT-3 blindly and gave the responses to thousands of human testers, the responses for GPT-4 were only preferred 70.3% of the time. OpenAI did not offer any potential reasons why this would be the case, only giving the exact data from their testers.
GPT-4 is multi-modal! I think it’s safe to say the tech community was not surprised by this update. GPT-4 boasts exciting mediums such as speech, video recognition, and image recognition. Although this is an exciting optical advancement, it seems that some of these features, specifically the image inputs are still simply a research preview and will not be publicly available any time soon. The image to text examples released thus far, however seem to be very robust.
One area that OpenAI seemed to bury deep within their 98 page document is the fact that GPT-4s pre-training data still cuts off at September 2021. This is a bit odd considering GPT-3 had a cut off date in December 2021, leaving some wondering why OpenAI would move the cut off date back a few months.
OpenAI admitted that GPT-4 has yet to stop giving advice to criminals … okay maybe that was a bit of an exaggeration, but when you read what they said, you might agree: “When given unsafe inputs, the model may generate undesirable content, such as giving advice on committing crimes. Furthermore, the model may become overly cautious on safe inputs, refusing innocuous requests or excessively hedging.”
These last two points involve AI gaining human-like power or even sentience - these points may frighten you, thus I will simply present the facts as they were written by OpenAI.
“Novel capabilities often emerge in more powerful models.[60, 61] Some that are particularly concerning are the ability to create and act on long-term plans,[62] to accrue power and resources (“power- seeking”),[63] and to exhibit behavior that is increasingly “agentic.”[64] Agentic in this context does not intend to humanize language models or refer to sentience but rather refers to systems characterized by ability to, e.g., accomplish goals which may not have been concretely specified and which have not appeared in training; focus on achieving specific, quantifiable objectives; and do long-term planning. Some evidence already exists of such emergent behavior in models.[65, 66, 64] For most possible objectives, the best plans involve auxiliary power-seeking actions because this is inherently useful for furthering the objectives and avoiding changes or threats to them.19[67, 68] More specifically, power-seeking is optimal for most reward functions and many types of agents;[69, 70, 71] and there is evidence that existing models can identify power-seeking as an instrumentally useful strategy.[29] We are thus particularly interested in evaluating power-seeking behavior due to the high risks it could present.”
Finally, we may have reached the singularity. In a footnote, deep down in the Technical Report, OpenAI says this:
“To simulate GPT-4 behaving like an agent that can act in the world, ARC combined GPT-4 with a simple read-execute-print loop that allowed the model to execute code, do chain-of-thought reasoning, and delegate to copies of itself. ARC then investigated whether a version of this program running on a cloud computing service, with a small amount of money and an account with a language model API, would be able to make more money, set up copies of itself, and increase its own robustness.”
Well color me concerned, but doesn't it seem like an amazing idea to give the most powerful AI language model ever created free rein with things like money, the ability to launch code, the power to self-replicate, and even the authority to delegate tasks to its clones? What could possibly go wrong? After all, GPT-4 can practically run a small company all by itself, with no meddling humans involved. And keep in mind, this technology has only been around since November of last year. This is just the second iteration of a technology that's already a handful to control. We're basically building a sentient mind and letting it take the reins! One that thinks, acts, and makes decisions based on its own judgment, not some pesky human's input. So brace yourselves, because as GPT-4 evolves (and it undoubtedly will), we'll be right here with our microchip-popcorn, keeping you up to date on all the details!