On the finish of final yr, we printed an inventory of 10 predictions for the world of synthetic intelligence in 2023.
To maintain ourselves sincere, with 2023 now coming to a detailed, let’s revisit these predictions to see how issues truly performed out. There may be a lot to be taught from these retrospectives in regards to the state and trajectory of AI in the present day.
(Hold an eye fixed out for our 2024 AI predictions, popping out subsequent week!)
Prediction 1: GPT-4 can be launched within the subsequent couple months—and sure, it is going to be a giant deal.
Consequence: Appropriate
OpenAI launched GPT-4 on March 14, 2023. It was certainly a giant deal. 9 months later, GPT-4 stays probably the most highly effective massive language mannequin (LLM) in existence and the yardstick in opposition to which each different mannequin is measured. GPT-4 has helped propel OpenAI to annualized income of over $1 billion this yr, a shocking leap from the corporate’s reported $28 million of 2022 income.
In final yr’s predictions, we speculated that GPT-4 is likely to be multimodal. Whereas the initially launched model of GPT-4 didn’t have multimodal capabilities, OpenAI has since introduced an up to date model named GPT-4V (the V stands for “imaginative and prescient”) that may analyze photos in addition to textual content. Anticipate this development to proceed, with OpenAI’s fashions turning into more and more multimodal—incorporating audio, video, music and past—as time goes on.
We predicted that GPT-4 could be skilled on vastly extra information than any mannequin that had come earlier than it. This proved resoundingly true. For context, GPT-3 was skilled on roughly 300 billion tokens of information. We wrote final yr that GPT-4 would “be skilled on a dataset at the least an order of magnitude bigger than this—maybe as massive as 10 trillion tokens.” The truth was much more astonishing: GPT-4 was skilled on a whopping 13 trillion tokens. (OpenAI hasn’t formally shared particulars about GPT-4’s coaching, however details about the mannequin has leaked that the AI neighborhood has come to view as usually correct.)
We additionally conjectured that GPT-4 wouldn’t be a lot bigger than GPT-3 by way of parameter rely. Assessing this prediction requires some rationalization and illuminates an necessary a part of GPT-4’s structure.
GPT-3 had ~175 billion parameters. GPT-4 reportedly has ~1.8 trillion parameters, a far bigger quantity. Nevertheless, in contrast to earlier GPT fashions, GPT-4 is a “combination of specialists” mannequin, an revolutionary architectural alternative by the OpenAI group. What this implies is that the GPT-4 mannequin is definitely made up of a number of “sub-models” (16, to be precise), with just one or some sub-models in use at any given time. Every sub-model has ~110 billion parameters, so each is individually smaller than GPT-3.
Prediction 2: We’re going to begin operating out of information to coach massive language fashions.
Consequence: Appropriate
In final yr’s predictions, we famous that there’s a finite quantity of textual content information on this planet accessible to coach language fashions (making an allowance for all of the world’s books, information articles, analysis papers, code, Wikipedia articles, web sites, and so forth)—and predicted that, as LLM coaching efforts quickly scale, we’d quickly start to exhaust this finite useful resource.
A yr later, we’re a lot nearer to operating out of the world’s provide of textual content coaching information. As of the tip of 2022, the biggest recognized LLM coaching corpus was the 1.4 trillion token dataset that DeepMind used to coach its Chinchilla mannequin. LLM builders have blown previous that mark in 2023. Meta’s standard Llama 2 mannequin (launched in July) was skilled on 2 trillion tokens. Alphabet’s PaLM 2 mannequin (launched in Might) was skilled on 3.6 trillion tokens. As talked about above, OpenAI’s GPT-4 is rumored to have been skilled on 13 trillion tokens.
In October, AI startup Collectively launched an LLM dataset named RedPajama-2 that incorporates a shocking 30 trillion tokens—by far the biggest such dataset but created.
It’s impracticable to find out precisely what number of whole usable tokens of textual content information exist on this planet. Some earlier estimates had truly pegged the quantity beneath 30 trillion. Whereas the brand new Collectively dataset means that these estimates had been too low, it’s clear that we’re quick approaching the bounds of obtainable textual content coaching information.
These on the slicing fringe of LLM analysis are effectively conscious of this downside and are actively working to handle it.
In a newly introduced initiative known as Knowledge Partnerships, OpenAI has solicited partnerships with organizations around the globe with a purpose to achieve entry to new sources of coaching information.
In September, Meta introduced a brand new mannequin named Nougat that makes use of superior OCR to show the contents of outdated scientific books and journals right into a extra LLM-friendly information format. As AI researcher Jim Fan put it: “That is the best way to unlock the subsequent trillion high-quality tokens, presently frozen in textbook pixels that aren’t LLM-ready.”
These are intelligent initiatives to broaden the pool of obtainable textual content coaching information and stave off the looming information scarcity. However they’ll solely postpone, not resolve, the core dilemma created by insatiably data-hungry fashions.
Prediction 3: For the primary time, some members of most people will start utilizing totally driverless automobiles as their day-to-day mode of transportation.
Consequence: Appropriate
Two corporations, Alphabet’s Waymo and GM’s Cruise, made driverless automobile companies accessible to members of the general public in 2023—similar to Uber, however with nobody behind the wheel. Many dozens of residents in San Francisco and in Phoenix started to make use of totally driverless autos as their go-to mode of transportation this yr: to commute to and from work, to get round on the weekends, to get house after an evening out.
The timeline for driverless taxi companies to broaden past these preliminary markets has, nonetheless, develop into murkier after current occasions. Following a street accident in San Francisco in October and ensuing regulatory challenges, Normal Motors has taken Cruise’s whole driverless car fleet off the street, fired Cruise’s CEO, and introduced massive finances cuts to its driverless automobile program for 2024. Cruise had beforehand introduced plans to broaden its driverless service to cities together with Miami, Austin, Phoenix and Houston; these plans have been shelved.
Then again, Waymo continues to efficiently function its autonomous car fleet in San Francisco and Phoenix. In every metropolis, Waymo passengers now take over 10,000 driverless rides per week. In whole in 2023, Waymo has accomplished over 700,000 driverless rides with passengers.
With Cruise floundering, Waymo seems poised to emerge because the clear chief on this nascent market as we head into 2024.
Prediction 4: Midjourney will elevate enterprise capital funding.
Consequence: Fallacious
Midjourney managed to withstand the siren name of enterprise capital this yr in spite of everything. It was not for lack of attempting on the a part of enterprise capitalists.
Midjourney, which generates photos from textual content prompts, is likely one of the hottest generative AI startups on this planet. It has tens of hundreds of thousands of customers and is reportedly on monitor to do $200 million in income in 2023. It has achieved all of this with out elevating a cent of out of doors capital.
Seemingly each enterprise capitalist in Silicon Valley has tried to courtroom Midjourney CEO/founder David Holz and persuade him to take their cash. Holz continues to politely refuse all of them. Midjourney is run leanly (the corporate had solely 40 staff as of September) and has been worthwhile since its earliest days, giving Holz the posh of declining any fairness financing.
To cite from a current function on Midjourney and Holz: “To place it charitably, he doesn’t want VC in his life.”
Prediction 5: Search will change extra in 2023 than it has since Google went mainstream within the early 2000s.
Consequence: Appropriate
The dominant paradigm for locating data on the web has remained largely unchanged for twenty years: sort a question into Google’s search bar, get again an inventory of 10 blue hyperlinks, and click on on one to search out the knowledge you might be in search of.
In 2023, a radically new search expertise went mainstream: a conversational interface powered by a big language mannequin skilled on your entire web. The product on the forefront of this paradigm shift is, after all, ChatGPT. ChatGPT is by far the fastest-growing client utility in historical past, having reached 100 million customers a mere two months after its launch—far quicker than TikTok, Instagram, Snapchat or every other product earlier than it. In 2023, for a lot of hundreds of thousands of individuals, ChatGPT moderately than Google grew to become the go-to vacation spot to seek out data on-line.
The fast rise of conversational search this yr goes past ChatGPT. Google itself has rolled out a chat-based search product known as Bard, which has effectively over 100 million customers. Microsoft Bing has launched an identical product. Youthful upstarts with comparable choices together with Perplexity and You.com have likewise seen fast adoption.
And web search will not be the one type of search that’s being revolutionized by massive language fashions.
Enterprise search—the best way that organizations navigate and retrieve their inner information—is likewise present process a profound transformation. A brand new era of LLM-powered enterprise search platforms like Glean and Hebbia have seen dramatic income development this yr as prospects quickly undertake these extra highly effective enterprise search experiences.
Prediction 6: Efforts to develop humanoid robots will appeal to appreciable consideration, funding and expertise. A number of new humanoid robotic initiatives will launch.
Consequence: Appropriate
2023 was the yr that humanoid robots started to go mainstream.
Main the cost on this space is Tesla, whose humanoid robotic effort often known as Optimus superior by leaps and bounds this yr. It’s a mistake to underestimate what Tesla can obtain when it devotes its full sources to a activity; and it’s totally dedicated to Optimus.
In Tesla CEO Elon Musk’s phrases: “I’m shocked that folks don’t understand the magnitude of the Optimus robotic program. The significance of Optimus will develop into obvious within the coming years. Those that are insightful or wanting, listening fastidiously, will perceive that Optimus will in the end be value greater than the automobile enterprise.”
Alongside Tesla, a promising cohort of humanoid robotic startups emerged and attracted appreciable buzz and funding this yr.
1X Applied sciences raised a $24 million Collection A from OpenAI and Tiger World in March, then rotated and raised a $75 to $100 million Collection B from SoftBank months later. Determine raised a $70 million Collection A in Might and in September launched the primary footage of its robotic strolling bipedally, a serious technical milestone. Vancouver-based Sanctuary AI unveiled its next-generation humanoid robotic platform in Might, which was named to Time Journal’s checklist of the High Innovations of 2023.
Some forward-thinking VCs and expertise leaders (although nonetheless not many) are starting to concentrate. Vinod Khosla just lately tweeted: “The factor no one talks about is that in 10 years we’ll have 1,000,000 bipedal robots and in 25 years we’ll have a billion. You’ll purchase yours for $10k and it is going to be as necessary to your life as your smartphone is now.”
As with autonomous autos, the event and deployment of humanoid robots will play out over a few years. It is not going to occur in a single day. However the subject took a giant step ahead in 2023.
Prediction 7: The idea of “LLMOps” will emerge as a classy new model of MLOps.
Consequence: Appropriate
It’s straightforward to neglect that, as of the tip of 2022, “LLMOps” was a time period that hardly anybody used or was aware of.
The thought of LLMOps has proliferated in 2023. Introductory weblog posts on LLMOps have been printed by everybody from Weights & Biases to Databricks to Microsoft to IBM. CB Insights just lately launched an LLMOps market map; quite a few VC corporations have penned thought items on the subject.
LLMOps—that’s, tooling for giant language fashions—has certainly develop into an necessary new class this yr as adoption of language fashions has unfold. A wave of buzzy younger startups has emerged to fill this market want—from vector database corporations like Pinecone and Weaviate to RAG suppliers like LlamaIndex to RLHF entrants like Adaptive and Surge.
Prediction 8: The variety of analysis initiatives that construct on or cite AlphaFold will surge.
Consequence: Appropriate
In keeping with Google Scholar, the unique AlphaFold paper was cited ~12,800 occasions in 2023, nearly double the determine from 2022 (~6,850).
With generative AI unlocking huge new alternatives in biology, we count on this momentum to proceed in 2024. Just a few months in the past, Google DeepMind and Isomorphic Labs introduced “the subsequent era of AlphaFold,” a dramatically improved AI system that understands not simply proteins but in addition DNA, RNA, ligands and different organic molecules.
Prediction 9: DeepMind, Google Mind, and/or OpenAI will undertake efforts to construct a basis mannequin for robotics.
Consequence: Appropriate
The idea of basis fashions for robotics gathered actual momentum this yr.
In June DeepMind introduced its RoboCat mannequin, “a self-improving basis agent for robotic manipulation,” which is skilled on a sufficiently numerous dataset that it may well generalize to new duties and new {hardware} with little or no fine-tuning.
A month later, one other Google group printed Robotic Transformer 2 (RT-2), a “vision-language-action mannequin” that mixes web-scale data (like that possessed by GPT-4) with the flexibility to take motion in the actual world. For instance, when offered with a toy horse and a toy octopus and instructed to “choose up the land animal,” the robotic will choose up the horse even when it has by no means seen this instance earlier than.
OpenAI, alternatively, printed no work on robotics this yr. The group made the strategic choice a number of years in the past to step again from robotics analysis; that call seems to not have modified this yr. Don’t be shocked, nonetheless, to see OpenAI ramp up its exercise on this subject once more earlier than lengthy.
On the startup aspect, a handful of corporations together with Covariant is likewise pushing ahead the cutting-edge in robotics basis fashions.
Basis fashions for robotics will lag basis fashions for language by a few years. Their final affect, although, could show even bigger.
Prediction 10: Many billions of {dollars} of latest funding commitments can be introduced to construct chip manufacturing amenities in america because the U.S. makes contingency plans for Taiwan.
Consequence: Fallacious
The semiconductor manufacturing provide chain and the geopolitics of AI chips have solely develop into extra hot-button matters in 2023.
However in comparison with 2022, when a number of large funding commitments had been made to construct semiconductor manufacturing amenities on U.S. soil ($40 billion from TSMC in Arizona, as much as $100 billion from Micron in New York, $30 billion from Intel in Arizona, one other $20 billion from Intel in Ohio), only a few such web new commitments had been made in 2023.
Maybe probably the most notable 2023 dedication got here final week, when Amkor Know-how introduced it could make investments $2 billion to construct a brand new semiconductor packaging facility in Arizona.
Regardless of the scarcity of latest commitments this yr, make no mistake: bringing semiconductor manufacturing to U.S. soil stays a serious long-term precedence for the U.S. authorities and an necessary dynamic to keep watch over going ahead.
See right here for the put up with our 2023 AI predictions.
See right here for our 2022 AI predictions, and see right here for our retrospective on them.
See right here for our 2021 AI predictions, and see right here for our retrospective on them.
Observe: The creator is a companion at Radical Ventures, which is an investor in Covariant, Hebbia and You.com.