Elon Musk concurs with different AI consultants that there’s little real-world knowledge left to coach AI fashions on.
“We’ve now exhausted principally the cumulative sum of human information …. in AI coaching,” Musk stated throughout a live-streamed dialog with Stagwell chairman Mark Penn streamed on X late Wednesday. “That occurred principally final 12 months.”
Musk, who owns AI firm xAI, echoed themes former OpenAI chief scientist Ilya Sutskever touched on at NeurIPS, the machine studying convention, throughout an tackle in December. Sutskever, who stated the AI business had reached what he known as “peak knowledge,” predicted an absence of coaching knowledge will pressure a shift away from the way in which fashions are developed at present.
Certainly, Musk prompt that artificial knowledge — knowledge generated by AI fashions themselves — is the trail ahead. “The one option to complement [real-world data] is with artificial knowledge, the place the AI creates [training data],” he stated. “With artificial knowledge … [AI] will type of grade itself and undergo this means of self-learning.”
Different corporations, together with tech giants like Microsoft, Meta, OpenAI, and Anthropic, are already utilizing artificial knowledge to coach flagship AI fashions. Gartner estimates 60% of the information used for AI and analytics tasks in 2024 have been synthetically generated.
Microsoft’s Phi-4, which was open-sourced early Wednesday, was educated on artificial knowledge alongside real-world knowledge. So have been Google’s Gemma fashions. Anthropic used some artificial knowledge to develop certainly one of its most performant methods, Claude 3.5 Sonnet. And Meta fine-tuned its most up-to-date Llama sequence of fashions utilizing AI-generated knowledge.
Coaching on artificial knowledge has different benefits, like value financial savings. AI startup Author claims its Palmyra X 004 mannequin, which was developed utilizing nearly completely artificial sources, value simply $700,000 to develop — in contrast to estimates of $4.6 million for a comparably-sized OpenAI mannequin.
However there as disadvantages as effectively. Some analysis means that artificial knowledge can result in mannequin collapse, the place a mannequin turns into much less “artistic” — and extra biased — in its outputs, ultimately severely compromising its performance. As a result of fashions create artificial knowledge, if the information used to coach these fashions has biases and limitations, their outputs will probably be equally tainted.
