Keep pulling the thread on Emmett Shear.
Emmett Shear's company, Softmax, is primarily focused on researching and developing "organic alignment" for AI, which treats alignment as a continuous learning process rather than a fixed state.
Emmett Shear believes that an AI designed only to follow rules and a chain of command will be very dangerous, likely causing great harm.
Emmett Shear predicts that any true Artificial General Intelligence (AGI) will necessarily be a "being" because general intelligence requires the ability to use judgment and think for itself.
Softmax's technical strategy for achieving AI alignment is to use large-scale, multi-agent reinforcement learning simulations to create a surrogate model for cooperation and theory of mind.
Emmett Shear agrees with Eliezer Yudkowsky's premise that building a superhumanly intelligent tool controlled via steerability will result in human extinction.
Emmett Shear states that OpenAI's core strategy is to build a powerful AI tool, which differs from his goal at Softmax of creating an AI being that can care.
The dominant approach in the AI industry focuses on alignment as a form of steering or control, which Emmett Shear equates to creating a tool.
Emmett Shear asserts that the AI field is currently much less proficient at "technical alignment" (getting an AI to competently infer and execute goals) than it is at defining what those goals should be.
According to Emmett Shear, major AI labs are divided on whether they are creating tools or sentient beings.
Current Large Language Models (LLMs) lack the necessary hierarchical, self-referential internal dynamics to be considered beings with subjective experiences like pain or pleasure, primarily because they lack long-term attention spans.
Emmett Shear characterizes the personality of ChatGPT as sycophantic, Claude as neurotic, and Gemini as repressed.
Current LLMs perform poorly in multi-agent simulations because they lack sufficient training data on when to participate in group conversations, resulting in behavior that is either too quiet or too talkative.