Keep pulling the thread on Edwin Chen.
A model's ranking on the LM Arena leaderboard can be artificially inflated by increasing the use of formatting, adding emojis, and making responses longer, even if this leads to hallucinations.
Researchers at frontier AI labs feel pressured to optimize models for leaderboard performance, such as on LM Arena, for career advancement, even when they believe it degrades the model's accuracy.
Surge AI achieved over $1 billion in revenue in 2023 with fewer than 100 employees.
Surge AI provides AI training data to every major frontier AI lab.
The LM Arena leaderboard is a flawed method for evaluating AI models because it relies on votes from casual users who skim responses and favor superficial qualities over accuracy.
Edwin Chen predicts that AI models will not converge into a commoditized technology but will instead become increasingly differentiated, with their behaviors and "personalities" reflecting the core values of the companies that train them.
Edwin Chen predicts that within one to two years, AI models will be capable of automating 80% of the tasks performed by an average L6 software engineer.
Surge AI is a completely bootstrapped company and has not raised any venture capital funding.
Reinforcement Learning (RL) within simulated environments is the next major stage in the evolution of AI model training, complementing existing methods like SFT and RLHF.
Edwin Chen believes that current Large Language Model (LLM) architectures alone will not be sufficient to achieve AGI and that a new technological breakthrough will be required.
Surge AI has been a profitable company since its first day of operation.
Surge AI's platform measures data quality by collecting thousands of signals on its human workers, projects, and individual tasks.