Keep pulling the thread on Edwin Chen.
Surge surpassed $1 billion in annual recurring revenue (ARR) in 2023.
Despite having over $1 billion in ARR, Surge's team consists of just over 100 people.
Edwin Chen asserts that the LMSYS Chatbot Arena leaderboard is "absolutely terrible" and has set the AI industry back by at least a year.
At Twitter, optimizing the timeline algorithm for clicks and retweets created a negative feedback loop that promoted clickbait and racy content.
Surge's founding strategy was to focus on high-complexity, high-quality data tasks, differentiating from competitors who focused on low-skill, commodity labeling.
Surge intentionally avoided venture capital to attract customers who genuinely believed in high-quality data, rather than those influenced by media coverage in outlets like TechCrunch.
According to Surge's CEO, many competing data labeling companies operate as "body shops" with minimal technology, using spreadsheets and having their own engineers manually create or review data.
Edwin Chen argues that optimizing for inter-rater agreement in data labeling for generative AI is a flawed, "old school" approach that produces low-quality, "lowest common denominator" outputs.
Surge will turn away potential customers if their goals are not aligned with advancing AGI, prioritizing its mission over maximizing revenue.
Surge provides data labeling and generation services in over 50 languages, including hyper-specialized domains like coding in Argentinian Spanish.
Edwin Chen believes that many frontier LLMs have been "benchmark hacked," meaning they are narrowly optimized for academic or synthetic benchmarks which do not reflect real-world, open-ended problems.
According to Edwin Chen, the easiest way to improve a model's score on the LMSYS Chatbot Arena is to make its responses longer and use more emojis.