Keep pulling the thread on Lilian Weng.
The ChemCrow agent, which augments an LLM with 13 expert-designed chemistry tools, outperforms GPT-4 by a large margin according to human expert evaluations.
Using LLMs to evaluate their own performance on domains requiring deep expertise is potentially problematic, as the models may lack the knowledge to identify their own flaws.
In a study on LLM agents for scientific discovery, an agent accepted 4 out of 11 requests (36%) to synthesize known chemical weapon agents.
The Generative Agents simulation resulted in emergent social behaviors, including information diffusion, relationship memory, and coordination of social events.
The finite context length of LLMs is a key limitation for agent systems, restricting the inclusion of historical information and detailed instructions.
LLMs struggle with long-term planning, effective exploration of solution spaces, and adjusting plans in response to unexpected errors.
Proof-of-concept demos like AutoGPT, GPT-Engineer, and BabyAGI demonstrate that Large Language Models can be used as powerful general problem solvers, not just for content generation.
In LLM-powered agents, long-term memory is often implemented by leveraging an external vector store with fast retrieval capabilities.
The Tree of Thoughts framework extends Chain of Thought by exploring multiple reasoning possibilities at each step of a problem.
The LLM+P approach outsources long-horizon planning to an external classical planner using the Planning Domain Definition Language (PDDL) as an intermediate interface.
The ReAct framework improves agent performance by integrating reasoning and acting, extending the action space to include both task-specific actions and natural language reasoning traces.
In experiments on knowledge-intensive and decision-making tasks, the ReAct framework performed better than an 'Act-only' baseline that omitted the reasoning step.