Keep pulling the thread on Claude Code.
Peter recommends an AI skill architecture where a separate agent with a clean context window is spun up to run evaluations on the primary agent's output.
By using a separate evaluation agent, an AI skill can be designed to iterate in a loop, automatically re-editing its output until all pass/fail checks are met.
Peter recommends separating personal context and examples from the main skill.md file to improve performance and maintain privacy when sharing AI skills.
Peter advises providing at least two different examples when creating an AI skill to prevent the model from overfitting to a single example.
In AI skill platforms like Claude Code or Codex, the system decides whether to trigger a skill by reading only its name and description, not the entire skill file.
Peter asserts that AI evaluation methods based on numerical scoring are not effective because AI models cannot reliably differentiate between close scores, such as 4 out of 5 versus 5 out of 5.
Peter recommends using pass/fail checks as a more straightforward and reliable method for AI evaluations instead of numerical scoring.
Peter recommends using a 'memory.md' file to improve an AI skill over time by storing concise, reverse-chronological summaries of lessons learned from user interactions.
Peter offers pre-built AI skills, including 'Skill Editor', 'Personal Advisor', 'Health Coach', and 'Infographic Designer', to paid subscribers of his newsletter at behindthecraft.com.
Peter states that even with advanced techniques, AI skills can only produce output that is 80% to 90% complete, requiring a final 10-20% of human review and handcrafting to achieve high quality.