Almost Timely News: ๐️ Highly Opinionated Method of Building AI Agent Skills (2026-10-04)
Almost Timely News: ๐️ Highly Opinionated Method of Building AI Agent Skills (2026-10-04)I have strong opinions about Agent Skills
Almost Timely News: ๐️ Highly Opinionated Method of Building AI Agent Skills (2026-10-04) :: View in Browser
The Big Plug✍️ Enroll in my newest course, Introduction to Deep Research With AI. ๐ This issue features a new paid download, the The Highly Opinionated Agent Skill Fixer Download! Content Authenticity Statement100% of this week’s newsletter was made by me, the human. Learn why this kind of disclosure is a good idea and might be required for anyone doing business in any capacity with the EU in the near future. Watch This Newsletter On YouTube ๐บ
Click here for the video ๐บ version of this newsletter on YouTube » Click here for an MP3 audio ๐ง only version » What’s On My Mind: Highly Opinionated Method of Building AI Agent SkillsThis week, let’s talk about building Agent Skills for level 3 and 4 AI systems. If you’re not familiar or you need a refresher, Agent Skills (hereafter Skills with a capital S) are a way to convert a repeatable process into a command you or an AI can use to automate the process. There are no shortage of guides about how to build Skills, but almost all of them are predicated on the same basic concept: have AI do repeatable tasks. The reason why this issue is titled “Highly Opinionated Skill Building” is because I have strong opinions about the state of most Skills, namely that they’re not great. So today, we’ll look at my particular point of view. I’m not saying it’s right or wrong beyond a certain point, but it works for me and if you share some of my values, it might work for you too. Part 1: A Brief Review of AI LevelsI mentioned level 3 and 4 systems above. There are, from my point of view, 5 levels of AI systems, and each level builds on the previous. Here’s what they are, in short:
In these 5 levels, each level builds on the previous level; you can’t build level 4 systems without having working levels 1-3 infrastructure, in the same way that you can’t bake a pizza if you have never baked anything before. Part 2: Non-NegotiablesThe first and foremost thing that any Skill has to do is conform to the Agent Skills specification. Anthropic built this originally, a year ago, and made it an open framework. Pretty much every major AI platform has adopted it since then, so much so that companies are deprecating their Level 2 platforms in favor of skills. You can read the standard at AgentSkills.io, but the short version is that a Skill has at a minimum a skill definition file called SKILL.md, plus can have things like templates, assets, references, and scripts (code). You can immediately make any skill better off the bat by telling your favorite AI agent that at a bare minimum, the skill should conform to the spec. That’s as simple as a prompt like this:
That alone will solve a fair number of problems. For example, I’ve seen Skills that have thousands of words in them, entire narratives, when the spec says that you can have a maximum of 1,024 characters. What’s really important about Skills, though, are those “optional” components:
It’s worth pointing out here that all of the agentic AI systems that support skills support running those code languages in their environments in a sandbox. So it’s safe to run that code. And the fact that they can run code is really important. These optional components, which to me aren’t really optional, are what gives rise to my opinions. Part 3: Skill Development is Software DevelopmentIf you’re just copy pasting your prompts into the SKILL.md file - the bare minimum definition of a Skill - then you’re not really taking advantage of what Skills can do. Skills are more than just prompts - they may start out life as prompts, but they should evolve and grow. Skills are software. Skills are software because of that scripts folder, because AI agents use Skills like functions in software (and so do you, when you invoke them with a slash command), and thus they should follow the software development lifecycle:
When you take a step back and consider the SDLC and your Skills, this is exactly the way great Skills should work. It’s not some random prompt you vomited at ChatGPT or Claude after your fifth Irish Espresso. It’s real, working software. For example, I built a Skill recently for a client that does B2B sales prospecting. Given an ideal customer profile and the client’s context, the Skill does its own fan-out queries, gathers data from several different APIs, collates it, scores it, and gives it to the client’s CRM so their sales team can take advantage of the freshest opportunities without a business development rep having to manually search all those data sources. That’s way beyond what a prompt can do, and the reason it can do that is because of those “optional” components in a Skill that unlock a tremendous amount of capability. The best way to build a Skill is to build it as a piece of software, collecting requirements, designing it, doing tests, iterations, bug fixes, and evaluations to make sure it’s doing what you want it to do. In turn, that means you should be using Katie Robbert’s 5P Framework by Trust Insights™ to build your Skills:
I like the 5P Framework by Trust Insights™ as the foundation for Skill building because it’s granular enough to nail down specifics but broad enough that you can either directly use or infer all 7 stages of the SDLC from a properly written 5P prompt. Skills are software, nothing less. If we treat them like software, we’ll build great Skills. What are some tangible specifcs? Part 4: My Skill DirectivesThe Root SKILL.md Should be Orchestration OnlyThe root SKILL.md file should be an orchestration layer only. Inside this file should be a map of what else is available in the Skill and all the details elsewhere and only the broadest possible process goes at the root file. This is because every time an agentic tool loads a Skill, it loads this file first. And if this thing is a massive monolith, you’re just chewing up tokens repeatedly for no good reason. For example, if you have a long detailed prompt right now, that should be split up and decomposed into sub-prompts that go in the references folder of a skill. If you have deterministic things that you want to be doing like calculations or analysis, that belongs in the scripts folder. If you have things that AI shouldn’t be changing, like style sheets or logos that belongs in the assets folder. And if you have deterministic outputs for how the results should look from the machine, that belongs in the templates folder. In fact, I would argue that not only should the root file be an orchestration layer, you should have an orchestration layer in each of the folders as well so that the AI doesn’t have to load every single file into memory to know what it’s supposed to be doing. An orchestration layer would tell the AI which specific files to load, and more importantly, which files not to bother with for a specific given task. I’ll give you a silly example. Let’s say you’re baking a cake. Your SKILL.md recipe would contain the broadest steps of baking a cake - mise en place, appliances and equipment, ingredients, and the high level steps of the recipe - prep, mix, bake, cool - with directives to read the Mixing instructions in refrences or the Baking instructions in references or the doneness calculator in the scripts folder. In my own Skill builder, I have a set of tests. If the SKILL.md file is longer than 200 lines, it means I’ve done something wrong. The standard, the specification says that it should be no longer than 500 lines, but by requiring AI to build Skills with the root Skill being 200 lines or less, it really forces it to use orchestration intelligently. This also matters because every AI model treats Skills slightly differently. Depending on how smart the model is, it might need to have a lot of hand holding or very little. If you use orchestration well, a big, smart, expensive model will be able to cruise through the Skill easily, gather up all the pieces it needs in one shot, and have a go. If a small, light, less intelligent model reads the skill, your Skill is decomposed enough that it can follow along easily and not have to think very hard and just follow instructions one piece at a time. Over the last week, I’ve been playing with Hermes Agent using the Qwen 3.6 model on my laptop. Qwen 3.6 is a small and fast model, and it’s intelligent enough to do most basic tasks, but it struggles at advanced reasoning. Because I built my Skills with strong decomposition, Qwen doesn’t have to think, it just has to do. That orchestration also works well with agentic models like Qwen by keeping token loads low. Skills Should Use As Little AI As PossibleMy second strong opinion is that Skills should use as little AI as possible. Many of the tasks we delegate to AI are deterministic in nature, meaning we don’t want randomness and we want a reliable outcome more often than not. Large language models are probabilistic in nature, meaning they generate randomness. That randomness is good for things like writing. It’s not so good for things like math, where 2 + 2 in a base 10 system should always equal 4. When I build Skills, in the planning process I ask how much of the task can be done deterministically and what parts must be done by an LLM. More often than not, more than half of any given task is deterministic in nature. And by using the scripts folder in the Skill and having my frontier model write code to power the Skill, I use very little AI, I use fewer tokens, and I get more reliable results because classical deterministic code is processing the data and giving the outputs for the LLM to incorporate in its language-based tasks. For example, if we ask a Skill to go search the web, it will do so and it will chew up a tremendous number of tokens to go and get data. If on the other hand we give it a Python script to execute those same searches, the language model calls the Python script and the Python script does all the searching and doesn’t chew up tokens - and the results are in a consistent format. Another example would be asking an AI Agent Skill to go do research as part of a task instead of providing that research in the references folder pre-baked so that the tool can focus on getting the job done and not doing a bunch of clerical work. The less we use AI for tasks it’s not suited for, the better. Because the Skill standard gives us the ability to run languages like node, TypeScript, JavaScript, and Python within a Skill, we can do so much more than just plain prompting alone. Individual Skills Should Be Like UNIX Commands and Do One Thing WellMy third strong opinion is that Skills should be like Unix commands. If you are unfamiliar with the Unix ecosystem, Unix was written in the 1970s, when compute power could be measured in kilobytes of memory and hertz clock cycles instead of today’s gigahertz CPUs and terabytes of memory. As a result, Unix evolved to have tiny little command line programs that did one thing very well. It was up to the operator of the computer to chain those commands together to accomplish more complex tasks in a highly resource constrained environment. For example, in most flavors of Unix (including MacOS), there’s an application called wc, which is short for word count. That application does one thing and one thing only - it counts words. It doesn’t write or edit or spell check or any of the other functions of writing - it only counts words. We should be building Skills in the same way. Rather than trying to build massive, all-encompassing Skills that do 40 different things at once, we should be building Skills to be focused and small in scope. Instead of trying to make a massive sales Skill, for example, that does everything, we might have a sales Skill that only does budget qualification for leads. We might have another Skill that only does needs identification. We might have a Skill that reads the transcripts of a conversation and determines timeliness. We might have a fourth Skill that looks at the job title of the person to decide authority. Put those four Skills together and you have the classic old school IBM BANT framework. Why approach Skill building this way instead of treating it more like an ensemble software package? The more focused you make any software product, the easier it is to debug and maintain, the fewer places there are for things to go wrong or to collide by having a Skill that is decomposed down into individual tasks. We can verify the outputs of each step and make repairs in just the targeted area that is failing. When you have many Skills broken down into individual little pieces, you can then start chaining them together in prompts or scheduled tasks. And the AI will load only the pieces it needs at the time it needs it to fulfill your requests, making it much more efficient and much faster. Skills Need TestsNo one in their right mind would release a piece of software without having tested it with things like unit tests, integration tests, end-to-end testing, user acceptance testing, etc. And yet we create Skills and just toss them out the door without ever having any kind of evaluation. Remember, the fifth P in the 5P Framework by Trust Insights™ is performance. Skills need tests and evaluations. At a bare minimum, a Skill should have three sets of evaluations:
In my Skills, there’s usually an evals in the scripts folder and sometimes in the references folder for the measurable way the Skill is supposed to perform and how it passed testing. Part 5: Wrapping Up and Grab My Skill FixerAs I said at the start, all of this is my opinions based on my experience with AI and my experiences with software development. It is not the “one true way” or any such bullshit. There are as many ways to develop Skills as there are ways to write software. Some of what I believe directly conflicts with what some of the AI industry espouses, in part because they want you using more AI, not less. What I love about Skills is that because it’s an industry standard, they are portable across so many platforms. This is especially important with Level 4 systems like Meta Muse and Grok Bot and OpenAI Dots, all of which can ingest Skills in some fashion. Anyone operating a Level 4 autonomous AI system while having only Level 1 prompting skills - or none at all - is going to create chaos by giving autonomous agents unbounded or poorly bounded tasks. We’re already seeing this happen in the industry. I was chatting with my friend and colleague Brooke Sellas the other day about how companies are getting slammed by Level 4 systems - especially Meta Muse - because consumers are giving them poorly bounded tasks with no Skills and the AI agents are eating customer support resources alive. If we can build great Skills and get them distributed as widely as these autonomous systems are being distributed, haphazardly and irresponsibly, we can at least help people realize that all AI, but especially agentic AI, needs guardrails and Skills are a great way to provide them. How Was This Issue?Rate this week’s newsletter issue with a single click/tap. Your feedback over time helps me figure out what content to create for you. Got More Feedback?๐ Please take my 6-question Reader Survey to tell me what you want! Here’s The UnsubscribeIt took me a while to find a convenient way to link it up, but here’s how to get to the unsubscribe.
If you don’t see anything, here’s the text link to copy and paste: https://almosttimely.substack.com/action/disable_email Share With a Friend or ColleaguePlease share this newsletter with two other people. Send this URL to your friends/colleagues: https://www.christopherspenn.com/newsletter For enrolled subscribers on Substack, there are referral rewards if you refer 100, 200, or 300 other readers. Visit the Leaderboard here. ICYMI: In Case You Missed ItHere’s content from the last week in case things fell through the cracks:
My Merch ShopI’ve been adding so much stuff that I’ve decided to bundle it all in what I call a Merch Shop, because otherwise there’s literally too much to keep track of and I run out of space in my own newsletter. So welcome to the Merch Shop! Courses: Books: Skills for Claude and Agentic AI:
Subscriptions: On The TubesHere’s what debuted on my YouTube channel this week:
Advertisement: New AI For Writers Course“It’s not X, it’s Y!” “XYZ is the load-bearing feature here.” “XYZ is the entire argument. Everything else is incidental.” Does AI writing grind your gears? When you use it at work and it spits out dreck despite your best efforts, does that frustrate you? It’s not your fault - and believe it or not, it’s not AI’s fault either. It’s that great writing is inherently low probability, and AI is a probability engine that trades in high probabilities. In the new Trust Insights AI for Writers Course, I teach you how to use tools, analysis, data, and code to force AI to write much more like you, giving it measurable requirements that it can obey to give you what you want - writing that doesn’t sound like the Terminator went on a bender with a thesaurus. The course is 22 lessons and includes tools, skills, plugins, and how to use them all to generate the kind of writing you’d want to read, and retails for USD 397. ✍️ Claim your seat here and start making AI write the way you expect. Get Back To Work!Folks who post jobs in the free Analytics for Marketers Slack community may have those jobs shared here, too. If you’re looking for work, check out these recent open positions, and check out the Slack group for the comprehensive list.
Disclosure: I source these links from LinkedIn every week on the following criteria: New in the past seven days, Easy Apply on, remote roles, USA geography. How to Stay in TouchLet’s make sure we’re connected in the places it suits you best. Here’s where you can find different content:
Listen to my theme song as a new single: Social Good: Ukraine ๐บ๐ฆ Humanitarian FundThe war to free Ukraine continues. If you’d like to support humanitarian efforts in Ukraine, the Ukrainian government has set up a special portal, United24, to help make contributing easy. The effort to free Ukraine from Russia’s illegal invasion needs your ongoing support. ๐ Donate today to the Ukraine Humanitarian Relief Fund » Events I’ll Be AtHere are the public events where I’m speaking and attending. Say hi if you’re at an event also:
There are also private events that aren’t open to the public. If you’re an event organizer, let me help your event shine. Visit my speaking page for more details. Can’t be at an event? Stop by my private Slack group instead, Analytics for Marketers. Required DisclosuresEvents with links have purchased sponsorships in this newsletter and as a result, I receive direct financial compensation for promoting them. Advertisements in this newsletter have paid to be promoted, and as a result, I receive direct financial compensation for promoting them. My company, Trust Insights, maintains business partnerships with companies including, but not limited to, Amazon, Talkwalker, MarketingProfs, Agorapulse, The Marketing AI Institute, Spin Sucks, and others. While links shared from partners are not explicit endorsements, nor do they directly financially benefit Trust Insights, a commercial relationship exists for which Trust Insights may receive indirect financial benefit, and thus I may receive indirect financial benefit from them as well. Thank YouThanks for subscribing and reading this far. I appreciate it. As always, thank you for your support, your attention, and your kindness. Please share this newsletter with two other people. See you next week, Christopher S. Penn Invite your friends and earn rewards
If you enjoy Almost Timely Newsletter, share it with your friends and earn rewards when they subscribe.
|
Comments