Almost Timely News: ๐️ How to Set Up Private, Local AI (2026-01-25)
Almost Timely News: 🗞️ How to Set Up Private, Local AI (2026-01-25)Do you really want tech bros to know what you're asking AI?
Almost Timely News: 🗞️ How to Set Up Private, Local AI (2026-01-25) :: View in Browser
The Big PlugI’ve got a sneak peek next week coming for a new AI GEO/AIO/whatever-EO basic auditing tool that I’ve been working on. If you want advance notice, join the Analytics for Marketers Slack group (free). That’s where I’ll be announcing it and asking for beta testers. Content Authenticity Statement100% of this week’s newsletter content was originated by me, the human. Learn why this kind of disclosure is a good idea and might be required for anyone doing business in any capacity with the EU in the near future. Watch This Newsletter On YouTube 📺 Click here for the video 📺 version of this newsletter on YouTube » Click here for an MP3 audio 🎧 only version » What’s On My Mind: How to Set Up Private, Local AIThis week, let’s talk about setting up completely private, local AI. There are many reasons to do this, from big tech and similar surveillance to working with the most confidential data that you cannot, for any reason, expose to a third party. This week’s issue is aimed at the individual, like you or me. It’s not aimed at companies and enterprises; company IT departments are responsible for assisting you with privacy protection at the organization level. Pro Tip: If you have access to a tool like Claude Code, Claude Cowork, Google Antigravity, Qwen Code, OpenAI Codex, or other agentic frameworks, you can just copy paste this entire newsletter in and have the tool work through it with you step by step. Part 0: Ethics and DisclaimersBefore we begin, it would be remiss if we didn’t touch on the ethics of completely private AI. AI is a crazy powerful tool, and today’s tutorial allows you to escape the watchful eye of anyone. This can enable good and bad things. Good, if you’re using private AI to protect sensitive data, or to work towards helping people make their lives better. Bad if you’re using it to make others’ lives worse, to do things that are blatantly harmful, etc. Because AI is only a tool, the onus of responsibility for its moral use lies with you and me, the users of it. Please use it to help, not harm. When it comes to AI models that you can run privately, some of the best models in the world come from China, particularly the Alibaba Qwen family of models. These models are largely uncensored and highly capable. There’s an important distinction between using AI hosted by companies in China and using Chinese AI models that you run on your own computer. In terms of data privacy and safety, Chinese AI companies make it very clear in their terms of service that they’re using your data. Using AI hosted by those companies with private, sensitive data is unsafe. Using models made by Chinese AI companies on your own hardware, as long as they’re from reputable sources like Alibaba, is as safe as the rest of your infrastructure. Finally, a disclaimer: you are solely responsible for how you use AI in general, and how you use these instructions specifically. Neither I nor my company provide support, guarantees, or assurances that any of this works, is safe, or will keep you hidden from entities you want to be hidden from. I’ve provided my best reasonable efforts, but nothing in life is guaranteed. If you use these instructions in any capacity, you are solely responsible for the consequences. Part 1: Foundation TechnologiesIf you want complete privacy in the use of AI and still have it be very capable, you’ll need to use AI tools and models that have access to the Internet in some capacity. The reason for this is that ALL smaller language models, ones that you’d run on your own infrastructure, are hallucination machines. They hallucinate a crazy amount of the time, anywhere from 5% to 90% of the time, because they simply aren’t large enough to store and have factually correct information in them. The smaller a model is, the more it is likely to hallucinate. To counter this, having search capabilities is essential. However, this raises a privacy risk - what you search for on the Internet can, without proper precautions, be traced back to you and your AI. To counteract this, you have to invest in some additional infrastructure - notably, a VPN and some privacy-safe search capabilities. When you’re shopping for a VPN, there are two phrases you want to look for in any commercial service: zero data retention, and no logging. Zero data retention means that the VPN provider retains absolutely no data about how you use the service. No logging means that they do not keep server logs of any kind, which in turn means that an employee cannot produce logs upon request from anyone else; many providers have paid for third party audits to prove they do not log user activities, and thus have no records to produce when asked. When choosing a VPN provider, look for one outside your jurisdiction. Here’s why: almost all technology companies of any kind state in their terms of service that they will comply with lawful requests from their respective governments to hand over user data, if they have the data (hence the no logging and zero data retention features). But that’s based on the locus of business for the provider. It is much more challenging for a government to obtain information from a foreign provider - even with mutual legal assistance treaties, a government cannot generally subpoena a foreign provider directly, instead having to work with the appropriate authorities in that jurisdiction, a maze of red tape. Combining provider features like no log retention / zero data retention with a VPN provider outside your jurisdiction provides you the most security. As a sidebar, having a VPN is useful ESPECIALLY if you travel. Places like airport and public wifi are incredibly dangerous for your devices, so using a VPN in those is almost always a good idea. Besides a VPN, you’re going to need the ability to use the Tor network. This is a planet-wide routing network that further masks activities. Tor is free to access and free to use, powered by millions of users around the globe. To install Tor services for use with your AI, you’ll want to use either Chocolatey (Windows) or Homebrew (Mac) or apt/yum/dnf (Linux, built in). You’ll also want to install Python and Node. If this sounds intimidating, don’t worry - help is one prompt away. In the AI of your choice, use a prompt like this to have it walk you through setting it up, step-by-step, selecting the options that make the most sense.
Let the AI of your choice guide you through the process. If you have access to tools like Claude Cowork or similar (Claude Code, Qwen Code, Google Antigravity) you could even ask it to help you by installing those things for you. Another piece of technology you’ll want is a network activity blocker. On the Mac, the best choice is Lulu. On Windows, it’s Simplewall. What these two free, open source packages do is monitor network traffic from your computer and allow you to block applications that are trying to call home. This is especially important if you have privacy-invading apps like those from Meta or similar big tech companies that are constantly calling home. If you’re unsure how to configure them, again, help is a prompt away. Paste this into the AI of your choice:
After that, you’ll want to switch your DNS provider to a privacy-safe DNS provider. DNS providers match things like typing christopherspenn.com with the numeric IP address of that server, which means if you’re using a Big Tech DNS provider (like Google), they’re logging and seeing every site and service you go to. The general best free service is Quad9 based in Switzerland, which offers 9.9.9.9 as a free DNS. If you’re not sure how to do this... you guessed it. Help is a prompt away:
Finally, if you’re going to be using a web browser at all during your work with AI, install a privacy-safe browser like Firefox and use it religiously during your private AI time. This gives you a solid, multi-layer foundation for privacy:
And this isn’t just for private AI usage - this works for any other activities you want to do privately on your computer. A note of caution: all your privacy protections can be negated by things like malware and spyware installed on your computer. Be certain the computer you’re working on is clean before doing anything requiring privacy. Part 2: Choosing the Local ModelNow that you’ve got the fundamentals installed, it’s time to choose a model. AS I wrote back in 2025, choosing a local model is largely a question of how much video memory (VRAM) your computer has. If your computer has very little or none, then you’ll only be able to run very small models. If your computer can play high end video games at max settings, like Call of Duty, etc., then chances are it has a great GPU and can run big local models. Here’s the easiest way to determine what size model your computer can run... with a prompt and some screenshots.
Once you get an answer for how much memory you have available for AI, then you can follow up with this prompt:
Let the AI of your choice guide you. Make a note of which model it chooses, because you’ll need that specific name in the next step. Part 3: Setting up the Local Model ServerNo one runs an AI model by itself. They’re basically just giant statistical databases, which means they need a server around them to work. Servers are applications which at a minimum serve the AI model up, and may have a user interface for them as well. Some popular servers include:
My recommendation is that if you’re on a Mac, use LM Studio. If you’re on a Windows or Linux machine, use AnythingLLM. You’ll download the software from the appropriate provider and install it, then fire up your AI of choice with this prompt:
Once you’ve completed this step, you’re ready to start working with the private AI model of your choice. Part 4: Setting up the Local Model InterfaceNext, we need to set up your AI environment. My recommendation right now if you want private, local, powerful agentic AI is to use Qwen Code, which is a terminal-based agentic AI framework. If you JUST want to use AI like you use ChatGPT, with a basic chat, then I recommend using LM Studio or AnythingLLM apps. They provide that familiar chat interface we all know and love. However, tools like Qwen Code allow you to have access to things like agents and other autonomous processes. If you’re a more advanced user, Qwen Code is the way to go. If you’re a less technical user, LM Studio or AnythingLLM is the way to go. Back in Part 1, you installed either Chocolatey or Homebrew, depending on whether your system was Windows or Mac along with npm, the node package manager. To install Qwen Code, open a new terminal and type:
This will install the software on your computer. Once you’ve done that, create a new folder somewhere on your computer (like your desktop) and call it something memorable, like aiplayground. In your command line interface, change to this folder. If you’re unsure how to do that, fire up the AI of your choice with this prompt:
Now that you’re in the right folder, start Qwen by typing:
On the command line. This will start the Qwen Code interface. It’ll ask you to choose an authentication method with two choices: Qwen OAuth and OpenAI. Despite the name, the second option, OpenAI, refers to the OpenAI compatible authentication, and it’s this version that we’ll use, because it will connect Qwen Code to either LM Studio or AnythingLLM. Once you choose it, it’ll give you three lines to enter data: API key, Base URL, and Model. Because you’re running local, private AI in LM Studio or AnythingLLM, there’s no API key. The Base URL will depend on the software you’re using:
The model name is optional but doesn’t hurt to put it in. If you use Claude Code and you have agents and skills you really like, you can literally just copy and paste your agent and skill files into your Qwen Code projects. For example, I have a fact check skill I made for Claude Code, and it works right out of the box in Qwen Code. Part 5: Setting up the Secure Search MCPThis brings us to the last piece, which is installing a search MCP service that’s private and secure. This is what LM Studio, AnythingLLM, or Qwen Code will use to conduct safe, secure web searches. To make this work, I built my own which is freely available on my Github repo, searchmcp. It uses DuckDuckGo as the default search engine with Tor routing and zero logging. As with any open source code from the Internet, it’s a good idea to put it through the AI coding tool of your choice with an evaluation prompt to decide how secure it is. Here’s an example:
Run this in a tool like Qwen Code, Claude Code, Google Antigravity, or any of the agentic coding tools. Don’t use regular AI chat tools like Gemini or ChatGPT - they’re not good at actually reading code repositories. Once you feel confident in its results, go ahead and install it. By the way, this is a general best practice for any open source software that you use. It’s always good to see what it’s monitoring and logging, what it’s collecting - and because it’s open source, you could tell your coding tool to turn those features off. If you’re unsure how to install it, there are instructions on the page itself that you paste into the AI tool of your choice that will walk you through the process or, if you’re using an agentic tool like Qwen Code, can do it for you. Scroll down to the section labeled “LLM Installation Assistant” and copy paste the instructions into your AI tool. Part 6: Wrapping UpThese directions, if you step through the whole process or use an AI tool to help you step through it all, greatly increase your security and privacy. You now have the ability to chat with an AI model that’s on your computer, using a secure form of web search to get additional information. With this, you can ask questions and think through things that you might not be comfortable asking an AI tool owned by a big tech company, or surveilled by a government that might not have your best interests in mind. It’s important to say that this isn’t foolproof - nothing is. If you do something that other people very much care about and would want to find you for, these steps will make it more difficult, but nothing is guaranteed. This is especially true if you do anything else while you’re working with private AI. We don’t often think about privacy when we’re using our computers, where our data is going, what data is being recorded or not recorded. If you have other applications open when you’re doing stuff you’d rather not be observed doing, you run the risk of leaking identifying information. Close out every other application until you’re done. If you’re working with REALLY sensitive information, you may want to set up this entire process on a different computer entirely to be sure that the environment is safe, or if you’ve only got one computer, at least on a different profile than your normal day to day profile. Some of the suggestions, like Quad9 DNS, are things you can use day to day with no change from how you normally operate, and they add a little extra privacy to your life with no additional effort on your part once you’ve got it set up. The same is true for using a privacy-safe browser, or privacy-safe VPN. These are generally good habits to get into. And as I said at the beginning, use AI tools to do more good than harm. We live in a world of great uncertainty, lots of conflict, and what feels like less and less empathy every day. Use the knowledge I’ve shared to do what you can to make your little patch of the world better, to help more than harm. How Was This Issue?Rate this week’s newsletter issue with a single click/tap. Your feedback over time helps me figure out what content to create for you. Thanks for reading Almost Timely Newsletter! Subscribe for free to receive new posts and support my work. Here’s The UnsubscribeIt took me a while to find a convenient way to link it up, but here’s how to get to the unsubscribe. If you don’t see anything, here’s the text link to copy and paste: https://almosttimely.substack.com/action/disable_email Share With a Friend or ColleaguePlease share this newsletter with two other people. Send this URL to your friends/colleagues: https://www.christopherspenn.com/newsletter For enrolled subscribers on Substack, there are referral rewards if you refer 100, 200, or 300 other readers. Visit the Leaderboard here. ICYMI: In Case You Missed ItHere’s content from the last week in case things fell through the cracks:
On The TubesHere’s what debuted on my YouTube channel this week: Skill Up With ClassesThese are just a few of the classes I have available over at the Trust Insights website that you can take. PremiumFree
Advertisement: New AI Book!In Almost Timeless, generative AI expert Christopher Penn provides the definitive playbook. Drawing on 18 months of in-the-trenches work and insights from thousands of real-world questions, Penn distills the noise into 48 foundational principles—durable mental models that give you a more permanent, strategic understanding of this transformative technology. In this book, you will learn to:
Stop feeling overwhelmed. Start leading with confidence. By the time you finish Almost Timeless, you won’t just know what to do; you will understand why you are doing it. And in an age of constant change, that understanding is the only real competitive advantage. 👉 Order your copy of Almost Timeless: 48 Foundation Principles of Generative AI today! Get Back To Work!Folks who post jobs in the free Analytics for Marketers Slack community may have those jobs shared here, too. If you’re looking for work, check out these recent open positions, and check out the Slack group for the comprehensive list. Advertisement: New AI Strategy CourseAlmost every AI course is the same, conceptually. They show you how to prompt, how to set things up - the cooking equivalents of how to use a blender or how to cook a dish. These are foundation skills, and while they’re good and important, you know what’s missing from all of them? How to run a restaurant successfully. That’s the big miss. We’re so focused on the how that we completely lose sight of the why and the what. This is why our new course, the AI-Ready Strategist, is different. It’s not a collection of prompting techniques or a set of recipes; it’s about why we do things with AI. AI strategy has nothing to do with prompting or the shiny object of the day — it has everything to do with extracting value from AI and avoiding preventable disasters. This course is for everyone in a decision-making capacity because it answers the questions almost every AI hype artist ignores: Why are you even considering AI in the first place? What will you do with it? If your AI strategy is the equivalent of obsessing over blenders while your steakhouse goes out of business, this is the course to get you back on course. How to Stay in TouchLet’s make sure we’re connected in the places it suits you best. Here’s where you can find different content:
Listen to my theme song as a new single: Advertisement: Ukraine 🇺🇦 Humanitarian FundThe war to free Ukraine continues. If you’d like to support humanitarian efforts in Ukraine, the Ukrainian government has set up a special portal, United24, to help make contributing easy. The effort to free Ukraine from Russia’s illegal invasion needs your ongoing support. 👉 Donate today to the Ukraine Humanitarian Relief Fund » Events I’ll Be AtHere are the public events where I’m speaking and attending. Say hi if you’re at an event also:
There are also private events that aren’t open to the public. If you’re an event organizer, let me help your event shine. Visit my speaking page for more details. Can’t be at an event? Stop by my private Slack group instead, Analytics for Marketers. Required DisclosuresEvents with links have purchased sponsorships in this newsletter and as a result, I receive direct financial compensation for promoting them. Advertisements in this newsletter have paid to be promoted, and as a result, I receive direct financial compensation for promoting them. My company, Trust Insights, maintains business partnerships with companies including, but not limited to, IBM, Cisco Systems, Amazon, Talkwalker, MarketingProfs, MarketMuse, Agorapulse, Hubspot, Informa, Demandbase, The Marketing AI Institute, and others. While links shared from partners are not explicit endorsements, nor do they directly financially benefit Trust Insights, a commercial relationship exists for which Trust Insights may receive indirect financial benefit, and thus I may receive indirect financial benefit from them as well. Thank YouThanks for subscribing and reading this far. I appreciate it. As always, thank you for your support, your attention, and your kindness. Please share this newsletter with two other people. See you next week, Christopher S. Penn Invite your friends and earn rewardsIf you enjoy Almost Timely Newsletter, share it with your friends and earn rewards when they subscribe.
|
Comments