Run Inkling Locally: A Practical Guide with Ollama & LM Studio

Thinking Machines just dropped Inkling on Hugging Face, and the official announcement is exactly what you'd expect: architecture highlights, benchmark numbers, a lot of excitement. What it doesn't include is a single line about how to actually run it on your own machine today. That gap is what this guide fills.

You'll get the fastest path from zero to running Inkling locally, honest hardware guidance on what this model actually needs to run, copy-paste system prompts you can use immediately, and a workflow integration pattern that connects it to tools like Notion and content calendars.

If you've been curious about why local inference is worth the setup, this breakdown of what cloud AI sees about you makes the privacy case clearly.

Quick Start: Download and Run Inkling

The fastest path uses Ollama. It handles model management, dependencies, and local server setup in one tool. If you're comfortable with a terminal, this is your quickest route.

One important reality check before you start: Inkling is a large model. Consumer GPUs with 8 or 12GB of VRAM are unlikely to run it fully in GPU memory. The practical approach for most users is CPU inference or CPU/GPU hybrid offloading, which means you'll need very substantial system RAM — 280GB or more depending on the quantization level you choose. If you have access to a machine with that kind of RAM, this guide walks you through the setup. If you're on a typical gaming laptop or workstation, the Phi-4 vs Gemma vs Llama breakdown covers smaller models that run comfortably on consumer hardware.

Option A: Ollama (Recommended for Beginners)

Install Ollama from ollama.com. Then check the Ollama model library for the current Inkling model identifier — Ollama's naming conventions can differ from the Hugging Face repo name, so confirm the exact identifier before running the pull command. Once you have it:

ollama pull [inkling-model-identifier]

Once the download completes, start a chat session:

ollama run [inkling-model-identifier]

Ollama spins up a local inference server automatically. You can query it from the terminal or from any tool that supports an OpenAI-compatible API endpoint. For large models, Ollama will automatically offload layers to CPU/RAM when VRAM is insufficient, so inference will work — it'll just be slower than full GPU acceleration.

Try this first prompt: Once the session loads, paste this in exactly:

"You are a creative storyteller specializing in whimsical fantasy. Write a short story about a robot discovering the beauty of nature. Keep it under 200 words and use vivid sensory language."

The system prompt line isn't necessary here, but setting the persona explicitly tends to tighten the output significantly compared to a bare question.

Option B: LM Studio (Better for Intermediate Users Who Want Control)

Download LM Studio from lmstudio.ai. Search for "Inkling" in the built-in model browser, or paste the Hugging Face repo URL directly. LM Studio lets you select quantization level before downloading, which matters a lot when you're working with a large model on limited hardware. More on that in the next section.

LM Studio also runs a local OpenAI-compatible server on port 1234 by default, so the same Python scripts and integrations work regardless of which tool you use.

What Format to Download

Look for GGUF files. GGUF is the standard format for CPU and hybrid CPU/GPU inference with tools like Ollama, LM Studio, and llama.cpp. Avoid PyTorch or Safetensors formats unless you're setting up a training or fine-tuning environment — they're not optimized for local inference.

Choosing the Right Quantization for Your Hardware

Quantization compresses the model weights, trading a small amount of output precision for meaningful reductions in memory requirements. For a large model like Inkling, this is what makes local inference possible at all on non-datacenter hardware.

The key thing to understand: Inkling's quantized file sizes are very large by the standards of smaller open models. Check the Hugging Face model page for exact GGUF variant sizes before you download anything. That file-size reality changes the hardware math significantly, and it's worth confirming what you're in for before committing to a multi-hour download.

Practical guidance by hardware tier:

Consumer GPU (8–16GB VRAM): Full GPU inference isn't realistic for Inkling at this VRAM level. Your best option is CPU/GPU hybrid offloading — load as many layers as fit in VRAM and handle the rest in system RAM. You'll need a very large RAM pool (280GB+) and fast storage for the initial load. Expect slower inference speeds.
High-VRAM workstation GPU or multi-GPU setup: Even heavily quantized versions of Inkling require 280GB or more of combined memory. Most workstation-class setups will still rely on system RAM to bridge the gap. Check the available GGUF variants on the Hugging Face model page for exact sizes before downloading, and plan your memory accordingly.
CPU-only with large RAM: Viable if you have 280GB+ of system RAM and patience. Inference will be slow but functional. Among the available GGUF quantizations, more aggressive compression levels will lower the memory floor — check the HF model page to compare what's available for Inkling specifically.

One practical tip: if inference causes slowdowns or crashes, reduce the context length in LM Studio's settings first. Context window size is often the real culprit when memory fills up, not just the quantization level itself.

Across all tiers, start with the most aggressively quantized GGUF version that's available and work up from there if your hardware can handle it. The quality difference between adjacent quantization levels is usually small on generative tasks — the speed and memory difference is not.

Prompt Engineering Templates That Work Well with Inkling

Running the model is step one. Getting consistent, high-quality output is step two, and it depends almost entirely on how you structure your prompts. A bare instruction gets you a generic response. A structured system prompt gets you something you can actually use.

Template 1: Creative Writing with a Fixed Persona

System: You are a creative storyteller specializing in whimsical fantasy tales. Your stories should be engaging, imaginative, and feature vivid sensory descriptions. Keep your tone warm and accessible — this is for a general adult audience, not children.

User: Write a story about a talking cat who becomes mayor of a small town.

The system prompt does the heavy lifting. Without it, you'll get a story. With it, you'll get a story in a consistent style you can replicate across sessions or hand off to clients.

Template 2: Chain-of-Thought for Structured Output

System: You are a content strategist. Before writing any content, think through the goal, the audience, and the key message. Show your reasoning in a brief planning section, then write the output below it.

User: Write an intro paragraph for a blog post about the benefits of local AI models for freelancers.

Chain-of-thought works especially well on longer outputs where coherence matters. Asking the model to show its reasoning before writing tends to reduce the drift you sometimes get in longer creative or analytical pieces.

Template 3: Few-Shot Formatting Guide

System: You write concise product descriptions for an e-commerce brand. Match the tone and length of the examples below exactly.

Example 1:
Product: Canvas tote bag
Description: Sturdy, oversized canvas with reinforced handles. Takes the weekly farmer's market run without complaint. Available in natural and black.

Example 2:
Product: Ceramic pour-over coffee dripper
Description: Clean pour, clean flavor. This ceramic dripper fits most standard mugs and rinses in seconds.

User: Write a description for a vintage leather jacket.

Few-shot examples do something instruction alone can't: they show the model exactly what "right" looks like. For brand voice consistency across content calendars or client projects, this pattern is more reliable than trying to describe the style in words.

Fine-Tuning Inkling on Your Own Niche Data

System prompts can get you most of the way there. But if you're using Inkling for something highly specific — a client's brand voice, a narrow technical domain, legal document formatting — fine-tuning on your own data closes the remaining gap.

The Method: LoRA

LoRA (Low-Rank Adaptation) is the standard approach for fine-tuning large open models without retraining all weights. Instead of updating everything, LoRA adds a small set of trainable parameters on top. The result: you can meaningfully adapt a model's behavior without retraining from scratch — though fine-tuning at this model's scale still requires substantial hardware. Check the Hugging Face repo for any Inkling-specific fine-tuning guidance before committing to a training setup, and plan your compute accordingly.

Preparing Your Dataset

Format your training data as instruction/output pairs. Plain text files work fine. Structure each example like this:

Instruction: Write a product description for a vintage leather jacket.
Output: This classic vintage leather jacket is crafted from supple, full-grain leather aged to develop natural character. The hardware is solid brass. The lining is original silk. It's the kind of piece that improves with every wear.

A few hundred high-quality pairs outperform thousands of mediocre ones. If you're training on client copy, curate carefully — the model will learn your bad examples as readily as your good ones.

Tools to Use

trl (Transformer Reinforcement Learning library) — handles LoRA fine-tuning with a clean Python API
Unsloth — significantly faster LoRA training, worth trying if trl feels slow
Axolotl — good for users who want more configuration control over the training loop

You don't need to know all three. Pick one, follow its documentation for LoRA on a large model, and run a small test before committing to a full training run.

Integrating Inkling into Real Workflows

A locally running model with no connection to your actual work is an interesting experiment, not a useful tool. Here's how to wire Inkling into content calendars, Notion, and client projects.

Step 1: Get the API Endpoint

Both Ollama and LM Studio expose a local REST API that's compatible with the OpenAI API format.

Ollama default endpoint: http://localhost:11434/api/generate
LM Studio runs its local server on port 1234 by default — check the LM Studio documentation for the current endpoint path, as it follows the OpenAI-compatible format.

You can test that it's working with a curl command before building anything else.

Step 2: Basic Python Integration

import requests

url = "http://localhost:11434/api/generate"
prompt = "Write a blog post intro about the benefits of running AI locally."

payload = {
    "model": "[inkling-model-identifier]",
    "prompt": prompt,
    "stream": False
}

response = requests.post(url, json=payload)
print(response.json()["response"])

Replace [inkling-model-identifier] with the confirmed Ollama model name from the library. This is the core loop. From here, you can wrap it in a function that pulls topics from a spreadsheet, a Notion database, or a CSV from your content calendar tool, and writes drafts back to wherever you need them.

Step 3: Connect to Notion or a Content Calendar

The two most practical connection patterns:

Notion API + Python: Read page properties (topic, tone, deadline) from a Notion database, pass them into Inkling as a structured prompt, write the response back to a new page. The Notion API documentation covers this setup; the Python notion-client library makes it straightforward.
Zapier or Make: Use a webhook trigger from your content calendar, pass the topic to a Python script running locally (via a tunnel like ngrok for testing), and route the response wherever it needs to go. This adds some complexity but keeps the model fully local.

The point isn't full automation. It's getting Inkling to handle the first-draft scaffolding so your time goes to the editing and judgment calls that actually require you. AI gets you to a working draft in a few minutes — you spend the next hour on the parts only you can do.

For a deeper look at that principle applied to reports and business writing, this piece on AI drafting reports covers the workflow logic well.

How Inkling Compares to Other Open Models

The competitive SERP for Inkling is currently empty, which means you're probably also wondering: why Inkling specifically, versus Phi-4, Gemma 2, or Qwen2?

The honest answer is that model selection depends heavily on your hardware and your task. Inkling is a general-purpose model from Thinking Machines, and its size makes it a very different hardware conversation than the smaller models in the open-source space. For structured reasoning, math, or code on consumer hardware, other models are more practical to run — and they don't require 280GB of RAM to get started.

The practical advice: run the same 5 prompts across whichever models you're considering and evaluate the outputs yourself. Benchmarks tell you how a model performs on standardized tests. Your actual prompts tell you how it performs on your work. For a structured way to do that kind of comparison, the prompt testing methodology here applies to any model, not just GPT-5.6.

Frequently Asked Questions

How do I run Inkling locally on my own computer?

Download the GGUF version of Inkling from the Hugging Face Hub, then install either Ollama or LM Studio. With Ollama, check the Ollama model library for the confirmed model identifier, then run ollama pull [identifier] followed by ollama run [identifier] in your terminal. LM Studio lets you browse and download the model through a graphical interface. Both tools create a local inference server, so once the model is downloaded you're not dependent on a cloud connection to generate responses. Be prepared for very large file sizes and significant RAM requirements at this model's scale.

What hardware do I need to run Inkling locally?

Inkling is a large model. Consumer GPUs with 8 or 12GB of VRAM won't run it fully in GPU memory, and even heavily quantized versions require 280GB or more of system RAM to operate. CPU/GPU hybrid offloading is the realistic path for most setups, with the same 280GB+ RAM requirement applying. If you want good local model performance on typical consumer hardware, the Phi-4 vs Gemma vs Llama breakdown covers smaller alternatives worth considering.

What quantization level should I use?

Check the available GGUF variants on the Hugging Face model page for exact file sizes before downloading. As a general rule for large models, starting with the most aggressively quantized version available lowers your memory requirements while keeping output quality reasonable. If inference causes slowdowns or crashes, reduce the model's context length in LM Studio's settings before trying a lower quantization level — context size is often the real bottleneck.

Can I fine-tune Inkling on my own data?

LoRA (Low-Rank Adaptation) is the standard method for fine-tuning large open models without retraining all weights. You'll need a dataset formatted as instruction/output pairs, a fine-tuning library like trl or Unsloth, and hardware capable of handling the training run at this model's scale. Check the Hugging Face repo for any Inkling-specific fine-tuning notes before starting. A few hundred well-curated examples is a realistic starting point for most niche applications.

Do I need an internet connection to use Inkling after setup?

Once the model is downloaded, inference runs on your local hardware through Ollama or LM Studio — neither tool requires a cloud connection to generate responses. This is one of the main reasons to run a model locally rather than through a cloud API. That said, confirm the specific behavior in the documentation for whichever tool you're using, since some features (like model updates or telemetry) may have their own connectivity requirements.

Is Inkling compatible with OpenAI API clients and tools?

Both Ollama and LM Studio expose OpenAI-compatible REST endpoints, so any tool or script written for the OpenAI API can be pointed at either local server with minimal changes — usually just swapping the base URL and model name. The specifics depend on which tool you're running; check each tool's documentation for the exact endpoint format.

The Short Version

Inkling is a capable model from Thinking Machines, and getting it running locally is straightforward once you have the right hardware in place. The RAM requirements are serious — plan for 280GB or more. A well-structured system prompt is what separates useful output from generic text. The model is a tool. Your prompts are the skill.

If you'd rather start with tested templates than build from scratch, Ultra Prompt's content creation prompt library has structures you can adapt for Inkling workflows today.

Ultra Prompt