Dot Net Perls Blog

I seem to be having more success (and fun) using agentic AI (locally with small models) than I have had with trying to use larger models for editing code.

I developed a local MCP server in Rust that has some useful tool calls for often-needed commands.

Large and powerful LLMs are great, but sometimes a smaller, local model is sufficient.

Now that I have written my Rust MCP server, I need a local agent to use it to call tools effectively.

I rewrote my MCP server to use Stdio input and output instead of HTTP.

Yesterday I developed a simple MCP server, which allows local LLMs to call tools that I have written in Rust code.

LLMs cannot actually do anything except generate text—unless you give them access to tools.

Tests are important for ensuring code quality, and are even more important for programming in the age of LLMs.

I am continually interested in having a small, local LLM (running in llama-cpp) that can edit and add simple features to programs written in Rust and similar languages.

Code is basically a burden or liability—it has to be maintained.

The new model Kimi K3 was released this week, and it is claimed to be near-frontier level quality similar to Fable and GPT.

It has become trendy to rewrite code using AI agents.

Unfortunately using programs like OpenCode with local LLMs and a mid-range Nvidia GPU can be rather slow.

Recently I have been experimenting with llama-cpp and its Tools, which are provided to the LLM to make system calls.

Speculative decoding is a huge speedup for local inference of LLMs in llama-cpp (and similar programs).

Currently LLMs generate text in an autoregressive way—one token after another.

Currently it is not feasible (for most people) to run frontier, state-of-the-art models (LLMs) locally.

I spent an hour trying to improve a function in my Rust program.

When an LLM is quantized, it becomes possible to fit it onto a consumer GPU.