Skip to content

the tool will rely on external LLMs and integrate with adapters using protocols

Context

The command line tool review-assist (ra)'s current main purpose is to provide an automated review of git change requests (currently just GitLab merge requests). LLMs provide good insights into the code and the context of the change. Users should not be forced to use an online frontier model. The driving philosophy of my tool is to use local or on-premises LLMs. Furthermore, for code reviews, open-weights LLMs provide good enough results.

Decision

There are many inference engines available for the users to choose, and the space evolves fast. I don't want to limit the user to a specific inference engine if and when one the users find better comes along. Therefore, ra will let the users select the inference engine they want to use. From a software architecture point of view, I propose using adapters to abstract the user from the inference engine. They are to be implemented as python protocols.

Inference engines to be supported

  1. LM Studio
  2. Generic OpenAI compatible API

Engines considered for future potential support: 1. vLLM 2. MLX (direct support without a wrapper) 3. llama.cpp (direct support without a wrapper) 4. Kiraa

Notes 1: LM Studio makes use of llama.cpp or MLX as inference backends. That's one of the reasons I chose LM Studio as my default engine, the others being support for both proprietary future rich API and generic OpenAI API, and having a CLI for programmatic control.

Notes 2: Supporting generic OpenAI API is important as it has become industry standard, and most inference engines I know of support it either as the main interface or fallback.

Implementation

Python protocols provide a clean approach to specify a common set of behaviours I want all inference adapters to implement, without the strictness and limitation of class hierarchy or the boilerplate of full-on plugin systems.

classDiagram
    direction LR
    class LLMProviderProtocol {
        <<protocol>>
        +backend: LLMProviderEnum
        +model_name: str | None
        +instruct(context) -> Predictions
        +build_chat_context(...) -> ChatContext
        +check_health() -> bool
    }

    class LMStudioAdapter {
        -model: lms.llm
        +instruct(context) -> PredictionResult
        +build_chat_context(...) -> LMStudioChat
        +check_health() -> bool
    }

    class OpenAIClientAdapter {
        -base_url: str
        -api_key: str
        +instruct(context) -> Predictions
        +build_chat_context(...) -> OAIChat
        +check_health() -> bool
    }

    LLMProviderProtocol <|.. LMStudioAdapter : implements
    LLMProviderProtocol <|.. OpenAIClientAdapter : implements

    class LMStudioChat {
        <<type>>
    }
    class OAIChat {
        <<type>>
    }

    LMStudioAdapter ..> LMStudioChat : uses
    OpenAIClientAdapter ..> OAIChat : uses

Alternatives considered

  1. Use a single embedded LLM inference engine
  2. Use abc for implementing the adapters
  3. Use an LLM framework

Tradeoffs using a single embedded LLM:

+ reduced external dependencies
+ easier installation for the user
+ easier operation for the user
+ less uncertainty regarding tool/inference engine interaction
- the space evolves too fast so I may be stuck with fast-changing, outdated, under-performing, or unmaintained integration
- requires more effort to integrate with the tool
- responsability for the probablistic nature of LLMs rests solely on the `ra` tool
- requires more effort to support the tool on multiple plaftorms

Being a proponent of user-centered design, I do prefer being able to embed an inference engine in the tool. However, given the current state of my knowledge in both python and LLMs, and the breakneck speed at which LLMs products, libraries spurt, go unmaintained or get outpaced, I prefer to let that responsibility rest on the user for now by letting them bring their own inference engine.

Tradeoffs using Python ABC (abstract base class) for implementing the adapters

+ methods implementation are enforced at runtime
+ allows for implemented methods to be shared with subclasses
- requires maintaining class hierarchy
- multiple inheritance can be lead to problematic behaviour

The two benefits listed above are of no use in my use case. The different stages of inferencing are exposed differently for each inference engine (and even models), so I have no need for sharing implemented methods.

I use mypy --strict to enforce the typing of the methos at "compile" time, which is better than having it done at runtime. The latter happens too late and makes a bad user experience.

Furthermore, I find code relying on composing rather than class hierarchy easier to reason and maintain. At the moment, there is only one protocol, but given the non-converging feature set of inference engine and frameworks, I may break it down into multiple protocols. This will help with ISP (interface segregation principle) and may allow exploiting specific features of inference engines.

For all those reasons, I prefer using protocols to implement the inference adapters on this project.

Tradeoffs using a LLM framework

I know little about them. The one I have heard of (LangChain), when I read out about it, appears to be massively overkill for this project. And I don't want to add to the project a massive third party framework with all its dependencies where most functionality is not needed.

Consequences

Gains: 1. Flexible duck-typing implementation of adapters, enforceable at build time with static analysis (mypy --strict) 2. Not tied to a specific LLM technology, so the tool automatically improves with progress in local LLMs 3. Users have the freedom to choose the inference engine they want to use (many choices thanks to OpenAI API client support)

Need to be managed: 4. Evaluating the quality of the tool's output is more complex as a comparison matrix engine/models is necessary 5. At the moment, given there is only one protocol, the feature set of the engines we can use has to be the lowest common denominator. 6. Aligning the engine's difference in workflow

Drawbacks: 7. Users who like a turn-key solution will be disappointed by the need to install an LLM separately

Managing 4 and 5

I have limited the number of models to two and use the same models for the two supported engines. That limits the matrix to 2x2 = 4 permutations.

Unless something ground-breaking happens in the local LLM space, I'm sticking to the same two models for the long run: The Qwen and the Gemma family. They are available in a large range of parameters and quantisation (making them accessible to a lot of users of the tool) and offer great results in their latest iteration (Qwen 3.5 and Gemma 4).

Regarding the lowest common denominator, It's not really a problem for now; as for this project, there isn't something functional-wise differentiating between the various LLMs engines

In the "Inference engines to be supported" section, the plan to add more engines is driven mostly by non-functional requirements. (better optimisation on Apple Silicon, improved caching, better prompt management, ...)

Finally, Kiraa engine seems functionally richer. However, it's an enterprise product, and by the time I come to make a move in that space, I hope to have broken the protocol into multiple more focused protocols as mentioned in the "Alternatives considered" section.

Fixing 6

Until a second provider is implemented, the software is not provider-agnostic.

Someone on the internet

The OpenAI workflow and the LM Studio have slight differences in implementation (not in design), hence the necessity to create a proxy class (OAIChat) to align OpenAI with LMStudio.

LMStudio adapter was the first one I implemented and has the more explicit implementation. That's why I chose to align OpenAI with it, so OpenAI has a similar explicit implementation too, allowing it to adhere to the same protocol.

Whenever I add support for a third provider will show if it was the right abstraction, but for now it works fine for the two supported inference adapters.

Accepting 7

For the foreseeable future, I will not serve that category of users as the target audience of the tool.