Creating a Custom VS Code Extension as a Personal AI Agent( Assistant)

As a developer, one tool that I spend most of my time using is VS Code. I found myself constantly switching windows and tabs to search for answers to my questions, which was quite distracting. Driven by my developer instincts, I decided to create a solution within VS Code itself—a bot with an interface similar to ChatGPT. This decision significantly elevated my productivity, eliminating the need to switch tabs and thereby reducing distractions.

During this journey, I discovered MindsDB. What makes MindsDB stand out is its simplicity in fine-tuning OpenAI models or training with custom data. It's adept at making analyses and predictions, which are crucial in the AI world. We truly need a product like this.

As a JavaScript developer with little to no Python knowledge, I was initially apprehensive about diving into this project. However, my concerns were alleviated when I discovered their JavaScript SDK. To say I was thrilled would be an understatement.

By the end of this blog, you'll see the final look of the product, which I hope you'll find exciting. Stay tuned for a deep dive into how it all comes together!

By the end of the blog, this is how the product will look like, so I hope you're excited about it:

https://youtu.be/pUwSmQl8o1E

And if you ever run into any error, you can always use my repo to get inspiration from. It's Open Source, so feel free to contribute :)) https://github.com/pritipsingh/vscode-assistant (leave a ⭐️ if you decide to check it out)

There are essentially three elements to building a vs code:

MindsDB Model which we'll be using to feed & extract responses from
The UI of the extension itself - it's the chatbot for us. I used a basic layout.
And the vs code extension. Learning about different elements of it. I found the officials docs of vs code along with chatgpt to be very helpful along the way :)

Setting Up MindsDB

There are three ways you can setup:

Cloud (you need access for it)
Docker
Your Own Server

I was initially using Docker but switched to cloud since when they provided special access for us. No matter what you're using, it's pretty much the same.

To create an AI agent. We need to create:

Model
Knowledge Base / DataBase
Skill - It can be an array of skills. This skill can be used to query your databases or knowledge bases with natural language.
AI Agent - After setting up the skills, you can create an AI agent that utilizes these skills.

We're using OpenAi as Knowledge Base

Once you've your MindsDB set up, you will need an OpenAI API key.I am using gpt-4 model, you can choose your model according to your need Let's get started:

First we create an OpenAI engine

  CREATE ML_ENGINE openai_engine
  FROM openai
  USING
      api_key = 'your-openai-api-key';

Then, create a model using this engine:

  CREATE MODEL openai_vscode
  PREDICT completion
  USING
  engine = 'openai_engine',
  model_name = 'gpt-4',
  temperature=0,
  max_tokens=2000,
  prompt_template = '{{prompt}}';

Check if the model is created using:
```
  DESCRIBE openai_vscode;
```
Once the model is created you can choose to fine tune your model. Connect your database and more.

Use the model openai_vscode to create the knowledge base:

  CREATE KNOWLEDGE BASE my_openai_knowlwdgebase
  USING
      model = openai_vscode;

Then we use my_openai_knowlwdgebase to create the skill.

  CREATE SKILL my_openai_skill_vscode_extension
  USING
      type = 'knowledge_base',
      source = 'my_openai_knowlwdgebase', 
      description = 'This skill utilizes the "my_openai_knowledgebase" to analyze code snippets. It leverages the OpenAI model to provide detailed coding analysis, identify errors, suggest best practices, and offer optimization tips. The skill aims to assist developers in improving code quality and efficiency by providing context-aware insights and recommendations.';

Now that we have the skill my_openai_skill_vscode_extension , we can use this to create our own agent! Isn't that exciting?

Note: You can create multiple skills to pass in the array.
```
  CREATE AGENT my_agent_vscode
  USING
     model = 'openai_vscode', -- this must be created with CREATE MODEL
     skills = ['my_openai_skill_vscode_extension'];
```
You've your agent ready !!!✨🚀🎉

Building the VS Code Extension

This was my first time making an extension and loved how straightforward the whole process was. To build one first:

Use the Yeoman Generator to bootstrap a new VS Code extension:

   npm install -g yo generator-code
   yo code

Choose "New Extension (TypeScript)" and follow the prompts.
Open the generated project in VS Code.

Use the comments here to understand what's going on:

import * as vscode from 'vscode';

 //the activate function in your extension.ts file to create and show a Webview
export function activate(context: vscode.ExtensionContext) {
    context.subscriptions.push(
        vscode.commands.registerCommand('extension.askQuestion', () => {
            const panel = vscode.window.createWebviewPanel(
                'askQuestion', // Identifies the type of the webview. Used internally
                'Ask a Question', // Title of the panel displayed to the user
                vscode.ViewColumn.One, // Editor column to show the new webview panel in.
                {
                    enableScripts: true  // Webview options.
                }
            );

            panel.webview.html = getWebviewContent();

            // Handle messages from the webview
            panel.webview.onDidReceiveMessage(
                message => {
                    switch (message.command) {
                        case 'askQuestion':
                            const response = getResponse(message.text);
                            panel.webview.postMessage({ command: 'showResponse', text: response });
                            return;
                    }
                },
                undefined,
                context.subscriptions
            );
        })
    );
}

function getWebviewContent() {
// UI of your bot
    return `
        <!DOCTYPE html>
        <html lang="en">
        <head>
            <meta charset="UTF-8">
            <meta name="viewport" content="width=device-width, initial-scale=1.0">
            <title>Ask a Question</title>
        </head>
        <body>
            <input type="text" id="question" placeholder="Type your question here...">
            <button onclick="askQuestion()">Ask</button>
            <div id="response"></div>

            <script>
                const vscode = acquireVsCodeApi();
                function askQuestion() {
                    const question = document.getElementById('question').value;
                    vscode.postMessage({
                        command: 'askQuestion',
                        text: question
                    });
                }

                window.addEventListener('message', event => {
                    const message = event.data;
                    switch (message.command) {
                        case 'showResponse':
                            const responseElement = document.getElementById('response');
                            responseElement.textContent = message.text;
                            break;
                    }
                });
            </script>
        </body>
        </html>
    `;
}

function getResponse(question: string) {
    // Dummy responses for demonstration - we'll re-write this
    const dummyResponses = [
        "Response to Question 1",
        "Response to Question 2",
        "Response to Question 3",
    ];

    // Simple logic to select a response (can be replaced with more complex logic)
    return dummyResponses[Math.floor(Math.random() * dummyResponses.length)];
}
//this is called whenever the extension is removed
export function deactivate() {}

Update Yourpackage.json

"activationEvents": [
    "onCommand:extension.askQuestion"
],
"contributes": {
    "commands": [
        {
            "command": "extension.askQuestion",
            "title": "Ask a Question"
        }
    ]
}

Testing the Extension
- Press F5 to run the extension in a new Extension Development Host window.
- Open the Command Palette (Ctrl+Shift+P or Cmd+Shift+P on macOS) and type "Ask a Question". Hit Enter.
- A webview panel should open where you can type a question and see a random response.

Now you'll be able to get dummy responses from the getResponse function. Isn't that cool? Let's get some real response now!

Connecting MindsDB and Extension

I am using their Javascript SDK to connect. Make a new file in src MindsDB.ts and use the following configuration :

import MindsDB from "mindsdb-js-sdk";
export async function getResponse(question: string) {
    try {
   //if you're using Docker or your own local machine, you'll need to write th url instead.

        await MindsDB.connect({
            user: 'myemail@gmail.com',
            password: 'mypassword'
        });

   const query = `SELECT prompt, completion FROM openai_vscode as m WHERE prompt ="${question}"`;
    const query1 = await MindsDB.SQL.runQuery(query);
    console.log("response", query1.rows[0].completion);
        return query1.rows[0].completion;

    } catch (error) {
      console.log(error);
      return "Issue Fetching The Response, Please Try Again";
    }
}

Now Import this in extension.ts and replace this with the dummy getResponse we had, reload the extension, adjust your css and enjoy your personal assistant inside your vscode extension.

If you enjoyed reading this, don't forget to leave a like and stay tuned for a new blog!!🚀

Creating a Custom VS Code Extension as a Personal AI Agent( Assistant)

Using Javascript, OpenAi and MindsDB

Setting Up MindsDB

You've your agent ready !!!✨🚀🎉

Building the VS Code Extension

Use the Yeoman Generator to bootstrap a new VS Code extension:

Connecting MindsDB and Extension