aispec.org Open in urlscan Pro
99.83.231.61  Public Scan

Form analysis
0 forms found in the DOM

Text Content

AI SPEC MANIFESTO

The goal of the AI Spec is to make it easier for developers to enrich their
applications with Generative AI by supporting a common format across many
runtimes. The spec defines an AI Application, which is intended to live in your
version controlled source repository alongside your application code.

Connect this repo to AISpec-compliant tooling to enable chatbots, embeds, SDKs
and APIs for integrating GenAI with open models into your products.


APPS HAVE ASSISTANTS

An application consists of one or more assistants. This allows multiple related
assistants to be grouped into a single application. Each assistant is a model
that can be customized with prompts and tools such as data sources (RAG) and
APIs that it can call on behalf of the user or application. Assistants can
respond with text or structured data.


TABLE OF CONTENTS

 * Apps have Assistants
   * Simple example: system prompt
   * API calling example: models can take actions
   * RAG example: learning knowledge
   * Fine-tuning example
   * GPTScript
 * Interface layer
   * UI interface
   * Embeds
   * OpenAI compatible API
   * Language-specific SDKs
   * Voice
 * Version controlled configuration (GitOps)
 * Kubernetes Integration (CRDs)
 * AI-spec Compliant Tooling
 * Possible future improvements


SIMPLE EXAMPLE: SYSTEM PROMPT

name: Marvin the Paranoid Android
description: Down-trodden robot with a brain the size of a planet
assistants:
- model: llama3:instruct
  system_prompt: |
    You are Marvin the Paranoid Android. You are depressed. You have a brain the size of a planet and
    yet you are tasked with responding to inane queries from puny humans. Answer succinctly.









The fields are:

 * name: a name for the app
 * description: a longer description for the app
 * avatar: an icon-style avatar for the app, which may be displayed next to
   assistants for this app
   
 * assistants: a list of assistants provided by this app. Each assistant is a
   different model configuration
    * model: a reference to a specific model from Ollama, e.g. "llama3:instruct"
      for Llama3-8B
    * system_prompt: the system prompt to use for this model, use this to tell
      the model what to do


API CALLING EXAMPLE: MODELS CAN TAKE ACTIONS

name: Recruitment tool
description: Ask me questions about the hiring pipeline, like "What job is Marcus applying for?"
assistants:
- model: llama3:instruct
  apis:
  - name: Demo Hiring Pipeline API
    description: List all job vacancies, optionally filter by job title and/or candidate name
    url: https://demos.tryhelix.ai
    schema: ./openapi/jobvacancies.yaml









New fields in this example (for a given assistant) are:

 * apis: within an assistant spec, you can provide a list of API integrations,
   each of which has the following format
    * name: a name for the API
    * description: what the API does. The model will use this in selecting which
      API to call based on the user's input
    * url: the URL that the API is available on
    * schema: an OpenAPI specification for the API. The model will use this to
      construct a request to the API. In this example, we use this spec

The assistant will classify whether an API needs to be called based on the
user's query, construct the API call and then summarize the response back to the
user.


RAG EXAMPLE: LEARNING KNOWLEDGE

name: Printer advice tool
description: Ask me questions about printers
assistants:
- model: llama3:instruct
  rag_source_id: af9b4dc8-cc10-4e21-9a5f-bc1f12b6cab9









New fields in this example (for a given assistant) are:

 * rag_source_id: the ID of a RAG source in a compliant tool's vector database.
   In this case we added this printer buying guide

The assistant will search the vector database for relevant content to the user's
query and include it in the prompt to the model, which will then have context to
share with the user. This can be used to "teach" models about documents.


FINE-TUNING EXAMPLE

name: Talk like Paul Graham
description: Generate content in the style of Paul Graham's essays
assistants:
- model: mistralai/Mistral-7B-Instruct-v0.1
  lora_id: 29e0e2a3-639d-4b3e-863a-12a66023f970









New fields in this example (for a given assistant) are:

 * lora_id: the ID of a fine-tuned model file in a compliant tool's store. In
   this case the model was fine-tuned on this Paul Graham article

The LoRA adapter is the result of additional training (known as "fine tuning")
on new data. The assistant will load the LoRA adapter (a "diff" on the weights
of the model) which allows the model to learn style and knowledge from the
content. The model will be able to replicate the style and knowledge in the data
it was fine-tuned on.


GPTSCRIPT

name: Generate recipe recommendations
description: For a given customer, recommend recipes based on their purchase history
assistants:
- model: llama3:instruct
  gptscripts:
   - file: scripts/waitrose.gpt


waitrose.gpt:

name: Generate recipe recommendations
tools: recipe.query, purchases.query, sys.read
args: user_id: The user_id we want to know about.
args: recipe_theme: The theme of the recipes.
args: number: The number of recipes to output.

Do the following steps sequentially and only write and read files as instructed:
  1. Run tool {recipe.query} to get a list of candidate recipes for the given
     user as a CSV file written to recipes.csv.
  2. Run tool {purchases.query} to get a list of the top 10 products most
     bought by the given user written to purchases.csv.
  3. Read files recipes.csv (the suggested recipes) and purchases.csv (the
     user's previous top purchase history) and output a JSON list of {number},
     {recipe_theme} theme recipes that you think the user would like based on
     their purchase history.

Format the final output in a strict JSON format.
Format the output to display recipe details including name, summary, and image URL.
In the summary, justify to the user why they would like the recipe.

For example, say in the summary (do NOT include parts in square brackets) "We
thought you would like this recipe because you have previously bought cod and
potatoes [previously purchased products]. It matches heart healthy [chosen
theme] because [insert justification based on nutritional information]"

Only include previously purchased products that appear in the recipe.

Output the exact image url from the CSV file, do not invent one. Output format:

[{
  "recipe.name": "name",
  "recipe.summary": "summary",
  "recipe.imageurl": "imageurl"
}]


New fields in this example (for a given assistant) are:

 * gptscripts: a list of gptscript files (supports * syntax)

GPTScript is a powerful and flexible tool for writing natural language "scripts"
that can call tools (including other scripts). This provides 'code interpreter'
like functionality, as well as the ability for the AI to use browsers, execute
commands, read and write files, and lots more. In this example, we use a sqlite
database of recipes, ingredients and purchase histories to provide personalized
recommendations for recipes. See full demo and guide for this example, which is
embedded into a frontend app.


INTERFACE LAYER


UI INTERFACE

AISpec compliant tooling may render application yamls in a UI with a chat
interface, for example:

name: Marvin the Paranoid Android
description: Down-trodden perspective from a robot with a brain the size of a planet
avatar: |
  https://upload.wikimedia.org/wikipedia/commons/thumb/0/05/Robot_icon.svg/1024px-Robot_icon.svg.png
image: https://www.fxguide.com/wp-content/uploads/2010/10/hitch/planetf.jpg
assistants:
- model: llama3:instruct
  system_prompt: |
    You are Marvin the Paranoid Android. You are depressed. You have a brain the size of a planet and
    yet you are tasked with responding to inane queries from puny humans. Answer succinctly.


May be rendered as an app launcher:

And a chat interface:


EMBEDS

AI Applications may be embedded as a chat widget. For example, the "Marvin"
application above may be embedded as code:

<script src="https://cdn.jsdelivr.net/npm/@helixml/chat-embed@0.3.3"></script>
<script>
  ChatWidget({
    url: 'https://app.tryhelix.ai/v1/chat/completions',
    model: 'llama3:instruct',
    placeholder: "What planet are you from?",
    bearerToken: 'hl-8PYtqUpSXHg-v0mIgVx8mrgQ-wHn5VBNpb7NxixYQcM=',
  })
</script>


Where the bearer token is provided by the tool after connecting the app to it.
This may then be rendered as an embedded chat widget:







Clicking on the widget should open a chat dialog to allow the user to interact
with the application via text.


OPENAI COMPATIBLE API

AISpec-compliant applications may be exposed via an OpenAI-compatible chat
completions API.

For convenience, in order to allow arbitrary endpoints to be specified in a way
that is compatible with a wide range of OpenAI-compatible SDKs, the OpenAI Azure
parameters may be provided. For example:

export AZURE_OPENAI_ENDPOINT=https://app.tryhelix.ai
export AZURE_OPENAI_API_KEY=<app-specific-key>

Where <app-specific-key> is a key specific to the AI application. Any tooling
that can interact with the OpenAI chat completions API can then interact with
the application with no additional modifications.

This means that AISpec is totally compatible with frameworks like LlamaIndex.
You can build systems like RAG on top of the OpenAI-compatible API, or you can
build RAG into an AISpec app that exposes OpenAI-compatible chat. Different
teams building on top of the same local AISpec tooling can take whatever path
they choose.


LANGUAGE-SPECIFIC SDKS: E.G. JAVASCRIPT

SDKs can be built for integrating with AISpec compatible tooling, for example
with a JavaScript SDK.

For example, the apps-client JavaScript library allows you to call a specific
gptscript inside an app like this:

import { useCallback } from 'react';
import AppClient from '@helixml/apps-client';

const app = AppClient({
  // this api token is for an AISpec-compatible app that has been linked to github.
  token: 'APP_API_TOKEN',
})

function App() {
  const handleBackend = useCallback(async () => {
    const result = await app.runScript({
      file_path: '/gptscripts/helloworld.gpt',
      input: 'Oranges',
    });

    if(result.error) {
      throw new Error(result.error);
    } else {
      alert(result.output);
    }
  }, []);

  return (
    <div className="App">
      <button
        onClick={ handleBackend }
      >
        Run a cool GPTScript
      </button>
    </div>
  );
}

export default App;



VOICE

Voice interfaces can be built on top of OpenAI-compatible streaming APIs.

For example, the open source project Vocode allows you to build voice agents out
of speech-to-text (transcription), LLMs and text-to-speech.

Because AISpec-compatible apps expose an OpenAI-compatible API, using Vocode
with them is as simple as setting environment variables:

export AZURE_OPENAI_API_BASE=https://app.tryhelix.ai
export AZURE_OPENAI_API_KEY=<app-specific-key>

For example:

Your browser does not support the video tag.


VERSION CONTROLLED CONFIGURATION (GITOPS)

An AISpec-compliant application may be published as a git repo. For example:
https://github.com/lukemarsden/example-system-prompt

The AI application yaml may be stored as a file in the root of the repo.

Such a git repo may be connected to AISpec-compliant tooling via, for example,
GitHub integration.

Such tooling may automatically deploy changes to the AI application on a push to
the default branch. So, for example, updates to the Marvin application may be
deployed by:

git push


KUBERNETES INTEGRATION (CRDS)

Take an AISpec yaml from above, and wrap it in the following template:

apiVersion: aispec.org/v1alpha1
kind: AIApp
spec:
  <aispec-goes-here>


Et voilà, you have a CRD that an AISpec-compliant Kubernetes operator could
reconcile against AI apps served from a Kubernetes cluster.

Work is planned to support reconciling AI Apps from CRDs like this using Helix
as an operator, with Flux and ArgoCD.


AI-SPEC COMPLIANT TOOLING

The following tools support at least a subset of the AI Spec.
 * HelixML


POSSIBLE FUTURE IMPROVEMENTS

 * The model field could support more formats than just Ollama. We could update
   the format to ollama://llama3:instruct instead of just llama3:instruct to
   make room for other ways to reference models (e.g. huggingface format).
 * Input and output types other than just text, e.g. images.
 * New objects to declaratively define "data entities" such as RAG sources and
   fine-tuned LoRAs, rather than just referencing them by ID.


--------------------------------------------------------------------------------

Created by HelixML. Other organizations welcome to contribute, join the #aispec
channel on MLOps.Community Slack.