{"id":3705,"date":"2024-12-10T08:32:03","date_gmt":"2024-12-10T16:32:03","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/semantic-kernel\/?p=3705"},"modified":"2024-12-10T08:32:03","modified_gmt":"2024-12-10T16:32:03","slug":"onnx-genai-connector-for-python-experimental","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/agent-framework\/onnx-genai-connector-for-python-experimental\/","title":{"rendered":"ONNX GenAI Connector for Python (Experimental)\u00a0"},"content":{"rendered":"

ONNX GenAI Connector for Python (Experimental)\u00a0<\/span><\/h3>\n

With the latest update we added support for running models locally with the onnxruntime-genai<\/a>. The onnxruntime-genai package is powered by the ONNX Runtime in the background, but first let\u2019s clarify what ONNX, ONNX Runtime and ONNX Runtime-GenAI are.\u00a0<\/span><\/p>\n

ONNX<\/strong><\/span><\/a><\/p>\n

ONNX is an open-source format for AI models, both for Deep Learning and traditional Machine Learning. It defines an extensible computation graph model, as well as definitions of built-in operators and standard data types.\u00a0<\/span><\/p>\n

ONNX Runtime<\/strong><\/span><\/a><\/p>\n

The ONNX Runtime executes the saved weighted operations stored using the ONNX format. The runtime is optimized to inference the model on different hardware’s like NVIDIA Cuda, Qualcom NPU\u2019s or Apple CoreML. The runtime is specific for each targeted hardware and choosing the right one for your hardware will run as fast as it can.\u00a0<\/span><\/p>\n

ONNX Runtime GenAI<\/strong><\/span><\/a><\/p>\n

The GenAI is an optimized version of the runtime dedicated for Generative AI Models. While the ONNX Runtime already allows us to run inference in a model, the GenAI version of the runtime runs language models in a very optimized way, improving the probability distribution of the next token, appending next tokens to the sentence as well as using caching tricks to boost the overall performance in a repetitive and iterative process.\u00a0\u00a0<\/span><\/p>\n

How can we use it in Semantic Kernel?\u00a0<\/strong><\/span><\/p>\n

With the ONNX Connector now available in the python version of Semantic Kernel, you are now enabled to use one of the fastest local inference engine on the market (Source: ONNX Runtime | Accelerating Phi-2, CodeLlama, Gemma and other Gen AI models with ONNX Runtime<\/a>). This will enable customers to run models offline with maximum speed, making Semantic Kernel a valuable orchestration engine for edge use cases.<\/span><\/p>\n

Use Cases<\/span><\/strong><\/p>\n

    \n
  1. Offline program to anonymize sentences before sending to the cloud to stay GDPR compliant. <\/span><\/li>\n
  2. On-Premises RAG Applications, using local memory connectors not exposing the data ensuring privacy.<\/li>\n
  3. High availability use-cases, deploying applications to Edge Devices, for example, a Jetson Nano that runs inside a car or elevator. Those are great examples where a stable internet connection is not guaranteed, ensuring 100% availability with a local Small Language Model (SLM).<\/li>\n
  4. The connector implements a Tokenizer and a Multiprocessor, which enables also Multimodality with the connector running a Phi3-Vision.<\/li>\n<\/ol>\n

    Running a Phi3-Vision model locally on your PC\u00a0<\/span><\/strong><\/p>\n

    The full Demo code can be found on here (github.com)<\/a>.\u00a0\u00a0<\/span><\/p>\n

    Note that we will run the model on our CPU since it\u2019s easier to setup, an inference is also possible to run on a GPU, using the onnxruntime-genai-cuda, be aware that you must install the corresponding versions of CUDA & CUDNN for this. A GPU inference is recommended for best performance.\u00a0<\/span><\/p>\n

    Install | onnxruntime<\/a>\u00a0<\/span><\/p>\n

    Install Dependencies\u00a0<\/span><\/strong><\/p>\n

    pip install semantic-kernel\u00a0\r\npip install onnxruntime-genai==0.4.0 or the GPU Package\u00a0<\/code><\/pre>\n
    Please note that when you are using a mac, the pip package needs to be built from source and is not available on pypi.\u00a0<\/span><\/p>\n
    Download the Model from Hugging Face\u00a0<\/span><\/strong><\/p>\n
    Make sure the hf-cli is installed\u00a0\u00a0<\/span><\/p>\n
    pip install -U \"huggingface_hub[cli]\"\u00a0<\/code><\/pre>\n
    Download Phi3-Vision\u00a0<\/span><\/p>\n
    huggingface-cli download microsoft\/Phi-3-vision-128k-instruct-onnx-cpu --include cpu-int4-rtn-block-32-acc-level-4\/* --local-dir .\u00a0<\/code><\/pre>\n
    Download an Example Image\u00a0<\/span><\/strong><\/p>\n
    curl https:\/\/onnxruntime.ai\/images\/coffee.png -o coffee.png\u00a0<\/code><\/pre>\n
    Use the connector in Semantic Kernel<\/span><\/strong><\/p>\n
    Step 1: Load the ONNX connector\u00a0<\/span><\/p>\n
    import asyncio\r\n\r\nfrom semantic_kernel.connectors.ai.onnx import OnnxGenAIChatCompletion, OnnxGenAIPromptExecutionSettings\r\nfrom semantic_kernel.contents import AuthorRole, ChatMessageContent, ImageContent, ChatHistory\r\n\r\nchat_completion = OnnxGenAIChatCompletion(\r\n    ai_model_path=\".\/cpu-int4-rtn-block-32-acc-level-4\",\r\n    template=\"phi3v\",\r\n)\r\n\r\n# Max length property is important to allocate RAM\r\n# If the value is too big, you ran out of memory\r\n# If the value is too small, your input is limited\r\nsettings = OnnxGenAIPromptExecutionSettings(\r\n    temperature=0.0,\r\n    max_length=7680,\r\n)<\/code><\/pre>\n
    Steps 2: Create the ChatHistory<\/span><\/p>\n
    system_message = \"\"\"\r\nYou are a helpful assistant.\r\nYou know about provided images and the history of the conversation.\r\n\"\"\"\r\n\r\nchat_history = ChatHistory(system_message=system_message)\r\nchat_history.add_message(\r\n    ChatMessageContent(\r\n        role=AuthorRole.USER,\r\n        items=[\r\n            ImageContent.from_image_path(image_path=\"coffee.png\"),\r\n        ],\r\n    ),\r\n)\r\nchat_history.add_user_message(\"Describe the image.\")<\/code><\/pre>\n
    Step 3: Run the Model<\/span><\/p>\n
    answer = asyncio.run(\r\n    chat_completion.get_chat_message_content(\r\n        chat_history=chat_history,\r\n        settings=settings,\r\n    )\r\n)\r\n\r\nprint(f\"Answer: {answer}\")<\/code><\/pre>\n
    Step 3.5: Running Multimodal Models on CPU can take some time (1-2 Minutes), make sure you grab yourself a coffee. \ud83d\ude42\u00a0<\/span><\/p>\n
    Answer: The image shows a cup of coffee with a latte art design on top.<\/code><\/pre>\n
    Running the Connector with Accelerated Hardware\u00a0<\/strong><\/span><\/p>\n
    The ONNX Runtime can also be run with NVIDIA CUDA, DirectML, or Qualcom NPU\u2019s currently. If you want to use the connector while leveraging your dedicated hardware, please check out the following sites and use the corresponding pip package.\u00a0<\/span><\/p>\n