\n \n By that logic, if we can figure out a way to pass image data to ChatGPT (Figure 2 & Figure 3), does that mean we\u2019ve made ChatGPT multimodal? Well, not really. But it\u2019s better than nothing.\n<\/p>\n \n \n In this blog we\u2019ll try to achieve the multimodal nature of KOSMOS-1 in the text-based model GPT 3.5.\n<\/p>\n \n Mobile users interact with their devices in a variety of ways. It\u2019s a language of taps, swipes, pictures, recordings, and quick messages. The language of mobile is complex, and the LLMs that support them should embrace as much of that as they can.\n<\/p>\n \n Since the LLM that we want to experiment with (GPT 3.5) has no concept of audio, video, images, or other complex data types, we need to process the data in some way to convert it into a text format before adding it to a prompt and passing it to the LLM.\n<\/p>\n Azure Cognitive Services<\/a> (ACS) is a set of APIs that developers can leverage to perform different AI tasks. These APIs cover a broad range of AI scenarios, including:\n<\/p>\n \n Input types like audio and video can be passed through Azure Cognitive Services, analyzed, and added as context to the LLM prompt (Figure 4).\n<\/p>\n \n \n Pictures and the camera are an integral part of the mobile experience, so we\u2019ll focus on passing images to our LLM.\n<\/p>\n \n Like the Graph Explorer for Microsoft Graph, Azure Cognitive Services has an explorer for vision endpoints called \u201cVision Studio\u201d – https:\/\/portal.vision.cognitive.azure.com\/<\/a> \n<\/p>\n \n When you first sign into Vision Studio, you will get prompted to choose a resource (Figure 5). This resource will be used to cover any costs accrued while using the Vision APIs.\n<\/p>\n \n Feel free to select \u201cDo this later\u201d, Vision Studio is a great way to explore what resources you would need to create for any given feature.\n<\/p>\n \n \n Since we\u2019re trying to provide as much information about images as we can to our LLM, the \u201cImage Analysis\u201d section is a good starting point (Figure 6).\n<\/p>\n \n \n Once we know what ACS features we\u2019d like to use, we can set up an Azure resource. This will work similarly to the API key needed for using OpenAI endpoints.\n<\/p>\n \n You can either follow the Vision Studio dialog boxes from Figure 5 to set up a resource, or follow this Microsoft Learn tutorial to Create a Cognitive Services Resource with Azure Portal<\/a>.\n<\/p>\n \n In our case, we want to use the Captions endpoint<\/a> to help our LLM understand what the image represents. After testing the endpoint in Vision Studio, we know there are only a couple of valid Azure regions our resource can be assigned to (Figure 7).\n<\/p>\n \n \n Now that we have a valid Azure resource and know that the Captions endpoint is the one for us, the next step is calling it in our app!\n<\/p>\n \n The Image Analysis APIs that we\u2019ve been testing in Vision Studio can be found here: https:\/\/learn.microsoft.com\/azure\/cognitive-services\/computer-vision\/how-to\/call-analyze-image-40?tabs=rest<\/a>.\n<\/p>\n \n Since there isn\u2019t an Android Client SDK to make these calls, we\u2019ll use the REST API to pass in our image, getting a JSON response from the Captions endpoint.\n<\/p>\n \n The \n Given the image in Figure 8…\n<\/p>\n \n \n \u2026 we get the following result from the Captions ACS endpoint!\n<\/p>\n \n This result can be passed to the LLM as the final prompt (Figure 9) or added to a prompt template, augmenting the prompt and helping add context to a different request.<\/p>\n \n And now our chatbot can accept images as input even though our LLM is text-based! Notice how the results are similar to Figure 1 \u2013 we have the Generative AI component of LLMs, with an extra added layer of AI preprocessing from ACS that traditional multimodal models don\u2019t have.\n<\/p>\n \n In this blog we\u2019ve been ignoring one big question \u2013 why would anyone want to do this instead of using a multimodal LLM?<\/em>\n<\/p>\n \n As with many other LLM evaluations, the decision comes down to a couple major factors.\n<\/p>\n \n Here\u2019s a summary of the links shared in this post:\n<\/p>\n \n If you have any questions, use the feedback forum<\/a> or message us on Twitter @surfaceduodev<\/a>.\n<\/p>\n \n There won\u2019t be a livestream this week, but you can check out the archives on YouTube<\/a>.\n<\/p>\n \n Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q., Aggarwal, K., Chi, Z., Bjorck, J., Chaudhary, V., Som, S., Song, X., & Wei, F. (2023, March 1). Language is not all you need: Aligning perception with language models. https:\/\/arxiv.org\/pdf\/2302.14045.pdf<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Hello AI enthusiasts, This week, we\u2019ll be talking about how you can use Azure Cognitive Services to enhance the types of inputs your Android AI scenarios can support. What makes an LLM multimodal? Popular LLMs like ChatGPT are trained on vast amounts of text from the internet. They accept text as input and provide text […]<\/p>\n","protected":false},"author":90683,"featured_media":3323,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[741],"tags":[739,734,733],"class_list":["post-3322","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-azure","tag-chatgpt","tag-openai"],"acf":[],"blog_post_summary":" Hello AI enthusiasts, This week, we\u2019ll be talking about how you can use Azure Cognitive Services to enhance the types of inputs your Android AI scenarios can support. What makes an LLM multimodal? Popular LLMs like ChatGPT are trained on vast amounts of text from the internet. They accept text as input and provide text […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/3322","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/users\/90683"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/comments?post=3322"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/3322\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media\/3323"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media?parent=3322"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/categories?post=3322"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/tags?post=3322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/word-image-3322-1.png\")
![\"Two](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/figure-2-3-600.png\")
<\/a>How does this apply to Android?<\/h2>\n
Analyzing complex data types<\/h2>\n
\n
![\"Data](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/data-is-passed-through-azure-cognitive-services-a.png\")
Figuring out which services to use<\/h2>\n
![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/word-image-3322-5.png\")
![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/word-image-3322-6.png\")
Setting up a Computer Vision resource<\/h2>\n
![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/word-image-3322-7.png\")
Analyzing an image<\/h2>\n
val request = Request.Builder()\r\n .url(\r\n \"${Constants.AZURE_ENDPOINT_WEST_US}computervision\/imageanalysis:analyze?api-version=2023-02-01-preview&features=caption\"\r\n )\r\n .addHeader(\"Ocp-Apim-Subscription-Key\", Constants.AZURE_SUBSCRIPTION_KEY)\r\n .addHeader(\"Content-Type\", \"application\/octet-stream\")\r\n .post(encodeToRequestBody(image))\r\n .build()\r\n\r\n val client = OkHttpClient.Builder()\r\n .build()\r\n\r\n val response = client.newCall(request).execute()<\/pre>\nencodeToRequestBody()<\/code> function converts our in-app Bitmap image into a PNG formatted ByteArray.\n<\/p>\n private fun encodeToRequestBody(image: Bitmap): RequestBody {\r\n ByteArrayOutputStream().use <\/em>{ <\/strong>baos -><\/strong>\r\n image.compress(Bitmap.CompressFormat.PNG<\/em>, 100, baos)\r\n return baos.toByteArray().toRequestBody<\/em>(\"image\/png\".toMediaType<\/em>())\r\n }<\/strong>\r\n }<\/strong><\/pre>\n![](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/word-image-3322-8.jpeg\")
{\"captionResult\":{\"text\":\"a drawing of mountains and trees on a whiteboard\"}}<\/pre>\n![\"Two](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/07\/figure-9-600.png\")
<\/a>Tradeoffs with Actual Multimodal Models<\/h2>\n
\n
Resources and feedback<\/h2>\n
\n
Citations<\/h2>\n