{"id":592,"date":"2023-11-15T08:00:12","date_gmt":"2023-11-15T16:00:12","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/windowsai\/?p=592"},"modified":"2023-11-16T11:26:33","modified_gmt":"2023-11-16T19:26:33","slug":"directml-llama2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/windowsai\/directml-llama2\/","title":{"rendered":"Announcing preview support for Llama 2 in DirectML"},"content":{"rendered":"

At Inspire this year we talked<\/a> about how developers will be able to run Llama 2 on Windows with DirectML and the ONNX Runtime and we’ve been hard at work to make this a reality.<\/p>\n

We now have a sample showing our progress with Llama 2 7B!<\/p>\n

See https:\/\/github.com\/microsoft\/Olive\/tree\/main\/examples\/directml\/llama_v2<\/a><\/p>\n

This sample relies on first doing an optimization pass on the model with Olive<\/a>, a powerful optimization tool for ONNX models. Olive utilizes powerful graph fusion optimizations from the ONNX Runtime and a model architecture optimized for DirectML to speed up inference times by up to 10X<\/strong>!<\/p>\n

After this optimization pass, Llama 2 7B runs fast enough that you can have a conversation in real time on multiple vendors\u2019 hardware!<\/p>\n

We\u2019ve also built a little UI to make it easy to see the optimized model in action.<\/p>\n

Thank you to our hardware partners who helped make this happen. For more on how Llama 2 lights up on our partners\u2019 hardware with DirectML, see:<\/p>\n