{"id":493,"date":"2022-12-06T15:22:48","date_gmt":"2022-12-06T23:22:48","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/windowsai\/?p=493"},"modified":"2022-12-06T15:22:48","modified_gmt":"2022-12-06T23:22:48","slug":"transformer-support-for-pytorch-with-directml-is-here","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/windowsai\/transformer-support-for-pytorch-with-directml-is-here\/","title":{"rendered":"Transformer support for PyTorch with DirectML is here!"},"content":{"rendered":"<p>The latest release of<a href=\"https:\/\/pypi.org\/project\/torch-directml\/\"> PyTorch with DirectML<\/a> is available today! This release brings support for training popular Transformer Models like GPT2, BERT, and Detection Transformers. To get started with training Transformer Models using PyTorch with DirectML, you can find a new sample on the <a href=\"https:\/\/github.com\/microsoft\/DirectML\/tree\/master\/PyTorch\/1.13\/attention_is_all_you_need\">DirectML GitHub<\/a>. The sample covers training a PyTorch implementation of the Transformer Model in the popular paper &#8220;<a href=\"https:\/\/arxiv.org\/abs\/1706.03762\">Attention is All You Need<\/a>\u00a0&#8221; (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, arxiv, 2017).<\/p>\n<p>This release of PyTorch with DirectML also includes improved memory consumption capabilities to unlock faster performance and the ability to use larger batch sizes.<\/p>\n<p>Finally, PyTorch with DirectML now follows a Plugin model with support for the latest version of PyTorch (1.13). After installing PyTorch, simply pip install torch-directml \u00a0to get started. Once you\u2019ve installed the Torch-DirectML plugin, you can begin training AI models starting with the following lines:<\/p>\n<p><span style=\"font-family: 'courier new', courier, monospace;\">import torch<\/span><\/p>\n<p><span style=\"font-family: 'courier new', courier, monospace;\">import torch_directml<\/span><\/p>\n<p><span style=\"font-family: 'courier new', courier, monospace;\">dml = torch_directml.device()<\/span><\/p>\n<p><span style=\"font-family: 'courier new', courier, monospace;\">tensor = torch.tensor([1]).to(dml)\u00a0 # Note that dml is a variable, not a string!<\/span><\/p>\n<p>&nbsp;<\/p>\n<p>Please note that this release of the Torch-DirectML plugin is mapped to the \u201cPrivateUse1\u201d Torch backend. The new torch.directml.device() API is a convenient wrapper for sending your tensors to the DirectML device. Now you\u2019re ready to train your models using PyTorch with DirectML!<\/p>\n<p>Please leave any questions, suggestions, or issues here on <a href=\"https:\/\/github.com\/microsoft\/DirectML\/issues\">GitHub<\/a>. Our team is constantly engaging with the community and would love to hear your input!<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The latest release of PyTorch with DirectML is available today! This release brings support for training popular Transformer Models like GPT2, BERT, and Detection Transformers. To get started with training Transformer Models using PyTorch with DirectML, you can find a new sample on the DirectML GitHub. The sample covers training a PyTorch implementation of the [&hellip;]<\/p>\n","protected":false},"author":60022,"featured_media":74,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-493","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-windows-ai"],"acf":[],"blog_post_summary":"<p>The latest release of PyTorch with DirectML is available today! This release brings support for training popular Transformer Models like GPT2, BERT, and Detection Transformers. To get started with training Transformer Models using PyTorch with DirectML, you can find a new sample on the DirectML GitHub. The sample covers training a PyTorch implementation of the [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/posts\/493","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/users\/60022"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/comments?post=493"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/posts\/493\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/media\/74"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/media?parent=493"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/categories?post=493"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/windowsai\/wp-json\/wp\/v2\/tags?post=493"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}