{"id":4733,"date":"2013-04-08T07:00:00","date_gmt":"2013-04-08T07:00:00","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/oldnewthing\/2013\/04\/08\/the-managed-way-to-retrieve-text-under-the-cursor-mouse-pointer\/"},"modified":"2013-04-08T07:00:00","modified_gmt":"2013-04-08T07:00:00","slug":"the-managed-way-to-retrieve-text-under-the-cursor-mouse-pointer","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/oldnewthing\/20130408-00\/?p=4733","title":{"rendered":"The managed way to retrieve text under the cursor (mouse pointer)"},"content":{"rendered":"<p><P>\nToday&#8217;s Little Program is a managed version of the\n<A HREF=\"http:\/\/blogs.msdn.com\/b\/oldnewthing\/archive\/2004\/04\/23\/118893.aspx\">\ntext-extraction program from\nseveral years ago<\/A>.\nIt turns out that it&#8217;s pretty easy in managed code\nbecause the accessibility folks sat down and wrote a whole framework for you,\nknown as\n<A HREF=\"http:\/\/msdn.microsoft.com\/library\/ms747327\">\nUI Automation<\/A>.\n<\/P>\n<P>\n(Some people are under the mistaken impression that UI Automation\nworks only for extracting data from applications written in managed code.\nThat is not true.\nNative code can also be a UI Automation provider.\nThe confusion arises because the name <I>UI Automation<\/I> is used\nboth for\n<A HREF=\"http:\/\/msdn.microsoft.com\/library\/ee684009\">\nthe underlying native technology<\/A>\nas well as for\n<A HREF=\"http:\/\/msdn.microsoft.com\/library\/ms747327\">\nthe managed wrappers<\/A>.)\n<\/P>\n<PRE>\nusing System;\nusing System.Windows;\nusing System.Windows.Forms;\nusing System.Windows.Automation;<\/p>\n<p>class Program\n{\n static Point MousePos {\n  get { var pos = Control.MousePosition;\n        return new Point(pos.X, pos.Y); }\n }<\/p>\n<p> public static void Main()\n {\n  for (;;) {\n   AutomationElement e = AutomationElement.FromPoint(MousePos);\n   if (e != null) {\n    Console.WriteLine(&#8220;Name: {0}&#8221;,\n     e.GetCurrentPropertyValue(AutomationElement.NameProperty));\n    Console.WriteLine(&#8220;Value: {0}&#8221;,\n     e.GetCurrentPropertyValue(ValuePattern.ValueProperty));\n    Console.WriteLine();\n   }\n   System.Threading.Thread.Sleep(1000);\n  }\n }\n}\n<\/PRE>\n<P>\nWe use the <CODE>From&shy;Point<\/CODE> method to locate the\nautomation element under the current mouse position and print\nits name and value.\n<\/P>\n<P>\nWell that was pretty simple.\nI may as well do something a little more challenging.\nSince the feature is known as UI <I>Automation<\/I>,\nI&#8217;ll try automating the <I>Run<\/I> dialog by\nprogrammatically entering some text and then clicking OK.\n<\/P>\n<PRE>\nusing System.Windows.Automation;<\/p>\n<p>class Program\n{\n static AutomationElement FindById(AutomationElement root, string id)\n {\n  return root.FindFirst(TreeScope.Children,\n   new PropertyCondition(AutomationElement.AutomationIdProperty, id));\n }<\/p>\n<p> public static void Main()\n {\n  var runDialog = AutomationElement.RootElement.FindFirst(\n   TreeScope.Children,\n   new PropertyCondition(AutomationElement.NameProperty, &#8220;Run&#8221;));\n  if (runDialog == null) return;<\/p>\n<p>  var commandBox = FindById(runDialog, &#8220;12298&#8221;);\n  var valuePattern = commandBox.GetCurrentPattern(ValuePattern.Pattern)\n                     as ValuePattern;\n  valuePattern.SetValue(&#8220;calc&#8221;);<\/p>\n<p>  var okButton = FindById(runDialog, &#8220;1&#8221;);\n  var invokePattern = okButton.GetCurrentPattern(InvokePattern.Pattern)\n                     as InvokePattern;\n  invokePattern.Invoke();\n }\n}\n<\/PRE>\n<P>\nThe program starts by looking for a window named <I>Run<\/I>\nby performing a children search on the root element\nfor an element whose <I>Name<\/I> property is equal to\n<CODE>&#8220;Run&#8221;<\/CODE>.\n<\/P>\n<P>\nAssuming it finds it,\nthe program looks for a child element whose\n<A HREF=\"http:\/\/msdn.microsoft.com\/en-us\/library\/system.windows.automation.automationelement.automationelementinformation.automationid.aspx\">\nautomation ID<\/A>\nis\n<CODE>&#8220;12298&#8221;<\/CODE>.\nHow did I know that was the automation ID to use?\n<A HREF=\"http:\/\/msdn.microsoft.com\/en-us\/library\/aa349646.aspx\">\nThe documentation for UI Automation<\/A>\nsuggests using a tool like UI Spy to look up the automation IDs.\n<\/P>\n<P>\nMind you, since I am automating something outside my control,\nI have to accept that the automation ID may change in future\nversions of Windows.\n(It&#8217;s not like they check with me before making changes.)\nBut this is a Little Program, not a production-level program,\nso that&#8217;s a limitation I will accept,\nsince I&#8217;m the only person who&#8217;s going to use this program,\nand if it stops working, I know who to talk to (namely, me).\n<\/P>\n<P>\nAnyway, afer we find the command box, I ask for its Value pattern.\nAutomation elements can support <I>patterns<\/I>\nwhich expose additional properties and methods specific to particular\nuses.\nIn our case, the Value pattern lets us get and set the value\nof an editable object,\nso we use the <CODE>Set&shy;Value<\/CODE> method to set the\ntext in the Run dialog to <TT>calc<\/TT>.\n<\/P>\n<P>\nNext, we look for the OK button,\nwhich UI Spy told me had automation ID&nbsp;1.\nWe ask for the Invoke pattern on the button\nand then call the <CODE>Invoke<\/CODE> method.\nThe Invoke pattern is the pattern for objects that do\njust one thing,\nand <CODE>Invoke<\/CODE> means &#8220;Do\n<A HREF=\"http:\/\/www.imdb.com\/title\/tt0117887\/\">\nthat thing that you do<\/A>.&#8221;\n<\/P>\n<P>\nOpen the <I>Run<\/I> dialog and run this program.\nIt should programmatically set the command line to\n<TT>calc<\/TT>, then click OK.\nHopefully, this will run the Calculator.\n<\/P>\n<P>\nJust for fun, here&#8217;s another program that just dumps the\nautomation properties and patterns for whatever object\nis under the mouse cursor:\n<\/P>\n<PRE>\nusing System;\nusing System.Windows;\nusing System.Windows.Forms;\nusing System.Windows.Automation;<\/p>\n<p>class Program\n{\n static Point MousePos {\n  get { var pos = Control.MousePosition;\n        return new Point(pos.X, pos.Y); }\n }<\/p>\n<p> public static void Main()\n {\n  for (;;) {\n   AutomationElement e = AutomationElement.FromPoint(MousePos);\n   if (e != null) {\n    <FONT COLOR=\"blue\">foreach (var prop in e.GetSupportedProperties()) {\n     object o = e.GetCurrentPropertyValue(prop);\n     if (o != null) {\n      var s = o.ToString();\n      if (s != &#8220;&#8221;) {\n       var id = o as AutomationIdentifier;\n       if (id != null) s = id.ProgrammaticName;\n       Console.WriteLine(&#8220;{0}: {1}&#8221;, Automation.PropertyName(prop), s);\n      }\n     }\n    }\n    foreach (var pattern in e.GetSupportedPatterns()) {\n     Console.WriteLine(&#8220;Pattern: {0}&#8221;, Automation.PatternName(pattern));\n    }<\/FONT>\n    Console.WriteLine();\n   }\n   System.Threading.Thread.Sleep(1000);\n  }\n }\n}\n<\/PRE><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Today&#8217;s Little Program is a managed version of the text-extraction program from several years ago. It turns out that it&#8217;s pretty easy in managed code because the accessibility folks sat down and wrote a whole framework for you, known as UI Automation. (Some people are under the mistaken impression that UI Automation works only for [&hellip;]<\/p>\n","protected":false},"author":1069,"featured_media":111744,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[25],"class_list":["post-4733","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-oldnewthing","tag-code"],"acf":[],"blog_post_summary":"<p>Today&#8217;s Little Program is a managed version of the text-extraction program from several years ago. It turns out that it&#8217;s pretty easy in managed code because the accessibility folks sat down and wrote a whole framework for you, known as UI Automation. (Some people are under the mistaken impression that UI Automation works only for [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/4733","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/users\/1069"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/comments?post=4733"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/posts\/4733\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media\/111744"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/media?parent=4733"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/categories?post=4733"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/oldnewthing\/wp-json\/wp\/v2\/tags?post=4733"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}