{"id":2677,"date":"2016-06-07T08:45:49","date_gmt":"2016-06-07T16:45:49","guid":{"rendered":"http:\/\/blogs.msdn.microsoft.com\/pythonengineering\/?p=725"},"modified":"2019-02-17T15:27:08","modified_gmt":"2019-02-17T22:27:08","slug":"lambda-exp-unleashed-2","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/python\/lambda-exp-unleashed-2\/","title":{"rendered":"Python lambda expressions unleashed"},"content":{"rendered":"<p><em>Carl Kadie, Ph.D., is a research developer in Microsoft Research\/TnR working on Genomics.<\/em><\/p>\n<p><a href=\"https:\/\/notebooks.azure.com\/library\/gOD27HzNZtg\/create_server?dest=notebooks\/LambdaUnleashed.ipynb\" target=\"_blank\"><img decoding=\"async\" class=\"alignnone size-full wp-image-452\" src=\"https:\/\/devblogs.microsoft.com\/python\/wp-content\/uploads\/sites\/12\/2016\/06\/Launch-Notebook-Now.png\" alt=\"Launch Notebook Now!\" width=\"338\" height=\"50\" \/><\/a><\/p>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p><a href=\"https:\/\/docs.python.org\/3\/reference\/expressions.html#lambda\">Lambda expressions<\/a> provide a way to pass functionality into a function. Sadly, Python puts two annoying restrictions on lambda expressions. First, lambdas can only contain an <a href=\"https:\/\/docs.python.org\/3\/glossary.html#term-expression\">expression<\/a>, not <a href=\"https:\/\/docs.python.org\/3\/glossary.html#term-statement\">statements<\/a>. Second, lambdas can&#8217;t be serialized to disk. This blog shows how we can work around these restrictions and unleash the full power of lambdas.<\/p>\n<p>So what are lambda&#8217;s good for? Suppose, you have a list of words from a string.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[1]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">()<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[1]:<\/div>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>[&#039;This&#039;, &#039;is&#039;, &#039;a&#039;, &#039;test&#039;, &#039;string&#039;, &#039;from&#039;, &#039;Carl&#039;]<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>You can sort the words with <code>sorted()<\/code>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[2]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">())<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[2]:<\/div>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>[&#039;Carl&#039;, &#039;This&#039;, &#039;a&#039;, &#039;from&#039;, &#039;is&#039;, &#039;string&#039;, &#039;test&#039;]<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Notice, however, that all the capitalized words, sort before all the lower-case words. This can be fixed by passing a lambda expression as the <em>key<\/em> argument to the <code>sorted()<\/code> function.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[3]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(),<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"n\">word<\/span><span class=\"o\">.<\/span><span class=\"n\">lower<\/span><span class=\"p\">())<\/span>  <span class=\"c1\"># `key=str.lower` also works.<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[3]:<\/div>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>[&#039;a&#039;, &#039;Carl&#039;, &#039;from&#039;, &#039;is&#039;, &#039;string&#039;, &#039;test&#039;, &#039;This&#039;]<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Lambda can be more complicated. Suppose we want to sort the words based on their (lower-case) back-to-front letters? As a reminder, here is a Python way to reverse the lower-case letters of a word:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[4]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"nb\">str<\/span><span class=\"o\">.<\/span><span class=\"n\">lower<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;Hello&quot;<\/span><span class=\"p\">)[::<\/span><span class=\"o\">-<\/span><span class=\"mi\">1<\/span><span class=\"p\">]<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[4]:<\/div>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>&#039;olleh&#039;<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>And here is how to pass this functionality to <code>sorted()<\/code> using a lambda:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[5]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(),<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"k\">lambda<\/span> <span class=\"n\">word<\/span><span class=\"p\">:<\/span> <span class=\"n\">word<\/span><span class=\"o\">.<\/span><span class=\"n\">lower<\/span><span class=\"p\">()[::<\/span><span class=\"o\">-<\/span><span class=\"mi\">1<\/span><span class=\"p\">])<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[5]:<\/div>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>[&#039;a&#039;, &#039;string&#039;, &#039;Carl&#039;, &#039;from&#039;, &#039;is&#039;, &#039;This&#039;, &#039;test&#039;]<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>But what if you want even more complex functionality? For example, functionality that requires <code>if<\/code> statements and multiple lines with unique scoping? Sadly, Python restricts lambdas to expressions only. But there is a workaround!<\/p>\n<p>Define a function that<\/p>\n<ul>\n<li>defines an inner function and &#8230;<\/li>\n<li>returns that inner function. <\/li>\n<\/ul>\n<p>Note that the inner function can refer to variables in the outer function, giving you that private scoping.<\/p>\n<p>In this example <code>lower_sorted()<\/code> is the outer function. It has an argument called <em>back_to_front<\/em>. Inside <em>lower_sorted<\/em>, we define and return an inner function called <code>inner_lower_sorted()<\/code>. That inner function has multiple lines including an <code>if<\/code> statement that references <em>back_to_front<\/em>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[6]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span> <span class=\"k\">def<\/span> <span class=\"nf\">lower_sorted<\/span><span class=\"p\">(<\/span><span class=\"n\">back_to_front<\/span><span class=\"o\">=<\/span><span class=\"kc\">False<\/span><span class=\"p\">):<\/span>\n    <span class=\"k\">def<\/span> <span class=\"nf\">inner_lower_sorted<\/span><span class=\"p\">(<\/span><span class=\"n\">word<\/span><span class=\"p\">):<\/span>\n        <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">word<\/span><span class=\"o\">.<\/span><span class=\"n\">lower<\/span><span class=\"p\">()<\/span>\n        <span class=\"k\">if<\/span> <span class=\"n\">back_to_front<\/span><span class=\"p\">:<\/span> <span class=\"c1\">#The inner function can refer to outside variables<\/span>\n            <span class=\"n\">result<\/span> <span class=\"o\">=<\/span> <span class=\"n\">result<\/span><span class=\"p\">[::<\/span><span class=\"o\">-<\/span><span class=\"mi\">1<\/span><span class=\"p\">]<\/span>\n        <span class=\"k\">return<\/span> <span class=\"n\">result<\/span>\n    <span class=\"k\">return<\/span> <span class=\"n\">inner_lower_sorted<\/span>\n\n<span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(),<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"n\">lower_sorted<\/span><span class=\"p\">()))<\/span>\n<span class=\"nb\">print<\/span><span class=\"p\">(<\/span><span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(),<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"n\">lower_sorted<\/span><span class=\"p\">(<\/span><span class=\"n\">back_to_front<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">)))<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt\"><\/div>\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre>[&#039;a&#039;, &#039;Carl&#039;, &#039;from&#039;, &#039;is&#039;, &#039;string&#039;, &#039;test&#039;, &#039;This&#039;]\n[&#039;a&#039;, &#039;string&#039;, &#039;Carl&#039;, &#039;from&#039;, &#039;is&#039;, &#039;This&#039;, &#039;test&#039;]\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>You may find lambdas and these inner functions handy enough that you&#8217;d like to serialize one to disk for use later. Sadly, if you try to seralize with <a href=\"https:\/\/docs.python.org\/3\/library\/pickle.html#module-pickle\"><code>pickle<\/code> module<\/a>, you&#8217;ll get an error message like &#8220;TypeError: can&#8217;t pickle function objects&#8221;.<\/p>\n<p>A nice workaround is to use the <a href=\"https:\/\/pypi.python.org\/pypi\/dill\"><code>dill<\/code> project<\/a> in place of <code>pickle<\/code>. The <code>dill<\/code> project is a third-party package that is now included in the standard <a href=\"https:\/\/www.continuum.io\/why-anaconda\">Anaconda distribution<\/a>. Here is an example:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[7]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"o\">!<\/span>pip install dill\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt\"><\/div>\n<div class=\"output_subarea output_stream output_stdout output_text\">\n<pre>Requirement already satisfied (use --upgrade to upgrade): dill in \/home\/nbcommon\/anaconda3_23\/lib\/python3.4\/site-packages\n<span class=\"ansi-yellow-fg\">You are using pip version 8.1.1, however version 8.1.2 is available.\nYou should consider upgrading via the &#039;pip install --upgrade pip&#039; command.<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing code_cell rendered\">\n<div class=\"input\">\n<div class=\"prompt input_prompt\">In&nbsp;[8]:<\/div>\n<div class=\"inner_cell\">\n<div class=\"input_area\">\n<div class=\" highlight hl-ipython3\">\n<pre><span><\/span><span class=\"kn\">import<\/span> <span class=\"nn\">dill<\/span> <span class=\"k\">as<\/span> <span class=\"nn\">pickle<\/span>\n\n<span class=\"k\">with<\/span> <span class=\"nb\">open<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;temp.p&quot;<\/span><span class=\"p\">,<\/span><span class=\"n\">mode<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;wb&quot;<\/span><span class=\"p\">)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">f<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">pickle<\/span><span class=\"o\">.<\/span><span class=\"n\">dump<\/span><span class=\"p\">(<\/span><span class=\"n\">lower_sorted<\/span><span class=\"p\">(<\/span><span class=\"n\">back_to_front<\/span><span class=\"o\">=<\/span><span class=\"kc\">True<\/span><span class=\"p\">),<\/span> <span class=\"n\">f<\/span><span class=\"p\">)<\/span>\n\n<span class=\"k\">with<\/span> <span class=\"nb\">open<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;temp.p&quot;<\/span><span class=\"p\">,<\/span> <span class=\"n\">mode<\/span><span class=\"o\">=<\/span><span class=\"s2\">&quot;rb&quot;<\/span><span class=\"p\">)<\/span> <span class=\"k\">as<\/span> <span class=\"n\">f<\/span><span class=\"p\">:<\/span>\n    <span class=\"n\">some_functionality<\/span> <span class=\"o\">=<\/span> <span class=\"n\">pickle<\/span><span class=\"o\">.<\/span><span class=\"n\">load<\/span><span class=\"p\">(<\/span><span class=\"n\">f<\/span><span class=\"p\">)<\/span>\n\n<span class=\"nb\">sorted<\/span><span class=\"p\">(<\/span><span class=\"s2\">&quot;This is a test string from Carl&quot;<\/span><span class=\"o\">.<\/span><span class=\"n\">split<\/span><span class=\"p\">(),<\/span> <span class=\"n\">key<\/span><span class=\"o\">=<\/span><span class=\"n\">some_functionality<\/span><span class=\"p\">)<\/span>\n<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"output_wrapper\">\n<div class=\"output\">\n<div class=\"output_area\">\n<div class=\"prompt output_prompt\">Out[8]:<\/div>\n<div class=\"output_text output_subarea output_execute_result\">\n<pre>[&#039;a&#039;, &#039;string&#039;, &#039;Carl&#039;, &#039;from&#039;, &#039;is&#039;, &#039;This&#039;, &#039;test&#039;]<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"cell border-box-sizing text_cell rendered\">\n<div class=\"prompt input_prompt\">\n<\/div>\n<div class=\"inner_cell\">\n<div class=\"text_cell_render border-box-sizing rendered_html\">\n<p>Serialization of lambdas and these inner functions opens exciting possibilities. For example, we use it in one of our libraries to run work in different processes and even on different machines in a cluster.<\/p>\n<p>We&#8217;ve seen that lambdas are a handy way to pass functionality into a function. Python&#8217;s implementation of lambdas has two restrictions, but each restriction has a workaround.<\/p>\n<ul>\n<li>Multiple lines not allowed.\n<ul>\n<li>Workaround: Define a function that defines and returns an inner function. The inner function can use variables outside itself.<\/li>\n<\/ul>\n<\/li>\n<li>Can&#8217;t pickle lambdas or inner functions.\n<ul>\n<li>Workaround: Replace <code>pickle<\/code> with <code>dill<\/code>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Python offers features such as <a href=\"https:\/\/devblogs.microsoft.com\/python\/2016\/03\/14\/idiomatic-python-comprehensions\/\"><em>list comprehensions<\/em><\/a> that makes lambdas less used that in other languages. When you do need lambdas, however, they will now be unleashed.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Carl Kadie, Ph.D., is a research developer in Microsoft Research\/TnR working on Genomics. Lambda expressions provide a way to pass functionality into a function. Sadly, Python puts two annoying restrictions on lambda expressions. First, lambdas can only contain an expression, not statements. Second, lambdas can&#8217;t be serialized to disk. This blog shows how we can [&hellip;]<\/p>\n","protected":false},"author":387,"featured_media":10119,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[14,17],"class_list":["post-2677","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-python","tag-jupyter","tag-python"],"acf":[],"blog_post_summary":"<p>Carl Kadie, Ph.D., is a research developer in Microsoft Research\/TnR working on Genomics. Lambda expressions provide a way to pass functionality into a function. Sadly, Python puts two annoying restrictions on lambda expressions. First, lambdas can only contain an expression, not statements. Second, lambdas can&#8217;t be serialized to disk. This blog shows how we can [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/posts\/2677","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/users\/387"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/comments?post=2677"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/posts\/2677\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/media\/10119"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/media?parent=2677"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/categories?post=2677"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/python\/wp-json\/wp\/v2\/tags?post=2677"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}