{"id":14021,"date":"2011-05-06T00:01:00","date_gmt":"2011-05-06T00:01:00","guid":{"rendered":"https:\/\/blogs.technet.microsoft.com\/heyscriptingguy\/2011\/05\/06\/use-powershell-to-copy-a-table-between-two-sql-server-instances\/"},"modified":"2011-05-06T00:01:00","modified_gmt":"2011-05-06T00:01:00","slug":"use-powershell-to-copy-a-table-between-two-sql-server-instances","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/scripting\/use-powershell-to-copy-a-table-between-two-sql-server-instances\/","title":{"rendered":"Use PowerShell to Copy a Table Between Two SQL Server Instances"},"content":{"rendered":"

Summary<\/b>: Learn how to use Windows PowerShell to quickly and efficiently copy a table between two SQL Server instances.<\/p>\n

\"Hey,  Hey, Scripting Guy! How can I copy a table between two SQL Server instances?<\/p>\n

—PY<\/p>\n

\"Hey, Hello PY, Microsoft Scripting Guy, Ed Wilson, is here. Today is the last day for this week’s SQL Guest Blogger series. Today our guest is Yan Pan. <\/p>\n

\"Photo<\/a><\/p>\n

\n

Yan Pan has 10 years working on various versions of SQL Server (MCITP SQL Server 2008, MCITP SQL Server 2005, MCDBA SQL Server 2000, plus OCA Oracle 10g), mainly in the financial services sector. She is a senior database architect, and she wrote a Wrox book titled Microsoft SQL Server 2008 Administration with Windows PowerShell<\/a>. Besides SQL Server and Windows PowerShell, she also develops complex BI solutions with Microsoft technologies, works on other databases, such as Oracle and columnar databases, programs in .NET, and performs financial data development and research. If you would like to read more articles by her, visit her website at http:\/\/yanpansql.com\/<\/a>.<\/p>\n<\/blockquote>\n

Very often, a DBA gets user requests to copy a table from a production to a development environment for testing purposes, or to synchronize a table in a reporting database with a production database. As a DBA, you can create an SSIS package with a Data Flow task in Business Intelligence Development Studio (BIDS), or you can launch the Import and Export Wizard from SQL Server Management Studio and let the wizard create an SSIS package for you. <\/p>\n

However, you cannot reuse the same SSIS package to copy a different table with a different structure. You have to create a new package for the new table or modify the column mappings of an existing package in BIDS. It would be far more convenient to have a script that takes the names of the source server, source database, source table, destination server, destination database, and destination table as the parameters, and can be reused by simply providing different names. <\/p>\n

Scripting can make a DBA’s life much easier. You might ask, “Why not use the bcp utility?” Yes, you can export the source table to a file and then import it to the destination table from the file. However, it requires creating a file on disk. We all know I\/O is typically the most expensive resource with data manipulation activities. Of course, you can also use non-Microsoft tools, such as SQL Data Compare from Red Gate, but these tools come at a cost, and they are not usually optimized for your special use cases. They can be slow and resource intensive when the target table size is big. <\/p>\n

Here I introduce a Windows PowerShell script that can be reused to copy the data from a source table to a destination table. This script uses the .NET SqlBulkCopy<\/a> class with a streaming IDataReader<\/b> to achieve the optimal performance (using a single thread).<\/p>\n

Let us first define the parameters in the following param block. As you can see, the script requires a source server name, a source database name, a source table name, and a destination server name. The names of the destination database and table are optional. If you do not provide them, the script assumes that they are the same as the source. If you have a table in a non-default schema, you also need to specify the table names as two-part names. For example, if the user account under which you run the script maps to a user in the destination database with the default schema dbo<\/b>, but you want to copy the data into an Order<\/b> table in a non-default schema Test<\/b>, then you need to specify the destination table name as Test.Order<\/b>. The script also uses a switch parameter, Truncate<\/b>. When this switch is included, the destination table is truncated and the existing data is removed before the copy.<\/p>\n

\n

Param<\/span><\/span><\/span> (<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $SrcServer<\/span><\/span>,<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $SrcDatabase,<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $SrcTable<\/span><\/span>,<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $DestServer<\/span><\/span>,<\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $DestDatabase<\/span><\/span>, # Name of the destination database is optional. When omitted, it is set to the source database name.<\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $DestTable<\/span><\/span>, # Name of the destination table is optional. When omitted, it is set to the source table name. <\/span><\/span><\/p>\n

      <\/span><\/span>[switch<\/span><\/span>] $Truncate<\/span><\/span> # Include this switch to truncate the destination table before the copy.<\/span><\/span><\/p>\n

  <\/span><\/span>)<\/span><\/span><\/p>\n

 <\/span><\/p>\n

  <\/span><\/span>If<\/span><\/span> ($DestDatabase<\/span><\/span>.Length<\/span><\/span> –eq 0) {<\/span><\/span><\/p>\n

    <\/span><\/span>$DestDatabase =<\/span><\/span> $SrcDatabase<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>If<\/span><\/span> ($DestTable<\/span><\/span>.Length<\/span><\/span> –eq 0) {<\/span><\/span><\/p>\n

    <\/span><\/span>$DestTable =<\/span><\/span> $SrcTable<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>If<\/span><\/span> ($Truncate<\/span><\/span>) { <\/span><\/span><\/p>\n

    <\/span><\/span>$TruncateSql =<\/span><\/span> “TRUNCATE TABLE “<\/span><\/span> +<\/span><\/span> $DestTable<\/span><\/span><\/p>\n

    <\/span><\/span>Sqlcmd -S $DestServer -d $DestDatabase -Q $TruncateSql<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n<\/blockquote>\n

Next, we prepare a SqlDataReader<\/b>, a forward-only stream of rows from the source table. In our example, we have an Order table in the AdventureWorks database on the source server, YYCenter. Notice that we also create the helper function ConnectionString<\/b> to return a connection string based on a server name and a database name.<\/p>\n

\n

Function<\/span><\/span><\/span> ConnectionString<\/span><\/span>([string<\/span><\/span>] $ServerName, [string<\/span><\/span>] $DbName) <\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>“Data Source=$ServerName<\/span><\/span>;Initial Catalog=$DbName<\/span><\/span>;Integrated Security=True;”<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/p>\n

  <\/span><\/span>$SrcConnStr =<\/span><\/span> ConnectionString<\/span><\/span> $SrcServer $SrcDatabase<\/span><\/span><\/p>\n

  <\/span><\/span>$SrcConn<\/span><\/span>  <\/span>=<\/span><\/span> New-Object<\/span><\/span><\/b> System.Data.SqlClient.SQLConnection($SrcConnStr<\/span><\/span>)<\/span><\/span><\/p>\n

  <\/span><\/span>$CmdText =<\/span><\/span> “SELECT * FROM “<\/span><\/span> +<\/span><\/span> $SrcTable<\/span><\/span><\/p>\n

  <\/span><\/span>$SqlCommand<\/span><\/span> =<\/span><\/span> New-Object<\/span><\/span><\/b> system.Data.SqlClient.SqlCommand($CmdText<\/span><\/span>, $SrcConn<\/span><\/span>)<\/span>  <\/span><\/span><\/span><\/p>\n

  <\/span><\/span>$SrcConn.Open<\/span><\/span>()<\/span><\/span><\/p>\n

  <\/span><\/span>[<\/span><\/span>System.Data.SqlClient.SqlDataReader] <\/span><\/span><\/span>$SqlReader<\/span><\/span> <\/span><\/span>=<\/span><\/span> <\/span><\/span>$SqlCommand<\/span><\/span>.<\/span><\/span>ExecuteReader<\/span><\/span><\/span>()<\/span><\/span><\/p>\n<\/blockquote>\n

We are now ready to do the bulk insert into the destination table.<\/p>\n

\n

Try<\/span><\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>$DestConnStr =<\/span><\/span> ConnectionString<\/span><\/span> $DestServer $DestDatabase<\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy<\/span><\/span> =<\/span><\/span> New-Object<\/span><\/span><\/b> Data.SqlClient.SqlBulkCopy<\/span><\/span>($DestConnStr<\/span><\/span>, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity<\/span><\/span>)<\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy<\/span><\/span>.DestinationTableName =<\/span><\/span> <\/span>$DestTable<\/span><\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy.WriteToServer<\/span><\/span>($sqlReader)<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

  <\/span><\/span>Catch<\/span><\/span> [System.Exception]<\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>$ex<\/span><\/span> =<\/span><\/span> $_<\/span><\/span>.Exception<\/span><\/span><\/p>\n

    <\/span><\/span>Write-Host<\/b> $ex<\/span><\/span>.<\/span>Message<\/span><\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

  <\/span><\/span>Finally<\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>Write-Host<\/span><\/span><\/b> “Table $SrcTable in $SrcDatabase database on $SrcServer has been copied to table $DestTable in $DestDatabase database on $DestServer”<\/span><\/span><\/p>\n

   <\/span><\/span> <\/span>$SqlReader.close<\/span><\/span>()<\/span><\/span><\/p>\n

    <\/span><\/span>$SrcConn<\/span><\/span>.Close<\/span><\/span>()<\/span><\/span><\/p>\n

    <\/span><\/span>$SrcConn<\/span><\/span>.Dispose<\/span><\/span>()<\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy.Close<\/span><\/span>()<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/span><\/p><\/blockquote>\n

You can also set a batch size, for example, 1000 rows, or increase the default time-out of 30 seconds to 60 seconds by setting the properties of the SqlBulkCopy<\/b> instance as shown here. <\/p>\n

\n

 <\/span><\/span><\/span><\/p>\n

$bulkCopy.BatchSize = 1000<\/span><\/span><\/span><\/p>\n

$bulkCopy.BulkCopyTimeout = 0<\/span><\/span><\/span><\/p>\n<\/blockquote>\n

Since Try\/Catch\/Finally<\/b> block is only available in Windows PowerShell 2.0, we add the version tag into the script. <\/p>\n

\n

#requires -version 2.0<\/span><\/span><\/span><\/p>\n<\/blockquote>\n

Let us put all the pieces of the script together. Here is the complete script Copy-SqlTable.ps1, and it can also be found in the Script Repository<\/a>.<\/p>\n

\n

  <\/span><\/span><\/span>#requires -version 2.0<\/span>  <\/span><\/span><\/span><\/span><\/span><\/p>\n

  <\/span><\/span>Param<\/span><\/span> (<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $SrcServer<\/span><\/span>,<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $SrcDatabase,<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $SrcTable<\/span><\/span>,<\/span><\/span><\/p>\n

      <\/span><\/span>[parameter(Mandatory =<\/span><\/span> $true<\/span><\/span>)] <\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $DestServer<\/span><\/span>,<\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $DestDatabase<\/span><\/span>, # Name of the destination database is optional. When omitted, it is set to the source database name.<\/span><\/span><\/p>\n

      <\/span><\/span>[string<\/span><\/span>] $DestTable<\/span><\/span>, # Name of the destination table is optional. When omitted, it is set to the source table name. <\/span><\/span><\/p>\n

      <\/span><\/span>[switch<\/span><\/span>] $Truncate<\/span><\/span> # Include this switch to truncate the destination table before the copy.<\/span><\/span><\/p>\n

  <\/span><\/span>)<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>Function<\/span><\/span> ConnectionString<\/span><\/span>([string<\/span><\/span>] $ServerName, [string<\/span><\/span>] $DbName) <\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>“Data Source=$ServerName;Initial Catalog=$DbName;Integrated Security=True;”<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>########## Main body ############ <\/span><\/span><\/p>\n

  <\/span><\/span>If<\/span><\/span> ($DestDatabase<\/span><\/span>.Length<\/span><\/span> –eq 0) {<\/span><\/span><\/p>\n

    <\/span><\/span>$DestDatabase =<\/span><\/span> $SrcDatabase<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>If<\/span><\/span> ($DestTable<\/span><\/span>.Length<\/span><\/span> –eq 0) {<\/span><\/span><\/p>\n

    <\/span><\/span>$DestTable =<\/span><\/span> $SrcTable<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>If<\/span><\/span> ($Truncate<\/span><\/span>) { <\/span><\/span><\/p>\n

    <\/span><\/span>$TruncateSql =<\/span><\/span> “TRUNCATE TABLE “<\/span><\/span> +<\/span><\/span> $DestTable<\/span><\/span><\/p>\n

    <\/span><\/span>Sqlcmd -S $DestServer -d $DestDatabase -Q $TruncateSql<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>$SrcConnStr =<\/span><\/span> ConnectionString<\/span><\/span> $SrcServer $SrcDatabase<\/span><\/span><\/p>\n

  <\/span><\/span>$SrcConn<\/span><\/span>  <\/span>=<\/span><\/span> New-Object<\/span><\/span><\/b> System.Data.SqlClient.SQLConnection($SrcConnStr<\/span><\/span>)<\/span><\/span><\/p>\n

  <\/span><\/span>$CmdText =<\/span><\/span> “SELECT * FROM “<\/span><\/span> +<\/span><\/span> $SrcTable<\/span><\/span><\/p>\n

  <\/span><\/span>$SqlCommand<\/span><\/span> =<\/span><\/span> New-Object<\/span><\/span><\/b> system.Data.SqlClient.SqlCommand($CmdText<\/span><\/span>, $SrcConn<\/span><\/span>)<\/span>  <\/span><\/span><\/span><\/p>\n

  <\/span><\/span>$SrcConn.Open<\/span><\/span>()<\/span><\/span><\/p>\n

  <\/span><\/span>[System.Data.SqlClient.SqlDataReader] $SqlReader<\/span><\/span> =<\/span><\/span> $SqlCommand<\/span><\/span>.ExecuteReader<\/span><\/span>()<\/span><\/span><\/p>\n

 <\/span><\/span><\/p>\n

  <\/span><\/span>Try<\/span><\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>$DestConnStr =<\/span><\/span> ConnectionString<\/span><\/span> $DestServer $DestDatabase<\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy<\/span><\/span> =<\/span><\/span> New-Object<\/span><\/span><\/b> Data.SqlClient.SqlBulkCopy<\/span><\/span>($DestConnStr<\/span><\/span>, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity<\/span><\/span>)<\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy<\/span><\/span>.DestinationTableName =<\/span><\/span> <\/span>$DestTable<\/span><\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy.WriteToServer<\/span><\/span>($sqlReader)<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

  <\/span><\/span>Catch<\/span><\/span> [System.Exception]<\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>$ex<\/span><\/span> =<\/span><\/span> $_<\/span><\/span>.Exception<\/span><\/span><\/p>\n

    <\/span><\/span>Write-Host<\/b> $ex<\/span><\/span>.<\/span>Message<\/span><\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n

  <\/span><\/span>Finally<\/span><\/span><\/p>\n

  <\/span><\/span>{<\/span><\/span><\/p>\n

    <\/span><\/span>Write-Host<\/span><\/span><\/b> “Table $SrcTable in $SrcDatabase database on $SrcServer has been copied to table $DestTable in $DestDatabase database on $DestServer”<\/span><\/span><\/p>\n

    <\/span><\/span>$SqlReader.close<\/span><\/span>()<\/span><\/span><\/p>\n

    <\/span><\/span>$SrcConn<\/span><\/span>.Close<\/span><\/span>()<\/span><\/span><\/p>\n

    <\/span><\/span>$SrcConn<\/span><\/span>.Dispose<\/span><\/span>()<\/span><\/span><\/p>\n

    <\/span><\/span>$bulkCopy.Close<\/span><\/span>()<\/span><\/span><\/p>\n

  <\/span><\/span>}<\/span><\/span><\/p>\n<\/blockquote>\n

I used the script to copy the Production.ProductInventory table in the AdventureWorks sample database on YYCenter to the Production.ProductInventory table in the AdventureWorks database on YYCenter\\SQL2008R2. The following image shows the output of the script on my computer.<\/p>\n

\"Image<\/a><\/p>\n

Thanks, Yan Pan—that was an excellent article. <\/p>\n

Well, this concludes SQL Guest Blogger Week. Join me tomorrow for the Weekend Scripter as I kick off the 2011 Scripting Games wrap-up articles. <\/p>\n

I invite you to follow me on Twitter<\/a> and Facebook<\/a>. If you have any questions, send email to me at scripter@microsoft.com<\/a>, or post your questions on the Official Scripting Guys Forum<\/a>. See you tomorrow. Until then, peace.<\/p>\n

Ed Wilson, Microsoft Scripting Guy<\/b><\/p>\n","protected":false},"excerpt":{"rendered":"

Summary: Learn how to use Windows PowerShell to quickly and efficiently copy a table between two SQL Server instances.   Hey, Scripting Guy! How can I copy a table between two SQL Server instances? —PY Hello PY, Microsoft Scripting Guy, Ed Wilson, is here. Today is the last day for this week’s SQL Guest Blogger […]<\/p>\n","protected":false},"author":595,"featured_media":87096,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[146,56,3,176,45,252],"class_list":["post-14021","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-scripting","tag-databases","tag-guest-blogger","tag-scripting-guy","tag-sql-server","tag-windows-powershell","tag-yan-pan"],"acf":[],"blog_post_summary":"

Summary: Learn how to use Windows PowerShell to quickly and efficiently copy a table between two SQL Server instances.   Hey, Scripting Guy! How can I copy a table between two SQL Server instances? —PY Hello PY, Microsoft Scripting Guy, Ed Wilson, is here. Today is the last day for this week’s SQL Guest Blogger […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/14021","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/users\/595"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/comments?post=14021"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/posts\/14021\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media\/87096"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/media?parent=14021"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/categories?post=14021"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/scripting\/wp-json\/wp\/v2\/tags?post=14021"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}