August 12th, 2016

SharePoint PnP Remote Provisioning Engine – August 2016 update

(Schema v201605 and new capabilities)

In this article we introduce the new features and capabilities that we provided in the PnP Provisioning Engine, since the release of the new XML schema version 201605 and of the new OpenXML .PNP package format.

What is SharePoint PnP Remote Provisioning Engine?

PnP Remote Provisioning Engine is open source, community driven remote provisioning engine for SharePoint Online and on-premises (2013 & 2016). It can be used to remotely extract and apply customizations and assets to/from SharePoint site in standardized way, without using classic feature framework elements. Provisioning Engine is available from PnP CSOM Core component as Nuget package and it’s evolved on monthly basis, based on the contributions and work by the community in PnP Sites Core GitHub repository together with members from SharePoint engineering. PnP Provisioning Engine can be used either using managed code or with PnP PowerShell to easily automate your site customization requirements. There are numerous advantages on using this remote API based model compared to classic feature framework based customizations.

Here’s few references around the engine

During July 2016, PnP Core Component was used in more than 2000 tenants in SharePoint Online. Majority of these operations were based on the usage of the PnP Remote Provisioning Egine.

The New Schema v201605

First of all, the new schema by itself deserves to be introduced. In fact, even if the PnP Provisioning Engine is totally independent from any schema and from any template formatting technique, at the time of this writing the main and most frequently used serialization format for a provisioning template is the XML one, eventually included in the fresh new OpenXML .PNP package format.

All of the schema versions are defined based on community efforts, collected through the dedicated GitHub repository. Whenever we start working on a new schema version, we collect feedbacks and features requests, then we release a proposed schema version, which after the community feedbacks becomes a final release. Any released version of the schema is afterwards implemented in the engine.

In particular, the new schema version 201605 (released in May 2016) is described through a .MD file, which illustrates all the elements and attributes that make up a template, and is supported by the PnP Provisioning Engine since the August 2016 release.

From a schema perspective, with the latest release, we introduced support for the following elements and attributes:

  • New security attributes at the provisioning template level, to manage security inheritance for the target sites to which the template will be applied
  • A new attribute to keep track of the base template ID of the site from which the template is generated
  • A new Navigation element, to represent site navigation settings
  • Some new attributes and elements to configure ForceCheckOut at list instance level, to remove content types from the collection of content type bindings, and to add list instance level user custom actions
  • Some new elements within the Files element, in order to support the capability to upload not only files but also directories of files, eventually with custom metadata fields
  • A refactored organization of the SearchSettings element, which now supports both site collection and site level search settings
  • Support for defining reusable and deprecated terms in the taxonomy
  • Support for Contributors and/or Managers for TermGroup elements in the taxonomy
  • Improved support for hierarchical site template, through sequences

And many other minor changes. 

If you’re interested, here you can find a full sample instance of the new 201605 schema.

New Functionalities at the Engine Level

As we said, the engine is independent from the schema, but both evolve almost in parallel. In fact, here follows the list – with some code samples – of the main functionalities introduced in the PnP Provisioning Engine accordingly to the previously defined new schema release.

Open XML .PNP package format

Since June 2016 the PnP Provisioning Engine supports the capability to store the templates within a unique file with extension .PNP, which internally is based on the OpenXML package format.

This is a great new capability, which allows to store a whole template (made of the XML template file, all the branding files, the .ASPX home page, any XAML file of workflows, etc.) within a unique file. Internally the .PNP file is a compressed (ZIP) file, structured in folders and with a manifest XML file. In the following figure you can see the structure of a .PNP file.

PnP file renamed as zip and opened to see the internal structure in windows explorer

Inside the ProvisioningTemplate folder there are some infrastructural files, while in the Files folder there are all the real content files related to the template, like possible branding assets. The persistence engine allows to store files organized in folders and sub-folders, however internally all the files are stored with a flat structure. Thus, you should never manually change the content of the .PNP file, instead you should use the object model provided by the engine to do proper files handling.

In order to export a template – using C# code – and leveraging the new .PNP file format you can use the following syntax.


using (var context = new ClientContext(sourceUrl)) {

context.Credentials = new SharePointOnlineCredentials(userName, password);

ProvisioningTemplateCreationInformation ptci =
new ProvisioningTemplateCreationInformation(web);

var fileSystemConnector = new FileSystemConnect-or(String.Format(@"{0}....Templates",
AppDomain.CurrentDomain.BaseDirectory),
"");

ptci.FileConnector = new OpenXMLConnector("PnPProvisioningDemo.pnp",
fileSystemConnector,
"Paolo Pialorsi");

ProvisioningTemplate template = web.GetProvisioningTemplate(ptci);

XMLTemplateProvider provider = new XMLOpenXMLTemplateProvider(
ptci.FileConnector as OpenXMLConnector);

provider.SaveAs(template, "PnPProvisioningDemo.xml");
}

As you can see, we need to create a concrete persistence provider connector, which will store the .PNP file somewhere. The previous example uses a FileSystemConnector to store the .PNP file in the local file system (you can use also SharePoint library or Azure storage as the location). Moreover, you have to create an instance of the OpenXMLConnector type, providing the name of the .PNP file, and a persistence connector that will be used to store the physical file. Loading the .PNP file and applying the template onto a target site is illustrated in the following code excerpt.


using (var context = new ClientContext(destinationUrl)) {

context.Credentials = new SharePointOnlineCredentials(userName, password);

var fileSystemConnector = new FileSystemConnect-or(String.Format(@"{0}....Templates",
AppDomain.CurrentDomain.BaseDirectory),
"");

XMLTemplateProvider provider = new XMLOpenXMLTemplateProvider(
new OpenXMLConnector("PnPProvisioningDemo.pnp",
fileSystemConnector));

ProvisioningTemplate template = provider.GetTemplate("PnPProvisioningDemo.xml");
ProvisioningTemplateApplyingInformation ptai =
new ProvisioningTemplateApplyingInformation();
template.Connector = provider.Connector;

web.ApplyProvisioningTemplate(template, ptai);
}

As like as before, you simply need to create a physical persistence connector and to provide it to the constructor of the OpenXMLConnector type.

If you rather prefer to use PnP PowerShell to handle .PNP files, you will simply need to invoke the Get-SPOProvisioningTemplate and Apply-SPOProvisioningTemplate cmdlets providing the name of a file with .PNP extension, instead of .XML. Internally the cmdlets will handle all the plumbing for you. Here you can see the very basic PowerShell syntax to export a template into a .PNP file.


Get-SPOProvisioningTemplate -Out .PnPDemo.pnp

While here you can see the very basic syntax to apply that template to a target site.


Apply-SPOProvisioningTemplate .PnPDemo.pnp

New capabilities at template level

At the root provisioning template level we introduced the capability to extract the base template ID, so that we can better target a template. For example, if you extracted a template from a Team Site (STS#0), before applying a template to a target, you can double-check if the target is based on the same base template. We do not enforce any strict rule on that, but we simply raise a warning if you try to apply a template to a site that doesn’t share the same base template of the template’s source. In the following screenshot you can see the behavior of the engine within a PowerShell session, where we apply a template taken from a Team Site (STS#0) onto a Blog (BLOG#0) site.

PnP PowerShell CmdLets used in PowerShell ISE editor for extracting and importing templates from existing sites

Moreover, and still at the provisioning template level we introduced the capability to provision sub-sites breaking role inheritance and eventually copying role assignments from the parent site. This will make much easier to create site hierarchies with special role assignments for sub-sites.

Taxonomy

Since this last version of the engine, we introduced support – in SharePoint Online only – for ex-porting and importing Managers and Contributors of term groups in the taxonomy. In fact, since CSOM for SharePoint Online version 16.1.5312.1200 (further details here) there is support for reading and writing term groups’ managers and contributors.

Moreover, thanks to the community contributions, the engine now supports reusable terms and deprecated terms.

Navigation settings

Based on community feedbacks, we introduced the capability to export the navigation settings from a site with the publishing features enabled. We export the settings of both the current and the global navigation. The engine supports all the various kinds of navigation settings, which are highlighted in the following list, for the sake of completeness.

  • Global Navigation
    • Inherit from parent site
    • Managed navigation
    • Structural navigation
  • Current Navigation
    • Inherit from parent site
    • Managed navigation
    • Structural navigation
    • Structural navigation with just local links

When the source site is configured for managed navigation, the engine exports the Term Store ID and the Term Set ID, which will be replaced with tokens, so that if you export the taxonomy set-tings within the same template, it will be possible to replicate the entire navigation structure onto the target, eventually across a different tenant (from Development, to Staging, and to Production).

When the source site is configured for structural navigation, the engine exports all the navigation nodes hierarchy within the template. Thus, while applying the template to the target the engine will create all of the navigation nodes from scratch. You can also enable the flag RemoveExistingNodes in order to completely remove any existing structural navigation node before applying the template.

Note: Be careful, in order to properly apply the navigation settings, you will need to have the same pages referenced in the navigation source site available in the target site, too. So, if you are for example exporting a site with some content pages referenced in the navigation settings, you will have to move the content pages between the source and the destination site before moving the navigation settings. You can achieve this result by using a custom extensibility provider for the en-gine and applying a two stages template handling. For example, during extraction you can also extract pages using a cus-tom extensibility provider. During template application, still using a custom extensibility provider, you will be able to cre-ate the pages on the target, executing the template application excluding the navigation handler. After a successful appli-cation of the template and thanks to the custom extensibility provider, the content pages will exist on the target. So, you can then apply the template one more time, just using the navigation handler, to provision the navigation settings on top of the target site.

Bulk upload of directories

Another feature very frequently requested by the community is the capability to import a full di-rectory into the target site, while applying a template. In fact, the PnP Provisioning Engine since a long time ago allows to upload single files, but quite often in real project is needed to upload a bunch of content files and custom pages, eventually structured in folders. Since this new release of the engine, we introduced support for uploading directories with or without recursive handling of sub-folders. In the following excerpt you can see the XML needed, at the template level, to import a full directory, with sub-folders, and with some custom metadata fields for the uploaded files.


<pnp:Provisioning xmlns:pnp="http://schemas.dev.office.com/PnP/2016/05/ProvisioningSchema">
<pnp:Preferences Author="John White" Version="1.0" Generator="Human being :-)" />
<pnp:Templates ID="DIRECTORY-SAMPLE-TEMPLATES">
<pnp:ProvisioningTemplate ID="DIRECTORY-SAMPLE" Version="1.0"
BaseSiteTemplate="STS#0"
DisplayName="Directory Sample"
Description="This is a sample about how to bulk upload a directory">
<pnp:Files>
<pnp:Directory Src=".DirectoryBulkLoadSample"
Folder="BulkLoaded"
Overwrite="true"
Recursive="true"
IncludedExtensions="*.docx,*.pdf"
ExcludedExtensions="*.xml,*.txt"
MetadataMappingFile=".BulkLoadedMetadata.json" />
</pnp:Files>
</pnp:ProvisioningTemplate>
</pnp:Templates>
</pnp:Provisioning>

Notice the Directory element in the XML file. The Src attribute declares the path of the source directory, which is relative to the root container of the connector used to load the template. The Folder attribute defines the target document library or folder in the target site. The Overwrite and Recursive attributes are self-explanatory. The IncludedExtensions and ExcludedExtensions attributes allow to do content inclusion or exclusion based on files’ extension. As you can see you can provide multiple included or excluded extensions. Keep in mind that the order of application of the inclusion/exclusion rules first of all applies the inclusion rules, if any, and after that applies the ex-clusion rules. Thus, if you include “*.docx,*.pdf” and you exclude “*.pdf” the result will be just “*.docx”.

Lastly, there is the MetadataMappingFile attribute, which allows you to upload files providing some metadata fields that will be set after uploading the files. The metadata file has to be a JSON file, which basically is a serialization of type Dictionary<String, Dictionary<String, String>>, where the first key of the outer dictionary item is the relative URL of the file, and the inner dictionary de-fines the tuple field name and field value. In the following excerpt you can see a sample JSON file for this purpose.


{
".\DirectoryBulkLoadSample\Sample-Document-01.docx":{
"Title":"Sample Document 1",
"CustomNotes":"Sample Notes for Document 1",
"Number":"1",
"Money":"10000"
},
".\DirectoryBulkLoadSample\Sample-Document-02.docx":{
"Title":"Sample Document 2",
"CustomNotes":"Sample Notes for Document 2",
"Number":"2",
"Money":"20000"
},
".\DirectoryBulkLoadSample\SubFolder\Sample-Document-03.pdf":{
"Title":"Sample Document 3",
"CustomNotes":"Sample Notes for Document 3",
"Number":"3",
"Money":"30000"
}
}

Important: The engine supports directories just during template application and it does not extract any folder structure from a live site when you get a provisioning template from it. So, in order to support the directory bulk upload capability, you will have to manually change the XML of the provisioning template, or to play at the code level with the provisioning template domain model.

New list instance capabilities

Some new features have been introduced at the list instance level, too. For example, now you can use the ForceCheckout flag property of the ListInstance type to enable the force checkout capability on the target library. You can now remove a content type binding from a list by adding – again by manually editing the XML file – the Remove attribute. Using this last capability, you can for example remove any default content type binding from a list or library.

Moreover, at the ListInstance level there is now support for user custom actions. For example, you can use this capability to provision a custom ECB (Edit Control Block) menu item for a list or library, which is supported also by the new modern UI of SharePoint Online.

Workflow handling improvements

With the latest release of the engine, the object handler that extracts the workflows supports extraction of workflows that reference other lists and libraries, with token replacement of lists’ IDs. Moreover, it also supports the extraction of initiation forms, including the .ASPX form file. Lastly the workflow object handler now supports the extraction and application of workflows that use single and multiple tasks.

With these new improvements, the engine is now ready to support almost any real business work-flows scenario, as long as the workflow has been defined using SharePoint Designer 2013.

Search settings

Based on some smart requests from the community, we changed the behavior of the SearchSettings property of a provisioning template. Now, we support the extraction and the application of search settings both at site collection level and at site level. In the past we supported only search settings at site collection level. This change is backward compatible with the previous versions of the schema and of the engine. So, you don’t need to be worried about already existing site templates. Nevertheless, you should consider using the Convert-SPOProvisioningTemplate cmdlet to convert old XML templates into the latest schema version. Here follows the sample syntax to use the conversion cmdlet:


Convert-SPOProvisioningTemplate -Path .source.xml -Out .destination.xml -ToSchema LATEST

Provider extensibility

Last but not least, we introduced the capability to plug extensibility handlers while loading or saving a provisioning template. In fact, quite often could be useful to be able to plug custom logic during template files handling. In particular we provide the following extensibility interface:


namespace OfficeDevPnP.Core.Framework.Provisioning.Providers {
/// <summary>
/// Interface for extending the XMLTemplateProvider while retrieving a template
/// </summary>
public interface ITemplateProviderExtension {
/// <summary>
/// Initialization method to setup the extension object
/// </summary>
/// <param name="settings"></param>
void Initialize(Object settings);

/// <summary>
/// Method invoked before deserializing the template from the source reposito-ry
/// </summary>
/// <param name="stream">The source stream</param>
/// <returns>The resulting stream, after pre-processing</returns>
Stream PreProcessGetTemplate(Stream stream);

/// <summary>
/// Method invoked after deserializing the template from the source repository
/// </summary>
/// <param name="template">The just deserialized template</param>
/// <returns>The resulting template, after post-processing</returns>
ProvisioningTemplate PostProcessGetTemplate(ProvisioningTemplate template);

/// <summary>
/// Method invoked before serializing the template and before
/// it is saved onto the target repository
/// </summary>
/// <param name="template">The template that is going to be serialized</param>
/// <returns>The resulting template, after pre-processing</returns>
ProvisioningTemplate PreProcessSaveTemplate(ProvisioningTemplate template);

/// <summary>
/// Method invoked after serializing the template and before
/// it is saved onto the target repository
/// </summary>
/// <param name="stream">The source stream</param>
/// <returns>The resulting stream, after pre-processing</returns>
Stream PostProcessSaveTemplate(Stream stream);

/// <summary>
/// Declares whether the object supports pre-processing during GetTemplate
/// </summary>
Boolean SupportsGetTemplatePreProcessing { get; }

/// <summary>
/// Declares whether the object supports post-processing during GetTemplate
/// </summary>
Boolean SupportsGetTemplatePostProcessing { get; }

/// <summary>
/// Declares whether the object supports pre-processing during SaveTemplate
/// </summary>
Boolean SupportsSaveTemplatePreProcessing { get; }

/// <summary>
/// Declares whether the object supports post-processing during SaveTemplate
/// </summary>
Boolean SupportsSaveTemplatePostProcessing { get; }
}
}

As you can see, the interface definition is straightforward and allows to support the following ex-tensibility points:

  • Pre-process a template before deserialization
  • Post-process a template after deserialization
  • Pre-process a template before serialization
  • Post-process a template after serialization

You can use such kind of extensibility providers to plug custom logic like digitally signing a template, encrypting/decrypting a template, supporting XInclude at multiple levels, etc. For example, in the following test method, which is available on the GitHub repo of the PnP Provisioning Engine, you can see how to use a custom extensibility provider that supports XML encryption of the provision-ing template using an X.509 certificate.


[TestMethod]
[TestCategory(TEST_CATEGORY)]
public void XMLEncryptionTest() {

X509Certificate2 certificate = RetrieveCertificateFromStore(new
X509Store(StoreLocation.CurrentUser), "PnPTestCertificate");

if (certificate == null) {
Assert.Inconclusive("Missing certificate with SN=PnPTestCertificate in Curren-tUser" +
"Certificate Store, so can't test");
}

XMLEncryptionTemplateProviderExtension extension =
new XMLEncryptionTemplateProviderExtension();

extension.Initialize(certificate);

ITemplateProviderExtension[] extensions = new ITemplateProviderExtension[] { exten-sion };

XMLTemplateProvider provider =
new XMLFileSystemTemplateProvider(
String.Format(@"{0}....Resources",
AppDomain.CurrentDomain.BaseDirectory),
"Templates");

var template = provider.GetTemplate("ProvisioningTemplate-2016-05-Sample-01.xml");
template.DisplayName = "Ciphered template";

provider.SaveAs(template, "ProvisioningTemplate-2016-05-Ciphered.xml", extensions);
var result = provider.GetTemplate("ProvisioningTemplate-2016-05-Ciphered.xml", ex-tensions);

provider.Delete("ProvisioningTemplate-2016-05-Ciphered.xml");
Assert.IsTrue(result.DisplayName == "Ciphered template");
}

New capabilities in PnP PowerShell

If you like, you can also use the PowerShell cmdlets to involve custom extensibility providers. In fact, both the Get-SPOProvisioningTemplate and Apply-SPOProvisioningTemplate cmdlets now support the TemplateProviderExtensions argument, which can be used to provide an array of one or more provider extensions. Here you can see the PowerShell syntax to involve the XML encryption capability illustrated before:


# Get the ciphered template from a source site
Connect-SPOnline "https://<tenant>.sharepoint.com/sites/PnPProvisioningSource/"

[System.Reflection.Assembly]::LoadFile("OfficeDevPnP.Core.Tests.dll")

$xmlEncryption = New-Object Of-ficeDevPnP.Core.Tests.Framework.Providers.Extensibility.XMLEncryptionTemplateProviderExtension
$x509Certificate = Get-ChildItem -Path cert:CurrentUserMy | Where-Object{$_.Subject -eq "CN=PnPTestCertificate"}
$xmlEncryption.Initialize($x509Certificate)
Get-SPOProvisioningTemplate -Out ".PnP-XML-Encryption-Demo.pnp" -TemplateProviderExtensions $xmlEncryption

# Apply the ciphered template to another site
Connect-SPOnline "https://<tenant>.sharepoint.com/sites/0fe6e9f7-f6c7-4e6c-b494-c8f5e59a6c86"

Apply-SPOProvisioningTemplate ".PnP-XML-Encryption-Demo.pnp" -TemplateProviderExtensions $xmlEncryption

There are some other new arguments available for the Get-SPOProvisioningTemplate cmdlet, which allow to create more complete template files. Here follows a list of the new arguments:

  • TemplateDisplayName: to set the display name of the template that will be extracted
  • TemplateImagePreviewUrl: to set the URL of the image preview of the template that will be extracted
  • TemplateProperties: to configure custom properties for the template that will be extracted
  • IncludeTermGroupsSecurity: exports the taxonomy term group security settings (manag-ers and contributors)

Wrap up

As you saw by reading this article, the PnP Provisioning Engine is growing fast and can satisfy most of the requirements of real business projects. The latest release of the schema, together with the new capabilities introduced at the engine level, and at the PnP PowerShell level will allow in the very near future to create and share professional and business-ready templates using the PnP Partner Pack for the benefit of the whole community. You can though already use all of these new capabilities in your own customizations just by updating to use the latest PnP Sites Core Nuget Package or the latest version of PnP PowerShell.

“Sharing is caring”


Author – Paolo Pialorsi, Senior Consultant, Piasys.com, PnP Core team member

Editor – Vesa Juvonen, Senior Program Manager, SharePoint, Microsoft

12th of August 2016