{"id":16155,"date":"2025-04-03T00:00:00","date_gmt":"2025-04-03T07:00:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/ise\/?p=16155"},"modified":"2025-04-03T01:30:02","modified_gmt":"2025-04-03T08:30:02","slug":"data-validations-with-great-expectations-in-ms-fabric","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/data-validations-with-great-expectations-in-ms-fabric\/","title":{"rendered":"Data Validations with Great Expectations in MS Fabric"},"content":{"rendered":"

Introduction<\/h2>\n

Data quality is a critical challenge for companies. Inaccurate, incomplete, or inconsistent data can lead to flawed decision making, operational inefficiency, and loss of trust. Poor data quality affects all aspects of a business.<\/p>\n

Great Expectations (GX) is a robust framework to check data quality and provide concrete, actionable results. Microsoft Fabric is an end-to-end data platform that integrates data engineering, data science, real-time analytics, and business intelligence into a unified environment. GX can be effectively utilized in the Microsoft Fabric environment, providing seamless solution for data validation and management. By validating datasets against defined expectations, companies can proactively identify and resolve issues, ensuring that their data is accurate, reliable, and fit for purpose.<\/p>\n

GX allows us to define test criteria programmatically using its open-source Python library, GX Core, and execute tests to ensure data quality. Also, GX offers GX Cloud, which is a fully managed cloud platform for data quality management.<\/p>\n

In this blog post, we will explore how to integrate GX Core within the Microsoft Fabric environment and use it to validate data programmatically.<\/p>\n

Sample data and quality requirements<\/h2>\n

To illustrate the problem and solution, let’s define the scenario for this post. We are working with the following test dataset:<\/p>\n\n\n\n\n\n\n\n\n\n
ID<\/th>\nName<\/th>\nAge<\/th>\nCity<\/th>\nScore<\/th>\n<\/tr>\n<\/thead>\n
1<\/td>\nAlice<\/td>\n25<\/td>\nNew York<\/td>\n85.5<\/td>\n<\/tr>\n
2<\/td>\nBob<\/td>\n30<\/td>\nLos Angeles<\/td>\n90.2<\/td>\n<\/tr>\n
3<\/td>\nCharlie<\/td>\n35<\/td>\nChicago<\/td>\n78.9<\/td>\n<\/tr>\n
4<\/td>\nDavid<\/td>\n40<\/td>\nHouston<\/td>\n88.3<\/td>\n<\/tr>\n
5<\/td>\nEva<\/td>\n45<\/td>\nPhoenix<\/td>\n92.5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

To validate the quality of this dataset, we define the following requirements:<\/p>\n

    \n
  1. The ID<\/code> column should have unique values.<\/li>\n
  2. The score<\/code> column should contain values greater than 80 (This expectation is intentionally set to fail to demonstrate how issues can be identified).<\/li>\n<\/ol>\n

    We will validate these requirements programmatically in a Microsoft Fabric Notebook using Great Expectations (GX). By integrating GX within this environment, we will demonstrate how to define expectations, apply validations, and analyze the results effectively.<\/p>\n

    Setting up GX in MS Fabric<\/h2>\n

    Create an Environment<\/h3>\n

    To use GX in MS Fabric, we need to add it to an Environment<\/a>. We can include the great_expectations<\/code> library with the desired version under the Public libraries > Public library tab. Once added, we must publish the updated environment to ensure the changes take effect.<\/p>\n

    \"Set<\/p>\n

    Create a Notebook<\/h3>\n

    We are using a Notebook to programmatically validate the data using GX. We create a Notebook, and select the Environment you set up for GX. Now, the Notebook is running on the environment that has GX imported.<\/p>\n

    \"Set<\/p>\n

    Load GX components<\/h2>\n

    Load data<\/h3>\n

    Let\u2019s begin by loading the data we want to validate into Microsoft Fabric. In this example, we upload a test dataset to the Lakehouse<\/a> and then read it into the Notebook<\/a> for processing using PySpark.<\/p>\n

    df = spark.read.table(\"test_dataset\")<\/code><\/pre>\n
    Create GX Context<\/h3>\n
    Next, we need to define a Data source in the GX context<\/strong>. The GX context contains all the necessary information to run GX within our workflow.<\/p>\n
    import great_expectations as gx\r\n\r\ncontext = gx.get_context()<\/code><\/pre>\n
    Create an expectation suite<\/h3>\n
    In GX, the validation criteria we want to test are referred to as expectations<\/strong>. The collection of these expectations is called expectation suite<\/strong>. The expectation suite contains all the expectations we intend to apply to the data. Also, it can be stored in GX context for future reuse and easier maintenance.<\/p>\n
    expectation_suite_name = \"test_data_validation_suite\"\r\ncontext.suites.add(gx.ExpectationSuite(name=expectation_suite_name))\r\nsuite = context.suites.get(name=expectation_suite_name)<\/code><\/pre>\n
    Create Data source and Batch<\/h3>\n
    We need to create a data source, a data asset and a batch definition. These elements reference the target dataset in GX context. In the Validation definition<\/strong>, all the references required to run the validation are included, such as the target dataset and the expectation suite.<\/p>\n
    data_source = context.data_sources.add_spark(name=\"test_data_source\")\r\ndata_asset = data_source.add_dataframe_asset(name=\"test_data\")\r\n\r\nbatch_definition = data_asset.add_batch_definition_whole_dataframe(\"test_batch_definition\")\r\n\r\nvalidation_definition = gx.ValidationDefinition(\r\n    data = batch_definition,\r\n    suite = suite,\r\n    name = \"test_validation_definition\",\r\n)<\/code><\/pre>\n
    Run GX in Fabric<\/h2>\n
    Add new expectations<\/h3>\n
    Now we are ready to start adding expectations. GX provides the pre-defined expectations that we can easily leverage. We can explore the complete list of available expectations in the GX Expectations Gallery<\/a> and find the ones which fit our requirements.<\/p>\n
    As mentioned in the requirements above, we will define two expectations : the ID<\/code> column should have unique values, and the score<\/code> column should be contain values greater than 80.<\/p>\n
    These expectations are defined and added in the expectations suite for validation.<\/p>\n
    unique_id = gx.expectations.ExpectColumnValuesToBeUnique(column=\"ID\")\r\nsuite.add_expectation(unique_id)\r\n\r\nscore_min = gx.expectations.ExpectColumnValuesToBeBetween(column=\"Score\", min_value=80)\r\nsuite.add_expectation(score_min)<\/code><\/pre>\n
    Run the validation<\/h2>\n
    Let’s validate the data(stored in the PySpark DataFrame) using the validation definition. This will apply the expectations from the expectation suite to the dataset, as defined in the validation definition, and generate a validation result.<\/p>\n
    results = validation_definition.run(batch_parameters = {\"dataframe\": df})<\/code><\/pre>\n
    \"Validation<\/p>\n
    Validation results<\/h2>\n
    Let’s take a closer look at the details of the validation results from the previous step. This will help us to understand how the validation works and effectively analyze the results.<\/p>\n
    Success<\/h3>\n
    We can see that the validation failed from the success<\/code> flag at the top of the result. This is expected as we intentionally included false expectations in the suite.<\/p>\n
    {\r\n  \"success\": false,\r\n  ...\r\n}<\/code><\/pre>\n
    Details for each expectation<\/h3>\n
    The detailed results are provided for each expectation. It allows us to see which expectations succeeded or failed individually. This breakdown also helps in understanding how each expectation was evaluated.<\/p>\n
    {\r\n    ...\r\n      \"results\": [\r\n    {\r\n      \"success\": true,\r\n      \"expectation_config\": {\r\n        \"type\": \"expect_column_values_to_be_unique\",\r\n        \"kwargs\": {\r\n          \"batch_id\": \"test_data_source-test_data\",\r\n          \"column\": \"ID\"\r\n        },\r\n        ...\r\n      },\r\n      \"result\": {\r\n        \"element_count\": 5,\r\n        \"unexpected_count\": 0,\r\n        \"unexpected_percent\": 0.0,\r\n        ...\r\n      },\r\n      ...\r\n    },\r\n    {\r\n      \"success\": false,\r\n      \"expectation_config\": {\r\n        \"type\": \"expect_column_values_to_be_between\",\r\n        \"kwargs\": {\r\n          \"batch_id\": \"test_data_source-test_data\",\r\n          \"column\": \"Score\",\r\n          \"min_value\": 80.0\r\n        },\r\n        ...\r\n      },\r\n      \"result\": {\r\n        \"element_count\": 5,\r\n        \"unexpected_count\": 1,\r\n        \"unexpected_percent\": 20.0,\r\n        \"partial_unexpected_list\": [\r\n          78.9000015258789\r\n        ],\r\n        ...\r\n      }\r\n      ...\r\n}<\/code><\/pre>\n
    In this example, for expect_column_values_to_be_unique<\/code>, all the data meets the expectation. We can confirm this by checking success<\/code> flag(true<\/code>), or by looking at the unexpected_count<\/code> and unexpected_percent<\/code>(both 0<\/code>).<\/p>\n
    However, for expect_column_values_to_be_between<\/code>, the result indicates a failure. We can see the failure in the success<\/code> flag(false<\/code>) and an unexpected_count<\/code> of 1, which corresponds to an unexpected_percent<\/code> of 20%. Additionally, we can identify the specific data that caused this failure by checking the partial_unexpected_counts<\/code>.<\/p>\n
    Further notes<\/h2>\n
    Result format<\/h3>\n
    It is also possible to get the primary key of the failed data by setting result_format<\/code> option that GX provides. With the primary key, you can retrieve the entire rows if needed. For more details, please refer to the GX Result Format<\/a>.<\/p>\n
    Custom expectations<\/h3>\n
    GX also provides options to create a custom expectations<\/a>. We can apply conditions to rows in a Batch, or customise an expectation class. This is particularly useful when we need to create tailored expectations for our dataset and reuse them across different validation scenarios.<\/p>\n
    Actions and Checkpoints<\/h3>\n
    We can define actions<\/strong> to be executed based on the validation results. GX offers pre-defined actions we can choose from. For instance, we can update Data Doc with the validation results or send a notification to Slack.<\/p>\n
    To enable actions<\/strong>, we need to create a checkpoint<\/strong>. With checkpoints, we can also run multiple validation definitions. For additional information, check out Create a checkpoint with actions<\/a>.<\/p>\n
    Conclusion<\/h2>\n
    The integration of Great Expectations(GX) within Microsoft Fabric is a powerful solution to validate data programmatically, and offers a seamless way to ensure data quality and consistency across your workflow. GX allows us to easily define, apply, and track expectations, while the validation results provide valuable insights. We can analyze failures in depth, create custom reports for your needs, and take corrective actions as required. This capability enables you to maintain accurate and reliable data, significantly enhancing the efficiency of our data management process within Fabric.<\/p>\n
    References<\/h2>\n