https:\/\/learn.microsoft.com\/en-us\/azure\/search\/search-indexer-overview<\/a><\/p>\nDeep Dive with Azure Function Approach<\/h2>\nOverview<\/h3>\n
Azure Functions<\/strong> provide a more powerful and customizable alternative in comparison to pre-built Indexers.\nWith Azure Functions, you can control how data is processed, apply custom logic, and ensure that the ingestion meets your specific needs.\nIt is easier to integrate any specific log\/data analytics tools in comparison to pre-build indexers.<\/p>\nArchitecture<\/h3>\n\n- Azure Blob Storage:<\/strong> Stores the data (e.g., PDFs).<\/li>\n
- Azure Function:<\/strong> Acts as a trigger(add new\/ or change file, metadata change) that processes the data and sends it to Azure AI Search.<\/li>\n
- Azure AI Search:<\/strong> Indexes the processed data, making it searchable.<\/li>\n<\/ol>\n
This architecture offers flexibility and scalability, especially when dealing with complex data structures or custom ingestion logic.<\/p>\n
![\"Architecture](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/02\/function-ingestion.png\")
<\/p>\n
Data Ingestion Pipeline<\/h3>\n
Let\u2019s build a data ingestion pipeline<\/strong> using Azure Functions. For this example, we\u2019ll demonstrate ingesting PDF documents<\/strong> from Azure Blob Storage into Azure AI Search using Python<\/strong>.<\/p>\nPrepare folder structure<\/h3>\n
Here\u2019s a breakdown of the folder structure for your Azure Function setup:<\/p>\n
![\"Common](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/02\/common-function-for-ingestion.png\")
<\/p>\n
\ncomponents\/<\/code><\/strong>: This folder contains all ingestion-related components, such as modules or classes, that your Azure Function will use.<\/li>\nfunction_app.py<\/code><\/strong>: This is the main entry point for the Azure Function. It contains the logic that triggers the ingestion process and orchestrates data flow from Blob Storage to Azure AI Search.<\/li>\nhost.json<\/code><\/strong>: This file defines the configuration settings for the Azure Function. You can customize it based on your business requirements.\n\nwatchDirectories<\/code><\/strong>: Specifies which directories should be monitored for changes (in this case, the components<\/code> folder).<\/li>\nlogLevel<\/code><\/strong>: Defines the logging level, so you can control the verbosity of logs. Here, all logs are set to Information<\/code> level, except for Azure.Core<\/code>, which is set to Error<\/code>.<\/li>\napplicationInsights<\/code><\/strong>: Enables logging for Application Insights, with sampling settings to exclude certain log types.<\/li>\nextensionBundle<\/code><\/strong>: Configures the extension bundle for the Azure Function runtime.<\/li>\n<\/ul>\nHere\u2019s a sample configuration:<\/p>\n
{\r\n\"version\": \"2.0\",\r\n\"watchDirectories\": [\r\n \"components\"\r\n],\r\n\"logging\": {\r\n \"logLevel\": {\r\n \"default\": \"Information\",\r\n \"Azure.Core\": \"Error\"\r\n },\r\n \"applicationInsights\": {\r\n \"samplingSettings\": {\r\n \"isEnabled\": true,\r\n \"excludedTypes\": \"Request\"\r\n }\r\n }\r\n},\r\n\"extensionBundle\": {\r\n \"id\": \"Microsoft.Azure.Functions.ExtensionBundle\",\r\n \"version\": \"[4.*, 5.0.0)\"\r\n}\r\n}<\/code><\/pre>\n<\/li>\ntests\/<\/code><\/strong>: This folder contains automated tests for your function, ensuring that your ingestion logic works as expected.<\/li>\nrequirements.txt<\/code><\/strong>: Lists all the dependencies (e.g., libraries and packages) required by the Azure Function. The function runtime uses this file to install the necessary dependencies during deployment.<\/li>\n<\/ul>\nRequired Environment Variables Setup<\/h3>\n
To ensure your ingestion pipeline functions properly, you need to configure several environment variables both locally (in a .env<\/code> file) and in the cloud (for the Azure Function). These variables provide access to essential services like Azure Blob Storage and Azure AI Search.<\/p>\nThe following environment variables are required:<\/p>\n
\nAPPLICATION_INSIGHTS_CONNECTION_STRING<\/code><\/strong>: Connection string for sending telemetry data to Application Insights.<\/li>\nAZURE_STORAGE_CONNECTION_STRING<\/code><\/strong>: Connection string to access Azure Blob Storage.<\/li>\nAZURE_SEARCH_SERVICE_NAME<\/code><\/strong>: The name of the Azure AI Search service.<\/li>\nAZURE_SEARCH_ADMIN_KEY<\/code><\/strong>: Admin key for managing the Azure Search service.<\/li>\nAZURE_SEARCH_SERVICE_ENDPOINT<\/code><\/strong>: The endpoint URL of the Azure AI Search service.<\/li>\nAZURE_SEARCH_INDEX_NAME<\/code><\/strong>: The name of the search index where the data will be ingested.<\/li>\nBLOB_STORAGE_DATA_CONNECTION_STRING<\/code><\/strong>: Connection string to the Azure Blob Storage that contains the data to be ingested.<\/li>\nBLOB_STORAGE_DATA_CONTAINER_NAME<\/code><\/strong>: The name of the Blob Storage container where the documents (e.g., PDFs) are stored.<\/li>\nAZURE_OPENAI_API_VERSION<\/code><\/strong>: The version of the Azure OpenAI API being used.<\/li>\nAZURE_OPENAI_ENDPOINT<\/code><\/strong>: The endpoint URL for the Azure OpenAI service.<\/li>\nAZURE_OPENAI_KEY<\/code><\/strong>: The API key for accessing the Azure OpenAI service.<\/li>\n<\/ul>\nThese environment variables will be accessed within the Azure Function to automate the connection and interaction with the respective services.<\/p>\n
Step-by-Step Guide to Azure Function Implementation<\/h2>\nStep 1: Setting Up the Azure Function in Azure Cloud<\/h3>\n
![\"Create](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/02\/create-function-app-in-azure.png\")
<\/p>\n
Step 2: Define the Context for Your Function App<\/h3>\n
At this stage, we created an independent PDFIngestionsService<\/code>, which serves as a dedicated service for handling PDF ingestion. This modular approach allows for easy future extension to support other file types, such as text, Excel, HTML, and more.<\/p>\nBy structuring the service in this way, you can easily add additional ingestion logic for other formats, ensuring flexibility and scalability in your data ingestion pipeline.<\/p>\n
import logging\r\n import os\r\n import pathlib\r\n\r\n import azure.functions as func\r\n\r\n DOCUMENT_CONTAINER_NAME = os.environ[\"BLOB_STORAGE_CONTAINER_NAME\"]\r\n app = func.FunctionApp()\r\n\r\n @app.blob_trigger(arg_name=\"blob\", path=f\"{DOCUMENT_CONTAINER_NAME}\", connection=\"BLOB_STORAGE_DOCS_CONNECTION_STRING\")\r\n async def handle_blob_change_function(blob: func.InputStream):\r\n # Load our code during execution to prevent host startup failures\r\n from components import logging\r\n from components.ingest_service import PDFIngestionsService\r\n\r\n configure_logging()\r\n logger = logging.getLogger(__name__)\r\n data_file_name = blob.name.lower()\r\n ingestion_service = None\r\n\r\n try:\r\n # Determine the file type and select the appropriate ingestion service\r\n file_extension = pathlib.Path(data_file_name).suffix[1:].lower()\r\n if file_extension in [\"pdf\"]:\r\n ingestion_service = PDFIngestionsService()\r\n else:\r\n raise ValueError(\"Unsupported file type\")\r\n\r\n # Ingest the blob using the selected service\r\n if ingestion_service:\r\n await ingestion_service.ingest(blob)\r\n\r\n except Exception as e:\r\n logger.exception(f\"An error occurred while processing data {data_file_name}: {e}\")\r\n raise<\/code><\/pre>\nStep 3: Create a Helper for Setting Up Application Insights Logging<\/h3>\n
To ensure proper logging and monitoring within your Azure Function app, we created a helper script that configures logging and integrates with Azure Application Insights<\/strong>. This allows you to capture logs, errors, and telemetry data in Application Insights for better observability and troubleshooting.<\/p>\nSuggested path:<\/strong> .\/components\/logging.py<\/code><\/p>\n import logging\r\n import os\r\n from opencensus.ext.azure.log_exporter import AzureLogHandler\r\n\r\n def configure_logging():\r\n # Configure the logger\r\n LOG_LEVEL = os.getenv(\"LOG_LEVEL\", \"INFO\").upper()\r\n LOG_FORMAT = os.getenv(\"LOG_FORMAT\", \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\")\r\n logging.basicConfig(level=LOG_LEVEL, format=LOG_FORMAT)\r\n\r\n # Silence noisy loggers\r\n logging.getLogger(\"azure.core.pipeline.policies.http_logging_policy\").setLevel(logging.WARNING)\r\n logging.getLogger(\"httpx\").setLevel(logging.WARNING)\r\n\r\n # Add Azure Application Insights handler to root logger\r\n root_logger = logging.getLogger()\r\n appinsights_connection_string = os.getenv(\"APPLICATION_INSIGHTS_CONNECTION_STRING\")\r\n if appinsights_connection_string and not any(isinstance(handler, AzureLogHandler) for handler in root_logger.handlers):\r\n azure_handler = AzureLogHandler(connection_string=appinsights_connection_string)\r\n root_logger.addHandler(azure_handler)<\/code><\/pre>\nStep 4: Configure Your File Ingestion Service Interface and Its Implementation<\/h3>\n
Now is the ideal time to set up the interface for your file ingestion service and implement it. This will allow you to streamline how different file types are processed, starting with PDFs, but leaving room for future extensions (e.g., text, Excel, HTML).<\/p>\n
Suggested folder structure for components and PDF ingestion service:<\/strong><\/p>\n![\"Common](\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2025\/02\/azure-function-components.png\")
<\/p>\n
Ingestion service interface (ingestion_service_interface.py<\/code>):<\/strong><\/p>\nIn this interface, we define the base structure for any ingestion service, enabling flexibility for handling different file types. The FileProcessor<\/code> class is also defined to process file streams and handle temporary file storage.<\/p>\nimport io\r\nimport logging\r\nimport os\r\nimport tempfile\r\nfrom abc import ABC, abstractmethod\r\n\r\nimport azure.functions as func\r\n\r\nclass IngestionServiceInterface(ABC):\r\n @abstractmethod\r\n async def ingest(blob: func.InputStream):\r\n pass\r\n\r\nclass FileProcessor(ABC):\r\n async def process_stream(self, blob: func.InputStream) -> str:\r\n \"\"\"\r\n Process a file stream by saving it to a temporary file and returning the file path.\r\n \"\"\"\r\n logger = logging.getLogger(__name__)\r\n\r\n try:\r\n # Extract the original filename from the blob metadata\r\n original_filename = os.path.basename(blob.name)\r\n input_stream = io.BytesIO(blob.read())\r\n\r\n # Read data from the stream\r\n file_data = input_stream.read()\r\n\r\n # Create a temporary file with the original filename and write the data to it\r\n temp_dir = tempfile.gettempdir()\r\n temporary_file_path = os.path.join(temp_dir, original_filename)\r\n\r\n with open(temporary_file_path, \"wb\") as temporary_file:\r\n temporary_file.write(file_data)\r\n\r\n # Return the path to the temporary file\r\n return temporary_file_path\r\n except Exception as error:\r\n logger.error(f\"An error occurred in process_stream: {str(error)}\", exc_info=True)\r\n raise\r\n\r\n @abstractmethod\r\n async def process_file(self, *, file_path: str, index_name: str):\r\n \"\"\"\r\n Process a file path by passing it to process_input.\r\n \"\"\"\r\n pass<\/code><\/pre>\nIngestionServiceInterface<\/strong>: Defines an abstract interface that any ingestion service (e.g., PDF, Excel) must implement. This provides flexibility for adding new file types in the future.<\/p>\nFileProcessor<\/strong>: Provides methods for processing file streams and saving them to a temporary directory. It also abstracts away the logic for file path handling, which will be reused in different implementations.<\/p>\nPDF ingestion service (.\/components\/PDFs\/ingest_service.py<\/code>):<\/strong><\/p>\nThis is a concrete implementation of the IngestionServiceInterface<\/code> for processing PDF files. It uses the PDFFileProcessor<\/code> to handle the file processing and ingestion.<\/p>\nStep 5: Next Steps: Logging and Local Deployment<\/h3>\nimport logging\r\n\r\nimport azure.functions as func\r\nfrom components.ingestion_service_interface import IngestionServiceInterface\r\nfrom components.PDF.build_pdf_index import PDFFileProcessor\r\n\r\nlogger = logging.getLogger(__name__)\r\n\r\nclass PDFIngestionsService(IngestionServiceInterface):\r\n async def ingest(self, myblob: func.InputStream):\r\n blob_name = myblob.name.lower()\r\n\r\n try:\r\n # Process the blob stream\r\n await self.process_blob(myblob)\r\n except Exception as e:\r\n logger.error(f\"Error ingesting PDF file '{blob_name}': {e}\", exc_info=True)\r\n raise\r\n\r\n async def process_blob(self, blob: func.InputStream):\r\n fileProcessor = PDFFileProcessor()\r\n\r\n try:\r\n file_path = await fileProcessor.process_stream(blob)\r\n await fileProcessor.process_file(file_path=file_path, index_name=\"ada002-sample-index\")\r\n except Exception as e:\r\n logger.error(f\"Error during data file processing: {e}\", exc_info=True)\r\n raise<\/code><\/pre>\nPDFIngestionsService<\/strong>: This class implements the IngestionServiceInterface<\/code> for PDF files. It handles blob ingestion and error logging.<\/p>\nprocess_blob<\/strong>: This method calls the PDFFileProcessor<\/code> to process the file stream and then processes the file using a specified index in Azure AI Search.<\/p>\nAt this point, the main focus is setting up logging<\/strong> capabilities within the process_blob()<\/code> method. This ensures that errors are properly logged during ingestion, making it easier to debug issues during file processing.<\/p>\nTo deploy<\/strong>, you can use the VSCode Azure Functions plugin<\/strong> to deploy the function from your local machine. This plugin provides a simple interface for deploying directly to Azure without needing additional scripts. Follow these steps:<\/p>\n\n- Install the Azure Functions Extension<\/strong> for VSCode.<\/li>\n
- Open the project in VSCode and click on the Azure Functions icon in the sidebar.<\/li>\n
- Log in to your Azure account if needed.<\/li>\n
- Right-click on your Azure Function in the Functions Explorer and select Deploy to Function App<\/strong>.<\/li>\n
- Choose your subscription and an existing or new function app for deployment.<\/li>\n<\/ol>\n
Once deployed, you can monitor the logs via Azure Portal or within VSCode’s Output window to ensure the function behaves as expected.<\/p>\n
An automated deployment script<\/strong> will be provided later in this guide to streamline the deployment process for CI\/CD pipelines.<\/p>\nStep 6: Deployment to Cloud<\/h3>\n
There are many ways to automate the deployment of your Azure Function app, and in this article, I\u2019ll demonstrate a sample deployment process using GitHub Actions<\/strong>.<\/p>\nGitHub Actions allows you to automate the deployment pipeline, ensuring your Azure Function app is continuously delivered to the cloud whenever changes are made.<\/p>\n
Below is a sample GitHub Actions workflow for deploying your function app:<\/p>\n
# Documentation for the Azure Web Apps Deploy action: https:\/\/github.com\/azure\/functions-action\r\n# Continuous delivery using GitHub Actions: https:\/\/learn.microsoft.com\/en-us\/azure\/azure-functions\/functions-how-to-github-actions?tabs=linux%2Cpython&pivots=method-manual\r\n\r\nname: Deploy document blob trigger\r\n\r\non:\r\n workflow_dispatch: # Allows triggering the workflow manually\r\n\r\nenv:\r\n PYTHON_VERSION: \"3.11\" # Set the Python version for the function app\r\n FUNCTION_APP_DIR: \"data-ingestion-function\" # Directory where the function app is located\r\n FUNCTION_ZIP_NAME: \"data-ingestion-function.zip\" # Name of the zip package for deployment\r\n\r\njobs:\r\n deploy:\r\n runs-on: ubuntu-latest\r\n\r\n steps:\r\n - name: Checkout repository\r\n uses: actions\/checkout@v4\r\n\r\n - name: Setup Python ${{ env.PYTHON_VERSION }} environment\r\n uses: actions\/setup-python@v5\r\n with:\r\n python-version: ${{ env.PYTHON_VERSION }}\r\n\r\n - name: Install project dependencies\r\n working-directory: .\/${{ env.FUNCTION_APP_DIR }}\r\n run: |\r\n python -m pip install --upgrade pip\r\n pip install -r requirements.txt --target=\".python_packages\/lib\/site-packages\"\r\n\r\n - name: Az CLI login\r\n uses: azure\/login@v2\r\n with:\r\n creds: ${{ secrets.AZURE_CREDENTIALS }}\r\n\r\n # Make environment variables dynamically or statically (Skipped in this example)\r\n # INTENTIONALLY SKIPPED STEP\r\n\r\n - name: AzFunctions deployment\r\n uses: Azure\/functions-action@v1\r\n with:\r\n app-name: ${{ env.FUNCTION_APP_DIR }} # Azure Function App name\r\n package: ${{ env.FUNCTION_APP_DIR }} # Directory containing the function code\r\n scm-do-build-during-deployment: true # Enable build on deployment\r\n enable-oryx-build: true # Enable Oryx build engine for deployment\r\n\r\n - name: Set function configuration\r\n run: |\r\n az functionapp config appsettings set \\\r\n --name \"${{ env.FUNCTION_APP_DIR }}\" \\\r\n --resource-group \"SOME_RESOURCE_GROUP\" \\\r\n --settings \\\r\n \"APPLICATION_INSIGHTS_CONNECTION_STRING=SOME\" \\\r\n \"AZURE_OPENAI_API_VERSION=SOME\" \\\r\n \"AZURE_OPENAI_ENDPOINT=SOME\" \\\r\n \"AZURE_OPENAI_KEY=SOME\" \\\r\n \"AZURE_SEARCH_ADMIN_KEY=SOME\" \\\r\n \"AZURE_SEARCH_SERVICE_ENDPOINT=SOME\" \\\r\n \"BLOB_STORAGE_DOCS_CONNECTION_STRING=SOME\" \\\r\n \"BLOB_STORAGE_DATA_CONTAINER_NAME=SOME\"<\/code><\/pre>\n\n- Checkout the repository<\/strong>: The workflow starts by checking out the repository to access the code.<\/li>\n
- Set up Python<\/strong>: The specified Python version (
3.11<\/code> in this case) is set up using setup-python<\/code>.<\/li>\n- Install project dependencies<\/strong>: The workflow installs the required dependencies listed in the
requirements.txt<\/code> file into a .python_packages<\/code> directory to be packaged with the function.<\/li>\n- Login to Azure CLI<\/strong>: The
azure\/login@v2<\/code> action logs in to Azure using credentials stored in GitHub secrets (AZURE_CREDENTIALS<\/code>).<\/li>\n- Azure Functions deployment<\/strong>: The
Azure\/functions-action@v1<\/code> action deploys the function app to Azure. The workflow enables build during deployment by setting scm-do-build-during-deployment<\/code> to true<\/code>.<\/li>\n- Set environment variables<\/strong>: The workflow configures environment variables for the function app using
az functionapp config appsettings set<\/code>.<\/li>\n<\/ul>\n
\nYou can manage the environment variables either dynamically or statically, so this part was skipped:<\/p>\n
\n- Dynamic setup<\/strong>: Use Azure Key Vault to securely pull secrets at runtime.<\/li>\n
- Static setup<\/strong>: Set the environment variables manually within the workflow.<\/li>\n<\/ul>\n
For simplicity, this example skips the environment setup step. You can choose the method that best fits your workflow.<\/p>\n
\nAn actual deployment for this workflow will:<\/p>\n
\n- Set up the Python environment.<\/li>\n
- Install dependencies.<\/li>\n
- Deploy the function app.<\/li>\n
- Configure the function app\u2019s environment settings.<\/li>\n<\/ul>\n
With this setup, your Azure Function will be automatically deployed whenever you manually trigger the workflow or add it to trigger on code pushes to the repository.<\/p>\n
Step 7: Verify Your Deployed Azure Function<\/h3>\n
Now that your Azure Function is deployed, it’s time to test it to ensure everything is working correctly.<\/p>\n
\n- Upload a Test PDF<\/strong>: Upload a PDF file into the configured Azure Blob Storage container. This should trigger the Azure Function if everything is set up correctly.<\/li>\n
- Check Function Logs<\/strong>\n
\n- Navigate to the Azure Portal<\/strong>.<\/li>\n
- Open your Function App<\/strong> and go to the Logs<\/strong> tab under the Monitoring<\/strong> section.<\/li>\n
- Monitor the logs to verify that the function has been triggered and processed the PDF successfully.<\/li>\n<\/ul>\n<\/li>\n
- Troubleshooting<\/strong>:\n
\n- If you don\u2019t see any logs or if the function isn\u2019t triggered, it might indicate an issue with the deployment or configuration.<\/li>\n
- Double-check your deployed files, environment variables, and ensure that the correct blob storage connection string is set.<\/li>\n
- You can also use Application Insights<\/strong> (if configured) to further analyze any errors or performance issues.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
Testing your function with a real PDF upload ensures that the entire pipeline\u2014from blob storage ingestion to logging\u2014functions as expected.<\/p>\n
Step 8: Building the Index<\/h3>\n
To enable full-text search<\/strong> on the ingested PDF documents, you\u2019ll need to build an index in Azure AI Search<\/strong>. The index defines the structure and schema of the data, making it searchable. This is a critical step in enabling efficient searches through the content of your documents.<\/p>\nWe recommend using Azure Document Intelligence<\/strong> (formerly known as Form Recognizer) to extract and structure the content of your PDF documents, such as the title, author, and text content. Document Intelligence can parse complex document layouts, allowing you to extract key-value pairs, tables, and paragraphs for indexing.<\/p>\n
\nHere\u2019s a sample index structure for PDF documents, which can be customized based on the specific fields you want to extract:<\/p>\n
\n- Id<\/strong>: A unique identifier for each document (used as a primary key).<\/li>\n