I have an azure Index based on azure blob storage container.

Here is part of definition from indexer.

"parameters": {
    "batchSize": null,
    "maxFailedItems": null,
    "maxFailedItemsPerBatch": null,
    "base64EncodeKeys": null,
    "configuration": {
      "parsingMode": "jsonArray"
    }
  },
  "fieldMappings": [],
  "outputFieldMappings": [],
  "cache": null,
  "encryptionKey": null

The blob contains json array and multiple index documents are created for each json record in the blob. Indexer works fine when new documents are added to container. However, we also need to "Remove" indexed documents if the associated blob gets deleted.

Is there any way which can help to auto-delete linked index documents from azure search on deletion of blob? I checked documentation from Microsoft, and there is a possibility to have mapping with "AzureSearch_DocumentKey". but it will only help for updates to the blob, but it will not react to blob deletion.

I have an azure Index based on azure blob storage container.

Here is part of definition from indexer.

"parameters": {
    "batchSize": null,
    "maxFailedItems": null,
    "maxFailedItemsPerBatch": null,
    "base64EncodeKeys": null,
    "configuration": {
      "parsingMode": "jsonArray"
    }
  },
  "fieldMappings": [],
  "outputFieldMappings": [],
  "cache": null,
  "encryptionKey": null

The blob contains json array and multiple index documents are created for each json record in the blob. Indexer works fine when new documents are added to container. However, we also need to "Remove" indexed documents if the associated blob gets deleted.

Is there any way which can help to auto-delete linked index documents from azure search on deletion of blob? I checked documentation from Microsoft, and there is a possibility to have mapping with "AzureSearch_DocumentKey". but it will only help for updates to the blob, but it will not react to blob deletion.

• azure
• azure-blob-storage
• azure-cognitive-search

Share Improve this question Follow asked Mar 7 at 8:49 K DK D 5,97911 gold badge2525 silver badges3636 bronze badges 2

• @K D Could you please share the full index and the indexer of Azure Search? – Sampath Commented Mar 7 at 10:19
• @K D Enable Soft Delete on Blob Storage and configure the indexer with @search.softDeleteColumnName to remove indexed documents when blobs are deleted – Pratik Jadhav Commented Mar 13 at 5:48

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Azure AI Search Does Not Automatically Delete Indexed Documents When a Blob is Deleted

However, the following methods ensure that indexed documents are removed:

• This method only works for Azure Blob Storage (not ADLS Gen2 or Azure Files).

• It requires enabling soft delete on the storage account.

• Azure AI Search detects soft-deleted blobs and removes the corresponding index documents.

Enabling Soft Delete in Data Sources

While creating the connection in Data Sources, make sure to:

• Select the Track deletions checkbox.

• Choose Native blob soft delete for soft deletion.

The following JSON configuration is supported starting from API version 2024-07-01. Ensure you update the version accordingly before applying the changes:

 "dataDeletionDetectionPolicy": {
    "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
    "softDeleteColumnName": "IsDeleted",
    "softDeleteMarkerValue": "true"
  },

You can use the following REST API request to create or update a data source:

PUT https://[service-name].search.windows/datasources/blob-datasource?api-version=2024-07-01
{
    "name": "blob-datasource",
    "type": "azureblob",
    "credentials": { "connectionString": "<your storage connection string>" },
    "container": { "name": "my-container" },
    "dataDeletionDetectionPolicy": {
        "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
        "softDeleteColumnName": "IsDeleted",
        "softDeleteMarkerValue": "true"
    }
}

When a file is deleted in Azure Storage, you need to reset the Indexer and run it to reflect the changes in the search index.