Summary:

Vertex AI Data Store indexing hangs on Document ingestion is working in progress. Document parsing and indexing will start later. for each document.

Details:

I use Airbyte to put Confluence pages into BigQuery. Then I use Vertex AI Data Store to index these documents - I have ~1300 documents imported into it and I see Document ingestion is working in progress. Document parsing and indexing will start later. next to each document in the status column.

The documents are imported into data store (when I click View document button I see the data), but the status is as described above. When I use Vertex AI search with Grounding, this data store is not giving any meaningful results.

Surprisingly that has worked for me well on another (private) account a month back, but here on another account (belonging to my company's anization) the indexing doesn't work. I have tried creating the data store in various regions, using periodical sync or one-off sync, using different schemas etc. It always ends up like this.

I confirmed that I can perform billable actions btw. When I create Data Store based on Cloud Storage bucket, it works fine, so it seems it's related to BigQuery only.

Screenshots

Today's import:

Import details:

Confirmed I have not hit the Discovery Engine limits:

Summary:

Vertex AI Data Store indexing hangs on Document ingestion is working in progress. Document parsing and indexing will start later. for each document.

Details:

I use Airbyte to put Confluence pages into BigQuery. Then I use Vertex AI Data Store to index these documents - I have ~1300 documents imported into it and I see Document ingestion is working in progress. Document parsing and indexing will start later. next to each document in the status column.

The documents are imported into data store (when I click View document button I see the data), but the status is as described above. When I use Vertex AI search with Grounding, this data store is not giving any meaningful results.

Surprisingly that has worked for me well on another (private) account a month back, but here on another account (belonging to my company's anization) the indexing doesn't work. I have tried creating the data store in various regions, using periodical sync or one-off sync, using different schemas etc. It always ends up like this.

I confirmed that I can perform billable actions btw. When I create Data Store based on Cloud Storage bucket, it works fine, so it seems it's related to BigQuery only.

Screenshots

Today's import:

Import details:

Confirmed I have not hit the Discovery Engine limits:

• google-cloud-platform
• google-cloud-vertex-ai
• vertex-ai-pipeline

Share Improve this question Follow edited Mar 23 at 10:29 sharq asked Mar 23 at 10:16 sharqsharq 15711 silver badge44 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Solved!

Terrible DevEx on Google's side on this one.

I needed to create a Vertex AI app that would actually use this data store before it started indexing the data. The fact that I used this data store in chat for grounding and for custom Cloud Run function did not seem to matter to the data store.

Since it was a data store with multiple entities I had to create a Search type of app.