Preparing Blobs for Fast Querying

Azure AirMettle Select generates small sidecar files for blobs in your storage accounts. A sidecar must be generated for a blob before it can be queried.

Location

These sidecar files are stored in the same container as the original blob, and the original blob is never modified in any way. These sidecar files are organized in a logical hidden directory, .airmettle/select/meta/, to avoid cluttering your container.

Generating Sidecar Metadata

To give you full control over which blobs are prepared, Azure AirMettle Select provides an API to selectively generate these sidecar files on-the-fly.

A sidecar only needs to be generated once per blob. Any queries to the same blob will reuse the existing sidecar.

You may delete the generated metadata file at any time without affecting the original blob. You can regenerate the metadata later using this API.

Endpoints

POST /prepare

Generate a sidecar for a blob.

The endpoint uses the blob’s URL to identify it.

Example of a blob URL for a blob named data.csv in the container container in the storage account teststore:

https://teststore.blob.core.windows.net/container/data.csv

Example command to generate a sidecar for a blob using its URL:

curl -X POST /prepare \
     -H 'Content-Type: application/json' \
     -d '{
           "blob_url": "URL",
           "subscription_id": "SUBSCRIPTION_ID",
           "api_key": "API_KEY"
         }'

The service accesses the blob with the access key stored in the storage account’s bucket registration (see Managing Your Azure AirMettle Select Resources with the Admin API), so the storage account in blob_url must be registered under the subscription before its blobs can be prepared. Requests for unregistered storage accounts return 403 Forbidden. When you rotate a storage account’s access key, update its bucket registration so prepares keep working. If a subscription has Marketplace contract status, that status must be active before metadata can be generated; inactive Marketplace contracts return 403 Forbidden.

If the sidecar was created successfully, the response will be 201 Created. In some cases, the response may be 202 Accepted, indicating that your request has been accepted for processing but sidecar generation is not yet complete. The initial 202 response includes a JSON document with the following fields:

{
  "workflow_id": "WORKFLOW_ID",
  "status_url": "https://<host>/status/WORKFLOW_ID"
}

The status_url value is the URL for the status endpoint. Calling that endpoint returns 200 OK with a JSON document containing a single state field. The state value will be one of:

• failed

• in_progress

• succeeded

If the provided workflow_id cannot be found, the status endpoint returns 404 Not Found.