Connect to Databricks Using Service Principal

See the following steps to configure a Service Principal for automated access to Azure Databricks, specifically to read billing and query history data from the system catalog. The following setup enables cost intelligence and usage analytics without requiring individual user credentials.

Prerequisites

Ensure you have the following:

  • Azure subscription access with permissions to create App Registrations.

  • Databricks Account Admin or Metastore Admin privileges. These privileges are required for system catalog grants.

  • Access to the target Databricks workspace and SQL warehouse.

Create a Service Principal in the Azure Portal

A Service Principal acts as a non-interactive service account for applications to authenticate Azure Databricks.

  1. Go to the Azure Portal > Azure Active Directory > App Registrations > New Registration.

  2. Provide a descriptive name (for example, Sentinel-Cost-Intelligence-SP) and complete the registration.

  3. Generate the client secret:

    1. Go to Certificates & Secrets and click New Client Secret.

    2. Set the expiration period.

    3. Copy the secret value immediately.

      This value only displays once.

  4. Add a Databricks API permission:

    1. Go to API Permissions > Add a permission > APIs my organization uses.

    2. Find and choose Azure Databricks.

    3. Select the user_impersonation delegated permission.

    4. Click Grant admin consent to activate the permission.

  5. Note the following credentials from the Overview page:

    • Application (client) ID

    • Directory (tenant) ID

Configure the Service Principal in Databricks

The Service Principal must be registered in Databricks and granted warehouse access.

  1. Add the service principal to workspace:

    1. In Databricks, go to Workspace > Settings > Identity & Access > Service Principals.

    2. Click Add Service Principal.

    3. Service principals can be created in Azure Databricks or linked from an existing Microsoft Entra ID. For Microsoft Entra ID managed service principal, enter the Application (client) ID from the Azure Portal.

    4. After creating the service principal, click the Secret tab and click Generate secret.

    5. Choose a secret lifetime and click Generate.

    6. Copy and save the client ID and secret for future use.

  2. Grant warehouse access:

    1. Go to SQL Warehouses and choose the target warehouse.

    2. Click Permissions and Add.

    3. Find the service principal and grant the Can Use permission.

Grant System Catalog Permissions

System catalog tables (such as system.billing.*, system.query.*) are managed at the account or metastore level and require Account Admin or Metastore Admin privileges to grant access.

Understand Permission Levels

Databricks has the following three permission scopes:

  • Workspace Admin: Can manage workspace users and settings, but cannot grant system catalog permissions.

  • Metastore Admin: Can grant permissions on Unity Catalog metastores and system catalog.

  • Account Admin: Full administrative access across all workspaces and account-leve features.

The Azure subscription Owner role does not automatically grant the Databricks Account Admin privileges. They are separate permission systems.

Required Grants

Grant permissions using the command line or from workspace:

  • Execute the following grant permissions as Account Admin or Metastore Admin:

    Copy 
    GRANT USE CATALOG ON CATALOG system TO `<client-id>`;
    GRANT USE SCHEMA ON SCHEMA system.billing TO `<client-id>`;
    GRANT SELECT ON TABLE system.billing.usage TO `<client-id>`;
    GRANT USE SCHEMA ON SCHEMA system.query TO `<client-id>`;
    GRANT SELECT ON TABLE system.query.history TO `<client-id>`;
    GRANT USE SCHEMA ON SCHEMA system.access TO `<client-id>`;
    GRANT SELECT ON TABLE system.access.workspaces_latest TO `<client-id>`;

  • Grant permissions from workspace: Go to Workspace > Catalog > Select a Catalog > Permissions > Grant and select the check box next to following privileges:

    • USE CATALOG

    • USE SCHEMA

    • APPLY TAG

    • BROWSE

    • READ VOLUME

    • SELECT

    • MODIFY

    • WRITE VOLUME

    • CREATE FUNCTION

    • CREATE MATERIALIZED VIEW

    • CREATE MODEL

    • CREATE SCHEMA

    • CREATE TABLE

    • CREATE VOLUME

Use M2M Authentication in Strategy

  1. In the Workstation Navigation pane, click , next to Datasets.

  2. Select Data Import Cube and click OK.

  3. Click , next to Data Catalog.

  4. Find and select the Databricks connector from the data source list.

  5. Type a Name and click Add New Database Connection.

  6. In the Add New Database Connection dialog, type a Host Name and Http Path.

  7. Expand the Authentication Mode drop-down list and choose Service Principal.

  8. Click Add New Database Login and paste the Client ID and Client Secret you copied above.

  9. Click Save.

  10. Click Save.

  11. Complete any additional required fields.

  12. Click Save.

  13. Import data using the data source.

Troubleshooting

Troubleshooting Account Admin Access

A common issue when setting up Service Principals is the inability to access Databricks Account Console at accounts.azuredatabricks.net. If you visit this URL and receive a 404 error or redirected to workspace selection, it indicates you lack Account Admin privileges.

Understand the Azure vs Databricks Role Separation

Azure and Databricks maintain independent permission systems. Being an Azure subscription Owner grants control over Azure resources (such as creating workspaces, managing billing, configuring networking) but does not automatically provide Databricks Account Admin privileges. Similarly, a Databricks Account Admin may have no Azure subscription access.

Scope Role Capabilities Location
Azure

Subscription Owner

 Manage Azure resources, billing, create/delete workspaces Azure > Subscriptions > IAM

Databricks

Account Admin

Manage Unity Catalog, system catalog permissions, account-level service principals

accounts.azuredatabricks.net > User Management

Why the error occurs: When you deploy a Databricks workspace through the Azure Portal (workspace-centric deployment), the initial Databricks Account Admin is typically whoever first activated the Databricks in the subscription or the service account used during deployment. This person may be different from the current Azure subscription owner.