Databricks SDKs & Libraries: Python, Java, Scala, R (2026)

SDKs overview

The Databricks Developer Tools encompass a range of SDKs and libraries designed to allow programmatic interaction with the Databricks Lakehouse Platform. These tools support common development workflows, including data engineering, machine learning, and data analytics. Developers can use these SDKs to automate tasks, manage resources, and deploy applications across various cloud environments where Databricks is supported, such as AWS, Azure, and Google Cloud.

Key functionalities provided by the SDKs include:

• Workspace Management: Programmatically create, update, and delete notebooks, jobs, and other workspace assets.
• Data Management: Interact with Delta Lake tables, including reading, writing, and managing schema.
• Job Orchestration: Define, schedule, and monitor Databricks jobs for automated data processing and ML workloads.
• MLflow Integration: Track machine learning experiments, package models, and deploy them for inference.
• Cluster Management: Configure and manage Databricks compute clusters.

The Databricks SDKs are primarily available for widely used programming languages, aligning with the platform's support for notebook-based development and integration with existing data science and engineering toolchains.

Official SDKs by language

Databricks provides official SDKs for Python, Java, Scala, and R, enabling developers to interact with the platform using their preferred language. These SDKs are maintained by Databricks and offer comprehensive functionality for managing resources and executing operations on the Lakehouse Platform.

For detailed information on each SDK, including API references and usage examples, refer to the Databricks developer documentation.

Installation

Installing Databricks SDKs typically involves using standard package managers for each respective language. Authentication methods for the SDKs generally include Databricks personal access tokens, Azure Active Directory tokens, or service principal credentials.

Python SDK

The Databricks Python SDK can be installed using pip. It is recommended to use a virtual environment to manage dependencies.

pip install databricks-sdk

After installation, you can configure authentication. A common method is to set environment variables or use a .databrickscfg file. For example, using environment variables:

export DATABRICKS_HOST=https://<your-workspace-url>
export DATABRICKS_TOKEN=<your-databricks-personal-access-token>

Alternatively, the Databricks configuration file (~/.databrickscfg) allows you to define multiple profiles:

[DEFAULT]
host = https://<your-workspace-url>
token = <your-databricks-personal-access-token>

Java/Scala SDK

For Java and Scala projects, the SDKs are distributed via Maven Central. You would typically add the dependency to your project's build file.

Maven (pom.xml):

<dependency>
    <groupId>com.databricks</groupId>
    <artifactId>databricks-sdk-java</artifactId>
    <version>0.2.0</version> <!-- Check for the latest version -->
</dependency>

Gradle (build.gradle):

implementation 'com.databricks:databricks-sdk-java:0.2.0' // Check for the latest version

SBT (build.sbt) for Scala:

libraryDependencies += "com.databricks" %% "databricks-sdk-scala" % "0.2.0" // Check for the latest version

Authentication typically uses the same host and token mechanisms as the Python SDK, often configured through environment variables or system properties in Java/Scala applications.

R SDK

The Databricks R SDK can be installed from CRAN or GitHub:

install.packages("databricks")
# Or from GitHub for the development version:
# devtools::install_github("databricks/databricks-sdk-r")

Authentication typically involves setting environment variables for DATABRICKS_HOST and DATABRICKS_TOKEN before initializing the client.

Quickstart example

This Python example demonstrates how to use the Databricks Python SDK to list all notebooks in a specified Databricks workspace. This requires configuring your Databricks host and a personal access token as environment variables or in a .databrickscfg file.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import workspace

# Initialize the WorkspaceClient. It will automatically pick up credentials
# from environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN)
# or from ~/.databrickscfg.
# For more explicit configuration, you can pass host and token directly:
# w = WorkspaceClient(host="https://your-workspace.cloud.databricks.com", token="dapi...")
w = WorkspaceClient()

try:
    # List all notebooks and folders in the workspace root path "/Users"
    # Adjust the path as needed, e.g., "/Shared" or a specific user's folder
    path_to_list = "/Users"
    print(f"Listing objects in {path_to_list}:")

    # The list_objects method returns an iterator of ObjectInfo objects
    objects = w.workspace.list_objects(path=path_to_list, recursive=True)

    found_objects = False
    for obj in objects:
        found_objects = True
        object_type = obj.object_type.value if isinstance(obj.object_type, workspace.ObjectType) else obj.object_type
        print(f"  Type: {object_type}, Path: {obj.path}")

    if not found_objects:
        print(f"No objects found in {path_to_list} or path does not exist.")

except Exception as e:
    print(f"An error occurred: {e}")
    print("Please ensure DATABRICKS_HOST and DATABRICKS_TOKEN are correctly configured.")
    print("You can generate a personal access token from your Databricks workspace user settings.")

To run this example:

1. Ensure the databricks-sdk is installed.
2. Set your DATABRICKS_HOST and DATABRICKS_TOKEN environment variables.
3. Execute the Python script.

This script connects to your Databricks workspace and prints a list of objects (notebooks, folders, etc.) found within the specified path. This demonstrates a basic interaction with the Databricks Workspace API using the Python SDK.

Community libraries

Beyond the official SDKs, the Databricks ecosystem benefits from a range of community-driven libraries and tools. These often extend functionality, provide specialized integrations, or offer alternative interfaces for specific use cases.

• Delta Lake Connectors: While Delta Lake is a core Databricks technology, community connectors exist for various data processing engines and programming languages, enabling broader integration beyond the Databricks platform. For instance, there are Python APIs for Delta Lake that can be used independently.
• MLflow Integrations: MLflow, an open-source platform for the machine learning lifecycle, is deeply integrated with Databricks. Its community-driven extensions and plugins allow for integration with diverse ML frameworks and deployment targets, such as MLflow tools for LLMs.
• Apache Spark Libraries: As Databricks is built on Apache Spark, any Spark-compatible library can generally be used within the Databricks environment. This includes a vast array of libraries for data manipulation, machine learning, and graph processing, often available for Python (PySpark), Scala, Java, and R.
• Data Connectors: Various community libraries facilitate connections from Databricks to external data sources and sinks, including databases, message queues, and other cloud services. These often leverage open standards for data interchange and connectivity.
• DevOps Tools: Community contributions also include tools for enhancing CI/CD pipelines and DevOps practices with Databricks, such as Terraform providers or GitHub Actions for deploying Databricks resources. The Databricks Terraform provider is a notable example.

Developers are encouraged to explore the broader Apache Spark and MLflow ecosystems for additional libraries that can complement their Databricks workflows. Community forums and GitHub repositories are common places to discover and contribute to these projects.