Getting started overview

This guide provides a structured approach to initiating interaction with the Humanitarian Data Exchange (HDX) API. The HDX platform centralizes humanitarian data, offering datasets relevant to global crises, disaster response, and related research. Programmatic access to this data is facilitated through the HDX API, enabling developers and data analysts to integrate humanitarian information into custom applications, dashboards, and analytical workflows.

The process outlined includes creating an HDX account, generating an API key for authentication, and executing an initial API request to confirm connectivity and data retrieval. While many datasets on HDX are publicly accessible without an API key, certain operations, such as accessing specific data or managing user-specific content, may require authenticated requests. The HDX API documentation provides comprehensive details on available endpoints and their respective authentication requirements HDX API documentation page.

To get started, follow these steps:

Step What to do Where
1. Create Account Register for a free HDX user account. HDX user registration page
2. Get API Key Locate or generate your API key in your user profile. HDX user profile settings
3. Make First Request Send a simple API request to a public endpoint using your API key. Using a tool like curl or a programming language like Python.

Create an account and get keys

Access to the Humanitarian Data Exchange API generally requires an HDX user account, particularly for operations that involve data contribution, specific dataset access, or personalized features. While numerous datasets are available for public download without an account, obtaining an API key ensures full programmatic access and helps HDX track API usage for service improvement.

1. Register for an HDX account

  1. Navigate to the HDX user registration page.
  2. Provide the required information, including a username, email address, and password.
  3. Complete any verification steps, such as email confirmation, as prompted by the system.

2. Locate your API key

Once your account is active, your API key can be found within your user profile settings:

  1. Log in to your HDX account.
  2. Click on your username in the top right corner of the page and select "My Profile" or navigate directly to the HDX user profile settings.
  3. Within your profile settings, look for a section labeled "API Key" or "API Token." The key is a unique alphanumeric string.
  4. Copy this key. It will be used to authenticate your API requests. Treat your API key as sensitive information, similar to a password, to prevent unauthorized access to your account or data.

Your first request

To verify your setup and API key, you can make a simple request to a public HDX endpoint. The HDX API is built on CKAN, an open-source data portal platform, and supports standard RESTful API interactions CKAN API documentation. This example demonstrates how to retrieve a list of available datasets.

Example: Listing datasets using curl

The curl command-line tool is a common method for making HTTP requests. Replace YOUR_API_KEY with the actual API key you obtained.

curl -H "Authorization: YOUR_API_KEY" https://data.humdata.org/api/3/action/package_list

This command sends a GET request to the package_list endpoint. The -H "Authorization: YOUR_API_KEY" header includes your API key, which is good practice even for public endpoints as it identifies your requests to the HDX system.

Expected response

A successful response will return a JSON object containing a list of dataset IDs. The structure will resemble the following (truncated for brevity):

{
  "help": "https://data.humdata.org/api/3/action/help_show?name=package_list",
  "success": true,
  "result": [
    "afghanistan-displacement-tracking-matrix-dtm-flow-monitoring-survey-fms-dataset",
    "afghanistan-humanitarian-country-team-hct-2023-humanitarian-response-plan-hrp-dataset",
    "afghanistan-humanitarian-response-plan-2023-people-in-need-and-people-targeted-dataset",
    // ... more dataset IDs
  ]
}

Example: Listing datasets using Python

Python is frequently used for interacting with data APIs due to its extensive libraries. This example utilizes the requests library.

First, ensure you have the requests library installed:

pip install requests

Then, execute the following Python script:

import requests
import json

api_key = "YOUR_API_KEY"  # Replace with your actual API key
base_url = "https://data.humdata.org"

headers = {
    "Authorization": api_key
}

endpoint = f"{base_url}/api/3/action/package_list"

try:
    response = requests.get(endpoint, headers=headers)
    response.raise_for_status()  # Raise an HTTPError for bad responses (4xx or 5xx)
    data = response.json()

    if data["success"]:
        print("Successfully retrieved dataset list:")
        # Print the first 5 dataset names for brevity
        for item in data["result"][:5]:
            print(f"- {item}")
    else:
        print(f"API request failed: {data.get('error', 'Unknown error')}")
	except requests.exceptions.RequestException as e:
    print(f"An error occurred during the request: {e}")
except json.JSONDecodeError:
    print("Failed to decode JSON response.")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

This Python script performs the same action as the curl command, demonstrating how to include the API key in the request headers and process the JSON response.

Common next steps

After successfully making your first API call, consider these next steps to further integrate with the Humanitarian Data Exchange:

  1. Explore the API Documentation: The HDX API documentation is the authoritative source for all available endpoints, parameters, and response structures. It details how to search for datasets, retrieve specific dataset metadata, access resources (actual data files), and perform other operations.
  2. Search for Specific Data: Use the package_search endpoint to find datasets relevant to your needs. You can filter by keywords, tags, organizations, and other metadata. For instance, you might search for datasets related to "Ebola" or "food security."
  3. Download Data Resources: Once you identify a dataset, you can retrieve its associated resources (e.g., CSV, Excel, Shapefile data files). The API allows you to get the download URL for these files, which you can then fetch programmatically.
  4. Integrate with Data Analysis Tools: Connect the HDX API with your preferred data analysis tools or programming environments. Python, R, and JavaScript are commonly used for data manipulation and visualization. Libraries like Pandas in Python can efficiently process the downloaded data. For examples of data manipulation in various languages, consult general programming resources like the MDN Web Docs on Fetch API for JavaScript or Python's requests library documentation.
  5. Contribute Data: If your organization generates humanitarian data, consider contributing it to HDX. The API supports data upload and management functionalities for registered organizations. This helps centralize critical information for the broader humanitarian community.
  6. Monitor for Updates: Some datasets are updated regularly. Implement mechanisms in your application to periodically check for new versions or changes to ensure you are always working with the most current information.

Troubleshooting the first call

Encountering issues during your initial API call is common. Here are some troubleshooting steps:

  • Incorrect API Key: Double-check that you have copied your API key correctly and that there are no leading or trailing spaces. Ensure it's placed correctly in the Authorization header. An invalid key often results in a 403 Forbidden error.
  • Missing Authorization Header: For authenticated endpoints, the Authorization header is mandatory. If you omit it, you will likely receive an authentication error.
  • Network Connectivity: Verify that your machine has an active internet connection and can reach https://data.humdata.org. Proxy settings or firewalls might block access.
  • Incorrect Endpoint URL: Ensure the URL you are calling is correct. Typos in the hostname or path (e.g., /api/3/action/package_list) will result in 404 Not Found errors. Refer to the official HDX API documentation for exact endpoint paths.
  • JSON Parsing Errors: If the API returns a non-JSON response or an improperly formatted JSON, your code might fail to parse it. Inspect the raw response content to identify the issue. This can happen if an error page (HTML) is returned instead of JSON.
  • Rate Limiting: While HDX is generally generous, excessive requests in a short period might trigger rate limiting, resulting in 429 Too Many Requests errors. If this occurs, implement exponential backoff in your retry logic.
  • Server-Side Issues: Occasionally, issues might stem from the HDX server itself. Check the HDX website for any announcements regarding maintenance or outages.

When troubleshooting, always examine the HTTP status code and the response body. These provide crucial clues about the nature of the problem. Tools like browser developer consoles (for front-end calls) or command-line tools like curl -v (for verbose output) can reveal detailed request and response information.