Getting started overview
URLhaus is a community-driven project that collects and shares malicious URLs to help network administrators and security researchers protect their networks from malware, phishing, and other cyber threats. The service maintains a database of URLs used for distributing malware, which is accessible via a public API. Unlike many API services, URLhaus is designed for immediate use without requiring account creation or API keys.
This guide focuses on the practical steps to initiate interaction with the URLhaus API. It covers the methods for querying the database to check the status of a URL or retrieve details about a known malicious URL. The API supports both individual URL lookups and bulk queries, making it suitable for various security applications, from real-time threat detection to retrospective analysis.
The primary interaction method involves sending HTTP POST requests to the designated API endpoint. Responses are typically provided in JSON format, which facilitates programmatic parsing and integration into existing systems. For those requiring large datasets, URLhaus also offers daily dumps of its entire database, accessible via separate download links rather than the API directly.
Before proceeding, ensure you have a basic understanding of making HTTP POST requests, which can be done using command-line tools like curl or programming language libraries (e.g., Python's requests library). Knowledge of JSON parsing is also beneficial for processing API responses effectively.
Create an account and get keys
A notable characteristic of the URLhaus API is that it does not require user registration or API keys to perform queries. This design choice aims to lower the barrier to entry for security researchers and developers, allowing immediate access to threat intelligence data. You can directly send requests to the API endpoint without prior setup steps like creating an account or generating authentication credentials.
This approach simplifies integration, as there's no need to manage API keys, handle authentication tokens, or configure authorization headers. Developers can focus directly on constructing their POST requests and parsing the responses.
While no explicit account or keys are needed for querying, users who wish to submit new malicious URLs to the database will interact with a separate submission interface, which may involve different procedures outlined in the URLhaus API documentation. However, for basic querying, the process is streamlined to be as straightforward as possible.
Your first request
To make your first request to the URLhaus API, you will send an HTTP POST request to the API endpoint with the necessary parameters. This example demonstrates how to check the status of a URL.
API Endpoint
https://urlhaus.abuse.ch/api/
Request Method
POST
Request Headers
Content-Type: application/x-www-form-urlencoded
Request Body Parameters
query: Set tourlfor a single URL query.url: The URL you want to query.
Example: Querying a single URL with curl
Let's check the status of a known malicious URL. Replace <MALICIOUS_URL> with an actual URL for testing, or use a placeholder like http://example.com/malware.exe for demonstration purposes. Note that URLhaus updates its database frequently, so results for specific URLs may change.
curl -X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "query=url" \
-d "url=http://example.com/malware.exe" \
https://urlhaus.abuse.ch/api/
Example: Querying a specific hash with curl
You can also query the database using a hash value. This is useful if you have a file hash and want to see if it's associated with any known malicious URLs in the URLhaus database.
curl -X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "query=hash" \
-d "hash=a50cdb13ac0bc87f176e58288540c14f4e1564f7" \
https://urlhaus.abuse.ch/api/
Example: Querying a single URL with Python
Using the requests library in Python, you can achieve the same result:
import requests
urlhaus_api_url = "https://urlhaus.abuse.ch/api/"
malicious_url_to_check = "http://example.com/malware.exe" # Replace with a URL for testing
payload = {
'query': 'url',
'url': malicious_url_to_check
}
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
try:
response = requests.post(urlhaus_api_url, data=payload, headers=headers)
response.raise_for_status() # Raise an exception for HTTP errors
data = response.json()
print(data)
except requests.exceptions.RequestException as e:
print(f"An error occurred: {e}")
Expected Response (JSON)
A successful response will typically be in JSON format, providing details about the queried URL. The structure and content will vary based on whether the URL is known to URLhaus and whether it's currently online or offline. Here's an example of what you might receive:
{
"query_status": "ok",
"id": "123456",
"url": "http://example.com/malware.exe",
"url_status": "online",
"threat": "malware_distribution",
"tags": [
"malware",
"exe"
],
"date_added": "2023-01-01 12:00:00 UTC",
"last_seen": "2023-01-05 15:30:00 UTC",
"reporter": "some_researcher",
"payloads": [
{
"first_seen": "2023-01-01 12:00:00 UTC",
"last_seen": "2023-01-05 15:30:00 UTC",
"filename": "malware.exe",
"file_type": "exe",
"file_size": 123456,
"crc32": "abcdef01",
"md5": "2f6b3e4a5c6d7e8f9a0b1c2d3e4f5a6b",
"sha1": "8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b",
"sha256": "1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b",
"signature": "Generic.Malware",
"urlhaus_download": "https://urlhaus.abuse.ch/download/2f6b3e4a5c6d7e8f9a0b1c2d3e4f5a6b/"
}
]
}
If the URL is not found in the database, the query_status will be no_results.
Common next steps
Once you've successfully made your first API call, consider these next steps to expand your use of URLhaus:
1. Explore additional query types
The URLhaus API supports querying by various parameters beyond just a single URL. You can query by:
- Host: Check all known URLs associated with a specific hostname.
- Tag: Retrieve URLs categorized with a particular tag (e.g.,
phishing,botnet). - Signature: Find URLs associated with specific malware signatures.
- Payoad hash: Query for URLs that distribute a file with a given MD5, SHA1, or SHA256 hash.
- Reporting software: Identify URLs submitted by a specific reporting tool.
Refer to the URLhaus API documentation for query parameters to understand the full range of options and required parameters for each query type.
2. Integrate into security tools
Integrate URLhaus data into your existing security infrastructure. This could involve:
- Firewalls and proxies: Automatically update blacklists to block access to known malicious URLs.
- SIEM systems: Enrich security event data with URLhaus threat intelligence for better detection and alerting.
- Incident response playbooks: Automate URL lookups during incident investigations to quickly assess threats.
3. Automate daily dumps
For high-volume requirements or offline analysis, consider downloading the daily URLhaus database dumps. These dumps provide the entire dataset in CSV or JSON format, allowing for comprehensive local analysis or integration into large-scale threat intelligence platforms. Automating the download process can ensure your systems always have the most current data.
4. Contribute to the database
If you discover new malicious URLs, consider contributing them to URLhaus. The community-driven nature of URLhaus relies on contributions from security professionals worldwide. Submitting new URLs helps improve the overall effectiveness of the threat intelligence shared by the platform. Details on how to submit URLs are available in the URLhaus API submission documentation.
5. Monitor API usage (rate limits)
While URLhaus does not explicitly publish strict rate limits for its public API, it's generally good practice to implement respectful usage patterns. Excessive requests in a short period might lead to temporary blocking of your IP address. For high-volume automated queries, consider spacing out requests or leveraging the daily dumps. Adhere to RFC 9110 guidelines for rate limiting to ensure reliable service.
Troubleshooting the first call
If your first API call to URLhaus doesn't return the expected results, consider the following common issues and troubleshooting steps:
1. Incorrect API Endpoint
- Issue: Using an old or incorrect API URL.
- Solution: Double-check that you are using
https://urlhaus.abuse.ch/api/.
2. Incorrect Request Method
- Issue: Sending a
GETrequest instead of aPOSTrequest. - Solution: Ensure your client (
curl, Pythonrequests, etc.) is configured to send an HTTPPOSTrequest.
3. Missing or Incorrect Content-Type Header
- Issue: The
Content-Typeheader is missing or set incorrectly. - Solution: The API expects
Content-Type: application/x-www-form-urlencoded. Verify this header is present and correctly spelled.
4. Incorrect Request Body Parameters
- Issue: Parameters like
queryorurlare misspelled, missing, or in the wrong format. - Solution: Ensure your request body correctly includes
query=urlandurl=<your_url>(or other valid parameters for different query types). Parameters should be URL-encoded.
5. Network Connectivity Issues
- Issue: Your system cannot reach the URLhaus server.
- Solution: Check your internet connection. If you are behind a corporate firewall or proxy, ensure it allows outbound connections to
https://urlhaus.abuse.ch/.
6. URL Encoding Issues
- Issue: Special characters in the URL being queried are not properly URL-encoded.
- Solution: Ensure that any URL you pass as a parameter (e.g., in the
urlfield) is correctly URL-encoded to prevent parsing errors. Many HTTP client libraries handle this automatically.
7. Rate Limiting
- Issue: Making too many requests in a short period, leading to a temporary block.
- Solution: If you receive an HTTP 429 (Too Many Requests) error or no response, wait a few minutes before trying again. Implement delays or backoff strategies for automated scripts. For bulk data needs, use the daily database dumps.
8. Empty or Unexpected Response
- Issue: The API returns an empty response, a non-JSON response, or an unexpected status like
no_results. - Solution: Verify the URL or hash you are querying is indeed known to URLhaus. Test with a known malicious URL from the URLhaus browse page to confirm the API is working as expected. If the URL is legitimate and not malicious, it will correctly return
query_status: no_results.
Quick Reference: Getting Started with URLhaus
| Step | What to Do | Where |
|---|---|---|
| 1. API Endpoint | Note the URLhaus API endpoint. | https://urlhaus.abuse.ch/api/ |
| 2. Authentication | No account or API keys required for queries. | N/A |
| 3. Request Method | Use HTTP POST. | Your HTTP client (curl, Python requests) |
| 4. Headers | Set Content-Type: application/x-www-form-urlencoded. |
Your HTTP client |
| 5. Body Parameters | Include query=url and url=<your_url>. |
Your HTTP client (as -d for curl, data for Python) |
| 6. First Call | Execute a curl or Python request. |
Terminal or Python script |
| 7. Parse Response | Expect JSON output. | Your application logic |