URLhaus: Malware URL Database and Threat Intelligence API

Overview

URLhaus, operated by abuse.ch, is a community-driven project dedicated to collecting and sharing intelligence on URLs used by cybercriminals to distribute malware. Established in 2014, its primary objective is to provide a comprehensive, real-time database of malicious URLs, including those hosting malware payloads or acting as command-and-control (C2) servers. This resource is made available to the public and security community without charge, supporting efforts to detect, prevent, and respond to cyber threats.

The service is designed for developers, security analysts, incident responders, and network administrators who require up-to-date threat intelligence. By offering an API, URLhaus enables automated integration into existing security infrastructure, allowing organizations to programmatically check URLs against its extensive database. This functionality is critical for applications such as email filters, web proxies, intrusion detection systems, and threat intelligence platforms that need to identify and block access to malicious content rapidly.

URLhaus collects URLs from various sources, including automated systems, honeypots, and contributions from a global community of security researchers. Each submitted URL undergoes analysis to determine if it hosts malware, phishing content, or other forms of malicious activity. Verified malicious URLs are then added to the database, along with contextual information such as the malware family associated with the URL, the timestamp of detection, and the current status of the URL (e.g., online, offline). This data is then disseminated through the API and daily dumps.

The platform shines in scenarios requiring rapid identification and blocking of known threats. For instance, an organization's security operations center (SOC) can integrate the URLhaus API into its SIEM (Security Information and Event Management) system to flag suspicious URLs encountered in network traffic or email. Similarly, security vendors can incorporate URLhaus data into their products to enhance their URL filtering and threat detection capabilities. Its straightforward API and comprehensive documentation make it accessible for developers seeking to build or enhance security tools. The entire service, including its API and data dumps, is offered for free, lowering the barrier to entry for security initiatives and research.

While URLhaus focuses specifically on malicious URLs, other services like VirusTotal offer broader malware analysis capabilities, including file scanning and reputation checks, providing a complementary approach to threat intelligence.

Key features

• Malware URL Database: A continuously updated repository of URLs identified as distributing malware, including drive-by downloads, phishing sites, and command-and-control infrastructure.
• URLhaus API: A RESTful API allowing programmatic submission of URLs for analysis and querying of the database to check the status and details of known malicious URLs (URLhaus API documentation).
• URLhaus Daily Dump: Provides daily updated lists of all currently active malicious URLs in various formats (CSV, JSON), enabling bulk integration into security systems.
• URL Submission: Users can submit suspicious URLs for analysis, contributing to the community's collective threat intelligence.
• Real-time Data: The database is updated frequently to reflect new threats and changes in URL status, providing current threat intelligence.
• Contextual Information: For each malicious URL, URLhaus provides details such as the malware family, payload information, and submission metadata.
• Free Access: All services, including the API and data dumps, are available without cost, promoting widespread adoption in security research and defense.

Pricing

URLhaus is a project offered entirely for free by abuse.ch.

For detailed information on usage policies, refer to the URLhaus API documentation.

Common integrations

• Security Information and Event Management (SIEM) Systems: Integrate the API to enrich log data with URL threat intelligence for incident detection and response.
• Firewalls and Proxies: Use daily dumps or API queries to block access to known malicious URLs at the network perimeter.
• Email Security Gateways: Scan URLs in incoming emails against the URLhaus database to detect and block phishing and malware distribution attempts.
• Threat Intelligence Platforms (TIPs): Incorporate URLhaus data feeds to augment existing threat intelligence sources.
• Endpoint Detection and Response (EDR) Solutions: Enhance endpoint protection by flagging or blocking processes attempting to connect to malicious URLs.
• Security Research Tools: Utilize the API for automated analysis and monitoring of suspicious URLs.

Alternatives

• VirusTotal: A service that analyzes suspicious files and URLs to detect types of malware, automatically share them with the security community, and provide contextual information.
• URLScan.io: A free service to analyze and screenshot websites, providing detailed information about the URL, IP addresses, domains, and network requests.
• PhishTank: A collaborative clearing house for data and information about phishing, providing an open API for developers to integrate anti-phishing capabilities.

Getting started

To get started with the URLhaus API, you can query the database for information about a specific URL. The following Python example demonstrates how to check if a URL is known to be malicious using the API's query interface.

import requests
import json

def check_url_status(url_to_check):
    api_url = "https://urlhaus.abuse.ch/api/v1/"
    headers = {"User-Agent": "URLhaus API Example (apispine)"}
    data = {"query": "url", "url": url_to_check}

    try:
        response = requests.post(api_url, headers=headers, data=data)
        response.raise_for_status()  # Raise an exception for HTTP errors
        result = response.json()

        if result.get("query_status") == "ok":
            if result.get("url_status") == "online":
                print(f"URL: {url_to_check} is ONLINE and listed as malicious.")
                print(f"Threat: {result.get('threat')}")
                print(f"Malware: {result.get('malware_family')}")
                print(f"First seen: {result.get('firstseen')}")
            elif result.get("url_status") == "offline":
                print(f"URL: {url_to_check} is OFFLINE but was listed as malicious.")
            else:
                print(f"URL: {url_to_check} is NOT listed as malicious in URLhaus.")
        else:
            print(f"Error querying URLhaus: {result.get('query_status')}")
            if result.get("error"): print(f"Details: {result.get('error')}")

    except requests.exceptions.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
    except requests.exceptions.ConnectionError as conn_err:
        print(f"Connection error occurred: {conn_err}")
    except requests.exceptions.Timeout as timeout_err:
        print(f"Timeout error occurred: {timeout_err}")
    except requests.exceptions.RequestException as req_err:
        print(f"An unexpected error occurred: {req_err}")
    except json.JSONDecodeError:
        print("Failed to decode JSON response from URLhaus API.")

# Example usage:
print("Checking a known malicious URL:")
check_url_status("http://example.com/malware.exe") # Replace with an actual known malicious URL for testing if desired

print("\nChecking a benign URL:")
check_url_status("https://www.google.com/")

This script sends a POST request to the URLhaus API with the target URL. It then parses the JSON response to determine the URL's status within the database, indicating whether it's known to be malicious, its threat type, and associated malware family. For more advanced usage, including submitting URLs or retrieving payload details, refer to the official URLhaus API documentation.