Getting started overview

Getting started with WebScraping.AI involves a sequence of steps to configure API access and initiate your first data extraction request. The process typically includes account registration, API key retrieval, and making an authenticated HTTP request to the WebScraping.AI endpoint. The API handles complexities such as proxy management, CAPTCHA resolution, and JavaScript rendering, allowing developers to focus on data parsing rather than infrastructure. WebScraping.AI offers a free tier for initial testing, providing 1,000 requests per month without charge.

This guide outlines the essential steps to get the WebScraping.AI API operational, from account setup to executing a basic cURL request. It also covers common issues and next steps for integrating the service into applications.

Here is a quick reference for the getting started process:

Step What to Do Where
1. Sign Up Create a WebScraping.AI account. WebScraping.AI homepage
2. Get API Key Locate your unique API key in the dashboard. WebScraping.AI dashboard
3. Construct Request Formulate an HTTP GET request with your API key and target URL. Your preferred HTTP client (e.g., cURL, Python requests)
4. Execute Request Send the request to the WebScraping.AI API endpoint. Terminal or code editor
5. Process Response Handle the JSON or HTML output returned by the API. Your application logic

Create an account and get keys

To begin using WebScraping.AI, you must first register for an account. This process establishes your identity and provides access to the WebScraping.AI dashboard, where your API key is managed.

  1. Navigate to the WebScraping.AI Website: Open your web browser and go to the WebScraping.AI homepage.

  2. Sign Up: Look for a "Sign Up" or "Get Started Free" button, typically located in the top right corner of the page. Click it to initiate the registration process.

  3. Provide Registration Details: You will likely be prompted to enter an email address and create a password. Some services may also offer sign-up options via Google or GitHub accounts.

  4. Verify Email (If Required): After submitting your details, check your email inbox for a verification link. Clicking this link confirms your email address and completes the account creation.

  5. Access Your Dashboard: Once registered and logged in, you will be directed to your personal WebScraping.AI dashboard.

  6. Locate Your API Key: Within the dashboard, your unique API key should be prominently displayed, often in a section labeled "API Key" or "Credentials." This key is essential for authenticating all your API requests.

  7. Secure Your API Key: Treat your API key as sensitive information. Do not embed it directly into client-side code, commit it to public repositories, or share it unnecessarily. Best practices include using environment variables or a secure configuration management system to store and access API keys, as recommended in general API key security guidelines from Google Cloud.

Your first request

Once you have your API key, you can make your first request to the WebScraping.AI API. The API uses a single primary endpoint, https://api.webscraping.ai/html, for most scraping operations. You pass the target URL and your API key as query parameters.

This example demonstrates how to make a basic request using cURL, a command-line tool for making HTTP requests, which is widely available on most operating systems. For other language examples, consult the WebScraping.AI documentation.

Example cURL Request:

Replace YOUR_API_KEY with the actual API key obtained from your dashboard and TARGET_URL with the URL of the webpage you wish to scrape (e.g., https://example.com).

curl -X GET "https://api.webscraping.ai/html?api_key=YOUR_API_KEY&url=https://example.com"

Breaking Down the Request:

  • curl -X GET: Specifies that this is an HTTP GET request.
  • "https://api.webscraping.ai/html?...": This is the WebScraping.AI API endpoint.
  • api_key=YOUR_API_KEY: Your unique API key for authentication.
  • url=https://example.com: The URL of the webpage you want to scrape.

Expected Response:

Upon a successful request, the API will return the HTML content of the target URL. The response will typically be a JSON object containing the HTML and potentially other metadata, or raw HTML if specified. For a simple request like the one above, the response will be the HTML of https://example.com.

{
  "content": "<!DOCTYPE html>\n<html>\n<head>\n<title>Example Domain</title>\n... (truncated HTML content) ...",
  "status": "success",
  "status_code": 200
}

This response structure provides the scraped HTML content within the content field, along with a status indicating the success of the operation.

Common next steps

After successfully making your first request, several common next steps can enhance your web scraping capabilities with WebScraping.AI:

  1. Explore API Parameters: The WebScraping.AI API provides various parameters to control scraping behavior. These include options for JavaScript rendering (&render_js=true), proxy location, custom headers, and more. Refer to the WebScraping.AI API documentation for a comprehensive list and examples.

  2. Integrate into a Programming Language: While cURL is useful for testing, most applications integrate the API using a programming language. WebScraping.AI provides code examples for Python, Node.js, PHP, and Ruby. Using libraries like Python's requests or Node.js's axios simplifies API calls and response handling.

  3. Parse Scraped Data: The raw HTML returned by the API needs to be parsed to extract specific data points. Libraries such as Beautiful Soup (Python), Cheerio (Node.js), or DOMDocument (PHP) are commonly used for this purpose. These tools allow you to navigate the HTML structure using CSS selectors or XPath expressions.

  4. Handle Pagination and Dynamic Content: For websites with multiple pages or content loaded dynamically via JavaScript, you will need to implement logic to iterate through pages and potentially enable JavaScript rendering in your API requests.

  5. Error Handling and Retries: Implement robust error handling to manage failed requests, rate limits, or unexpected responses. Consider implementing retry mechanisms with exponential backoff to handle transient network issues or API limits, a common practice in distributed systems as described by Microsoft Azure.

  6. Monitor Usage: Regularly check your WebScraping.AI dashboard to monitor your request usage and ensure you stay within your plan limits. This helps prevent service interruptions and manage costs.

Troubleshooting the first call

When making your first API call, you might encounter issues. Here are some common problems and their solutions:

  • Invalid API Key (HTTP 401 Unauthorized):

    • Issue: The API returns a 401 error, indicating that your API key is incorrect or missing.
    • Solution: Double-check that you have copied the API key exactly as it appears in your WebScraping.AI dashboard. Ensure there are no leading or trailing spaces. Confirm that the api_key parameter is correctly included in your request URL.

  • Missing URL Parameter (HTTP 400 Bad Request):

    • Issue: The API returns a 400 error, stating that the url parameter is missing or malformed.
    • Solution: Verify that the url parameter is present in your request and that its value is a properly encoded, absolute URL (e.g., https://example.com, not just example.com).

  • Rate Limit Exceeded (HTTP 429 Too Many Requests):

    • Issue: You receive a 429 error, indicating you've sent too many requests in a short period or exceeded your plan's monthly limit.
    • Solution: If you are on the free tier, you might have exceeded 1,000 requests. Wait for a short period before retrying, or consider upgrading your WebScraping.AI plan if you consistently hit this limit. Implement delays between your requests to avoid hitting rate limits.

  • Network Issues or Connection Timeouts:

    • Issue: Your client (e.g., cURL) reports a connection error or timeout before receiving an API response.
    • Solution: Check your internet connection. Ensure there are no local firewall rules or proxy settings preventing your client from reaching api.webscraping.ai. Try the request again after a short delay.

  • Unexpected HTML Content:

    • Issue: The returned HTML is not what you expect (e.g., a CAPTCHA page, an empty page, or an error page from the target website).
    • Solution: The target website might be detecting the scraping attempt or requires JavaScript rendering. Try adding &render_js=true to your request parameters. If the issue persists, the target site might have advanced anti-bot measures, and you may need to consult the WebScraping.AI documentation for more advanced options or consider alternative approaches.