Getting started overview

Integrating ScraperAPI involves a few key steps: account creation, obtaining your API key, and constructing your first API request. ScraperAPI functions as an intermediary proxy, simplifying web scraping challenges such as IP rotation, CAPTCHA solving, and browser rendering (ScraperAPI documentation). Developers send their target URL to ScraperAPI, which then handles the request to the target website and returns the HTML content.

This guide provides a structured approach to making your initial call, with code examples in both Python and Node.js, ScraperAPI's primary example languages. The overall process is designed to be straightforward, primarily requiring adjustment of request parameters to suit specific scraping needs (ScraperAPI developer experience notes).

Quick reference table

Step What to do Where
1. Sign Up Create a ScraperAPI account ScraperAPI homepage
2. Get API Key Locate your unique API key ScraperAPI dashboard
3. Construct Request Build your API call with the target URL Your code editor
4. Execute Request Run your code to fetch data Your terminal or application
5. Process Response Handle the returned HTML or JSON Your code editor

Create an account and get keys

To begin using ScraperAPI, you must first create an account. This process grants access to the ScraperAPI dashboard, where your unique API key is provided. The API key is essential for authenticating all your requests to the ScraperAPI service.

Account creation

  1. Navigate to the ScraperAPI homepage.
  2. Click on the "Sign Up" or "Get Started Free" button.
  3. Provide the required information, such as your email address and a password.
  4. Complete any verification steps, if prompted.
  5. Upon successful registration, you will be redirected to your ScraperAPI dashboard.

Obtaining your API key

Once logged into your ScraperAPI dashboard:

  • Your API key will be prominently displayed, often in a section labeled "Your API Key" or similar.
  • The key is a long alphanumeric string. Copy this key, as it will be included in every API request you make.
  • ScraperAPI offers a free tier that includes 1,000 API calls, allowing you to test the service without immediate financial commitment.

Your first request

After acquiring your API key, you can construct your first API call. ScraperAPI works by accepting a URL as a parameter, then fetching the content of that URL through its proxy network. The API returns the raw HTML content of the target page.

API endpoint

The base endpoint for ScraperAPI is http://api.scraperapi.com/. All requests are made to this endpoint, with your API key and the target URL appended as query parameters.

Request parameters

  • api_key: (Required) Your unique API key obtained from the dashboard.
  • url: (Required) The URL of the webpage you wish to scrape. Ensure this URL is properly URL-encoded if it contains special characters.
  • render: (Optional) Set to true to enable JavaScript rendering for dynamic content.
  • country_code: (Optional) Specify the country for the proxy.
  • premium: (Optional) Set to true to use premium proxies.

For a complete list of available parameters and their functions, consult the ScraperAPI documentation.

Example requests

Below are examples demonstrating how to make a basic request to ScraperAPI using Python and Node.js. For these examples, replace YOUR_API_KEY with your actual API key and https://example.com with your target URL.

Python example

This Python example uses the requests library to make an HTTP GET request to ScraperAPI. This method is common for interacting with HTTP-based APIs (MDN Web Docs on HTTP GET).


import requests

api_key = 'YOUR_API_KEY' # Replace with your actual API key
target_url = 'https://example.com' # Replace with the URL you want to scrape

payload = {'api_key': api_key, 'url': target_url}
r = requests.get('http://api.scraperapi.com/', params=payload)

print(r.text)

Node.js example

This Node.js example uses the built-in https module for making the request. Alternatively, libraries like axios or node-fetch could be used for more streamlined HTTP requests in Node.js environments.


const https = require('https');

const api_key = 'YOUR_API_KEY'; // Replace with your actual API key
const target_url = 'https://example.com'; // Replace with the URL you want to scrape

const options = {
  hostname: 'api.scraperapi.com',
  path: `/?api_key=${api_key}&url=${encodeURIComponent(target_url)}`,
  method: 'GET'
};

const req = https.request(options, (res) => {
  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('end', () => {
    console.log(data);
  });
});

req.on('error', (e) => {
  console.error(e);
});

req.end();

Common next steps

Once you have successfully made your first request, consider these next steps to enhance your scraping capabilities with ScraperAPI:

  • Parse HTML Content: The API returns raw HTML. You will need a parsing library (e.g., Beautiful Soup in Python, Cheerio in Node.js) to extract specific data points from the HTML structure.
  • Handle Dynamic Content: If the target website relies heavily on JavaScript to load content, use the render=true parameter to enable ScraperAPI's browser rendering engine (ScraperAPI JavaScript rendering guide).
  • Manage Concurrency: For large-scale scraping, implement asynchronous requests and manage the rate at which you send requests to stay within ScraperAPI's rate limits and prevent overwhelming target servers.
  • Error Handling: Implement robust error handling in your code to manage network issues, API errors, and unexpected responses from target websites.
  • Proxy Options: Explore advanced proxy options like country-specific proxies or premium proxies to adapt to different website protections.
  • Integrate with Data Storage: Store the extracted data in a database (e.g., SQL, NoSQL), a CSV file, or another suitable format for further analysis or use.
  • Review Pricing: If your usage exceeds the free tier, review the ScraperAPI pricing plans to choose a tier that matches your expected call volume.

Troubleshooting the first call

If your first API call doesn't return the expected results, consider the following troubleshooting steps:

  • Check API Key: Double-check that your API key is correct and has not expired. A common issue is a typo in the key.
  • Verify URL Encoding: Ensure the target URL is correctly URL-encoded, especially if it contains special characters like &, ?, or =.
  • Inspect ScraperAPI Dashboard: Visit your ScraperAPI dashboard to check your API call usage and ensure you haven't exceeded your plan limits. The dashboard may also show recent request logs or error messages.
  • Review Response Status Code: Examine the HTTP status code returned by ScraperAPI. A 200 OK indicates success, while 4xx or 5xx codes point to client-side or server-side issues, respectively (MDN Web Docs on HTTP status codes).
  • Test with render=true: If you're scraping a modern website, it might be dynamically loading content with JavaScript. Try adding render=true to your request parameters to see if that resolves the issue.
  • Consult ScraperAPI Docs: Refer to the official ScraperAPI documentation for detailed explanations of parameters, error codes, and advanced usage patterns.
  • Isolate the Problem: Try scraping a simpler, static website (like https://example.com) first to confirm that the basic ScraperAPI integration is working before attempting more complex targets.