Getting started overview
To begin using scrapestack, the initial steps involve setting up an account, retrieving your unique API access key, and constructing your first API call. This process typically takes a few minutes and allows for immediate interaction with the service's core functionality: making proxied HTTP/HTTPS requests to retrieve web page content. The API supports various parameters for customizing requests, such as specifying target URLs, enabling JavaScript rendering, and configuring proxy location. Successful integration begins with a basic request and then scales to more complex configurations as needed for specific web scraping tasks.
The following table provides a quick reference for the essential steps to get started:
| Step | What to do | Where |
|---|---|---|
| 1. Sign Up | Create a new scrapestack account. | scrapestack pricing page or scrapestack homepage |
| 2. Get API Key | Locate your personal API access key. | scrapestack Account Dashboard |
| 3. Construct Request | Build your first API call with the key and a target URL. | Local development environment or browser |
| 4. Execute Request | Send the API call and observe the response. | Terminal (cURL), code editor (SDKs), or browser |
Create an account and get keys
Access to the scrapestack API requires an account and a valid API access key. This key authenticates your requests and links them to your usage plan. scrapestack offers a free tier that includes 10,000 requests per month, which is sufficient for initial testing and development.
- Sign Up: Navigate to the scrapestack homepage. Click on a 'Sign Up' or 'Get Started Free' button. You will typically be prompted to enter an email address and create a password.
- Account Activation: After signing up, you may need to verify your email address. Check your inbox for an activation link from scrapestack.
- Locate API Key: Once your account is active and you have logged into the scrapestack dashboard, your unique API access key will be prominently displayed. It is usually labeled 'Your API Access Key' or similar. Keep this key secure, as it grants access to your account's API usage. The scrapestack documentation provides details on locating this key.
The API key is a crucial component for every request you send to the scrapestack API. It is passed as a query parameter named access_key.
Your first request
Once you have your API access key, you can make your first request to the scrapestack API. The API is primarily accessed via a simple HTTP GET request to a single endpoint, with parameters defining the target URL and other scraping options.
API Endpoint Structure
http://api.scrapestack.com/scrape
?access_key=YOUR_ACCESS_KEY
&url=TARGET_URL
[ &render_js=1 ]
[ &proxy_location=COUNTRY_CODE ]
Replace YOUR_ACCESS_KEY with the key obtained from your dashboard and TARGET_URL with the URL of the website you wish to scrape. Optional parameters like render_js=1 enable JavaScript rendering for dynamic content, and proxy_location allows specifying a proxy country (e.g., us for United States).
Example with cURL
cURL is a command-line tool and library for transferring data with URLs, commonly used for making initial API requests without writing code. You can execute this command in your terminal.
curl 'http://api.scrapestack.com/scrape?access_key=YOUR_ACCESS_KEY&url=http://example.com'
Replace YOUR_ACCESS_KEY with your actual access key. This command will fetch the HTML content of http://example.com through the scrapestack proxy and print it to your terminal.
Example with Python
Python is a popular language for web scraping. The requests library simplifies making HTTP requests.
import requests
ACCESS_KEY = 'YOUR_ACCESS_KEY'
TARGET_URL = 'http://example.com'
params = {
'access_key': ACCESS_KEY,
'url': TARGET_URL
}
api_result = requests.get('http://api.scrapestack.com/scrape', params)
print(api_result.text)
Ensure you have the requests library installed (pip install requests). This Python script performs the same action as the cURL example, printing the scraped HTML content.
Example with Node.js
For Node.js environments, you can use the built-in https module or a third-party library like axios.
const axios = require('axios'); // npm install axios
const ACCESS_KEY = 'YOUR_ACCESS_KEY';
const TARGET_URL = 'http://example.com';
axios.get('http://api.scrapestack.com/scrape', {
params: {
access_key: ACCESS_KEY,
url: TARGET_URL
}
})
.then(response => {
console.log(response.data);
})
.catch(error => {
console.error('Error:', error.message);
});
Install axios if you haven't already (npm install axios).
Interpreting the Response
A successful request will return the HTML content of the TARGET_URL in the response body. If there's an issue, scrapestack will return a JSON object containing an error field with details about what went wrong. For a comprehensive list of error codes, refer to the scrapestack API error codes documentation.
Common next steps
After successfully making your first request, consider these common next steps to enhance your web scraping capabilities with scrapestack:
- Enable JavaScript Rendering: Many modern websites rely heavily on JavaScript to load content dynamically. To scrape such sites, include the
render_js=1parameter in your API calls. This instructs scrapestack to fully render the page, including executing client-side scripts, before returning the HTML. The scrapestack JavaScript rendering documentation provides more details. - Utilize Proxy Locations: If you need to access geo-restricted content or simulate requests from specific regions, use the
proxy_locationparameter with a two-letter country code (e.g.,us,de,gb). This allows you to route your request through a proxy server in that country. - Handle Blocked Requests: Websites can implement anti-scraping measures. If you encounter frequent blocks or CAPTCHAs, consider adjusting parameters like user agents or referrers, or rotating proxies more frequently. While scrapestack manages proxy rotation internally, understanding potential blocking mechanisms can help refine your scraping strategy. Resources like the Cloudflare bot management documentation discuss common strategies for identifying and mitigating automated traffic, which can inform your approach to avoiding blocks.
- Integrate with a Parser: The raw HTML returned by scrapestack often needs to be parsed to extract specific data. Libraries like Beautiful Soup (Python), Cheerio (Node.js), or DOMDocument (PHP) are commonly used for this purpose. These tools allow you to navigate the HTML document structure and select elements based on CSS selectors or XPath expressions.
- Monitor Usage: Regularly check your scrapestack dashboard to monitor your API usage and ensure you stay within your plan's limits. This helps prevent unexpected service interruptions or overage charges.
- Explore Advanced Features: Review the comprehensive scrapestack documentation for advanced features such as custom headers, request timeouts, and handling redirects.
Troubleshooting the first call
When making your first API call, you might encounter issues. Here are some common problems and their solutions:
- Invalid Access Key:
- Symptom: API returns an error message indicating an invalid or missing access key.
- Solution: Double-check that you have copied your API access key correctly from your scrapestack dashboard. Ensure there are no leading or trailing spaces, and that it is correctly passed as the
access_keyquery parameter. The scrapestack error documentation lists specific codes for invalid authentication.
- Incorrect Target URL:
- Symptom: API returns a 400-level client error or an empty/unexpected response.
- Solution: Verify that the
urlparameter contains a valid, fully qualified URL (e.g.,http://example.com, not justexample.com). Test the target URL directly in your browser to confirm it's accessible.
- HTTP vs. HTTPS Endpoint:
- Symptom: Connection errors or unexpected behavior.
- Solution: The scrapestack API supports both HTTP and HTTPS. While
http://api.scrapestack.com/scrapeworks, usinghttps://api.scrapestack.com/scrapeis generally recommended for secure communication, especially when transmitting your API key.
- Rate Limiting:
- Symptom: API returns a 429 Too Many Requests error.
- Solution: If you are on the free tier or a basic plan, you might hit rate limits quickly. Pause between requests or upgrade your plan. The scrapestack pricing page details request limits for each tier.
- Network Issues:
- Symptom: Connection timeouts or network unreachable errors from your client.
- Solution: Ensure your local network connection is stable and that no firewalls or proxies are blocking outbound connections to
api.scrapestack.com. Tools likepingortraceroutecan help diagnose network connectivity.
- JavaScript Rendering Not Enabled:
- Symptom: Scraped content is incomplete or missing dynamic elements that appear in a browser.
- Solution: If the target website uses JavaScript to load content, include the
render_js=1parameter in your request. Note that JavaScript rendering consumes more API credits and may take longer to process.
For further assistance, consult the official scrapestack documentation or their support channels.