Getting started overview
Enigma Public provides access to a wide range of public data for enterprise applications, primarily supporting data science projects, business intelligence, economic research, and competitive analysis. Unlike self-service APIs, access to Enigma Public's data and APIs requires an initial consultation to determine specific data needs and establish a customized agreement. This process ensures that clients receive tailored data sets and appropriate access mechanisms, which may include Secure File Transfer Protocol (SFTP), Amazon S3 buckets, or direct API integration.
The typical getting started workflow involves:
- Contacting Sales: Initiating a discussion with the Enigma Public sales team to define data requirements.
- Agreement & Customization: Establishing a data licensing agreement and configuring the data delivery method.
- Credential Provisioning: Receiving the necessary API keys, tokens, or access credentials.
- Integration: Developing code to integrate with the provided data feeds or API endpoints.
This page focuses on the steps following the initial agreement, assuming credentials have been provided.
Create an account and get keys
Enigma Public does not offer a self-service sign-up portal for immediate API key generation. Account creation and credential provisioning are part of a managed process tailored for enterprise clients. Access to Enigma Public's data products, such as Enigma Public Enterprise Data, necessitates direct engagement with their sales team to define specific data requirements and licensing terms. This approach ensures that data access is customized to each client's unique use case and compliance needs, including adherence to SOC 2 Type II compliance standards.
To begin the process of obtaining access and credentials:
- Initiate Contact: Visit the Enigma Public contact sales page.
- Consultation: Engage with the Enigma Public team to discuss your organization's data needs, use cases, and integration requirements. This consultation will help determine the most suitable data sets and delivery mechanisms (e.g., SFTP, AWS S3, or API).
- Agreement: Once data requirements are finalized, a formal data licensing agreement will be established.
- Credential Provisioning: Upon completion of the agreement, Enigma Public will provide the necessary access credentials. These credentials typically include API keys, authentication tokens, or secure access details for SFTP or AWS S3 buckets, depending on the agreed-upon delivery method. Specific instructions for credential usage will be supplied at this stage.
For API integrations, the provided credentials will be essential for authenticating requests to the Enigma Public API endpoints. For data delivered via SFTP or AWS S3, the credentials will enable secure file transfers or access to designated data storage locations.
Your first request
Making your first request to Enigma Public's data services depends on the agreed-upon data delivery method. This section outlines how to approach a first request for both API-based access and file-based data delivery.
API-based access
If your agreement includes API access, Enigma Public will provide specific API documentation relevant to your licensed datasets and endpoints. While the exact structure varies, most enterprise APIs follow common patterns for authentication and data retrieval. For example, many APIs use API keys in request headers or OAuth 2.0 for token-based authentication, as described in the OAuth 2.0 specification.
Example API Request (Conceptual)
Assuming you have received an API key (YOUR_API_KEY) and an endpoint (https://api.enigma.com/v1/data/dataset_name), a conceptual request using curl might look like this:
curl -X GET \
'https://api.enigma.com/v1/data/dataset_name?param1=value1¶m2=value2' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Accept: application/json'
Explanation:
-X GET: Specifies the HTTP GET method for retrieving data.'https://api.enigma.com/v1/data/dataset_name?param1=value1¶m2=value2': The specific API endpoint provided by Enigma Public, including any necessary query parameters to filter or specify the data.-H 'Authorization: Bearer YOUR_API_KEY': An HTTP header containing your API key, typically prefixed withBearerfor token-based authentication. The exact header name and format will be specified in your Enigma Public API documentation.-H 'Accept: application/json': Indicates that you prefer the response in JSON format.
Upon a successful request, the API will return data in a structured format, commonly JSON, which your application can then parse and process.
File-based data delivery (SFTP/AWS S3)
For data delivered via SFTP or AWS S3, your first "request" involves connecting to the designated server or bucket using the provided credentials.
SFTP Example
If you receive SFTP credentials (hostname, username, password/SSH key), you can connect using an SFTP client or command-line tools:
sftp YOUR_USERNAME@YOUR_SFTP_HOST
After connecting, you can navigate directories and download files using standard SFTP commands (e.g., ls, get filename.csv). For secure file transfers, it is common practice to use SSH keys rather than passwords, as detailed in SSH.com's SFTP guide.
AWS S3 Example
If data is delivered via an AWS S3 bucket, you will receive AWS access keys (Access Key ID and Secret Access Key) and the bucket name. You can use the AWS CLI or an AWS SDK in your preferred programming language to access the data. Ensure your AWS CLI is configured with your credentials:
aws s3 ls s3://YOUR_S3_BUCKET_NAME
This command lists the contents of your designated S3 bucket. You can then use aws s3 cp or SDK methods to download specific files. For more details on configuring the AWS CLI, refer to the AWS CLI documentation on quick configuration.
Quick Reference: First Steps
| Step | What to do | Where to find info |
|---|---|---|
| 1. Contact Sales | Initiate discussion on data needs. | Enigma Public Contact Sales |
| 2. Agreement & Customization | Finalize data licensing and delivery method. | Direct communication with Enigma Public team |
| 3. Credential Provisioning | Receive API keys, tokens, or SFTP/S3 access details. | Provided directly by Enigma Public |
| 4. First Request (API) | Authenticate and retrieve data using provided API docs. | Enigma Public API documentation (provided post-agreement) |
| 4. First Request (SFTP) | Connect via SFTP client with provided credentials. | Enigma Public SFTP access details |
| 4. First Request (AWS S3) | Configure AWS CLI/SDK and access bucket. | Enigma Public S3 access details, AWS CLI documentation |
Common next steps
After successfully making your first request and accessing Enigma Public data, consider these common next steps to integrate the data more deeply into your applications and workflows:
- Data Parsing and Transformation: Raw data often requires parsing, cleaning, and transformation to fit your specific analytical models or application schemas. This may involve using data processing libraries in Python (e.g., Pandas) or R, or specialized ETL (Extract, Transform, Load) tools.
- Error Handling and Logging: Implement robust error handling for API requests or file transfers to gracefully manage issues like network failures, authentication errors, or rate limits. Integrate logging to monitor data ingestion processes and troubleshoot problems efficiently.
- Scheduling and Automation: For recurring data updates, set up automated scripts or workflows to fetch new data on a schedule. This could involve cron jobs, cloud-based schedulers (e.g., AWS EventBridge, Google Cloud Scheduler), or workflow orchestration tools like Tray.io.
- Data Storage and Management: Determine how you will store and manage the Enigma Public data locally. Options include relational databases (e.g., PostgreSQL, MySQL), data warehouses (e.g., Snowflake, Google BigQuery), or data lakes (e.g., AWS S3, Azure Data Lake Storage), depending on scale and query requirements.
- Security Best Practices: Review and implement security best practices for handling API keys, tokens, and sensitive data. This includes using environment variables for credentials, limiting access permissions, and encrypting data at rest and in transit.
- Monitoring and Alerts: Set up monitoring for your data pipelines to track data freshness, transfer success rates, and API response times. Configure alerts to notify you of any anomalies or failures.
- Scalability Considerations: As your data usage grows, consider the scalability of your integration. This might involve optimizing data retrieval queries, implementing caching strategies, or distributing data processing across multiple resources.
- Compliance and Governance: Ensure your usage of Enigma Public data complies with your organization's internal data governance policies and any relevant industry regulations. Enigma Public's SOC 2 Type II compliance can support your own compliance efforts.
Troubleshooting the first call
Encountering issues during your first interaction with Enigma Public's data services is common. Here's a guide to troubleshooting typical problems:
API-based access issues
- Authentication Errors (401 Unauthorized, 403 Forbidden):
- Check Credentials: Verify that your API key or authentication token is correct and has not expired. Ensure it is included in the request header exactly as specified in the Enigma Public documentation.
- Token Format: Confirm the token format (e.g.,
Bearer YOUR_API_KEY) is correct. - Permissions: Ensure your credentials have the necessary permissions for the specific endpoint you are trying to access.
- Bad Request (400):
- Parameters: Check that all required query parameters or request body fields are present and correctly formatted according to the API documentation.
- Data Types: Verify that parameter values match the expected data types (e.g., integer, string, date format).
- Not Found (404):
- Endpoint URL: Double-check the API endpoint URL for typos. Ensure you are using the correct version of the API (e.g.,
/v1/). - Resource Existence: Confirm that the specific dataset or resource you are requesting exists and is accessible with your credentials.
- Endpoint URL: Double-check the API endpoint URL for typos. Ensure you are using the correct version of the API (e.g.,
- Server Errors (5xx):
- Retries: Server-side errors can sometimes be transient. Implement a retry mechanism with exponential backoff.
- Contact Support: If errors persist, gather details (request, response, timestamp) and contact Enigma Public support.
- Network Issues:
- Connectivity: Ensure your network connection is stable and there are no firewalls or proxies blocking outbound requests to the API endpoint.
File-based data delivery issues (SFTP/AWS S3)
- Connection Refused/Timeout (SFTP):
- Hostname/Port: Verify the SFTP hostname and port number are correct.
- Firewall: Check local or network firewalls that might be blocking the SFTP port (typically 22).
- IP Whitelisting: Confirm if Enigma Public requires your IP address to be whitelisted for SFTP access.
- Authentication Failed (SFTP/AWS S3):
- Username/Password: Double-check the SFTP username and password.
- SSH Key: If using SSH keys, ensure the correct private key is being used and it's not password-protected (or you're providing the passphrase).
- AWS Credentials: For S3, verify your AWS Access Key ID and Secret Access Key are correct and properly configured in your environment or AWS CLI.
- Permission Denied (SFTP/AWS S3):
- Directory/Bucket Access: Ensure your credentials have read/list permissions for the specific directory or S3 bucket you are trying to access.
- File Permissions: For SFTP, check file-level permissions if you cannot download specific files.
- Empty Directory/Bucket (SFTP/AWS S3):
- Data Availability: Confirm with Enigma Public that data has been loaded into your designated location.
- Path: Ensure you are navigating to the correct directory path within the SFTP server or S3 bucket.
Always refer to the specific documentation provided by Enigma Public for your licensed data products, as it will contain the most accurate and up-to-date information for troubleshooting.