# Endpoints
Source: https://developers.gopher-ai.com/docs/data/endpoints





# Introduction
Source: https://developers.gopher-ai.com/docs/data/intro



Welcome to the Gopher Data API! It's the fastest and simplest way to get clean, AI-ready data. Stream straight into models, apps and agents with our secure, real-time search and indexing API.

<Card title="Get Access" icon="key">
  [Visit our dashboard](https://data.gopher-ai.com/) and get personalized access:

  <br />

  ✓ Access real-time data for X/Twitter, TikTok, Reddit, Web and more\
  ✓ Search across our millions of indexed records\
  ✓ Stream data directly into your agents with our API
</Card>

## Key Features

<CardGroup cols={2}>
  <Card title="Real-Time Search" icon="magnifying-glass">
    ✓ Easy to use UI\
    ✓ Structured queries\
    ✓ Advanced filtering options
  </Card>

  <Card title="Security & Scalability" icon="bolt">
    ✓ TEE-powered node system\
    ✓ Real-time data indexing\
    ✓ Secure scraping environment
  </Card>

  <Card title="Storage & Indexing" icon="database">
    ✓ Efficient vector storage\
    ✓ Optimized search capabilities\
    ✓ Scalable retrieval system
  </Card>

  <Card title="Developer Experience" icon="code">
    ✓ Interactive Swagger docs\
    ✓ Code examples (cURL, Python)\
    ✓ Step-by-step guides
  </Card>
</CardGroup>

## How It Works

1. **Get an API Key** – Head to [data.gopher-ai.com](https://data.gopher-ai.com/) to get started
2. **Submit a Search Job** – Use our live search endpoint to request real-time data from X/Twitter, TikTok, Reddit, or Web sources
3. **Monitor Progress** – Track your job status using the returned UUID
4. **Retrieve Results** – Fetch structured JSON results when your job completes

## What You Get

* **Real-time data scraping** across multiple platforms
* **Structured search queries** with advanced filtering
* **Vector similarity search** for finding related content
* **Hybrid search capabilities** combining text and semantic matching
* **Data analysis tools** for extracting insights from your results

This API is designed for **fast, reliable, and controlled access to data**, making it a powerful tool for developers building AI applications that need real-time insights from a wide variety of sources. 🚀


# LinkedIn
Source: https://developers.gopher-ai.com/docs/data/linkedin



All job types use the `/api/v1/search/live` POST endpoint and follow the same structure: a top-level `type` to define the data source, and a set of `arguments` defining the parameters of the job.

<CardGroup cols={2}>
  <Card title="Profile Search" icon="magnifying-glass">
    Search LinkedIn profiles with advanced filtering
  </Card>

  <Card title="Professional Data" icon="briefcase">
    Extract detailed professional information
  </Card>

  <Card title="Contact Discovery" icon="envelope">
    Find contact information and company details
  </Card>

  <Card title="Talent Research" icon="users">
    Research professionals by role, industry, and experience
  </Card>
</CardGroup>

## Overview

The LinkedIn API provides comprehensive profile search capabilities with advanced filtering options. It supports three different scraper modes to balance data depth with performance, and offers extensive filtering by professional criteria.

### `searchbyprofile`

Searches LinkedIn profiles based on professional criteria including experience, seniority, industry, function, and location. This is the primary job type for LinkedIn data extraction.

**Use Cases:**

* Talent acquisition and recruitment
* Professional network analysis
* Market research and competitive intelligence
* Lead generation and sales prospecting
* Industry trend analysis

## Parameters

### Common Parameters

| Parameter            | Type    | Required | Default   | Description                                                   |
| -------------------- | ------- | -------- | --------- | ------------------------------------------------------------- |
| `type`               | string  | Yes      | -         | The job type: `searchbyprofile`                               |
| `profileScraperMode` | string  | No       | `"Short"` | Scraper mode: `"Short"`, `"Full"`, or `"Full + email search"` |
| `searchQuery`        | string  | Yes      | -         | Search query or keywords                                      |
| `maxItems`           | integer | No       | `10`      | Maximum number of profiles to return (max 1000)               |

### Search Filters

| Parameter             | Type           | Required | Default | Description                                    |
| --------------------- | -------------- | -------- | ------- | ---------------------------------------------- |
| `locations`           | array\[string] | No       | -       | Filter by location (e.g., "San Francisco, CA") |
| `currentCompanies`    | array\[string] | No       | -       | Filter by current company names                |
| `pastCompanies`       | array\[string] | No       | -       | Filter by past company names                   |
| `currentJobTitles`    | array\[string] | No       | -       | Filter by current job titles                   |
| `pastJobTitles`       | array\[string] | No       | -       | Filter by past job titles                      |
| `schools`             | array\[string] | No       | -       | Filter by educational institutions             |
| `firstNames`          | array\[string] | No       | -       | Filter by first names                          |
| `lastNames`           | array\[string] | No       | -       | Filter by last names                           |
| `recentlyChangedJobs` | boolean        | No       | -       | Filter for profiles that recently changed jobs |
| `startPage`           | integer        | No       | -       | Starting page for pagination                   |

### Professional Criteria Filters

| Parameter                  | Type           | Required | Default | Description                                                            |
| -------------------------- | -------------- | -------- | ------- | ---------------------------------------------------------------------- |
| `yearsOfExperienceIds`     | array\[string] | No       | -       | Years of experience (see [Experience Levels](#experience-levels))      |
| `yearsAtCurrentCompanyIds` | array\[string] | No       | -       | Years at current company (see [Experience Levels](#experience-levels)) |
| `seniorityLevelIds`        | array\[string] | No       | -       | Seniority levels (see [Seniority Levels](#seniority-levels))           |
| `functionIds`              | array\[string] | No       | -       | Job functions (see [Job Functions](#job-functions))                    |
| `industryIds`              | array\[string] | No       | -       | Industries (see [Industries](#industries))                             |

## Scraper Modes

### `"Short"` (Default)

Returns basic profile information including:

* Name, headline, location
* Current position summary
* Basic profile metrics
* Profile URL and public identifier

**Best for:** Quick searches, large result sets, basic contact information

### `"Full"`

Returns comprehensive profile data including:

* All Short mode data
* Complete work experience history
* Education details
* Skills and endorsements
* Certifications and projects
* Publications and patents
* Recommendations

**Best for:** Detailed professional analysis, comprehensive talent research

### `"Full + email search"`

Returns all Full mode data plus:

* Email addresses (when available)
* Company website information
* Enhanced contact discovery

**Best for:** Lead generation, sales prospecting, direct outreach

## Experience Levels

| ID    | Description        |
| ----- | ------------------ |
| `"1"` | Less than 1 year   |
| `"2"` | 1-2 years          |
| `"3"` | 3-5 years          |
| `"4"` | 6-10 years         |
| `"5"` | More than 10 years |

## Seniority Levels

| ID      | Description         |
| ------- | ------------------- |
| `"100"` | In Training         |
| `"110"` | Entry Level         |
| `"120"` | Senior              |
| `"130"` | Strategic           |
| `"200"` | Entry Level Manager |
| `"210"` | Experienced Manager |
| `"220"` | Director            |
| `"300"` | Vice President      |
| `"310"` | CXO (C-Level)       |
| `"320"` | Partner             |

## Job Functions

| ID     | Function                      | ID     | Function                         |
| ------ | ----------------------------- | ------ | -------------------------------- |
| `"1"`  | Accounting                    | `"14"` | Legal                            |
| `"2"`  | Administrative                | `"15"` | Marketing                        |
| `"3"`  | Arts and Design               | `"16"` | Media and Communication          |
| `"4"`  | Business Development          | `"17"` | Military and Protective Services |
| `"5"`  | Community and Social Services | `"18"` | Operations                       |
| `"6"`  | Consulting                    | `"19"` | Product Management               |
| `"7"`  | Education                     | `"20"` | Program and Project Management   |
| `"8"`  | Engineering                   | `"21"` | Purchasing                       |
| `"9"`  | Entrepreneurship              | `"22"` | Quality Assurance                |
| `"10"` | Finance                       | `"23"` | Real Estate                      |
| `"11"` | Healthcare Services           | `"24"` | Research                         |
| `"12"` | Human Resources               | `"25"` | Sales                            |
| `"13"` | Information Technology        |        |                                  |

## Industries

The LinkedIn API supports 100+ industries. Here are some key categories:

### Technology & Software

* `"4"` - Software Development
* `"3"` - Computer Hardware Manufacturing
* `"96"` - IT Services and IT Consulting
* `"6"` - Technology Information and Internet
* `"8"` - Telecommunications

### Financial Services

* `"43"` - Financial Services
* `"41"` - Banking
* `"42"` - Insurance
* `"45"` - Investment Banking
* `"106"` - Venture Capital and Private Equity

### Healthcare & Medical

* `"2081"` - Hospitals
* `"13"` - Medical Practices
* `"12"` - Biotechnology Research
* `"17"` - Medical Equipment Manufacturing

### Professional Services

* `"1810"` - Professional Services
* `"47"` - Accounting
* `"10"` - Legal Services
* `"11"` - Business Consulting and Services
* `"80"` - Advertising Services

*\[Note: This is a partial list. The API supports 100+ industry categories.]*

## Examples

### Basic Profile Search

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "software engineer",
    "maxItems": 20
  }
}
```

### Advanced Filtered Search

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "machine learning",
    "profileScraperMode": "Full",
    "maxItems": 50,
    "locations": ["San Francisco, CA", "New York, NY"],
    "currentCompanies": ["Google", "Microsoft", "Apple"],
    "seniorityLevelIds": ["120", "130"],
    "functionIds": ["8", "19"],
    "industryIds": ["4", "96"],
    "yearsOfExperienceIds": ["3", "4"]
  }
}
```

### Lead Generation Search

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "marketing director",
    "profileScraperMode": "Full + email search",
    "maxItems": 25,
    "locations": ["Austin, TX"],
    "currentCompanies": ["Dell", "Tesla"],
    "seniorityLevelIds": ["220", "300"],
    "functionIds": ["15"],
    "recentlyChangedJobs": true
  }
}
```

### Industry-Specific Search

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "data scientist",
    "profileScraperMode": "Full",
    "maxItems": 30,
    "industryIds": ["43", "4", "96"],
    "functionIds": ["8", "24"],
    "yearsOfExperienceIds": ["3", "4", "5"],
    "schools": ["Stanford University", "MIT", "UC Berkeley"]
  }
}
```

### Geographic Talent Search

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "product manager",
    "profileScraperMode": "Short",
    "maxItems": 100,
    "locations": ["Seattle, WA", "Portland, OR"],
    "currentJobTitles": ["Product Manager", "Senior Product Manager"],
    "seniorityLevelIds": ["120", "130", "200"],
    "yearsAtCurrentCompanyIds": ["1", "2"]
  }
}
```

## Use Case Examples

### Talent Acquisition

Find senior software engineers in specific locations:

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "senior software engineer",
    "profileScraperMode": "Full",
    "maxItems": 50,
    "locations": ["Remote", "San Francisco, CA"],
    "seniorityLevelIds": ["120", "130"],
    "functionIds": ["8"],
    "industryIds": ["4", "96"],
    "yearsOfExperienceIds": ["4", "5"]
  }
}
```

### Competitive Analysis

Research professionals at competitor companies:

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "marketing",
    "profileScraperMode": "Full",
    "maxItems": 30,
    "currentCompanies": ["Competitor A", "Competitor B"],
    "functionIds": ["15"],
    "seniorityLevelIds": ["200", "220", "300"]
  }
}
```

### Market Research

Analyze industry trends and talent distribution:

```json  theme={null}
{
  "type": "linkedin",
  "arguments": {
    "type": "searchbyprofile",
    "searchQuery": "artificial intelligence",
    "profileScraperMode": "Short",
    "maxItems": 200,
    "industryIds": ["4", "96", "43"],
    "functionIds": ["8", "24", "19"],
    "yearsOfExperienceIds": ["2", "3", "4", "5"]
  }
}
```

## Best Practices

### 1. Query Optimization

* Use specific, relevant keywords in `searchQuery`
* Combine multiple filters for more targeted results
* Start with broader searches and narrow down with filters

### 2. Scraper Mode Selection

* Use `"Short"` for large-scale searches and basic contact info
* Use `"Full"` for detailed professional analysis
* Use `"Full + email search"` only when contact discovery is critical

### 3. Filter Strategy

* Combine location and industry filters for geographic targeting
* Use seniority and experience filters to target appropriate levels
* Leverage company filters for competitive analysis

### 4. Performance Considerations

* Set appropriate `maxItems` based on your analysis needs
* Use pagination with `startPage` for large result sets
* Cache results to avoid redundant API calls

### 5. Data Quality

* Validate filter values against supported IDs
* Use multiple related search terms for comprehensive coverage
* Consider both current and past company filters

## Response Data

The LinkedIn API returns detailed profile information including:

### Basic Profile Information

* **Personal**: Name, headline, location, profile photo
* **Contact**: Profile URL, public identifier
* **Metrics**: Connection count, follower count, premium status

### Professional Information

* **Current Position**: Company, title, tenure, location
* **Experience**: Complete work history with descriptions
* **Education**: Schools, degrees, fields of study
* **Skills**: Endorsed skills and expertise areas

### Additional Data (Full Mode)

* **Certifications**: Professional certifications and licenses
* **Projects**: Personal and professional projects
* **Publications**: Research papers and articles
* **Patents**: Intellectual property and inventions
* **Recommendations**: Received professional recommendations

### Contact Information (Email Mode)

* **Emails**: Professional email addresses
* **Company Websites**: Associated company domains

## Limitations

* Maximum 1000 profiles per request (`maxItems` limit)
* Search results depend on LinkedIn's search algorithm
* Some profile data may be limited by privacy settings
* Email discovery success varies by profile privacy settings
* Rate limiting applies to prevent excessive API usage

## Error Handling

### Common Error Scenarios

#### Invalid Filter Values

Ensure all filter IDs match the supported values:

* Experience IDs: `"1"` through `"5"`
* Seniority IDs: `"100"`, `"110"`, `"120"`, `"130"`, `"200"`, `"210"`, `"220"`, `"300"`, `"310"`, `"320"`
* Function IDs: `"1"` through `"25"`
* Industry IDs: Various values (see industries list)

#### Invalid Scraper Mode

The `profileScraperMode` must be one of: `"Short"`, `"Full"`, or `"Full + email search"`.

#### Exceeding Limits

* `maxItems` cannot exceed 1000
* Large requests may timeout or be rate limited

#### Missing Required Parameters

* `searchQuery` is required for all searches
* `type` must be `"searchbyprofile"`

## Notes

* All filter arrays accept multiple values for OR logic
* String filters (locations, companies, etc.) support partial matching
* Boolean filters (`recentlyChangedJobs`) work as exact matches
* Pagination is supported via `startPage` parameter
* Results are returned in LinkedIn's default sort order


# Reddit
Source: https://developers.gopher-ai.com/docs/data/reddit



All job types use the `/api/v1/search/live` POST endpoint and follow the same structure: a top-level `type` to define the data source, and a set of `arguments` defining the parameters of the job.

<CardGroup cols={2}>
  <Card title="URL Scraping" icon="link">
    Extract content from specific Reddit posts and comments
  </Card>

  <Card title="Post Search" icon="magnifying-glass">
    Search across Reddit posts and comments
  </Card>

  <Card title="User Search" icon="user">
    Find and retrieve information about Reddit users
  </Card>

  <Card title="Community Search" icon="users">
    Discover Reddit communities (subreddits)
  </Card>
</CardGroup>

### `scrapeurls`

Retrieves the complete content of one or more Reddit URLs. This job type is specifically designed for extracting data from Reddit posts and comments.

**Supported URLs:**

* Post URLs: `https://reddit.com/r/{community}/comments/{post_id}/...`
* Comment URLs: `https://reddit.com/r/{community}/comments/{post_id}/{title}/{comment_id}/...`

**Not Supported:**

* User profile URLs
* Community/subreddit URLs
* Reddit homepage or feed URLs

### `searchposts`

Performs comprehensive searches across Reddit posts and comments based on keywords or phrases.

### `searchusers`

Searches for Reddit user profiles and optionally retrieves their post history.

### `searchcommunities`

Discovers Reddit communities (subreddits) based on search queries.

## Parameters

### Common Parameters

| Parameter      | Type    | Required | Default       | Description                                                                             |
| -------------- | ------- | -------- | ------------- | --------------------------------------------------------------------------------------- |
| `type`         | string  | Yes      | -             | The job type: `scrapeurls`, `searchposts`, `searchusers`, or `searchcommunities`        |
| `sort`         | string  | No       | `"relevance"` | Sort order for results. Options: `relevance`, `hot`, `top`, `new`, `rising`, `comments` |
| `include_nsfw` | boolean | No       | `false`       | Whether to include NSFW (Not Safe For Work) content                                     |
| `after`        | string  | No       | -             | ISO8601 timestamp to filter results created after this date/time                        |
| `max_items`    | integer | No       | `10`          | Maximum number of items to load in server cache                                         |
| `max_results`  | integer | No       | `10`          | Maximum number of results to return per page                                            |
| `next_cursor`  | string  | No       | -             | Pagination cursor for retrieving subsequent pages                                       |

### Job-Specific Parameters

#### `scrapeurls` Parameters

| Parameter | Type           | Required | Default | Description                                 |
| --------- | -------------- | -------- | ------- | ------------------------------------------- |
| `urls`    | array\[string] | Yes      | -       | Array of Reddit post/comment URLs to scrape |

#### Search Job Parameters (`searchposts`, `searchusers`, `searchcommunities`)

| Parameter | Type           | Required | Default | Description             |
| --------- | -------------- | -------- | ------- | ----------------------- |
| `queries` | array\[string] | Yes      | -       | Array of search queries |

#### Additional Search-Specific Parameters

| Parameter         | Type    | Applies To          | Default | Description                          |
| ----------------- | ------- | ------------------- | ------- | ------------------------------------ |
| `skip_posts`      | boolean | `searchusers`       | `false` | Skip user posts when searching users |
| `max_posts`       | integer | `searchposts`       | `10`    | Maximum posts per page               |
| `max_comments`    | integer | `searchposts`       | `10`    | Maximum comments per page            |
| `max_communities` | integer | `searchcommunities` | `2`     | Maximum communities per page         |
| `max_users`       | integer | `searchusers`       | `2`     | Maximum users per page               |

## Examples

### `scrapeurls`

Scrape Reddit URLs

```json  theme={null}
{
  "type": "reddit",
  "arguments": {
    "type": "scrapeurls",
    "urls": [
      "https://reddit.com/r/ArtificialIntelligence/comments/1n1dwzv/what_math_should_i_focus_on_for_ai_and_why/",
      "https://reddit.com/r/programming/comments/abc123/example_post/"
    ],
    "sort": "new",
    "include_nsfw": false,
    "max_items": 100
  }
}
```

### `searchposts`

Search Reddit posts

```json  theme={null}
{
  "type": "reddit",
  "arguments": {
    "type": "searchposts",
    "queries": [
      "machine learning",
      "artificial intelligence"
    ],
    "sort": "top",
    "after": "2024-01-01T00:00:00Z",
    "max_items": 100,
    "max_results": 20,
    "max_posts": 10,
    "max_comments": 10
  }
}
```

### `searchusers`

Search Reddit users

```json  theme={null}
{
  "type": "reddit",
  "arguments": {
    "type": "searchusers",
    "queries": [
      "NASA",
      "SpaceX"
    ],
    "sort": "relevance",
    "skip_posts": false,
    "max_items": 50,
    "max_users": 5
  }
}
```

### `searchcommunities`

Search Reddit communities

```json  theme={null}
{
  "type": "reddit",
  "arguments": {
    "type": "searchcommunities",
    "queries": [
      "artificial intelligence",
      "machine learning"
    ],
    "sort": "relevance",
    "max_items": 100,
    "max_results": 10,
    "max_communities": 5
  }
}
```

## Pagination

The Reddit API supports cursor-based pagination for handling large result sets. After receiving your initial results, use the `next_cursor` value from the response to fetch subsequent pages:

```json  theme={null}
{
  "type": "reddit",
  "arguments": {
    "type": "searchposts",
    "queries": ["technology"],
    "max_results": 20,
    "next_cursor": "eyJvZmZzZXQiOjIwfQ=="
  }
}
```

## Best Practices

### 1. Efficient Querying

* Use specific search terms to get more relevant results
* Combine related queries in a single request when possible
* Set appropriate `max_*` parameters based on your needs

### 2. Rate Limiting

* Be mindful of API rate limits
* Implement exponential backoff for retries
* Cache results when appropriate

### 3. Content Filtering

* Use the `after` parameter to get only recent content
* Set `include_nsfw` appropriately for your use case
* Use `skip_posts` when you only need user profile information

### 4. URL Validation

* Ensure URLs are properly formatted Reddit URLs before scraping
* Validate that URLs point to posts or comments, not profiles or communities

## Error Handling

Common error scenarios and how to handle them:

### Invalid URL Format

When using `scrapeurls`, ensure all URLs follow the correct Reddit URL pattern for posts and comments.

### Missing Required Parameters

* `scrapeurls` requires `urls` array
* All search operations require `queries` array
* These parameters are mutually exclusive

### Invalid Job Type

Ensure the `type` parameter is one of: `scrapeurls`, `searchposts`, `searchusers`, or `searchcommunities`.

## Response Structure

While the exact response structure may vary, typical responses include:

* **Posts**: Title, content, author, timestamp, score, comments
* **Comments**: Content, author, timestamp, score, parent context
* **Users**: Username, karma, account age, recent posts (unless `skip_posts` is true)
* **Communities**: Name, description, subscriber count, creation date

## Limitations

* URL scraping is limited to posts and comments only
* Search results are subject to Reddit's search algorithm
* NSFW content is excluded by default
* Maximum limits apply to all result types to prevent excessive API usage


# Similarity Search
Source: https://developers.gopher-ai.com/docs/data/similarity



<CardGroup cols={2}>
  <Card title="What is Indexed Search?" icon="database">
    The indexed search endpoints provide powerful vector-based search over previously collected X/Twitter data. Unlike live search, indexed search queries our secure vector database for lightning-fast results with no polling required.
  </Card>

  <Card title="Key Features" icon="stars">
    • Instant results from vector database

    • Semantic similarity matching

    • Hybrid search combining keywords and vectors

    • Secure TEE-protected data storage
  </Card>

  <Card title="Search Types" icon="magnifying-glass">
    • **Similarity Search**: Semantically related content

    • **Hybrid Search**: Combine semantic and keyword matching
  </Card>

  <Card title="Use Cases" icon="lightbulb">
    Perfect for applications needing:

    * Fast historical data access
    * Semantic content discovery
    * Pattern and trend analysis
  </Card>
</CardGroup>

## Indexed Search (Similarity & Hybrid)

The indexed search endpoints allow you to query **stored data** via a powerful vector-based search engine. These endpoints are ideal for applications that require high-speed access to previously fetched data, enabling use cases like:

* Semantic similarity detection
* Hybrid relevance ranking
* Near-instant responses (no polling or job queues)

<Info>
  All searches operate over tweets already collected and stored in the Gopher vector store (Milvus).
</Info>

## Similarity Search (X/Twitter)

Use this endpoint to perform semantic searches on previously indexed X/Twitter data. Results are ranked based on vector similarity.

### Endpoint

```http  theme={null}
POST /v1/search/similarity
```

### Authentication

* **Type:** Bearer Token
* **Header:** `Authorization: Bearer <API_KEY>`

### Request Parameters

The request parameters allow you to customize your semantic search query. The `query` field is used to find semantically similar content, while `sources` specifies which data sources to search. Optional `keywords` can filter results, and you can use `keyword_operator` to control how multiple keywords are combined. Use `max_results` to limit the number of returned items.

| Field             | Type           | Required | Description                                                                                |
| ----------------- | -------------- | -------- | ------------------------------------------------------------------------------------------ |
| query             | string         | ✅        | The semantic search text                                                                   |
| sources           | string\[]      | ❌        | Data sources to search: "twitter", "web", "tiktok". If not specified, searches all sources |
| keywords          | string\[]      | ❌        | Keywords to filter results                                                                 |
| keyword\_operator | "and" \| "or"  | ❌        | Default: "and"                                                                             |
| max\_results      | number (1-100) | ❌        | Defaults to environment variable or 10                                                     |

### Example Request

The following example demonstrates how to perform a semantic search for content related to "open source LLM models", filtered to include tweets containing either "AI" or "NLP" keywords. The request limits results to 10 tweets and uses the "or" operator to match tweets containing any of the specified keywords.

```bash  theme={null}
curl https://data.gopher-ai.com/api/v1/search/similarity \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "query": "bitcoin",
    "sources": [
      "twitter"
    ],
    "keywords": [
      "price",
      "$BTC",
      "sentiment"
    ],
    "keyword_operator": "and",
    "max_results": 50
  }'
```

### Example Response

```json  theme={null}
{
    "ID": "1234567890",
    "Content": "Open source LLMs like Falcon and LLaMA are changing the game.",
    "Metadata": null,
    "Score": 0.87
  },
```

<Note>
  Metadata will be included in future versions.
</Note>

## Hybrid Search (X/Twitter)

Hybrid search combines vector similarity and keyword-based full-text search for more powerful and flexible querying. By assigning weights to both semantic similarity and keyword matching, you can precisely tune how results are ranked. This allows you to balance finding content that is conceptually related (via vector similarity) with content containing specific keywords (via full-text search) to get the most relevant results for your use case.

### Endpoint

```http  theme={null}
POST /v1/search/hybrid/twitter
```

### Authentication

* **Type:** Bearer Token
* **Header:** `Authorization: Bearer <API_KEY>`

\###Request Structure

| Field                    | Type           | Required | Description                                                                |
| ------------------------ | -------------- | -------- | -------------------------------------------------------------------------- |
| similarity\_query.query  | string         | ✅        | Text for semantic matching                                                 |
| similarity\_query.weight | number (0–1)   | ✅        | Weight to apply to vector score                                            |
| text\_query.query        | string         | ✅        | Text for full-text keyword matching                                        |
| text\_query.weight       | number (0–1)   | ✅        | Weight to apply to text score                                              |
| sources                  | string\[]      | ❌        | Data sources: "twitter", "web", "tiktok". Defaults to all if not specified |
| keywords                 | string\[]      | ❌        | Optional filter keywords                                                   |
| keyword\_operator        | "and" \| "or"  | ❌        | Default: "and"                                                             |
| max\_results             | number (1–100) | ❌        | Defaults to env value or 10                                                |

Example request

```bash  theme={null}
curl https://data.gopher-ai.com/api/v1/search/hybrid \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "similarity_query": {
      "query": "artificial intelligence developments",
      "weight": 0.7
    },
    "text_query": {
      "query": "deepseek llama ChatGPT comparison",
      "weight": 0.2
    },
    "sources": [
      "twitter"
    ],
    "keywords": ["LLM", "Large"],
    "keyword_operator": "or",
    "max_results": 25
  }'
```

Example response

```bash  theme={null}
  {
    "ID": "987654321",
    "Content": "LLaMA-3 is being positioned as a serious open source ChatGPT competitor.",
    "Metadata": null,
    "Score": 0.91
  },
```

### Choosing Between Search Types

| Use Case                                   | Similarity Search | Hybrid Search |
| ------------------------------------------ | ----------------- | ------------- |
| Quick semantic matching without keywords   | ✅                 | —             |
| Blend semantic meaning with keyword search | —                 | ✅             |
| Fine-tune relevance scoring                | —                 | ✅             |
| Approximate text matching                  | ✅                 | —             |

<Note>
  Tips for Better Results

  • Use short, descriptive phrases in query
  fields.

  • Use max\_results: 1 for "I'm feeling lucky"
  style lookups.

  • Try Hybrid Search if you want balance
  between context and keyword targeting.
</Note>


# TikTok
Source: https://developers.gopher-ai.com/docs/data/tiktok



All job types use the `/api/v1/search/live` POST endpoint and follow the same structure: a top-level `type` to define the data source, and a set of `arguments` defining the parameters of the job.

<CardGroup cols={2}>
  <Card title="Query Search" icon="magnifying-glass">
    Search across TikTok videos by query
  </Card>

  <Card title="Trending Search" icon="bolt">
    Find videos that are trending
  </Card>

  <Card title="Transcription" icon="text">
    Transcribe TikTok videos
  </Card>
</CardGroup>

### `searchbyquery`

Searches TikTok content based on keywords, hashtags, or phrases. This job type allows you to find specific content across the TikTok platform.

**Use Cases:**

* Finding content about specific topics
* Discovering videos with certain hashtags
* Researching content trends for particular keywords

### `searchbytrending`

Retrieves trending TikTok content sorted by various engagement metrics. This job type helps you discover what's currently popular on TikTok.

**Use Cases:**

* Monitoring viral content
* Tracking engagement trends
* Analyzing popular content by country
* Understanding what content formats are gaining traction

## Parameters

### Common Parameters

| Parameter   | Type    | Required | Default | Description                                         |
| ----------- | ------- | -------- | ------- | --------------------------------------------------- |
| `type`      | string  | Yes      | -       | The job type: `searchbyquery` or `searchbytrending` |
| `max_items` | integer | No       | `10`    | Maximum number of items to return                   |

### `searchbyquery` Parameters

| Parameter | Type           | Required | Default | Description                                    |
| --------- | -------------- | -------- | ------- | ---------------------------------------------- |
| `search`  | array\[string] | Yes      | -       | Array of search queries (keywords or hashtags) |

### `searchbytrending` Parameters

| Parameter      | Type   | Required | Default | Description                                                                                 |
| -------------- | ------ | -------- | ------- | ------------------------------------------------------------------------------------------- |
| `sort_by`      | string | No       | `vv`    | Sort trending content by metric. See [Sort Options](#sort-options)                          |
| `period`       | string | No       | `7`     | Time period for trending content: `"7"` (week) or `"30"` (month)                            |
| `country_code` | string | No       | `"US"`  | Country code for regional trending content. See [Supported Countries](#supported-countries) |

## Sort Options

The `sort_by` parameter for trending searches supports the following values:

| Value     | Description                      | Best For                      |
| --------- | -------------------------------- | ----------------------------- |
| `vv`      | Sort by views/trending score     | Finding viral content         |
| `like`    | Sort by number of likes          | High engagement content       |
| `comment` | Sort by number of comments       | Discussion-generating content |
| `repost`  | Sort by number of reposts/shares | Highly shareable content      |

### `searchbyquery`

Searches for TikTok videos based on keywords and phrases

#### Basic Keyword Search

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbyquery",
    "search": ["AI", "crypto"],
    "max_items": 10
  }
}
```

#### Hashtag Search

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbyquery",
    "search": ["#artificialintelligence", "#machinelearning", "#tech"],
    "max_items": 25
  }
}
```

#### Multi-Topic Search

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbyquery",
    "search": [
      "sustainable fashion",
      "eco friendly clothing",
      "thrift haul"
    ],
    "max_items": 50
  }
}
```

### `searchbytrending`

Searches for TikTok videos based on popular trends

#### Most Shared Content (Last Week)

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbytrending",
    "sort_by": "repost",
    "period": "7",
    "max_items": 10
  }
}
```

#### Most Commented Videos (Last Month)

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbytrending",
    "sort_by": "comment",
    "period": "30",
    "max_items": 20
  }
}
```

#### Regional Trending Content

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbytrending",
    "sort_by": "vv",
    "period": "7",
    "country_code": "JP",
    "max_items": 15
  }
}
```

#### Most Liked Content by Country

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbytrending",
    "sort_by": "like",
    "period": "7",
    "country_code": "BR",
    "max_items": 25
  }
}
```

## Use Case Examples

### Content Research

Discover what topics are gaining traction:

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbyquery",
    "search": ["climate change", "sustainability", "green energy"],
    "max_items": 30
  }
}
```

### Competitive Analysis

Monitor trending content in specific markets:

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbytrending",
    "sort_by": "vv",
    "period": "7",
    "country_code": "KR",
    "max_items": 20
  }
}
```

### Engagement Analysis

Find highly engaging content formats:

```json  theme={null}
{
  "type": "tiktok",
  "arguments": {
    "type": "searchbytrending",
    "sort_by": "comment",
    "period": "30",
    "country_code": "US",
    "max_items": 50
  }
}
```

## Supported Countries

The TikTok API supports trending content from the following countries:

### Americas

* `US` - United States
* `CA` - Canada
* `BR` - Brazil

### Europe

* `GB` - United Kingdom
* `FR` - France
* `DE` - Germany
* `IT` - Italy
* `ES` - Spain
* `RU` - Russia
* `TR` - Turkey

### Asia-Pacific

* `AU` - Australia
* `JP` - Japan
* `KR` - South Korea
* `SG` - Singapore
* `MY` - Malaysia
* `ID` - Indonesia
* `TH` - Thailand
* `PH` - Philippines
* `VN` - Vietnam
* `TW` - Taiwan
* `IL` - Israel

### Middle East & Africa

* `AE` - United Arab Emirates
* `SA` - Saudi Arabia
* `EG` - Egypt

## Best Practices

### 1. Query Optimization

* Use specific keywords for targeted results
* Combine related search terms in a single request
* Include both general terms and specific hashtags

### 2. Trending Analysis

* Use `period: "7"` for current trends
* Use `period: "30"` for more stable trend analysis
* Compare different `sort_by` metrics to understand content performance

### 3. Regional Insights

* Always specify `country_code` when analyzing regional markets
* Compare trending content across different countries
* Consider cultural context when interpreting results

### 4. Data Collection

* Set appropriate `max_items` based on your analysis needs
* For comprehensive analysis, consider multiple queries with different parameters
* Cache results to avoid redundant API calls

## Response Data

While the exact response structure may vary, typical TikTok data includes:

* **Video Details**: Title, description, hashtags, duration
* **Engagement Metrics**: Views, likes, comments, shares/reposts
* **Creator Information**: Username, follower count
* **Timestamps**: Creation date, trending period
* **Media URLs**: Video URLs, thumbnail images
* **Audio Information**: Music/sound used in the video

## Limitations

* Search results are subject to TikTok's algorithm and availability
* Historical trending data is limited to 7 or 30-day periods
* Country-specific trending may not reflect global trends
* Content availability may vary by region
* Maximum item limits apply to prevent excessive API usage

## Error Handling

### Common Error Scenarios

#### Invalid Country Code

Ensure the `country_code` parameter uses one of the supported country codes listed above.

#### Invalid Sort Option

The `sort_by` parameter must be one of: `vv`, `like`, `comment`, or `repost`.

#### Invalid Period

The `period` parameter only accepts `"7"` or `"30"` as string values.

#### Missing Required Parameters

* `searchbyquery` requires the `search` array
* `searchbytrending` has all optional parameters but benefits from explicit configuration

## Notes

* The `vv` sort option represents TikTok's trending/view score algorithm
* Period values must be strings (`"7"` not `7`)
* Search queries can include hashtags with or without the `#` symbol
* Multiple search terms in the `search` array will return combined results


# Twitter
Source: https://developers.gopher-ai.com/docs/data/twitter



All job types use the `/api/v1/search/live` POST endpoint and follow the same structure: a top-level `type` to define the data source, and a set of `arguments` defining the parameters of the job.

<CardGroup cols={2}>
  <Card title="Tweet Search" icon="magnifying-glass">
    Search tweets using Twitter query syntax
  </Card>

  <Card title="User Profiles" icon="user">
    Get user profile information and timelines
  </Card>

  <Card title="Tweet Interactions" icon="people">
    Get replies, retweets, and specific tweets by ID
  </Card>

  <Card title="Social Networks" icon="users">
    Get followers, following, and trending topics
  </Card>

  <Card title="Media Content" icon="image">
    Extract media from tweets and profiles
  </Card>

  <Card title="Trending Topics" icon="trending-up">
    Discover trending topics and hashtags
  </Card>
</CardGroup>

## Overview

The Twitter API provides comprehensive access to Twitter data including tweets, user profiles, social networks, and trending content. It supports multiple authentication methods and offers extensive search and interaction capabilities.

### `twitter`

The primary job type for Twitter data extraction, automatically using the best available authentication method.

**Use Cases:**

* Social media monitoring and analysis
* Content discovery and research
* User engagement analysis
* Trend monitoring and hashtag tracking
* Media content extraction
* Social network analysis

## Parameters

### Common Parameters

| Parameter     | Type    | Required | Default | Description                                                |
| ------------- | ------- | -------- | ------- | ---------------------------------------------------------- |
| `type`        | string  | Yes      | -       | The operation type (see capabilities below)                |
| `query`       | string  | No       | -       | The query to execute (username, tweet ID, or search terms) |
| `max_results` | integer | No       | `1000`  | Maximum number of results to return (max 1000)             |
| `count`       | integer | No       | -       | Number of results per request (max 1000)                   |
| `next_cursor` | string  | No       | -       | Pagination cursor for retrieving subsequent pages          |
| `start_time`  | string  | No       | -       | ISO timestamp to filter results created after this time    |
| `end_time`    | string  | No       | -       | ISO timestamp to filter results created before this time   |

### Query Types

* **Search Queries**: Use Twitter search syntax for finding tweets
* **Usernames**: Use @username format for user-specific operations
* **Tweet IDs**: Use numeric IDs for specific tweet operations
* **User IDs**: Use numeric IDs for user profile operations

## Capabilities

### Tweet Operations

| Capability            | Description                                      | Query Type   | Max Results |
| --------------------- | ------------------------------------------------ | ------------ | ----------- |
| `searchbyquery`       | Search tweets using Twitter query syntax         | Search terms | 1000        |
| `searchbyfullarchive` | Search historical tweets (requires elevated API) | Search terms | 1000        |
| `getbyid`             | Get specific tweet by ID                         | Tweet ID     | 1           |
| `getreplies`          | Get replies to a specific tweet                  | Tweet ID     | 1000        |
| `getretweeters`       | Get users who retweeted a tweet                  | Tweet ID     | 1000        |
| `getmedia`            | Get media from tweets                            | Username/ID  | 1000        |
| `gettweets`           | Get tweets from user timeline                    | Username/ID  | 1000        |

### Profile Operations

| Capability        | Description                           | Query Type  | Max Results |
| ----------------- | ------------------------------------- | ----------- | ----------- |
| `searchbyprofile` | Get user profile information          | Username    | 1           |
| `getprofilebyid`  | Get user profile by user ID           | User ID     | 1           |
| `getfollowers`    | Get followers of a profile            | Username/ID | 1000        |
| `getfollowing`    | Get users that a profile is following | Username/ID | 1000        |

### Special Operations

| Capability  | Description                    | Query Type | Max Results |
| ----------- | ------------------------------ | ---------- | ----------- |
| `gettrends` | Get trending topics            | None       | Varies      |
| `getspace`  | Get Twitter Spaces information | Space ID   | 1           |

## Examples

### Tweet Search Operations

#### Basic Tweet Search

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyquery",
    "query": "gopher_ai",
    "max_results": 10
  }
}
```

#### Advanced Tweet Search with Time Filters

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyquery",
    "query": "artificial intelligence",
    "max_results": 100,
    "start_time": "2024-01-01T00:00:00Z",
    "end_time": "2024-01-31T23:59:59Z"
  }
}
```

#### Historical Tweet Search (Elevated API)

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyfullarchive",
    "query": "machine learning",
    "max_results": 50,
    "start_time": "2023-01-01T00:00:00Z"
  }
}
```

#### Get Specific Tweet

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getbyid",
    "query": "1881258110712492142"
  }
}
```

#### Get Tweet Replies

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getreplies",
    "query": "1234567890",
    "max_results": 20
  }
}
```

#### Get Tweet Retweeters

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getretweeters",
    "query": "1234567890",
    "max_results": 50
  }
}
```

#### Get User Timeline

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "gettweets",
    "query": "gopher_ai",
    "max_results": 50
  }
}
```

#### Get Media from Tweets

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getmedia",
    "query": "gopher_ai",
    "max_results": 25
  }
}
```

### Profile Operations

#### Get User Profile by Username

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyprofile",
    "query": "gopher_ai"
  }
}
```

#### Get User Profile by ID

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getprofilebyid",
    "query": "44196397"
  }
}
```

#### Get User Followers

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getfollowers",
    "query": "gopher_ai",
    "max_results": 200
  }
}
```

#### Get User Following

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getfollowing",
    "query": "gopher_ai",
    "max_results": 200
  }
}
```

### Special Operations

#### Get Trending Topics

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "gettrends"
  }
}
```

#### Get Twitter Space

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getspace",
    "query": "1YqKDqWqdPLsV"
  }
}
```

## Use Case Examples

### Social Media Monitoring

Monitor mentions and hashtags:

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyquery",
    "query": "#AI OR #MachineLearning OR #ArtificialIntelligence",
    "max_results": 100,
    "start_time": "2024-01-01T00:00:00Z"
  }
}
```

### Competitor Analysis

Analyze competitor mentions:

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyquery",
    "query": "@competitor OR competitor_name",
    "max_results": 50
  }
}
```

### Influencer Research

Research user engagement:

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "getfollowers",
    "query": "influencer_username",
    "max_results": 1000
  }
}
```

### Content Discovery

Find viral content:

```json  theme={null}
{
  "type": "twitter",
  "arguments": {
    "type": "searchbyquery",
    "query": "viral OR trending",
    "max_results": 200
  }
}
```

## Best Practices

### 1. Query Optimization

* Use specific keywords and hashtags for targeted results
* Combine multiple terms with OR/AND operators
* Use quotes for exact phrase matching
* Include relevant filters (lang:, from:, etc.)

### 2. Rate Limiting

* Be mindful of API rate limits
* Use appropriate `max_results` values
* Implement pagination with `next_cursor`
* Cache results when possible

### 3. Time Filtering

* Use `start_time` and `end_time` for historical data
* Consider timezone implications
* Use ISO 8601 format for timestamps

### 4. Result Management

* Set appropriate `max_results` based on your needs
* Use pagination for large result sets
* Consider the cost of high-volume requests

## Response Data

The Twitter API returns structured data including:

### Tweet Data

* **Content**: Tweet text, media, links
* **Metadata**: Timestamps, engagement metrics
* **User Info**: Author details and verification status
* **Interactions**: Reply, retweet, like counts

### Profile Data

* **Basic Info**: Username, display name, bio
* **Metrics**: Follower count, following count, tweet count
* **Verification**: Account verification status
* **Media**: Profile and banner images

### Social Network Data

* **Followers**: User profiles of followers
* **Following**: User profiles being followed
* **Engagement**: Interaction patterns and metrics

## Limitations

* Maximum 1000 results per request (`max_results` limit)
* Historical search requires elevated API access
* Some data may be limited by privacy settings
* Rate limiting applies to prevent excessive API usage
* Real-time data availability may vary

## Error Handling

### Common Error Scenarios

#### Invalid Query Format

Ensure queries follow Twitter search syntax:

* Use proper operators (OR, AND, NOT)
* Quote phrases that need exact matching
* Use valid hashtag and mention formats

#### Exceeding Limits

* `max_results` cannot exceed 1000
* `count` cannot exceed 1000
* Large requests may be rate limited

#### Missing Required Parameters

* `type` is required for all operations
* `query` is required for most operations (except `gettrends`)

#### Authentication Issues

* Some capabilities require specific authentication methods
* Elevated API access needed for historical search
* Rate limits vary by authentication type

## Notes

* All timestamps use ISO 8601 format
* Pagination is supported via `next_cursor` parameter
* Search results are returned in reverse chronological order
* Media URLs may have expiration times
* Some capabilities may require specific API access levels


# Web
Source: https://developers.gopher-ai.com/docs/data/web



All job types use the `/api/v1/search/live` POST endpoint and follow the same structure: a top-level `type` to define the data source, and a set of `arguments` defining the parameters of the job.

<CardGroup cols={2}>
  <Card title="URL Scraping" icon="link">
    Extract content from specific web pages as markdown
  </Card>

  <Card title="LLM Processing" icon="computer">
    Get LLM processed text from the scraped content
  </Card>

  <Card title="Multi-Page Crawling" icon="sitemap">
    Crawl multiple pages with configurable depth
  </Card>

  <Card title="Content Extraction" icon="file-text">
    Extract structured content and metadata
  </Card>
</CardGroup>

## Overview

The Web API provides comprehensive web scraping capabilities for extracting content from websites. It supports single-page scraping as well as multi-page crawling with configurable depth and page limits.

### `scraper`

Scrapes content from web pages and returns metadata, markdown, and LLM-processed text. This is the primary job type for web data extraction.

**Use Cases:**

* Content extraction and analysis
* Website documentation scraping
* Data collection from web sources
* Multi-page site crawling
* Content preprocessing for LLM analysis

## Parameters

### Common Parameters

| Parameter   | Type    | Required | Default | Description                                            |
| ----------- | ------- | -------- | ------- | ------------------------------------------------------ |
| `type`      | string  | Yes      | -       | The job type: `scraper`                                |
| `url`       | string  | Yes      | -       | The URL to scrape (must include http\:// or https\://) |
| `max_depth` | integer | No       | `0`     | Maximum crawl depth from the starting URL              |
| `max_pages` | integer | No       | `1`     | Maximum number of pages to scrape                      |

### URL Requirements

* **Scheme Required**: URLs must include `http://` or `https://` scheme
* **Valid Format**: Must be a properly formatted URL
* **Accessible**: The URL must be accessible and return valid content

### Crawling Behavior

* **Depth Control**: `max_depth` controls how many levels deep to crawl from the starting URL
* **Page Limits**: `max_pages` prevents excessive scraping by limiting total pages processed
* **Robots.txt**: Currently respects robots.txt by default (configurable)
* **Markdown Output**: Content is automatically converted to markdown format

## Examples

### Basic Single Page Scraping

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://docs.learnbittensor.org",
    "max_depth": 0,
    "max_pages": 1
  }
}
```

### Multi-Page Crawling

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://docs.learnbittensor.org",
    "max_depth": 2,
    "max_pages": 10
  }
}
```

### Deep Site Crawling

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://example.com/sitemap",
    "max_depth": 3,
    "max_pages": 50
  }
}
```

### Documentation Site Scraping

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://api.example.com/docs",
    "max_depth": 1,
    "max_pages": 25
  }
}
```

## Use Case Examples

### API Documentation Extraction

Extract complete API documentation from a documentation site:

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://docs.api.example.com",
    "max_depth": 2,
    "max_pages": 100
  }
}
```

### News Article Collection

Scrape multiple articles from a news site:

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://news.example.com/tech",
    "max_depth": 1,
    "max_pages": 20
  }
}
```

### Product Catalog Scraping

Extract product information from an e-commerce site:

```json  theme={null}
{
  "type": "web",
  "arguments": {
    "type": "scraper",
    "url": "https://shop.example.com/products",
    "max_depth": 2,
    "max_pages": 50
  }
}
```

## Best Practices

### 1. URL Validation

* Always include the full URL with `http://` or `https://`
* Test URLs manually before scraping to ensure accessibility
* Use specific URLs rather than redirects when possible

### 2. Depth and Page Limits

* Start with `max_depth: 0` for single pages
* Use `max_depth: 1-2` for most multi-page scenarios
* Set appropriate `max_pages` to avoid excessive resource usage
* Consider the site structure when setting depth limits

### 3. Performance Considerations

* Use `max_pages` to control resource usage
* Start with smaller limits and increase as needed
* Consider the target site's size and structure
* Be respectful of server resources

### 4. Content Quality

* Target specific sections or pages when possible
* Use appropriate depth to capture related content
* Consider the site's navigation structure

## Response Data

The Web API returns structured content including:

### Basic Information

* **URL**: The scraped page URL
* **Title**: Page title and metadata
* **Content**: Extracted text content
* **Markdown**: Formatted markdown version of the content

### Technical Details

* **HTTP Status**: Response status codes
* **Content Type**: MIME type of the content
* **Encoding**: Character encoding information
* **Links**: Internal and external links found

### Crawling Information

* **Depth**: How deep the page was found from the starting URL
* **Crawl Order**: Order in which pages were processed
* **Parent URLs**: Links that led to each page

## Limitations

* Maximum crawl depth is limited by the `max_depth` parameter
* Page limits are enforced by the `max_pages` parameter
* Some sites may block automated scraping
* JavaScript-heavy sites may not be fully scraped
* Rate limiting may apply to prevent excessive requests

## Error Handling

### Common Error Scenarios

#### Invalid URL Format

```json  theme={null}
{
  "error": "url must include a scheme (http:// or https://)"
}
```

#### URL Not Accessible

```json  theme={null}
{
  "error": "invalid URL format: [specific error details]"
}
```

#### Invalid Parameters

* `max_depth` must be non-negative
* `max_pages` must be at least 1
* `url` is required and cannot be empty

#### Missing Required Parameters

* `url` parameter is required for all scraping operations
* `type` must be `"scraper"`

## Notes

* All scraped content is automatically converted to markdown format
* The scraper respects robots.txt by default
* Multi-page crawling follows internal links found on pages
* External links are not followed during crawling
* Content extraction focuses on main page content, not navigation or ads


# null
Source: https://developers.gopher-ai.com/docs/gotrader/API-guide



# Gopher API Guide

Welcome to the Gopher API. This guide explains how to request API access, make authenticated calls, and interpret JSON responses.

***

<Steps>
  <Step title="Request an API Key">
    To begin using the Gopher API, request an API key from the Gopher team.

    ### How to Request

    Fill out the [API Key Request Form](https://docs.google.com/forms/d/e/1FAIpQLSc4tIzeDrV9La2DVx2ubX2fMabHp4T5OIXPuXBvdJr8pO0KGw/viewform?usp=publish-editor) to request access.

    The team will review your request and send you an API key.
  </Step>

  <Step title="Making API Requests">
    All requests require an `X-API-Key` header.

    ### cURL Request Template

    ```bash  theme={null}
    curl -H "X-API-Key: <API_KEY>" \
    "https://<GOPHER_URL>/api/market-data/candles?symbol=<EXCHANGE:ASSET_PAIR>&timeframe=<TIMEFRAME>&limit=<ROWS>"
    ```

    ### Parameters

    | Field     | Description             | Example         |
    | --------- | ----------------------- | --------------- |
    | symbol    | Exchange + asset pair   | BINANCE:BTCUSDT |
    | timeframe | Candle interval         | 15m, 1h, 1d     |
    | limit     | Number of rows (1-1000) | 100             |

    ### Supported Exchanges

    * ✅ BINANCE
    * ✅ COINBASE

    ### Supported Timeframes

    Only the following timeframes are accepted:

    `1m`, `5m`, `15m`, `1h`, `4h`, `1d`

    ### Row Limit

    * Maximum: 1000
    * Requests above this return an error.

    ### Example Error Response

    ```json  theme={null}
    {"status":"error","error":{"code":"INVALID_LIMIT","message":"Limit must be between 1 and 1000"}}
    ```
  </Step>

  <Step title="Understanding the JSON Response">
    ### Example Response

    ```json  theme={null}
    {
      "status": "success",
      "data": {
        "symbol": "COINBASE:ETHUSDT",
        "timeframe": "4h",
        "count": 1,
        "candles": [
          {
            "timestamp": 1762329600,
            "open": 3331.06,
            "high": 3331.06,
            "low": 3282.84,
            "close": 3295.26,
            "volume": 1201.34418178
          }
        ]
      }
    }
    ```

    ### Response Fields

    | Field      | Meaning                   |
    | ---------- | ------------------------- |
    | status     | success/error             |
    | symbol     | Exchange + pair           |
    | timeframe  | Candle timeframe          |
    | count      | Total rows returned       |
    | candles\[] | OHLCV objects + timestamp |
  </Step>
</Steps>

***

## Example API Calls

Below are example calls using the development environment:

### Binance — 15m, 100 candles

```bash  theme={null}
curl -H "X-API-Key: <API_KEY>" \
"https://gotrader.gopher-ai.com/api/market-data/candles?symbol=BINANCE:BTCUSDT&timeframe=15m&limit=100"
```

### Binance — 1h, 10 candles

```bash  theme={null}
curl -H "X-API-Key: <API_KEY>" \
"https://gotrader.gopher-ai.com/api/market-data/candles?symbol=BINANCE:BTCUSDT&timeframe=1h&limit=10"
```

### Coinbase — 4h, 20 candles

```bash  theme={null}
curl -H "X-API-Key: <API_KEY>" \
"https://gotrader.gopher-ai.com/api/market-data/candles?symbol=COINBASE:ETHUSDT&timeframe=4h&limit=20"
```

***

## Summary

* Request an API key from the Gopher team
* Authenticate using `X-API-Key`
* Query candles using `<EXCHANGE:PAIR>`
* Max 1000 rows per request


# Welcome to Gopher Network
Source: https://developers.gopher-ai.com/docs/intro



### Build AI applications and agents with ease

Gopher Network empowers developers to create, deploy, and scale next-generation AI applications and intelligent agents. Whether you're building innovative tools or contributing as a miner or validator, Gopher connects you with real-time data and powerful frameworks to bring your vision to life.

<img className="block dark:hidden" src="https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=4594431f8821275fd72f53bb4ea1ad24" alt="Hero Light" data-og-width="700" width="700" data-og-height="320" height="320" data-path="images/hero-light.svg" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?w=280&fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=3fdcfa26393a9a50b89cdd02bd8bacdf 280w, https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?w=560&fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=0dcc42bcfd85b9f00dfe3cd4cdc40ff4 560w, https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?w=840&fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=4469b79d7b51a875280e774d3c0cad47 840w, https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?w=1100&fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=95114645838355e5fda4fb1f1ba32654 1100w, https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?w=1650&fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=9c18f6ef3fce236abd9aa6267f68d028 1650w, https://mintcdn.com/masa/gLrbfgQXRJWG3sbW/images/hero-light.svg?w=2500&fit=max&auto=format&n=gLrbfgQXRJWG3sbW&q=85&s=ff82c698aaab2a930b73cd4a457591b9 2500w" />

<img className="hidden dark:block" src="https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=25711cc1f20e6e21122b019d26cf8252" alt="Hero Dark" data-og-width="2124" width="2124" data-og-height="963" height="963" data-path="images/AI-ready.png" data-optimize="true" data-opv="3" srcset="https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?w=280&fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=f2d57db29e910bb2ca4f84db4afd401b 280w, https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?w=560&fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=c5c0122904e9baecf6dd04c9d7dc1029 560w, https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?w=840&fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=f40cc57fc0b5244a44e92dbeb84c7251 840w, https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?w=1100&fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=8428fcf1e570f7a9e857b6f695847a23 1100w, https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?w=1650&fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=48fe52c51ba970cb3f51b02a9f504b30 1650w, https://mintcdn.com/masa/LZWz1UzBs0onM_se/images/AI-ready.png?w=2500&fit=max&auto=format&n=LZWz1UzBs0onM_se&q=85&s=7d58ac88b0de378120221dcb1e4e6888 2500w" />

**FOR DEVELOPERS**

<CardGroup cols={2}>
  <Card title="Real-Time data" icon="cubes" iconType="light" href="/docs/index-API/masa-api-search">
    Access real-time data streams from X/Twitter and the web through simple APIs to build smarter, data-driven AI applications and agents.
  </Card>

  <Card title="Gopher Data Dashboard" icon="cube" iconType="light" href="https://data.gopher-ai.com">
    Access real-time X/Twitter and web data through a unified API. Create an API key, explore endpoints, and start building powerful AI agents and applications.
  </Card>
</CardGroup>

**FOR MINERS & VALIDATORS**

<CardGroup cols={2}>
  <Card title="Subnet 42 Miner" icon="solar-system" iconType="light" href="/docs/subnet/intro-42">
    Earn TAO tokens on the Bittensor Subnet for completing data requests.
  </Card>

  <Card title="Subnet Validator" icon="circle-nodes" iconType="light" href="/docs/subnet/intro">
    Distribute work to miners and maintain the network’s reliability.
  </Card>
</CardGroup>


# Community
Source: https://developers.gopher-ai.com/docs/subnet/community



Join the **[Bittensor Discord server](https://discord.gg/bittensor)** and chat with us in the **[#gopher](https://discord.com/channels/799672011265015819/1275149537547391058)** channel to learn about the latest developments in our subnet.

<Info>
  You can also head to the **[Gopher Discord server](https://discord.gg/HyHGaKhaKs)** and join the **[#nodes-miners](https://discord.com/channels/848996376896602163/1247449212124401756)** channel to chat about all things Gopher.
</Info>


# Create Wallet
Source: https://developers.gopher-ai.com/docs/subnet/create-wallet



If you have not created a wallet for your miner or validator, create one now. If you have an existing wallet, you can skip this step and move on to the next section.

## Creating Your Wallet

Follow these steps to create your wallet:

<Steps>
  <Step title="Create a Cold Wallet">
    First, create a cold wallet for your `miner` or `validator`:

    <CodeBlock language="bash">
      {`btcli wallet new_coldkey --wallet.name miner`}
    </CodeBlock>

    <Note>
      Replace `miner` with `validator` if you're setting up a validator wallet.
    </Note>
  </Step>

  <Step title="Create a Hot Wallet">
    Next, create a hot wallet for your cold wallet:

    <CodeBlock language="bash">
      {`btcli wallet new_hotkey --wallet.name miner --wallet.hotkey default`}
    </CodeBlock>

    <Note>
      Make sure to use the same wallet name (miner/validator) that you used in Step 1.
    </Note>
  </Step>

  <Step title="Verify Your Wallets">
    Finally, verify that both wallets were created successfully:

    <CodeBlock language="bash">
      btcli wallet list
    </CodeBlock>

    You should see your `miner` or `validator` wallet listed, along with its associated hotkey (`default`).

    <Tip>
      Keep your wallet passwords safe! You'll need them for future operations.
    </Tip>
  </Step>
</Steps>


# Delegation
Source: https://developers.gopher-ai.com/docs/subnet/delegation



## Validator delegation using child hotkeys

This guide helps you delegate a portion of your validator stake to the official Gopher validator using a child hotkey. Delegating through child hotkeys enhances security, simplifies operations, and enables direct support of the Gopher network.

### What is a Child Hotkey?

A **child hotkey** is a hotkey that receives a delegated portion of stake from a **parent hotkey**. This lets you safely stake across subnets and validators without exposing your primary hotkey to risk.

### Why Delegate to the Gopher Validator?

<CardGroup cols={2}>
  <Card title="2% Take Rate" icon="percent">
    ✓ Keep 98% of your rewards
  </Card>

  <Card title="Trusted Infrastructure" icon="server">
    ✓ Run by Gopher core team\
    ✓ Enterprise-grade reliability\
    ✓ Professional monitoring
  </Card>

  <Card title="Simple Setup" icon="gear">
    ✓ Delegate in minutes\
    ✓ Clear documentation\
    ✓ Easy to get started
  </Card>

  <Card title="Scalable Design" icon="chart-line">
    ✓ Multi-subnet staking\
    ✓ Isolated operational risk\
    ✓ Flexible delegation
  </Card>
</CardGroup>

<Note>
  Gopher is currently one of the most delegator-friendly validators on Bittensor Subnet 42 with a 2% take rate, though this rate may be adjusted in the future.
</Note>

### 🪪 Gopher Validator Hotkey for Delegation

To delegate to the official Gopher Validator on Subnet 42, use this hotkey address:

```text  theme={null}
5CZv4oXgYsAFjJj8rLmmbLG29y7x9RGEc6hM9tzCwRU8NeDe
```

### 🛠️ Step-by-Step Guide

<Steps>
  <Step title="Register Your Child Hotkey">
    Run the following command to register your child hotkey:

    ```bash  theme={null}
    btcli register --wallet.name <child_wallet_name> --netuid 42
    ```
  </Step>

  <Step title="Delegate Stake to Gopher Validator">
    Execute this command to delegate your stake:

    ```bash  theme={null}
    btcli stake set_children \
      --netuid 42 \
      --children <your_child_hotkey_ss58> \
      --proportions 1.0 \
      --hotkey 5CZv4oXgYsAFjJj8rLmmbLG29y7x9RGEc6hM9tzCwRU8NeDe \
      --wallet.name <your_coldkey_name>
    ```
  </Step>
</Steps>

<Note>
  Tip: The `--proportions` flag should reflect the portion of stake you want to delegate (e.g., 0.5 for 50%).
</Note>

<Note>
  By default, the child hotkey starts with a take rate of **0%**, and the Gopher validator does **not** charge any fee. You keep 100% of your rewards.
</Note>

### View Your Delegation

Check that your delegation is active:

```bash  theme={null}
btcli stake get_children --netuid 42 --hotkey 5CZv4oXgYsAFjJj8rLmmbLG29y7x9RGEc6hM9tzCwRU8NeDe --all
```

## Update or Revoke Delegation

**Update proportions**:

```bash  theme={null}
btcli stake set_children \
  --netuid 42 \
  --children <your_child_hotkey> \
  --proportions <new_proportion> \
  --hotkey 5CZv4oXgYsAFjJj8rLmmbLG29y7x9RGEc6hM9tzCwRU8NeDe \
  --wallet.name <your_coldkey_name>
```

**Revoke delegation**:

```bash  theme={null}
btcli stake revoke_children --hotkey 5CZv4oXgYsAFjJj8rLmmbLG29y7x9RGEc6hM9tzCwRU8NeDe --netuid 42
```

***

### Optional: Set Take Rate for Your Child Hotkey

Control how much your child key keeps from rewards:

**Set take rate**:

```bash  theme={null}
btcli stake set_childkey_take --hotkey <your_child_hotkey> --netuid 42 --take 0.18
```

**Check take rate**:

```bash  theme={null}
btcli stake get_childkey_take --hotkey <your_child_hotkey> --netuid 42
```

*Note: You can only update the take rate once every 30 days.*

## Important Considerations

* The total proportion across child keys must be ≤ 1.0
* You must have registered your child hotkey on Subnet 42
* Minimum stake may apply depending on network settings
* Gopher validator’s default take rate is **0%**, so you keep **100% of your earnings**


# Fund Wallet
Source: https://developers.gopher-ai.com/docs/subnet/fund-wallet



## Get Tokens

### Testnet

To get tTAO to register your miner on testnet, head to **[Bittensor's Testnet Discord](https://discord.com/channels/799672011265015819/1190048018184011867)** channel and request tTAO! You will need to provide your miner's coldkey.

### Mainnet

You can purchase TAO from many exchanges, including **[Coin Marketcap](https://coinmarketcap.com/currencies/bittensor/#Markets)** and **[Changenow.io](https://changenow.io/?from=eth\&to=tao)**. Copy your wallet's coldkey and send TAO to the address!  Subnet registration is a variable cost, so be sure to factor this into your budget.

### Registration Costs

* 📊 View registration costs and network stats
* 📈 Monitor price fluctuations in real-time
* 💰 Plan your registration budget accordingly

<CardGroup cols={2}>
  <Card title="Subnet 42" href="https://taostats.io/subnets/42/metagraph">
    View registration costs and network stats for Masa's data scraping subnet
  </Card>
</CardGroup>


# Introduction
Source: https://developers.gopher-ai.com/docs/subnet/intro



Join the data revolution with the Gopher Bittensor Subnet 🎉, where miners use a secure Trusted Execution Environment (TEE) to extract AI-ready data with built-in integrity. By leveraging hardware-level security enclaves, this subnet delivers high-quality, real-time data with minimal latency and industry-leading security guarantees.

<Note>Before you can dive into Subnet 42, you must complete this Getting Started section. These steps are critical to setting up your environment and ensuring seamless interaction with the subnet.</Note>

## What you’ll accomplish here 🚀

This section will guide you through:

1. CLI Installation: follow the [Bittensor BTCLI docs](https://docs.bittensor.com/getting-started/install-btcli).
2. Wallet Configuration: Prepare your wallet for transactions and interactions within Gopher's ecosystem.


# Miners
Source: https://developers.gopher-ai.com/docs/subnet/miners



Miners use the Gopher Worker to scrape data from a secure TEE enclave. It uses the ego framework to build, run and sign the binary.

## Hardware Prerequisites

### Required Hardware

<CardGroup cols={2}>
  <Card title="🔒 Intel SGX Support" icon="microchip">
    * Intel SGX 2.0+ enabled CPU required
    * Hardware-level security enclave
    * Secure computation environment
  </Card>

  <Card title="💻 Compatible Processors" icon="server">
    * 11th Gen+ Intel Core series
    * Modern Intel Xeon processors
    * Must have SGX feature enabled in BIOS
  </Card>
</CardGroup>

<Note>
  Verify your CPU supports Intel SGX 2.0 or higher before setting up mining operations. The SGX feature must also be enabled in your system BIOS.
</Note>

### Cloud Provider Recommendations

For optimal TEE mining performance and reliability, we recommend using Microsoft Azure. This provider has been extensively tested and validated by our team to ensure seamless compatibility with Masa Network's TEE mining requirements:

<CardGroup cols={2}>
  <Card title="Microsoft Azure" icon="microsoft">
    ✓ Offers SGX-enabled virtual machines\
    ✓ Proven reliability for TEE workloads\
    ✓ Validated for Masa Network mining\
    ✓ Our recommended provider
  </Card>

  <Card title="Alibaba Cloud" icon="ban" color="red">
    ✗ Not supported for TEE mining\
    ✗ Uses outdated SGX API (v3)\
    ✗ Incompatible with Masa requirements\
    ✗ Please use Azure or other validated providers
  </Card>
</CardGroup>

### Azure instance capacity planning

When running TEE miners on Azure, it's important to understand the capacity limits to optimize costs and performance. Here's what we've validated:

<CardGroup cols={2}>
  <Card title="Instance Specifications" icon="server">
    Standard DC4s v2

    * 4 vCPUs
    * 16 GiB Memory
    * SGX-enabled
  </Card>

  <Card title="Miner Capacity" icon="chart-line">
    * Maximum 4 TEE miners per instance
    * 1 miner per vCPU recommended
    * Attempting to run >4 miners causes automatic termination
  </Card>
</CardGroup>

<Tip>
  To optimize costs, you can run multiple miners on a single Azure instance as long as you stay within the vCPU limits. This approach is more cost-effective than running individual instances for each miner.
</Tip>

## System Setup & Configuration

### Prerequisites

Before starting TEE mining operations, ensure you have:

<CardGroup cols={2}>
  <Card title="Docker Setup" icon="docker">
    ✓ Docker installed and running\
    ✓ Permissions to run containers
  </Card>

  <Card title="Network Keys" icon="key">
    ✓ BTCLI to installation to manage your wallet\
    ✓ Generate coldkey and hotkey mnemonics and register your miner
  </Card>
</CardGroup>

<AccordionGroup>
  <Accordion icon="code" title="Installing the Bittensor BTCLI">
    <Tabs>
      <Tab title="masOS and Linux">
        To install the BTCLI please follow the [Bittensor BTCLI docs](https://docs.bittensor.com/getting-started/install-btcli).
      </Tab>

      <Tab title="Windows">
        Note that there is limited support for Windows at the moment. We recommend using WSL2 to run the BTCLI.
        Please reference the Bittensor CLI where it covers windows installation  [here](https://docs.bittensor.com/getting-started/install-btcli).
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

<AccordionGroup>
  <Accordion icon="code" title="Installing Docker">
    <Tabs>
      <Tab title="Linux">
        To install Docker, follow these steps:

        **Linux:**

        ```bash  theme={null}
        # Install Docker using snap
        sudo snap install docker
        ```

        ```bash  theme={null}
        # Add user to docker group and set permissions
        sudo usermod -aG docker $USER
        ```

        After running these commands, restart your system for the changes to take effect.

        Verify installation by running:

        ```bash  theme={null}
        docker --version
        ```
      </Tab>

      <Tab title="Windows">
        For windows users, we recommend using WSL2 to run Docker. Please reference the Docker docs where it covers windows installation  [here](https://docs.docker.com/desktop/install/windows-install/).
        Then use the linux instructions above to install docker.
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

<AccordionGroup>
  <Accordion icon="code" title="Generate coldkey and hotkey mnemonics and register your miner">
    ### Creating your wallet mnemonics and register your miner

    Follow these steps to create your wallet keys and register as a miner on the network. You'll need to generate both a coldkey (for secure storage) and hotkey (for active mining operations).

    <Steps>
      <Step title="Create a Cold Wallet">
        First, create a cold wallet for your `miner`:

        <CodeBlock language="bash">
          {`btcli wallet new_coldkey --wallet.name miner`}
        </CodeBlock>
      </Step>

      <Step title="Create a Hot Wallet">
        Next, create a hot wallet for your cold wallet:

        <CodeBlock language="bash">
          {`btcli wallet new_hotkey --wallet.name miner --wallet.hotkey default`}
        </CodeBlock>

        <Note>
          Make sure to use the same wallet name (miner) that you used in Step 1.
        </Note>
      </Step>

      <Step title="Verify Your Wallets">
        Finally, verify that both wallets were created successfully:

        <CodeBlock language="bash">
          {`btcli w overview`}
        </CodeBlock>

        You should see your `miner` wallet listed, along with its associated hotkey (`default`).

        <Tip>
          Keep your wallet passwords safe! You'll need them for future operations.
        </Tip>
      </Step>

      <Step title="Register Your Node">
        Register your node as a miner on the subnet:

        ```sh  theme={null}
        btcli subnet register --netuid 42
        ```

        <Note>Replace the --netuid with 42 for mainnet or 165 for testnet</Note>
        This command will:

        * Configure your node as a miner
        * Set up the necessary network connections
      </Step>

      <Step title="Verify Registration">
        Check that your miner is properly registered on the subnet:

        ```sh  theme={null}
        btcli s metagraph --netuid 165 --network test
        ```

        <Note>Replace the --netuid with 42 for mainnet or 165 for testnet</Note>

        This command will display the subnet metagraph, showing:

        * All registered nodes and their roles
        * Your miner's status and registration
        * Current network topology

        <Note>
          Look for your hotkey address in the output to confirm successful registration. If you don't see your node listed, wait a few minutes and try again.
        </Note>
      </Step>
    </Steps>
  </Accordion>
</AccordionGroup>

<Note>
  Keep your coldkey and hotkey mnemonics secure and backed up. These are required for mining operations and cannot be recovered if lost.
</Note>

## Subnet 42 Miner setup

To begin mining on Subnet 42, follow these steps:

<Steps>
  <Step title="Clone Repository">
    First, clone the Subnet 42 repository which contains the docker compose file and .env example:

    ```sh  theme={null}
    git clone https://github.com/gopher-lab/subnet-42.git
    ```
  </Step>

  <Step title="Navigate to Directory">
    Change into the repository directory:

    ```sh  theme={null}
    cd subnet-42
    ```
  </Step>

  <Step title="Copy Environment File">
    Create your environment configuration file:

    ```sh  theme={null}
    cp .env.example .env
    ```
  </Step>

  <Step title="Edit Environment File">
    Wallet Setup - you can add either your mnemonics, or load your wallet from disk:

    ```bash  theme={null}
    COLDKEY_MNEMONIC="your_coldkey_mnemonic"
    HOTKEY_MNEMONIC="your_hotkey_mnemonic"
    ```

    or

    ```bash  theme={null}
    WALLET_NAME="your_wallet_name"
    HOTKEY_NAME="your_hotkey_name"
    WALLET_PATH="~/.bittensor"
    ```

    Next, add your role and network configuration:

    ```bash  theme={null}
    ROLE="miner"
    NETUID="42" #(165 for testnet)
    SUBTENSOR_NETWORK="finney" #(test for testnet)
    OVERRIDE_EXTERNAL_IP="your_external_ip_address"
    MINER_PORT="your_miner_port"
    LISTEN_ADDRESS=":your_tee_worker_port"
    MINER_TEE_ADDRESS="your_tee_worker_address:<your_tee_worker_port>"
    ```

    Finally, add credentials for your TEE worker:

    ```bash  theme={null}
    TWITTER_ACCOUNTS="username:password,username2:password2..." #(to scrape twitter with cookies)
    TWITTER_API_KEYS="key1,key2..." #(to scrape twitter with API keys)
    APIFY_API_KEY="key" #(to support Apify actors)
    CLAUDE_API_KEY="key" #(for web / llm processing)
    ```
  </Step>

  <Step title="Start Your Node">
    Start your node with Docker Compose:

    ```sh  theme={null}
    docker compose --profile miner up -d
    ```

    This will:

    * Launch the required containers for your node type
    * Initialize your miner node
    * Initialize your TEE worker
    * Connect to the subnet network
  </Step>

  <Step title="Verify TEE Worker">
    Once your node is running, verify the TEE worker is accessible by visiting your configured address in a browser:

    ```
    https://<your-ip-address>:8080
    ```

    You should see a response indicating the TEE worker is running. If you get a connection error:

    * Verify your IP address is correct in the .env file
    * Check that port 8080 is open in your firewall
    * Ensure the TEE worker container is running properly

    <Note>
      The browser may show a security warning due to the self-signed certificate. This is expected and you can proceed to verify the endpoint is responding.
    </Note>
  </Step>

  <Step title="Monitor Logs">
    Monitor your miner's logs to ensure everything is running properly.

    ```sh  theme={null}
    docker logs <container-name>
    ```

    Note that subnet miners run with two containers, one for the neuron (miner) and one for the worker (tee)
  </Step>
</Steps>

## Apify Support

Miners are able to support various Apify actors by setting an `APIFY_API_KEY` in your .env file.  This will give you default access to the Apify API.  Gopher supports the following actors - it is the miner's responsibility to rent the actor if needed!

* `trudax~reddit-scraper`
* `epctex~tiktok-search-scraper`
* `lexis-solutions~tiktok-trending-videos-scraper`
* `dusan.vystrcil~llm-dataset-processor`
* `kaitoeasyapi~premium-x-follower-scraper-following-data`
* `apify~website-content-crawler`

## LLM Support

Some jobs include additional LLM processing, such as `web` `scraper` jobs.  Miners may add either a `CLAUDE_API_KEY` or `GEMINI_API_KEY` to their .env file to support this type of work.

## Cookie Management

If you have `TWITTER_ACCOUNTS` defined in your .env file, you need to provide relevant cookies in the `.masa` folder of your TEE worker.  Each JSON file must be named with the `username` of the account and a `_twitter_cookies.json` suffix.(This will of course soon change to `.gopher`)
Gopher provides helpful tools via [Docker commands](https://github.com/gopher-lab/subnet-42/blob/main/docker-compose.yml), and a Python script built with Chromium [here](https://github.com/gopher-lab/subnet-42/blob/main/scripts/cookie_grabber.py).

## Upgrading your miner

<Steps>
  <Step title="Stop current miner">
    ```sh  theme={null}
    docker compose -p <container-name> down
    ```
  </Step>

  <Step title="Start new version">
    ```sh  theme={null}
    docker compose --profile miner up -d
    ```

    The latest docker image will automatically be pulled and used, as defined in the docker-compose.yml file.
  </Step>
</Steps>

### Frequently Asked Questions

<AccordionGroup>
  ### ⚙️ Setup & Configuration

  <Accordion title="Can I run multiple miners on the same instance?">
    Yes, multiple miners can run on the same machine. However, each miner must have its own TEE worker. A single TEE worker cannot be shared across multiple miners.
  </Accordion>

  <Accordion title="How many miners can I run per instance?">
    The number of miners per instance depends on the system resources and performance considerations. Each miner needs its own TEE worker, so the setup must account for computational capacity.
  </Accordion>

  <Accordion title="Can I use a single scraper for multiple miners?">
    No, the scraper (worker) is tied to a specific miner's TEE worker. Each miner must have its own dedicated scraping logic within its TEE environment.
  </Accordion>

  <Accordion title="Do I need separate Twitter/X accounts per miner?">
    Yes, but a miner can have multiple Twitter accounts/API keys assigned. It can be X/Twitter accounts you scrape and/or X/Twitter API accounts. We recommend the Twitter API Pro tier for full-archive search. or a combination of tiers for higher stability.
  </Accordion>

  <Accordion title="Can I authenticate via Twitter/X API instead of premium account scraing with logins?">
    Yes, you can authenticate via the X/Twitter API instead of relying on premium account logins, which can be unstable. However, premium accounts remain an option.
  </Accordion>

  <Accordion title="Should I invest in a Twitter API subscription?">
    Yes, it is recommended to consider Twitter/X API Pro or a combination of tiers. This provides better access to full-archive search, improves data reliability, and reduces the risk of scraping failures.
  </Accordion>

  <Accordion title="Will miners be able to share TEE workers in the future?">
    No, when moving to mainnet, miners will not be able to share TEE worker instances. This is possible only in the testnet environment and not recommended.
  </Accordion>

  <Accordion title="How does Twitter/X data access affect performance?">
    Both stable authentication and reliable data access are critical for miner performance and rewards. Your miner's ability to consistently return high-quality Twitter/X data impacts scoring. Poor performance can result from:

    * Authentication failures
    * Slow response times
    * Missing or incomplete data
    * Rate limiting issues
    * API quota exhaustion

    We recommend using the Twitter API with the following tiers:

    * **Basic Tier (\$100/month)**
      * Good starting point for testing
      * Limited to 500k tweets/month
      * Recent search only (7 days)
    * **Pro Tier (\$5000/month)**
      * Full archive search access
      * Up to 1M tweets/month
      * Higher rate limits
      * More reliable for production

    You can use a combination of tiers to achieve the best performance and stability. You also have the option to use premium accounts with logins but can be more unstable..

    A well-optimized setup with reliable API access or stable premium accounts, combined with efficient data retrieval and processing, is key to maintaining high performance scores and maximizing rewards.
  </Accordion>
</AccordionGroup>


# Register Wallet
Source: https://developers.gopher-ai.com/docs/subnet/register-wallet



<Tabs>
  <Tab title="Miner registration">
    Register your `miner` to the subnet:

    <Note>
      Replace `<netuid>` with the network id for the subnet.  Gopher testnet is `165` and mainnet is `42`.
    </Note>

    ```bash  theme={null}
    btcli subnet register --netuid <netuid> --wallet.name miner --wallet.hotkey default
    ```

    When prompted to continue, type `y` and press enter again. Confirm your password and press enter to continue. Wait for the miner to be registered to the subnet.

    <Warning>
      You may encounter an error about exceeding blocks. This means that many people have registered to the subnet in the last hour, and you need to wait. This is normal; wait for one tempo (approximately 1 hour).
    </Warning>
  </Tab>

  <Tab title="Validator registration">
    Register your `validator` to the subnet:

    <Note>
      Replace `<netuid>` with the network id for the subnet.  Gopher testnet is `165` and mainnet is `42`.
    </Note>

    ```bash  theme={null}
    btcli subnet register --netuid <netuid> --wallet.name validator --wallet.hotkey default
    ```

    When prompted to continue, type `y` and press enter again. Confirm your password and press enter to continue. Wait for the validator to be registered to the subnet.

    <Warning>
      You may encounter an error about exceeding blocks. This means that many people have registered to the subnet in the last hour, and you need to wait. This is normal; wait for one tempo (approximately 1 hour).
    </Warning>
  </Tab>
</Tabs>


# Release Notes
Source: https://developers.gopher-ai.com/docs/subnet/release-notes



To view the latest release notes and updates for Subnet 42, please visit our [GitHub Releases page](https://github.com/gopher-lab/subnet-42/releases). All new features, improvements, and bug fixes are documented there as they are published.


# Scoring
Source: https://developers.gopher-ai.com/docs/subnet/scoring



## Overview

The Subnet 42 scoring system evaluates miner performance by analyzing telemetry data collected from their TEE (Trusted Execution Environment) workers. This scoring mechanism is designed to reward miners that successfully process data collection tasks.

### How telemetry data is collected and processed

Telemetry Data Sources: Each registered TEE worker periodically reports telemetry data that includes:

<CardGroup cols={2}>
  <Card title="Twitter Metrics" icon="twitter">
    ✓ Tweet collection statistics\
    ✓ Profile data retrieval metrics\
    ✓ API usage and rate limit tracking
  </Card>

  <Card title="Web Metrics" icon="globe">
    ✓ Success and failure counts for web scraping operations\
    ✓ Performance tracking across different target sites
  </Card>

  <Card title="TikTok Metrics" icon="tiktok">
    ✓ Success and failure counts for web scraping operations\
    ✓ Performance tracking across different target sites
  </Card>

  <Card title="Reddit Metrics" icon="reddit">
    ✓ Success and failure counts for web scraping operations\
    ✓ Performance tracking across different target sites
  </Card>

  <Card title="Error Monitoring" icon="triangle-exclamation">
    ✓ Authentication failures\
    ✓ Rate limit exceeded events\
    ✓ Network and connectivity issues\
    ✓ Other operational errors
  </Card>

  <Card title="Timing Data" icon="clock">
    ✓ Operation start/end timestamps\
    ✓ Processing duration metrics\
    ✓ Interval between data collections
  </Card>
</CardGroup>

### Scoring Algorithm

<Steps>
  <Step title="Telemetry Data Collection">
    Analyzes changes in miner performance metrics over time using delta-based calculations
  </Step>

  <Step title="Extract & Normalize">
    Standardizes raw metrics into comparable values across different data types
  </Step>

  <Step title="Apply Weighting">
    Uses source-based weighting to reward miners based on the demand they fulfill
  </Step>

  <Step title="Calculate Scores">
    Combines weighted metrics into comprehensive performance scores
  </Step>

  <Step title="Kurtosis Weighting">
    A custom kurtosis function weights top performers more heavily in the final scoring calculations.
  </Step>

  <Step title="Generate Weights">
    Converts final scores into network weight allocations for rewards
  </Step>
</Steps>

## Conclusion

For detailed information about the specific weights and scoring parameters each data source, check the [configuration file](https://github.com/gopher-lab/subnet-42/blob/main/validator/platform_config.py), which contains the exact weight values used by the subnet validators.


# Validators
Source: https://developers.gopher-ai.com/docs/subnet/validators



Validators on Subnet 42 run in a secure TEE (Trusted Execution Environment) enclave to validate and verify data from miners. This ensures secure and tamper-proof validation of network data through hardware-level security.

## Hardware Prerequisites

### Required Hardware

<CardGroup cols={2}>
  <Card title="🔒 Intel SGX Support" icon="microchip">
    * Intel SGX 2.0+ enabled CPU required
    * Hardware-level security enclave
    * Secure computation environment
  </Card>

  <Card title="💻 Compatible Processors" icon="server">
    * 11th Gen+ Intel Core series
    * Modern Intel Xeon processors
    * Must have SGX feature enabled in BIOS
  </Card>
</CardGroup>

<Note>
  Verify your CPU supports Intel SGX 2.0 or higher before setting up mining operations. The SGX feature must also be enabled in your system BIOS.
</Note>

### Cloud Provider Recommendations

For optimal TEE mining performance and reliability, we recommend using Microsoft Azure. This provider has been extensively tested and validated by our team to ensure seamless compatibility with Gopher Network's TEE mining requirements:

<CardGroup cols={2}>
  <Card title="Microsoft Azure" icon="microsoft">
    ✓ Offers SGX-enabled virtual machines\
    ✓ Proven reliability for TEE workloads\
    ✓ Validated for Masa Network mining\
    ✓ Our recommended provider
  </Card>
</CardGroup>

## System Setup & Configuration

### Prerequisites

Before starting TEE validator operations, ensure you have:

<CardGroup cols={2}>
  <Card title="Docker Setup" icon="docker">
    ✓ Docker installed and running\
    ✓ Permissions to run containers
  </Card>

  <Card title="Network Keys" icon="key">
    ✓ BTCLI to installation to manage your wallet\
    ✓ Generate coldkey and hotkey mnemonics and register your validator
  </Card>
</CardGroup>

<AccordionGroup>
  <Accordion icon="code" title="Installing the Bittensor BTCLI">
    <Tabs>
      <Tab title="masOS and Linux">
        To install the BTCLI please follow the [Bittensor BTCLI docs](https://docs.bittensor.com/getting-started/install-btcli).
      </Tab>

      <Tab title="Windows">
        Note that there is limited support for Windows at the moment. We recommend using WSL2 to run the BTCLI.
        Please reference the Bittensor CLI where it covers windows installation  [here](https://docs.bittensor.com/getting-started/install-btcli).
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

<AccordionGroup>
  <Accordion icon="code" title="Installing Docker">
    <Tabs>
      <Tab title="Linux">
        To install Docker, follow these steps:

        **Linux:**

        ```bash  theme={null}
        # Install Docker using snap
        sudo snap install docker
        ```

        ```bash  theme={null}
        # Add user to docker group and set permissions
        sudo usermod -aG docker $USER
        ```

        After running these commands, restart your system for the changes to take effect.

        Verify installation by running:

        ```bash  theme={null}
        docker --version
        ```
      </Tab>

      <Tab title="Windows">
        For windows users, we recommend using WSL2 to run Docker. Please reference the Docker docs where it covers windows installation  [here](https://docs.docker.com/desktop/install/windows-install/).
        Then use the linux instructions above to install docker.
      </Tab>
    </Tabs>
  </Accordion>
</AccordionGroup>

<AccordionGroup>
  <Accordion icon="code" title="Generate coldkey and hotkey mnemonics and register your validator">
    ### Creating your wallet mnemonics and register your validator

    Follow these steps to create your wallet keys and register as a validator on the network. You'll need to generate both a coldkey (for secure storage) and hotkey (for active mining operations).

    <Steps>
      <Step title="Create a Cold Wallet">
        First, create a cold wallet for your `validator`:

        <CodeBlock language="bash">
          {`btcli wallet new_coldkey --wallet.name validator`}
        </CodeBlock>
      </Step>

      <Step title="Create a Hot Wallet">
        Next, create a hot wallet for your cold wallet:

        <CodeBlock language="bash">
          {`btcli wallet new_hotkey --wallet.name validator --wallet.hotkey default`}
        </CodeBlock>

        <Note>
          Make sure to use the same wallet name (validator) that you used in Step 1.
        </Note>
      </Step>

      <Step title="Verify Your Wallets">
        Finally, verify that both wallets were created successfully:

        <CodeBlock language="bash">
          {`btcli w overview`}
        </CodeBlock>

        You should see your `validator` wallet listed, along with its associated hotkey (`default`).

        <Tip>
          Keep your wallet passwords safe! You'll need them for future operations.
        </Tip>
      </Step>

      <Step title="Register Your Node">
        Register your node as a validator on the subnet:

        ```sh  theme={null}
        btcli subnet register --netuid 42
        ```

        <Note>Replace the --netuid with 42 for mainnet or 165 for testnet</Note>
        This command will:

        * Configure your node as a validator
        * Set up the necessary network connections
      </Step>

      <Step title="Verify Registration">
        Check that your validator is properly registered on the subnet:

        ```sh  theme={null}
        btcli s metagraph --netuid 165 --network test
        ```

        <Note>Replace the --netuid with 42 for mainnet or 165 for testnet</Note>

        This command will display the subnet metagraph, showing:

        * All registered nodes and their roles
        * Your validator's status and registration
        * Current network topology

        <Note>
          Look for your hotkey address in the output to confirm successful registration. If you don't see your node listed, wait a few minutes and try again.
        </Note>
      </Step>
    </Steps>
  </Accordion>
</AccordionGroup>

<Note>
  Keep your coldkey and hotkey mnemonics secure and backed up. These are required for mining operations and cannot be recovered if lost.
</Note>

### Optional: Delegate with Child Key

If you're running a validator, you can optionally delegate using a child hotkey for enhanced security. This allows you to:

* Keep your cold key secure
* Isolate operational risk
* Manage multiple delegations
* Set custom take rates (up to 18% of rewards)

<Note>
  The Gopher validator charges 2% take rate - you keep 98% of your rewards when delegating to us! See our [Validator Child Key Delegation Guide](/docs/subnet/validator-child-key) for detailed instructions on secure delegation with child hotkeys. The take rate is subject to change in the near future.
</Note>

## Subnet 42 Validator setup

To begin validating on Subnet 42, follow these steps:

<Steps>
  <Step title="Clone Repository">
    First, clone the Subnet 42 repository which contains the docker compose file and .env example:

    ```sh  theme={null}
    git clone https://github.com/gopher-lab/subnet-42.git
    ```
  </Step>

  <Step title="Navigate to Directory">
    Change into the repository directory:

    ```sh  theme={null}
    cd subnet-42
    ```
  </Step>

  <Step title="Copy Environment File">
    Create your environment configuration file:

    ```sh  theme={null}
    cp .env.example .env
    ```
  </Step>

  <Step title="Edit Environment File">
    Wallet Setup - you can add either your mnemonics, or load your wallet from disk:

    ```bash  theme={null}
    COLDKEY_MNEMONIC="your_coldkey_mnemonic"
    HOTKEY_MNEMONIC="your_hotkey_mnemonic"
    ```

    or

    ```bash  theme={null}
    WALLET_NAME="your_wallet_name"
    HOTKEY_NAME="your_hotkey_name"
    WALLET_PATH="~/.bittensor"
    ```

    Next, add your role and network configuration:

    ```bash  theme={null}
    ROLE="validator"
    NETUID="42" #(165 for testnet)
    SUBTENSOR_NETWORK="finney" #(test for testnet)
    VALIDATOR_PORT="your_validator_port"
    LISTEN_ADDRESS=":your_tee_worker_port"
    TELEMETRY_RESULT_WORKER_ADDRESS="your_tee_worker_address:<your_tee_worker_port>"
    ```
  </Step>

  <Step title="Start Your Node">
    Start your node with Docker Compose:

    ```sh  theme={null}
    docker compose --profile validator-tee up -d
    ```

    This will:

    * Launch the required containers
    * Initialize your validator node
    * Initialize your TEE worker
    * Connect to the subnet network
  </Step>

  <Step title="Verify TEE Worker">
    Once your node is running, verify the TEE worker is accessible by visiting your configured address in a browser:

    ```
    <TELEMETRY_RESULT_WORKER_ADDRESS>
    ```

    You should see a response indicating the TEE worker is running. If you get a connection error:

    * Verify your IP address is correct in the .env file
    * Check that your tee worker port is open in your firewall
    * Ensure the TEE worker container is running properly

    <Note>
      The browser may show a security warning due to the self-signed certificate. This is expected and you can proceed to verify the endpoint is responding.
    </Note>
  </Step>

  <Step title="Monitor Logs">
    Monitor your miner's logs to ensure everything is running properly.

    ```sh  theme={null}
    docker logs <container-name>
    ```

    Note that subnet validators run with two containers, one for the neuron (validator) and one for the worker (tee)
  </Step>
</Steps>

## Upgrading your validator

<Steps>
  <Step title="Stop current validator">
    ```sh  theme={null}
    docker compose -p <container-name> down
    ```
  </Step>

  <Step title="Start new version">
    ```sh  theme={null}
    docker compose --profile validator-tee up -d
    ```

    The latest docker image will automatically be pulled and used, as defined in the docker-compose.yml file.
  </Step>
</Steps>