Skip to content

Update README.md #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
NoahDrucker merged 1 commit into from
Jul 3, 2026
Merged

Update README.md #44

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 32 additions & 26 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,11 @@
<a href="https://dashboard.decodo.com/integrations?utm_source=github&utm_medium=social&utm_campaign=cli"> <img src="https://github.com/user-attachments/assets/a1e52a9e-3da1-4081-b3c6-053aafb8f196"/></a>
</p>

A web scraping CLI for the [Decodo Web Scraping API](https://decodo.com/scraping/web). Scrape websites, search engines, eCommerce platforms, and social media from your terminal without building proxy rotation, JavaScript rendering, anti-bot bypass, or CAPTCHA solving from scratch. Get structured data in JSON, NDJSON, Markdown, or PNG screenshots for shell scripts, CI/CD pipelines, data pipelines, and AI coding agents.
A web scraping CLI for the [Decodo Web Scraping API](https://decodo.com/scraping/web).

Scrape websites, search engines, eCommerce platforms, and social media from your terminal without building proxy rotation, JavaScript rendering, anti-bot bypass, or CAPTCHA solving from scratch.

Get data in JSON, NDJSON, Markdown, or PNG screenshots for shell scripts, CI/CD pipelines, data pipelines, and AI coding agents.

- Structured output in JSON, Markdown, NDJSON, and screenshots
- Server-side JavaScript rendering, anti-bot bypass, and CAPTCHA handling
Expand All @@ -16,7 +20,9 @@ A web scraping CLI for the [Decodo Web Scraping API](https://decodo.com/scraping

## What is the Decodo CLI?

The Decodo CLI is a web scraping CLI and command-line interface for the Decodo Web Scraping API. It wraps every API target as a subcommand and provides shell-native output modes for structured data extraction, scripting, shell automation, and AI agent workflows.
The Decodo CLI is a web scraping CLI and command-line scraper for the Decodo Web Scraping API.

It wraps every API target as a subcommand and provides shell-native output modes for structured data extraction, scripting, shell automation, and AI agent workflows.

Instead of maintaining scraping infrastructure, you get a single binary for reliable web data access from the terminal.

Expand All @@ -29,7 +35,9 @@ Use the CLI when you need web scraping outside an IDE or MCP client:
- **Coding agents**. Invoke scraping as a subprocess from Cursor, Claude Code, Codex, Gemini CLI, or Windsurf.
- **Quick terminal access**. Scrape websites, search the web, or capture screenshots with a single command.

For MCP-based scraping inside an IDE, see the [Decodo MCP server for IDE-based agent scraping](https://github.com/Decodo/mcp-server). The CLI is the right tool when you need direct shell access, scripting, or automation.
For MCP-based scraping inside an IDE, see the [Decodo MCP server for IDE-based agent scraping](https://github.com/Decodo/mcp-server).

The CLI is the right tool when you need direct shell access, scripting, or automation.

## Key features

Expand All @@ -39,32 +47,32 @@ For MCP-based scraping inside an IDE, see the [Decodo MCP server for IDE-based a

- **Built-in support for popular targets**. Tier-1 commands for scrape, search, and screenshot, plus schema-driven subcommands for Google Search and Bing, Amazon, Walmart, and Target, Reddit, TikTok, YouTube, and more. Run `decodo targets` to list everything available.

- **Global proxy infrastructure**. 115M+ residential IPs across 195+ locations with IP rotation, geo-targeting, and a 99.99% success rate on even the most protected targets, all via the Decodo Web Scraping API.
- **Global proxy infrastructure**. Access 115M+ residential IPs across 195+ locations with IP rotation and geo-targeting. All requests are powered by the Decodo Web Scraping API with a 99.99% success rate.

- **Pipe-friendly by design**. Compact JSON when piped, human-readable output in a TTY, and explicit exit codes for authentication, validation, rate limits, and API errors.

- **Fast time to value**. From API token to first scrape in minutes. Install with one command or use `npx` with zero setup.
- **Fast time to value**. From API token to your first scrape in minutes. Install with one command or use `npx` with zero setup.

## Use cases

Use the Decodo CLI when you need web scraping from the shell, structured data extraction in automation, reliable access to dynamic websites, or an alternative to building scraping infrastructure from scratch. Common scenarios:
Use the Decodo CLI when you need web scraping from the shell, structured data extraction in automation, reliable access to dynamic websites, or an alternative to building scraping infrastructure from scratch.

Common scenarios:

- **Shell scripts and CI/CD**. Scrape or search in pipelines without embedding SDK logic.
- **Data pipelines**. Stream NDJSON results into `jq`, databases, or downstream tools.
- **Coding agent subprocesses**. Let AI agents call `decodo search` or `decodo scrape` directly.
- **eCommerce intelligence**. Query Amazon, Walmart, and Target targets from the terminal.
- **Social media data collection**. Gather posts and metadata from Reddit, TikTok, and YouTube.
- **SERP monitoring**. Search Google and Bing programmatically with geo-targeting and parsed output.
- **SERP monitoring**. Search Google and Bing programmatically as a SERP scraper with geo-targeting and parsed output.

## Quick start

1. **Create a free account** at [dashboard.decodo.com](https://dashboard.decodo.com/) — up to 2K
free requests, no credit card required.
2. **Get your API key.** Obtain a Web Scraping API basic authentication token from the
[playground](https://dashboard.decodo.com/playground).
3. **Install Node.js 18+** from [nodejs.org](https://nodejs.org/) (required for npm/npx installs).
4. **Install the CLI** (pick one method below).
5. **Configure auth** and run your first scrape:
1. **Create a free account** at [dashboard.decodo.com](https://dashboard.decodo.com/) — get up to 2K free requests with no credit card required.
2. **Get your Web Scraping API token** from the Decodo [Playground](https://dashboard.decodo.com/playground).
3. **Install Node.js 18+** from [nodejs.org](https://nodejs.org/) (required for npm or `npx`).
4. **Install the CLI** using one of the methods below.
5. **Authenticate and run** your first scrape:

```bash
decodo setup
Expand Down Expand Up @@ -103,7 +111,7 @@ npx @decodo/cli scrape https://ip.decodo.com --token "$DECODO_AUTH_TOKEN"

## Authentication

Get a Basic auth token from the [Decodo playground](https://dashboard.decodo.com/playground).
Get a basic auth token from the Decodo [Playground](https://dashboard.decodo.com/playground).

```bash
# Interactive — saves token to config
Expand Down Expand Up @@ -132,16 +140,15 @@ decodo scrape https://ip.decodo.com
decodo google-search "top articles hacker news" --page-count 5 --parse
```

You should see markdown or parsed JSON within seconds. If you see an auth error, double-check your
token from the dashboard.
You should see Markdown or parsed JSON within seconds. If you see an auth error, double-check your token from the dashboard.

## Commands

### Tier-1 commands

| Command | Description |
| --- | --- |
| `decodo scrape <url>` | Scrape a URL (markdown by default) |
| `decodo scrape <url>` | Scrape a URL (Markdown by default) |
| `decodo search <query>` | Web search (`--engine google\|bing`, `--geo`, `--limit`) |
| `decodo screenshot <url>` | Capture a PNG screenshot (`-o` file or directory) |
| `decodo targets` | List all scrape targets by group |
Expand All @@ -151,7 +158,7 @@ token from the dashboard.

### Schema-driven target commands

Every API target is also available as its own subcommand (kebab-case name from `decodo targets`):
Every API target is also available as its own subcommand using the kebab-case name shown by `decodo targets`:

```bash
decodo google-search "decodo scraping api"
Expand All @@ -163,7 +170,7 @@ Use `decodo <target> --help` for target-specific flags (`--parse`, `--geo`, and

## Output modes

By default, scrape commands print the first result's `content` (parsed JSON when the target supports `--parse`, markdown for `decodo scrape`).
By default, scrape commands print the first result's `content` (parsed JSON when the target supports `--parse`, Markdown for `decodo scrape`).

| Flag | Effect |
| --- | --- |
Expand All @@ -173,7 +180,7 @@ By default, scrape commands print the first result's `content` (parsed JSON when
| `-o, --output <path>` | Write to a file instead of stdout |
| `-v, --verbose` | Print debug logs to stderr |

**TTY vs pipe:** When stdout is a terminal, human-readable output is used where possible. When piped or redirected, raw bytes or compact JSON is written. Screenshot output must go to `-o` or a redirect — writing binary PNG to a TTY is rejected.
**TTY vs. pipe:** When stdout is a terminal, human-readable output is used where possible. When piped or redirected, raw bytes or compact JSON is written. Screenshot output must go to `-o` or a redirect — writing binary PNG to a TTY is rejected.

**NDJSON line contract:** With `--format ndjson`, stdout is one JSON object per API result line. Without `--full`, each line is that result's `content`. With `--full`, each line is the full result entry (e.g. `content`, `status_code`, `url`). There is no envelope-level `.results[]` on a single line — pipe each line through `jq` individually.

Expand Down Expand Up @@ -271,7 +278,7 @@ Run `decodo setup` or export `DECODO_AUTH_TOKEN`.

The [install script](https://decodo.github.io/cli/install.sh) configures your PATH automatically. If `decodo` isn't available, restart your terminal or re-run the install script. You can also use `npx @decodo/cli`.

**Validation / API errors**
**Validation or API errors**

Read the `Error:` message on stderr. Use `--full` to inspect the raw API response.

Expand Down Expand Up @@ -316,17 +323,16 @@ pnpm test

## Related repositories

- [Web Scraping API](https://github.com/Decodo/Web-Scraping-API)
- [Decodo MCP server](https://github.com/Decodo/mcp-server)
- [Decodo Web Scraping API](https://github.com/Decodo/Web-Scraping-API)
- [Decodo MCP server for IDE-based agent scraping](https://github.com/Decodo/mcp-server)
- [Decodo OpenClaw skill](https://github.com/Decodo/decodo-openclaw-skill)
- [Decodo SDK for TypeScript](https://github.com/Decodo/sdk-ts)


## Try it

Install the CLI and start scraping from your terminal in minutes.

[Start scraping for free](https://dashboard.decodo.com/) | [Web Scraping API documentation](https://help.decodo.com/docs/introduction)
[Start scraping for free](https://dashboard.decodo.com/) | [Web Scraping API documentation](https://help.decodo.com/docs/web-scraping-api-introduction)
| [Discord](https://discord.gg/Ja8dqKgvbZ)

## License
Expand Down
Loading