Auto Proxy

Automatic proxy rotation with intelligent failover. Let Ujeebu automatically cycle through available proxies until your request succeeds.

Overview

Auto Proxy is a powerful feature that automatically rotates through available proxy servers when scraping websites. Instead of manually selecting a proxy and handling failures, enable auto_proxy=true and Ujeebu will:

1. Automatically select the next available proxy from the proxy pool
2. Retry failed requests with different proxies until one succeeds
3. Handle proxy failures transparently without returning errors
4. Fall back to direct connection if all proxies fail

TIP — When to Use Auto Proxy

Auto Proxy is ideal for:

• Scraping websites that block or rate-limit requests
• Handling geo-restricted content
• Improving success rates for difficult-to-scrape sites
• Avoiding IP bans from aggressive scraping

How It Works

When you enable auto_proxy=true, Ujeebu follows this retry logic:

Retry Flow

1. Select Next Proxy: Choose the next proxy from the configured proxy pool
2. Make Request: Attempt to scrape the URL using the selected proxy
3. Check Result:
   • ✅ Success (status 200-399): Return the response
   • ❌ Failure (error, timeout, or status ≥ 400 except 404): Retry with next proxy
4. Repeat steps 1-3 until:
   • A proxy succeeds, OR
   • All proxies have been tried
5. Final Fallback: If all proxies fail, make one final attempt without any proxy (direct connection)

What Triggers a Retry?

Auto Proxy retries when:

• Network errors occur (connection timeout, DNS failure, etc.)
• Response is null (no response received)
• Status code ≥ 400 (except 404 which is considered a valid "not found" response)
• Proxy connection fails (proxy unreachable, authentication failed, etc.)

What Doesn't Trigger a Retry?

Auto Proxy will NOT retry for:

• 404 Not Found - Valid response indicating the page doesn't exist
• 2xx or 3xx status codes - Successful responses

INFO — Smart Retry Logic

Auto Proxy is intelligent enough to distinguish between proxy failures and actual website responses. A 403 Forbidden or 500 Internal Server Error from the target website will trigger a retry, but a 404 Not Found will not.

Usage

Enable Auto Proxy by adding the auto_proxy=true parameter to any scrape request:

GET https://api.ujeebu.com/scrape?url=https://example.com&auto_proxy=true

curl -X GET 'https://api.ujeebu.com/scrape?url=https://example.com&auto_proxy=true' \
  -H "ApiKey: YOUR_API_KEY"

import { UjeebuClient } from '@ujeebu-org/ujeebu-sdk';

const client = new UjeebuClient(process.env.UJEEBU_API_KEY);

const response = await client.scrape({
  url: 'https://example.com',
  auto_proxy: true
});

console.log(response);

from ujeebu_python import UjeebuClient

ujeebu = UjeebuClient(api_key="YOUR_API_KEY")

response = ujeebu.scrape(
    url='https://example.com',
    auto_proxy=True
)

print(response)

package main

import (
	"fmt"
	"github.com/ujeebu/ujeebu-go"
)

func main() {
	client, _ := ujeebu.NewClient("YOUR-API-KEY")

	response, credits, err := client.Scrape(ujeebu.ScrapeParams{
		URL:       "https://example.com",
		AutoProxy: true,
	})
	if err != nil {
		panic(err)
	}

	fmt.Printf("Credits used: %d\n", credits)
	fmt.Println(response)
}

With Extract Rules

Auto Proxy works seamlessly with extract rules and other scraping features:

import { UjeebuClient } from '@ujeebu-org/ujeebu-sdk';

const client = new UjeebuClient(process.env.UJEEBU_API_KEY);

const rules = {
  title: { selector: 'h1', type: 'text' },
  price: {
    selector: '.price',
    type: 'text',
    transform: ['currency']
  },
  images: {
    selector: 'img.product',
    type: 'image',
    multiple: true
  }
};

const result = await client.scrapeWithRules(
  'https://example.com/product',
  rules,
  { auto_proxy: true }
);

console.log(result);

from ujeebu_python import UjeebuClient

ujeebu = UjeebuClient(api_key="YOUR_API_KEY")

rules = {
    'title': {'selector': 'h1', 'type': 'text'},
    'price': {
        'selector': '.price',
        'type': 'text',
        'transform': ['currency']
    },
    'images': {
        'selector': 'img.product',
        'type': 'image',
        'multiple': True
    }
}

result = ujeebu.scrape_with_rules(
    url='https://example.com/product',
    extract_rules=rules,
    params={'auto_proxy': True}
)

print(result)

curl -X GET 'https://api.ujeebu.com/scrape' \
  --data-urlencode 'url=https://example.com/product' \
  --data-urlencode 'auto_proxy=true' \
  --data-urlencode 'extract_rules={"title":{"selector":"h1","type":"text"},"price":{"selector":".price","type":"text","transform":["currency"]}}' \
  -H "ApiKey: YOUR_API_KEY"

Response Headers

When Auto Proxy is used, the response includes information about which proxy was successful:

{
  "success": true,
  "result": {
    "data": "...",
    "status": 200,
    "headers": {
      "content-type": "text/html",
      "x-proxy-used": "residential"
    }
  },
  "credits_used": 5
}

The x-proxy-used header (or similar proxy indicator in logs) tells you which proxy type successfully completed the request.

Best Practices

When to Use Auto Proxy

✅ Use Auto Proxy when:

• Scraping websites known to block requests
• You need high success rates for critical scraping tasks
• Dealing with rate-limited or geo-restricted content
• Scraping multiple pages from the same domain in succession

❌ Don't use Auto Proxy when:

• Scraping public, open APIs that don't require proxies
• Cost is a primary concern (auto proxy may use more credits due to retries)
• You have a specific proxy requirement (use manual proxy selection instead)
• Scraping simple, unrestricted content

Combining with Other Features

Auto Proxy works well with:

JavaScript Rendering:

{
  url: 'https://example.com',
  auto_proxy: true,
  js: true,
  wait_for: '.dynamic-content'
}

Screenshots:

{
  url: 'https://example.com',
  auto_proxy: true,
  screenshot: true,
  screenshot_fullpage: true
}

PDF Generation:

{
  url: 'https://example.com',
  auto_proxy: true,
  pdf: true
}

Performance Considerations

• Retry Overhead: Auto Proxy may take longer than a single request due to retries
• Credit Usage: Failed proxy attempts still consume credits
• Success Rate: Significantly higher success rates justify the additional cost

WARNING — Credit Consumption

Each proxy attempt (including failed ones) may consume credits. While Auto Proxy improves success rates, it may use more credits than a single successful request. Balance reliability needs with cost considerations.

Troubleshooting

All Proxies Failed

If all proxies fail and you still receive an error:

1. Check URL accessibility: Verify the URL is publicly accessible
2. Review rate limits: The target site might have aggressive blocking
3. Try without auto_proxy: Test if a direct connection works
4. Check proxy pool: Ensure your account has access to healthy proxies

Slow Response Times

If Auto Proxy requests are taking too long:

1. Reduce timeout values: Set lower timeouts to fail faster
2. Use specific proxy type: Instead of auto_proxy, choose the best proxy type manually
3. Check proxy health: Some proxies in the pool might be slow or unreachable

Unexpected Proxy Selection

Auto Proxy cycles through all available proxies in your pool. To use a specific proxy type:

// Instead of auto_proxy, use a specific proxy type
{
  url: 'https://example.com',
  proxy_type: 'residential',  // or 'premium', 'mobile', etc.
  proxy_country: 'us'
}

Parameters

INFO — Compatibility

Auto Proxy is compatible with all scrape endpoints and features including extract rules, screenshots, PDFs, and JavaScript rendering.

Examples

Basic Auto Proxy

curl -X GET 'https://api.ujeebu.com/scrape?url=https://difficult-site.com&auto_proxy=true' \
  -H "ApiKey: YOUR_API_KEY"

With JavaScript Rendering

response = ujeebu.scrape(
    url='https://spa-website.com',
    auto_proxy=True,
    js=True,
    wait_for='.content-loaded'
)

E-commerce Product Scraping

const rules = {
  product: {
    selector: '.product-detail',
    type: 'obj',
    children: {
      name: { selector: 'h1', type: 'text' },
      price: {
        selector: '.price',
        type: 'text',
        transform: ['currency', ['round', 2]]
      },
      availability: {
        selector: '.stock',
        type: 'text',
        transform: ['boolean']
      },
      images: {
        selector: 'img.gallery',
        type: 'image',
        multiple: true
      }
    }
  }
};

const result = await client.scrapeWithRules(
  'https://shop.example.com/product/123',
  rules,
  { 
    auto_proxy: true,
    js: true,
    wait_for: '.price'
  }
);

console.log(result.product);