The Web Scrape step fetches a single URL and extracts the page content along with metadata like title, description, author, and published date.

Fetch a URL and extract its content in your chosen format (Markdown, plain text, or HTML) along with page metadata. Markdown is the default and best format for feeding content into AI steps.

• Fetch article or blog post content to summarize or analyze with AI
• Pull product page details for competitive analysis
• Extract page content from URLs found in an RSS feed or spreadsheet
• Capture page metadata (title, description, author) for content cataloguing

Configure a URL (static or referenced from upstream data via {{stepId.fieldName}}). The step fetches the page, extracts the main content, and returns it in your chosen format. Output fields are locked: content, title, description, url (final URL after redirects), published_date, and author.

What format should I use for AI processing?

Markdown (the default). It preserves page structure without HTML noise, making it ideal for downstream AI steps.

Can I scrape pages behind login walls?

No. The step fetches publicly accessible URLs. Pages requiring authentication or with aggressive anti-bot measures may not return content.