Octocom
AI Knowledge & Logic

Website Sources

List, add, remove and rescrape the website pages scraped into the knowledge base through MCP or REST, and keep them in sync with an external feed of links.

Website sources are the pages Octocom scrapes into a business's knowledge base. Each source is one URL. Scraping turns the page into searchable knowledge for the bot; removing a source also removes the knowledge scraped from it.

Every source has an origin:

OriginMeaning
manualAdded one by one, from the dashboard, MCP, REST or a feed sync. Once scraped, it refreshes about once a day
sitemapBelongs to a sitemap configured in the dashboard. Its weekly sitemap check adds, removes and rescrapes them

And a status:

StatusMeaning
scrapedThe last scrape succeeded at lastScrapedAt
not_scrapedNo scrape has succeeded yet. The page may still be in the queue, or fetching it failed (for example a 403, a 404 or a redirect loop). A page that has never scraped is not retried until you rescrape it

URLs are stored exactly as given. https://shop.example/a and https://shop.example/a/ are two different sources. Only http and https URLs are accepted.

MCP tools

ToolPurpose
list_website_sourcesList sources by URL with origin and status. Pass nextCursor back as cursor to page
add_website_sourcesAdd up to 500 pages and queue a scrape for each new one. Existing URLs are not rescraped
set_website_sourcesMake the manual sources exactly this list (up to 2,000) in one call; dryRun: true previews
remove_website_sourcesRemove sources by ids and/or exact urls, with their scraped knowledge
rescrape_website_sourcesQueue a fresh scrape of up to 100 existing sources

REST operations

All paths below start with /rest/v1 and require X-API-Key. Use the REST reference for complete schemas. Copilot and MCP can discover these endpoints in the Website Sources group with list_rest_api_operations.

MethodPathPurpose
GET/website-sources?businessSlug=shopList sources; optional origin, limit (up to 500) and cursor
POST/website-sourcesAdd {businessSlug, urls}; returns added and alreadyPresent URLs
PUT/website-sourcesSet the exact manual list {businessSlug, urls, dryRun?}
POST/website-sources/removeRemove {businessSlug, ids?, urls?}; returns removed and notFound
POST/website-sources/rescrapeRescrape {businessSlug, ids?, urls?}; returns queued and notFound

Sources are scoped to the API key's organization and the requested business. IDs or URLs from another business are reported as notFound and never touched. Archived businesses remain readable but reject changes with 409, and their pages are not refreshed.

Scrapes run in a shared, rate-limited queue. A manual page that fails during its daily refresh keeps its last good content and is tried again a day later. Add or rescrape only pages whose content you actually need refreshed.

Keep website knowledge in sync with a feed

The set operation (PUT /website-sources or set_website_sources) compares the list you send with the business's current manual sources:

  • URLs that are new are added and scraped.
  • Manual sources missing from the list are removed, together with their scraped knowledge.
  • Everything else is left untouched and is not rescraped.

Sitemap sources are never added or removed by this call. A URL that already exists as a sitemap source counts as present. An empty list is rejected, so a failed feed download cannot wipe the knowledge base by accident.

A Python recurring job can run the sync daily with its injected OCTOCOM_ORG_API_KEY and OCTOCOM_API_BASE_URL; no extra secret is needed. Fetch and validate the whole feed first, and stop without changing anything when it looks wrong:

import xml.etree.ElementTree as ET
import requests

FEED_URL = "https://shop.example/feeds/links.xml"
BUSINESS_SLUG = "shop"

def run_job(context):
    response = requests.get(FEED_URL, timeout=30)
    response.raise_for_status()
    links = [el.text.strip() for el in ET.fromstring(response.content).iter("LINK") if el.text and el.text.strip()]
    if not links:
        raise RuntimeError("Feed returned no links; keeping the current sources")

    result = requests.put(
        f"{OCTOCOM_API_BASE_URL}/rest/v1/website-sources",
        headers={"X-API-Key": OCTOCOM_ORG_API_KEY},
        json={"businessSlug": BUSINESS_SLUG, "urls": links},
        timeout=60,
    )
    result.raise_for_status()
    diff = result.json()["data"]
    return {"added": len(diff["added"]), "removed": len(diff["removed"]), "unchanged": diff["unchanged"]}

Run it once with "dryRun": True to check the difference before the first real sync. A second run against an unchanged feed reports zero additions and removals.

On this page

MCP toolsREST operationsKeep website knowledge in sync with a feed