Deploy to static hosting

A SpiderCaster site is a folder of plain files: pages with the style built in, an icon, and robots.txt. Any host that serves files can serve it. There is no server code, no database, and no login to keep up.

Before you upload

Drafts are built to stay private. Going live is a short checklist with one rebuild in it.

Pass the gate

Run the checker with --expect-noindex while the site is a draft. Nothing moves until it shows 0 fail.

Get the words approved

Pages are noindex until a person has read them. That is the default for every build.

Rebuild for search

Add --indexable to the build. Add --host with your address, such as https://example.com, to write sitemap.xml and point robots.txt at it.

Check again

python spidercaster.py check sites/my-site --expect-indexable

A build for search

With --indexable and --host, the build writes sitemap.xml and opens robots.txt to crawlers.

build output
$ python spidercaster.py build projects/my-site --offline --indexable --host https://example.com --out sites/my-site-live
built sites/my-site-live (mode: offline)
  pages: index.html, favicon.svg
  tokens: requests=0 ok=0 in=0 out=0 est_cost=$0.0000
  index: indexable
  robots: sites/my-site-live/robots.txt
  sitemap: sites/my-site-live/sitemap.xml
  next: python spidercaster.py check sites/my-site-live
        python spidercaster.py serve sites/my-site-live

Exit code: 0

The robots file after an indexable build
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Upload the folder

The engine does not upload anything. You copy the files with the tool your host already gives you.

What to copy

Every file in the site folder. site.json, report.json, and the _attempts folder are build records that no page links to, so you can leave them out.

Where it can go

Shared hosting through FTP or a file manager, a storage bucket set up to serve a website, or any static host.

Preview first

python spidercaster.py serve sites/my-site serves the folder on localhost so you can click through every page before upload.

Error pages

Version 0.1.0 does not write a 404 page. Make one as its own project, and on Apache point to it with an ErrorDocument line in .htaccess.

Publishing is a human step

No command in version 0.1.0 publishes a site. The deployer seat writes a release ticket and stops there, and a person makes the final move.

That is on purpose. A checked draft can still say something its owner did not mean, so the last look belongs to a person and not to the engine.

How this site ships

spidercaster.com is a real example of static hosting. Every page is built offline by SpiderCaster 0.1.0, one project per page, and checked by its own gate. The finished folder is served by MochaHost shared hosting, with cPanel and Apache, and Cloudflare sits in front as the CDN.

The host

MochaHost serves the plain files from the domain's web folder with Apache. No server code runs and there is no database.

The CDN

Cloudflare takes every request first, handles the secure connection, and passes it to the host.

Error pages

Apache points errors at the site's own pages with ErrorDocument lines in .htaccess, and keeps the real error status.

Draft rules

Until the words are signed off, every page carries noindex and robots.txt blocks crawlers.