Aevornix Strata

AevornixStrataBot

The web crawler for Aevornix Strata, the search engine built into Aevornix Browser. It fetches public pages to build indexes that are installed and searched on people's own devices.

Control crawling Request a removal

Robots compliantRFC 9309

One request at a timePer host

Curated scopeListed sites only

At a glance

Who operates the crawler and how to recognise it.

Operator: Aevornix, an operating division of Oxford Private Studios Ltd, company number 16271033, 20 Wenlock Road, London, England, N1 7GU
User agent: AevornixStrataBot/1.0 (+https://strata.aevornix.com)
Robots token: AevornixStrataBot
Request rate: At most one request in flight to a host, and at least one second between requests to it
Contact: enquiries@oxfordprivatestudios.com

Purpose

Strata indexes publicly accessible pages so people can search them. The index is packaged, signed by Aevornix and distributed with Aevornix Browser, and every search runs on the user's device.

Collects: What search needs: the address, title, description, headings, main text, language, publication date and links
Does not collect: Images, video, downloads or personal accounts. The crawler is not an archive.
Scope: A curated list of reference, public service, standards and technical documentation sites. The crawler does not follow links away from them; if your site is being crawled, it is on that list.

robots.txt

The crawler follows the Robots Exclusion Protocol, RFC 9309, and reads robots.txt before any other request to a host.

Fetch: /robots.txt comes first on every host, and the answer is kept for up to 24 hours.
Groups: The group naming AevornixStrataBot applies. When no group names it, the * group applies.
Precedence: The most specific rule wins, and Allow wins a tie. * and $ are supported.
Unavailable: A 4xx answer for robots.txt is treated as no restriction. A 5xx answer, or no answer, is treated as a complete disallow, and the host is not crawled.
Crawl delay: Crawl-delay is honoured. A delay above 30 seconds causes the host to be skipped.
Sitemaps: Sitemap lines are read to discover pages on the same host. Sitemap entries are still checked against robots.txt.

robots.txtBlock all crawling

User-agent: AevornixStrataBot
Disallow: /

Page controls

Signals on an individual page, honoured as the crawler reads it.

noindex: <meta name="robots" content="noindex"> or <meta name="AevornixStrataBot" content="noindex">: the page is fetched but not indexed.
nofollow: nofollow in either tag: links on the page are not followed.
Link rel: Links marked rel="nofollow", rel="ugc" or rel="sponsored" are not followed.
Canonical: A page that names another address on the same site as canonical is indexed at that address instead.

How it fetches

Deliberately slow, conditional and limited to what a search index needs.

Pace: One request at a time per host, at least one second apart.
Back off: On 429 or 503 it waits, honouring Retry-After up to five minutes, and it leaves the host after three failures in a row.
Revisits: Conditional: it sends If-None-Match and If-Modified-Since, and a 304 answer leaves the stored copy as it is.
Content: It skips addresses ending in image, media, archive, font, PDF and office document extensions, reads at most 5 MB of a response, and indexes HTML and plain text only.
Never: It does not run JavaScript, submit forms, sign in, store cookies, or follow addresses carrying a query string.
Redirects: It follows redirects itself, and checks each destination against robots.txt before fetching it.

Verification

AevornixStrataBot does not yet crawl from a published range of addresses, so it cannot be verified by IP address or reverse DNS. Anyone can send this user agent.

Behaviour: A request that ignores the behaviour described on this page is not from Aevornix.
Confirm a request: Write to enquiries@oxfordprivatestudios.com with the time of the request, the address requested and the source IP address. Aevornix keeps records of what the crawler fetched and when, and will check them.

Removal

Blocking the crawler in robots.txt, or marking a page noindex, removes the page from the next index Strata builds. Indexes already on people's devices are updated with the next index release, which carries signed removal records that Strata applies before it shows any result.

Personal data: dpo@oxfordprivatestudios.com
Legal: legal@oxfordprivatestudios.com for copyright, legal orders and other legal requests
Everything else: enquiries@oxfordprivatestudios.com, including crawl rate concerns
What to send: The exact address of each page. Aevornix does not charge for removal, and does not charge for inclusion.