StaticSearch v0.9.2 update

StaticSearch v0.9.2 was released on November 10, 2026. There should be no breaking changes but you should check your robots.txt and <meta> tags to ensure pages are indexed – or not indexed – as you expect.

Improved robots.txt parsing #

Previous releases of StaticSearch used simpler robots.txt parsing that incorrectly merged * and staticsearch user agent sections and only processed Disallow rules.

StaticSearch v0.9.2 parses robots.txt according to the Google specification processing rules:

1. StaticSearch parses any User-agent: staticsearch sections.

2. Where none exist, it parses any User-agent: * sections (* and staticsearch sections are not merged).

3. You can define any number of sections for the same user agent, e.g.

   robots.txt

   # both disallow rules applied
   User-agent: staticsearch
   Disallow: /tag/
   
   User-agent: staticsearch
   Disallow: /news/
   

4. You can specify more than one User-agent per section, e.g.

   robots.txt

   # applies to both agents
   User-agent: staticsearch
   User-agent: somebot
   Disallow: /news/
   

5. You can set * wildcard characters in Allow and Disallow paths, e.g. /*/2026*/.

6. You can set an explicit Allow and Disallow end of path markers with $, e.g. /news/$.

7. Allow and Disallow override each other when they are more explicit (have a longer rule length). Allow takes precedence when rule lengths are identical.

   robots.txt

   # disallow all paths except those starting /news/
   User-agent: staticsearch
   Allow: /news/
   Disallow: /
   

The following examples are also shown in the StaticSearch file indexing documentation.

Example 1 #

Do not index pages in the /secret/ and /personal/ paths:

robots.txt

# do not index pages with paths starting
# /secret/ and /personal/
User-agent: *
Disallow: /secret/
Disallow: /personal/

Example 2 #

Do not index pages in /personal/. StaticSearch indexes paths starting with /secret/ because only the User-agent: staticsearch section applies (it is not merged with the * section):

robots.txt

# non-staticsearch agents: ignored
User-agent: *
Disallow: /secret/

# staticsearch agent
# do not index /personal/
User-agent: staticsearch
Disallow: /personal/

Example 3 #

Do not index the root /news/ page but permit any sub-pages such as /news/post1/. $ indicates the explicit end of a path:

robots.txt

# do not index /news/ page only
# sub-pages of /news/ are indexed
User-agent: *
Disallow: /news/$

Example 4 #

Do not index pages starting /news/ except those where the path ends 2026*. * is a wildcard character:

robots.txt

# do not index /news/ pages
# unless it's a sub-page ending /2026*/, e.g. /news/2026-01-01/
User-agent: *
Disallow: /news/
Allow: /*/2026*/$

Improved meta tag parsing #

Previous releases of StaticSearch only parsed robots noindex meta tag rules and they could be applied even when in HTML <!-- --> comments.

StaticSearch v0.9.2 now parses robots <meta> tags according to the following rules:

1. <meta> tags only apply when robots.txt has not blocked page indexing.

2. StaticSearch only processes <meta> tags in the HTML <head> unless they are commented out with <!-- -->.

3. StaticSearch indexes a page unless content contains "noindex" in the robots or staticsearch meta tags:

   <meta name="robots" content="noindex,nofollow">
   <!-- AND/OR -->
   <meta name="staticsearch" content="noindex">
   

4. StaticSearch indexes a page when the more explicit staticsearch meta tag sets "index" irrespective of the robots meta tag:

   <!-- not indexed by search engines -->
   <meta name="robots" content="noindex">
   
   <!-- but is indexed by staticsearch -->
   <meta name="staticsearch" content="index">
   

Refer to the StaticSearch file indexing options documentation.

Get started #

The StaticSearch documentation provides a quick start guide, and details about the indexer, single script tag, web component,web component, bind module, and JavaScript API.

StaticSearch works well with Publican sites. The Publican documentation provides a quick start guide, a detailed set-up guide, API references, and common recipes you can use and adapt for your own projects.

• Get started
• Donate