Forums | MacLife

I'm currently working on a rewrite of the open source php search engine sphyder.

The main goal of the rewrite -

1) give it an option to use a sitemap.xml either to build initial list of pages to index or list of only pages to index (I'll have it just check the root level robots.txt file and only use sitemap files declared there)

2) port the backend indexer to DOMDocument parsing of a document.

3) port it use either flat file xml or database for customizations to the configuration. Right now, configuration file, which is parsed as php, has to be writeable by the web server in order for the admin interface to change some configurations, and that's bad. I'm not sure whether storing custom values in xml or database would be faster, I suspect database would be faster, but may be moot.

4) port the database to either pear::mdb2 or pdo, maybe both.

5) teach it to understand a new html attribute to tell it whether or not a portion of a page should be indexed.

Right now, #4 is done via very sphyder specific html comments.

<p>This content is indexed</p>
<!--sphider_noindex-->
<p>This content is not indexed</p>
<!--/sphider_noindex-->
<p>This content is again indexed</p>

It should be done via an attribute of the node.

<p>This content is indexed</p>
<p spider="off">This content is not indexed</p>
<p>This content is again indexed</p>

The spider attribute is custom, but I found no official attribute that does what it needs to do.
Adding custom attributes to xhtml is cake, you can just add it in the DOCTYPE node and it will validate.
Adding custom attributes to html requires using a custom DTD for it to validate.

Right now - I'm thinking about just adding it as a coreattrs attribute that can simply have a value of either on or off. Default value is on unless a parent node has set it to off.

I'm not sure that spider is the best thing to use as an attribute name, and would be willing to entertain alternative suggestions.

Also, I think a simple binary on/off would be sufficient, but would of course like to know if there are other values that should be considered?

I'm hoping that by having a generic attribute name, other search engines can be encouraged to update their indexing code and maybe eventually the attribute can become part of a W3C standard (probably too late for (X)HTML 5 consideration)

Any thoughts on either of those considerations?