| Internet-Draft | aipref-autoctl | April 2025 |
| Peiyuan | Expires 10 October 2025 | [Page] |
This document specifies the core automation-preferences.txt protocol, a machine-readable document that defines server-side automation permissions with a focus on essential controls for AI and automation use cases. Unlike the traditional robots.txt, which governs crawling, this protocol addresses a broader range of automation activities while maintaining simplicity for initial implementation. It defines the fundamental file format, policy declarations, HTTP method restrictions, and purpose requirements. Advanced features are addressed in a separate extension specification.¶
This note is to be removed before publishing as an RFC.¶
The latest revision of this draft can be found at https://datatracker.ietf.org/doc/draft-liao-aipref-autoctl-core/. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-liao-aipref-autoctl-core/.¶
Discussion of this document takes place on the AI Preferences Working Group mailing list (mailto:[email protected]), which is archived at https://mailarchive.ietf.org/arch/browse/ai-control/. Subscribe at https://www.ietf.org/mailman/listinfo/ai-control/.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 10 October 2025.¶
Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
The evolution of web automation has outpaced the capabilities of existing standards such as robots.txt, which only provides for crawler permissions. This document introduces the core automation-preferences.txt protocol that enables server operators to explicitly define basic policies governing automated interactions. These core policies cover fundamental aspects of automation control, including HTTP method restrictions and automation purpose declarations.¶
The automation-preferences.txt protocol is designed with extensibility in mind, allowing future enhancements while maintaining backward compatibility. This specification focuses on essential controls to facilitate initial adoption and implementation, with more advanced features defined in a separate extension document.¶
The automation-preferences.txt protocol applies to automated systems interacting with web servers, especially those driven by foundation models or other types of advanced AI models. It is designed to benefit both content owners-by allowing them to specify acceptable automation behaviors-and developers of automated systems, who can use these directives to ensure compliance.¶
This document defines the core functionality of the automation-preferences.txt protocol. A separate document, "Protocol Extension for Advanced AI Automation Control," extends this core specification with additional directives and capabilities. The extension specification builds upon this core document without modifying its requirements, providing a path for progressive implementation.¶
Implementations conforming to only this core specification are considered compliant with the automation-preferences.txt protocol. The extension specification defines optional enhancements that may be implemented once the core functionality is established.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The following terms are used in this document:¶
All terminology defined in this core specification applies to the extension specification without redefinition. The extension specification may introduce additional terms for concepts not covered in this document.¶
The automation-preferences.txt file MUST be hosted at the root of the domain, in the same manner as robots.txt. The file is structured as a series of key-value pairs that specify automation permissions.¶
The file MUST be served with the text/plain MIME type. Lines beginning with the hash symbol (#) are considered comments and MUST be ignored by parsers. Each directive consists of a field name, followed by a colon, followed by a value. Multiple values MAY be separated by commas.¶
Parsers MUST silently ignore any directives they do not recognize. This enables future extensions to add new capabilities without breaking compatibility with existing implementations.¶
A top-level directive, AutomationPolicy, indicates the overall stance
of the server regarding automated interactions and state-changing requests.
The following values are defined:¶
If the AutomationPolicy directive is not present, clients SHOULD assume a
default value of "limited".¶
Example:¶
AutomationPolicy: limited
The protocol MUST explicitly list allowed and disallowed HTTP methods using
the AllowedMethods and DisallowedMethods directives. Typically, GET and
HEAD are permitted while methods such as POST, PUT, DELETE, and PATCH are
disallowed for automated processing.¶
If no HTTP method directives are specified, clients SHOULD assume that only GET and HEAD methods are permitted.¶
Example:¶
AllowedMethods: GET, HEAD DisallowedMethods: POST, PUT, DELETE, PATCH
The protocol enables automation clients to declare their intended usage purposes. This is managed using the following directives:¶
RequireAutomationPurpose: Boolean value indicating whether clients must
declare a purpose for automation.¶
AllowedPurposes: Comma-separated list of permitted purposes using standardized
vocabulary terms.¶
DisallowedPurposes: Comma-separated list of prohibited purposes.¶
The specific vocabulary terms for automation purposes are intentionally not defined in this protocol specification. Instead, this protocol provides a mechanism for expressing allowed and disallowed purposes, which can be populated with terms from any widely accepted vocabulary standard. This approach ensures that the protocol remains flexible and can adapt to evolving vocabulary standards while maintaining the essential structure for purpose declarations.¶
Example:¶
RequireAutomationPurpose: true AllowedPurposes: [PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2] DisallowedPurposes: [PLACEHOLDER_PURPOSE3]
In a manner similar to robots.txt, the automation-preferences.txt file is divided into groups, each of which applies to a specific subset of content. Each group begins with one or more scope directives that define the target of the preferences. The following directives MAY be used within a group:¶
Scope: Specifies the URL pattern (e.g., /admin/) to which the group
applies. Wildcards MAY be used to indicate variable components of the URL.¶
Host: Specifies a subdomain or host. If present, the group applies only
to the indicated subdomain; if omitted, the group is assumed to apply to the
entire host.¶
Groups are processed in order of specificity. When multiple groups match a given request, the group with the longest matching Scope directive SHALL take precedence.¶
Example:¶
# Group 1: Applies to the entire site Host: example.com Scope: / AutomationPolicy: limited AllowedMethods: GET, HEAD DisallowedMethods: POST, PUT, DELETE, PATCH # Group 2: Specific preferences for the /admin/ path Host: example.com Scope: /admin/ AutomationPolicy: strict AllowedMethods: GET DisallowedMethods: POST, PUT, DELETE, PATCH
The automation-preferences.txt protocol is designed for extensibility. The protocol defines a forward-compatible approach where implementations:¶
This enables future extensions to add new capabilities while maintaining compatibility with implementations of this core specification. Extended directives may include:¶
Servers implementing this protocol SHOULD:¶
Clients consuming this protocol SHOULD:¶
Implementations SHOULD cache the automation-preferences.txt file to reduce server load, but SHOULD NOT cache it for longer than 24 hours to ensure timely policy updates.¶
The use of machine-readable automation policies introduces security considerations that must be addressed by implementations:¶
This document has no IANA actions.¶
The following is an example of a automation-preferences.txt file that adheres to this specification:¶
# Automation preferences for example.com # Version: 1.0 # Last updated: 2025-04-08 # Group 1: Applies to the entire site Host: example.com Scope: / AutomationPolicy: limited AllowedMethods: GET, HEAD DisallowedMethods: POST, PUT, DELETE, PATCH RequireAutomationPurpose: true AllowedPurposes: [PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2] DisallowedPurposes: [PLACEHOLDER_PURPOSE3] ContactEmail: [email protected] # Group 2: Specific preferences for the /admin/ path Host: example.com Scope: /admin/ AutomationPolicy: strict AllowedMethods: GET DisallowedMethods: POST, PUT, DELETE, PATCH AllowedPurposes: [PLACEHOLDER_PURPOSE1] DisallowedPurposes: [PLACEHOLDER_PURPOSE2], [PLACEHOLDER_PURPOSE3]