Protocol Extension for Advanced Automation Control

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 10 October 2025.¶

Copyright Notice

Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶

Table of Contents

1. Introduction

This document extends the automation-preferences.txt protocol defined in "Protocol for Basic Automation Control" [CORE-SPEC] by introducing advanced directives and capabilities for more sophisticated control over automated interactions. These extensions address complex automation scenarios while maintaining backward compatibility with implementations of the core specification.¶

The extensions defined in this document enable content providers to exercise more granular control over automated access, including rate limiting, specific technology restrictions, API usage policies, session validation requirements, and asset-level annotation methods. These capabilities are designed to complement the basic controls provided by the core specification, offering a progressive path to more comprehensive automation management.¶

2. Conventions and Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

This document uses the terminology defined in the automation-preferences.txt protocol [CORE-SPEC]. The following additional terms are introduced in this document:¶

• Rate limiting: Constraints on the frequency or concurrency of automated requests to prevent excessive server load.¶

• Automation technology: Specific tools or frameworks used for automation, such as headless browsers or browser automation protocols.¶

• XHR/Fetch: XMLHttpRequest or Fetch API calls performed programmatically.¶

• Session validation: Mechanisms to verify that automated requests are part of a legitimate user session.¶

• Asset annotation: Metadata embedded within HTML documents to specify automation policies for individual content elements.¶

3. Extended Protocol Specification

This section defines additional directives that extend the automation-preferences.txt protocol. These directives may be used alongside the core directives in any group within the automation-preferences.txt file.¶

RequestLimit: 60/minute
ConcurrentLimit: 5

AllowCDP: false
AllowHeadless: false
AllowSelenium: false
AllowPuppeteer: false
AllowPlaywright: false

APIAutomation: with-key-only
AllowXHR: read-only
DisallowFetchFrom: /account/*, /checkout/*, /admin/*

RequireHumanInitiatedSession: true
SessionValidation: cookie-based
SessionTTL: 1h
RequireUserAgent: true

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "AutomationPolicyAnnotation",
  "automationPolicy": "limited",
  "allowCDP": false,
  "allowHeadless": false,
  "automationPurpose": {
    "require": true,
    "allowed": [[PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]],
    "disallowed": [[PLACEHOLDER_PURPOSE3]]
  },
  "contactEmail": "[email protected]"
}
</script>

4. Backward Compatibility

The extensions defined in this document maintain backward compatibility with implementations of the core specification. This compatibility is achieved through the following mechanisms:¶

• All directives defined in this document are OPTIONAL. Implementations that support only the core specification can safely ignore these directives, as specified in the core specification's extension mechanism.¶

• The extensions do not modify or override the behavior of any directives defined in the core specification.¶

• Extended directives enhance but do not replace core functionality.¶

Implementations supporting these extensions SHOULD degrade gracefully when interacting with servers or clients that support only the core specification:¶

• Servers supporting extensions SHOULD still process all core directives correctly, even if extended directives are also present.¶

• Clients supporting extensions SHOULD still honor all core directives, even if they do not recognize extended directives in a file.¶

• When HTML asset annotations are not supported by a client, the client SHOULD fall back to the site-level automation-preferences.txt file for guidance.¶

This approach ensures that the introduction of extensions does not break existing implementations while providing a path for enhanced functionality.¶

5. Implementation and Enforcement

Servers implementing the extensions defined in this document SHOULD:¶

• Employ detection mechanisms (e.g., CDP fingerprinting, headless browser detection) to identify automated clients using specific technologies.¶

• Implement rate limiting according to the specified directives.¶

• Validate sessions as required by the session requirement directives.¶

• Process HTML asset annotations when interpreting automation policies for specific content.¶

• Respond with appropriate HTTP status codes for non-compliant requests, such as:¶

  • 429 Too Many Requests for rate limit violations.¶

  • 403 Forbidden for unauthorized automation technology use.¶

  • 401 Unauthorized for missing or invalid authentication.¶

Clients supporting these extensions SHOULD:¶

• Honor rate limiting directives by self-throttling requests.¶

• Respect automation technology restrictions by avoiding prohibited tools.¶

• Adhere to API and XHR permissions as specified.¶

• Establish and maintain valid sessions when required.¶

• Parse and respect HTML asset annotations when present.¶

Both servers and clients MAY implement additional detection and enforcement mechanisms beyond those explicitly described in this document, as long as they maintain compatibility with the specified directives.¶

6. Security Considerations

In addition to the security considerations mentioned in the core specification, the extensions defined in this document introduce the following considerations:¶

• Rate Limiting: Implementations of rate limiting SHOULD use secure methods to track request counts and prevent circumvention through IP spoofing or other means.¶

• Technology Detection: Methods used to detect specific automation technologies MAY be circumvented by sophisticated clients. Servers SHOULD employ multiple detection approaches and adapt to evolving evasion techniques.¶

• Session Validation: Session validation mechanisms SHOULD be resistant to replay attacks and session hijacking attempts.¶

• HTML Asset Annotations: Parsing of JSON-LD annotations MUST be performed securely to prevent injection attacks or denial-of-service through malformed input.¶

The extensions provide more granular control over automated access, which can enhance security, but they also introduce complexity that may lead to misconfiguration. Implementers SHOULD carefully test and validate their configurations to ensure they provide the intended protections.¶

7. IANA Considerations

This document has no IANA actions.¶

8. Future Work

Future enhancements to the automation-preferences.txt protocol MAY include:¶

• Soliciting further feedback from browser vendors, content owners, AI model and automation tool developers.¶

• Developing reference implementations and comprehensive detection libraries.¶

• Formalizing the protocol in collaboration with the IETF and W3C.¶

• Expanding interoperability with related protocols for consistent content preference signaling across the web.¶

• Standardizing the HTML asset annotation schema through formal registration with schema.org or similar organizations.¶

Sample Extended automation-preferences.txt File

The following is an example of a automation-preferences.txt file that includes both core and extended directives:¶

# Automation preferences for example.com
# Version: 2.0
# Last updated: 2025-04-08

# Group 1: Applies to the entire site
Host: example.com
Scope: /
AutomationPolicy: limited
AllowedMethods: GET, HEAD
DisallowedMethods: POST, PUT, DELETE, PATCH
RequireAutomationPurpose: true
AllowedPurposes: [PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]
DisallowedPurposes: [PLACEHOLDER_PURPOSE3]
ContactEmail: [email protected]

# Extended directives
RequestLimit: 60/minute
ConcurrentLimit: 5
AllowCDP: false
AllowHeadless: false
AllowSelenium: false
AllowPuppeteer: false
AllowPlaywright: false
APIAutomation: with-key-only
RequireUserAgent: true
AllowXHR: read-only
DisallowFetchFrom: /account/*, /checkout/*, /admin/*
RequireHumanInitiatedSession: true
SessionValidation: cookie-based
SessionTTL: 1h

# Group 2: Specific preferences for the /admin/ path
Host: example.com
Scope: /admin/
AutomationPolicy: strict
AllowedMethods: GET
DisallowedMethods: POST, PUT, DELETE, PATCH
AllowedPurposes: [PLACEHOLDER_PURPOSE1]
DisallowedPurposes: [PLACEHOLDER_PURPOSE2], [PLACEHOLDER_PURPOSE3]

# Extended directives for admin path
RequestLimit: 10/minute
ConcurrentLimit: 2
RequireHumanInitiatedSession: true
SessionValidation: token-based
SessionTTL: 30m

Author's Address