DevCentr Docs
    • Business Bootstrap
    • General Knowledge
    • Home
    • Platforms
    • Templates
    • Tools
    • ~
  • /
  • Keyword: robots.txt
  • Home
    • Overview
    • Activity Log
    • Where Documentation Lives
    • Todo & Roadmap
    • Software publishing
      • Code signing in the release pipeline
      • Antora UI branding
      • Antora essential extensions
      • Antora diagram formats
      • Nav scroll performance A/B
    • Tools
  • General Knowledge
    • Overview
    • Changelog
    • Tutorials
      • Overview
      • Developer Onboarding
      • Secrets from vendor to deploy
      • Build a search-ready Next.js listing site
    • How-to Guides
      • Overview
      • Nushell Setup
      • Flutter Setup
      • GitHub Pages Custom Domain
      • Cloudflare Access for static sites
      • Git Release Workflow
      • Concurrent Agent Workspaces
      • Org-scoped Git worktrees
      • Automated Releases
      • Polyglot CI matrix
      • Prevent text encoding corruption
      • Enforce org-wide checks on GitHub Free
      • Keep GitHub Actions current
      • Binary Tailor
      • Winget Registration
      • Finding Windows Internals Docs
      • About Screen and Debug Dump
      • Windows Auto-Updates
      • Unix/macOS Auto-Updates
      • Product Essentials Checklists
      • App Installer Packaging
      • Authoritative installations
      • WSL Setup
      • Context7 MCP and Skills Setup
      • Composio MCP Setup
      • Composio Cloudflare Connection
      • Google Cloud OAuth (personal tools)
      • Bootstrap Cursor Skills
      • Secrets distribution (CLI · MCP)
      • Bitwarden CLI (agents)
      • gcloud CLI (agents)
    • Reference
      • Overview
      • Recommended Tools
      • AI Studio Tooling Catalog
      • Decentralized video platforms
      • Google Cloud OAuth concept map
      • Shells
        • Nushell
      • Languages
        • C Language
      • Git Performance
      • pnpm run routing
      • Package Registries
      • App Essentials Checklist
      • Auxiliary Features Checklist
      • Checklist Sidecar
      • Mojibake signatures
    • Explanation
      • Overview
      • Windows
        • ClearType
        • Windows color management pipeline
      • Languages
        • Language recommendations
        • C and D
        • C++ and D
        • Julia, D, and Java
        • D and Rust
      • Internet Architecture
        • Overview
        • Internet Reliability
          • Internet Reliability
          • Three altitudes
          • Reference reliability
          • Identity stack
          • Near-term absorb
          • NDN and friends
        • Provider search and mirror backups
        • Hash-matched binary conversion profiles
      • Architecture
        • Architecture overview
        • Software Product Essentials
          • Overview
          • Core Principles
          • By type
            • Libraries
            • GUI Apps
            • CLI
            • TUI
            • Games
            • Services
          • Delivery classes
            • Overview
            • Desktop
            • Web
            • Mobile
        • Auxiliary Features
        • Marketing for engineers
        • About and Build Info
        • Automatic Updates
        • Distribution and Packaging
          • Overview
          • Installation architecture
            • Overview
            • Windows installation
            • macOS installation
            • Linux installation
            • BSD installation
            • Unity Install Layout (UIL)
        • Release target patterns
        • Open Canvas (OCIF)
        • Polyglot distribution (experimental)
        • Build Pipeline
        • Diagnostics and Support
        • Config Lifecycle Management
        • Graphics rendering engines
        • Aspect-Oriented Programming
        • UI-heavy web applications
        • Subscription management platforms
      • Infrastructure
        • Nushell (recommended shell)
        • Nushell vs PowerShell
        • Nushell and AI agents
        • Text encoding and AI agents
        • Agent control planes
        • UniProvider / Open Provider Registry
        • Vibe coding bootstrap
        • Composio and MCP connectors
        • Google Cloud project mental model
        • Studio MCP install strategies
        • Secrets distribution hub
        • NVIDIA OpenShell
        • Dev-Centr and OpenShellOrg
        • Tool-run embedding for agents
        • Shell Environment Refresh
        • Abstract playbooks
        • Toolchain Management Pattern
        • Toolchain Control Plane
        • Toolchain Framework (TCF)
        • OS install concurrency
        • Infrastructure as Code (IaC)
        • Access in front of static sites
        • API key vs token
        • Git concurrency enshittification
        • Git worktrees long-range
        • Git’s file-shaped change model
        • VCS over FS editions
        • pnpm Mechanics
        • npm package managers
        • WSL Distro Comparison
      • AI literacy
        • Curated watching (AI)
      • Programming
        • Curated watching (programming)
      • Open source / licensing posture
        • Ecosystem failures and successes
      • Web publishing
        • What SEO does for a public website
      • Documentation
        • Writing Clear Docs with AI
        • Literate Programming (LP)
        • Brand and UI Motion Formats
        • Documentation markup
        • Markdown editors on local disk
        • Structured data formats
  • Business Bootstrap
    • Overview
    • Changelog
    • Fast org bootstrap
    • Software stack
      • Overview
      • Defaults by stage
      • Email
        • Overview
        • Human vs product messaging
        • Email decision matrix
        • Bootstrap org email
        • ZeptoMail + Gmail send-as
        • Personal Gmail send-as gotchas
        • Cloudflare Email Routing limits
        • Escalate DMARC policy
        • Resend domain DNS roles
        • Email provider comparison
        • DMARC policy path
        • Resend subdomain DNS roles
        • ITIL-shaped email services
      • Scheduling
        • Overview
        • Cal.com and Cal.diy
      • Design
        • Overview
        • Penpot
      • Vault
        • Overview
        • Bitwarden and Vaultwarden
      • Chat
        • Overview
        • Mattermost self-host
      • Org infra
        • Overview
        • Org infra repo pattern
        • Pulumi Cloud org model
        • Pulumi secrets vs stack state
        • Back up Pulumi stack state
        • Transfer a project into an org
        • Pulumi + Cloudflare email checklist
        • Cloudflare Access (static sites)
      • Funding
        • Overview
        • OSS funding landscape
        • Donation ask ideas
  • Platforms
    • Overview
    • devcentr.org
      • Overview
      • Changelog
      • Site architecture
      • News & changelog pipeline
      • Theme reveal
      • Stack Advisor
      • Agent skills
      • Local development
      • Help & support plans
      • VS Code & CI walkthrough
  • Tools
    • Overview
    • Overview
    • Overview
    • Overview
    • Overview
    • Overview
    • Overview
    • Changelog
    • Changelog
    • Changelog
    • Changelog
    • Changelog
    • Agent Rules
    • Editorial titles
    • Case study: agent voice vs reader voice
    • Harness-neutral architecture
    • End-of-run commit and push
    • Agent harness
    • Mixr routing and cost
    • Shadow workspace and LSP
    • Actor-model agentic UI
    • Usage
    • Org layer
    • Bootstrap agent skills
    • Skills catalog
    • Bitwarden CLI (agents)
    • gcloud CLI (agents)
    • Architecture
    • Install
    • Introduction
    • How-to: in-place PATH
    • How-to: installer projects
    • How-to: emit CI pipelines
    • Plugins
    • How-to: plugin tools
    • Plugin ABI
    • Shell integration
    • Install Coordinator / UIL
    • Introduction
    • Architecture
    • Distribution
    • Framework + UIL
    • UIL reference
    • Connectome-fs mapping
    • Switchyard
    • Getting started
    • RepoDrive
    • Installation
    • Schema
    • Usage
    • About / help
    • Introduction
    • Forge metadata backup
    • Installation
    • Usage
    • Architecture
    • Installation
    • Usage
    • Library API
    • Introduction
    • Getting Started
    • AI Evaluation
    • API Reference
    • Security & Verification
    • Scriptbook
    • PlayTime
    • Host Awareness
    • Playbook profile
    • Sidecar schema
    • Installation & Setup
    • CLI Reference
    • Host Awareness
    • CI Action Integration
    • D Library Integration
    • Code Rulesets Guide
    • Filesystem Rules Guide
    • Introduction
    • Why a generic config panel
    • How-to
      • Open and register files
      • Ship vocabulary with your app
      • Embed or launch ConfigUI
      • Engine and UI components
    • CLI and formats
    • Architecture
    • Open Provider Registry
    • JS / Tauri bridge
    • Binary Tailor
    • macho-memload
    • polyglot-debug
    • Toolchain Framework
    • Themed SVG Studio
      • Overview
      • Changelog
      • Build a first themed SVG project
      • Use the CLI
      • Architecture
      • Protocol version 1
    • Tutorials
      • First mail check
    • How-to
      • Set up Gmail with Himalaya
      • Run the checker on Windows
      • Wire into docs.devcentr.org
    • Explanation
      • Why integration, not disposable inboxes
      • Architecture
    • Reference
      • CLI
      • Changelog
      • Config
    • Infra template
    • Usage
    • Email DNS roles
  • Templates
    • Overview
    • Changelog
    • Getting Started
    • VS Code workspace
    • Committing code
    • Environment setup
    • Documentation preview
    • VCS configuration
    • Architecture overview
    • Using Corepack
    • Tool hierarchy
    • SSH keys on Windows
    • Issues repository pattern
    • Terminology and Stylistic Choices

Keyword: robots.txt

Articles that use the keyword robots.txt.

general-knowledge

  • What SEO does for a public website

Related projects

DevCentr · OpenShellOrg · dlang-supplemental

Extensions

Antora: lunr-extension · site-nav-tree · nav-typology · nav-typology-diataxis · page-edit · link-validator · linkinator · extension-lister · orphan-finder/extension · page-context/antora · mermaid-client/antora · diagram-lightbox
AsciiDoc: asciidoctor-kroki · page-context

Built with Antora (3.1.14). UI: valentus-theme (MIT).