User-agent: * Disallow: /api/ User-agent: ChatGPT-User User-agent: GPTBot User-agent: CCbot Disallow: / User-agent: Mediapartners-Google Allow: /