Add site map and search engine robots

The pub directory is read only in the Magento Commerce (Cloud) Integration, Staging, and Production environments. An attempt to generate and write the sitemap.xml file to the root directory results in the following error:

The path ".:///app/app/" is not writable.
Path "/app/pub/static/sitemap.xml" is protected and cannot be used.

To avoid this error, you can only write to specific directories, such as var, pub/media, or app/etc. When using Magento Commerce (Cloud), you must use a redirect in the read-only environments. Generate the files on your local workstation and upload them to one of the write-enabled directories using the following, appropriate environment:

• For Starter, access the staging branch
• For Pro, access the integration branch

To add a sitemap.xml file:

1. On your local branch, access the Magento Admin panel.
2. On the Marketing menu, click Site Map.
3. In the Site Map view, click Add Sitemap.

4. In the New Site Map view, enter the following values:

   • Filename:sitemap.xml
   • Path:/pub/media/
5. Click Save & Generate.
6. Import the generated files to the pub/media directory on the server.
7. Log in to the remote environment using SSH.

8. Move the files into the /pub/media/ directory using the rsync command:

   rsync -azvP pub/media/ <user_name>@<environment_IP_address>:pub/media/
   

9. This results in a www.example.com/media/sitemap.xml path, so you must submit a Support ticket to apply a redirect for your generated files.

To add a robots.txt file:

1. On your local branch, access the Magento Admin panel.
2. On the Content menu, click Configuration.
3. In the Design Configuration view, click Edit for the Global Website.
4. In the Main Website view, click Search Engine Robots in the Other Settings section.
5. Select options and update the robots.txt file.
6. Click Save Configuration.

Rewrite using Fastly VCL snippet

You can generate the sitemap.xml file in the Magento Admin panel as described above, but you can avoid a Support ticket by creating a custom Fastly VCL snippet to manage the redirect. See Custom Fastly VCL snippets.

To use a Fastly VCL snippet for redirect:

Create a custom VCL snippet to rewrite the path for sitemap.xml to /media/sitemap.xml using the type and content key and value pairs.

{ 
  "name": "sitemapxml_rewrite",
  "dynamic": "0",
  "type": "recv",
  "priority": "90",
  "content": "if ( req.url.path ~ \"^/?sitemap.xml$\" ) { set req.url = \"/media/sitemap.xml\"; }" 
}