Submitting Bing IndexNow url requests for updated sitemaps using Python

December 19th, 2021

bing-indexnow-python

IndexNow is a service that is currently supported by Microsoft Bing and Yandex - the Russian search engine - that defines a standard way for site owners to submit urls to be indexed, presumably on demand.

There is a standard API which can be invoked using a key generated in the link. But before any request can be made, a utf-8 {key}.txt file with the key inside the file can be placed in a website's root and that will allow the Bing & Yandex to know it was a valid request to the site.

We just need to submit the IndexNow request to either Bing or Yandex - no need to do it twice - would be great if Google joins this party but probably not going to happen, so we are going to have to continue to ping Google separately.

This post outlines how to generate a valid IndexNow API json in Python and then submit the request to Bing. The script will read a sitemap.xml to check for new and updated urls and only send those.

Lets review.

Generating an IndexNow API key and adding it to the root

Use the link here to generate a key.

gen indexnow api key

Copy the value of the key - we will need this in the API request - but before we do that, create a text file titled {key}.txt with the same {key} value as a single line as the file's content.

Copy the {key}.txt to the public folder of your website and deploy it.

Python code to read a sitemap and check for updates against the last read value

The first part of the script is going to read the current updated sitemap.xml from a defined location - check against another file which is it's own record of what it saw previously in the sitemap - and generate a list of urls that will be submitted in the next step.

We will use xmltodict - a Python library that is great for working with XMLs as dictionaries to read the sitemap and the requests library to make the API calls in the next step.

This is how to do this in Python.

import requests
import xmltodict
from pathlib import Path
from datetime import date

sitemap = Path(f"sitemap.xml")
last_checked_list = Path(f"sitemap-check.txt")

root_domain = "https://www.{your_domain}.com"

f = open(sitemap)

doc = xmltodict.parse(f.read())

f.close()

sitemap_urls = {}
last_checked_urls = {}

for urlset in doc["urlset"]["url"]:
    url = urlset["loc"]
    lastmod = date.fromisoformat(urlset["lastmod"])
    sitemap_urls[url] = lastmod

f = open(last_checked_list, 'r')
lines = [line.rstrip() for line in f]

for line in lines:
    split = line.split("|")
    url = split[0]
    lastmod = date.fromisoformat(split[1])
    last_checked_urls[url] = lastmod

f.close()

new_updated_urls = []

# Save the current sitemap to a text file
# while checking for updates
with open(last_checked_list, 'w') as f:
    for url, lastmod in sitemap_urls.items():
        if url not in last_checked_urls:
            new_updated_urls.append(url)
        else:
            if lastmod != last_checked_urls[url]:
                new_updated_urls.append(url)

        f.write(f"{url}|{lastmod.isoformat()}\n")

Python code to make the call to IndexNow API

Now that we have the list of new/updated urls, we can use the requests API to submit them to Bing like this. Be sure to update the {key} with your actual key.

if len(new_updated_urls) > 0:

    index_now = {
      "host": root_domain,
      "key": "{key}",
      "keyLocation": f"{root_domain}/{key}.txt",
      "urlList": new_updated_urls
    }

    bing_now = requests.post("https://www.bing.com/indexnow", json=index_now)

    print(bing_now)