TaskBlinkDataBot

How to recognise it

Every request it makes carries this user agent:

Mozilla/5.0 (compatible; TaskBlinkDataBot/1.0; +https://data.taskblink.com/bot)

It sends no cookies, never signs in, never fills in or submits forms, and only reads pages (GET and HEAD requests).

What it reads, and why

TaskBlink Data customers build lists of local businesses and can ask for a check of each business's website: which website builder and tools it uses, whether it has a booking or chat tool, basic search and speed signals, and whether it advertises. To answer, the bot visits the business's public website and reads:

• the homepage, robots.txt and the sitemap;
• up to two contact or about pages linked from the menu;
• the site's news or blog feed, when it has one;
• the Google Tag Manager containers the homepage loads, to see which tags and pixels are installed;
• a small sample of links on the site, checked with HEAD requests to find broken links.

When a customer checks a business's Google Maps rank, the bot also reads the websites of up to 3 competitors at the top of that Maps search in the same way, so the customer can compare them. A competitor's site is then read even though nobody asked for a check of that business, under the same rules, pace and robots.txt handling as every other site.

It also asks public DNS for the domain's mail server, SPF and DMARC records. Whether ads for the website are listed in Google's Ads Transparency Center, the public record of ads that run on Google, is checked through a data provider (DataForSEO), or directly when that lookup is switched on.

We keep what we learn about a site, not copies of its pages: facts such as the tools found and the checks that passed or failed, plus a short extract of the visible text for the AI features our customers use. We don't keep phone numbers from website pages.

How often, and how fast

• At most 20 requests per site in one check, one request at a time, with a pause between them.
• Results are reused for up to 30 days, so a site is not checked again for every customer or every list that includes it.
• A customer can ask for a fresh check, but a site is refreshed at most once a day whatever the number of requests.
• When we check Google's Ads Transparency Center directly, it is asked about one domain at a time, at a low rate. Through DataForSEO, the provider sends the lookups.

Limiting or stopping it

The bot follows the Disallow rules in your robots.txt, both the ones for TaskBlinkDataBot and the ones for all robots (User-agent: *), for every page except your homepage. It reads the homepage once per check, the way a visitor would, so it can tell whether the site is up.

To keep it off everything beyond your homepage, add this to your robots.txt:

User-agent: TaskBlinkDataBot
Disallow: /

To keep it off your homepage as well, refuse requests from the TaskBlinkDataBot user agent at your server or firewall (a 403 answer works). The bot records that the site could not be checked and does not try again another way: it never switches to a different user agent, uses proxies or other addresses, or solves CAPTCHAs.

If our bot's visits cause you a problem, email banks@taskblink.com with your domain and we will look into it.

More about TaskBlink Data

Our Privacy Policy explains what TaskBlink Data collects and who helps us run it. A business that doesn't want its number supplied by TaskBlink Data can ask us to remove it at data.taskblink.com/remove.