security.txt: A Welcome Mat or a Red Flag for Hackers?

Boost Your Website’s Security with security.txt - But Beware of the Risks!

Alright folks, let’s talk about something that’s been making waves in the cybersecurity world: security.txt. Now, before your eyes glaze over, I promise to keep things beginner-friendly. Think of security.txt as a digital welcome mat for those ethical hackers – the good guys who help companies find and fix security holes before the bad guys exploit them.

In the old days (and sadly, still on many sites today), if a security researcher stumbled upon a vulnerability, they’d have to jump through hoops to report it. Imagine trying to find the right contact at a massive company — talk about a headache! Enter security.txt, a simple text file that websites can place on their servers to provide clear contact information for reporting vulnerabilities. Sounds great, right? Well, like most things in cybersecurity, it's not that black and white.

A little trip down memory lane:

The idea for security.txt was born out of frustration. Security researchers were tired of jumping through hoops to report vulnerabilities, and website owners were often unaware of security holes until it was too late. So, a group of security experts got together and said, "There has to be a better way!" And thus, security.txt was born.

After years of development and collaboration, security.txt was officially recognized as an RFC (9116) in 2021. It was a big win for the security community, and it paved the way for wider adoption of this simple yet powerful tool.

But here’s the catch:

While security.txt has gained significant traction, it's still not as widely adopted as you might think. I was curious about just how many websites were actually using it, but I couldn't find any reliable statistics (not that my test was scientific in anyway). So, being the curious cybersecurity enthusiast that I am, I decided to do a little digging myself.

I found a list of the top 1,000 websites (from CloudFlare Radar) and wrote a Python script to check for the presence of security.txt. (Checking 1,000 addresses is a lot of work, let me tell you!) The results? Out of those 1,000 sites, only 20% had a security.txt file. That's a surprisingly low number, considering the potential benefits.

If you’re interested in the technical details, you can check out the script I used on my GitHub gist or at the bottom of this post.

This little experiment really highlights the need for greater awareness and adoption of security.txt. It's a simple yet powerful tool that can make a real difference in website security.

Why **security.txt** is a thumbs up:

• Think efficiency: No more endless searching for the right email address or contact form. security.txt puts the information front and center, making it super easy for researchers to report vulnerabilities.
• Speed is key: The faster a vulnerability is reported, the faster it can be fixed. This means less time for the bad guys to exploit it and cause damage.
• Proactive is the name of the game: Having a security.txt file shows that a company is serious about security and encourages them to be more proactive in finding and fixing vulnerabilities.
• Building trust: Transparency is key in today’s world. By using security.txt, companies can build trust with their users and show that they're committed to keeping their data safe.

But hold on… there’s a flip side:

• A beacon for the bad guys? Unfortunately, having a security.txt file could potentially attract malicious actors. It's like putting a sign up saying, "Hey, we're trying to be secure, but we might have weaknesses!"
• Don’t cry wolf: There’s always the risk of false reports or spam, which can overwhelm security teams and waste valuable resources.
• Too much information? Some worry that security.txt could inadvertently reveal sensitive information about a company's internal security practices.
• Tech headaches: Implementing security.txt correctly requires some technical know-how, and it might not be suitable for all websites.

So, what’s the verdict?

Like I said, it’s a double-edged sword. security.txt has the potential to significantly improve website security, but it's important to weigh the pros and cons carefully. If you're considering implementing it, make sure you understand the potential risks and take steps to mitigate them.

Disclaimer: The information and opinions expressed in this blog post are solely my own and do not reflect the views of my employer. This post is intended for educational purposes only.

Read More:

• https://krebsonsecurity.com/2021/09/does-your-organization-have-a-security-txt-file/
• https://blog.cloudflare.com/security-txt/
• https://www.cisa.gov/news-events/news/securitytxt-simple-file-big-value

Script:

import concurrent.futures
import time
import csv
import traceback
import socket
import requests
import threading

def check_security_txt(url):
"""Checks if a website has a security.txt file."""
try:
# Enforce https:// and lowercase domain
if not url.startswith(("http://", "https://")):
url = "https://" + url
url = url.lower()

response = requests.get(
f"{url}/.well-known/security.txt",
timeout=10,
headers={"User-Agent": "Security Scanner Bot"}
)
return response.status_code == 200
except requests.exceptions.Timeout:
# Suppress timeout error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - TimeoutError for {url}\n")
return False
except requests.exceptions.ConnectionError as e:
try:
if isinstance(e.args[0].reason, socket.gaierror):
# Suppress DNS resolution error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - DNS resolution error for {url}: {e.args[0].reason}\n")
else:
# Suppress connection error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - Connection error for {url}: {e}\n")
except (AttributeError, IndexError):
# Suppress connection error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - Connection error for {url}: {e}\n")
return False
except requests.exceptions.SSLError as e:
# Suppress SSL error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - SSL error for {url}: {e}\n")
return False
except requests.exceptions.RequestException as e:
# Suppress client error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - ClientError for {url}: {e}\n")
return False
except Exception as e:
# Suppress unexpected error messages
with open("security_txt_errors.log", "a") as error_log:
error_log.write(f"{time.strftime('%Y-%m-%d %H:%M:%S')} - Unexpected error for {url}: {e}\n")
traceback.print_exc()
return False

def main():
print("Starting script...")

try:
top_1m_websites = []
with open("top-1m.csv", "r") as f:
print("Opened top-1m.csv")
next(f) # Skip the header row
for line in f:
line = line.strip()
try:
top_1m_websites.append(line.split(",")[1])
except IndexError:
top_1m_websites.append(line)

print("Loaded websites:", len(top_1m_websites))

total_websites = len(top_1m_websites)
count = 0
positive_results = []
negative_results = []
results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=100) as executor:
futures = [executor.submit(check_security_txt, url) for url in top_1m_websites]
for i, future in enumerate(concurrent.futures.as_completed(futures)):
results.append(future.result())
if results[-1]:
count += 1
percentage = (count / (i + 1)) * 100
print(f"Scanned: {i+1}/{total_websites} - Percent with security.txt: {percentage:.2f}%", end='\r')

with open("security_txt_log.txt", "w") as log_file:
log_writer = csv.writer(log_file)
for website, has_security_txt in zip(top_1m_websites, results):
log_writer.writerow([website, "Found" if has_security_txt else "Not Found"])

for website, has_security_txt in zip(top_1m_websites, results):
if has_security_txt:
positive_results.append(website)
else:
negative_results.append(website)

with open("security_txt_positive.csv", "w", newline="") as positive_csv, \
open("security_txt_negative.csv", "w", newline="") as negative_csv:

positive_writer = csv.writer(positive_csv)
negative_writer = csv.writer(negative_csv)
positive_writer.writerows([[website] for website in positive_results])
negative_writer.writerows([[website] for website in negative_results])

percentage = (count / total_websites) * 100
print(f"\nFinished checking {total_websites} websites.")
print(f"Final percentage with security.txt: {percentage:.2f}%")
print("Log file saved as 'security_txt_log.txt'")
print("Positive results saved as 'security_txt_positive.csv'")
print("Negative results saved as 'security_txt_negative.csv'")

except FileNotFoundError:
print("Error: top-1m.csv not found.")
except Exception as e:
print(f"An unexpected error occurred: {e}")
traceback.print_exc()

if __name__ == "__main__":
try:
main()
except Exception as e:
print(f"An unexpected error occurred: {e}")
traceback.print_exc()