Release 0.2.4 (#96)

- Increase timeout for requests to check web urls alive or not, defaults to 15 seconds.
- Consider status codes between 2xx and 3xx as valid URLs.
- Add headers Accept and User-Agent headers to the requests which are required by some websites.

Author: john0isaac

.github/workflows/python-tests.yaml

@@ -42,13 +42,13 @@ jobs:
       run: |
         cd azure-search-openai-demo
         markdown-checker -d . -f check_urls_locale -gu ''
- 
+
     - name: Test Check Broken URLs in azure-search-openai-demo
       if: always()
       run: |
         cd azure-search-openai-demo
         markdown-checker -d . -f check_broken_urls -gu ''
- 
+
   tests-phicookbook-repo:
     name: Python Tests on Phi-3CookBook
     runs-on: ubuntu-latest

CHANGELOG.md

@@ -12,6 +12,12 @@ All notable changes to this project will be documented in this file.
 
 ### Other Changes
 
+## [v0.2.4] 26 Jan 2025
+
+- Increase timeout for requests to check web urls alive or not, defaults to 15 seconds.
+- Consider status codes between 2xx and 3xx as valid URLs.
+- Add headers Accept and User-Agent headers to the requests which are required by some websites.
+
 ## [v0.2.3] 26 Nov 2024
 
 - Skip another domain by @IsuminI

docs/mkdocs.yml

@@ -35,7 +35,7 @@ nav:
   - About: index.md
   - Usage: usage.md
   - Advanced Usage: advanced.md
-  - API Reference: 
+  - API Reference:
     - API Reference: api.md
     - Main: ./api/main.md
     - Urls: ./api/urls.md

docs/source/advanced.md

@@ -1,104 +1,119 @@
+<!-- markdownlint-disable MD041 -->
 ## Advanced Usage
 
 To further customize your experience with the Markdown Checker, you can utilize additional command-line interface (CLI) options.
 
 ## Command Line Options
 
 ### `-d`, `--dir`
+
 - **Type**: `click.Path`
 - **Description**: Path to the root directory to check.
 - **Required**: Yes
 
 ### `-f`, `--func`
+
 - **Type**: `click.Choice`
 - **Description**: Function to be executed.
 - **Choices**:
-    - `check_broken_paths`
-    - `check_broken_urls`
-    - `check_paths_tracking`
-    - `check_urls_tracking`
-    - `check_urls_locale`
+  - `check_broken_paths`
+  - `check_broken_urls`
+  - `check_paths_tracking`
+  - `check_urls_tracking`
+  - `check_urls_locale`
 - **Required**: Yes
 
 ### `-ext`, `--extensions`
+
 - **Type**: `list[str]`
 - **Description**: File extensions to filter the files.
-- **Default**: 
-    - `.md`
-    - `.ipynb`
+- **Default**:
+  - `.md`
+  - `.ipynb`
 - **Required**: No
 
 ### `-td`, `--tracking-domains`
+
 - **Type**: `list[str]`
 - **Description**: List of tracking domains to check.
-- **Default**: 
-    - `github.com`
-    - `microsoft.com`
-    - `visualstudio.com`
-    - `aka.ms`
-    - `azure.com`
+- **Default**:
+  - `github.com`
+  - `microsoft.com`
+  - `visualstudio.com`
+  - `aka.ms`
+  - `azure.com`
 - **Required**: No
 
 ### `-sf`, `--skip-files`
+
 - **Type**: `list[str]`
 - **Description**: List of file names to skip check.
-- **Default**: 
-    - `CODE_OF_CONDUCT.md`
-    - `SECURITY.md`
+- **Default**:
+  - `CODE_OF_CONDUCT.md`
+  - `SECURITY.md`
 - **Required**: No
 
 ### `-sd`, `--skip-domains`
+
 - **Type**: `list[str]`
 - **Description**: List of domains to skip checking.
 - **Default**: `[]`
 - **Required**: No
 
 ### `-suc`, `--skip-urls-containing`
+
 - **Type**: `list[str]`
 - **Description**: List of strings to skip checking if their urls are working or not.
-- **Default**: 
-    - `https://www.microsoft.com/en-us/security/blog`
-    - `video-embed.html`
+- **Default**:
+  - `https://www.microsoft.com/en-us/security/blog`
+  - `video-embed.html`
 - **Required**: No
 
 ### `-gu`, `--guide-url`
+
 - **Type**: `str`
 - **Description**: Full URL of your contributing guide.
 - **Required**: No
 
 ### `-to`, `--timeout`
+
 - **Type**: `Click.IntRange`
 - **Description**: Timeout in seconds for the requests before retrying.
-- **Default**: `10`
+- **Default**: `15`
 - **Range**: `0-50`
 - **Required**: No
 
 ### `-rt`, `--retries`
+
 - **Type**: `Click.IntRange`
 - **Description**: Number of retries for the requests before flagging a url as broken.
 - **Default**: `3`
 - **Range**: `0-10`
 - **Required**: No
 
 ### `-o`, `--output-file-name`
+
 - **Type**: `str`
 - **Description**: Name of the output file.
 - **Default**: `comment`
 - **Required**: No
 
 ### `SRC ...`
+
 - **Type**: `click.Path`
 - **Description**: Source files or directories to check.
 - **Required**: No
 
 ## Other Options
 
 ### `--version`
+
 - **Type**: `bool`
 - **Description**: Show the version and exit.
 - **Required**: No
 
 ### `--help`
+
 - **Type**: `bool`
 - **Description**: Show the help message and exit.
 - **Required**: No

docs/source/api.md

@@ -1,3 +1,4 @@
+<!-- markdownlint-disable MD041 -->
 - [Main](./api/main.md)
 - [Urls](./api/urls.md)
 - [Paths](./api/paths.md)

docs/source/api/main.md

@@ -1 +1 @@
-::: markdown_checker
+::: markdown_checker

docs/source/api/markdown_link_base.md

@@ -1 +1 @@
-::: markdown_checker.markdown_link_base
+::: markdown_checker.markdown_link_base

docs/source/api/paths.md

@@ -1 +1 @@
-::: markdown_checker.paths
+::: markdown_checker.paths

docs/source/api/urls.md

@@ -1 +1 @@
-::: markdown_checker.urls
+::: markdown_checker.urls

docs/source/api/utils.md

@@ -4,4 +4,4 @@
 
 ::: markdown_checker.utils.list_files
 
-::: markdown_checker.utils.spinner
+::: markdown_checker.utils.spinner

docs/source/usage.md

@@ -4,11 +4,11 @@ The library provides the following functions:
 
 [Usage](#usage):
 
-  - [`check_broken_paths`](#check_broken_paths)
-  - [`check_broken_urls`](#check_broken_urls)
-  - [`check_urls_locale`](#check_urls_locale)
-  - [`check_paths_tracking`](#check_paths_tracking)
-  - [`check_urls_tracking`](#check_urls_tracking)
+- [`check_broken_paths`](#check_broken_paths)
+- [`check_broken_urls`](#check_broken_urls)
+- [`check_urls_locale`](#check_urls_locale)
+- [`check_paths_tracking`](#check_paths_tracking)
+- [`check_urls_tracking`](#check_urls_tracking)
 
 ## `check_broken_paths`
 
@@ -60,4 +60,4 @@ Example:
 markdown-checker -d . -f check_urls_tracking -gu https://github.com/john0isaac/markdown-checker/blob/main/CONTRIBUTING.md
 ```
 
-## Want to do more? Check out the [Advanced Usage](./advanced.md) page.
+## Want to do more? Check out the [Advanced Usage](./advanced.md) page

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "markdown-checker"
 description= "A markdown link validation reporting tool."
-version = "0.2.3"
+version = "0.2.4"
 authors = [{ name = "John Aziz", email = "[email protected]" }]
 maintainers = [{ name = "John Aziz", email = "[email protected]" }]
 license = {file = "LICENSE"}

src/markdown_checker/__init__.py

@@ -76,32 +76,15 @@ def detect_issues(
         # currently these domains are known to have restrictions on the requests
         skip_domains.extend(
             [
-                "platform.openai.com",
-                "help.openai.com",
+                "openai.com",
                 "beta.openai.com",
-                "marketplace.visualstudio.com",
-                "huggingface.co",
+                "help.openai.com",
+                "platform.openai.com",
+                "vscode.dev",
                 "en.wikipedia.org",
-                "twitter.com",
-                "www.linkedin.com",
-                "make.powerautomate.com",
-                "make.powerapps.com",
                 "www.midjourney.com",
-                "vscode.dev",
+                "www.linkedin.com",
                 "rodtrent.substack.com",
-                "example.com",
-                "www.nuget.org",
-                "www.docker.com",
-                "build.nvidia.com",
-                "dotnet.microsoft.com",
-                "www.gemini.com",
-                "upload.wikimedia.org",
-                "medium.com",
-                "blogs.nvidia.com",
-                "blog.gopenai.com",
-                "towardsdatascience.com",
-                "code.visualstudio.com",
-                "opensource.org",
             ]
         )
         with concurrent.futures.ProcessPoolExecutor() as executor:
@@ -260,7 +243,7 @@ def type_cast_value(self, ctx, value):
     "-to",
     "--timeout",
     type=click.IntRange(0, 50),
-    default=10,
+    default=15,
     help="Timeout in seconds for the requests.",
     required=False,
 )
@@ -289,7 +272,7 @@ def type_cast_value(self, ctx, value):
     required=False,
 )
 @click.version_option(
-    message=(f"%(prog)s, %(version)s\n" f"Python ({platform.python_implementation()}) {platform.python_version()}"),
+    message=(f"%(prog)s, %(version)s\nPython ({platform.python_implementation()}) {platform.python_version()}"),
 )
 def main(
     src: tuple[Path, ...],

src/markdown_checker/reports/md_reports/templates/paths/broken.md

@@ -3,4 +3,4 @@
 We have automatically detected the following broken relative paths in your files.
 Review and fix the paths to resolve this issue.
 
-Check the file paths and associated broken paths inside them.
+Check the file paths and associated broken paths inside them.

src/markdown_checker/reports/md_reports/templates/paths/tracking.md

@@ -3,4 +3,4 @@
 We have automatically detected missing tracking IDs from the following relative paths in your files.
 Review and add tracking to paths to resolve this issue.
 
-Check the file paths and associated paths inside them.
+Check the file paths and associated paths inside them.

src/markdown_checker/reports/md_reports/templates/urls/broken.md

@@ -2,4 +2,4 @@
 
 We have automatically detected the following broken URLs in your files. Review and fix the paths to resolve this issue.
 
-Check the file paths and associated broken URLs inside them.
+Check the file paths and associated broken URLs inside them.

src/markdown_checker/reports/md_reports/templates/urls/locale.md

@@ -3,4 +3,4 @@
 We have automatically detected added country locale to URLs in your files.
 Review and remove country-specific locale from URLs to resolve this issue.
 
-Check the file paths and associated URLs inside them.
+Check the file paths and associated URLs inside them.

src/markdown_checker/reports/md_reports/templates/urls/tracking.md

@@ -3,4 +3,4 @@
 We have automatically detected missing tracking IDs from the following URLs in your files.
 Review and add tracking to URLs to resolve this issue.
 
-Check the file paths and associated URLs inside them.
+Check the file paths and associated URLs inside them.

src/markdown_checker/urls.py

@@ -27,7 +27,7 @@ def host_name(self) -> str:
         """
         return self.parsed_url.netloc
 
-    def is_alive(self, timeout: int = 10, retries: int = 3) -> bool:
+    def is_alive(self, timeout: int = 15, retries: int = 3) -> bool:
         """
         Check if the URL is alive
 
@@ -38,14 +38,18 @@ def is_alive(self, timeout: int = 10, retries: int = 3) -> bool:
         Returns:
             bool: True if the URL is alive, False otherwise
         """
+        headers = {
+            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)",
+        }
         for _ in range(retries):
             try:
-                response = requests.head(self.link, timeout=timeout, allow_redirects=True)
-                if response.status_code == 200:
+                response = requests.head(self.link, timeout=timeout, allow_redirects=True, headers=headers)
+                if 200 <= response.status_code < 300:
                     return True
                 else:
-                    response = requests.get(self.link, timeout=timeout, allow_redirects=True)
-                    if response.status_code == 200:
+                    response = requests.get(self.link, timeout=timeout, allow_redirects=True, headers=headers)
+                    if 200 <= response.status_code < 300:
                         return True
             except requests.RequestException:
                 continue

src/markdown_checker/utils/logging.py

@@ -0,0 +1,19 @@
+"""
+Utilities for logging config.
+"""
+
+import logging
+
+
+def setup_logging(
+    level: int = logging.INFO,
+    format: str = "[%(asctime)s] %(levelname)s in %(module)s: %(message)s",
+) -> None:
+    """
+    Setup logging
+
+    Args:
+        level (int): The logging level
+        format (str): The logging format
+    """
+    logging.basicConfig(level=level, format=format)