-
Notifications
You must be signed in to change notification settings - Fork 347
ci: catch broken markdown links #403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
x-n2o
wants to merge
3
commits into
ml-explore:main
Choose a base branch
from
x-n2o:markdown-link-check
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Paths that tools/check_links.py should skip: markdown files matching | ||
| # a pattern are not scanned, and link targets matching a pattern are not | ||
| # flagged as broken. One fnmatch glob per line, relative to the repository | ||
| # root. Lines starting with # are comments. | ||
|
|
||
| # Third-party sources -- we don't build docs for these: | ||
| Source/Cmlx/fmt/* | ||
| Source/Cmlx/json/* | ||
| Source/Cmlx/metal-cpp/* | ||
|
|
||
| # DocC cross-doc-package links; these resolve in the built documentation | ||
| # (https://swiftpackageindex.com/ml-explore/mlx-swift/main/documentation/mlx): | ||
| Source/*/Documentation.docc/mlx | ||
| Source/*/Documentation.docc/mlxnn | ||
| Source/*/Documentation.docc/mlxoptimizers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,150 @@ | ||
| #!/usr/bin/env python3 | ||
| """ | ||
| Check for broken internal markdown links. | ||
| Scans .md files for links like [text](Something.md) and reports | ||
| any that point to files that don't exist. Links with a URL scheme | ||
| or host (http, https, mailto, ...) and bare anchors are skipped, | ||
| as are targets that resolve outside the repository (they cannot | ||
| be validated from a checkout). | ||
|
|
||
| Paths listed in .check-links-ignore at the repository root (one | ||
| fnmatch glob per line, relative to the root) are also skipped: | ||
| markdown files matching a pattern are not scanned (e.g. vendored | ||
| third-party docs), and link targets matching a pattern are not | ||
| flagged as broken (e.g. targets that moved to another repository). | ||
|
|
||
| Usage: python3 check_links.py <directory> | ||
| python3 check_links.py <file.md> [<file.md> ...] | ||
|
|
||
| The file form is used by pre-commit, which passes the tracked | ||
| markdown files to check. | ||
|
|
||
| Kept in sync between ml-explore/mlx-swift-examples (support/) | ||
| and ml-explore/mlx-swift (tools/). | ||
| """ | ||
|
|
||
| import fnmatch | ||
| import os | ||
| import re | ||
| import sys | ||
| from pathlib import Path | ||
| from urllib.parse import unquote, urlsplit | ||
|
|
||
|
|
||
| # Directories to skip when scanning a directory | ||
| SKIP_DIRS = {'.git', '.build', '.swiftpm', 'vendor', 'node_modules'} | ||
|
|
||
| IGNORE_FILE = '.check-links-ignore' | ||
|
|
||
| LINK_PATTERN = re.compile(r'\[([^\]]*)\]\(([^)]+)\)') | ||
|
|
||
|
|
||
| def load_ignore_patterns(base): | ||
| """Load path patterns to skip from .check-links-ignore.""" | ||
| ignore_file = base / IGNORE_FILE | ||
| if not ignore_file.is_file(): | ||
| return [] | ||
| patterns = [] | ||
| for line in ignore_file.read_text(encoding='utf-8').splitlines(): | ||
| line = line.strip() | ||
| if line and not line.startswith('#'): | ||
| patterns.append(line) | ||
| return patterns | ||
|
|
||
|
|
||
| def is_ignored(path, base, ignore_patterns): | ||
| """True if path (relative to base) matches an ignore pattern.""" | ||
| if not path.is_relative_to(base): | ||
| return False | ||
| rel = str(path.relative_to(base)) | ||
| return any(fnmatch.fnmatch(rel, p) for p in ignore_patterns) | ||
|
|
||
|
|
||
| def find_all_md_files(directory): | ||
| """Build a set of all .md file paths under the directory.""" | ||
| md_files = set() | ||
| for dirpath, dirnames, filenames in os.walk(directory): | ||
| dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS] | ||
| md_files.update( | ||
| Path(dirpath) / name for name in filenames if name.endswith('.md')) | ||
| return md_files | ||
|
|
||
|
|
||
| def check_file(filepath, directory, ignore_patterns): | ||
| """Check all internal links in a file. Returns a list of | ||
| (line_num, link_text, target) tuples for broken links.""" | ||
| broken = [] | ||
| in_code_block = False | ||
| with open(filepath, 'r', encoding='utf-8') as f: | ||
| for line_num, line in enumerate(f, 1): | ||
| # Skip fenced code blocks -- their contents are not links | ||
| if line.lstrip().startswith(('```', '~~~')): | ||
| in_code_block = not in_code_block | ||
| continue | ||
| if in_code_block: | ||
| continue | ||
| for match in LINK_PATTERN.finditer(line): | ||
| display = match.group(1) | ||
| target = urlsplit(match.group(2)) | ||
|
|
||
| # Skip external links and bare anchors | ||
| if target.scheme or target.netloc or not target.path: | ||
| continue | ||
|
|
||
| # URL-decode (e.g. %20 -> space) and resolve relative | ||
| # to the file's directory | ||
| target_path = (filepath.parent / unquote(target.path)).resolve() | ||
|
|
||
| # Targets outside the repository cannot be validated here | ||
| if not target_path.is_relative_to(directory): | ||
| continue | ||
|
|
||
| if is_ignored(target_path, directory, ignore_patterns): | ||
| continue | ||
|
|
||
| if not target_path.exists(): | ||
| broken.append((line_num, display, target.path)) | ||
|
|
||
| return broken | ||
|
|
||
|
|
||
| def main(): | ||
| args = sys.argv[1:] | ||
| if len(args) == 1 and Path(args[0]).is_dir(): | ||
| directory = Path(args[0]).resolve() | ||
| md_files = find_all_md_files(directory) | ||
| elif args: | ||
| directory = Path.cwd() | ||
| md_files = {Path(arg).resolve() for arg in args} | ||
| else: | ||
| print("Usage: python3 check_links.py <directory or markdown files>") | ||
| sys.exit(1) | ||
|
|
||
| ignore_patterns = load_ignore_patterns(directory) | ||
|
|
||
| by_file = {} | ||
| for filepath in sorted(md_files): | ||
| if is_ignored(filepath, directory, ignore_patterns): | ||
| continue | ||
| broken = check_file(filepath, directory, ignore_patterns) | ||
| if broken: | ||
| rel = (filepath.relative_to(directory) | ||
| if filepath.is_relative_to(directory) else filepath) | ||
| by_file[rel] = broken | ||
|
|
||
| if not by_file: | ||
| print("No broken links found.") | ||
| sys.exit(0) | ||
|
|
||
| for source, broken in by_file.items(): | ||
| print(f"\n{source}:") | ||
| for line_num, display, target in broken: | ||
| print(f" line {line_num}: [{display}]({target}) -> NOT FOUND") | ||
|
|
||
| total = sum(len(broken) for broken in by_file.values()) | ||
| print(f"\nTotal: {total} broken link(s) in {len(by_file)} file(s)") | ||
| sys.exit(1) | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| main() |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.