Revert "Update the link from blockchain.info to blockchain.com"

This commit is contained in:
Pawel Borkar
2021-03-31 14:15:04 +05:30
committed by GitHub
parent 53b482c1b8
commit d715512429
19 changed files with 3732 additions and 1175 deletions
+6 -16
View File
@@ -4,43 +4,35 @@
opened to market company APIs that offer paid solutions. This API list is not a marketing tool, but a tool to help the
community build applications and use free, public APIs quickly and easily. Pull requests that are identified as marketing attempts will not be accepted.
>
> Please make sure the API you want to add has a full free access or at least a free tier before submitting.
>
> Thanks for understanding! :)
## Formatting
Current API entry format:
| API | Description | Auth | HTTPS | CORS |
| API | Description | Auth | HTTPS | Link |
| --- | --- | --- | --- | --- |
| API Title(Link to API documentation) | Description of API | Does this API require authentication? * | Does the API support HTTPS? | Does the API support [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)? * |
| API Title | Description of API | Does this API require authentication? * | Does the API support HTTPS? | Link to API webpage |
Example entry:
```
| [NASA](https://api.nasa.gov) | NASA data, including imagery | No | Yes | Yes |
| NASA | NASA data, including imagery | No | Yes | [Go!](https://api.nasa.gov) |
```
\* Currently, the only accepted inputs for the `Auth` field are as follows:
Currently, the only accepted inputs for this field are as follows:
* `OAuth` - _the API supports OAuth_
* `apiKey` - _the API uses a private key string/token for authentication - try and use the correct parameter_
* `X-Mashape-Key` - _the name of the header which may need to be sent_
* `No` - _the API requires no authentication to run_
\* Currently, the only accepted inputs for the `CORS` field are as follows:
* `Yes` - _the API supports CORS_
* `No` - _the API does not support CORS_
* `Unknown` - _it is unknown if the API supports CORS_
_Without proper [CORS configuration](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) an API will only be usable server side._
Please continue to follow the alphabetical ordering that is in place per section. Each table column should be padded with one space on either side.
If an API seems to fall into multiple categories, please place the listing within the section most in line with the services offered through the API. For example, the Instagram API is listed under `Social` since it is mainly a social network, even though it could also apply to `Photography`.
This project now offers a [JSON version](json/) of the list of services. These files are automatically updated as part of the Continuous Integration process, so there is no need to update them as part of the contribution.
## Pull Request
After you've created a branch on your fork with your changes, it's time to [make a pull request][pr-link]!
@@ -56,7 +48,6 @@ merge conflicts will be less likely. See more detailed instructions [here][synci
* Create a [branch][branch-link] for your edits.
* Contribute in the style of the project as outlined above. This makes it easier for the collaborators to merge
and for others to understand and maintain in the future.
* Please make sure you squash all commits together before opening a pull request. If your pull request requires changes upon review, please be sure to squash all additional commits as well. [This wiki page][squash-link] outlines the squash process.
### Open Pull Requests
@@ -79,5 +70,4 @@ Thanks for being a part of this project, and we look forward to hearing from you
[pr-link]: <https://help.github.com/articles/creating-a-pull-request/>
[remote-link]: <https://help.github.com/articles/configuring-a-remote-for-a-fork/>
[syncing-link]: <https://help.github.com/articles/syncing-a-fork>
[squash-link]: <https://github.com/todotxt/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit>
+10 -11
View File
@@ -1,12 +1,11 @@
<!-- Thank you for taking the time to work on a Pull Request for this project! -->
<!-- To ensure your PR is dealt with swiftly please check the following: -->
- [ ] My submission is formatted according to the guidelines in the [contributing guide](CONTRIBUTING.md)
- [ ] My addition is ordered alphabetically
- [ ] My submission has a useful description
- [ ] The description does not end with punctuation
- [ ] Each table column is padded with one space on either side
- [ ] I have searched the repository for any relevant issues or pull requests
- [ ] Any category I am creating has the minimum requirement of 3 items
- [ ] All changes have been [squashed][squash-link] into a single commit
Thank you for taking the time to work on a Pull Request for this project!
[squash-link]: <https://github.com/todotxt/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit>
To ensure your PR is dealt with swiftly please check the following:
- [ ] Your submissions are formatted according to the guidelines in the [contributing guide](CONTRIBUTING.md).
- [ ] Your changes are made in the [README](../README.md) file, not the auto-generated JSON.
- [ ] Your additions are ordered alphabetically.
- [ ] Your submission has a useful description.
- [ ] Each table column should be padded with one space on either side.
- [ ] You have searched the repository for any relevant issues or PRs.
- [ ] Any category you are creating has the minimum requirement of 3 items.
-29
View File
@@ -1,29 +0,0 @@
name: "Run tests"
on:
schedule:
- cron: '0 0 * * *'
push:
branches:
- master
pull_request:
branches:
- master
env:
FORMAT_FILE: README.md
jobs:
test:
name: 'Validate README.md'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Validate Markdown format
run: build/validate_format.py ${FORMAT_FILE}
- name: Validate pull request changes
run: build/github-pull.sh ${{ github.repository }} ${{ github.event.pull_request.number }} ${FORMAT_FILE}
if: github.event_name == 'pull_request'
-22
View File
@@ -1,22 +0,0 @@
name: "Validate links"
on:
schedule:
- cron: '0 0 * * *'
push:
branches:
- master
env:
FORMAT_FILE: README.md
jobs:
test:
name: 'Validate links'
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Validate all links from README.md
run: build/validate_links.py ${FORMAT_FILE}
+4 -7
View File
@@ -1,14 +1,11 @@
language: python
python: "3.6"
language: node_js
notifications:
email: true
before_install:
- rvm install 2.4.0
install:
- pip install -r build/requirements.txt
- gem install httparty ruby-progressbar
before_script:
# stop the build if there are Python syntax errors or undefined names
- flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
- flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- cd build
script:
- ./main.sh
+417 -774
View File
File diff suppressed because it is too large Load Diff
Executable
+9
View File
@@ -0,0 +1,9 @@
#!/bin/bash
# create json directory if not already present
mkdir -p ../json
# parse API README and print (minified) JSON to stdout, redirect to /json
node condenseMd.js ../README.md > single_table.md
node md2json.js single_table.md > ../json/entries.min.json
# beautify the previously created JSON file, redirect to /json
python -m json.tool ../json/entries.min.json > ../json/entries.json
+63
View File
@@ -0,0 +1,63 @@
fs = require('fs')
function setupMd(filename, anchor) {
fs.readFile(filename, 'utf8', function (err,text) {
if (err) {
return console.log(err);
}
var lines = text.split("\n"),
cur_line = 0,
line = "",
table_name = "",
col_num = 0,
cols = [],
rows = [],
arr = [];
function read_line() {
return lines[cur_line++];
}
while (true) {
var cols = [];
var rows = [];
while (line.indexOf(anchor) == -1 && cur_line != lines.length) {
line = read_line();
}
if (cur_line == lines.length) {
break;
}
table_name = line.split(anchor)[1];
read_line()
read_line()
while (true) {
line = read_line()
if (line.length < 2 || cur_line == lines.length) {
break
}
if (line.indexOf("|") == 0) {
arr.push(line + table_name)
}
}
}
console.log(anchor + " entries")
console.log("API | Description | Auth | HTTPS | Link | Category")
console.log("|---|---|---|---|---|---|")
for (i = 0; i < arr.length; i++) {
console.log(arr[i])
}
});
}
if (process.argv.length < 3) {
console.log("No .md file passed!");
return;
}
if (process.argv.length < 4) {
anchorText = "###";
} else {
anchorText = process.argv[3];
}
setupMd(process.argv[2].toString(), anchorText);
+29
View File
@@ -0,0 +1,29 @@
#!/bin/bash
set -o errexit -o nounset
if [ "$TRAVIS_BRANCH" != "master" ]
then
echo "This commit was made against $TRAVIS_BRANCH and not master! No deploy!"
exit 0
fi
rev=$(git rev-parse --short HEAD)
mkdir deploy
cd deploy
git init
git config --global user.name $GH_USER
git config --global user.email $GH_EMAIL
git remote add upstream "https://$GH_TOKEN@github.com/toddmotto/public-apis.git"
git fetch upstream
git reset upstream/master
mv ../../json .
git add json/
git commit -m "rebuild JSON at ${rev}" -m "[ci skip]"
git push upstream HEAD:master
-57
View File
@@ -1,57 +0,0 @@
#!/usr/bin/env bash
set -e
# Argument validation
if [ $# -ne 3 ]; then
echo "Usage: $0 <github-repo> <pull-number> <format-file>"
exit 1
fi
# Assign variables
GITHUB_REPOSITORY="$1"
GITHUB_PULL_REQUEST="$2"
FORMAT_FILE="$3"
# Move to root of project
cd "$GITHUB_WORKSPACE"
# Determine files
FORMAT_FILE="$( realpath "${FORMAT_FILE}" )"
# Skip if build number could not be determined
if [ -z "$GITHUB_REPOSITORY" -o -z "$GITHUB_PULL_REQUEST" ]; then
echo "No pull request and/or repository is provided"
exit 1
fi
# Pull changes on PR
echo "running on Pull Request #$GITHUB_PULL_REQUEST"
# Trick the URL validator python script into not seeing this as a URL
DUMMY_SCHEME="https"
DIFF_URL="$DUMMY_SCHEME://patch-diff.githubusercontent.com/raw/$GITHUB_REPOSITORY/pull/$GITHUB_PULL_REQUEST.diff"
curl -L -o diff.txt "$DIFF_URL"
# Construct diff
echo "------- BEGIN DIFF -------"
cat diff.txt
echo "-------- END DIFF --------"
cat diff.txt | egrep "\+" > additions.txt
echo "------ BEGIN ADDITIONS -----"
cat additions.txt
echo "------- END ADDITIONS ------"
LINK_FILE=additions.txt
# Validate links
echo "Running link validation..."
./build/validate_links.py "$LINK_FILE"
# Vebosity
if [[ $? != 0 ]]; then
echo "link validation failed!"
exit 1
else
echo "link validation passed!"
fi
+27 -19
View File
@@ -1,32 +1,40 @@
#!/bin/bash
FORMAT_FILE=../README.md
if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then
echo "running on $TRAVIS_BRANCH branch"
LINK_FILE=../README.md
else
echo "running on Pull Request #$TRAVIS_PULL_REQUEST"
DIFF_URL="https://patch-diff.githubusercontent.com/raw/toddmotto/public-apis/pull/$TRAVIS_PULL_REQUEST.diff"
curl $DIFF_URL > diff.txt
echo "------- BEGIN DIFF -------"
cat diff.txt
echo "-------- END DIFF --------"
cat diff.txt | egrep "\+" > additions.txt
echo "------ BEGIN ADDITIONS -----"
cat additions.txt
echo "------- END ADDITIONS ------"
LINK_FILE=additions.txt
fi
echo "running format validation..."
./validate_format.py $FORMAT_FILE
./validate_format.rb $FORMAT_FILE
if [[ $? != 0 ]]; then
echo "format validation failed!"
exit 1
else
echo "format validation passed!"
./build.sh && ./deploy.sh
if [[ $? != 0 ]]; then
echo "JSON build and deploy failed!"
else
echo "JSON build and deploy success!"
fi
fi
echo "format validation passed!"
if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then
echo "running on $TRAVIS_BRANCH branch - skipping Pull Request logic"
exit 0
fi
echo "running on Pull Request #$TRAVIS_PULL_REQUEST"
DIFF_URL="https://patch-diff.githubusercontent.com/raw/toddmotto/public-apis/pull/$TRAVIS_PULL_REQUEST.diff"
curl $DIFF_URL > diff.txt
echo "------- BEGIN DIFF -------"
cat diff.txt
echo "-------- END DIFF --------"
cat diff.txt | egrep "\+" > additions.txt
echo "------ BEGIN ADDITIONS -----"
cat additions.txt
echo "------- END ADDITIONS ------"
LINK_FILE=additions.txt
echo "running link validation..."
./validate_links.py $LINK_FILE
./validate_links.rb $LINK_FILE
if [[ $? != 0 ]]; then
echo "link validation failed!"
exit 1
+134
View File
@@ -0,0 +1,134 @@
fs = require('fs')
function md_trim(str, context) {
str = str.replace(/(^\s+)|(\s+$)/g, "");
if (context == 1) { // Name
// placeholder for any formatting on name value
} else if (context == 2) { // Description
str = str.replace(".", ""); // remove ending periods on descriptions
} else if (context == 3) { // Auth
if (str.toUpperCase() == "NO") {
str = null
} else {
str = str.replace("`", "").replace("`", "")
}
} else if (context == 4) { // HTTPS
if (str.toUpperCase() == "YES") {
str = true
} else {
str = false
}
} else if (context == 5) { // Link
str = str.replace("[Go!]", "").slice(1, -1);
}
return str;
}
function handle(filename, anchor) {
fs.readFile(filename, 'utf8', function (err,text) {
if (err) {
return console.log(err);
}
var lines = text.split("\n");
var cur_line = 0;
var line = ""
var table_name = "";
var col_num = 0;
var cols = [];
var rows = [];
var entry_count = 0;
function read_line() {
return lines[cur_line++];
}
var root = {};
while (true) {
var cols = [];
var rows = [];
while (line.indexOf(anchor) == -1 && cur_line != lines.length) {
line = read_line();
}
if (cur_line == lines.length) {
break;
}
table_name = line.split(anchor)[1];
table_name = md_trim(table_name, 0)
line = read_line()
if (line) {
line = line.split("|")
for (var j in line) {
line[j] = md_trim(line[j], 0)
if ((j == 0 || j == line.length - 1) && line[j] === "") {
} else {
cols.push(line[j]);
}
}
if (line.length) {
cols = line;
rows.push(cols)
} else {
console.error("markdown expect column title")
break;
}
} else {
console.error("markdown expect table content")
break;
}
line = read_line()
if (!line) {
console.error("markdown expect table spliter")
break;
}
line = read_line()
while (line.indexOf("|") != -1 && cur_line != lines.length) {
var line_this = line.split("|")
var row = []
for (var j in line_this) {
line_this[j] = md_trim(line_this[j], j)
if ((j == 0 || j == line_this.length - 1) && line_this[j] === "") {
} else {
row.push(line_this[j]);
}
}
rows.push(row);
entry_count++;
line = read_line()
}
var data=[];
for (var j in rows) {
if (j != 0) {
var ele = {};
for (var k in rows[j]) {
ele[rows[0][k]] = rows[j][k];
}
data.push(ele);
}
}
root["count"] = entry_count;
root[table_name] = data;
}
console.log(JSON.stringify(root));
});
}
if (process.argv.length < 3) {
console.log("No .md file passed!");
return;
}
if (process.argv.length < 4) {
anchorText = "###";
} else {
anchorText = process.argv[3];
}
handle(process.argv[2].toString(), anchorText);
-2
View File
@@ -1,2 +0,0 @@
flake8>=3.5.0
httplib2==0.18.0
-175
View File
@@ -1,175 +0,0 @@
#!/usr/bin/env python3
import re
import sys
anchor = '###'
min_entries_per_section = 3
auth_keys = ['apiKey', 'OAuth', 'X-Mashape-Key', 'No']
punctuation = ['.', '?', '!']
https_keys = ['Yes', 'No']
cors_keys = ['Yes', 'No', 'Unknown']
index_title = 0
index_desc = 1
index_auth = 2
index_https = 3
index_cors = 4
index_link = 5
num_segments = 5
errors = []
title_links = []
previous_links = []
anchor_re = re.compile(anchor + '\s(.+)')
section_title_re = re.compile('\*\s\[(.*)\]')
link_re = re.compile('\[(.+)\]\((http.*)\)')
def add_error(line_num, message):
"""adds an error to the dynamic error list"""
err = '(L{:03d}) {}'.format(line_num + 1, message)
errors.append(err)
def check_alphabetical(lines):
"""
checks if all entries per section are in alphabetical order based in entry title
"""
sections = {}
section_line_num = {}
for line_num, line in enumerate(lines):
if line.startswith(anchor):
category = line.split(anchor)[1].strip()
sections[category] = []
section_line_num[category] = line_num
continue
if not line.startswith('|') or line.startswith('|---'):
continue
raw_title = [x.strip() for x in line.split('|')[1:-1]][0]
title_re_match = link_re.match(raw_title)
if title_re_match:
sections[category].append(title_re_match.group(1).upper())
for category, entries in sections.items():
if sorted(entries) != entries:
add_error(section_line_num[category], "{} section is not in alphabetical order".format(category))
def check_entry(line_num, segments):
# START Title
raw_title = segments[index_title]
title_re_match = link_re.match(raw_title)
# url should be wrapped in '[TITLE](LINK)' Markdown syntax
if not title_re_match:
add_error(line_num, 'Title syntax should be "[TITLE](LINK)"')
else:
# do not allow "... API" in the entry title
title = title_re_match.group(1)
if title.upper().endswith(' API'):
add_error(line_num, 'Title should not end with "... API". Every entry is an API here!')
# do not allow duplicate links
link = title_re_match.group(2)
if link in previous_links:
add_error(line_num, 'Duplicate link - entries should only be included in one section')
else:
previous_links.append(link)
# END Title
# START Description
# first character should be capitalized
char = segments[index_desc][0]
if char.upper() != char:
add_error(line_num, "first character of description is not capitalized")
# last character should not punctuation
char = segments[index_desc][-1]
if char in punctuation:
add_error(line_num, "description should not end with {}".format(char))
desc_length = len(segments[index_desc])
if desc_length > 100:
add_error(line_num, "description should not exceed 100 characters (currently {})".format(desc_length))
# END Description
# START Auth
# values should conform to valid options only
auth = segments[index_auth]
if auth != 'No' and (not auth.startswith('`') or not auth.endswith('`')):
add_error(line_num, "auth value is not enclosed with `backticks`")
if auth.replace('`', '') not in auth_keys:
add_error(line_num, "{} is not a valid Auth option".format(auth))
# END Auth
# START HTTPS
# values should conform to valid options only
https = segments[index_https]
if https not in https_keys:
add_error(line_num, "{} is not a valid HTTPS option".format(https))
# END HTTPS
# START CORS
# values should conform to valid options only
cors = segments[index_cors]
if cors not in cors_keys:
add_error(line_num, "{} is not a valid CORS option".format(cors))
# END CORS
def check_format(filename):
"""
validates that each line is formatted correctly,
appending to error list as needed
"""
with open(filename) as fp:
lines = list(line.rstrip() for line in fp)
check_alphabetical(lines)
# START Check Entries
num_in_category = min_entries_per_section + 1
category = ""
category_line = 0
for line_num, line in enumerate(lines):
if section_title_re.match(line):
title_links.append(section_title_re.match(line).group(1))
# check each section for the minimum number of entries
if line.startswith(anchor):
match = anchor_re.match(line)
if match:
if match.group(1) not in title_links:
add_error(line_num, "section header ({}) not added as a title link".format(match.group(1)))
else:
add_error(line_num, "section header is not formatted correctly")
if num_in_category < min_entries_per_section:
add_error(category_line, "{} section does not have the minimum {} entries (only has {})".format(
category, min_entries_per_section, num_in_category))
category = line.split(' ')[1]
category_line = line_num
num_in_category = 0
continue
# skips lines that we do not care about
if not line.startswith('|') or line.startswith('|---'):
continue
num_in_category += 1
segments = line.split('|')[1:-1]
if len(segments) < num_segments:
add_error(line_num, "entry does not have all the required sections (have {}, need {})".format(
len(segments), num_segments))
continue
# START Global
for segment in segments:
# every line segment should start and end with exactly 1 space
if len(segment) - len(segment.lstrip()) != 1 or len(segment) - len(segment.rstrip()) != 1:
add_error(line_num, "each segment must start and end with exactly 1 space")
# END Global
segments = [seg.strip() for seg in segments]
check_entry(line_num, segments)
# END Check Entries
def main():
if len(sys.argv) < 2:
print("No file passed (file should contain Markdown table syntax)")
sys.exit(1)
check_format(sys.argv[1])
if len(errors) > 0:
for err in errors:
print(err)
sys.exit(1)
if __name__ == "__main__":
main()
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env ruby
auth_keys = ['apiKey', 'OAuth', 'X-Mashape-Key', 'No']
punctuation = ['.', '?', '!']
https_keys = ['Yes', 'No']
INDEX_TITLE = 1
INDEX_DESCRIPTION = 2
INDEX_AUTH = 3
INDEX_HTTPS = 4
INDEX_LINK = 5
filename = ARGV[0]
$errors = []
def add_error(line_num, val_index, message)
case val_index
when INDEX_TITLE
segment = "Title"
when INDEX_DESCRIPTION
segment = "Description"
when INDEX_AUTH
segment = "Auth"
when INDEX_HTTPS
segment = "HTTPS"
when INDEX_LINK
segment = "Link"
end
$errors.push("(L%03d) %-14.14s #{message}" % [line_num, segment])
end
File.foreach(filename).with_index do | line, line_num |
line_num += 1
# Skip non-markdown table lines and table schema lines
if !line.start_with?('|') || line.eql?("|---|---|---|---|---|\n")
next
end
values = line.split("|")
################### GLOBAL ###################
values.each.with_index do |val, val_index|
msg = ""
case val_index
when INDEX_TITLE..INDEX_LINK
# every line segment should start and end with exactly 1 space
if val[/\A */].size != 1 || val[/ *\z/].size != 1
add_error(line_num, val_index, "string should start and end with exactly 1 space")
end
end
end
################# DESCRIPTION ################
# First character should be capitalized
desc_val = values[INDEX_DESCRIPTION].lstrip.chop
if !/[[:upper:]]/.match(desc_val[0])
add_error(line_num, INDEX_DESCRIPTION, "first char not uppercase")
end
# value should not be punctuated
last_char = desc_val[desc_val.length-1]
if punctuation.include?(last_char)
add_error(line_num, INDEX_DESCRIPTION, "description should not end with \"#{last_char}\"")
end
#################### AUTH ####################
# Values should conform to valid options only
auth_val = values[INDEX_AUTH].lstrip.chop.tr('``', '')
if !auth_keys.include?(auth_val)
add_error(line_num, INDEX_AUTH, "not a valid option: #{auth_val}")
end
#################### HTTPS ###################
# Values should be either "Yes" or "No"
https_val = values[INDEX_HTTPS].lstrip.chop
if !https_keys.include?(https_val)
add_error(line_num, INDEX_HTTPS, "must use \"Yes\" or \"No\": #{https_val}")
end
#################### LINK ####################
# Url should be wrapped in "[Go!]" view
link_val = values[INDEX_LINK].lstrip.chop
if !link_val.start_with?("[Go!](") || !link_val.end_with?(')')
add_error(line_num, INDEX_LINK, "format should be \"[Go!](<LINK>)\": #{link_val}")
end
end
$errors.each do | e |
puts e
end
exit($errors.length)
-63
View File
@@ -1,63 +0,0 @@
#!/usr/bin/env python3
import httplib2
import re
import socket
import sys
def parse_links(filename):
"""Returns a list of URLs from text file"""
with open(filename) as fp:
data = fp.read()
raw_links = re.findall(
'((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'\".,<>?«»“”‘’]))',
data)
links = [raw_link[0] for raw_link in raw_links]
return links
def validate_links(links):
"""Checks each entry in JSON file for live link"""
print(f'Validating {len(links)} links...')
hasError = False
for link in links:
h = httplib2.Http(disable_ssl_certificate_validation=True, timeout=25)
try:
resp = h.request(link, headers={
# Faking user agent as some hosting services block not-whitelisted UA
'user-agent': 'Mozilla/5.0'
})
code = int(resp[0]['status'])
# Checking status code errors
if (code >= 300):
hasError = True
print(f"ERR:CLT:{code} : {link}")
except TimeoutError:
hasError = True
print(f"ERR:TMO: {link}")
except socket.error as socketerror:
hasError = True
print(f"ERR:SOC: {socketerror} : {link}")
except Exception as e:
hasError = True
# Ignore some exceptions which are not actually errors.
# The list below should be extended with other exceptions in the future if needed
if (-1 != str(e).find("[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)")):
print(f"ERR:SSL: {e} : {link}")
elif (-1 != str(e).find("Content purported to be compressed with gzip but failed to decompress.")):
print(f"ERR:GZP: {e} : {link}")
elif (-1 != str(e).find("Unable to find the server at")):
print(f"ERR:SRV: {e} : {link}")
else:
print(f"ERR:UKN: {e} : {link}")
return hasError
if __name__ == "__main__":
num_args = len(sys.argv)
if num_args < 2:
print("No .md file passed")
sys.exit(1)
hasError = validate_links(parse_links(sys.argv[1]))
if hasError:
sys.exit(1)
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env ruby
require 'httparty'
require 'ruby-progressbar'
require 'uri'
allowed_codes = [200, 302, 403, 429]
allowed_links = ["https://www.yelp.com/developers/documentation/v3"]
args = ARGV
filename = args[0]
contents = File.open(filename, 'rb') { |f| f.read }
raw_links = URI.extract(contents, ['http', 'https'])
# Remove trailing ')' from entry URLs
links = []
raw_links.each do |link|
if link.end_with?(')')
links.push(link[0...-1])
else
links.push(link)
end
end
if links.length <= 0
puts "no links to check"
exit(0)
end
fails = []
# Fail on any duplicate elements
dup = links.select{|element| links.count(element) > 1}
if dup.uniq.length > 0
dup.uniq.each do |e|
fails.push("(DUP): #{e}")
end
end
# Remove any duplicates from array
links = links.uniq
count = 0
total = links.length
progressbar = ProgressBar.create(:total => total,
:format => "%a %P% | Processed: %c from %C")
# GET each link and check for valid response code from allowed_codes
links.each do |link|
begin
count += 1
if allowed_links.include?(link)
next
end
res = HTTParty.get(link, timeout: 10)
if res.code.nil?
fails.push("(NIL): #{link}")
next
end
if !allowed_codes.include?(res.code)
fails.push("(#{res.code}): #{link}")
end
rescue Net::ReadTimeout
fails.push("(TMO): #{link}")
rescue Net::OpenTimeout
fails.push("(TMO): #{link}")
rescue OpenSSL::SSL::SSLError
fails.push("(SSL): #{link}")
rescue SocketError
fails.push("(SOK): #{link}")
rescue Errno::ECONNREFUSED
fails.push("(CON): #{link}")
end
progressbar.increment
end
puts "#{count}/#{total} links checked"
if fails.length <= 0
puts "all links valid"
exit(0)
else
puts "-- RESULTS --"
fails.sort!
fails.each do |e|
puts e
end
exit(1)
end
+2869
View File
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long