mirror of
https://github.com/toddmotto/public-apis.git
synced 2021-09-13 01:36:33 +00:00
Merge pull request #1609 from public-apis/revert-1563-master
Revert "Update the link from blockchain.info to blockchain.com"
This commit is contained in:
+6
-16
@@ -4,43 +4,35 @@
|
||||
opened to market company APIs that offer paid solutions. This API list is not a marketing tool, but a tool to help the
|
||||
community build applications and use free, public APIs quickly and easily. Pull requests that are identified as marketing attempts will not be accepted.
|
||||
>
|
||||
> Please make sure the API you want to add has a full free access or at least a free tier before submitting.
|
||||
>
|
||||
> Thanks for understanding! :)
|
||||
|
||||
## Formatting
|
||||
|
||||
Current API entry format:
|
||||
|
||||
| API | Description | Auth | HTTPS | CORS |
|
||||
| API | Description | Auth | HTTPS | Link |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| API Title(Link to API documentation) | Description of API | Does this API require authentication? * | Does the API support HTTPS? | Does the API support [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)? * |
|
||||
| API Title | Description of API | Does this API require authentication? * | Does the API support HTTPS? | Link to API webpage |
|
||||
|
||||
Example entry:
|
||||
|
||||
```
|
||||
| [NASA](https://api.nasa.gov) | NASA data, including imagery | No | Yes | Yes |
|
||||
| NASA | NASA data, including imagery | No | Yes | [Go!](https://api.nasa.gov) |
|
||||
```
|
||||
|
||||
\* Currently, the only accepted inputs for the `Auth` field are as follows:
|
||||
Currently, the only accepted inputs for this field are as follows:
|
||||
|
||||
* `OAuth` - _the API supports OAuth_
|
||||
* `apiKey` - _the API uses a private key string/token for authentication - try and use the correct parameter_
|
||||
* `X-Mashape-Key` - _the name of the header which may need to be sent_
|
||||
* `No` - _the API requires no authentication to run_
|
||||
|
||||
\* Currently, the only accepted inputs for the `CORS` field are as follows:
|
||||
|
||||
* `Yes` - _the API supports CORS_
|
||||
* `No` - _the API does not support CORS_
|
||||
* `Unknown` - _it is unknown if the API supports CORS_
|
||||
|
||||
_Without proper [CORS configuration](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) an API will only be usable server side._
|
||||
|
||||
Please continue to follow the alphabetical ordering that is in place per section. Each table column should be padded with one space on either side.
|
||||
|
||||
If an API seems to fall into multiple categories, please place the listing within the section most in line with the services offered through the API. For example, the Instagram API is listed under `Social` since it is mainly a social network, even though it could also apply to `Photography`.
|
||||
|
||||
This project now offers a [JSON version](json/) of the list of services. These files are automatically updated as part of the Continuous Integration process, so there is no need to update them as part of the contribution.
|
||||
|
||||
## Pull Request
|
||||
|
||||
After you've created a branch on your fork with your changes, it's time to [make a pull request][pr-link]!
|
||||
@@ -56,7 +48,6 @@ merge conflicts will be less likely. See more detailed instructions [here][synci
|
||||
* Create a [branch][branch-link] for your edits.
|
||||
* Contribute in the style of the project as outlined above. This makes it easier for the collaborators to merge
|
||||
and for others to understand and maintain in the future.
|
||||
* Please make sure you squash all commits together before opening a pull request. If your pull request requires changes upon review, please be sure to squash all additional commits as well. [This wiki page][squash-link] outlines the squash process.
|
||||
|
||||
### Open Pull Requests
|
||||
|
||||
@@ -79,5 +70,4 @@ Thanks for being a part of this project, and we look forward to hearing from you
|
||||
[pr-link]: <https://help.github.com/articles/creating-a-pull-request/>
|
||||
[remote-link]: <https://help.github.com/articles/configuring-a-remote-for-a-fork/>
|
||||
[syncing-link]: <https://help.github.com/articles/syncing-a-fork>
|
||||
[squash-link]: <https://github.com/todotxt/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit>
|
||||
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
<!-- Thank you for taking the time to work on a Pull Request for this project! -->
|
||||
<!-- To ensure your PR is dealt with swiftly please check the following: -->
|
||||
- [ ] My submission is formatted according to the guidelines in the [contributing guide](CONTRIBUTING.md)
|
||||
- [ ] My addition is ordered alphabetically
|
||||
- [ ] My submission has a useful description
|
||||
- [ ] The description does not end with punctuation
|
||||
- [ ] Each table column is padded with one space on either side
|
||||
- [ ] I have searched the repository for any relevant issues or pull requests
|
||||
- [ ] Any category I am creating has the minimum requirement of 3 items
|
||||
- [ ] All changes have been [squashed][squash-link] into a single commit
|
||||
Thank you for taking the time to work on a Pull Request for this project!
|
||||
|
||||
[squash-link]: <https://github.com/todotxt/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit>
|
||||
To ensure your PR is dealt with swiftly please check the following:
|
||||
|
||||
- [ ] Your submissions are formatted according to the guidelines in the [contributing guide](CONTRIBUTING.md).
|
||||
- [ ] Your changes are made in the [README](../README.md) file, not the auto-generated JSON.
|
||||
- [ ] Your additions are ordered alphabetically.
|
||||
- [ ] Your submission has a useful description.
|
||||
- [ ] Each table column should be padded with one space on either side.
|
||||
- [ ] You have searched the repository for any relevant issues or PRs.
|
||||
- [ ] Any category you are creating has the minimum requirement of 3 items.
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
name: "Run tests"
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 0 * * *'
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
|
||||
env:
|
||||
FORMAT_FILE: README.md
|
||||
|
||||
jobs:
|
||||
test:
|
||||
name: 'Validate README.md'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Validate Markdown format
|
||||
run: build/validate_format.py ${FORMAT_FILE}
|
||||
|
||||
- name: Validate pull request changes
|
||||
run: build/github-pull.sh ${{ github.repository }} ${{ github.event.pull_request.number }} ${FORMAT_FILE}
|
||||
if: github.event_name == 'pull_request'
|
||||
@@ -1,22 +0,0 @@
|
||||
name: "Validate links"
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 0 * * *'
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
|
||||
env:
|
||||
FORMAT_FILE: README.md
|
||||
|
||||
jobs:
|
||||
test:
|
||||
name: 'Validate links'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Validate all links from README.md
|
||||
run: build/validate_links.py ${FORMAT_FILE}
|
||||
+4
-7
@@ -1,14 +1,11 @@
|
||||
language: python
|
||||
python: "3.6"
|
||||
language: node_js
|
||||
notifications:
|
||||
email: true
|
||||
before_install:
|
||||
- rvm install 2.4.0
|
||||
install:
|
||||
- pip install -r build/requirements.txt
|
||||
- gem install httparty ruby-progressbar
|
||||
before_script:
|
||||
# stop the build if there are Python syntax errors or undefined names
|
||||
- flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
|
||||
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
|
||||
- flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
|
||||
- cd build
|
||||
script:
|
||||
- ./main.sh
|
||||
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/bin/bash
|
||||
|
||||
# create json directory if not already present
|
||||
mkdir -p ../json
|
||||
# parse API README and print (minified) JSON to stdout, redirect to /json
|
||||
node condenseMd.js ../README.md > single_table.md
|
||||
node md2json.js single_table.md > ../json/entries.min.json
|
||||
# beautify the previously created JSON file, redirect to /json
|
||||
python -m json.tool ../json/entries.min.json > ../json/entries.json
|
||||
@@ -0,0 +1,63 @@
|
||||
fs = require('fs')
|
||||
|
||||
function setupMd(filename, anchor) {
|
||||
fs.readFile(filename, 'utf8', function (err,text) {
|
||||
if (err) {
|
||||
return console.log(err);
|
||||
}
|
||||
var lines = text.split("\n"),
|
||||
cur_line = 0,
|
||||
line = "",
|
||||
table_name = "",
|
||||
col_num = 0,
|
||||
cols = [],
|
||||
rows = [],
|
||||
arr = [];
|
||||
|
||||
|
||||
function read_line() {
|
||||
return lines[cur_line++];
|
||||
}
|
||||
|
||||
while (true) {
|
||||
var cols = [];
|
||||
var rows = [];
|
||||
while (line.indexOf(anchor) == -1 && cur_line != lines.length) {
|
||||
line = read_line();
|
||||
}
|
||||
if (cur_line == lines.length) {
|
||||
break;
|
||||
}
|
||||
table_name = line.split(anchor)[1];
|
||||
read_line()
|
||||
read_line()
|
||||
while (true) {
|
||||
line = read_line()
|
||||
if (line.length < 2 || cur_line == lines.length) {
|
||||
break
|
||||
}
|
||||
if (line.indexOf("|") == 0) {
|
||||
arr.push(line + table_name)
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
console.log(anchor + " entries")
|
||||
console.log("API | Description | Auth | HTTPS | Link | Category")
|
||||
console.log("|---|---|---|---|---|---|")
|
||||
for (i = 0; i < arr.length; i++) {
|
||||
console.log(arr[i])
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
if (process.argv.length < 3) {
|
||||
console.log("No .md file passed!");
|
||||
return;
|
||||
}
|
||||
if (process.argv.length < 4) {
|
||||
anchorText = "###";
|
||||
} else {
|
||||
anchorText = process.argv[3];
|
||||
}
|
||||
setupMd(process.argv[2].toString(), anchorText);
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/bin/bash
|
||||
|
||||
set -o errexit -o nounset
|
||||
|
||||
if [ "$TRAVIS_BRANCH" != "master" ]
|
||||
then
|
||||
echo "This commit was made against $TRAVIS_BRANCH and not master! No deploy!"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
rev=$(git rev-parse --short HEAD)
|
||||
|
||||
mkdir deploy
|
||||
cd deploy
|
||||
|
||||
git init
|
||||
git config --global user.name $GH_USER
|
||||
git config --global user.email $GH_EMAIL
|
||||
|
||||
git remote add upstream "https://$GH_TOKEN@github.com/toddmotto/public-apis.git"
|
||||
git fetch upstream
|
||||
git reset upstream/master
|
||||
|
||||
mv ../../json .
|
||||
|
||||
git add json/
|
||||
git commit -m "rebuild JSON at ${rev}" -m "[ci skip]"
|
||||
git push upstream HEAD:master
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
# Argument validation
|
||||
if [ $# -ne 3 ]; then
|
||||
echo "Usage: $0 <github-repo> <pull-number> <format-file>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Assign variables
|
||||
GITHUB_REPOSITORY="$1"
|
||||
GITHUB_PULL_REQUEST="$2"
|
||||
FORMAT_FILE="$3"
|
||||
|
||||
# Move to root of project
|
||||
cd "$GITHUB_WORKSPACE"
|
||||
|
||||
# Determine files
|
||||
FORMAT_FILE="$( realpath "${FORMAT_FILE}" )"
|
||||
|
||||
# Skip if build number could not be determined
|
||||
if [ -z "$GITHUB_REPOSITORY" -o -z "$GITHUB_PULL_REQUEST" ]; then
|
||||
echo "No pull request and/or repository is provided"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pull changes on PR
|
||||
echo "running on Pull Request #$GITHUB_PULL_REQUEST"
|
||||
|
||||
# Trick the URL validator python script into not seeing this as a URL
|
||||
DUMMY_SCHEME="https"
|
||||
DIFF_URL="$DUMMY_SCHEME://patch-diff.githubusercontent.com/raw/$GITHUB_REPOSITORY/pull/$GITHUB_PULL_REQUEST.diff"
|
||||
curl -L -o diff.txt "$DIFF_URL"
|
||||
|
||||
# Construct diff
|
||||
echo "------- BEGIN DIFF -------"
|
||||
cat diff.txt
|
||||
echo "-------- END DIFF --------"
|
||||
cat diff.txt | egrep "\+" > additions.txt
|
||||
|
||||
echo "------ BEGIN ADDITIONS -----"
|
||||
cat additions.txt
|
||||
echo "------- END ADDITIONS ------"
|
||||
LINK_FILE=additions.txt
|
||||
|
||||
# Validate links
|
||||
echo "Running link validation..."
|
||||
./build/validate_links.py "$LINK_FILE"
|
||||
|
||||
# Vebosity
|
||||
if [[ $? != 0 ]]; then
|
||||
echo "link validation failed!"
|
||||
exit 1
|
||||
else
|
||||
echo "link validation passed!"
|
||||
fi
|
||||
+27
-19
@@ -1,32 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
FORMAT_FILE=../README.md
|
||||
if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then
|
||||
echo "running on $TRAVIS_BRANCH branch"
|
||||
LINK_FILE=../README.md
|
||||
else
|
||||
echo "running on Pull Request #$TRAVIS_PULL_REQUEST"
|
||||
DIFF_URL="https://patch-diff.githubusercontent.com/raw/toddmotto/public-apis/pull/$TRAVIS_PULL_REQUEST.diff"
|
||||
curl $DIFF_URL > diff.txt
|
||||
echo "------- BEGIN DIFF -------"
|
||||
cat diff.txt
|
||||
echo "-------- END DIFF --------"
|
||||
cat diff.txt | egrep "\+" > additions.txt
|
||||
echo "------ BEGIN ADDITIONS -----"
|
||||
cat additions.txt
|
||||
echo "------- END ADDITIONS ------"
|
||||
LINK_FILE=additions.txt
|
||||
fi
|
||||
|
||||
echo "running format validation..."
|
||||
./validate_format.py $FORMAT_FILE
|
||||
./validate_format.rb $FORMAT_FILE
|
||||
if [[ $? != 0 ]]; then
|
||||
echo "format validation failed!"
|
||||
exit 1
|
||||
else
|
||||
echo "format validation passed!"
|
||||
./build.sh && ./deploy.sh
|
||||
if [[ $? != 0 ]]; then
|
||||
echo "JSON build and deploy failed!"
|
||||
else
|
||||
echo "JSON build and deploy success!"
|
||||
fi
|
||||
fi
|
||||
echo "format validation passed!"
|
||||
if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then
|
||||
echo "running on $TRAVIS_BRANCH branch - skipping Pull Request logic"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "running on Pull Request #$TRAVIS_PULL_REQUEST"
|
||||
DIFF_URL="https://patch-diff.githubusercontent.com/raw/toddmotto/public-apis/pull/$TRAVIS_PULL_REQUEST.diff"
|
||||
curl $DIFF_URL > diff.txt
|
||||
echo "------- BEGIN DIFF -------"
|
||||
cat diff.txt
|
||||
echo "-------- END DIFF --------"
|
||||
cat diff.txt | egrep "\+" > additions.txt
|
||||
echo "------ BEGIN ADDITIONS -----"
|
||||
cat additions.txt
|
||||
echo "------- END ADDITIONS ------"
|
||||
LINK_FILE=additions.txt
|
||||
|
||||
echo "running link validation..."
|
||||
./validate_links.py $LINK_FILE
|
||||
./validate_links.rb $LINK_FILE
|
||||
if [[ $? != 0 ]]; then
|
||||
echo "link validation failed!"
|
||||
exit 1
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
fs = require('fs')
|
||||
|
||||
function md_trim(str, context) {
|
||||
str = str.replace(/(^\s+)|(\s+$)/g, "");
|
||||
|
||||
if (context == 1) { // Name
|
||||
// placeholder for any formatting on name value
|
||||
} else if (context == 2) { // Description
|
||||
str = str.replace(".", ""); // remove ending periods on descriptions
|
||||
} else if (context == 3) { // Auth
|
||||
if (str.toUpperCase() == "NO") {
|
||||
str = null
|
||||
} else {
|
||||
str = str.replace("`", "").replace("`", "")
|
||||
}
|
||||
} else if (context == 4) { // HTTPS
|
||||
if (str.toUpperCase() == "YES") {
|
||||
str = true
|
||||
} else {
|
||||
str = false
|
||||
}
|
||||
} else if (context == 5) { // Link
|
||||
str = str.replace("[Go!]", "").slice(1, -1);
|
||||
}
|
||||
return str;
|
||||
}
|
||||
|
||||
function handle(filename, anchor) {
|
||||
fs.readFile(filename, 'utf8', function (err,text) {
|
||||
if (err) {
|
||||
return console.log(err);
|
||||
}
|
||||
var lines = text.split("\n");
|
||||
var cur_line = 0;
|
||||
var line = ""
|
||||
var table_name = "";
|
||||
var col_num = 0;
|
||||
var cols = [];
|
||||
var rows = [];
|
||||
var entry_count = 0;
|
||||
|
||||
function read_line() {
|
||||
return lines[cur_line++];
|
||||
}
|
||||
var root = {};
|
||||
while (true) {
|
||||
var cols = [];
|
||||
var rows = [];
|
||||
while (line.indexOf(anchor) == -1 && cur_line != lines.length) {
|
||||
line = read_line();
|
||||
}
|
||||
if (cur_line == lines.length) {
|
||||
break;
|
||||
}
|
||||
table_name = line.split(anchor)[1];
|
||||
table_name = md_trim(table_name, 0)
|
||||
|
||||
line = read_line()
|
||||
|
||||
if (line) {
|
||||
line = line.split("|")
|
||||
for (var j in line) {
|
||||
|
||||
line[j] = md_trim(line[j], 0)
|
||||
if ((j == 0 || j == line.length - 1) && line[j] === "") {
|
||||
|
||||
} else {
|
||||
cols.push(line[j]);
|
||||
}
|
||||
}
|
||||
if (line.length) {
|
||||
cols = line;
|
||||
rows.push(cols)
|
||||
} else {
|
||||
console.error("markdown expect column title")
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
console.error("markdown expect table content")
|
||||
break;
|
||||
}
|
||||
|
||||
line = read_line()
|
||||
|
||||
if (!line) {
|
||||
console.error("markdown expect table spliter")
|
||||
break;
|
||||
}
|
||||
line = read_line()
|
||||
while (line.indexOf("|") != -1 && cur_line != lines.length) {
|
||||
|
||||
var line_this = line.split("|")
|
||||
var row = []
|
||||
for (var j in line_this) {
|
||||
line_this[j] = md_trim(line_this[j], j)
|
||||
if ((j == 0 || j == line_this.length - 1) && line_this[j] === "") {
|
||||
|
||||
} else {
|
||||
row.push(line_this[j]);
|
||||
}
|
||||
|
||||
}
|
||||
rows.push(row);
|
||||
entry_count++;
|
||||
line = read_line()
|
||||
}
|
||||
|
||||
var data=[];
|
||||
for (var j in rows) {
|
||||
if (j != 0) {
|
||||
var ele = {};
|
||||
for (var k in rows[j]) {
|
||||
ele[rows[0][k]] = rows[j][k];
|
||||
}
|
||||
data.push(ele);
|
||||
}
|
||||
}
|
||||
root["count"] = entry_count;
|
||||
root[table_name] = data;
|
||||
}
|
||||
console.log(JSON.stringify(root));
|
||||
});
|
||||
}
|
||||
|
||||
if (process.argv.length < 3) {
|
||||
console.log("No .md file passed!");
|
||||
return;
|
||||
}
|
||||
if (process.argv.length < 4) {
|
||||
anchorText = "###";
|
||||
} else {
|
||||
anchorText = process.argv[3];
|
||||
}
|
||||
handle(process.argv[2].toString(), anchorText);
|
||||
@@ -1,2 +0,0 @@
|
||||
flake8>=3.5.0
|
||||
httplib2==0.18.0
|
||||
@@ -1,175 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import re
|
||||
import sys
|
||||
|
||||
anchor = '###'
|
||||
min_entries_per_section = 3
|
||||
auth_keys = ['apiKey', 'OAuth', 'X-Mashape-Key', 'No']
|
||||
punctuation = ['.', '?', '!']
|
||||
https_keys = ['Yes', 'No']
|
||||
cors_keys = ['Yes', 'No', 'Unknown']
|
||||
|
||||
index_title = 0
|
||||
index_desc = 1
|
||||
index_auth = 2
|
||||
index_https = 3
|
||||
index_cors = 4
|
||||
index_link = 5
|
||||
num_segments = 5
|
||||
|
||||
errors = []
|
||||
title_links = []
|
||||
previous_links = []
|
||||
anchor_re = re.compile(anchor + '\s(.+)')
|
||||
section_title_re = re.compile('\*\s\[(.*)\]')
|
||||
link_re = re.compile('\[(.+)\]\((http.*)\)')
|
||||
|
||||
|
||||
def add_error(line_num, message):
|
||||
"""adds an error to the dynamic error list"""
|
||||
err = '(L{:03d}) {}'.format(line_num + 1, message)
|
||||
errors.append(err)
|
||||
|
||||
|
||||
def check_alphabetical(lines):
|
||||
"""
|
||||
checks if all entries per section are in alphabetical order based in entry title
|
||||
"""
|
||||
sections = {}
|
||||
section_line_num = {}
|
||||
for line_num, line in enumerate(lines):
|
||||
if line.startswith(anchor):
|
||||
category = line.split(anchor)[1].strip()
|
||||
sections[category] = []
|
||||
section_line_num[category] = line_num
|
||||
continue
|
||||
if not line.startswith('|') or line.startswith('|---'):
|
||||
continue
|
||||
raw_title = [x.strip() for x in line.split('|')[1:-1]][0]
|
||||
title_re_match = link_re.match(raw_title)
|
||||
if title_re_match:
|
||||
sections[category].append(title_re_match.group(1).upper())
|
||||
|
||||
for category, entries in sections.items():
|
||||
if sorted(entries) != entries:
|
||||
add_error(section_line_num[category], "{} section is not in alphabetical order".format(category))
|
||||
|
||||
|
||||
def check_entry(line_num, segments):
|
||||
# START Title
|
||||
raw_title = segments[index_title]
|
||||
title_re_match = link_re.match(raw_title)
|
||||
# url should be wrapped in '[TITLE](LINK)' Markdown syntax
|
||||
if not title_re_match:
|
||||
add_error(line_num, 'Title syntax should be "[TITLE](LINK)"')
|
||||
else:
|
||||
# do not allow "... API" in the entry title
|
||||
title = title_re_match.group(1)
|
||||
if title.upper().endswith(' API'):
|
||||
add_error(line_num, 'Title should not end with "... API". Every entry is an API here!')
|
||||
# do not allow duplicate links
|
||||
link = title_re_match.group(2)
|
||||
if link in previous_links:
|
||||
add_error(line_num, 'Duplicate link - entries should only be included in one section')
|
||||
else:
|
||||
previous_links.append(link)
|
||||
# END Title
|
||||
# START Description
|
||||
# first character should be capitalized
|
||||
char = segments[index_desc][0]
|
||||
if char.upper() != char:
|
||||
add_error(line_num, "first character of description is not capitalized")
|
||||
# last character should not punctuation
|
||||
char = segments[index_desc][-1]
|
||||
if char in punctuation:
|
||||
add_error(line_num, "description should not end with {}".format(char))
|
||||
desc_length = len(segments[index_desc])
|
||||
if desc_length > 100:
|
||||
add_error(line_num, "description should not exceed 100 characters (currently {})".format(desc_length))
|
||||
# END Description
|
||||
# START Auth
|
||||
# values should conform to valid options only
|
||||
auth = segments[index_auth]
|
||||
if auth != 'No' and (not auth.startswith('`') or not auth.endswith('`')):
|
||||
add_error(line_num, "auth value is not enclosed with `backticks`")
|
||||
if auth.replace('`', '') not in auth_keys:
|
||||
add_error(line_num, "{} is not a valid Auth option".format(auth))
|
||||
# END Auth
|
||||
# START HTTPS
|
||||
# values should conform to valid options only
|
||||
https = segments[index_https]
|
||||
if https not in https_keys:
|
||||
add_error(line_num, "{} is not a valid HTTPS option".format(https))
|
||||
# END HTTPS
|
||||
# START CORS
|
||||
# values should conform to valid options only
|
||||
cors = segments[index_cors]
|
||||
if cors not in cors_keys:
|
||||
add_error(line_num, "{} is not a valid CORS option".format(cors))
|
||||
# END CORS
|
||||
|
||||
|
||||
def check_format(filename):
|
||||
"""
|
||||
validates that each line is formatted correctly,
|
||||
appending to error list as needed
|
||||
"""
|
||||
with open(filename) as fp:
|
||||
lines = list(line.rstrip() for line in fp)
|
||||
check_alphabetical(lines)
|
||||
# START Check Entries
|
||||
num_in_category = min_entries_per_section + 1
|
||||
category = ""
|
||||
category_line = 0
|
||||
for line_num, line in enumerate(lines):
|
||||
if section_title_re.match(line):
|
||||
title_links.append(section_title_re.match(line).group(1))
|
||||
# check each section for the minimum number of entries
|
||||
if line.startswith(anchor):
|
||||
match = anchor_re.match(line)
|
||||
if match:
|
||||
if match.group(1) not in title_links:
|
||||
add_error(line_num, "section header ({}) not added as a title link".format(match.group(1)))
|
||||
else:
|
||||
add_error(line_num, "section header is not formatted correctly")
|
||||
if num_in_category < min_entries_per_section:
|
||||
add_error(category_line, "{} section does not have the minimum {} entries (only has {})".format(
|
||||
category, min_entries_per_section, num_in_category))
|
||||
category = line.split(' ')[1]
|
||||
category_line = line_num
|
||||
num_in_category = 0
|
||||
continue
|
||||
# skips lines that we do not care about
|
||||
if not line.startswith('|') or line.startswith('|---'):
|
||||
continue
|
||||
num_in_category += 1
|
||||
segments = line.split('|')[1:-1]
|
||||
if len(segments) < num_segments:
|
||||
add_error(line_num, "entry does not have all the required sections (have {}, need {})".format(
|
||||
len(segments), num_segments))
|
||||
continue
|
||||
# START Global
|
||||
for segment in segments:
|
||||
# every line segment should start and end with exactly 1 space
|
||||
if len(segment) - len(segment.lstrip()) != 1 or len(segment) - len(segment.rstrip()) != 1:
|
||||
add_error(line_num, "each segment must start and end with exactly 1 space")
|
||||
# END Global
|
||||
segments = [seg.strip() for seg in segments]
|
||||
check_entry(line_num, segments)
|
||||
# END Check Entries
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("No file passed (file should contain Markdown table syntax)")
|
||||
sys.exit(1)
|
||||
check_format(sys.argv[1])
|
||||
if len(errors) > 0:
|
||||
for err in errors:
|
||||
print(err)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env ruby
|
||||
|
||||
auth_keys = ['apiKey', 'OAuth', 'X-Mashape-Key', 'No']
|
||||
punctuation = ['.', '?', '!']
|
||||
https_keys = ['Yes', 'No']
|
||||
|
||||
INDEX_TITLE = 1
|
||||
INDEX_DESCRIPTION = 2
|
||||
INDEX_AUTH = 3
|
||||
INDEX_HTTPS = 4
|
||||
INDEX_LINK = 5
|
||||
filename = ARGV[0]
|
||||
$errors = []
|
||||
|
||||
def add_error(line_num, val_index, message)
|
||||
case val_index
|
||||
when INDEX_TITLE
|
||||
segment = "Title"
|
||||
when INDEX_DESCRIPTION
|
||||
segment = "Description"
|
||||
when INDEX_AUTH
|
||||
segment = "Auth"
|
||||
when INDEX_HTTPS
|
||||
segment = "HTTPS"
|
||||
when INDEX_LINK
|
||||
segment = "Link"
|
||||
end
|
||||
|
||||
$errors.push("(L%03d) %-14.14s #{message}" % [line_num, segment])
|
||||
end
|
||||
|
||||
File.foreach(filename).with_index do | line, line_num |
|
||||
line_num += 1
|
||||
|
||||
# Skip non-markdown table lines and table schema lines
|
||||
if !line.start_with?('|') || line.eql?("|---|---|---|---|---|\n")
|
||||
next
|
||||
end
|
||||
|
||||
values = line.split("|")
|
||||
|
||||
################### GLOBAL ###################
|
||||
values.each.with_index do |val, val_index|
|
||||
msg = ""
|
||||
case val_index
|
||||
when INDEX_TITLE..INDEX_LINK
|
||||
# every line segment should start and end with exactly 1 space
|
||||
if val[/\A */].size != 1 || val[/ *\z/].size != 1
|
||||
add_error(line_num, val_index, "string should start and end with exactly 1 space")
|
||||
end
|
||||
end
|
||||
end
|
||||
################# DESCRIPTION ################
|
||||
# First character should be capitalized
|
||||
desc_val = values[INDEX_DESCRIPTION].lstrip.chop
|
||||
if !/[[:upper:]]/.match(desc_val[0])
|
||||
add_error(line_num, INDEX_DESCRIPTION, "first char not uppercase")
|
||||
end
|
||||
# value should not be punctuated
|
||||
last_char = desc_val[desc_val.length-1]
|
||||
if punctuation.include?(last_char)
|
||||
add_error(line_num, INDEX_DESCRIPTION, "description should not end with \"#{last_char}\"")
|
||||
end
|
||||
#################### AUTH ####################
|
||||
# Values should conform to valid options only
|
||||
auth_val = values[INDEX_AUTH].lstrip.chop.tr('``', '')
|
||||
if !auth_keys.include?(auth_val)
|
||||
add_error(line_num, INDEX_AUTH, "not a valid option: #{auth_val}")
|
||||
end
|
||||
#################### HTTPS ###################
|
||||
# Values should be either "Yes" or "No"
|
||||
https_val = values[INDEX_HTTPS].lstrip.chop
|
||||
if !https_keys.include?(https_val)
|
||||
add_error(line_num, INDEX_HTTPS, "must use \"Yes\" or \"No\": #{https_val}")
|
||||
end
|
||||
#################### LINK ####################
|
||||
# Url should be wrapped in "[Go!]" view
|
||||
link_val = values[INDEX_LINK].lstrip.chop
|
||||
if !link_val.start_with?("[Go!](") || !link_val.end_with?(')')
|
||||
add_error(line_num, INDEX_LINK, "format should be \"[Go!](<LINK>)\": #{link_val}")
|
||||
end
|
||||
end
|
||||
$errors.each do | e |
|
||||
puts e
|
||||
end
|
||||
exit($errors.length)
|
||||
@@ -1,63 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import httplib2
|
||||
import re
|
||||
import socket
|
||||
import sys
|
||||
|
||||
|
||||
def parse_links(filename):
|
||||
"""Returns a list of URLs from text file"""
|
||||
with open(filename) as fp:
|
||||
data = fp.read()
|
||||
raw_links = re.findall(
|
||||
'((?:https?://|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'\".,<>?«»“”‘’]))',
|
||||
data)
|
||||
links = [raw_link[0] for raw_link in raw_links]
|
||||
return links
|
||||
|
||||
|
||||
def validate_links(links):
|
||||
"""Checks each entry in JSON file for live link"""
|
||||
print(f'Validating {len(links)} links...')
|
||||
hasError = False
|
||||
for link in links:
|
||||
h = httplib2.Http(disable_ssl_certificate_validation=True, timeout=25)
|
||||
try:
|
||||
resp = h.request(link, headers={
|
||||
# Faking user agent as some hosting services block not-whitelisted UA
|
||||
'user-agent': 'Mozilla/5.0'
|
||||
})
|
||||
code = int(resp[0]['status'])
|
||||
# Checking status code errors
|
||||
if (code >= 300):
|
||||
hasError = True
|
||||
print(f"ERR:CLT:{code} : {link}")
|
||||
except TimeoutError:
|
||||
hasError = True
|
||||
print(f"ERR:TMO: {link}")
|
||||
except socket.error as socketerror:
|
||||
hasError = True
|
||||
print(f"ERR:SOC: {socketerror} : {link}")
|
||||
except Exception as e:
|
||||
hasError = True
|
||||
# Ignore some exceptions which are not actually errors.
|
||||
# The list below should be extended with other exceptions in the future if needed
|
||||
if (-1 != str(e).find("[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)")):
|
||||
print(f"ERR:SSL: {e} : {link}")
|
||||
elif (-1 != str(e).find("Content purported to be compressed with gzip but failed to decompress.")):
|
||||
print(f"ERR:GZP: {e} : {link}")
|
||||
elif (-1 != str(e).find("Unable to find the server at")):
|
||||
print(f"ERR:SRV: {e} : {link}")
|
||||
else:
|
||||
print(f"ERR:UKN: {e} : {link}")
|
||||
return hasError
|
||||
|
||||
if __name__ == "__main__":
|
||||
num_args = len(sys.argv)
|
||||
if num_args < 2:
|
||||
print("No .md file passed")
|
||||
sys.exit(1)
|
||||
hasError = validate_links(parse_links(sys.argv[1]))
|
||||
if hasError:
|
||||
sys.exit(1)
|
||||
Executable
+77
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env ruby
|
||||
require 'httparty'
|
||||
require 'ruby-progressbar'
|
||||
require 'uri'
|
||||
allowed_codes = [200, 302, 403, 429]
|
||||
allowed_links = ["https://www.yelp.com/developers/documentation/v3"]
|
||||
args = ARGV
|
||||
filename = args[0]
|
||||
contents = File.open(filename, 'rb') { |f| f.read }
|
||||
raw_links = URI.extract(contents, ['http', 'https'])
|
||||
# Remove trailing ')' from entry URLs
|
||||
links = []
|
||||
raw_links.each do |link|
|
||||
if link.end_with?(')')
|
||||
links.push(link[0...-1])
|
||||
else
|
||||
links.push(link)
|
||||
end
|
||||
end
|
||||
if links.length <= 0
|
||||
puts "no links to check"
|
||||
exit(0)
|
||||
end
|
||||
fails = []
|
||||
# Fail on any duplicate elements
|
||||
dup = links.select{|element| links.count(element) > 1}
|
||||
if dup.uniq.length > 0
|
||||
dup.uniq.each do |e|
|
||||
fails.push("(DUP): #{e}")
|
||||
end
|
||||
end
|
||||
# Remove any duplicates from array
|
||||
links = links.uniq
|
||||
count = 0
|
||||
total = links.length
|
||||
progressbar = ProgressBar.create(:total => total,
|
||||
:format => "%a %P% | Processed: %c from %C")
|
||||
# GET each link and check for valid response code from allowed_codes
|
||||
links.each do |link|
|
||||
begin
|
||||
count += 1
|
||||
if allowed_links.include?(link)
|
||||
next
|
||||
end
|
||||
res = HTTParty.get(link, timeout: 10)
|
||||
if res.code.nil?
|
||||
fails.push("(NIL): #{link}")
|
||||
next
|
||||
end
|
||||
if !allowed_codes.include?(res.code)
|
||||
fails.push("(#{res.code}): #{link}")
|
||||
end
|
||||
rescue Net::ReadTimeout
|
||||
fails.push("(TMO): #{link}")
|
||||
rescue Net::OpenTimeout
|
||||
fails.push("(TMO): #{link}")
|
||||
rescue OpenSSL::SSL::SSLError
|
||||
fails.push("(SSL): #{link}")
|
||||
rescue SocketError
|
||||
fails.push("(SOK): #{link}")
|
||||
rescue Errno::ECONNREFUSED
|
||||
fails.push("(CON): #{link}")
|
||||
end
|
||||
progressbar.increment
|
||||
end
|
||||
puts "#{count}/#{total} links checked"
|
||||
if fails.length <= 0
|
||||
puts "all links valid"
|
||||
exit(0)
|
||||
else
|
||||
puts "-- RESULTS --"
|
||||
fails.sort!
|
||||
fails.each do |e|
|
||||
puts e
|
||||
end
|
||||
exit(1)
|
||||
end
|
||||
+2869
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user