Loading repository data…
google / repository
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++14).
The Robots Exclusion Protocol (REP) is a standard that enables website owners to control which URLs may be accessed by automated clients (i.e. crawlers) through a simple text file with a specific syntax. It's one of the basic building blocks of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to confusion. This project aims to fix that by releasing the parser that Google uses.
The library is slightly modified (i.e. some internal headers and equivalent symbols) production code used by Googlebot, Google's crawler, to determine which URLs it may access based on rules provided by webmasters in robots.txt files. The library is released open-source to help developers build tools that better reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a single URL and user-agent against a robots.txt.
We included with the library a small binary to test a local robots.txt against a user-agent and URL. Running the included binary requires:
Bazel is the official build system for the library, which is supported on most major platforms (Linux, Windows, MacOS, for example) and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with URI 'https://example.com/url': ALLOWED
CMake is the community-supported build system for the library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build && cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with URI 'https://example.com/url': ALLOWED
Note: If the robots file is empty, the parser also prints:
notice: robots file is empty so all user-agents are allowed
Exit codes:
0= ALLOWED,1= DISALLOWED
Parsing of robots.txt files themselves is done exactly as in the production version of Googlebot, including how percent codes and unicode characters in patterns are handled. The user must ensure however that the URI passed to the AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter of the robots tool, follows the format specified by RFC3986, since this library will not perform full normalization of those URI parameters. Only if the URI is in this format, the matching will be done according to the REP specification.
Also note that the library, and the included binary, do not handle
implementation logic that a crawler might apply outside of parsing and matching,
for example: Googlebot-Image respecting the rules specified for User-agent: Googlebot if not explicitly defined in the robots.txt file being tested.
The robots.txt parser and matcher C++ library is licensed under the terms of the Apache license. See LICENSE for more information.
To learn more about this project:
Selected from shared topics, language and repository description—not editorial ratings.
wisdompeak /
This repository contains the solutions and explanations to the algorithm problems on LeetCode. Only medium or above are included. All are written in C++/Python and implemented by myself. The problems attempted multiple times are labelled with hyperlinks.
80/100 healthbrandonprry /
This repository contains full code examples from the book Gray Hat C#
85/100 healthblack-shadows /
The repository contains solutions to various problems on interviewbit. The code is merely a snippet (as solved on InterviewBit) & hence is not executable in a c++ compiler. The code written is purely original and completely my own.
62/100 healthevgenkarlson /
| SCHOOL_42_UPDATE 2020 | This repository contains many versions of completed and commented out tasks for each day of the very first "Piscine C"( Program | Course | Programing | Coding | School 42 | Ecole 42 | School 21 | Школа 42 | Школа 21 ) Этот репозиторий содержит множество версий выполненных и закомментированных задач на каждый день самого первого "Бассейна по Си".
85/100 healthblack-shadows /
The repository contains solutions to various problems on leetcode. The code is merely a snippet (as solved on LeetCode) & hence is not executable in a c++ compiler. The code written is purely original and completely my own.
71/100 healthYou can run it on pynq z1. The repository contains the relevant Verilog code, Vivado configuration and C code for sdk testing. The size of the systolic array can be changed, now it is 16X16.
67/100 health