gh-60055: Allow passing a Request instance for RobotParser URLs#103753
Conversation
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
|
All commit authors signed the Contributor License Agreement. |
|
This PR is stale because it has been open for 30 days with no activity. |
orsenthil
force-pushed
the
gh-60055
branch
2 times, most recently
from
cf08d64 to
40444c2
Compare
Documentation build overview
|
serhiy-storchaka
left a comment
serhiy-storchaka
left a comment
There was a problem hiding this comment.
The direction is fine, but I have more than a couple of remarks.
| This class provides methods to read, parse and answer questions about the | ||
| :file:`robots.txt` file at *url*. | ||
| :file:`robots.txt` file at *url* or a :class:`urllib.request.Request` object | ||
| with additional user-agent headers populated. |
There was a problem hiding this comment.
Add versionchanged.
Headers aren't limited to user-agent.
The wording in general could be improved.
There was a problem hiding this comment.
I have simplified it :file:robots.txt file at url or a :class:urllib.request.Request object. Please suggest if it should be anything else.
|
@serhiy-storchaka , I have addressed your review comments. Thank you for those. |
serhiy-storchaka
left a comment
serhiy-storchaka
left a comment
There was a problem hiding this comment.
LGTM, besides few nitpicks.
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
4 participants
RobotParseronly needs to know the type of the URL param in order to set the undocumentedhostandpathproperties, so when callingset_urluse the pre-parsed attributes if given aRequestobject instance. Otherwise the only use of the url is when it is passed tourllib.request.urlopenwhich accepts eitherstrorRequestURLs.Main driver for this change was to set the User-Agent header so #60055 seemed like the appropriate issue to link.