Add headers parameter on RobotFileParser

[eamanu]

BPO	38067
Nosy	@eamanu
PRs	some website requests User-Agent field in headers #15726

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-09-09.12:28:17.485>
labels = ['type-feature', 'library', '3.9']
title = 'Add headers parameter on RobotFileParser'
updated_at = <Date 2019-09-09.12:28:17.485>
user = 'https://github.com/eamanu'

bugs.python.org fields:

activity = <Date 2019-09-09.12:28:17.485>
actor = 'eamanu'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-09-09.12:28:17.485>
creator = 'eamanu'
dependencies = []
files = []
hgrepos = []
issue_num = 38067
keywords = []
message_count = 1.0
messages = ['351440']
nosy_count = 1.0
nosy_names = ['eamanu']
pr_nums = ['15726']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38067'
versions = ['Python 3.9']

[eamanu]

Hello I open an issue from a PR that I think need discussion:

I write here the @fuwaraido's PR description:

I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.

[orsenthil]

Addressed here #103753