[FEATURE] Clear `url` ambiguity between `config.yaml` and `jobs.yaml`

[drws]

Is your feature request related to a problem? Please describe.

Compare these two cases of setting User Agent string:

1. Setting default UA in config.yaml:

job_defaults:
  url:
    headers:
      User-Agent: 'Mozilla/5.0'

2. Setting job-specific UA in jobs.yaml:

url: https://lynx.invisible-island.net
headers:
  User-Agent: 'Lynx'

The difference in url structure between config.yaml and jobs.yaml introduces unnecessary ambiguity.

Describe the solution you'd like.

For example:

config.yaml:

job_defaults:
  http:
    headers:
      User-Agent: 'Mozilla/5.0'

jobs.yaml:

url: https://lynx.invisible-island.net
http:
  headers:
    User-Agent: 'Lynx'

[mborsetti]

@drws Thank you for your contribution(s)!

Agree that the legacy jobs.config is not ideal.

However, your suggestion would make it hard if not impossible to have different headers for url and browser jobs, which is a useful and sometimes necessary requirement.

Will let this stew a bit...

Thanks for your contribution!

[drws]

There is no need to throw away any modularity, the solution could be:

config.yaml:

job_defaults:
  http:
    headers:
      User-Agent: 'Mozilla/5.0'
  browser:
    http:
      headers:
        User-Agent: 'Chrome'

jobs.yaml:

url: https://lynx.invisible-island.net
http:
  headers:
    User-Agent: 'Lynx'