Skip to content

Allow for 0 params in config-schema files of component repos #57

Open
Open
Allow for 0 params in config-schema files of component repos#57
Assignees
maany
Labels
bugSomething isn't working
Milestone
1.1.0
@maany

Description

@maany
maany
opened on Oct 25, 2019

Describe the bug
A clear and concise description of what the bug is.
The HTCondor-Batch component repository at present uses 0 parameters in the config-schema. Therefore, the following line in the compiler:

print " No of expected params: " + str(len(expected_params))

fails.

In theory, we should allow for component repositories that need absolutely zero additional configs from the site admins.

What input files were supplied to the compiler?
What output/error did you get?

To Reproduce

### url of schema
schema:  "https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml"

### Variable declaration:
global_variables:
  - &lightweight_component01_ip_address 188.185.112.251
  - &lightweight_component01_fqdn simple-condor-ce.cern.ch
  - &lightweight_component02_ip_address 188.185.84.16
  - &lightweight_component02_fqdn simple-condor-batch.cern.ch
  - &lightweight_component03_ip_address 188.185.70.127
  - &lightweight_component03_fqdn simple-condor-worker.cern.ch
  - &lightweight_component04_ip_address 188.185.85.178
  - &lightweight_component04_fqdn simple-condor-submit.cern.ch
htcondor_ce_runtime_variables:
  - &htcondor_ce_runtime_var_ce_host simple-condor-ce.cern.ch

htcondor_batch__runtime_variables:
  - &htcondor_runtime_var_batch_host simple-condor-batch.cern.ch

site:
  name: 'simple-dev-site'
  email: 'mayank.sharma@cern.ch'
  latitude: 46.3
  longitude: 6.2
  location: CERN, Geneva, Switzerland
  description: 'A demo site demonstrating the SIMPLE framework for site setup'
  website: 'https://home.cern'
  support_website: 'https://groups.google.com/forum/#!forum/wlcg-lightweight-sites'
  support_email: 'mayank.sharma@cern.ch'
  security_email: 'mayank.sharma@cern.ch'
  grid: 'wlcg' #site_other_grid: str(), wlcg, egi,osg
  tier: 3
  bdii_host: bdii.cern.ch
  cvmfs_http_proxy_list:
    - "http://cvmfs.cat.cbpf.br:3128"
    - "http://cvmfs-sec.cat.cbpf.br:3128"
  use_argus: no

preferred_tech_stack:
  level_1_configuration: puppet
  level_2_configuration: sh
  container_orchestration: docker-swarm
  container: docker

site_infrastructure:
  - fqdn: *lightweight_component01_fqdn
    ip_address: *lightweight_component01_ip_address
  - fqdn: *lightweight_component02_fqdn
    ip_address: *lightweight_component02_ip_address
  - fqdn: *lightweight_component03_fqdn
    ip_address: *lightweight_component03_ip_address
  - fqdn: *lightweight_component04_fqdn
    ip_address: *lightweight_component04_ip_address

lightweight_components:
  - type: compute_element
    name: HTCondor-CE
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce"
    repository_revision: "master"
    execution_id: 0
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    deploy:
      - node: *lightweight_component01_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    preferred_tech_stack:
      level_2_configuration: ""
    supplemental_config:
      some_additional_parameter: ""

  - name: HTCondor-Batch
    type: batch_system
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch"
    repository_revision: "master"
    execution_id: 1
    deploy:
      - node: *lightweight_component02_fqdn
        container_count: 1
    config:
      condor_host: ""
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""
    supplemental_config:
      some_additional_parameter: ""


  - type: worker_node
    name: HTCondor-Execute
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker"
    repository_revision: "master"
    execution_id: 2
    deploy:
      - node: *lightweight_component03_fqdn
        container_count: 1
      - node: *lightweight_component04_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""
    supplemental_config:
      some_additional_parameter: ""

supported_virtual_organizations:
  - *default_vo_alice
  - *default_vo_dteam
  - *default_vo_ops

voms_config:
  - voms_fqan: '/alice'
    pool_accounts:
      - *default_pool_accounts_alice
    vo: *default_vo_alice
  - voms_fqan: '/alice/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_alice
      - *default_pool_accounts_alicesgm
    vo: *default_vo_alice
    comment: sgm
  - voms_fqan: '/dteam'
    pool_accounts:
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
  - voms_fqan: '/dteam/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_dteamsgm
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
    comment: sgm
  - voms_fqan: '/ops'
    vo: *default_vo_ops
    pool_accounts:
      - *default_pool_accounts_ops
  - voms_fqan: '/ops/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_opssgm
      - *default_pool_accounts_ops
    vo: *default_vo_ops
    comment: sgm

Logs

/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/.virtualenv/bin/python /Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py ./tests/resources/condor-config.yaml -o ./tests/output.yaml -s ./tests/schema.yaml
[{'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker', 'revision': 'master'}]
RUNTIME$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Param 1: schema
     - required: True
     - use_default: True
     - specified: True
     - default_value: http://github.com/WLCG-lightweight-Sites/simple_grid_site_repo/simple-base-schema.yaml
     - user_defined_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
     - final_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
Param 2: global_variables
     - required: False
     - use_default: False
     - specified: True
     - user_defined_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
     - final_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
Param 3: site_infrastructure
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-ce.cern.ch
          - final_value: simple-condor-ce.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.112.251
          - final_value: 188.185.112.251
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-batch.cern.ch
          - final_value: simple-condor-batch.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.84.16
          - final_value: 188.185.84.16
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-worker.cern.ch
          - final_value: simple-condor-worker.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.70.127
          - final_value: 188.185.70.127
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-submit.cern.ch
          - final_value: simple-condor-submit.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.85.178
          - final_value: 188.185.85.178
Param 4: lightweight_components
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-CE
          - final_value: HTCondor-CE
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: compute_element
          - final_value: compute_element
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 0
          - final_value: 0
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-ce.cern.ch
               - final_value: simple-condor-ce.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host_execution_id', 1)])
          - final_value: ordereddict([('condor_host_execution_id', 1)])
     Param 10: supplemental_config
          Param 1: some_additional_parameter
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     -----------
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-Batch
          - final_value: HTCondor-Batch
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: batch_system
          - final_value: batch_system
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 1
          - final_value: 1
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-batch.cern.ch
               - final_value: simple-condor-batch.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host', '')])
          - final_value: ordereddict([('condor_host', '')])
     Param 10: supplemental_config
          Param 1: some_additional_parameter
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     -----------
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-Execute
          - final_value: HTCondor-Execute
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: worker_node
          - final_value: worker_node
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 2
          - final_value: 2
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-worker.cern.ch
               - final_value: simple-condor-worker.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
          -----------
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-submit.cern.ch
               - final_value: simple-condor-submit.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host_execution_id', 1)])
          - final_value: ordereddict([('condor_host_execution_id', 1)])
     Param 10: supplemental_config
          Param 1: some_additional_parameter
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
Component: HTCondor-CE
 No of expected params: 4
  - Param 1 :condor_host_execution_id
     - required: True
     - specified: True
     - user-defined-value: 1
     - Final Value: 1
  - Param 2 :uid_domain
     - required: True
     - specified: False
     - use-default: True
     - default-value: cern.ch
     - Final Value: cern.ch
  - Param 3 :gss_assist_gridmap_cache_expiration
     - required: True
     - specified: False
     - use-default: True
     - default-value: 7200
     - Final Value: 7200
  - Param 4 :per_job_history_dir
     - required: True
     - specified: False
     - use-default: True
     - default-value: /var/lib/gratia/condorce_data
     - Final Value: /var/lib/gratia/condorce_data
{'gss_assist_gridmap_cache_expiration': 7200, 'condor_host_execution_id': 1, 'per_job_history_dir': '/var/lib/gratia/condorce_data', 'uid_domain': 'cern.ch'}
Component: HTCondor-Batch
Traceback (most recent call last):
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 185, in <module>
    main()
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 181, in main
    execute_compiler(site_level_configuration_file, output, schema)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 163, in execute_compiler
    phase_5_output_file = phase_5(phase_4_output, runtime_vars, yaml)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 94, in phase_5
    augmented_config = processor_config_schemas.process_config_schema(lightweight_component[component_section], config_schema_file, default_data_runtime_file, meta_info)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 94, in process_config_schema
    final_config_for_component = get_final_config_values_for_component(component, expected_params, defaults_for_expected_params, meta_info)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 35, in get_final_config_values_for_component
    print " No of expected params: " + str(len(expected_params))
TypeError: object of type 'NoneType' has no len()

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Python
  • Compiler: v1.0.0

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions