Adding GPU testing to Github workflows

[moazreyad]

This issue is open to discuss different options for adding GPU build and test to Github workflows.

To enable this feature, SINGA must provide a real or virtual machine with GPU as host machine for running the workflow. Then use the self-hosted runner feature of Github Actions. See also this MLOps video tutorial.

The team need to take some decisions:

1. Which machine(s) should we use? (e.g. virtual machine(s) on AWS, dedicated server(s) at NUS, ..)
2. Which operating systems we test on? (Only Linux? or also Mac).
3. When we run the GPU build and test workflow? (with every pull request? once per day at night? once per week? only on master branch?, ...)
4. Should we keep the machines always on? or use them only when the scheduled test is running and shut down them when there is no workflow runs? Assuming we run the GPU build and test only at scheduled time.
5. Should we add workflows to run examples and test the Jupyter notebooks? note that some examples may take hours or days to complete the training. But automating the test of examples can be very useful to speed up the development.

What do you think?

[chrishkchris]

I believe that what you propose in this issue is very important for us.
We need to safeguard our codebase, because a wrong PR could destroy the whole system functionality.
Therefore I believe that GPU testing is necessary.

Here are my opinions on the questions:

1. prefer dedicated server at NUS (not prefer cloud service providers because I think they are too expensive)
2. Linux is enough (ubuntu)
3. For every PR we need to run the test workflow (just like the current test settings for CPU versions, but need to upgrade the test to GPUs)
4. Prefer to keep the test server on so when someone submits a PR it will be checked immediately.
5. Maybe consider in the future to run the examples. I think the priority is to enable GPU build and test first.

[chrishkchris]

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

[chrishkchris]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

[NLGithubWP]

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

Ok, i will look into it

[chrishkchris]

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

Ok, i will look into it

FYI, the different between CPU version and GPU version in building SINGA is just the flag we used in cmake
GPU version: https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cuda10/Dockerfile#L104
cmake -DENABLE_TEST=ON -DUSE_CUDA=ON -DUSE_PYTHON3=ON -DUSE_DNNL=ON ..

CPU version: https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cpu/Dockerfile#L67
cmake -DENABLE_TEST=ON -DUSE_PYTHON3=ON -DUSE_DNNL=ON ..

In GPU version, it include all the GPU test case when we run test/python/run.py
In CPU version, it automatically skiped all the GPU test case

[moazreyad]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:

First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)

To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.

Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

[chrishkchris]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:

First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)

To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.

Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

@moazreyad thanks a lot for the information, I will consider it. Also I am thinking about if GPU emulator is another option? https://developer.nvidia.com/nvemulate

@naili-xing do you think the github actions can access our server to do gpu test, does it need to connect to nus VPN?

[NLGithubWP]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:
First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)
To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.
Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

@moazreyad thanks a lot for the information, I will consider it. Also I am thinking about if GPU emulator is another option? https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

@naili-xing do you think the github actions can access our server to do gpu test, does it need to connect to nus VPN?

I think it needs, if accessing the server from outside of nus, it needs vpn, if access from inside, using nus network, i think there is no needs

[chrishkchris]

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

[joddiy]

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

Hi, @chrishkchris , these two GPU emulators haven't been updated for more than 5 years. And since we have our own machine, the self-hosted GitHub Actions is a better solution.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

[chrishkchris]

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

Hi, @chrishkchris , these two GPU emulators haven't been updated for more than 5 years. And science we have our own machine, the self-hosted GitHub Actions is a better solution.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

@joddiy OK, then we go for the solution using our own machine gpu for github actions test first.

For our reference: communication between our server and github actions
https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github

[chrishkchris]

https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
FYI
"The self-hosted runner uses a HTTPS long poll that opens a connection to GitHub for 50 seconds"

[chrishkchris]

@moazreyad I have opened a ticket to ask the infrastructure and hopes they can give us a solution

https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21115?filter=allopenissues

[chrishkchris]

@moazreyad
They told us that we can use the API in the following link to add github secrets (to store token like username and password)
https://developer.github.com/v3/actions/secrets/

@joddiy
Could you help test the API to see if we can store some random token, and see if we are able to access what we have stored?
(e.g. you can use postman https://www.postman.com/ to test the restful API)