Skip to content

ci: rewrite deploy scripts with python3 #7046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ElectricalBoy wants to merge 40 commits into from
Closed

ci: rewrite deploy scripts with python3 #7046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
1790105
rewrite login_and_get_token
ElectricalBoy Feb 3, 2026
f5b7092
add deploy script
ElectricalBoy Feb 3, 2026
5569f26
syntax
ElectricalBoy Feb 4, 2026
37161d5
disable async deploy
ElectricalBoy Feb 4, 2026
5da3213
rewrite with requests
ElectricalBoy Feb 4, 2026
945712e
memoize tokens
ElectricalBoy Feb 4, 2026
5a14d10
res deploy script to python
ElectricalBoy Feb 4, 2026
c699946
refactor
ElectricalBoy Feb 4, 2026
965097e
protect_page script to python
ElectricalBoy Feb 4, 2026
308f91c
protect script to python
ElectricalBoy Feb 4, 2026
7c2fd6d
remove dev script to python
ElectricalBoy Feb 4, 2026
ec68df6
adjust main call
ElectricalBoy Feb 4, 2026
6d7e592
update gitignore
ElectricalBoy Feb 4, 2026
ab8a99e
update workflows
ElectricalBoy Feb 4, 2026
e3a97bc
throw away shellscript versions
ElectricalBoy Feb 4, 2026
515a4fb
disable python cache in GHA
ElectricalBoy Feb 4, 2026
151fc48
update workflows
ElectricalBoy Feb 4, 2026
c1a656a
bump deploy bot version
ElectricalBoy Feb 5, 2026
4114277
kick debugging print
ElectricalBoy Feb 5, 2026
cfde037
suppress python stdout buffering
ElectricalBoy Feb 5, 2026
ae49e94
use rglob
ElectricalBoy Feb 5, 2026
25532aa
cleanup
ElectricalBoy Feb 5, 2026
c5ac6d7
lint
ElectricalBoy Feb 5, 2026
7a20721
key safety
ElectricalBoy Feb 5, 2026
511fbfd
add python check workflow
ElectricalBoy Feb 5, 2026
1b2db14
adjust workflow
ElectricalBoy Feb 5, 2026
621bcd1
format
ElectricalBoy Feb 5, 2026
f057d8f
safer no change detect
ElectricalBoy Feb 5, 2026
22027dd
sort paths before looping
ElectricalBoy Feb 5, 2026
6eac891
sort resources too
ElectricalBoy Feb 5, 2026
5dd3643
adjust workflow
ElectricalBoy Feb 5, 2026
ba980c2
extract wiki deploy function
ElectricalBoy Feb 5, 2026
6e7bc44
extract sleep duration to constant
ElectricalBoy Feb 5, 2026
4ed1707
use ruff for styling too
ElectricalBoy Feb 5, 2026
200b156
style
ElectricalBoy Feb 5, 2026
baf8d81
oops
ElectricalBoy Feb 5, 2026
2267ac1
adjust workflow trigger
ElectricalBoy Feb 5, 2026
30f2b1a
code conciseness
ElectricalBoy Feb 5, 2026
b075a7f
fix login in dev remove script
ElectricalBoy Feb 6, 2026
984ede5
fix incorrect condition reading
ElectricalBoy Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .github/workflows/delete dev env.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,14 @@ jobs:
with:
fetch-depth: 0

- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.14'

- name: Install Python Dependency
run: pip install requests

- name: Remove Lua Dev Env Modules
env:
WIKI_USER: ${{ secrets.LP_BOTUSER }}
Expand All @@ -33,4 +41,5 @@ jobs:
LUA_DEV_ENV_NAME: "dev/${{ github.event.inputs.luadevenv }}"
INCLUDE_COMMONS: ${{ github.event.inputs.includecommons }}
INCLUDE_SUB_ENVS: ${{ github.event.inputs.includesubenvs }}
run: bash scripts/remove_dev.sh
PYTHONUNBUFFERED: 1
run: python3 ./scripts/remove_dev.py
13 changes: 12 additions & 1 deletion .github/workflows/deploy test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,16 @@ jobs:
files: |
lua/wikis/**/*.lua

- name: Setup Python
if: steps.changed-files.outputs.any_changed == 'true'
uses: actions/setup-python@v6
with:
python-version: '3.14'

- name: Install Python Dependency
if: steps.changed-files.outputs.any_changed == 'true'
run: pip install requests

- name: Personal Lua Deploy
if: steps.changed-files.outputs.any_changed == 'true'
env:
Expand All @@ -35,4 +45,5 @@ jobs:
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
LUA_DEV_ENV_NAME: "/dev/${{ github.event.inputs.luadevenv }}"
run: bash scripts/deploy.sh "${{ steps.changed-files.outputs.all_changed_files }}"
PYTHONUNBUFFERED: 1
run: python3 ./scripts/deploy.py ${{ steps.changed-files.outputs.all_changed_files }}
19 changes: 16 additions & 3 deletions .github/workflows/deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ jobs:
stylesheets/**/*.scss
javascript/**/*.js

- name: Setup Python
if: steps.res-changed-files.outputs.any_changed == 'true' || steps.lua-changed-files.outputs.any_changed == 'true'
uses: actions/setup-python@v6
with:
python-version: '3.14'

- name: Install Python Dependency
if: steps.res-changed-files.outputs.any_changed == 'true' || steps.lua-changed-files.outputs.any_changed == 'true'
run: pip install requests

- name: Resource Deploy
if: steps.res-changed-files.outputs.any_changed == 'true'
env:
Expand All @@ -37,7 +47,8 @@ jobs:
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
DEPLOY_TRIGGER: ${{ github.event_name }}
run: bash scripts/deploy_res.sh "${{ steps.res-changed-files.outputs.all_changed_files }}"
PYTHONUNBUFFERED: 1
run: python3 ./scripts/deploy_res.py ${{ steps.res-changed-files.outputs.all_changed_files }}

- name: Lua Deploy
if: steps.lua-changed-files.outputs.any_changed == 'true'
Expand All @@ -47,7 +58,8 @@ jobs:
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
DEPLOY_TRIGGER: ${{ github.event_name }}
run: bash scripts/deploy.sh "${{ steps.lua-changed-files.outputs.all_changed_files }}"
PYTHONUNBUFFERED: 1
run: python3 ./scripts/deploy.py ${{ steps.lua-changed-files.outputs.all_changed_files }}

- name: Lua Protect
if: steps.lua-changed-files.outputs.added_files_count != 0 || steps.lua-changed-files.outputs.renamed_files_count != 0
Expand All @@ -56,4 +68,5 @@ jobs:
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
run: bash scripts/protect.sh "${{ steps.lua-changed-files.outputs.added_files }} ${{ steps.lua-changed-files.outputs.renamed_files }}"
PYTHONUNBUFFERED: 1
run: python3 ./scripts/protect.py ${{ steps.lua-changed-files.outputs.added_files }} ${{ steps.lua-changed-files.outputs.renamed_files }}
11 changes: 10 additions & 1 deletion .github/workflows/protect new wiki modules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,20 @@ jobs:
steps:
- uses: actions/checkout@v6

- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.14'

- name: Install Python Dependency
run: pip install requests

- name: Lua Protect
env:
WIKI_USER: ${{ secrets.LP_BOTUSER }}
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
WIKI_TO_PROTECT: ${{ github.event.inputs.wiki }}
run: bash scripts/protect.sh
PYTHONUNBUFFERED: 1
run: python3 ./scripts/protect.py
11 changes: 10 additions & 1 deletion .github/workflows/protect new wiki templates.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,20 @@ jobs:
steps:
- uses: actions/checkout@v6

- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.14'

- name: Install Python Dependency
run: pip install requests

- name: Template Protect
env:
WIKI_USER: ${{ secrets.LP_BOTUSER }}
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
WIKI_TO_PROTECT: ${{ github.event.inputs.wiki }}
run: bash scripts/protect_templates.sh
PYTHONUNBUFFERED: 1
run: python3 ./scripts/protect_templates.py
22 changes: 22 additions & 0 deletions .github/workflows/pythoncheck.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: Python Code Style

on:
pull_request:
paths:
- '**.py'
- '.github/workflows/*.yml'
workflow_dispatch:

jobs:
python-code-style:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v6
- name: Run linter
uses: astral-sh/ruff-action@v3
with:
src: scripts/
- name: Check styling
run: ruff format --check --diff scripts/
14 changes: 12 additions & 2 deletions .github/workflows/scheduled.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,23 @@ jobs:
with:
fetch-depth: 2

- name: Setup Python
uses: actions/setup-python@v6
with:
python-version: '3.14'

- name: Install Python Dependency
run: pip install requests

- name: Resource Deploy
env:
WIKI_USER: ${{ secrets.LP_BOTUSER }}
WIKI_PASSWORD: ${{ secrets.LP_BOTPASSWORD }}
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
DEPLOY_TRIGGER: ${{ github.event_name }}
run: bash scripts/deploy_res.sh
PYTHONUNBUFFERED: 1
run: python3 ./scripts/deploy_res.py

- name: Lua Deploy
env:
Expand All @@ -31,4 +40,5 @@ jobs:
WIKI_UA_EMAIL: ${{ secrets.LP_UA_EMAIL }}
WIKI_BASE_URL: ${{ secrets.LP_BASE_URL }}
DEPLOY_TRIGGER: ${{ github.event_name }}
run: bash scripts/deploy.sh
PYTHONUNBUFFERED: 1
run: python3 ./scripts/deploy.py
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,6 @@
/package-lock.json
/luacov.report.out
/luacov.stats.out
**/__pycache__/
.env
*.ck
73 changes: 73 additions & 0 deletions scripts/deploy.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
import itertools
import os
import pathlib
import re
import sys

from typing import Iterable

import requests

from deploy_util import (
get_git_deploy_reason,
deploy_file_to_wiki,
read_cookie_jar,
read_file_from_path,
write_to_github_summary_file,
)
from login_and_get_token import get_token

HEADER_PATTERN = re.compile(
r"\A---\n" r"-- @Liquipedia\n" r"-- page=(?P<pageName>[^\n]*)\n"
)


def deploy_all_files_for_wiki(
wiki: str, file_paths: Iterable[pathlib.Path], deploy_reason: str
) -> bool:
all_modules_deployed = True
token = get_token(wiki)
with requests.Session() as session:
session.cookies = read_cookie_jar(wiki)
Comment on lines +29 to +31
Copy link
Collaborator

@mbergen mbergen Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could also move this into a dedicated context manager to avoid repetition of cookie reading, but is also a bit overkill

@contextmanager
def authenticated_session(wiki: str):
    token = get_token(wiki)
    with requests.Session() as session:
        session.headers.update(HEADERS)
        session.cookies = read_cookie_jar(wiki)
        yield session, token
Suggested change
token = get_token(wiki)
with requests.Session() as session:
session.cookies = read_cookie_jar(wiki)
with authenticated_session(wiki) as session, token:

In case we'd then also move the opening of this context manager to be in each main function, and then passed everywhere, we could also get rid of the cookie jar as the session will persist the cookies.

Copy link
Collaborator Author

@ElectricalBoy ElectricalBoy Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think it's a bit overkill

for file_path in file_paths:
print(f"::group::Checking {str(file_path)}")
file_content = read_file_from_path(file_path)
header_match = HEADER_PATTERN.match(file_content)
if not header_match:
print("...skipping - no magic comment found")
write_to_github_summary_file(f"{str(file_path)} skipped")
else:
page = header_match.groupdict()["pageName"] + (
os.getenv("LUA_DEV_ENV_NAME") or ""
)
module_deployed, _ = deploy_file_to_wiki(
session, file_path, file_content, wiki, page, token, deploy_reason
)
all_modules_deployed &= module_deployed
print("::endgroup::")
return all_modules_deployed


def main():
all_modules_deployed = True
lua_files: Iterable[pathlib.Path]
git_deploy_reason: str
if len(sys.argv[1:]) == 0:
lua_files = pathlib.Path("./lua/wikis/").rglob("*.lua")
git_deploy_reason = "Automated Weekly Re-Sync"
else:
lua_files = [pathlib.Path(arg) for arg in sys.argv[1:]]
git_deploy_reason = get_git_deploy_reason()

for wiki, files in itertools.groupby(sorted(lua_files), lambda path: path.parts[2]):
all_modules_deployed &= deploy_all_files_for_wiki(
wiki, list(files), git_deploy_reason
)

if not all_modules_deployed:
print("::warning::Some modules were not deployed!")
sys.exit(1)


if __name__ == "__main__":
main()
92 changes: 0 additions & 92 deletions scripts/deploy.sh
View file
Open in desktop

This file was deleted.

Loading
Loading