Improve document for IDE support and simplify doc build#654
Improve document for IDE support and simplify doc build#654yaooqinn merged 12 commits intoapache:asf-sitefrom
Conversation
README.md
Outdated
| --mount type=bind,source="/path/to/spark-website",target="/spark-website" \ | ||
| -w /spark-website \ | ||
| docs-builder:latest \ | ||
| /bin/bash -c "sh .github/run-in-container.sh" |
There was a problem hiding this comment.
.github? Maybe it is better to be in dev?
developer-tools.md
Outdated
| <h4>IntelliJ</h4> | ||
|
|
||
| While many of the Spark developers use SBT or Maven on the command line, the most common IDE we | ||
| use is IntelliJ IDEA. You can get the community edition for free (Apache committers can get |
There was a problem hiding this comment.
there is no IDEA community edition now ...
https://blog.jetbrains.com/idea/2025/07/intellij-idea-unified-distribution-plan/
developer-tools.md
Outdated
| choose "Detect automatically" for `Generated source folders`, and choose "generate sources" for | ||
| `Phase to be used for folders update`. | ||
| - Go to `Settings -> Build, Execution, Deployment -> Compiler -> Scala Compiler -> Scala Compiler Server`, | ||
| pick a large enough number for `Maximum heap size, MB`, such as "16000". |
There was a problem hiding this comment.
the default is 2g, which does not suffice for Spark, but 16g seems to be too aggressive, this might scare off many developers, I tested 4g also works
developer-tools.md
Outdated
|
|
||
| Make sure you have a clean start before setting up the IDE: A clean git clone of the Spark repo, install the latest | ||
| version of the IDE. If something goes wrong, clear the build outputs by `./build/sbt clean` and `./build/mvn clean`, | ||
| clear the m2 cache by `rm -rf ~/.m2/repository/*`, remove the IDE folder such as `.idea`, re-import the project into |
There was a problem hiding this comment.
Looks it's IDEA-specific, can we move this to IntelliJ section?
| # | ||
|
|
||
| # 1.Set env variable. | ||
| export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-arm64 |
There was a problem hiding this comment.
@cloud-fan this assumes the developer uses arm machine ...
BTW, is Java required in website building?
There was a problem hiding this comment.
This script runs in the docker, so the local env does not matter.
There was a problem hiding this comment.
this depends on the cpu platform, for intel cpu, the suffix is -amd64, not -arm64
| Once the image is built, navigate to the `spark-website` root directory, run the script which processes | ||
| the Markdown files in the Docker container. | ||
| ``` | ||
| SPARK_WEBSITE_PATH="/path/to/spark-website" sh .dev/build-docs.sh |
There was a problem hiding this comment.
I think env var SPARK_WEBSITE_PATH is not used. and grant x permission to .dev/*.sh then we don't need use sh foo.sh to run the script
5 participants
This PR does two things:
developer-tools.mdto give more details about IDE setup, w.r.t. my success tries