docker.io/apache/spark:3.4.0-python3 linux/amd64

docker.io/apache/spark:3.4.0-python3 - 国内下载镜像源 浏览次数:73 安全受验证的发布者-apache

Apache Spark 镜像

该镜像包含 Apache Spark 的预构建环境,可用于运行 Spark 作业和应用程序。它提供了 Spark 的核心组件,包括:

  • Spark Core
  • Spark SQL
  • Spark Streaming
  • Spark MLlib
  • Spark GraphX

使用该镜像可以快速开始使用 Spark,无需手动安装和配置。它是一个易于使用且可靠的解决方案,适用于各种 Spark 应用场景。

源镜像 docker.io/apache/spark:3.4.0-python3
国内镜像 swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3
镜像ID sha256:31ca9f801ec3d73d5c8e4cc6048557723fd72c5492f79e409f387f4d5272506a
镜像TAG 3.4.0-python3
大小 963.13MB
镜像源 docker.io
项目信息 Docker-Hub主页 🚀项目TAG 🚀
CMD
启动入口 /opt/entrypoint.sh
工作目录 /opt/spark/work-dir
OS/平台 linux/amd64
浏览量 73 次
贡献者 ha*********i@zmn.cn
镜像创建 2023-04-23T02:52:24.920212021Z
同步时间 2025-02-26 17:21
更新时间 2025-03-30 18:56
环境变量
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JAVA_HOME=/opt/java/openjdk LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8 JAVA_VERSION=jdk-11.0.18+10 SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz.asc GPG_KEY=CC68B3D16FE33A766705160BA7E57908C7A4E1B1 SPARK_HOME=/opt/spark
镜像标签
ubuntu: org.opencontainers.image.ref.name 20.04: org.opencontainers.image.version
镜像安全扫描 查看Trivy扫描报告

系统OS: ubuntu 20.04 扫描引擎: Trivy 扫描时间: 2025-02-26 17:22

低危漏洞:255 中危漏洞:2125 高危漏洞:113 严重漏洞:5

Docker拉取命令 无权限下载?点我修复

docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3
docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3  docker.io/apache/spark:3.4.0-python3

Containerd拉取命令

ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3
ctr images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3  docker.io/apache/spark:3.4.0-python3

Shell快速替换命令

sed -i 's#apache/spark:3.4.0-python3#swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3#' deployment.yaml

Ansible快速分发-Docker

#ansible k8s -m shell -a 'docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3 && docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3  docker.io/apache/spark:3.4.0-python3'

Ansible快速分发-Containerd

#ansible k8s -m shell -a 'ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3 && ctr images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3  docker.io/apache/spark:3.4.0-python3'

镜像构建历史


# 2023-04-23 10:52:24  0.00B 配置容器启动时运行的命令
ENTRYPOINT ["/opt/entrypoint.sh"]
                        
# 2023-04-23 10:52:24  3.84KB 执行命令并创建新的镜像层
RUN |1 spark_uid=185 /bin/sh -c chmod a+x /opt/entrypoint.sh # buildkit
                        
# 2023-04-23 10:52:24  0.00B 执行命令并创建新的镜像层
RUN |1 spark_uid=185 /bin/sh -c chmod a+x /opt/decom.sh # buildkit
                        
# 2023-04-23 10:52:24  0.00B 执行命令并创建新的镜像层
RUN |1 spark_uid=185 /bin/sh -c chmod g+w /opt/spark/work-dir # buildkit
                        
# 2023-04-23 10:52:24  0.00B 设置工作目录为/opt/spark/work-dir
WORKDIR /opt/spark/work-dir
                        
# 2023-04-23 10:52:24  0.00B 设置环境变量 SPARK_HOME
ENV SPARK_HOME=/opt/spark
                        
# 2023-04-23 10:52:24  3.84KB 复制新文件或目录到容器中
COPY entrypoint.sh /opt/ # buildkit
                        
# 2023-04-23 10:52:24  347.24MB 执行命令并创建新的镜像层
RUN |1 spark_uid=185 /bin/sh -c set -ex;     export SPARK_TMP="$(mktemp -d)";     cd $SPARK_TMP;     wget -nv -O spark.tgz "$SPARK_TGZ_URL";     wget -nv -O spark.tgz.asc "$SPARK_TGZ_ASC_URL";     export GNUPGHOME="$(mktemp -d)";     gpg --keyserver hkps://keys.openpgp.org --recv-key "$GPG_KEY" ||     gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys "$GPG_KEY";     gpg --batch --verify spark.tgz.asc spark.tgz;     gpgconf --kill all;     rm -rf "$GNUPGHOME" spark.tgz.asc;         tar -xf spark.tgz --strip-components=1;     chown -R spark:spark .;     mv jars /opt/spark/;     mv bin /opt/spark/;     mv sbin /opt/spark/;     mv kubernetes/dockerfiles/spark/decom.sh /opt/;     mv examples /opt/spark/;     mv kubernetes/tests /opt/spark/;     mv data /opt/spark/;     mv python/pyspark /opt/spark/python/pyspark/;     mv python/lib /opt/spark/python/lib/;     cd ..;     rm -rf "$SPARK_TMP"; # buildkit
                        
# 2023-04-23 10:48:07  0.00B 设置环境变量 SPARK_TGZ_URL SPARK_TGZ_ASC_URL GPG_KEY
ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz.asc GPG_KEY=CC68B3D16FE33A766705160BA7E57908C7A4E1B1
                        
# 2023-04-23 10:48:07  359.59MB 执行命令并创建新的镜像层
RUN |1 spark_uid=185 /bin/sh -c set -ex &&     apt-get update &&     ln -s /lib /lib64 &&     apt install -y gnupg2 wget bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools gosu &&     apt install -y python3 python3-pip &&     mkdir -p /opt/spark &&     mkdir /opt/spark/python &&     mkdir -p /opt/spark/examples &&     mkdir -p /opt/spark/work-dir &&     touch /opt/spark/RELEASE &&     chown -R spark:spark /opt/spark &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd &&     rm -rf /var/cache/apt/* &&     rm -rf /var/lib/apt/lists/* # buildkit
                        
# 2023-04-23 10:47:37  64.84KB 执行命令并创建新的镜像层
RUN |1 spark_uid=185 /bin/sh -c groupadd --system --gid=${spark_uid} spark &&     useradd --system --uid=${spark_uid} --gid=spark spark # buildkit
                        
# 2023-04-23 10:47:37  0.00B 定义构建参数
ARG spark_uid=185
                        
# 2023-04-18 08:44:20  0.00B 
/bin/sh -c echo Verifying install ...     && fileEncoding="$(echo 'System.out.println(System.getProperty("file.encoding"))' | jshell -s -)"; [ "$fileEncoding" = 'UTF-8' ]; rm -rf ~/.java     && echo java --version && java --version     && echo Complete.
                        
# 2023-04-18 08:44:19  139.75MB 
/bin/sh -c set -eux;     ARCH="$(dpkg --print-architecture)";     case "${ARCH}" in        aarch64|arm64)          ESUM='89bc6e93d48a37a5eff7ec5afa515c60eb3369d106d1736f5e845b3dcf8fb72c';          BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jre_aarch64_linux_hotspot_11.0.18_10.tar.gz';          ;;        armhf|arm)          ESUM='949482ac232e756f342de6a8592d56b58803e10d3956abff14c4958e711a0b7c';          BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jre_arm_linux_hotspot_11.0.18_10.tar.gz';          ;;        ppc64el|powerpc:common64)          ESUM='df32e80d5b6db60a1ed9ed04eaf267eaf17835ed2ae2c1708d8d94328c03a0a5';          BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jre_ppc64le_linux_hotspot_11.0.18_10.tar.gz';          ;;        s390x|s390:64-bit)          ESUM='c07df5081f78021bed0d169622e470ebd4a4525a45f84192a52580ddb912959b';          BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jre_s390x_linux_hotspot_11.0.18_10.tar.gz';          ;;        amd64|i386:x86-64)          ESUM='0e7b196ef8603ac3d38caaf7768b7b0a3c613d60e15a6511bcfb2c894b609e99';          BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.18%2B10/OpenJDK11U-jre_x64_linux_hotspot_11.0.18_10.tar.gz';          ;;        *)          echo "Unsupported arch: ${ARCH}";          exit 1;          ;;     esac; 	  wget -O /tmp/openjdk.tar.gz ${BINARY_URL}; 	  echo "${ESUM} */tmp/openjdk.tar.gz" | sha256sum -c -; 	  mkdir -p "$JAVA_HOME"; 	  tar --extract 	      --file /tmp/openjdk.tar.gz 	      --directory "$JAVA_HOME" 	      --strip-components 1 	      --no-same-owner 	  ;     rm -f /tmp/openjdk.tar.gz ${JAVA_HOME}/src.zip;     find "$JAVA_HOME/lib" -name '*.so' -exec dirname '{}' ';' | sort -u > /etc/ld.so.conf.d/docker-openjdk.conf;     ldconfig;     java -Xshare:dump;
                        
# 2023-04-18 08:43:54  0.00B 
/bin/sh -c #(nop)  ENV JAVA_VERSION=jdk-11.0.18+10
                        
# 2023-04-18 08:43:22  43.68MB 
/bin/sh -c apt-get update     && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata curl wget ca-certificates fontconfig locales     && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen     && locale-gen en_US.UTF-8     && rm -rf /var/lib/apt/lists/*
                        
# 2023-04-18 08:42:52  0.00B 
/bin/sh -c #(nop)  ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
                        
# 2023-04-18 08:42:52  0.00B 
/bin/sh -c #(nop)  ENV PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
                        
# 2023-04-18 08:42:52  0.00B 
/bin/sh -c #(nop)  ENV JAVA_HOME=/opt/java/openjdk
                        
# 2023-04-13 21:05:15  0.00B 
/bin/sh -c #(nop)  CMD ["/bin/bash"]
                        
# 2023-04-13 21:05:15  72.79MB 
/bin/sh -c #(nop) ADD file:d05d1c0936b046937bd5755876db2f8da3ed8ccbcf464bb56c312fbc7ed78589 in / 
                        
# 2023-04-13 21:05:13  0.00B 
/bin/sh -c #(nop)  LABEL org.opencontainers.image.version=20.04
                        
# 2023-04-13 21:05:13  0.00B 
/bin/sh -c #(nop)  LABEL org.opencontainers.image.ref.name=ubuntu
                        
# 2023-04-13 21:05:13  0.00B 
/bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH
                        
# 2023-04-13 21:05:13  0.00B 
/bin/sh -c #(nop)  ARG RELEASE
                        
                    

镜像信息

{
    "Id": "sha256:31ca9f801ec3d73d5c8e4cc6048557723fd72c5492f79e409f387f4d5272506a",
    "RepoTags": [
        "apache/spark:3.4.0-python3",
        "swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark:3.4.0-python3"
    ],
    "RepoDigests": [
        "apache/spark@sha256:55e67077c859e14b6ea5600dc9dfd67219054c6feed0689558ff07bb4b2c9579",
        "swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/spark@sha256:e154fc458f8f59facfe41d4836ca69f4dcb0403f72889f32acf7a68f5e14ec34"
    ],
    "Parent": "",
    "Comment": "buildkit.dockerfile.v0",
    "Created": "2023-04-23T02:52:24.920212021Z",
    "Container": "",
    "ContainerConfig": null,
    "DockerVersion": "",
    "Author": "",
    "Config": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "JAVA_HOME=/opt/java/openjdk",
            "LANG=en_US.UTF-8",
            "LANGUAGE=en_US:en",
            "LC_ALL=en_US.UTF-8",
            "JAVA_VERSION=jdk-11.0.18+10",
            "SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz",
            "SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.4.0/spark-3.4.0-bin-hadoop3.tgz.asc",
            "GPG_KEY=CC68B3D16FE33A766705160BA7E57908C7A4E1B1",
            "SPARK_HOME=/opt/spark"
        ],
        "Cmd": null,
        "Image": "",
        "Volumes": null,
        "WorkingDir": "/opt/spark/work-dir",
        "Entrypoint": [
            "/opt/entrypoint.sh"
        ],
        "OnBuild": null,
        "Labels": {
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "20.04"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Size": 963129009,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/9ecda8f548379a4a659571c46cc72fb2e9827c215abd7e652529479d03087a9c/diff:/var/lib/docker/overlay2/38cc796ffb62c4b1a6ce7c17b3bb310d9d27cb6ad7e4db1a25d29a35791c9629/diff:/var/lib/docker/overlay2/d6220289a83b16a7ca854906385fef34302579ff63616966be3d09086298b761/diff:/var/lib/docker/overlay2/29833d274051999f4a254637ff767804be6af60e1ad54fffbe4428956e48f2c0/diff:/var/lib/docker/overlay2/2c03b2f4a28b749dc8757d7e1acb0b8521d37e4ca03b3040f2eb79ea7043d135/diff:/var/lib/docker/overlay2/a80fb101e0623c5a54ed21ff6d78b00e29935c0acc8b71edfe9a533489c28051/diff:/var/lib/docker/overlay2/6d5fb62174cc3b2e8d19f2a0aeed155eaf4e0058223dd07d21e10a0da9b29eb1/diff:/var/lib/docker/overlay2/9f6b230bbd8137737e591d3373db9a021ab5880a877250eed71668d08f3c2e84/diff:/var/lib/docker/overlay2/9a0898b9b552ef7e5e73e27166b93097b09bc28830c30df3bb42612fd9a126c1/diff:/var/lib/docker/overlay2/1c2d2f16e5b2e575099e36c31f6b0700f5baae337690ecc923c2716b56adee05/diff:/var/lib/docker/overlay2/475045c1e801a70019813b58c6ee75700ff42c863bcb5f170ec11a3765430350/diff",
            "MergedDir": "/var/lib/docker/overlay2/971b33d7773b87894200560e936bcb1592edda3a5488721b5ef18f6a45362a96/merged",
            "UpperDir": "/var/lib/docker/overlay2/971b33d7773b87894200560e936bcb1592edda3a5488721b5ef18f6a45362a96/diff",
            "WorkDir": "/var/lib/docker/overlay2/971b33d7773b87894200560e936bcb1592edda3a5488721b5ef18f6a45362a96/work"
        },
        "Name": "overlay2"
    },
    "RootFS": {
        "Type": "layers",
        "Layers": [
            "sha256:6f37ca73c74f2cef0ddefd960260f2033c16c84583c5507a4f37b1cf7631dc20",
            "sha256:78dd8d8d8e00be6af681d7af9f7adaccea2441e2b6c8212b21521eef6d2c5e93",
            "sha256:879dcc8a370d97cf77de6abcf511633815ea82722377f7e8860dd3d70c664109",
            "sha256:12248c2c2572c8f1f4bdd3605cddd54c9098b62239de13ff2d2d21b4aceebdee",
            "sha256:c423a729b73880559ab5a9dd4b406da3f6df5051a744bcfe67332b1892c72b48",
            "sha256:894fafa63dd4bbd8da5b4f072a6887913bd16b04a4d5fcafb70f1899d33b3908",
            "sha256:1be33f7385a2bb4cd31694cb9f0f6972b03b6fdfe8e2c87b0eb7bf8a369dd3ff",
            "sha256:028167662abef659ca625d0532ac5da8efce63bb48194df4eb4a9170a40f6175",
            "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
            "sha256:def2e738c6a0cf50ed8283d105fee293ad1897bd7db9730475736f93dddf0784",
            "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
            "sha256:c64a3fc48e85458e36779b5a74fe88f54b1c32837f1c3c70790a985adea4d66e"
        ]
    },
    "Metadata": {
        "LastTagTime": "2025-02-26T17:20:08.137081731+08:00"
    }
}

更多版本

docker.io/apache/spark:3.4.4

linux/amd64 docker.io974.48MB2024-10-29 01:26
198

docker.io/apache/spark:v3.2.3

linux/amd64 docker.io612.37MB2024-11-11 16:11
122

docker.io/apache/spark:3.5.3

linux/amd64 docker.io984.74MB2024-11-23 19:03
90

docker.io/apache/spark:3.5.3-java17

linux/amd64 docker.io1.15GB2024-11-24 00:41
99

docker.io/apache/spark:3.5.3-scala2.12-java11-r-ubuntu

linux/amd64 docker.io1.32GB2024-11-24 01:13
90

docker.io/apache/spark:3.5.3-scala2.12-java11-ubuntu

linux/amd64 docker.io681.32MB2024-11-24 01:31
108

docker.io/apache/spark:3.5.3-scala2.12-java17-ubuntu

linux/amd64 docker.io828.98MB2024-11-24 01:33
87

docker.io/apache/spark:3.3.3

linux/amd64 docker.io939.31MB2024-12-03 11:56
116

docker.io/apache/spark-py:v3.1.3

linux/amd64 docker.io886.30MB2024-12-11 19:44
94

docker.io/apache/spark:3.4.0-python3

linux/amd64 docker.io963.13MB2025-02-26 17:21
72