docker.io/spark:3.5.3 linux/amd64

docker.io/spark:3.5.3 - 国内下载镜像源 浏览次数:46
Apache Spark 是一个快速、通用的、分布式内存处理引擎。

Spark Docker 镜像

这是 Apache Spark 的 Docker 镜像,提供了一个完整的 Spark 集群环境。

  • 基于 OpenJDK 11 和 Java 8
  • 包含 Scala 2.12 和 Python 3.x 运行时
  • 支持 HDFS、HBase 和 Cassandra 等存储系统
  • 可以通过 Docker Compose 或者 Kubernetes 部署
源镜像 docker.io/spark:3.5.3
国内镜像 swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3
镜像ID sha256:5b54ea65484b7a183382c67ecf04ea512e3a1dba2809ac0e1e0893295816e187
镜像TAG 3.5.3
大小 982.46MB
镜像源 docker.io
项目信息 Docker-Hub主页 🚀项目TAG 🚀
CMD
启动入口 /opt/entrypoint.sh
工作目录 /opt/spark/work-dir
OS/平台 linux/amd64
浏览量 46 次
贡献者
镜像创建 2024-10-10T06:58:10Z
同步时间 2024-10-29 01:25
更新时间 2024-11-22 11:58
环境变量
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JAVA_HOME=/opt/java/openjdk LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8 JAVA_VERSION=jdk-11.0.25+9 SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz.asc GPG_KEY=0A2D660358B6F6F8071FD16F6606986CF5A8447C SPARK_HOME=/opt/spark
镜像标签
ubuntu: org.opencontainers.image.ref.name 20.04: org.opencontainers.image.version

Docker拉取命令 无权限下载?点我修复

docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3
docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3  docker.io/spark:3.5.3

Containerd拉取命令

ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3
ctr images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3  docker.io/spark:3.5.3

Shell快速替换命令

sed -i 's#spark:3.5.3#swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3#' deployment.yaml

Ansible快速分发-Docker

#ansible k8s -m shell -a 'docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3 && docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3  docker.io/spark:3.5.3'

Ansible快速分发-Containerd

#ansible k8s -m shell -a 'ctr images pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3 && ctr images tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3  docker.io/spark:3.5.3'

镜像历史

大小 创建时间 层信息
0.00B 2024-10-10 14:58:10 USER spark
303.03MB 2024-10-10 14:58:10 RUN /bin/sh -c set -ex; apt-get update; apt-get install -y python3 python3-pip; rm -rf /var/lib/apt/lists/* # buildkit
0.00B 2024-10-10 14:58:10 USER root
0.00B 2024-10-10 14:58:10 ENTRYPOINT ["/opt/entrypoint.sh"]
0.00B 2024-10-10 14:58:10 USER spark
0.00B 2024-10-10 14:58:10 WORKDIR /opt/spark/work-dir
0.00B 2024-10-10 14:58:10 ENV SPARK_HOME=/opt/spark
4.74KB 2024-10-10 14:58:10 COPY entrypoint.sh /opt/ # buildkit
361.77MB 2024-10-10 14:58:10 RUN |1 spark_uid=185 /bin/sh -c set -ex; export SPARK_TMP="$(mktemp -d)"; cd $SPARK_TMP; wget -nv -O spark.tgz "$SPARK_TGZ_URL"; wget -nv -O spark.tgz.asc "$SPARK_TGZ_ASC_URL"; export GNUPGHOME="$(mktemp -d)"; gpg --batch --keyserver hkps://keys.openpgp.org --recv-key "$GPG_KEY" || gpg --batch --keyserver hkps://keyserver.ubuntu.com --recv-keys "$GPG_KEY"; gpg --batch --verify spark.tgz.asc spark.tgz; gpgconf --kill all; rm -rf "$GNUPGHOME" spark.tgz.asc; tar -xf spark.tgz --strip-components=1; chown -R spark:spark .; mv jars /opt/spark/; mv RELEASE /opt/spark/; mv bin /opt/spark/; mv sbin /opt/spark/; mv kubernetes/dockerfiles/spark/decom.sh /opt/; mv examples /opt/spark/; ln -s "$(basename /opt/spark/examples/jars/spark-examples_*.jar)" /opt/spark/examples/jars/spark-examples.jar; mv kubernetes/tests /opt/spark/; mv data /opt/spark/; mv python/pyspark /opt/spark/python/pyspark/; mv python/lib /opt/spark/python/lib/; mv R /opt/spark/; chmod a+x /opt/decom.sh; cd ..; rm -rf "$SPARK_TMP"; # buildkit
0.00B 2024-10-10 14:58:10 ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz.asc GPG_KEY=0A2D660358B6F6F8071FD16F6606986CF5A8447C
50.89MB 2024-10-10 14:58:10 RUN |1 spark_uid=185 /bin/sh -c set -ex; apt-get update; apt-get install -y gnupg2 wget bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools gosu libnss-wrapper; mkdir -p /opt/spark; mkdir /opt/spark/python; mkdir -p /opt/spark/examples; mkdir -p /opt/spark/work-dir; chmod g+w /opt/spark/work-dir; touch /opt/spark/RELEASE; chown -R spark:spark /opt/spark; echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su; rm -rf /var/lib/apt/lists/* # buildkit
64.84KB 2024-10-10 14:58:10 RUN |1 spark_uid=185 /bin/sh -c groupadd --system --gid=${spark_uid} spark && useradd --system --uid=${spark_uid} --gid=spark spark # buildkit
0.00B 2024-10-10 14:58:10 ARG spark_uid=185
0.00B 2024-10-10 14:58:10 ENTRYPOINT ["/__cacert_entrypoint.sh"]
5.31KB 2024-10-10 14:58:10 COPY --chmod=755 entrypoint.sh /__cacert_entrypoint.sh # buildkit
0.00B 2024-10-10 14:58:10 RUN /bin/sh -c set -eux; echo "Verifying install ..."; echo "java --version"; java --version; echo "Complete." # buildkit
141.01MB 2024-10-10 14:58:10 RUN /bin/sh -c set -eux; ARCH="$(dpkg --print-architecture)"; case "${ARCH}" in amd64) ESUM='84cd7101f39172a4db085fb52940595bb14dad6bc3afb5bf82ee497eceaf86d3'; BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.25%2B9/OpenJDK11U-jre_x64_linux_hotspot_11.0.25_9.tar.gz'; ;; arm64) ESUM='e37ba6636e31f3c9191ac7e3fd0ab7fb354a2f3b320d68bfb95efd1e053134c9'; BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.25%2B9/OpenJDK11U-jre_aarch64_linux_hotspot_11.0.25_9.tar.gz'; ;; armhf) ESUM='6b7b1297da762cf2b1eb4834073e6a45cda82a359efb17a89eba3fc6b59b4d26'; BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.25%2B9/OpenJDK11U-jre_arm_linux_hotspot_11.0.25_9.tar.gz'; ;; ppc64el) ESUM='7e7edaf34c29c304514d60f40f6c9ce58eb3e32b0dec20bb6ccd1cfbb4456698'; BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.25%2B9/OpenJDK11U-jre_ppc64le_linux_hotspot_11.0.25_9.tar.gz'; ;; s390x) ESUM='4ec884fe3874e258ae2253d535d3d92d6c313542fd973e8963c2eb87d68fb273'; BINARY_URL='https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.25%2B9/OpenJDK11U-jre_s390x_linux_hotspot_11.0.25_9.tar.gz'; ;; *) echo "Unsupported arch: ${ARCH}"; exit 1; ;; esac; wget --progress=dot:giga -O /tmp/openjdk.tar.gz ${BINARY_URL}; wget --progress=dot:giga -O /tmp/openjdk.tar.gz.sig ${BINARY_URL}.sig; export GNUPGHOME="$(mktemp -d)"; gpg --batch --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B; gpg --batch --verify /tmp/openjdk.tar.gz.sig /tmp/openjdk.tar.gz; rm -r "${GNUPGHOME}" /tmp/openjdk.tar.gz.sig; echo "${ESUM} */tmp/openjdk.tar.gz" | sha256sum -c -; mkdir -p "$JAVA_HOME"; tar --extract --file /tmp/openjdk.tar.gz --directory "$JAVA_HOME" --strip-components 1 --no-same-owner ; rm -f /tmp/openjdk.tar.gz ${JAVA_HOME}/lib/src.zip; find "$JAVA_HOME/lib" -name '*.so' -exec dirname '{}' ';' | sort -u > /etc/ld.so.conf.d/docker-openjdk.conf; ldconfig; java -Xshare:dump; # buildkit
0.00B 2024-10-10 14:58:10 ENV JAVA_VERSION=jdk-11.0.25+9
52.87MB 2024-10-10 14:58:10 RUN /bin/sh -c set -eux; apt-get update; DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends curl wget gnupg fontconfig ca-certificates p11-kit tzdata locales ; echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen; locale-gen en_US.UTF-8; rm -rf /var/lib/apt/lists/* # buildkit
0.00B 2024-10-10 14:58:10 ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
0.00B 2024-10-10 14:58:10 ENV PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
0.00B 2024-10-10 14:58:10 ENV JAVA_HOME=/opt/java/openjdk
0.00B 2024-10-10 14:58:10 /bin/sh -c #(nop) CMD ["/bin/bash"]
72.81MB 2024-10-10 14:58:10 /bin/sh -c #(nop) ADD file:7486147a645d8835a5181c79f00a3606c6b714c83bcbfcd8862221eb14690f9e in /
0.00B 2024-10-10 14:58:10 /bin/sh -c #(nop) LABEL org.opencontainers.image.version=20.04
0.00B 2024-10-10 14:58:10 /bin/sh -c #(nop) LABEL org.opencontainers.image.ref.name=ubuntu
0.00B 2024-10-10 14:58:10 /bin/sh -c #(nop) ARG LAUNCHPAD_BUILD_ARCH
0.00B 2024-10-10 14:58:10 /bin/sh -c #(nop) ARG RELEASE

镜像信息

{
    "Id": "sha256:5b54ea65484b7a183382c67ecf04ea512e3a1dba2809ac0e1e0893295816e187",
    "RepoTags": [
        "spark:3.5.3",
        "swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark:3.5.3"
    ],
    "RepoDigests": [
        "spark@sha256:77f191f91b11a1a4f414655cd9e88cc8199b64cf20cf67057940d318b4f22190",
        "swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/spark@sha256:e44f89eb0a58694b66bacbdfb1b4f67c201d0cbc20fbf7072d0061eb22461727"
    ],
    "Parent": "",
    "Comment": "buildkit.dockerfile.v0",
    "Created": "2024-10-10T06:58:10Z",
    "Container": "",
    "ContainerConfig": null,
    "DockerVersion": "",
    "Author": "",
    "Config": {
        "Hostname": "",
        "Domainname": "",
        "User": "spark",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "JAVA_HOME=/opt/java/openjdk",
            "LANG=en_US.UTF-8",
            "LANGUAGE=en_US:en",
            "LC_ALL=en_US.UTF-8",
            "JAVA_VERSION=jdk-11.0.25+9",
            "SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz",
            "SPARK_TGZ_ASC_URL=https://archive.apache.org/dist/spark/spark-3.5.3/spark-3.5.3-bin-hadoop3.tgz.asc",
            "GPG_KEY=0A2D660358B6F6F8071FD16F6606986CF5A8447C",
            "SPARK_HOME=/opt/spark"
        ],
        "Cmd": null,
        "Image": "",
        "Volumes": null,
        "WorkingDir": "/opt/spark/work-dir",
        "Entrypoint": [
            "/opt/entrypoint.sh"
        ],
        "OnBuild": null,
        "Labels": {
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "20.04"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Size": 982457417,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/3e73a20e74a265b659937e2e616c4a3e3f8774239a1e5b43ef3317abafda1b12/diff:/var/lib/docker/overlay2/49fb6f446d904870148cc98c4c20a40acee9aeb20a1585efded85b64bc7f8c66/diff:/var/lib/docker/overlay2/a3306523ac33bafba634154eeda623cd2c5259b4861c4e48035746149a8ba5a2/diff:/var/lib/docker/overlay2/70b5a85f64f7f3020c0186dcd8fba13c487a57060bb960731132812a718191c8/diff:/var/lib/docker/overlay2/6762153de47840755ad81600cd8c065bdbbd3a89568e2ff02aac40986206d839/diff:/var/lib/docker/overlay2/cd023733e09d9aa5aa433a00b06c186fce77bf78ebb70e1185dcd2dd487d0b32/diff:/var/lib/docker/overlay2/51fc2c99fad7d2d368a37408b2ece92cac7ec15771834feb6d471fb823d7215b/diff:/var/lib/docker/overlay2/c89fd6a7cda448f92910ce9eef4e887b5446231484dfa3d79d1f53787211b90a/diff:/var/lib/docker/overlay2/6b12eaed56e61480ee49879a72f3fefc60095458af1352e2b7eae1450ef8c8fd/diff:/var/lib/docker/overlay2/16a86e622827aceef4f9b6bf6e57b09dea22f712e99b64eb203c4ed3c43d14a5/diff",
            "MergedDir": "/var/lib/docker/overlay2/fcc7bdccc84310e7044d8caf14bff8ad97f441fe224085ae8d3cf9dd1e3a9fd3/merged",
            "UpperDir": "/var/lib/docker/overlay2/fcc7bdccc84310e7044d8caf14bff8ad97f441fe224085ae8d3cf9dd1e3a9fd3/diff",
            "WorkDir": "/var/lib/docker/overlay2/fcc7bdccc84310e7044d8caf14bff8ad97f441fe224085ae8d3cf9dd1e3a9fd3/work"
        },
        "Name": "overlay2"
    },
    "RootFS": {
        "Type": "layers",
        "Layers": [
            "sha256:fffe76c64ef2dee2d80a8bb3ad13d65d596d04a45510b1956a976a69215dae92",
            "sha256:1a6e7e1fc78a1c01ecbc8ce3693faf9816b1380df134458cac72f340d43d1fa8",
            "sha256:60b89701d33eefe6014db3d21b24c8bdb8c0054848cfea3ce44d07e1a22ecf7b",
            "sha256:2a574e81bd82d2c5c8d8a482e12bb8fa85d206da01f4da10ff5b39c0059d1629",
            "sha256:06f79ba26fd42b919085db9e9f953097c2779853058475d8bcbd08e26ddc07a4",
            "sha256:5785eadf7f8e11009e0d1edf0d3fb0347fa29f5a0d5881f85dfdcf2e2cca1da7",
            "sha256:2628edda9834fb4146690501cec2f7a7c4b2709bcddff1dcf0dadc674939caf8",
            "sha256:331e29774e06a2cc1ce82ca53448bb311f0aab48ce19dbbc6a01b2d2372a6ffe",
            "sha256:ab601ad9ad6709daa012040c40e5b314385e76c1df5381a6aed9a3bab8659900",
            "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
            "sha256:e9bb2ff0a7e7ad52f4e6866f04a5da6561a68e4ddd0a8bcab9d4af26e8054df3"
        ]
    },
    "Metadata": {
        "LastTagTime": "2024-10-29T01:23:46.970052599+08:00"
    }
}

更多版本

docker.io/spark:latest

linux/amd64 docker.io984.32MB2024-09-25 17:19
133

docker.io/spark:3.5.3

linux/amd64 docker.io982.46MB2024-10-29 01:25
45