ARTICLE DETAIL

资讯详情

深耕网站建设、视觉设计与SEO优化的一线实战洞察。

Huggingface镜像网站下载语言模型方法

Huggingface镜像网站下载语言模型方法

通常通过镜像网站下载https://hf-mirror.com/。
在链接页面有介绍方法,对于不大的模型可以直接下载。这里介绍比较常用且方便的下载方法。

使用huggingface 官方提供的 huggingface-cli 命令行工具

安装(huggingface_hub、hf_transfer安装可以使用-i命令从镜像网站下载)

pip install -U huggingface_hub hf_transfer

基本命令(每次打开远程链接都要输入)

export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_ENDPOINT=https://hf-mirror.com

下载模型(下载NousResearch/Llama-2-13b-chat-hf,放在当前目录文件夹Llama-2-13b-chat-hf下)

huggingface-cli download --resume-download NousResearch/Llama-2-13b-chat-hf --local-dir Llama-2-13b-chat-hf --local-dir-use-symlinks False

携带参数下载

huggingface-cli download --token hf_*** --resume-download meta-llama/Llama-2-7b-hf --local-dir Llama-2-7b-hf --local-dir-use-symlinks False

对于下载中断的问题,由于huggingface-cli本身能断点重传,但下载界面关闭后只能重新下载,因而采用选择性下载

huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "model-00013-of-00015.safetensors"huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "pytorch_model*.bin"

其他用法

huggingface-cli download --helpsusage: huggingface-cli <command> [<args>] download [-h] [--repo-type {model,dataset,space}][--revision REVISION] [--include [INCLUDE ...]][--exclude [EXCLUDE ...]] [--cache-dir CACHE_DIR]  [--local-dir LOCAL_DIR]   [--local-dir-use-symlinks {auto,True,False}]   [--force-download] [--resume-download][--token TOKEN]                                                         [--quiet]                                                                                                      repo_id [filenames ...]
positional arguments:                                                                                            repo_id               ID of the repo to download from (e.g. `username/repo-name`).filenames             Files to download (e.g. `config.json`, `data/metadata.jsonl`).options:-h, --help            show this help message and exit--repo-type {model,dataset,space}Type of repo to download from (e.g. `dataset`).--revision REVISION   An optional Git revision id which can be a branch name, a tag, or a commit hash.--include [INCLUDE ...]Glob patterns to match files to download.--exclude [EXCLUDE ...]Glob patterns to exclude from files to download.--cache-dir CACHE_DIRPath to the directory where to save the downloaded files.--local-dir LOCAL_DIRIf set, the downloaded file will be placed under this directory either as a symlink(default) or a regular file. Check outhttps://huggingface.co/docs/huggingface_hub/guides/download#download-files-to-local-folder for more details.--local-dir-use-symlinks {auto,True,False}To be used with `local_dir`. If set to 'auto', the cache directory will be used andthe file will be either duplicated or symlinked to the local directory depending onits size. It set to `True`, a symlink will be created, no matter the file size. Ifset to `False`, the file will either be duplicated from cache (if already exists) ordownloaded from the Hub and not cached.--force-download      If True, the files will be downloaded even if they are already cached.--resume-download     If True, resume a previously interrupted download.--token TOKEN         A User Access Token generated from https://huggingface.co/settings/tokens--quiet               If True, progress bars are disabled and only the path to the download files isprinted.
返回列表