I was trying to use the ViTT transfomer. I got the following error with code:
from pathlib import Path
import torchvision
from typing import Callable
root = Path("~/data/").expanduser()
# root = Path(".").expanduser()
train = torchvision.datasets.CIFAR100(root=root, train=True, download=True)
test = torchvision.datasets.CIFAR100(root=root, train=False, download=True)
img2tensor: Callable = torchvision.transforms.ToTensor()
from transformers import ViTFeatureExtractor
feature_extractor = ViTFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
x, y = train_ds[0]
print(f'{y=}')
print(f'{type(x)=}')
x = img2tensor(x)
x = x.unsqueeze(0) # add batch size 1
out_cls: ImageClassifierOutput = model(x)
print(f'{out_cls.logits=}')
error
Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 11, in <module>
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/__init__.py", line 30, in <module>
from . import dependency_versions_check
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/dependency_versions_check.py", line 36, in <module>
from .utils import is_tokenizers_available
ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils' (/Users/brandomiranda/opt/anaconda3/envs/meta_learning/lib/python3.9/site-packages/transformers/utils/__init__.py)
I tried upgrading everything but it still failed. Upgrade commands:
/Users/brandomiranda/opt/anaconda3/envs/meta_learning/bin/python -m
pip install --upgrade pip
pip install --upgrade pip
pip install --upgrade huggingface-hub
pip install --upgrade transformers
pip install --upgrade huggingface-hub
pip install --upgrade datasets
pip install --upgrade tokenizers
pip install pytorch-transformers
pip install --upgrade torch
pip install --upgrade torchvision
pip install --upgrade torchtext
pip install --upgrade torchaudio
# pip install --upgrade torchmeta
pip uninstall torchmeta
Why and how to fix it?
Pip list:
(meta_learning) ❯ pip list
Package Version Editable project location
------------------------------------------------- ---------- ------------------------------------------------------------------------------
absl-py 1.0.0
aiohttp 3.8.1
aiosignal 1.2.0
antlr4-python3-runtime 4.8
argcomplete 2.0.0
async-timeout 4.0.1
attrs 21.4.0
automl-meta-learning 0.1.0 /Users/brandomiranda/automl-meta-learning/automl-proj-src
bcj-cffi 0.5.1
boto 2.49.0
boto3 1.24.85
botocore 1.27.85
Bottleneck 1.3.4
Brotli 1.0.9
brotlicffi 1.0.9.2
brotlipy 0.7.0
cachetools 4.2.4
certifi 2022.9.14
cffi 1.15.1
charset-normalizer 2.0.9
cherry-rl 0.1.4
click 8.0.3
cloudpickle 2.0.0
colorama 0.4.4
configparser 5.2.0
conllu 4.4.1
crcmod 1.7
cryptography 37.0.1
cycler 0.11.0
Cython 0.29.25
dataclasses 0.6
datasets 2.5.1
dill 0.3.4
diversity-for-predictive-success-of-meta-learning 0.0.1 /Users/brandomiranda/diversity-for-predictive-success-of-meta-learning/div_src
docker-pycreds 0.4.0
editdistance 0.6.0
et-xmlfile 1.1.0
fairseq 0.10.0
fastcluster 1.2.4
fasteners 0.17.3
filelock 3.6.0
fonttools 4.28.3
frozenlist 1.2.0
fsspec 2022.7.1
gcs-oauth2-boto-plugin 3.0
gitdb 4.0.9
GitPython 3.1.24
google-apitools 0.5.32
google-auth 2.3.3
google-auth-oauthlib 0.4.6
google-reauth 0.1.1
grpcio 1.42.0
gsutil 5.6
gym 0.21.0
h5py 3.6.0
higher 0.2.1
httplib2 0.20.4
huggingface-hub 0.10.0
hydra-core 1.1.1
idna 3.3
importlib-metadata 4.11.3
jmespath 1.0.1
joblib 1.1.0
kiwisolver 1.3.2
lark-parser 0.12.0
learn2learn 0.1.7
lxml 4.8.0
Markdown 3.3.6
matplotlib 3.5.1
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
monotonic 1.6
multidict 5.2.0
multiprocess 0.70.12.2
multivolumefile 0.2.3
munkres 1.1.4
networkx 2.6.3
numexpr 2.8.1
numpy 1.21.5
oauth2client 4.1.3
oauthlib 3.1.1
omegaconf 2.1.1
openpyxl 3.0.10
ordered-set 4.0.2
packaging 21.3
pandas 1.4.2
pathtools 0.1.2
Pillow 9.0.1
pip 22.2.2
plotly 5.4.0
portalocker 2.3.2
progressbar2 3.55.0
promise 2.3
protobuf 3.19.1
psutil 5.8.0
py7zr 0.16.1
pyarrow 9.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
pycryptodomex 3.15.0
pyOpenSSL 22.0.0
pyparsing 3.0.6
pyppmd 0.16.1
PySocks 1.7.1
python-dateutil 2.8.2
python-utils 2.5.6
pytorch-transformers 1.2.0
pytz 2021.3
pyu2f 0.1.5
PyYAML 6.0
pyzstd 0.14.4
qpth 0.0.15
regex 2021.11.10
requests 2.28.1
requests-oauthlib 1.3.0
responses 0.18.0
retry-decorator 1.1.1
rsa 4.7.2
s3transfer 0.6.0
sacrebleu 2.0.0
sacremoses 0.0.46
scikit-learn 1.0.1
scipy 1.7.3
seaborn 0.11.2
sentencepiece 0.1.97
sentry-sdk 1.5.1
setproctitle 1.2.2
setuptools 58.0.4
shortuuid 1.0.8
six 1.16.0
sklearn 0.0
smmap 5.0.0
subprocess32 3.5.4
tabulate 0.8.9
tenacity 8.0.1
tensorboard 2.7.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
termcolor 1.1.0
texttable 1.6.4
threadpoolctl 3.0.0
tokenizers 0.13.0
torch 1.12.1
torchaudio 0.12.1
torchtext 0.13.1
torchvision 0.13.1
tornado 6.1
tqdm 4.62.3
transformers 4.22.2
typing_extensions 4.3.0
ultimate-anatome 0.1.1 /Users/brandomiranda/ultimate-anatome
ultimate-aws-cv-task2vec 0.0.1 /Users/brandomiranda/ultimate-aws-cv-task2vec
ultimate-utils 0.6.1 /Users/brandomiranda/ultimate-utils/ultimate-utils-proj-src
urllib3 1.26.11
wandb 0.13.3
Werkzeug 2.0.2
wheel 0.37.0
xxhash 2.0.2
yarl 1.8.1
yaspin 2.1.0
zipp 3.8.0
hugging face (HF) related gitissues:
- https://github.com/huggingface/transformers/issues/15062
- https://github.com/huggingface/tokenizers/issues/120
- https://github.com/huggingface/tokenizers/issues/585
- https://github.com/huggingface/transformers/issues/11262
- SO related: Transformer: Error importing packages. "ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler'"
- HF discuss: https://discuss.huggingface.co/t/how-to-resolve-the-hugging-face-error-importerror-cannot-import-name-is-tokenizers-available-from-transformers-utils/23957
- reddit: https://www.reddit.com/r/pytorch/comments/xusuuy/how_to_resolve_the_hugging_face_error_importerror/
- https://github.com/huggingface/tokenizers/issues/1080
from How to resolve the hugging face error ImportError: cannot import name 'is_tokenizers_available' from 'transformers.utils'?
No comments:
Post a Comment