I have gone through similar questions that has been asked before (for example [1] [2]). However, none of them completely relevant for my problem.

I am trying to calculate a dot product between two large matrices and I have some memory constraint that I have to meet.

I have a numpy sparse matrix, which is a shape of (10000,600000). For example,

from scipy import sparse as sps
x = sps.random(m=10000, n=600000, density=0.1).toarray()

The second numpy matrix is of size (600000, 256), which consists of only (-1, 1).

import numpy as np
y = np.random.choice([-1,1], size=(600000, 256))

I need dot product of x and y at lowest possible memory required. Speed is not the primary concern.

Here is what I have tried so far:

Scipy Sparse Format:

Naturally, I converted the numpy sparse matrix to scipy csr_matrix. However, task is still getting killed due to memory issue. There is no error, I just get killed on the terminal.

from scipy import sparse as sps
sparse_x = sps.csr_matrix(x, copy=False)
z = sparse_x.dot(y)
# killed

Decreasing dtype precision + Scipy Sparse Format:

from scipy import sparse as sps

x = x.astype("float16", copy=False)
y = y.astype("int8", copy=False)

sparse_x = sps.csr_matrix(x, copy=False)
z = sparse_x.dot(y)
# Increases the memory requirement for some reason and dies

np.einsum

Not sure if it helps/works with sparse matrix. Found something interesting in this answer. However, following doesn't help either:

z = np.einsum('ij,jk->ik', x, y)
# similar memory requirement as the scipy sparse dot

Suggestions?

If you have any suggestions to improve any of these. Please let me know. Further, I am thinking in the following directions:

It would be great If I can get rid of dot product itself somehow. My second matrix (i.e. y is randomly generated and it just has [-1, 1]. I am hoping if there is way I could take advantage of its features.
May be diving dot product into several small dot product and then, aggregate.

from Memory efficient dot product between a sparse matrix and a non-sparse numpy matrix

TL;DR - This is a PyCharm remote interpreter question.

Remote libraries are not properly synced, and PyCharm is unable to index properly when using remote interpreter. Everything runs fine.

Following is the entire (currently unsuccessful) debug process

I am using a virtual environment created with python -m venv venv, then pointing to it as I always have using ssh interpreter. The exact same happens with conda as well.

After configuring the interpreter, many of the installed packages are marked red by PyCharm, not giving auto complete, and not knowing these packages.

Here is pip freeze

absl-py==1.3.0
aggdraw==1.3.15
aiohttp==3.8.3
aiosignal==1.3.1
albumentations==1.3.0
algo-api==2.5.0
algo-flows==2.4.0
allegroai==3.6.1
altair==4.2.0
amqp==5.1.1
anomalib==0.3.2
antlr4-python3-runtime==4.9.3
anyio==3.6.2
astunparse==1.6.3
async-timeout==4.0.2
attrs==20.3.0
bcrypt==4.0.1
bleach==5.0.1
boto3==1.26.34
botocore==1.29.34
cachetools==5.2.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
clearml==1.8.3
click==8.1.3
commonmark==0.9.1
contourpy==1.0.6
cpu-cores==0.1.3
cryptography==38.0.4
cycler==0.11.0
DateTime==4.7
decorator==5.1.1
deepmerge==1.1.0
dnspython==2.2.1
docker-pycreds==0.4.0
docopt==0.6.2
docutils==0.19
dotsi==0.0.3
efficientnet==1.0.0
einops==0.6.0
entrypoints==0.4
fastapi==0.88.0
ffmpy==0.3.0
fire==0.5.0
Flask==2.2.2
flatbuffers==1.12
focal-loss==0.0.7
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
furl==2.1.3
future==0.18.2
gast==0.4.0
gitdb==4.0.10
GitPython==3.1.29
google-auth==2.15.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
gradio==3.15.0
grpcio==1.51.1
gunicorn==20.1.0
h11==0.14.0
h5py==3.7.0
httpcore==0.16.3
httpx==0.23.1
humanfriendly==9.2
idna==3.4
image-classifiers==1.0.0
imageio==2.23.0
imagesize==1.4.1
imgaug==0.4.0
importlib-metadata==5.2.0
importlib-resources==5.10.1
imutils==0.5.4
inflection==0.5.1
iniconfig==1.1.1
itsdangerous==2.1.2
jaraco.classes==3.2.3
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==3.2.0
keras==2.9.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
keyring==23.13.1
kiwisolver==1.4.4
kmeans1d==0.3.1
kornia==0.6.8
libclang==14.0.6
linkify-it-py==1.0.3
luqum==0.11.0
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe==2.1.1
maskrcnn-benchmark==1.1.2+cu113
matplotlib==3.6.2
mdit-py-plugins==0.3.3
mdurl==0.1.2
ml-distillery==1.0.1
more-itertools==9.0.0
multidict==6.0.3
networkx==2.8.8
numpy==1.23.1
oauthlib==3.2.2
omegaconf==2.3.0
opencv-python==4.6.0.66
opencv-python-headless==4.6.0.66
opt-einsum==3.3.0
orderedmultidict==1.0.1
orjson==3.8.3
overrides==6.1.0
packaging==22.0
pandas==1.5.2
paramiko==2.12.0
pathlib==1.0.1
pathlib2==2.3.7.post1
pathtools==0.1.2
pika==1.3.1
Pillow==9.3.0
pkginfo==1.9.2
pluggy==1.0.0
ply==3.11
promise==2.3
protobuf==3.19.6
psd-tools==1.9.23
psutil==5.9.4
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyclipper==1.3.0.post4
pycocotools==2.0.6
pycparser==2.21
pycpd==2.0.0
pycryptodome==3.16.0
pydantic==1.9.0
pyDeprecate==0.3.2
pydub==0.25.1
pygit2==1.11.1
Pygments==2.13.0
pyhumps==3.8.0
PyJWT==2.4.0
pymongo==4.1.1
PyNaCl==1.5.0
pyparsing==2.4.7
pyrsistent==0.19.2
pytest==7.1.2
python-dateutil==2.8.2
python-multipart==0.0.5
pytorch-lightning==1.6.4
pytz==2022.7
PyWavelets==1.4.1
PyYAML==6.0
qudida==0.0.4
readme-renderer==37.3
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
rfc3986==1.5.0
rich==12.6.0
rsa==4.9
s3transfer==0.6.0
scikit-image==0.19.3
scikit-learn==1.1.3
scipy==1.9.3
SecretStorage==3.3.3
segmentation-models==1.0.1
sentry-sdk==1.12.1
setproctitle==1.3.2
shapely==2.0.0
shortuuid==1.0.11
six==1.16.0
sklearn==0.0.post1
smmap==5.0.0
sniffio==1.3.0
starlette==0.22.0
tensorboard==2.9.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.1
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.29.0
termcolor==2.1.1
threadpoolctl==3.1.0
tifffile==2022.10.10
timm==0.5.4
tinytree==0.2.1
tomli==2.0.1
toolz==0.12.0
torch==1.10.2+cu113
torchmetrics==0.9.0
torchtext==0.11.2
torchvision==0.11.3+cu113
tqdm==4.64.0
twine==4.0.2
typing-utils==0.1.0
typing_extensions==4.4.0
uc-micro-py==1.0.1
urllib3==1.26.13
uv-build-utils==1.4.0
uv-envyaml==2.0.1
uv-python-serving==2.0.1
uv-python-utils==1.12.0
uvicorn==0.20.0
uvrabbit==1.4.1
validators==0.20.0
vine==5.0.0
wandb==0.12.17
webencodings==0.5.1
websockets==10.4
Werkzeug==2.2.2
windshield-grid-localisation==1.0.0.dev5
wrapt==1.14.1
yacs==0.1.8
yarl==1.8.2
zipp==3.11.0
zope.interface==5.5.2

The following minimal test program

import pytest
import uv_python_utils
from importlib_metadata import version as version_query
from pkg_resources import parse_version
import requests


installed_pytest_version = parse_version(version_query('pytest'))
installed_uv_python_utils_version = parse_version(version_query('uv_python_utils'))
installed_importlib_metadata_version = parse_version(version_query('importlib_metadata'))
print(installed_pytest_version)
print(installed_uv_python_utils_version)
print(installed_importlib_metadata_version)

runs with output

7.1.2
1.12.0
5.2.0

but in the IDE, it looks like this:

Here is the support ticket for JetBrains (not sure if visible for everyone or not). They were not able to help yet.

They offered, and I have done all of the following which did not help:

Delete ~/.pycharm_helpers on remote
Go to Help | Find Action... and search for "Registry...". In the registry, search for python.use.targets.api and disable it. Reconfigure your project interpreter.

I also tried to purge everything from Pycharm from both local and remote, and reinstall, and this persists.

Uninstall PyCharm, resinstall an older version that works for a colleague (works on the same remote in the same directory for the colleague, so the problem is local)
Delete .idea
Delete C:\Users\noam.s\AppData\Roaming\JetBrains
Obviously I tried invalidate caches & restart.

The libraries just don't get downloaded to the External Libraries, as shown in the Project menu, which doesn't agree with pip freeze

In the venv case:

In the conda case, the downloaded remote libraries don't even agree with the Pycharm interpreter screen!

This really makes it hard for me to work and I am not able to find any workaround. Any ideas?

For the bounty, it is enough for me to know how to fix it, though it would be preferable to know:

Why does this happen?
How to fix it?
How to avoid it?

from Pycharm doesn't recognize packages with remote interpreter

Hemant Vishwakarma

Saturday, 24 December 2022

Type hint that excludes a certain type

Memory efficient dot product between a sparse matrix and a non-sparse numpy matrix

Scipy Sparse Format:

Decreasing dtype precision + Scipy Sparse Format:

np.einsum

Suggestions?

Pycharm doesn't recognize packages with remote interpreter

TL;DR - This is a PyCharm remote interpreter question.