Wednesday, 2 August 2023

Error "'jupyter' is not recognized as an internal or external command, operable program or batch file" when installing Jupyter Notebook

I am trying to install Jupyter Notebook without installing Anaconda on my Windows. I have followed the steps in https://jupyter.org/install but seems not to work. I have tried to close & reopen the command prompt and restart the Windows but didn't work too. What did I miss?

C:\Users\xxxxxx>pip install notebook
Requirement already satisfied: notebook in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (6.5.4)
Requirement already satisfied: jinja2 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (3.1.2)
Requirement already satisfied: tornado>=6.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (6.2)
Requirement already satisfied: pyzmq>=17 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (25.0.2)
Requirement already satisfied: argon2-cffi in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (21.3.0)
Requirement already satisfied: traitlets>=4.2.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (5.9.0)
Requirement already satisfied: jupyter-core>=4.6.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (5.3.0)
Requirement already satisfied: jupyter-client>=5.3.4 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (8.1.0)
Requirement already satisfied: ipython-genutils in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.2.0)
Requirement already satisfied: nbformat in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (5.8.0)
Requirement already satisfied: nbconvert>=5 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (7.3.1)
Requirement already satisfied: nest-asyncio>=1.5 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (1.5.6)
Requirement already satisfied: ipykernel in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from notebook) (6.22.0)
Requirement already satisfied: Send2Trash>=1.8.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (1.8.0)
Requirement already satisfied: terminado>=0.8.3 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.17.1)
Requirement already satisfied: prometheus-client in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.16.0)
Requirement already satisfied: nbclassic>=0.4.7 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from notebook) (0.5.5)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-client>=5.3.4->notebook) (2.8.2)
Requirement already satisfied: platformdirs>=2.5 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from jupyter-core>=4.6.1->notebook) (3.1.1)
Requirement already satisfied: pywin32>=300 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from jupyter-core>=4.6.1->notebook) (305)
Requirement already satisfied: jupyter-server>=1.8 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbclassic>=0.4.7->notebook) (2.5.0)
Requirement already satisfied: notebook-shim>=0.1.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbclassic>=0.4.7->notebook) (0.2.3)
Requirement already satisfied: beautifulsoup4 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (4.11.2)
Requirement already satisfied: bleach in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (6.0.0)
Requirement already satisfied: defusedxml in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (0.7.1)
Requirement already satisfied: jupyterlab-pygments in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (0.2.2)
Requirement already satisfied: markupsafe>=2.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (2.1.2)
Requirement already satisfied: mistune<3,>=2.0.3 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (2.0.5)
Requirement already satisfied: nbclient>=0.5.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (0.7.4)
Requirement already satisfied: packaging in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (21.3)
Requirement already satisfied: pandocfilters>=1.4.1 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (1.5.0)
Requirement already satisfied: pygments>=2.4.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from nbconvert>=5->notebook) (2.14.0)
Requirement already satisfied: tinycss2 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbconvert>=5->notebook) (1.2.1)
Requirement already satisfied: fastjsonschema in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbformat->notebook) (2.16.3)
Requirement already satisfied: jsonschema>=2.6 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from nbformat->notebook) (4.17.3)
Requirement already satisfied: pywinpty>=1.1.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from terminado>=0.8.3->notebook) (2.0.10)
Requirement already satisfied: argon2-cffi-bindings in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from argon2-cffi->notebook) (21.2.0)
Requirement already satisfied: comm>=0.1.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipykernel->notebook) (0.1.2)
Requirement already satisfied: debugpy>=1.6.5 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipykernel->notebook) (1.6.6)
Requirement already satisfied: ipython>=7.23.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipykernel->notebook) (8.11.0)
Requirement already satisfied: matplotlib-inline>=0.1 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipykernel->notebook) (0.1.6)
Requirement already satisfied: psutil in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipykernel->notebook) (5.9.4)
Requirement already satisfied: backcall in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipython>=7.23.1->ipykernel->notebook) (0.2.0)
Requirement already satisfied: decorator in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from ipython>=7.23.1->ipykernel->notebook) (5.1.1)
Requirement already satisfied: jedi>=0.16 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipython>=7.23.1->ipykernel->notebook) (0.18.2)
Requirement already satisfied: pickleshare in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipython>=7.23.1->ipykernel->notebook) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipython>=7.23.1->ipykernel->notebook) (3.0.38)
Requirement already satisfied: stack-data in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from ipython>=7.23.1->ipykernel->notebook) (0.6.2)
Requirement already satisfied: colorama in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from ipython>=7.23.1->ipykernel->notebook) (0.4.5)
Requirement already satisfied: attrs>=17.4.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (22.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (0.19.3)
Requirement already satisfied: anyio>=3.1.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (3.6.2)
Requirement already satisfied: jupyter-events>=0.4.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (0.6.3)
Requirement already satisfied: jupyter-server-terminals in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (0.4.4)
Requirement already satisfied: websocket-client in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (1.5.1)
Requirement already satisfied: six>=1.5 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from python-dateutil>=2.8.2->jupyter-client>=5.3.4->notebook) (1.16.0)
Requirement already satisfied: cffi>=1.0.1 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from argon2-cffi-bindings->argon2-cffi->notebook) (1.15.1)
Requirement already satisfied: soupsieve>1.2 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from beautifulsoup4->nbconvert>=5->notebook) (2.4)
Requirement already satisfied: webencodings in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from bleach->nbconvert>=5->notebook) (0.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from packaging->nbconvert>=5->notebook) (3.0.9)
Requirement already satisfied: idna>=2.8 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (3.4)
Requirement already satisfied: sniffio>=1.1 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (1.3.0)
Requirement already satisfied: pycparser in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook) (2.21)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from jedi>=0.16->ipython>=7.23.1->ipykernel->notebook) (0.8.3)
Requirement already satisfied: python-json-logger>=2.0.4 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (2.0.7)
Requirement already satisfied: pyyaml>=5.3 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (6.0)
Requirement already satisfied: rfc3339-validator in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (0.1.4)
Requirement already satisfied: rfc3986-validator>=0.1.1 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jupyter-events>=0.4.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook) (0.1.1)
Requirement already satisfied: wcwidth in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=7.23.1->ipykernel->notebook) (0.2.6)
Requirement already satisfied: executing>=1.2.0 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from stack-data->ipython>=7.23.1->ipykernel->notebook) (1.2.0)
Requirement already satisfied: asttokens>=2.1.0 in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from stack-data->ipython>=7.23.1->ipykernel->notebook) (2.2.1)
Requirement already satisfied: pure-eval in c:\users\xxxxxx\appdata\roaming\python\python310\site-packages (from stack-data->ipython>=7.23.1->ipykernel->notebook) (0.2.2)
Requirement already satisfied: fqdn in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (1.5.1)
Requirement already satisfied: isoduration in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (20.11.0)
Requirement already satisfied: jsonpointer>1.13 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (2.3)
Requirement already satisfied: uri-template in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (1.2.0)
Requirement already satisfied: webcolors>=1.11 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from jsonschema>=2.6->nbformat->notebook) (1.13)
Requirement already satisfied: arrow>=0.15.0 in c:\users\xxxxxx\appdata\local\programs\python\python310\lib\site-packages (from isoduration->jsonschema>=2.6->nbformat->notebook) (1.2.3)

C:\Users\xxxxxx>jupyter notebook
'jupyter' is not recognized as an internal or external command,
operable program or batch file.


from Error "'jupyter' is not recognized as an internal or external command, operable program or batch file" when installing Jupyter Notebook

Tuesday, 1 August 2023

Vectorized sum and product with list of pandas data frames

I have a list of list of Data Frames, each corresponding to a different time period t from 0 to N. Each data frame has multiple types, i need to preform the finally for each type in the data frame.

An example data set would be as follows, i made each df in the list the same values for simplicity but the calculation would remain the same.

d = {'type': ['a', 'b', 'c'], 'x': [1, 2, 3], 'y':[3,4,5]}
df = pd.DataFrame(data=d)

l = []
window = 20
[l.append(df.copy(deep=True)) for i in range(window)]

I need to compute a vectored sum $$\sum$$ and product $$\prod$$ using the above list of dataframes for each type (a, b, c) in an efficient manner. e.g. for each calculation below I need to filter by a single type df[df['type'] == 'a'] for every df in the list.

enter image description here

If i use a df.groupby('type') on every df in the list it could be very slow using a larger data set. The same goes if i use nested for loops for the sum and the product and filtering by type in each iteration of the for loop. How can I compute this sum product in an efficient manner ?

Update

One possible way as suggested in the comments is below, however this will start to be very inefficient if I use a larger dataset or window:

import pandas as pd
import math

d = {'type': ['a', 'b', 'c'], 'x': [1, 2, 3], 'y':[3, 4, 5]}
df = pd.DataFrame(data=d)

l = []
window = 20
for i in range(window):
    df['window'] = i
    l.append(df.copy(deep=True))
df = pd.concat(l)

sums = {}
types = df['type'].unique()
for s in types:
    sums[s] = 0
    tdf = df[df['type'] == s]
    for i in range(window):
        sums[s] += tdf[tdf['window'] == i]['x'].values[0] * math.prod(2 - tdf[tdf['window'] == i+1]['x'].values[0] for i in range(0, window-1)) * tdf[tdf['window'] == i]['y'].values[0]


from Vectorized sum and product with list of pandas data frames

PyCharm add python extra path: Custom module path for missing import resolution

I'm coming from VSCode into PyCharm as my team wants me to use PyCharm. It's a hard requirements.

In my current python project, I have a python code where we didn't commit all the generated (grpc) python code. To generate this, let's say we run a command like: bazel build //...

The project will run only if you use bazel, but bear with me. It's supposedly not a complex question.

The directory looks like this:

    - workspaceFolder (source root)
    |-- models
      |-- dao_user.py
      |-- dao_accounts.py
    |-- database
      |-- transaction.py
      |-- google_spanner.py
    |-- service
      |-- grpc_user_svc.py
    ...
    |-- .bazel-bin (auto-generated by bazel build)
      |-- api
        |-- protos
          |-- awesome_pb2.py
          |-- awesome_service_pb2.py
          |-- authentication.py

However, this will output the generated code into a different folder inside ${workspaceFolder}/.bazel-bin/. Of course, this should give an issue in the PyCharm interpreter code where we would be missing some imports because the python code is not in the workspace folder (a.k.a the project source root).

# Pycharm has issues finding this modules
from api.protos.authentication import AuthenticationRequest 
from api.protos import awesome_pb2 as api_awersome_pb2
from api.protos import awesome_service_pb2

My question is, how can tell PyCharm to get the above imports to be analysed from .bazel-bin folders?

  • I don't want to move the code from .bazel-bin.
  • I want PyCharm interpreter to understand the extra source code path from .bazel-bin/

I'm pretty sure it's possible. I have made it work in VSCode. I can do this in VSCode by having the following configuration settings:

// .vscode/settings.json
{ 
 "python.analysis.extraPaths": [
    "${workspaceFolder}/bazel-bin/api",
    "${workspaceFolder}/bazel-bin/cloud",
    "${workspaceFolder}/bazel-bin/storage"
  ],
  "python.linting.pylintPath": "pylint",
  "python.linting.pydocstyleEnabled": false,
  "python.linting.banditEnabled": false,
  "python.linting.flake8Enabled": false,
  "python.linting.prospectorEnabled": false,
  "python.languageServer": "Pylance",
  "python.analysis.autoImportCompletions": false,
}

Any help will be appreciated. My team also doesn't know how to do this in PyCharm.



from PyCharm add python extra path: Custom module path for missing import resolution