Friday 30 June 2023

Google Sign-In Not Working in Next.js with Firebase due to Cross-Origin-Opener-Policy Error

I am developing a web application using Next.js and Firebase. I have successfully implemented Google sign-in in my application, but I've encountered an issue which is preventing the sign-in process from completing as expected.

When I attempt to sign in, a popup window appears as intended. However, in the console, I see the following error:

Cross-Origin-Opener-Policy policy would block the window.closed call.

Because of this error, the sign-in process doesn't complete and the user is not properly signed in.

Some further context: I'm testing this functionality in a development environment, on localhost. The issue is active on Chrome, but not on Firefox.

Could this error be due to a misconfiguration of the Cross-Origin-Opener-Policy? If so, how can I correctly configure this policy for Google Sign-in with Firebase in a Next.js application?

enter image description here



from Google Sign-In Not Working in Next.js with Firebase due to Cross-Origin-Opener-Policy Error

Can not install module I uploaded

I am trying to upload my package to PyPI and it uploads successfully every time though when I try to install it, the following error is thrown

ERROR: Could not find a version that satisfies the requirement Google-Ads-Transparency-Scraper==1.4 (from versions: none)
ERROR: No matching distribution found for Google-Ads-Transparency-Scraper==1.4

The following is the directory structure I have

  • GoogleAds
    • GoogleAdsTransparency
      • __init __.py
      • main.py
      • regions.py
    • setup.py
    • setup.cfg
    • license.txt
    • README.md

The setup.py has the following content

"""Install packages as defined in this file into the Python environment."""
from setuptools import setup, find_packages

setup(
    name="Google Ads Transparency Scraper",
    author="Farhan Ahmed",
    author_email="jattfarhan10@gmail.com",
    url="https://github.com/faniAhmed/GoogleAdsTransparencyScraper",
    description="A scraper for getting Ads from Google Transparency",
    version="1.4",
    packages=find_packages(),
    download_url= 'https://github.com/faniAhmed/GoogleAdsTransparencyScraper/archive/refs/tags/v1.2.tar.gz',
    keywords= ['Google', 'Transparency', 'Scraper', 'API', 'Google Ads', 'Ads', 'Google Transparency', 'Google Transparency Scraper', 'Google Ads Scrapre'],
    license='Securely Incorporation',
    install_requires=[
        "setuptools>=45.0",
        "beautifulsoup4>=4.12.2",
        "Requests>=2.31.0",
        "lxml>=4.6.3",
    ],
    classifiers=[
        "Development Status :: 5 - Production/Stable",
        "Environment :: Console",
        "Intended Audience :: Developers",
        "License :: Free for non-commercial use",
        "Natural Language :: Urdu",
        "Operating System :: OS Independent",
        "Programming Language :: Python",
    ],
    platforms=["any"],
)

The setup.cfg

[metadata]
description-file = README.md

The init.py has the following content

from GoogleAdsTransparency.main import GoogleAds
from GoogleAdsTransparency.main import show_regions_list

I run the following commands to upload the package

python setup.py sdist
twine upload dist/*

The link to the package is https://pypi.org/project/Google-Ads-Transparency-Scraper/

Thanks for any help in advance



from Can not install module I uploaded

Inserting data as vectors from SQL Database to Pinecone

I have a profiles table in SQL with around 50 columns, and only 244 rows. I have created a view with only 2 columns, ID and content and in content I concatenated all data from other columns in a format like this: FirstName: John. LastName: Smith. Age: 70, Likes: Gardening, Painting. Dislikes: Soccer.

Then I created the following code to index all contents from the view into pinecone, and it works so far. However I noticed something strange.

  1. There are over 2000 vectors and still not finished, the first iterations were really fast, but now each iteration is taking over 18 seconds to finish and it says it will take over 40 minutes to finish upserting. (but for 244 rows only?)

What am I doing wrong? or is it normal?

 pinecone.init(
        api_key=PINECONE_API_KEY,  # find at app.pinecone.io
        environment=PINECONE_ENV  # next to api key in console
    )

    import streamlit as st
    st.title('Work in progress')
    embed = OpenAIEmbeddings(deployment=OPENAI_EMBEDDING_DEPLOYMENT_NAME, model=OPENAI_EMBEDDING_MODEL_NAME, chunk_size=1)
   
    cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+DATABASE_SERVER+'.database.windows.net;DATABASE='+DATABASE_DB+';UID='+DATABASE_USERNAME+';PWD='+ DATABASE_PASSWORD)
    query = "SELECT * from views.vwprofiles2;"
    df = pd.read_sql(query, cnxn)
    index = pinecone.Index("default")
   
    batch_limit = 100

    texts = []
    metadatas = []

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=400,
        chunk_overlap=20,
        length_function=tiktoken_len,
        separators=["\n\n", "\n", " ", ""]
    )
    

    for _, record in stqdm(df.iterrows(), total=len(df)):
        # First get metadata fields for this record
        metadata = {
            'IdentityId': str(record['IdentityId'])
        }
        # Now we create chunks from the record text
        record_texts = text_splitter.split_text(record['content'])
        # Create individual metadata dicts for each chunk
        record_metadatas = [{
            "chunk": j, "text": text, **metadata
        } for j, text in enumerate(record_texts)]
        # Append these to the current batches
        texts.extend(record_texts)
        metadatas.extend(record_metadatas)
        # If we have reached the batch_limit, we can add texts
        if len(texts) >= batch_limit:
            ids = [str(uuid4()) for _ in range(len(texts))]
            embeds = embed.embed_documents(texts)
            index.upsert(vectors=zip(ids, embeds, metadatas))    
            texts = []
            metadatas = []

        if len(texts) > 0:
            ids = [str(uuid4()) for _ in range(len(texts))]
            embeds = embed.embed_documents(texts)
            index.upsert(vectors=zip(ids, embeds, metadatas))


from Inserting data as vectors from SQL Database to Pinecone

How do I print the wandb sweep url in python?

For runs I do:

wandb.run.get_url()

how do I do the same but for sweeps given the sweep_id?


fulls sample run:

"""
Main Idea:
- create sweep with a sweep config & get sweep_id for the agents (note, this creates a sweep in wandb's website)
- create agent to run a setting of hps by giving it the sweep_id (that mataches the sweep in the wandb website)
- keep running agents with sweep_id until you're done

note:
    - Each individual training session with a specific set of hyperparameters in a sweep is considered a wandb run.

ref:
    - read: https://docs.wandb.ai/guides/sweeps
"""

import wandb
from pprint import pprint
import math
import torch

sweep_config: dict = {
    "project": "playground",
    "entity": "your_wanbd_username",
    "name": "my-ultimate-sweep",
    "metric":
        {"name": "train_loss",
         "goal": "minimize"}
    ,
    "method": "random",
    "parameters": None,  # not set yet
}

parameters = {
    'optimizer': {
        'values': ['adam', 'adafactor']}
    ,
    'scheduler': {
        'values': ['cosine', 'none']}  # todo, think how to do
    ,
    'lr': {
        "distribution": "log_uniform_values",
        "min": 1e-6,
        "max": 0.2}
    ,
    'batch_size': {
        # integers between 32 and 256
        # with evenly-distributed logarithms
        'distribution': 'q_log_uniform_values',
        'q': 8,
        'min': 32,
        'max': 256,
    }
    ,
    # it's often the case that some hps we don't want to vary in the run e.g. num_its
    'num_its': {'value': 5}
}
sweep_config['parameters'] = parameters
pprint(sweep_config)

# create sweep in wandb's website & get sweep_id to create agents that run a single agent with a set of hps
sweep_id = wandb.sweep(sweep_config)
print(f'{sweep_id=}')


def my_train_func():
    # read the current value of parameter "a" from wandb.config
    # I don't think we need the group since the sweep name is already the group
    run = wandb.init(config=sweep_config)
    print(f'{run=}')
    pprint(f'{wandb.config=}')
    lr = wandb.config.lr
    num_its = wandb.config.num_its

    train_loss: float = 8.0 + torch.rand(1).item()
    for i in range(num_its):
        # get a random update step from the range [0.0, 1.0] using torch
        update_step: float = lr * torch.rand(1).item()
        wandb.log({"lr": lr, "train_loss": train_loss - update_step})
    run.finish()


# run the sweep, The cell below will launch an agent that runs train 5 times, usingly the randomly-generated hyperparameter values returned by the Sweep Controller.
wandb.agent(sweep_id, function=my_train_func, count=5)

cross: https://community.wandb.ai/t/how-do-i-print-the-wandb-sweep-url-in-python/4133



from How do I print the wandb sweep url in python?

Why is useEffect render-blocking(paint-blocking) in the tooltip example?

I am following the tooltip example for the useLayoutEffect hook from the new react docs. From their explanation, I assume that the following is the order of execution of things:

react render() --> reconciliation --> update DOM if virtual dom is changed --> DOM update finishes --> useLayoutEffect is executed --> broswer paints and triggers some sort of LayoutPaint event --> useEffect is triggered

To verify this, I swapped useLayouteffect hook with useEffect hook in the tooltip.js file of the docs example.

This is the modified example

I have put a few for loops to slow down the execution of useEffect hook. Now when you load the modifed example and move your mouse over to any of the three buttons, you will see first paint with tooltip in wrong position and then the useEffect takes 1-2s and then you will see another repaint with the tooltip at correct position. So far so good, but now any later mouseovers on the same button, you will see that the wrong-position-paint waits for useEffect to finish and then the correct-position-paint happens within a few miliseconds. So I have two questions:

  1. Why does later mouseovers cause useEffect to become render-blocking?

  2. How does react make sure that useLayoutEffect stops browser paint from happening and if any state update within useLayoutEffect is mentioned then triggers another render->repaint while completely disallowing the previous browser paint to occur at all. In our case the tooltip at -60px is not painted at all.



from Why is useEffect render-blocking(paint-blocking) in the tooltip example?

Thursday 29 June 2023

Error in r.listen(source) while using the speech recognition module

I am trying to run a basic code for speech recognition. But I am getting an error in the following line:

audio = r.listen(source)

The small piece of code that I am trying is:

import speech_recognition as s_r
print(s_r.__version__) # just to print the version not required
r = s_r.Recognizer()
my_mic = s_r.Microphone() #my device index is 1, you have to put your device index
with my_mic as source:
    print("Say now!!!!")
    audio = r.listen(source) #take voice input from the microphone
print(r.recognize_google(audio)) #to print voice into text

I have developed multiple projects using this code but this time I am getting the following error:

3.10.0
Say now!!!!
Traceback (most recent call last):
  File "D:\professional\JARVIS\assistant\test.py", line 7, in <module>
    audio = r.listen(source) #take voice input from the microphone
  File "C:\Users\theas\AppData\Roaming\Python\Python38\site-packages\speech_recognition\__init__.py", line 465, in listen
    assert source.stream is not None, "Audio source must be entered before listening, see documentation for ``AudioSource``; are you using ``source`` outside of a ``with`` statement?"
AssertionError: Audio source must be entered before listening, see documentation for ``AudioSource``; are you using ``source`` outside of a ``with`` statement?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\professional\JARVIS\assistant\test.py", line 7, in <module>
    audio = r.listen(source) #take voice input from the microphone
  File "C:\Users\theas\AppData\Roaming\Python\Python38\site-packages\speech_recognition\__init__.py", line 189, in __exit__
    self.stream.close()
AttributeError: 'NoneType' object has no attribute 'close'
>>> 

I am using this code on a new system with Windows 11. I have installed a fresh Python with default settings with version 3.11.4. The following library versions have been installed for speech recognition:

SpeechRecognition: 3.10.0
Pyaudio: 0.2.13

The same code is working in my other system. Please help me find this silly mistake I made during the setup.

UPDATE: When I try to print the objects of my_mic. I get the following output:

['CHUNK', 'MicrophoneStream', 'SAMPLE_RATE', 'SAMPLE_WIDTH', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'audio', 'device_index', 'format', 'get_pyaudio', 'list_microphone_names', 'list_working_microphones', 'pyaudio_module', 'stream']

When I try to get a list of all the microphones present, I get a large list. But when I try to get the list of working microphones, that is empty.



from Error in r.listen(source) while using the speech recognition module

On the web how do you make the code be highlighted as the user is typing it? I only managed to make it that the code is highlighted on button click

In my PicoBlaze Simulator in JavaScript, I managed to program that, when the user presses the button "Highlight Assembly", the assembly code is highlighted. But the cursor is then moved to the beginning of the assembly program. How could I make it so that the assembly code is highlighted as the user is typing it?

I have tried it this way:
In the footerscript.js I added the following (and I commented it out when I realized it isn't working):

// For now, attempting to highlight code as the user is typing is worse
// than useless, because it moves the cursor to the beginning.
/*
document.getElementById("assemblyCode").
       oninput=syntaxHighlighter;
*/

In the footerScript.js file, I added the following code at the beginning of the syntaxHighlighter function:

function syntaxHighlighter(/*edit*/) {
  //"edit" should contain the
  // cursor position, but that
  // seems not to work.
  if (areWeHighlighting)
    return;
  areWeHighlighting = true;
  const assemblyCodeDiv = document.getElementById("assemblyCode");
  const assemblyCode = assemblyCodeDiv.innerText.replace(/&/g, "&amp;")
                           .replace(/</g, "&lt;")
                           .replace(/>/g, "&gt;");
  // const start=edit.selectionStart,
  //  end=edit.selectionEnd; //Cursor position.

And, at the end of the syntaxHighlighter function, in that same file, I added the following:

  assemblyCodeDiv.innerHTML = highlightedText;
  // The following code is supposed to move the cursor to the correct
  // position, but it doesn't work.
  /*
  const range=document.createRange();
  range.setStart(assemblyCodeDiv,start);
  range.setEnd(assemblyCodeDiv,end);
  const selection=window.getSelection();
  selection.removeAllRanges();
  selection.addRange(range);
  */

However, that's not working, so I commented it out. Why doesn't it work? How do I make it work?

It is important to me that my PicoBlaze Simulator continues working in Firefox 52 (the last version of Firefox to work on Windows XP), as many computers at my university run Windows XP and use Firefox 52 as the browser.



from On the web, how do you make the code be highlighted as the user is typing it? I only managed to make it that the code is highlighted on button click

REST API call in OpenText TeamSite

Is there any way to call REST API in TeamSite? For example in .Net application we can use visual studio solution and call REST API via HTTPClient but working pattern in TeamSite is completely different. TeamSite has one admin portal where we can see files organized in tree structure for example index.aspx but there is no VS solution to write any code. It keeps everything in admin portal. The only way is using inline code approach as explained in Write Inline code

Is there any better way that TeamSite support for REST API call. Basically we need to call REST API which will return data in JSON format and set data in JavaScript variable. We need to use returned data for further processing.

TeamSite version: 20.2.0.5



from REST API call in OpenText TeamSite

Cmake problems after upgrading to MacOS 13.0

As mentioned on the title, CMake seems to be broken after upgrading to MacOS 13.0.

Trying to install something that requires Cmakes takes unusually long then the following pop-up shows up.

“CMake” is damaged and can’t be opened. You should move it to the Trash. 

This file was downloaded on an unknown date.  # this txt is grey and smaller font

Pop-up Options
1. Move to Trash  2. Cancel

Steps to reproduce Err

  1. Cloned EasyOCR

    1.1 git clone ...

  2. Made Python venv

    2.0. cd EasyOCR/

    2.1. python3 -m venv venv

    2.2. source venv/bin/activate

    2.3. venv info

    python --version && pip --version
    # output   
    Python 3.10.6
    pip 22.3 from ...  # path to venv dir
    
  3. pip install -r requirements.txt

# requirements.txt content
torch
torchvision>=0.5
opencv-python-headless<=4.5.4.60
scipy
numpy
Pillow
scikit-image
python-bidi
PyYAML
Shapely
pyclipper
ninja
  1. After a while, the aforementioned pop-up, shows up. Clicking on either option will result in the following error.
Building wheels for collected packages: opencv-python-headless
  Building wheel for opencv-python-headless (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for opencv-python-headless (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
        File "/private/var/folders/5h/36chnb_s3b5fpqmqgt_7cz_m0000gn/T/pip-build-env-ea_5u80v/overlay/lib/python3.10/site-packages/skbuild/setuptools_wrap.py", line 613, in setup
          cmkr = cmaker.CMaker(cmake_executable)
        File "/private/var/folders/5h/36chnb_s3b5fpqmqgt_7cz_m0000gn/T/pip-build-env-ea_5u80v/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 141, in __init__
          self.cmake_version = get_cmake_version(self.cmake_executable)
        File "/private/var/folders/5h/36chnb_s3b5fpqmqgt_7cz_m0000gn/T/pip-build-env-ea_5u80v/overlay/lib/python3.10/site-packages/skbuild/cmaker.py", line 95, in get_cmake_version
          raise SKBuildError(
      Traceback (most recent call last):
      
      Problem with the CMake installation, aborting build. CMake executable is cmake
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for opencv-python-headless
Failed to build opencv-python-headless
ERROR: Could not build wheels for opencv-python-headless, which is required to install pyproject.toml-based projects

Any thoughts on how to fix or get around this? This is the first time I see this pop up.



from Cmake problems after upgrading to MacOS 13.0

Avoiding extra next call after yield from in Python generator

Please see the below snippet, run with Python 3.10:

from collections.abc import Generator

DUMP_DATA = 5, 6, 7

class DumpData(Exception):
    """Exception used to indicate to yield from DUMP_DATA."""

def sample_gen() -> Generator[int | None, int, None]:
    out_value: int | None = None
    while True:
        try:
            in_value = yield out_value
        except DumpData:
            yield len(DUMP_DATA)
            yield from DUMP_DATA
            out_value = None
            continue
        out_value = in_value

My question pertains to the DumpData path where there is a yield from. After that yield from, there needs to be a next(g) call, to bring the generator back to the main yield statement so we can send:

def main() -> None:
    g = sample_gen()
    next(g)  # Initialize
    assert g.send(1) == 1
    assert g.send(2) == 2

    # Okay let's dump the data
    num_data = g.throw(DumpData)
    data = tuple(next(g) for _ in range(num_data))
    assert data == DUMP_DATA

    # How can one avoid this `next` call, before it works again?
    next(g)
    assert g.send(3) == 3

How can this extra next call be avoided?



from Avoiding extra `next` call after `yield from` in Python generator

Wednesday 28 June 2023

cytoscape.js problems with autoungrabify or autolock?

I am using Cytoscape.js to create a web application. To enable users to add edges, I have integrated the Edgehandles extension. Two kinds of edges can be added: undirected and directed. To add directed edges, the user holds down the control or apple key. To add undirected edges, the space key is used. Adding directed edges works perfectly. However, when I add an undirected edge, the nodes cannot be moved anymore. node.grabbable() returns false for all nodes. While this seems to always return true before adding an undirected edge.

I am surprised since all I do to make an edge undirected is add a class (see cy.on('add', 'edge', function (event) { even listener). At no point do I explicitly modify node.grabbable(). I thus highly suspect this has to do with the autoungrabify, or autolock functionality of cytoscape.js. However, as you can see in the code, I set those to false.

Any help would be greatly appreciated!

It seems like the even listeners are not working here. So, you probably have to download the files to reproduce the problem.

var cy = cytoscape({
  container: document.getElementById('cy'),
  elements: [],
  autoungrabify: false,
  autolock: false,
  style: [{
      selector: 'edge',
      style: {
        'target-arrow-shape': 'triangle',
        'curve-style': 'bezier',
      }
    },
    {
      selector: 'edge.undirected',
      style: {
        'target-arrow-shape': 'triangle',
        'source-arrow-shape': 'triangle'
      }
    }
  ]
});

var layout = cy.layout({
  name: 'grid'
});
var eh = cy.edgehandles();

var nodeIdCounter = 0;

function addNode(nodeType, position) {
  var finalPosition = {
    x: Math.random() * 400 + 50,
    y: Math.random() * 400 + 50
  };
  cy.add({
    group: 'nodes',
    classes: nodeType,
    position: finalPosition
  });
}

var spaceKeyDown = false;
document.addEventListener('keydown', function(event) {
  // Check if the Command key was pressed
  if (event.key === 'Meta' || event.key === 'Control' || event.key === ' ') {
    eh.enableDrawMode();
  }
  if (event.key === ' ') {
    spaceKeyDown = true;
  }
});

// Listen for keyup event
document.addEventListener('keyup', function(event) {
  // Check if the Command key was released
  if (event.key === 'Meta' || event.key === 'Control' || event.key === ' ') {
    eh.disableDrawMode();
  }
  if (event.key === ' ') {
    spaceKeyDown = false;
  }
});

cy.on('add', 'edge', function(event) {
  var edge = event.target;
  if (spaceKeyDown) {
    edge.addClass('undirected');
  } else {
    edge.addClass('directed');
  }
});

addNode('observed-variable');
addNode('observed-variable');
addNode('observed-variable');
<!DOCTYPE html>
<html>

<head>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.20/lodash.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/cytoscape/3.2.9/cytoscape.min.js"></script>
  <script src="https://cdn.rawgit.com/cytoscape/cytoscape.js-edgehandles/master/cytoscape-edgehandles.js"></script>
  <script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
  <style>
    #cy {
      z-index: 1;
      width: 70%;
      height: 80vh;
      padding: 10px;
    }
  </style>
</head>

<body>
  <div id="cy"></div>
  <script src="javascript.js"></script>
</body>

</html>


from cytoscape.js problems with autoungrabify or autolock?

How to enter data to IWA fields via Playwright

I need to automate a test that uses IWA (Integrated Windows Authentication).

I know that the the prompt that opens up is not part of the HTML page, yet why my code is not working:

login_page.click_iwa()
sleep(5)
self.page.keyboard.type('UserName')
sleep(5)
self.page.keyboard.press('Tab')
self.page.keyboard.type('Password')


from How to enter data to IWA fields via Playwright

error exporting matplotlib file to kml - python

I'm trying to export a matplotlib figure using a kml format. The question is identical to the following:

Export Python Plot to KML

I've outlined the exact function but I can't get the kml output to work. Conversely, if I export a simplekml function, it's working fine.

I've attached both outputs below. Output 1 one works but 2 doesn't.

Output 1:

import simplekml
kml = simplekml.Kml()
kml.newpoint(name="Kirstenbosch", coords=[(18.432314,33.988862)])
kml.save("botanicalgarden.kml")

enter image description here

But when trying to pass a matplotlib function to simplekml function, I'm returning the following output. What am I doing wrong?

Output 2:

import matplotlib
from mpl_toolkits.basemap import Basemap
import matplotlib.pyplot as ppl
from pylab import rcParams
import simplekml
rcParams['figure.figsize'] = (8,8)

# create rectangle over 0 to 10 degrees longitude and 0 to 10 degrees latitude
x = [0, 10, 10, 0, 0]
y = [10, 10, 0, 0, 10]
x1 = range(0,11)    # to draw a diagonal line

fig = ppl.figure(1)
ax = fig.add_axes([0,0,1,1])
ax.axis('off')
fig.patch.set_facecolor('blue')  # so we can see the true extent

ppl.plot(x, y, 'r', linewidth=3)
ppl.plot(x, y, '.b', linewidth=3)
ppl.plot(x1, x1, 'g', linewidth=3)

ppl.axis('off')
border1 = ppl.axis()

if False:
    ppl.show()
else:
    pngName = 'Overlay.png'
    fig.savefig(pngName, facecolor=fig.get_facecolor(), transparent=False)

bottomleft  = (border1[0],border1[2])
bottomright = (border1[1],border1[2])
topright    = (border1[1],border1[3])
topleft     = (border1[0],border1[3])

kml = simplekml.Kml()
ground = kml.newgroundoverlay(name='GroundOverlay')
ground.icon.href = pngName
ground.gxlatlonquad.coords =[bottomleft, bottomright, topright, topleft]
kml.save("GroundOverlay.kml")

enter image description here



from error exporting matplotlib file to kml - python

Tuesday 27 June 2023

Deep Layout Parsing Invalid argument

I've been trying to run layout parser example from here https://layout-parser.readthedocs.io/en/latest/example/deep_layout_parsing/index.html

The problem is that when I try:

model = lp.Detectron2LayoutModel('lp://PubLayNet/faster_rcnn_R_50_FPN_3x/config',
                                 extra_config=["MODEL.ROI_HEADS.SCORE_THRESH_TEST", 0.8],
                                 label_map={0: "Text", 1: "Title", 2: "List", 3:"Table", 
                                 4:"Figure"})

I get this error:

OSError: [Errno 22] Invalid argument: 'C:\\Users\\user/.torch/iopath_cache\\s/f3b12qc4hc0yh4m\\config.yml?dl=1.lock'

Originally, config.yml was not in the expected folder. But downloading it did not solve the problem.

I would appreciate a solution for this, or an alternative suggestion if it exists.



from Deep Layout Parsing Invalid argument

Why project.addSourceFilesAtPaths("../../tsconfig.json"); cannot get the correct result source files?

I use ts-morph to find all the source files:

the key code:

let tsmorph = require('ts-morph') 

const project = new tsmorph.Project();

project.addSourceFilesAtPaths("../../tsconfig.json");

const sourceFiles = project.getSourceFiles();

console.log(sourceFiles)  // there get empty array:[]

when I run npx ts-node test/unit/ts-morph-test.ts: but there get empty array:[]

why it cannot get the correct results?

my code uploaded to codesandbox: https://codesandbox.io/s/2ntc8q

the test project code structure: you see it obviously has ts files.

enter image description here



from Why `project.addSourceFilesAtPaths("../../tsconfig.json");` cannot get the correct result source files?

Can't spawn background processes after the app was running for some time

In our Python application we spawn some background processes when the program starts up. Then in some cases we need to shutdown one of them and some time later we want to re-start (= spawn) another background process again.

However, it seems that sometimes, when the application was running for some days already (it's a server application), then it cannot spawn another process. In this case we get the following error log at the call of new_proc.start():

Jun 23 14:23:46 srv gunicorn[2717986]: Traceback (most recent call last):
Jun 23 14:23:46 srv gunicorn[2717986]:   File "<string>", line 1, in <module>
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
Jun 23 14:23:46 srv gunicorn[2717986]:     exitcode = _main(fd, parent_sentinel)
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
Jun 23 14:23:46 srv gunicorn[2717986]:     self = reduction.pickle.load(from_parent)
Jun 23 14:23:46 srv gunicorn[2717986]:   File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
Jun 23 14:23:46 srv gunicorn[2717986]:     self._semlock = _multiprocessing.SemLock._rebuild(*state)
Jun 23 14:23:46 srv gunicorn[2717986]: FileNotFoundError: [Errno 2] No such file or directory

So apparently the Python multiprocessing module cannot create the file required for a semaphore (or something similar). When I then shutdown the whole application and restart it, then again all subprocesses can be spawned again. So it really seems to be related to the fact that the app was running for some days already ...

Any ideas what might be the issue here?



from Can't spawn background processes after the app was running for some time

Converting a real-time MP3 audio stream to 8000/mulaw in Python

I'm working with an API that streams real-time audio in the MP3 format (44.1kHz/16bit) and I need to convert this stream to 8000/mulaw. I've tried several solutions, but all have run into issues due to the structure of the MP3 data.

My current approach is to decode and process each chunk of audio as it arrives, using PyDub and Python's audioop module. However, I often encounter errors that seem to arise from trying to decode a chunk of data that doesn't contain a complete MP3 frame.

Here's a simplified version of my current code:

from pydub import AudioSegment
import audioop
import io

class StreamConverter:
    def __init__(self):
        self.state = None  
        self.buffer = b''  

    def convert_chunk(self, chunk):
        # Add the chunk to the buffer
        self.buffer += chunk

        # Try to decode the buffer
        try:
            audio = AudioSegment.from_mp3(io.BytesIO(self.buffer))
        except CouldntDecodeError:
            return None

        # If decoding was successful, empty the buffer
        self.buffer = b''

        # Ensure audio is mono
        if audio.channels != 1:
            audio = audio.set_channels(1)

        # Get audio data as bytes
        raw_audio = audio.raw_data

        # Sample rate conversion
        chunk_8khz, self.state = audioop.ratecv(raw_audio, audio.sample_width, audio.channels, audio.frame_rate, 8000, self.state)

        # μ-law conversion
        chunk_ulaw = audioop.lin2ulaw(chunk_8khz, audio.sample_width)

        return chunk_ulaw

# This is then used as follows:
for chunk in audio_stream:
    if chunk is not None:
        ulaw_chunk = converter.convert_chunk(chunk)
        # do something with ulaw_chunk

I believe my issue stems from the fact that MP3 data is structured in frames, and I can't reliably decode the audio if a chunk doesn't contain a complete frame. Also, a frame could potentially be split between two chunks, so I can't decode them independently.

Does anyone have any ideas on how I can handle this? Is there a way to process an MP3 stream in real-time while converting to 8000/mulaw, possibly using a different library or approach?



from Converting a real-time MP3 audio stream to 8000/mulaw in Python

Sticky headers with overflow?

I have a table where I want to make the header and first column sticky (as shown in the example).

It works as intended, until I add overflow-x: scroll; (uncomment line 3 in styles.css). overflow-x: scroll retains the first columns stickiness but breaks the top headers stickiness.

A solution I tried but doesn't quite work:

I've read into this topic a bit and found an article that describes a solution to this exact problem. The article suggests extracting your headers into a separate div and then using a JS scroll sync library to synchronize the overflow-x scrolling of your headers and your rows (example 1, example 2).

This solves the stickiness + overflow problem, but introduces a new problem. The column width now can't adjust based on the length of data in the cells (as it does in my initial example) and computing this manually for every column seems sketchy.

How can I add overflow-x: scroll; to my table while retaining the following:

  • Sticky header
  • Sticky first column
  • Column widths that adjust based on data provided
  • No nested vertical scroll (the window scrollbar should be the only vertical scroll)

I'm open to hacks with JS.

EDIT: I found another article that suggests using a JS onScroll event listener to continuously translate the table headers to the top of the screen. I tried this solution and the effect is too janky to use with the amount of data I'm using.



from Sticky headers with overflow?

What is the cause of "RuntimeWarning: invalid value encountered in matmul ret = a @ b" when using sklearn?

I encountered RuntimeWarning when using Ridge().fit(X_train, y_train) on a dataset where X_train.shape = (9440, 1900)

/Users/username/micromamba/envs/data-science/lib/python3.11/site-packages/sklearn/utils/extmath.py:189: RuntimeWarning: invalid value encountered in matmul
  ret = a @ b

When I reduce the size of dataset X_train.shape = (1000, 1900) the RuntimeWarning goes away.

What might be causing this warning and how can I avoid it? I made sure there are no np.nan, np.inf, and -np.inf.



from What is the cause of "RuntimeWarning: invalid value encountered in matmul ret = a @ b" when using sklearn?

Monday 26 June 2023

How to disable / hide the mouse cursor to be recorded

I want to record the screen of the user, but I want to hide / exclude the cursor to be recorded. I researched about this and found out this article :-

https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackSettings/cursor

The above can be used to disable the mouse to be recorded from the recording and in the official documents it is mentioned cursor as a available values :- enter image description here

I am not sure where and how to use the above parameter to disable or hide the mouse cursor. I am using tabCapture api which basically is like getUserDisplay :-

chrome.tabCapture.capture({ 
     video: true,
     audio: false,
     videoConstraints:{
                  
     mandatory:
          {cursor: 'never', minFrameRate:70,maxFrameRate:120,maxWidth:1920,maxHeight:1080}} 
              
          }

I used the cursor parameter like above, but it didn't work, the cursor still showing up in the recording, It would be a great help if you please show a small demo for the recording of the screen but excluding the cursor

Thank you for reading :)



from How to disable / hide the mouse cursor to be recorded

Plotly plot single trace with 2 yaxis

I have a plotly graph object bar chart for which I want to display 2 y-axis (different currencies, so the conversion factor ist constant).

Currently I plot 1 trace each, while for the second one I set opacity to 0, disable the legend and hoverinfo. This hack works, but is ugly to maintain.

I'm aware of https://plotly.com/python/multiple-axes/

Is there an easier way to do this?

Goal: enter image description here

The conversion-Factor here is 3.9 CHF per credit



from Plotly plot single trace with 2 yaxis

How to change Traceback back to normal?

My most recent env prints traceback like this

╭───────────────────── Traceback (most recent call last) ──────────────────────╮

which is beyond useless.

I've already looked at this How to make typer traceback look normal but it doesn't help.

My hunch is it may be about Huggingface but maybe something else like datasets or evaluate but I can't find anything useful so far.

How to make stacktrace print everything it need to again?



from How to change Traceback back to normal?

Add livedoc-mocha to existing project

I'm trying add livedoc-mocha (https://github.com/dotnetprofessional/LiveDoc/tree/master/packages/livedoc-mocha) to an existing project here: https://github.com/rocket-pool/rocketpool

I installed livedoc with npm install --save-dev livedoc-mocha but as for this part I'm not sure what to do

mocha --ui livedoc-mocha --reporter livedoc-mocha/livedoc-spec --recursive path-to-my-tests/

Is there a way to modify an existing configuration file within this repo so that when I run npm test it will run mocha with livedoc-mocha? This is what the test field in the scripts section looks like in package.json:

"test": "hardhat test test/rocket-pool-tests.js",

There is also a hardhat.config.js file which has a mocha section that looks like this:

mocha: {
    timeout: 0,
},


from Add livedoc-mocha to existing project

automatically create Python class factory from __init__.py

I have this code that assigns the class name in a dict to the class. I've been adding to feature_expanded_factory manually and find this is inefficient especially if the class name change or a class is added.

Instead I'd like to create feature_expander_factory from the __init__.py below. So it should take every class from the __init__.py file then create a dict where the class name is assigned the to the class.

from data_processing.feature_expanders import CategoricalToOneHot, RFMSplitter, RFMSplitterAndOneHot, \
    StrToListToColumns

feature_expander_factory = dict(CategoricalToOneHot=CategoricalToOneHot, RFMSplitter=RFMSplitter,
                                RFMSplitterAndOneHot=RFMSplitterAndOneHot, ListToColumns=StrToListToColumns)

__init__.py

from data_processing.feature_expanders.AbstractFeatureExpander import AbstractFeatureExpander
from data_processing.feature_expanders.CategoricalToOneHot import CategoricalToOneHot
from data_processing.feature_expanders.RFMSplitter import RFMSplitter
from data_processing.feature_expanders.RFMSplitterAndOneHot import RFMSplitterAndOneHot
from data_processing.feature_expanders.StrToListToColumns import StrToListToColumns


from automatically create Python class factory from __init__.py

Saturday 24 June 2023

Django oauth2 request.user returns AnonymousUser

I'm a Django newbie and am trying to create an API to the backend for use by iOS client. Currently I am testing my API access with curl.

I have successfully generated access tokens using:

curl -X POST -d "grant_type=password&username=example_user&password=example_password" http://clientID:clientSecret@localhost:8000/o/token/

which generated the following response -

{
    "access_token": "oAyNlYuGVk1Sr8oO1EsGnLwOTL7hAY", 
    "scope": "write read", 
    "expires_in": 36000, 
    "token_type": "Bearer", 
    "refresh_token": "pxd5rXzz1zZIKEtmqPgE608a4H9E5m"
}

I then used the access token to try to access the following class view:

from django.http import JsonResponse
from oauth2_provider.views.generic import ProtectedResourceView
from django.views.decorators.csrf import csrf_exempt
from django.utils.decorators import method_decorator

class post_test(ProtectedResourceView):

    def get(self, request, *args, **kwargs):
        print(request.user)
        return JsonResponse({'Message': "You used a GET request"})

    def post(self, request, *args, **kwargs):
        print(request.user)
        return JsonResponse({'Message': 'You used a POST request'})

    @method_decorator(csrf_exempt)
    def dispatch(self, *args, **kwargs):
        return super(post_test, self).dispatch(*args, **kwargs)

with the following curl request:

curl -H "Authorization: Bearer oAyNlYuGVk1Sr8oO1EsGnLwOTL7hAY" -X GET http://localhost:8000/api/post-test/

Which properly responds to the client with:

{"Message": "You used a GET request"}

But in the console, where I expect the request.user variable, I get AnonymousUser.

Isn't the token supposed to be assigned to example_user? Shouldn't that be what request.user returns?



from Django oauth2 request.user returns AnonymousUser

Dynamic JS content not loading when testing with Selenium Chrome headless browser in Java

I'm trying to test a website I made as part of a Spring Boot project, and I'm using Selenium. I'm able to test basic stuff like loading the page and the title, but I'm struggling with the actual content of the page.

My page has the following section:

<div id="main">
    <div id="div_1"></div>
    <div id="div_2"></div>
    <div id="div_3"></div>
    <div id="div_4"></div>
</div>

The content is loaded from a JS script:

document.addEventListener("DOMContentLoaded", function(event) {
    populateDivs()
})

I'm initialising the WebDriver with the following options gathered from similar questions (other Selenium tests continue passing, so I don't think the options are conflicting):

final ChromeOptions options = new ChromeOptions();
options.addArguments(
    "--headless",
    "--nogpu",
    "--disable-gpu",
    "--enable-javascript",
    "--no-sandbox",
    "--disable-extensions",
    "--disable-blink-features=AutomationControlled",
    "--disable-features=NetworkService,NetworkServiceInProcess",
     "start-maximized",
     "disable-infobars"
);

I've also added a wait in my test case to give time for the content to load:

final WebDriverWait wait = new WebDriverWait(driver, Duration.ofMinutes(1L));
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("div_1_content")));

Nothing I try changes the fact that the no content is loaded even after waiting. Not really sure where to go from here - am I using Selenium incorrectly? Or should I load my content in a different way on the JS side?

Thanks!



from Dynamic JS content not loading when testing with Selenium Chrome headless browser in Java

Friday 23 June 2023

ReactNative App Crashes when on WallpaperManager.setbitmap - Android native module

Aim:
Passing a Base64 from react native app to a native module, which will set it as wallpaper.

Currently:
When I am passing a Base64 String, the wallpaper is set successfully but, the app crashes(closes) or resets ( like reopen state, back to app's home screen ).

Problem:
the app crashes after successfully setting the wallpaper.

React Native Code

...
import {NativeModules} from 'react-native';
const {WallpaperMod} = NativeModules;
...
const Page = (...) => {

const base64String = '...'

 useEffect(()=>{
   WallpaperMod.setWallpaper(base64String);  👈 
 },[])

return(...)

export default Page
}

Android module code (WallpaperMod.java)

package com.zenwalls;

...

public class WallpaperMod extends ReactContextBaseJavaModule {
    ReactApplicationContext context;
    WallpaperMod(ReactApplicationContext context) {
        super(context);
        this.context = context;
    }

    @Override
    public String getName() {
        return "WallpaperMod";
    }

    @ReactMethod
    public void setWallpaper(String image64) {
        Log.d("testing", "Print: a"+ image64);
        byte[] decodedString = Base64.decode(image64, Base64.DEFAULT);
        Bitmap bitmap = BitmapFactory.decodeByteArray(decodedString, 0, decodedString.length);
        WallpaperManager wallpaperManager = WallpaperManager.getInstance(context);
        try {
            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {

                wallpaperManager.setBitmap(bitmap, null, false, WallpaperManager.FLAG_SYSTEM);  👈 

                return;
            }
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }
}

Error

Fatal Exception

FATAL EXCEPTION: main
                                                                                           Process: com.zenwalls, PID: 5202
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.zenwalls/com.zenwalls.MainActivity}: androidx.fragment.app.Fragment$InstantiationException: Unable to instantiate fragment com.swmansion.rnscreens.ScreenFragment: calling Fragment constructor caused an exception
    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3815)
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3977)
    at android.app.ActivityThread.handleRelaunchActivityInner(ActivityThread.java:6013)
    at android.app.ActivityThread.handleRelaunchActivity(ActivityThread.java:5902)
    at android.app.servertransaction.ActivityRelaunchItem.execute(ActivityRelaunchItem.java:71)
    at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45)
    at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2374)
    at android.os.Handler.dispatchMessage(Handler.java:106)
    at android.os.Looper.loopOnce(Looper.java:233)
    at android.os.Looper.loop(Looper.java:344)
    at android.app.ActivityThread.main(ActivityThread.java:8249)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:589)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1071)
Caused by: androidx.fragment.app.Fragment$InstantiationException: Unable to instantiate fragment com.swmansion.rnscreens.ScreenFragment: calling Fragment constructor caused an exception
    at androidx.fragment.app.Fragment.instantiate(Fragment.java:631)
    at androidx.fragment.app.FragmentContainer.instantiate(FragmentContainer.java:57)
    at androidx.fragment.app.FragmentManager$3.instantiate(FragmentManager.java:483)
    at androidx.fragment.app.FragmentStateManager.<init>(FragmentStateManager.java:85)
    at androidx.fragment.app.FragmentManager.restoreSaveState(FragmentManager.java:2728)
    at androidx.fragment.app.FragmentController.restoreSaveState(FragmentController.java:198)
    at androidx.fragment.app.FragmentActivity$2.onContextAvailable(FragmentActivity.java:149)
    at androidx.activity.contextaware.ContextAwareHelper.dispatchOnContextAvailable(ContextAwareHelper.java:99)
    at androidx.activity.ComponentActivity.onCreate(ComponentActivity.java:322)
    at androidx.fragment.app.FragmentActivity.onCreate(FragmentActivity.java:273)
    at com.facebook.react.ReactActivity.onCreate(ReactActivity.java:45)
    at android.app.Activity.performCreate(Activity.java:8130)
    at android.app.Activity.performCreate(Activity.java:8110)
    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1343)
    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3781)
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3977) 
    at android.app.ActivityThread.handleRelaunchActivityInner(ActivityThread.java:6013) 
    at android.app.ActivityThread.handleRelaunchActivity(ActivityThread.java:5902) 
    at android.app.servertransaction.ActivityRelaunchItem.execute(ActivityRelaunchItem.java:71) 
    at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45) 
    at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135) 
    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95) 
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2374) 
    at android.os.Handler.dispatchMessage(Handler.java:106) 
    at android.os.Looper.loopOnce(Looper.java:233) 
    at android.os.Looper.loop(Looper.java:344) 
    at android.app.ActivityThread.main(ActivityThread.java:8249) 
    at java.lang.reflect.Method.invoke(Native Method) 
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:589) 
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1071) 
Caused by: java.lang.reflect.InvocationTargetException
    at java.lang.reflect.Constructor.newInstance0(Native Method)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
    at androidx.fragment.app.Fragment.instantiate(Fragment.java:613)
    at androidx.fragment.app.FragmentContainer.instantiate(FragmentContainer.java:57) 
    at androidx.fragment.app.FragmentManager$3.instantiate(FragmentManager.java:483) 
    at androidx.fragment.app.FragmentStateManager.<init>(FragmentStateManager.java:85) 
    at androidx.fragment.app.FragmentManager.restoreSaveState(FragmentManager.java:2728) 
    at androidx.fragment.app.FragmentController.restoreSaveState(FragmentController.java:198) 
    at androidx.fragment.app.FragmentActivity$2.onContextAvailable(FragmentActivity.java:149) 
    at androidx.activity.contextaware.ContextAwareHelper.dispatchOnContextAvailable(ContextAwareHelper.java:99) 
    at androidx.activity.ComponentActivity.onCreate(ComponentActivity.java:322) 
    at androidx.fragment.app.FragmentActivity.onCreate(FragmentActivity.java:273) 
    at com.facebook.react.ReactActivity.onCreate(ReactActivity.java:45) 
    at android.app.Activity.performCreate(Activity.java:8130) 
    at android.app.Activity.performCreate(Activity.java:8110) 
    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1343) 
    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3781) 
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3977) 
    at android.app.ActivityThread.handleRelaunchActivityInner(ActivityThread.java:6013) 
    at android.app.ActivityThread.handleRelaunchActivity(ActivityThread.java:5902) 
    at android.app.servertransaction.ActivityRelaunchItem.execute(ActivityRelaunchItem.java:71) 
    at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45) 
    at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135) 
    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95) 
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2374) 
    at android.os.Handler.dispatchMessage(Handler.java:106) 
    at android.os.Looper.loopOnce(Looper.java:233) 
    at android.os.Looper.loop(Looper.java:344) 
    at android.app.ActivityThread.main(ActivityThread.java:8249) 
    at java.lang.reflect.Method.invoke(Native Method) 
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:589) 
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1071) 
Caused by: java.lang.IllegalStateException: Screen fragments should never be restored. Follow instructions from https://github.com/software-mansion/react-native-screens/issues/17#issuecomment-424704067 to properly configure your main activity.
    at com.swmansion.rnscreens.ScreenFragment.<init>(ScreenFragment.kt:54)
    at java.lang.reflect.Constructor.newInstance0(Native Method) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:343) 
    at androidx.fragment.app.Fragment.instantiate(Fragment.java:613) 
    at androidx.fragment.app.FragmentContainer.instantiate(FragmentContainer.java:57) 
    at androidx.fragment.app.FragmentManager$3.instantiate(FragmentManager.java:483) 
    at androidx.fragment.app.FragmentStateManager.<init>(FragmentStateManager.java:85) 
    at androidx.fragment.app.FragmentManager.restoreSaveState(FragmentManager.java:2728) 
    at androidx.fragment.app.FragmentController.restoreSaveState(FragmentController.java:198) 
    at androidx.fragment.app.FragmentActivity$2.onContextAvailable(FragmentActivity.java:149) 
    at androidx.activity.contextaware.ContextAwareHelper.dispatchOnContextAvailable(ContextAwareHelper.java:99) 
    at androidx.activity.ComponentActivity.onCreate(ComponentActivity.java:322) 
    at androidx.fragment.app.FragmentActivity.onCreate(FragmentActivity.java:273) 
    at com.facebook.react.ReactActivity.onCreate(ReactActivity.java:45) 
    at android.app.Activity.performCreate(Activity.java:8130) 
    at android.app.Activity.performCreate(Activity.java:8110) 
    at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1343) 
    at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3781) 
    at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3977) 
    at android.app.ActivityThread.handleRelaunchActivityInner(ActivityThread.java:6013) 
    at android.app.ActivityThread.handleRelaunchActivity(ActivityThread.java:5902) 
    at android.app.servertransaction.ActivityRelaunchItem.execute(ActivityRelaunchItem.java:71) 
    at android.app.servertransaction.ActivityTransactionItem.execute(ActivityTransactionItem.java:45) 
    at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135) 
    at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95) 
    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2374) 
    at android.os.Handler.dispatchMessage(Handler.java:106) 
    at android.os.Looper.loopOnce(Looper.java:233) 
    at android.os.Looper.loop(Looper.java:344) 
    at android.app.ActivityThread.main(ActivityThread.java:8249) 
    at java.lang.reflect.Method.invoke(Native Method) 
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:589) 
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1071) ```


  [1]: https://i.stack.imgur.com/ymWCh.jpg


from ReactNative App Crashes when on WallpaperManager.setbitmap - Android, native module

Plotly Sankey Diagram: How to display the value for each links and node on the link/node without hover?

In the Plotly Sankey diagram, you are able to see the 'value' of a link/node by hovering over it. I want the image to display the values without hovering though.

I've looked through the documentation and see virtually no way of doing this beside replacing the labels themselves with the desired value. That is not a good option, as nothing would then by labeled. Short of making dynamic labels that include both and and value, I'm not sure how to approach this.

Examples below...

Sample Sankey Diagram (source):

import plotly.graph_objects as go

fig = go.Figure(data=[go.Sankey(
    node = dict(
      pad = 15,
      thickness = 20,
      line = dict(color = "black", width = 0.5),
      label = ["A1", "A2", "B1", "B2", "C1", "C2"],
      customdata = ["Long name A1", "Long name A2", "Long name B1", "Long name B2",
                    "Long name C1", "Long name C2"],
      hovertemplate='Node %{customdata} has total value %{value}<extra></extra>',
      color = "blue"
    ),
    link = dict(
      source = [0, 1, 0, 2, 3, 3],
      target = [2, 3, 3, 4, 4, 5],
      value = [8, 4, 2, 8, 4, 2],
      customdata = ["q","r","s","t","u","v"],
      hovertemplate='Link from node %{source.customdata}<br />'+
        'to node%{target.customdata}<br />has value %{value}'+
        '<br />and data %{customdata}<extra></extra>',
  ))])

fig.update_layout(title_text="Basic Sankey Diagram", font_size=10)
fig.show()

Actual Output: Actual Output

Desired Output: Desired Output



from Plotly Sankey Diagram: How to display the value for each links and node on the link/node without hover?

How to do Google docs like pagination using React

In Google docs, If you create a document. You will start writing in Page 1. If the page is filled with content, then automatically Page 2 is created and the rest of the contents goes to page 2 and so on. If you remove some lines from Page 1, then the contents from Page 2 (If there are any) will be moved to the bottom of the page 1.

So in my scenario, I have some presentational components (Around 10) which gets filled by a form. Consider Left Pane has a form and updating them shows live preview of Docs like UI on the right pane.

I tried by directly manipulating DOM using some methods like insertBefore in a react app. I read that doing DOM manipulation directly on a React App is error prone and it is not recommended. Is there any other pattern or method to achieve this use case?



from How to do Google docs like pagination using React

Why does this shgo minimization fail?

I am trying to use linear constraints with shgo. Here is a simple MWE:

from scipy.optimize import shgo,  rosen


# Set up the constraints list
constraints = [{'type': 'ineq', 'fun': lambda x, i=i: x[i+1] - x[i] - 0.1} for i in range(2)]

# Define the variable bounds
bounds = [(0, 20)]*3

# Call the shgo function with constraints
result = shgo(rosen, bounds, constraints=constraints)

# Print the optimal solution
print("Optimal solution:", result.x)
print("Optimal value:", result.fun)

An example solution satisfying these constraints is:

rosen((0.1, 0.21, 0.32))
13.046181

But if you run the code you get:

Optimal solution: None
Optimal value: None

It doesn't find a feasible solution at all! Is this a bug?



from Why does this shgo minimization fail?

CornerstoneJS select segmentation by mouse and change the color

Im using cornerstoneTools with cornerstoneJS dicom view, I draw multiple segments by cornerstone segment tool and load the segments from RLE, I need to select existing segment by click it by mouse on the dicom viewer



from CornerstoneJS select segmentation by mouse and change the color

Selenium - Getting the Text in span tag

I am trying to get the followers count from Instagram. I am using selenium to do the task. Now the structure in which the followers is as follows(This is just to give you an idea. Please check the Instagram website using inspect tool)

[...]
<span class="_ac2a">
<span> 216 </span>
</span>
[...]

The above is the rough structure. I want 216. When I try the following code I get [] as the result The code:

        username = self.username
        driver.get(f"https://www.instagram.com/{username}/")
        try:
            #html = self.__scrape_page()
            #page = self.__parse_page(html)

            #followers = page.find_all("meta", attrs={"name": "description"})
            followers = driver.find_elements(By.XPATH, '//span[@class="_ac2a"]/span')
            return followers
        #     followers_count = (followers[0]["content"].split(",")[0].split(" ")[0])
        #     return {
        #         "data": followers_count,
        #         "message": f"Followers found for user {self.username}",
        #     }
        except Exception as e:
            message = f"{self.username} not found!"
            return {"data": None, "message": message}

How do I get the followers?



from Selenium - Getting the Text in span tag

Thursday 22 June 2023

Unable to delete cookie using Next.js server side action

I'm trying to delete a cookie using the next/headers module in a Next.js application, but it doesn't seem to work as expected. Here's the code snippet:

import {cookies} from "next/headers";
export default async function Signout() {
    async function deleteTokens() {
       "use server"

        cookies().delete('accessToken')
    }

  await deleteTokens()
  return (
      <></>
  );
}

I expected the cookies().delete('accessToken') line to delete the cookie named "accessToken", but it doesn't seem to have any effect. The cookie is still present after the function is executed.

I am getting following error: Error: Cookies can only be modified in a Server Action or Route Handler. Read more: https://nextjs.org/docs/app/api-reference/functions/cookies#cookiessetname-value-options

But i have set the "use server" and enabled it in the next.config.js

#using Next.js 13.4.4



from Unable to delete cookie using Next.js server side action

duckdb query takes too long to process and return inside Flask application

I have a Flask app and want to use duckdb as a database for several endpoints. My idea is to query the data and return it as a .parquet file. When I test my database with a simple Python script outside of the Flask app, it can query the data and save it as a .parquet in under a second. When I bring that same methodology to the Flask app, it still successfully queries the data and returns it as a .parquet file but it takes roughly 45 seconds. Other endpoints that return a .parquet file -- ones that are pre-staged and do not need to be queried -- can do so in just a second or two. So the issue, apparently, is incorporating duckdb inside my Flask application. Here is a sample boiler plate of what I have:

@test.route('/duckdb', methods = ['GET'])
def duckdb_test():

    con = duckdb.connect(database = '~/flask_db/test.db')

    # get tempfile .parquet
    tmp = tempfile.NamedTemporaryFile(suffix = '.parquet', mode = 'w+b', delete = False)

    # get data
    df = con.sql("SELECT * FROM tbl WHERE name = 'John'").to_df()

    # write to temporary .parquet
    df.to_parquet(tmp.name, engine='pyarrow', index=False)

    return send_file(tmp.name, mimetype='application/octet-stream', as_attachment=True, download_name="request.parquet")

I want to save it as a temp file. Not really sure what's wrong here. Again, it does work, but it just takes way, way too much time. The data being returned is about 12,000 rows in a ~5.5M row database -- but given that it works fairly quickly outside of the Flask app, on the same VM, the size itself shouldn't be an issue.



from duckdb query takes too long to process and return inside Flask application

Plot regression confidence interval using seaborn.objects interface (v0.12)

I'm trying to use the objects API to plot a regression line with confidence limits (like sns.regplot / sns.lmplot).

Based on the so.Band fmri example, I thought it would be something like this:

import seaborn as sns
import seaborn.objects as so

(so.Plot(sns.load_dataset("tips"), x="total_bill", y="tip")
   .add(so.Dots())
   .add(so.Line(), so.PolyFit()) # regression line
   .add(so.Band(), so.Est())     # confidence band
)

But I'm getting an unexpected result (left). I'm expecting something like the regplot band (right).

current and expected output



from Plot regression confidence interval using seaborn.objects interface (v0.12)

How to update user session from backend after updating user's metadata?

I´m using Next.js for the client side, auth0 to handle authentication and Django Rest Framework for the backend. Following the Auth0's Manage Metadta Using the Management API guide, I achieved to set new metadata values (I have checked it from the Dashboard). As it is spectated, if the user refresh its profile page (pages/profile.js), the old session metadata is rendered.

So my question is, how can I update user session after the metadata values has been set?

I have tried to use updateSession from @auth/nextjs-auth I also have tried nextjs-auth0: update user session (without logging out/in) after updating user_metadata , but checkSession() is not defined so I got lost there.

index.js

   async function handleAddToFavourite() {
      if (selected) {
         const data = await axios.patch("api/updateUserSession",)
         // Update the user session for the client side
         checkSession() //this function is not defined
         
      }
   }

api/updateUserSession.js

async function handler(req, res) {

  const session = await getSession(req, res);

  if (!session || session === undefined || session === null) {
    return res.status(401).end();
  }

  const id = session?.user?.sub;
  const { accessToken } = session;

  const currentUserManagementClient = new ManagementClient({
    token: accessToken,
    domain: auth0_domain.replace('https://', ''),
    scope: process.env.AUTH0_SCOPE,
  });

  const user = await currentUserManagementClient.updateUserMetadata({ id }, req.body);
  await updateSession(req, res, { ...session, user }); // Add this to update the session

  return res.status(200).json(user);
}

export default withApiAuthRequired(handler);

api/auth/[...auth0].js

const afterRefetch = (req, res, session) => {
     const newSession = getSession(req, res)
     if (newSession) {
          return newSession as Promise<Session>
     }
     return session
}


export default handleAuth({
  async profile(req, res) {
    try {
      await handleProfile(req, res, {
        refetch: true,
        afterRefetch 
      });
    } catch (error: any) {
      res.status(error.status || 500).end(error.message);
    }
  },

});

But I still have to log out and log in to see the new metadata values. Is there any way to update user's session from Django Rest Framework after performing the metadata update? If not, how can I update it using updateSession in Node.js?

Thank you in advance for your time.



from How to update user session from backend after updating user's metadata?

How can I prevent the address bar in Safari iOS from reappearing when a dialog is open similar to the functionality on Airbnb?

I'm trying to prevent the scrolling of the body element in the background when modals or overlays are opened, similar to what's implemented on Airbnb's website. Here's the method I've been using:

Opening a Modal

I add the locked class to either the body or html element:

html {
  min-height: 100%;
}

body {
  height: 100%;
  margin: 0;
}

.locked {
  overflow: hidden;
  position: fixed;
  inset: 0 0 0 0;
}

Then, I use scrollTop to update the CSS top value, which keeps the current position in view.

Closing a Modal

Upon closing the modal, I remove the locked class and restore scrollTop.


This method has been generally successful, but there is an issue on iOS where the address bar reappears when the view is locked, causing a jarring jump in the content.

Interestingly, Airbnb seems to have overcome this issue without apparent deviation from this method.

What is the solution to prevent the reappearing address bar on mobile browsers, thereby eliminating the content jump?


Here is a visual representation of the current state when I open a dialog:

enter image description here

What I aim to achieve is demonstrated below:

enter image description here



from How can I prevent the address bar in Safari iOS from reappearing when a dialog is open, similar to the functionality on Airbnb?

setfit training with a pandas dataframe

I would like to train a zero shot classifier on an annotated sample dataset.

I am following some tutorials but as all use their own data and the same pretarined model, I am trying to confirm: Is this the best approach?

Data example: 

import pandas as pd
from datasets import Dataset
    
# Sample feedback data, it will have 8 samples per label
feedback_dict = [
    {'text': 'The product is great and works well.', 'label': 'Product Performance'},
    {'text': 'I love the design of the product.', 'label': 'Product Design'},
    {'text': 'The product is difficult to use.', 'label': 'Usability'},
    {'text': 'The customer service was very helpful.', 'label': 'Customer Service'},
    {'text': 'The product was delivered on time.', 'label': 'Delivery Time'}
]

# Create a DataFrame with the feedback data
df = pd.DataFrame(feedback_dict)

# convert to Dataset format
df = Dataset.from_pandas(df)

By having the previous data format, this is the approach for model finetunning:

from setfit import SetFitModel, SetFitTrainer

# Select a model
model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")

# training with Setfit
trainer = SetFitTrainer(
    model=model,
    train_dataset=df, # to keep the code simple I do not create the df_train
    eval_dataset=df, # to keep the code simple I do not create the df_eval
    column_mapping={"text": "text", "label": "label"} 
)

trainer.train()

The issue here is that the process never ends after more than 500 hours in a laptop, and the dataset it is only about 88 records with 11 labels.



from setfit training with a pandas dataframe

Web-scrapping ASPX page where loop over page number from Network>Payload

I am trying to scrap the table with column office, cadre, designation, name, and asset_details:

Here P2 ranges from 1 to 38.

In the Network > Payload > Form Data has the page number (__EVENTARGUMENT: Page$x), which can't be input and has to be ascertained.

http://bpsm.bihar.gov.in/Assets2020/AssetDetails.aspx?P1=2&P2=7&P3=0&P4=0

Here is my attempt at code:

import json
import requests
import pandas as pd


api_url = (
    "http://bpsm.bihar.gov.in/Assets2019/AssetDetails.aspx?P1=2&P2=33&P3=0&P4=0"
)
payload = {"P1": "2", "P2": "33", "P3": "0", "P4": "0"}


all_data = []
for P2 in range(1, 39):  # <-- increase from 1 to 200
    print(P2)
    payload['P2'] = P2
    data = requests.post(api_url, json=payload).json()
        data = json.loads(data['d'])
        if not data:
            break
        for name, count in data[0].items():
            all_data.append({
            })


from Web-scrapping ASPX page where loop over page number from Network>Payload

Wednesday 21 June 2023

Issue in setting up a tf.data pipeline for training TensorFlow model

I have an application in which I need to setup a pipeline using tf.data. The data I have is stored in .mat files created in Matlab and contains three variables "s_matrix" which is a 224x224x3 double array, a "frame" which is 1024x1 complex double and finally a numeric label. The pipeline is to be loaded as such so that I can feed the data to the model.fit function. The code I have been using so far to load and process the data is added below but I keep getting several errors of type and unexpected byte errors.

# Define the shape of the input image
input_shape = (224, 224, 3)

# Define the shape of the complex vector after conversion
complex_shape = (1024, 2, 1)


def load_and_preprocess_sample(sample_path):

    # Load the sample from the mat file
    sample = scipy.io.loadmat(sample_path)
    # Extract the matrix, complex vector and label from the sample
    matrix = sample['s_matrix']
    complex_vector = sample['frame']
    label = sample['numeric_label']

    # Convert the complex vector to the appropriate shape
    #complex_shape = (1024, 2, 1)
    #complex_vector = np.reshape(complex_vector, complex_shape)
    
    real = tf.reshape(tf.math.real(complex_vector), [1024, 1])
    imag = tf.reshape(tf.math.imag(complex_vector), [1024, 1])
    signal_tensor = tf.concat([real, imag], axis=-1)
    signal_tensor = tf.reshape(signal_tensor, [1024, 2, 1])
    signal = signal_tensor
    # Normalize the matrix values between 0 and 1
    matrix = matrix / 255.0

    # Convert the label to one-hot encoding
    label = tf.one_hot(label - 1, num_classes)

   
    return matrix, complex_vector, label

# Define a function to create a dataset from a list of file paths
def create_dataset(file_paths):
    # Create a dataset from the file paths
    dataset = tf.data.Dataset.from_tensor_slices(file_paths)

    # Shuffle the dataset
    dataset = dataset.shuffle(buffer_size=len(file_paths))

    # Load and preprocess each sample in parallel using CPU cores
    dataset = dataset.map(
        lambda x: tf.numpy_function(load_and_preprocess_sample, [x], [tf.float32, tf.float32, tf.float32]),
        num_parallel_calls=tf.data.AUTOTUNE)

    # Batch and prefetch the dataset for performance optimization
    dataset = dataset.batch(batch_size)
    dataset = dataset.prefetch(buffer_size=tf.data.AUTOTUNE)

    return dataset

when I try to extract the data for passing to model with below code, it all crashes when the zip function is called.

X_train_a, X_train_b, y_train = zip(*train_dataset)
X_val_a, X_val_b, y_val = zip(*val_dataset)
...
...
...
model.fit([X_train_a, X_train_b], y_train, batch_size=5, epochs=5, validation_data=([X_test_a, X_test_b], y_test))

the error output is shared below:

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
Cell In[6], line 84
     78 val_dataset = create_dataset(val_file_paths)
     82 #Y: attempting segragation of inputs
     83 # Convert the dataset into separate inputs
---> 84 X_train_a, X_train_b, y_train = zip(*train_dataset)
     85 X_val_a, X_val_b, y_val = zip(*val_dataset)
     87 # Define your model architecture here...

File ~\miniconda3\envs\tf2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py:766, in OwnedIterator.__next__(self)
    764 def __next__(self):
    765   try:
--> 766     return self._next_internal()
    767   except errors.OutOfRangeError:
    768     raise StopIteration

File ~\miniconda3\envs\tf2\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py:749, in OwnedIterator._next_internal(self)
    746 # TODO(b/77291417): This runs in sync mode as iterators use an error status
    747 # to communicate that there is no more data to iterate over.
    748 with context.execution_mode(context.SYNC):
--> 749   ret = gen_dataset_ops.iterator_get_next(
    750       self._iterator_resource,
    751       output_types=self._flat_output_types,
    752       output_shapes=self._flat_output_shapes)
    754   try:
    755     # Fast path for the case `self._structure` is not a nested structure.
    756     return self._element_spec._from_compatible_tensor_list(ret)  # pylint: disable=protected-access

File ~\miniconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py:3016, in iterator_get_next(iterator, output_types, output_shapes, name)
   3014   return _result
   3015 except _core._NotOkStatusException as e:
-> 3016   _ops.raise_from_not_ok_status(e, name)
   3017 except _core._FallbackException:
   3018   pass

File ~\miniconda3\envs\tf2\lib\site-packages\tensorflow\python\framework\ops.py:7209, in raise_from_not_ok_status(e, name)
   7207 def raise_from_not_ok_status(e, name):
   7208   e.message += (" name: " + name if name is not None else "")
-> 7209   raise core._status_to_exception(e) from None

InvalidArgumentError:  0-th value returned by pyfunc_0 is double, but expects float
     [[]] [Op:IteratorGetNext]

I am definitely doing something wrong, what is the solution?



from Issue in setting up a tf.data pipeline for training TensorFlow model

Use python AutoGPT and ChatGPT to extract data from downloaded HTML page

I already use Scrapy to crawl websites successfully. I extract specific data from a webpage using CSS selectors. However, it's time consuming to setup and error prone. I want to be able to pass the raw HTML to chatGPT and ask a question like

"Give me in a JSON object format the price, array of photos, description, key features, street address, and zipcode of the object"

Right now I run into the max chat length of 4096 characters. So I decided to send the page in chunks. However even with a simple question like "What is the price of this object?" I'd expect the answer to be "$945,000" but I'm just getting a whole bunch of text. I'm wondering what I'm doing wrong. I heard that AutoGPT offers a new layer of flexibility so was also wondering if that could be a solution here.

My code:

import requests
from bs4 import BeautifulSoup, Comment
import openai
import json

# Set up your OpenAI API key
openai.api_key = "MYKEY"

# Fetch the HTML from the page
url = "https://www.corcoran.com/listing/for-sale/170-west-89th-street-2d-manhattan-ny-10024/22053660/regionId/1"
response = requests.get(url)

# Parse and clean the HTML
soup = BeautifulSoup(response.text, "html.parser")

# Remove unnecessary tags, comments, and scripts
for script in soup(["script", "style"]):
    script.extract()

# for comment in soup.find_all(text=lambda text: isinstance(text, Comment)):
#     comment.extract()

text = soup.get_text(strip=True)

# Divide the cleaned text into chunks of 4096 characters
def chunk_text(text, chunk_size=4096):
    chunks = []
    for i in range(0, len(text), chunk_size):
        chunks.append(text[i:i+chunk_size])
    return chunks

print(text)

text_chunks = chunk_text(text)

# Send text chunks to ChatGPT API and ask for the price
def get_price_from_gpt(text_chunks, question):
    for chunk in text_chunks:
        prompt = f"{question}\n\n{chunk}"
        response = openai.Completion.create(
            engine="text-davinci-002",
            prompt=prompt,
            max_tokens=50,
            n=1,
            stop=None,
            temperature=0.5,
        )

        answer = response.choices[0].text.strip()
        if answer.lower() != "unknown" and len(answer) > 0:
            return answer

    return "Price not found"

question = "What is the price of this object?"
price = get_price_from_gpt(text_chunks, question)
print(price)


from Use python, AutoGPT and ChatGPT to extract data from downloaded HTML page

web3.js ETH events log - How to convert javascript big number into WEI

I have a js script that used to work fine with getting events in the proper format -

receipt.events.MyEvent.returnValues.marketShare

but reading events that way seems to no longer work. So I edited my script to read the logs and use web3.eth.abi.decodeLog.

var eventAbi = [
            {
                "indexed": false,
                "internalType": "uint256",
                "name": "earnings",
                "type": "uint256"
            },
            {
                "indexed": false,
                "internalType": "uint256",
                "name": "marketShare",
                "type": "uint256"
            },
    ];

var myEvent = web3.eth.abi.decodeLog(eventAbi, receipt.logs[1].data, receipt.logs[1].topics);   

This seems to return fine, however my numbers are returning as such -

earnings: 32n
marketShare: 47691443057146912922899395050909650362856399929838832537703884369061582418996n

I need it to return the WEI or ETH value (at this point I'll take either or and convert later). So I've tried -

    console.log(myEvent.marketShare.toLocaleString());

    console.log(web3.utils.fromWei(myEvent.marketShare, "ether"));

    console.log(myEvent.marketShare.toNumber());

    console.log(myEvent.marketShare.toFixed());

    console.log(web3.utils.toBN(myEvent.marketShare));

I'm not having any luck getting the correct values, any suggestions?



from web3.js ETH events log - How to convert javascript big number into WEI

ContextVars across modules

I am completely newb on asyncio and ContextVars, I just read up what's new in 3.7 and discovered ContextVars, I struggle to understand it's usage, all I know it's helpful in coroutines, instead of using thread.local should use ContextVars. But none of the official doc and top google search result could help me truely understand its purpose.

So is convextvars shared across modules? I tried:

example.py

from contextvars import ContextVar

number = ContextVar('number', default=100)
number.set(1)

then I try to import number.py

(playground) Jamess-MacBook-Pro-2:playground jlin$ python3.7
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import example
>>> from contextvars import ContextVar
>>> number = ContextVar('number', default=200)
>>> number.get()
200

I was expecting number.get() would return 1, but obviously I have understood its purpose wrong.

Could someone please help me understand this?



from ContextVars across modules

NextAuth with Wildcard / Custom Domains

I have a NextJS application. It's a multi-tenant SaaS application.

The app provides each customer with the option to use a subdomain on our site or to map their custom domain vis CNAME.

I want to allow our customers to allow their employees to login on their subdomain site or custom domain.

export const authOptions: NextAuthOptions = {
  // Configure one or more authentication providers
  providers: [
     EMAIL PROVIDER
    // ...add more providers here
  ],
  pages: {
    signIn: `/login`,
    verifyRequest: `/verify`,
  },
  adapter: PrismaAdapter(prisma),
  callbacks: {

  },
  cookies: {
    sessionToken: {
      name: 'next-auth.session-token',
      options: {
        httpOnly: true,
        sameSite: 'lax',
        path: '/',
        domain: process.env.NODE_ENV === 'production' ? '.mysaas.com' : undefined,
        secure: process.env.NODE_ENV && process.env.NODE_ENV === 'production' ? true : false
      }
    },
    callbackUrl: {
      name: 'next-auth.callback-url',
      options: {
        sameSite: 'lax',
        path: '/',
        domain: process.env.NODE_ENV === 'production' ? '.mysaas.com' : undefined,
        secure: process.env.NODE_ENV && process.env.NODE_ENV === 'production' ? true : false
      }
    },
    csrfToken: {
      name: 'next-auth.csrf-token',
      options: {
        sameSite: 'lax',
        path: '/',
        domain: process.env.NODE_ENV === 'production' ? '.mysaas.com' : undefined,
        secure: process.env.NODE_ENV && process.env.NODE_ENV === 'production' ? true : false
      }
    }
  }  
}

export default NextAuth(authOptions)

With the above [...nextauth] file, I'm able to make it work with subdomains as I'm using '.mysaas.com' for domain cookie.

However, it doesn't work with a custom domain mapped to a subdomain? How can I achieve that?

If I can set the cookie domain dynamically so that I can dynamically set the domain to the actual domain, then it will work. Like instead of .mysaas.com, if I could set it to .mycustomdomain.com, if the login page is called from this custom domain, then the problem gets resolved.

However, I cannot find a way to set this cookie domain dynamically. Any help is appreciated.



from NextAuth with Wildcard / Custom Domains

scrapy spider working locally but resulting in 403 error when running on Zyte

The spider is setup in a way where it reads the links to scrape and finally, makes a post request, and the data is parsed.

The spider is able to collect data locally, but when deployed to ZYTE it results in the error shown below..

```
              yield scrapy.Request(
                    url=STORE_URL.format(zip_code),
                    headers=headers_1,
                    meta={"item_id": item_id, "zip_code": zip_code},
                    dont_filter=True,
                    callback=self.parse_a
                )
```
                yield scrapy.Request(
                       url=API_URL,
                       method="POST",
                       headers=headers,
                 body=json.dumps(payload(item_id,zip_code, store_id)),
                       meta={"prod_code": item_id,    "zip_code": zip_code},
                       dont_filter=True,
                       callback=self.parse)
    
 USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36'
14: 2023-06-18 03:10:58 INFO    [scrapy.extensions.telnet] Telnet console listening on 0.0.0.0:6023
15: 2023-06-18 03:10:58 INFO    [scrapy.spidermiddlewares.httperror] Ignoring response <403 https://www.homedepot.com/StoreSearchServices/v2/storesearch?address=30308&radius=50&pagesize=30>: HTTP status code is not handled or not allowed
16: 2023-06-18 03:11:04 INFO    [scrapy.spidermiddlewares.httperror] Ignoring response <403 https://www.homedepot.com/StoreSearchServices/v2/storesearch?address=2125&radius=50&pagesize=30>: HTTP status code is not handled or not allowed
17: 2023-06-18 03:11:11 INFO    [scrapy.spidermiddlewares.httperror] Ignoring response <403 https://www.homedepot.com/StoreSearchServices/v2/storesearch?address=60607&radius=50&pagesize=30>: HTTP status code is not handled or not allowed


from scrapy spider working locally but resulting in 403 error when running on Zyte

Tuesday 20 June 2023

Fitting pmdarima auto_arima to time series with missing values

I have some time series data at weekly granularity, but some weeks have NaN values.

The pmdarima auto_arima documentation says that the input time series data should not contain any np.nan or np.inf values.

It looks like the R ARIMA package allows you to fit time series with missing values: https://stats.stackexchange.com/questions/346225/fitting-arima-to-time-series-with-missing-values

So, without having to use imputed values in my training data, what are my options if I want to use auto_arima on my time series data?



from Fitting pmdarima auto_arima to time series with missing values