My steps:
- Build image
docker build . -t scrapy - Run a container
docker run -it -p 8050:8050 --rm scrapy - In container run scrapy project:
scrapy crawl foobar -o allobjects.json
This works locally, but on my production server I get error:
[scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 1 times): Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
Note: I'm NOT using Docker Desktop, neither can I on this server.
Dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN setx /M PATH $('C:\Users\ContainerAdministrator\miniconda3\Library\bin;C:\Users\ContainerAdministrator\miniconda3\Scripts;C:\Users\ContainerAdministrator\miniconda3;' + $Env:PATH)
RUN Invoke-WebRequest "https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Windows-x86_64.exe" -OutFile miniconda3.exe -UseBasicParsing; \
Start-Process -FilePath 'miniconda3.exe' -Wait -ArgumentList '/S', '/D=C:\Users\ContainerAdministrator\miniconda3'; \
Remove-Item .\miniconda3.exe; \
conda install -y -c conda-forge scrapy;
RUN pip install scrapy-splash
RUN pip install scrapy-user-agents
#creates root directory if not exists, then enters it
WORKDIR /root/scrapy
COPY scrapy /root/scrapy
settings.py
SPLASH_URL = 'http://localhost:8050/'
OUTPUT with command scrapy crawl foobar -o allobjects.json
2021-09-15 20:12:16 [scrapy.core.engine] INFO: Spider opened
2021-09-15 20:12:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min
)
2021-09-15 20:12:16 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2021-09-15 20:12:16 [py.warnings] WARNING: C:\Users\ContainerAdministrator\miniconda3\lib\site-packages\scrapy_splash\re
quest.py:41: ScrapyDeprecationWarning: Call to deprecated function to_native_str. Use to_unicode instead.
url = to_native_str(url)
2021-09-15 20:12:16 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36
2021-09-15 20:12:16 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36
2021-09-15 20:12:17 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 1 times): Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2021-09-15 20:12:17 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) App
leWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
2021-09-15 20:12:18 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 2 times): Connection was refused by other side: 10061: No connection
could be made because the target machine actively refused it..
2021-09-15 20:12:18 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64
) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36
2021-09-15 20:12:19 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.example.com via http://localhost:8050/execute> (failed 3 times): Connection was refused by other side: 10061: No con
nection could be made because the target machine actively refused it..
2021-09-15 20:12:19 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.example.com via http://localhost:8050/execute>
Traceback (most recent call last):
File "C:\Users\ContainerAdministrator\miniconda3\lib\site-packages\scrapy\core\downloader\middleware.py", line 45, in
process_request
return (yield download_func(request=request, spider=spider))
twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 10061: No connection could be made
because the target machine actively refused it..
2021-09-15 20:12:19 [scrapy.core.engine] INFO: Closing spider (finished)
2021-09-15 20:12:19 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 3,
'downloader/exception_type_count/twisted.internet.error.ConnectionRefusedError': 3,
'downloader/request_bytes': 4632,
'downloader/request_count': 3,
'downloader/request_method_count/POST': 3,
'elapsed_time_seconds': 3.310168,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2021, 9, 15, 18, 12, 19, 605641),
'log_count/DEBUG': 6,
'log_count/ERROR': 2,
'log_count/INFO': 10,
'log_count/WARNING': 46,
'retry/count': 2,
'retry/max_reached': 1,
'retry/reason_count/twisted.internet.error.ConnectionRefusedError': 2,
'scheduler/dequeued': 4,
'scheduler/dequeued/memory': 4,
'scheduler/enqueued': 4,
'scheduler/enqueued/memory': 4,
'splash/execute/request_count': 1,
'start_time': datetime.datetime(2021, 9, 15, 18, 12, 16, 295473)}
2021-09-15 20:12:19 [scrapy.core.engine] INFO: Spider closed (finished)
What am I missing?
I already checked here:
- Scrapy, Splash and Connection was refused by other side: 10061
- Scrapy + Splash = Connection Refused
- How to run splash using docker toolbox
UPDATE 1
I included EXPOSE 8050 in my Dockerfile, but get the same error. I tried netstat -a inside the docker container, but 8050 seems not to be in there?
C:\root\scrapy>netstat -a
Active Connections
Proto Local Address Foreign Address State
TCP 0.0.0.0:135 c60d48724046:0 LISTENING
TCP 0.0.0.0:5985 c60d48724046:0 LISTENING
TCP 0.0.0.0:47001 c60d48724046:0 LISTENING
TCP 0.0.0.0:49152 c60d48724046:0 LISTENING
TCP 0.0.0.0:49153 c60d48724046:0 LISTENING
TCP 0.0.0.0:49154 c60d48724046:0 LISTENING
TCP 0.0.0.0:49155 c60d48724046:0 LISTENING
TCP 0.0.0.0:49159 c60d48724046:0 LISTENING
TCP [::]:135 c60d48724046:0 LISTENING
TCP [::]:5985 c60d48724046:0 LISTENING
TCP [::]:47001 c60d48724046:0 LISTENING
TCP [::]:49152 c60d48724046:0 LISTENING
TCP [::]:49153 c60d48724046:0 LISTENING
TCP [::]:49154 c60d48724046:0 LISTENING
TCP [::]:49155 c60d48724046:0 LISTENING
TCP [::]:49159 c60d48724046:0 LISTENING
UDP 0.0.0.0:5353 *:*
UDP 0.0.0.0:5355 *:*
UDP 127.0.0.1:51352 *:*
UDP [::]:5353 *:*
UDP [::]:5355 *:*
from Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it
No comments:
Post a Comment