Deploying projects and dependencies

Deploying projects

To deploy a Scrapy project to Scrapy Cloud, navigate into the project’s folder and run:

shub deploy [TARGET]

where [TARGET] is either a project name defined in scrapinghub.yml or a numerical Scrapinghub project ID. If you have configured a default target in your scrapinghub.yml, you can leave out the parameter completely:

$ shub deploy
Packing version 3af023e-master
Deploying to Scrapy Cloud project "12345"
{"status": "ok", "project": 12345, "version": "3af023e-master", "spiders": 1}
Run your spiders at: https://app.zyte.com/p/12345/

You can also build your project’s Python egg without deploying it, and deploy your project from an existing egg:

$ shub build-egg egg_name
Writing egg to egg_name

$ shub deploy --egg egg_name --version 1.0.0
Using egg: egg_name
Deploying to Scrapy Cloud project "12345"
{"status": "ok", "project": 12345, "version": "1.0.0", "spiders": 1}
Run your spiders at: https://app.zyte.com/p/12345/

shub build-egg builds the egg of your project only. It does not build or deploy the libraries your project depends on, see Deploying dependencies for that. Unlike shub deploy, it does not need a target or an API key, because it does not talk to Scrapy Cloud.

If your project is inside a git repository, the egg is built from a copy of the working directory that leaves out anything git considers ignored (e.g. via .gitignore), so build artifacts, local secrets, and stray virtualenvs don’t end up in the deploy. Uncommitted changes to tracked files, and new untracked files that aren’t gitignored, are still included. Projects that aren’t inside a git repository deploy the working directory as-is, unchanged from before.

To deploy only the files committed to git HEAD, leaving out uncommitted changes and untracked files, use --from-head:

shub deploy --from-head

Deploying dependencies

Sometimes your project will depend on third party libraries that are not available on Scrapy Cloud. You can easily upload these by specifying a requirements file:

# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: requirements.txt

Your requirements file should also include the packages of your Scrapy Cloud stack, pinned to their stack versions, and freeze all indirect dependencies, e.g. compiled with pip-tools from a requirements.in file that lists your stack packages and your additional packages. That way, your additional packages get versions compatible with those of your stack packages. See SCP24: Missing stack requirements and SCP13: Incomplete requirements freeze.

In case you use pipenv you may also specify a Pipfile:

# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: Pipfile

In this case the Pipfile must be locked and pipenv available in the environment.

Note

To install pipenv tool, use pip install pipenv or check its documentation.

A requirements.txt file will be created out of the Pipfile, so like the requirements file above, it should include the packages of your stack.

If you use Poetry you can specify your pyproject.toml:

# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: pyproject.toml

This will use Poetry’s export command to create a requirements.txt file. For Poetry >= 2.0 this command is no longer installed by default and needs to manually added as described in the plugin’s documentation. If poetry.lock does not exist yet, it will be created during this process.

Note

Poetry is a tool for dependency management and packaging in Python.

If you declare your dependencies in setup.py (install_requires) or in the [project] table of a pyproject.toml file without a [tool.poetry] table (dependencies), you can specify that file instead:

Added in version VERSION.

scrapinghub.yml

requirements:
  file: setup.py

Those dependencies are not frozen, so prefer a requirements file as described above.

When your dependencies cannot be specified in a requirements file, e.g. because they are not publicly available, you can supply them as Python eggs:

# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: requirements.txt
  eggs:
    - privatelib.egg
    - path/to/otherlib.egg

Alternatively, if you cannot or don’t want to supply Python eggs, you can also build your own Docker image to be used on Scrapy Cloud. See Deploying custom Docker images.

Choosing a Scrapy Cloud stack

You can specify the Scrapy Cloud stack to deploy your spider to by adding a stack entry to your configuration:

# project_directory/scrapinghub.yml

projects:
  default: 12345
stack: scrapy:1.3-py3

It is also possible to define the stack per project for advanced use cases:

# project_directory/scrapinghub.yml

projects:
  default:
    id: 12345
    stack: scrapy:1.3-py3
  prod: 33333  # will use Scrapinghub's default stack

Run shub list-stacks to see the available stacks.

Projects without a configured stack use Scrapy Cloud’s default stack, which can be much older than the latest one. That is why, when shub deploy creates a new scrapinghub.yml for a project that does not use a custom Docker image, it sets stack to the latest Scrapy stack released in the scrapinghub-stack-scrapy repository, unless a default stack is already configured, e.g. in ~/.scrapinghub.yml. The stack is frozen to its release date, e.g. scrapy:2.18-20261026, so that your deploys keep using the same environment until you change it. If the list of stacks cannot be fetched, e.g. while offline, the latest stack at the time your version of shub was released is used instead. shub never adds a stack to an existing scrapinghub.yml.