# Deploying projects and dependencies

## Deploying projects

To deploy a Scrapy project to Scrapy Cloud, navigate into the project’s folder
and run:

```
shub deploy [TARGET]
```

where `[TARGET]` is either a project name defined in `scrapinghub.yml` or a
numerical Scrapinghub project ID. If you have configured a default target in
your `scrapinghub.yml`, you can leave out the parameter completely:

```
$ shub deploy
Packing version 3af023e-master
Deploying to Scrapy Cloud project "12345"
{"status": "ok", "project": 12345, "version": "3af023e-master", "spiders": 1}
Run your spiders at: https://app.zyte.com/p/12345/
```

You can also build your project’s Python egg without deploying it, and
deploy your project from an existing egg:

```
$ shub build-egg egg_name
Writing egg to egg_name
```

```
$ shub deploy --egg egg_name --version 1.0.0
Using egg: egg_name
Deploying to Scrapy Cloud project "12345"
{"status": "ok", "project": 12345, "version": "1.0.0", "spiders": 1}
Run your spiders at: https://app.zyte.com/p/12345/
```

`shub build-egg` builds the egg of your project only. It does not build or
deploy the libraries your project depends on, see Deploying dependencies
for that. Unlike `shub deploy`, it does not need a target or an API key,
because it does not talk to Scrapy Cloud.

If your project is inside a git repository, the egg is built from a copy of
the working directory that leaves out anything git considers ignored (e.g.
via `.gitignore`), so build artifacts, local secrets, and stray
virtualenvs don’t end up in the deploy. Uncommitted changes to tracked
files, and new untracked files that aren’t gitignored, are still included.
Projects that aren’t inside a git repository deploy the working directory
as-is, unchanged from before.

To deploy only the files committed to git `HEAD`, leaving out uncommitted
changes and untracked files, use `--from-head`:

```
shub deploy --from-head
```

## Deploying dependencies

Sometimes your project will depend on third party libraries that are not
available on Scrapy Cloud. You can easily upload these by specifying a
[requirements file](https://pip.pypa.io/en/stable/user_guide/#requirements-files):

```
# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: requirements.txt
```

Your requirements file should also include the packages of your [Scrapy Cloud
stack](https://helpdesk.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks), pinned to their stack versions, and freeze all indirect dependencies,
e.g. compiled with [pip-tools](https://pip-tools.readthedocs.io/en/stable/) from a `requirements.in` file that lists
your stack packages and your additional packages. That way, your additional
packages get versions compatible with those of your stack packages. See
[SCP24: Missing stack requirements](https://scrapy-lint.readthedocs.io/en/stable/rules/scp24.html#scp24) and [SCP13: Incomplete requirements freeze](https://scrapy-lint.readthedocs.io/en/stable/rules/scp13.html#scp13).

In case you use [pipenv](https://github.com/pypa/pipenv) you may also specify a `Pipfile`:

```
# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: Pipfile
```

In this case the `Pipfile` must be locked and `pipenv` available in the
environment.

> [!NOTE]
> To install pipenv tool, use `pip install pipenv` or check [its documentation](https://pipenv.readthedocs.io/).

A requirements.txt file will be created out of the `Pipfile`, so like the
requirements file above, it should include the packages of your stack.

If you use [Poetry](https://poetry.eustace.io/) you can specify your `pyproject.toml`:

```
# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: pyproject.toml
```

This will use Poetry’s `export` command to create a requirements.txt file. For
Poetry >= 2.0 this command is no longer installed by default and needs to manually
added as described in the
[plugin’s documentation](https://github.com/python-poetry/poetry-plugin-export).
If `poetry.lock` does not exist yet, it will be created during this process.

> [!NOTE]
> [Poetry](https://poetry.eustace.io/) is a tool for dependency management and packaging in Python.

If you declare your dependencies in `setup.py` (`install_requires`) or
in the `[project]` table of a `pyproject.toml` file without a
`[tool.poetry]` table (`dependencies`), you can specify that file instead:

> [!WARNING]
> Added in version VERSION.

`scrapinghub.yml`
```yaml
requirements:
  file: setup.py
```

Those dependencies are not frozen, so prefer a requirements file as described
above.

When your dependencies cannot be specified in a requirements file, e.g.
because they are not publicly available, you can supply them as Python eggs:

```
# project_directory/scrapinghub.yml

projects:
  default: 12345
  prod: 33333

requirements:
  file: requirements.txt
  eggs:
    - privatelib.egg
    - path/to/otherlib.egg
```

Alternatively, if you cannot or don’t want to supply Python eggs, you can also
build your own Docker image to be used on Scrapy Cloud. See
[Deploying custom Docker images](deploy-custom-image.md).

## Choosing a Scrapy Cloud stack

You can specify the [Scrapy Cloud stack](https://helpdesk.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks) to deploy your spider to by adding a
`stack` entry to your configuration:

```
# project_directory/scrapinghub.yml

projects:
  default: 12345
stack: scrapy:1.3-py3
```

It is also possible to define the stack per project for advanced use cases:

```
# project_directory/scrapinghub.yml

projects:
  default:
    id: 12345
    stack: scrapy:1.3-py3
  prod: 33333  # will use Scrapinghub's default stack
```

Run `shub list-stacks` to see the available stacks.

Projects without a configured stack use Scrapy Cloud’s default stack, which
can be much older than the latest one. That is why, when `shub deploy`
creates a new `scrapinghub.yml` for a project that does not use a
[custom Docker image](deploy-custom-image.md), it sets `stack` to the
latest Scrapy stack released in the [scrapinghub-stack-scrapy](https://github.com/scrapinghub/scrapinghub-stack-scrapy/tags) repository,
unless a default stack is already configured, e.g. in `~/.scrapinghub.yml`.
The stack is frozen to its release date, e.g. scrapy:2.18-20261026, so that
your deploys keep using the same environment until you change it. If the list
of stacks cannot be fetched, e.g. while offline, the latest stack at the time
your version of shub was released is used instead. shub never adds a stack to
an existing `scrapinghub.yml`.