Skip to content
Paquetier

Upstream Mirrors

A repository can mirror an upstream package repository. Paquetier copies what changed upstream into the repository’s own storage and serves it under /cran/{slug} like any other repository, so R installs from your storage and never reaches the upstream. The whole sync runs inside Paquetier: no rsync job, cron container or extra tool is needed next to it.

Mirrored packages are imported, not uploaded, so they count against no package or repository limit.

Supported Upstreams

Upstream Examples
Repository URL CRAN (https://cloud.r-project.org), a CRAN mirror, R-universe (https://<user>.r-universe.dev), another Paquetier (https://<host>/cran/<slug>)
Storage connection A prefix in a bucket that holds a CRAN layout, read the way a scan reads it

A repository URL is the one R would use in options(repos = ...). Mirroring from a storage connection takes the connection’s owner or an admin.

Setting Up a Mirror

  1. Open the repository and go to Settings.
  2. Under Upstream mirror, fill in the settings below.
  3. Click Set up mirror, then Sync now to start the first sync instead of waiting for the schedule.
Setting Description
Upstream A repository URL (HTTP or HTTPS) or a storage connection
Repository URL The upstream’s root, such as https://cloud.r-project.org
Storage connection, Prefix The connection and the directory that holds src/contrib; an empty prefix is the bucket root
Index paths One per line: src/contrib for sources, bin/<platform>/contrib/<R minor> for binaries, such as bin/windows/contrib/4.5 or bin/macosx/big-sur-arm64/contrib/4.5
Packages to mirror Selected packages and what they need to install, or everything in the upstream index
Packages The packages to start from, separated by commas or new lines, at most 2000
Copy older releases on the first sync Also copies each package’s upstream Archive/ once
Sync on a schedule Checks the upstream regularly and syncs when it changed
Check every (minutes) How often the upstream is checked, at least 5 minutes, 60 by default
Parallel downloads Files copied at once, 8 by default, at most 64
Requests per second A cap on requests to a repository URL; 0 is unlimited

Configuring or removing a mirror takes a repository owner or an admin. Anyone with write access can see its settings and start a sync.

Each index path keeps its layout under the repository, so R finds it under /cran/{slug}/src/contrib and /cran/{slug}/bin/... without any extra configuration. Linux binary layouts such as Posit Package Manager’s __linux__/<distro>/latest/src/contrib are not supported yet.

Choosing What to Mirror

With Selected packages and what they need to install, Paquetier starts from the packages you list and follows their Depends, Imports and LinkingTo in the upstream index until nothing new turns up. Suggests is left out, since following it pulls in a large part of CRAN. A dependency the upstream does not have and that is not part of R is listed under missing_dependencies in the sync’s result.

With Everything in the upstream index, the whole index is copied. All of CRAN’s sources are about 22,000 packages and several GB, and its binaries are much larger, so start with a list unless the repository really is meant to be a full mirror.

How a Sync Works

For each index path, in order:

  • Check: the upstream’s PACKAGES.gz (or PACKAGES) is checked against what the last sync saw. An unchanged upstream costs one request per index and copies nothing.
  • Compare: every package in scope is compared by name, version and MD5 checksum with what the repository already holds. Only new or changed files are downloaded.
  • Copy: each file streams from the upstream straight into the repository’s storage and is checked against the upstream’s MD5 checksum. A file that does not match is discarded. Failed downloads are retried with backoff.
  • Archive: a release that drops out of the upstream index moves to Archive/<package>/, as CRAN moves it, instead of being deleted. An upstream index with nothing in scope archives nothing, so a broken upstream cannot empty a mirror.
  • Publish: once every file is in place, the repository’s PACKAGES indexes are rewritten, so R never sees a package whose file is still on its way.

A sync shows up in the repository’s activity with live progress, like a scan. Only one sync runs per repository at a time. If the instance running it stops, another instance picks it up, and the new run skips every file that was already copied.

When the sync finishes, Paquetier reads authors, URLs and bug report links from the copied packages, since CRAN’s PACKAGES does not carry them.

Schedule and Failures

With Sync on a schedule, Paquetier checks the upstream at the chosen interval and starts a sync only when an index changed, the mirror’s settings changed or the last sync failed. A mirror in step with its upstream leaves nothing in the activity feed.

If the upstream index cannot be read, the sync fails and the repository keeps serving what it had. If some files fail to download, the rest are still published, the mirror’s Last error says how many failed, and the next check starts a sync that fetches only what is still missing.

What a Mirror Owns

A mirror owns the files under its index paths. It never overwrites a package that was uploaded: an upload with the same file name is kept and the sync reports it as a failure. Removing a mirror stops syncing and leaves everything it copied in the repository.

API

Method Path Description
GET /api/v1/repos/{slug}/mirror The mirror’s settings and the state of its last sync
PUT /api/v1/repos/{slug}/mirror Create or change the mirror
DELETE /api/v1/repos/{slug}/mirror Remove the mirror
POST /api/v1/repos/{slug}/mirror/sync Start a sync; returns a job_id whose progress streams at /api/v1/jobs/{id}/stream
# mirror cli, dplyr and their dependencies from CRAN, as sources and Windows binaries for R 4.5
curl -X PUT https://paquetier.example.com/api/v1/repos/cran/mirror \
  -H "Authorization: Bearer $PAQUETIER_TOKEN" -H 'Content-Type: application/json' \
  -d '{"upstream_type": "http", "upstream_url": "https://cloud.r-project.org",
    "paths": ["src/contrib", "bin/windows/contrib/4.5"],
    "scope": "list", "packages": ["cli", "dplyr"], "enabled": true}'
# start a sync now instead of waiting for the schedule
curl -X POST https://paquetier.example.com/api/v1/repos/cran/mirror/sync \
  -H "Authorization: Bearer $PAQUETIER_TOKEN"

The request fields are upstream_type (http or storage), upstream_url, storage_connection_id, upstream_prefix, paths, scope (list or all), packages, backfill_archive, enabled, sync_interval_seconds, concurrency and rate_limit.

Private Networks

By default a repository URL may not resolve to a private, loopback, link-local or carrier-grade NAT address, since the server fetches whatever URL a repository owner enters. To mirror a Paquetier or CRAN mirror on your own network, an administrator sets PAQUETIER_MIRROR_ALLOW_PRIVATE_UPSTREAMS (see Configuration).