Share via

Upstream sources

  • Article

TFS 2017

Upstream sources enable you to use a single feed to store packages from different sources: the ones you publish and the ones you consume from feeds and public registries such as NuGet.org, npmjs.com, Maven Central, and PyPI. Once you enable an upstream source, any user connected to your feed can install a package from upstream and a copy will be saved to your feed.

Advantages

Upstream sources enable you to manage all of your product's dependencies in a single feed. Publishing all your packages to a single feed has a few benefits:

  • Simplicity: your config file such as NuGet.config, .npmrc, or settings.xml will contain only one feed so it less prone to mistakes and bugs.
  • Determinism: your feed resolves package requests in order, so rebuilding your code will be more consistent.
  • Provenance: your feed knows the provenance of the packages it saved from upstream sources, so you can verify that you are using the original package and not a copy or malicious package.
  • Peace of mind: a copy will be saved to your feed for any package installed from upstream sources. So if the upstream source is disabled, removed, or undergoing maintenance, you can still continue to develop and build because you have a copy of that package in your feed.

Best practices - package consumers

To take full advantage of the benefits of upstream sources as a package consumer, follow these best practices:

Use a single feed in your config file

In order for your feed to provide deterministic restore, it is important to ensure that your configuration file such as nuget.config or .npmrc references only one feed with the upstream sources enabled.

Example:

  • nuget.config

    <packageSources>
  <clear />
  <add key="FabrikamFiber" value="https://pkgs.dev.azure.com/fabrikam/_packaging/FabrikamFiber/nuget/v3/index.json" />
</packageSources>

Note

The <clear /> tag is required because NuGet composes several configuration files to determine the full set of options to use. <clear /> tells NuGet to ignore all other <packageSources> defined in higher-level configuration files.

  • .npmrc:

    registry=https://pkgs.dev.azure.com/fabrikam/_packaging/FabrikamFiber/npm/registry/
always-auth=true

Order your upstream sources intentionally

If you are only using public registries such as nuget.org or npmjs.com, the order of your upstream sources is irrelevant. Requests to the feed will follow the search order.

If you are using multiple sources such as a mixture of feeds and public registries, then each upstream is searched in the order it's listed in the feed's configuration settings. In this case, we recommend placing the public registries first in the list of upstream sources.

In rare cases, some organizations choose to modify OSS packages to fix security issues, to add functionality, or to satisfy requirements that the package is built from scratch internally, rather than consumed directly from the public repository. If your organization follows this pattern, place the upstream source that contains these modified OSS packages before the public package managers to ensure you use your organization's modified versions.

Use the suggested default view

When you add a remote feed as an upstream source, you must select its feed's view. This enables the upstream sources to construct a set of available packages. See complete package graphs for more details.

Best practices: feed owners/package publishers

Note

Upstream sources are not supported in public feeds.

To make sure your feed is easily configured as an upstream source, consider applying the following best practices:

Use the default view

The @local view is the default view for all newly created feeds. It contains all the packages published to your feed or saved from upstream sources.

If you want to use views to release new package versions, you can promote your package to a view such as @release and make it available to your consumers.

Construct a package graph

When a feed query its upstream source for a package, Azure Artifacts return the packages in the view that was configured for that specific upstream source. To construct a package graph, simply connect to the feed's default view and install the package you wish to share. When the package is installed correctly in the default view, users who want to consume it will be able to resolve the package graph and install the desired package.

Search order

For public package managers that support multiple feeds (NuGet and Maven), the order in which feeds are queried is sometimes unclear or non-deterministic. For example in NuGet, parallel queries are made to all the feeds in the config file, and the responses are processed first-In, first-out FIFO.

Upstream sources prevent this non-deterministic behavior by searching the feed and its upstream sources using the following order:

  1. Packages pushed to the feed.

  2. Packages saved from an upstream source.

  3. Packages available from upstream sources: each upstream is searched in the order it is listed in the feed's configuration

To take full advantage of the fast lookup feature, we recommend that you only include one feed in your config file.

Save packages from upstream sources

When you enable upstream sources for your feed, packages installed from upstream sources will be automatically saved to your feed. These packages could be installed directly from the upstream as follows npm install express or they could be installed as part of a dependency resolution (installing express would also save dependencies like accepts).

Saving packages can improve download performance and save network bandwidth especially for TFS servers in internal networks.

Override packages from upstream sources

When you enable upstream sources, you must be aware that publishing a package version that already exists in upstream will not be possible. For instance, when you enable the NuGet.org upstream, you cannot publish the Newtonsoft.Json 10.0.3 package because that same package version is already present NuGet.org.

If you must publish a package version that already exists on one of your upstream sources, you must disable that upstream source, publish your package, and then re-enable the upstream source.

Note

You can only publish a package version that wasn't previously saved from upstream. Saved packages remain in the feed even if the upstream source is disabled or removed.

Health status

If a feed has a failing upstream source, the metadata can no longer be refreshed for packages of the same protocol. To view your upstream source's health status, select the gear icon gear icon to access your Feed settings, and then select Upstream sources.

If there are any failures, a warning message will be displayed. The settings page will also indicate which one of the upstream sources is failing. Selecting the failing upstream will provide more details such as the reason of failure and instructions on how to solve it.

Screenshot showing the upstream source's health status

Note

For public registries such as NuGet.org, there is a 3-6 hour delay between when a package is pushed to the public registry and when it is available for download by your feed. This delay depends on job timing and data propagation. The is no latency when the upstream source is an Azure Artifacts feed.

Offline upstream sources

Upstream sources are a great way to protect your consumers and infrastructure from unplanned outages. When you install a package from an upstream source, a copy of that package is saved to your feed. If the upstream source is down, undergoing maintenance, or not available, you can still access the packages you need from your feed.

FAQs

Q: I cannot find my package even though I can see it in one of my feed's upstreams?

A: Packages belonging to an upstream are available downstream soon after they are published. However the package will only show up in your feed's UI once they have been ingested, which requires installing the package version for the first time in the downstream feed.

Q: What are feed views?

A: Views enable developers to only share a subset of package versions that have been tested and validated and excluding any packages that are still under development and/or did not meet the quality bar. See What are feed views for more details.

Q: I cannot find the feed that I want to configure as an upstream source?

A: Make sure that the feed's owner is sharing a view as an upstream source.

Q: Can a user with Reader role download packages from an upstream source?

A: No. A user with Reader role in an Azure Artifacts feed can only download packages that have been saved to the feed. Packages are saved to the feed when a Collaborator, a Contributor, or an Owner install those packages from upstream.

Q: What happens when a user delete or unpublish a package saved from an upstream source?

A: The package will not be available for download from the feed and the version number gets reserved permanently. The package also will no longer be saved from the upstream source. Earlier and later versions of the package will not be affected.

Q: What happens when a user deprecate a package saved from an upstream source?

A: A warning message gets added to the package's metadata and displayed whenever the package is viewed or installed from the feed.