Share via

Azure Artifacts Upstream Sources

  • Article

TFS 2018

Using upstream sources, you can conveniently store packages from various sources in a single feed. This includes packages you publish and those you consume from external feeds and public registries like NuGet.org, npmjs.com, Maven Central, and PyPI. Once you've enabled upstream sources, any package installed from these upstream sources, a copy will be automatically saved to your feed.

Note

To save packages from upstream sources, you must be a Collaborator or higher. See Permissions for more details.

Advantages

Enabling upstream sources offers several advantages for managing your product's dependencies within a single feed:

  • Simplicity: When you publish all your packages to a single feed, it simplifies your configuration files like NuGet.config, npmrc, or settings.xml. With just one feed in your config file, you reduce the chances of errors and bugs, streamlining your setup.

  • Determinism: your feed resolves package requests in order, resulting in more consistency when rebuilding your code.

  • Provenance: Your feed retains information about the packages it saved from upstream sources. This allows you to verify that you're using the original package and not a copy or a potentially malicious version.

  • Peace of mind: Every package installed from upstream sources is automatically saved to your feed. This means that even if the upstream source is disabled, removed, or undergoing maintenance, you can continue developing and building with confidence because you have a copy of that package in your feed.

Best practices - package consumers

To take full advantage of the benefits of upstream sources as a package consumer, follow these best practices:

Use a single feed in your config file:

In order for your feed to provide a deterministic restore, make sure that your configuration file such as nuget.config or .npmrc references only one feed with the upstream sources enabled. See the example below:

<packageSources>
  <clear />
  <add key="FabrikamFiber" value="https://pkgs.dev.azure.com/fabrikam/_packaging/FabrikamFiber/nuget/v3/index.json" />
</packageSources>

Note

NuGet compiles several configuration files to determine the complete set of options to apply. By using <clear />, you can effectively ignore all other package sources specified in higher-level configuration files.

registry=https://pkgs.dev.azure.com/fabrikam/_packaging/FabrikamFiber/npm/registry/
always-auth=true

Order your upstream sources intentionally:

If you're exclusively using public registries like nuget.org or npmjs.com, the order of your upstream sources is irrelevant. Requests to the feed follow the sequence detailed in the search order section.

However, when you're managing multiple sources, which might include a combination of feeds and public registries, each upstream source is searched in the order it's listed in the feed's configuration settings. In this case, we recommend placing the public registries first in the list of upstream sources.

In some unique scenarios, certain organizations choose to customize open-source software (OSS) packages. This could involve addressing security concerns, enhancing functionality, or meeting specific requirements that necessitate internally rebuilding the package rather than directly obtaining it from a public repository. If your organization follows this practice, it's advisable to position the upstream source containing these modified OSS packages ahead of the public package managers. This arrangement ensures the use of your organization's customized versions.

Use the suggested default view:

When you add a remote feed as an upstream source, you must select its feed's view. This enables the upstream sources to construct a set of available packages. See How upstreams construct the set of available packages for more details.

Best practices: feed owners/package publishers

To make sure your feed is easily configured as an upstream source, consider applying the following best practices:

Use the default view:

The default view for all newly created feeds is the @local view, which contains all the packages published to your feed or saved from upstream sources.

If you want to use other views such as a view for newly released package versions, you can promote your package to the @release view and then make that view available for your package consumers.

Construct a package graph:

To construct a package graph, simply connect to the feed's default view and install the package you wish to share. When the package is saved to the default view, users who want to consume it will be able to resolve the package graph and install the desired package. Packages from upstream sources are displayed based on the configured view for the corresponding upstream source.

Search order

For public package managers that support multiple feeds like NuGet and Maven, the order in which feeds are queried can sometimes be unclear or nondeterministic. For example, in NuGet, parallel queries are sent to all the feeds in the configuration file, and the responses are processed in a first-in, first-out (FIFO) manner.

Upstream sources address this non-deterministic behavior by searching the feed and its upstream sources in the following order:

  1. Packages that have been published directly to the feed.

  2. Packages that have been saved from an upstream source.

  3. Packages available from upstream sources. Each upstream source is searched in the order it's listed in the feed's configuration.

To take full advantage of the fast lookup feature, we recommend that you include only one feed in your configuration file.

Note

Searching for packages in upstream sources using the NuGet Package Explorer is not supported.

Save packages from upstream sources

When you enable upstream sources for your feed and install a package from an upstream source, an automatic copy of that package is saved to your feed. This practice enhances download performance and conserves network bandwidth, making it particularly beneficial for TFS servers in internal networks.

For instance, you can install packages directly from the upstream source using a command like npm install express. Alternatively, packages might be installed as part of a dependency resolution process. In the latter scenario, installing express would also save its dependencies, such as accepts.

Override packages from upstream sources

When enabling upstream sources, it's important to note that publishing a package version that already exists in an upstream source will not be possible. For example, if you enable the NuGet.org upstream, you won't be able to publish the Newtonsoft.Json 10.0.3 package, as that exact version already exists in NuGet.org.

If you need to publish a package version that's already present in one of your upstream sources, you must follow these steps:

  1. Disable the relevant upstream source.
  2. Publish your package.
  3. Re-enable the upstream source.

Note

Package versions are immutable. Saved packages remain in the feed even if the upstream source is disabled or removed.

Upstream sources health status

If a feed has a failing upstream source, the metadata for packages of the same protocol can no longer be refreshed. To check the health status of your upstream sources, follow these steps:

  1. Sign in to your Azure DevOps organization, and then navigate to your project.

  2. Select Artifacts, and then select your feed from the dropdown menu.

  3. Select the gear icon gear icon to navigate to your Feed settings, and then select Upstream sources.

    A screenshot showing the upstream sources last sync up status.

  4. If any failures occur, a warning message will be displayed. Clicking on the Failed status will provide additional details, including the cause of the failure and instructions on how to resolve it.

    A screenshot displaying details of the sync up failure.

Note

In the case of public registries like NuGet.org, there is a 3-6 hour delay between the time a package is pushed to the public registry and when it becomes available for download. This delay depends on job timing and data propagation. However, when the upstream source is an Azure Artifacts feed, the latency is usually no more than a few minutes.

Offline upstream sources

Upstream sources serve as a valuable safeguard for your consumers and infrastructure, shielding them from unforeseen outages. When you install a package from an upstream source, a copy of that package is saved to your feed. In case the upstream source experiences downtime, undergoes maintenance, or becomes temporarily unavailable, you can still retrieve the necessary packages from your feed and continue your development.

FAQs

Q: I can't find my package even though I can see it in one of my feed's upstreams?

A: Packages from upstream sources become available downstream as soon as they're published. However, the package will only appear in your feed's UI after it has been ingested. Ingestion is triggered when a Collaborator or higher installs the package version from upstream.

Q: What are feed views?

A: Views allow developers to selectively share a subset of package versions that have been tested and validated, excluding any packages that are still under development or haven't met the quality criteria. For more information, see What are feed views.

Q: I can't find the feed that I want to configure as an upstream source?

A: Make sure that the feed's owner is sharing a view as an upstream source.

Q: Can a user with Reader role download packages from an upstream source?

A: No. A user with Reader role in an Azure Artifacts feed can only download packages that have been saved to the feed. Packages are saved to the feed when a Collaborator, a Contributor, an Administrator, or an Owner installs those packages from upstream.

Q: What happens when a user deletes or unpublishes a package saved from an upstream source?

A: The package becomes unavailable for download from the feed, and the version number is permanently reserved. Additionally, the package will no longer be saved from the upstream source. Earlier and later versions of the package will remain unaffected.

Q: What happens when a user deprecates a package saved from an upstream source?

A: When a user deprecates a package, a warning message is added to the package's metadata, which is displayed whenever the package is viewed or installed from the feed.