Share via

Merge vs. Replace Semantics for Update Operations

  • Article

So far Astoria has used “merge” semantics for update. That is, an “update” operation (an HTTP PUT request) replaces the values of the properties for the target entity that are specified in the input payload; this applies both to properties and links. If a property or link is not present in a PUT operation, it means “leave it with its current value”. To null-out a property or link it has to be present in the PUT body and has to have a null marker (null attribute for properties, null href for links).

While supporting merge is important and will remain part of Astoria, there are scenarios where we need “replace” semantics. That is, an operation where the entity properties are entirely replaced by those in the request body, even if the request body is partial. In particular, the AtomPub protocol requires PUT to have replace semantics.

Operation semantics

In a “replace” operation, each property in the target entity either takes the value specified in the payload (if any) or its default value. The meaning of “default value” for a “replace” operation is server implementation-specific. The Astoria server will likely use the CLR default values for each value type and null for references.

For operations on primitive values (e.g. PUT /Customers(123)/CompanyName), there is no practical difference between “merge” and “replace”.

A “replace” operation, just like a “merge” operation, cannot specify different values in the key properties. That is, keys remain non-updatable in “replace”. Similarly, a “replace” operation is subject to the same requirements from the ETags perspective as a “merge” operation.

About links inside entity payloads: “replace” operations replace the properties in entities themselves, not its links. If you include a link in a “merge” or “replace” operation we’ll wire it up, but if you don’t it’ll maintain the existing links.

About directly-addressed links: links are currently atomic values (just the link itself), so there is no difference between replacing it and merging it when operating on link resources (e.g. /$links/…). Astoria servers should support both operations but keep identical semantics. The rest of this discussion won’t touch on links as stand-alone resources anymore.

Note that these operational semantics are the same regardless of the actual format (Atom, JSON, etc.) used to represent the resources being exchanged.

Finally, this in general does not apply to service operations, as the meaning of service operations is service-defined. It’s interesting to think about whether in the Astoria server library we introduce some mechanism to allow it to expose a MERGE-enabled URL, but that would still require the user-implementation to make sense of it.

HTTP interface

AtomPub specifically requires HTTP PUT to mean replace. So we adjusted the way Astoria interprets PUT to mean “replace”.

In order to request a “merge” operation we have two options:

1. Introduce a new HTTP method, “MERGE”, and rely on verb tunneling (POST + a header) for the cases where custom methods are not allowed. There has been talk about a PATCH verb in multiple circles including the AtomPub community, but it seems to be going in a somewhat different direction.

2. Introduce a new custom header, “DataServices-Merge” or something, that when set to “1” in a PUT request indicates that the server should merge the body with the server entity instead of replacing it.

While we’re not thrilled with the idea of introducing a new HTTP method, overloading PUT with an extra header seems to be very problematic. If anything else, a server that does not support “merge” through headers would see PUT as a regular “replace” request and perform an operation that’s not what the client expected. Also other things break. For example, if a server sees an actual MERGE request and cannot handle it then it can respond with 405 – method not supported.

So we’re leaning toward MERGE and tunneling (we already support tunneling for PUT/DELETE in Astoria servers and clients).

Responses to “merge” and “replace” requests are identical.

.NET client

Using “merge” from the client has several advantages:

1. The server does not know what a client considers a “whole” entity. The client may be using entity types that contain a subset of the properties of the server-side version, either due to versioning mismatches or because the client is not interested in all of the properties.

2. The client is pretty much required to use “merge” to make links work. Since an entity might have been brought down to the client without its related entities expanded, one or more links won’t be present, and if we used replace we’d lose information on the server.

Based on the above, it would seem that the client should do “merge”, which will effectively result in “replacing the subset the client knows about” because the client always sends all the fields.

The problem now is servers that don’t implement “merge” operation.  One option is to require “merge” for the client to work, but that leaves too many interesting scenarios out. Since we already had a SaveChangesOptions enum that’s used as an argument to SaveChanges/BeginSaveChanges, we introduced SaveChangeOptions.UpdateAsReplace to indicate that you want the client to use PUT.

AJAX client

The AJAX client’s DataService.update method can be extended with a new argument “UpdateAsReplace” that enables use of replace when set to true. By default we would continue to do “merge” (this requires tweaking the library because currently DataService.update generates a PUT request).

If the new boolean adds one too many arguments for update(), alternatively we could add a knob to the DataService class, we still made this change.

Astoria runtime-data source interaction

The only change in the interaction between Astoria and the data source should be that for the resource being replaced the system will call IUpdatable.ReplaceResource instead of IUpdatable.GetResource.

Pablo Castro
Software Architect
Microsoft Corporation

This post is part of the transparent design exercise in the Astoria Team. To understand how it works and how your feedback will be used please look at this post.

Comments

  • Anonymous
    May 20, 2008
    Merge vs. Replace Semantics for Update OperationsThe Project Astoria Team is researching the semantic...
  • Anonymous
    May 20, 2008
    This is a beautiful example of sticking to a good standard... I'm interested in finding a solution for this (a good solution), so I'll start digging!
  • Anonymous
    May 20, 2008
    Glad to see that Astoria is now aligned with 2616 and AtomPub as far as PUT is concerned.Your comment about PATCH is interesting. I think your proposed MERGE is a subset of what PATCH can be used for.Referring to the PATCH I-D, PATCH just says that "a set of changes described in the request entity be applied to the resource identified by the Request-URI". The set of changes contained in the request body could very well say "merge this stuff with the current state". There is no need for a new MERGE verb for this use case. Choosing a right patch format for the particular representations should be good enough to deal with not just merge but partial updates in general.Subbu
  • Anonymous
    May 21, 2008
    I agree with Subbu; use PATCH. If the current PATCH I-D doesn't serve your needs, give input on it. Collaboarte with the editors of the I-D to move it in the direction you want. The semantics of PATCH and MERGE are so close that it would be really unfortunate to end up with both methods serving 99% of the same purpose, only completely uninteroperably.
  • Anonymous
    May 22, 2008
    Subbu/Asbjørn: we read the PATCH I-D in detail and in its current form it makes sense and it matches our current behavior almost exactly.The concern that we have is that PATCH is still in draft, and we need to close down the code base in the very short term. If we use PATCH and then there is a change in the PATCH I-D that causes our implementation to be non-compliant, we'd be in a tough spot: if we fix it to be compliant we break existing clients, and if we don't we have a non-compliant server.By using MERGE, as ugly as it is to introduce a custom method, we make sure that when PATCH comes out of the draft state we can introduce a compliant implementation.So, as you can see, we are between a rock and a hard place...any ideas that sorts this out in a day or two would be welcome :)-pablo
  • Anonymous
    May 29, 2008
    Pablo - have you considered bringing this up APP list? IMO, the I-D is stable, and is ready for adoption. I believe a number of folks (myself included) are interested in getting PATCH done. Since you are actively looking into this, it would be great if you can kick off a discussion.SincerelySubbu
  • Anonymous
    May 30, 2008
    Subbu - I wish we had enough time. Unfortunately we're closing down the code right now, so even if we get this agreed upon as soon as by next week it would be very challenging for us to make a change that's not a critical bug fix at that point.This is very unfortunate timing. I would have loved to have the product using PATCH in alignment with everybody else. Even if we don't do this now, we'll likely do it next round, probably putting "MERGE" into deprecation path quickly.I know it's not perfect, but we have to close and ship this thing at some point so folks can start to use it. There is always a next release, and I can assure you that myself and the whole Astoria team has a strong commitment to align with internet community efforts as much as possible.-pablo
  • Anonymous
    June 10, 2008
    I'd say that using a custom method without any kind of IETF review is an extremely bad idea. Don't do it.Either just use PATCH as already defined in RFC2068, or - by all means - use the generic purpose POST.Julian
  • Anonymous
    June 13, 2008
    Pablo Castro explains the options ahead for how updates might be improved with Astoria. Currently, Astoria
  • Anonymous
    August 24, 2008
    I am using ExtJS to hit ADO.NET Data Services without the AJAX Client For ADO.NET Data Services (DataService.js).Everything worked fine in SP1 Beta but when I installed SP1 RTM all the PUTs that involved foreign keys stopped working. I replaced PUT with MERGE and it now works in FF but not in IE. Any ideas?
  • Anonymous
    August 24, 2008
    Maybe XMLHTTP is not liking the custom verb. Since we expect many cases where nothing except GET/POST will work, we built verb tunneling into Data Services, you can use POST and set the x-http-method header to the method you actually meant (e.g. MERGE in this case).-pablo
  • Anonymous
    August 25, 2008
    So I would do method 'POST' but I would send 'x-http-method': 'MERGE' in the header?
  • Anonymous
    August 25, 2008
    Yep... that's what you must've meant 'cause it worked! Thnak you so much...
  • Anonymous
    September 26, 2008
    For V1 of ADO.NET Data Services (aka Project Astoria) we introduced a process we called Transparent Design
  • Anonymous
    September 26, 2009
    Definitely hoping you look to use PATCH in the future.
  • Anonymous
    August 23, 2010
    Great blog...Supporting merge is important and will remain part of Astoria, there are scenarios where we need “replace” semantics. That is, an operation where the entity properties are entirely replaced by those in the request body, even if the request body is partial. In particular, the AtomPub protocol requires PUT to have replace semantics. thanks for sharing nice information...
  • Anonymous
    December 12, 2010
    i came to your blog instantly and amazed to see that its full of info what i was searching for
  • Anonymous
    February 03, 2011
    I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.  <a href="http://www.ciplex.com/" mce_href="http://www.ciplex.com/" rel="follow">Web Design</a>