blambert/bugreport - ADO.NET Data Services Client URL encoding bug.

Today I was working with the ADO.NET Data Services Client, performing queries against Azure Table Storage, and I discovered that there is a bug in the URL encoding of some queries.  (Though I was performing queries against Azure Table Storage, this bug is in the ADO.NET Data Services Client; so it will manifest itself in any query, against any service.  I think.  Certainly it affects Azure Table Storage.)

I was able to write this up and get it to the ADO.NET Data Services Client team today.

I am writing it up here, in my blog, to get the information out there, share a workaround that works, but needs work.

The essence of the bug is this:  The ADO.NET Data Services Client does not properly URL encode certain values.

Steps to reproduce:

We’ll add two DataEntity values, one with a RowKey of “foo%bar” and the other with a RowKey of “foo_bar”:

 context.AddObject(EntitySetName, new DataEntity()
{
    PartitionKey = "SomeValue",
    RowKey = "foo%bar"
});
context.AddObject(EntitySetName, new DataEntity()
{
    PartitionKey = "SomeValue",
    RowKey = "foo_bar"
});
context.SaveChanges();

Next, we’ll query for these two values:

foreach (DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == "foo%bar").Execute())
{
    Console.WriteLine("Query for foo%bar returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}

foreach (DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == "foo_bar").Execute())
{
    Console.WriteLine("Query for foo_bar returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}

And the output of this operation looks like this:

Query for foo_bar returned SomeValue-foo_bar
Press any key to continue . . .

QUERY FAIL. We didn’t get the “foo%bar” entry back…

If we fiddler this, we see the following requests:

https://xyz.table.core.windows.net/brian603()?$filter=RowKey%20eq%20'foo%bar'

https://xyz.table.core.windows.net/brian603()?$filter=RowKey%20eq%20'foo_bar'

And there’s the issue!  The % in “foo%bar” in the first query was not URL encoded.  You can’t have a % in a URL.  It’s a reserved character.

If we change that first query to look like this:

string value = HttpUtility.UrlEncode("foo%bar");
foreach(DataEntity dataEntry in context.CreateQuery<DataEntity>(EntitySetName).Where(q => q.RowKey == value).Execute())
{
    Console.WriteLine("Query for HttpUtility.UrlEncode(\"foo%bar\") returned "+ dataEntry.PartitionKey + "-"+ dataEntry.RowKey);
}

The fiddler traces looks like:

https://xyz.table.core.windows.net/brian604()?$filter=RowKey%20eq%20'foo%25bar'

https://xyz.table.core.windows.net/brian604()?$filter=RowKey%20eq%20'foo_bar'

And we can see that “foo%bar” is now properly URL encoded as “foo%25bar”.

The output of this operation looks like this:

Query for HttpUtility.UrlEncode("foo%bar") returned SomeValue-foo%bar
Query for foo_bar returned SomeValue-foo_bar
Press any key to continue . . .

It works.

My testing indicates that of the reserved characters, the following errors will occur:

For ! result SUCCESS
For * result SUCCESS
For ' result SUCCESS
For ( result SUCCESS
For ) result SUCCESS
For ; result SUCCESS
For : result SUCCESS
For @ result SUCCESS
For & result EXCEPTION An error occurred while processing this request.
For = result SUCCESS
For + result FAIL
For $ result SUCCESS
For , result SUCCESS
For / result SUCCESS
For ? result SUCCESS
For % result FAIL
For # result EXCEPTION An error occurred while processing this request.
For [ result SUCCESS
For ] result SUCCESS

So, it appears that & + % and # are broken.

The quick and dirty workaround is to call HttpUtility.UrlEncode(…) on every input value being compared using Where’s with values that contains these characters.

Since % is not being encoded, we have a workaround, as every input value that is URL encoded before being passed into the ADO.NET Data Services Client will pass right through.

So your Where’s would look like:

Where(q => q.X == HttpUtility.UrlEncode(x) && q.Y == HttpUtility.UrlEncode(y))

The downside of this quick and dirty workaround is that when this bug gets fixed, queries with this workaround will break, because the %’s will be double URL encoded.  So a more sophisticated workaround will be needed…

Disclaimer: I don’t work on the ADO.NET Data Services team, I’m on another team; so I am not speaking authoritatively for Microsoft in this area. I am sharing my experiences as a developer who ran into a bug, and needed to understand it, report it, and work around it.

Brian

Comments

• Anonymous
  February 26, 2009
  PingBack from http://www.clickandsolve.com/?p=15475
• Anonymous
  October 01, 2009
  I'm running into the same thing.  However, instead of encoding all of the offending arguments in the Linq queries, I'm going to try to do it in one place, namely the overridden OnStartProcessingRequest(...) method within the service.