Share via

Streaming large files with WSE

  • Article

We have a few articles that describe how to enable large file transfers with WSE 3.0(see references section below).
The main thing to note here is that WSE by default buffers the attachments before they are sent, so when the file size is large(typically anything above 10MB can be considered as large), this can lead to high memory usage and can also cause a System.OutOfMemoryException to be thrown.
This condition worsens as the file size increases or the number of simultaneous requests increases.

The reason for this is that the entire amount of data that will be sent in the response is buffered into the Web service computer's memory. This means that the process that is serving the file needs to be able to allocate one contiguous block of memory to accommodate the buffer. If the process cannot find a large enough contiguous block it throws an System.OutOfMemoryException.
Depending on the current state of the process memory, it may be able to accommodate a fairly large buffer or may not be able to accommodate even a small buffer.

So, the best way to avoid this issue is to stream the files. There is a sample for streaming using WSE 3.0 at "WSE Installation folder\Samples\CS\QuickStart\Basic\BinaryDataMTOM" (usually this maps to C:\Program Files\Microsoft WSE\v3.0\Samples\CS\QuickStart\Basic\BinaryDataMTOM).
This requires writing a custom class that implements the IXmlSerializable interface. Additionally, you must stream the MTOM attachments by using the WriteXml method
NOTE: This sample has been updated with the hotfix https://support.microsoft.com/kb/943508 to include an implementation on how to stream from client side as well.

Note:
1) Streaming cannot be implemented when you use message-level security because the entire soap message needs to be buffered into the memory to perform the cryptographic operations. So, if you want to use security the only option is to use tansport-level security(SSL).
2) The BinaryDataMTOM sample says that the streaming can be done only on the server side and not on the client and on the client the whole file is buffered. But, there is a hotfix https://support.microsoft.com/kb/943508 that provides the streaming on client side. You would need to apply this hotfix and then, you must set the SendChunkWithoutBuffering boolean property to the true value in the WSE Web service proxy class. The sample now has a RunPutStreaming method that does this.

Scenarios:
1) Sending files from client to a webservice. (Upload)
    a) secured : No streaming support using message-level security.
    b) unsecured: Streaming is supported.
2) Sending files from webservice to client. (Download)
    a) secured:  No streaming support using message-level security.
    b) unsecured: Streaming is supported 

References:
How to: Stream Large Amounts of Data from a Web Service: https://msdn.microsoft.com/en-us/library/aa528818.aspx
How to: Enable a Web Service to Send and Receive Large Amounts of Data: https://msdn.microsoft.com/en-us/library/aa528822.aspx

Comments

  • Anonymous
    March 14, 2011
    I know this is an older article, but is buffering from the client truly possible?  Everything I read seems to contradict this article about client buffering.

  • Anonymous
    March 16, 2011
    As the support.microsoft.com/.../943508 states, you can use the sample application from WSE for streaming which does not buffer on the client side.                        Some sample applications are provided in this hotfix. After you apply this hotfix, you can find the sample applications  in the following location:          WSE Installation folderSamplesCSQuickStartBasicBinaryDataMTOM For WCF, the following articles show streaming samples which do not buffer data on the client. msdn.microsoft.com/.../ms789010.aspx msdn.microsoft.com/.../ms751463.aspx

  • Anonymous
    November 29, 2011
    The comment has been removed

  • Anonymous
    July 26, 2012
    And the answer is (a small change you can find in the sample code that's not mentioned in the KB article):        _fs = new FileStream(_fileName, FileMode.Open, FileAccess.Read);        // cast the writer to IStreamingWriter        IStreamXmlWriter streamingWriter = w as IStreamXmlWriter;        if (streamingWriter != null)        {            // here we can do the real streaming            streamingWriter.WriteStream(_fs);        }        else        {            byte[] buf = new byte[1024];            int numRead = 0;            while ((numRead = _fs.Read(buf, 0, 1024)) > 0)            {                w.WriteBase64(buf, 0, numRead);            }        }        

  • Anonymous
    October 30, 2012
    Summary: property not in MSIL, is in native code(??), or something like that Details: Since, according to both Google and Bing, this is the only thread out there that discusses SendChunkWithoutBuffering, I thought would post a related piece of supreme weirdness here. I had gotten past the various hurdles enumerated above, through unit testing, through helping a partner develop a streaming WCF client for my streaming WSE3 service (another very interesting exercise, worthy of a fairly length rant of it own). So I put together a test harness for our QA people.  The first version didn't stream, but, to better represent the way the code would be used, I put streaming into the 2nd version. Boom.  System.MissingMethodException was unhandled Message=Method not found: 'Void Microsoft.Web.Services3.WebServicesClientProtocol.set_SendChunkWithoutBuffering(Boolean)'. I look with the debugger, in the immediate window:  ...GetType().GetProperties().  The property is present on my (64 bit Windows 7) desktop, absent on the (32 bit Server 2003R2) system where I'm reproducing the problem (the QA dude's desktop was Windows XP, so I figured 2003R2 was a good candidate for a similar environment).  Verified the same in PowerShell on both systems. But I notice, as I'm listing the appdomain's assemblies, that on the systems where it's working Microsoft.Web.Services3 is in the GAC.  After fumbling a bit for the right gacutil (the 2003R2 system had only the 1.1 gacutil, which wouldn't install Services3), I got Microsoft.Web.Services into the GAC on the 2003R2 system and it started working (QA dude had the same experience). In fact, it continued to work on the 2003R2 system even after I uninstalled Services3 from the GAC.  Leading to my theory about MSIL vs. (cached?) JITted code. But who knows what it really is.  And I'm not going to dig into it any farther right now.  But it's certainly something I'm going to remember for a long time.