Large File Transfer in B2B – Choking with a Risk of Asphyxiation

It was almost 10 years ago when the file sharing revolution started by Napster first began to surge.  Napster enabled the everyday consumer to simply and conveniently share MP3s with other end-users throughout the world.   In recent years digital media models have evolved as peer-to-peer exchange has given favor to centralized, commercial sites such as Apple’s iTunes.  Download sizes have increased dramatically as well.  Full-length movies can be now just as easily downloaded as individual music tracks.  The key drivers have been declining storage prices and nearly ubiquitous broadband connections, which continue to enable better performance and capacity for large file transfer.  Interestingly, however, this file sharing phenomenon has not developed as rapidly in the business community.  In fact, I would argue that the corporate world is 10 years behind consumers in its ability to share large files.

Growing Demand Large File Transfer

This is a growing problem for IT professionals as there has been a substantial increase in the demand for exchanging larger files between business partners in recent years.   Examples of the types of files that need to be shared in various industries include:

• Check image files in banking
• Call detail records in telecommunications
• Satellite images in logistics
• CAD diagrams in manufacturing
• Point-of-sale data in retail
• Radiology images in health care

 Shoving a Gigabyte-Sized File through a Kilobyte-Sized Hole

Most of the infrastructure through which B2B transmissions occur was designed for payloads on the order of kilobytes.  However, today it is not uncommon to see file transfers on the orders of multi-gigabytes.  These large file transfers in the business- to-business context create significant issues, particularly if the receiving entities are not expecting such large payloads.  For example, suppose a company is hosting a general-purpose B2B gateway to exchange XML and EDI documents with its trading partners.  However, one day a key customer decides to consolidate a large batch of messages into one file along with associated images attachments.  This multi-gigabyte file transfer can wreak havoc with a company’s B2B integration infrastructure if it was not appropriately designed.  For example successive, unexpected large file transfers might lead to:

• Disks become full if capacity planning did not account for the storage of very large files
• Firewall sessions timing out over the unusually long period required for transmission
• Inability to service other connection requests, in effect creating a denial of service scenario

Choking with a Risk of Asphyxiation

The irony is that from my investigations there is rarely a reason why the files need to be this large.  Using simple free utilities such as HJsplit, these files could be parsed easily into smaller parts for transmission purposes.  Unfortunately, suggesting such a technique to a customer is tantamount to political suicide.  Who wants to be the IT manager that goes to the sales team with a message:

“Can you please ask your customers to send smaller files, because they are choking our infrastructure?” 

While most technology practitioners would appreciate the dilemma, business users (who are accustomed to downloading 4GB movies at home) would probably laugh at them.  The reality is that it would be cheaper and simpler in many cases to copy files on a memory stick and physically transport them to a business partner than to exchange them electronically.  The whole scenario reminds of me something Scott Adams might depict in a Dilbert cartoon. 

More thoughts in an upcoming post…