One of the biggest challenges in building integrations is getting the data from A to B.  There are several possibilities for this in the 21^st century with modern web services, direct database connections through VPNs and firewalls, email, and traditional File Transfer Protocol (FTP) being the most popular.  You would probably imagine that the majority of pasUNITY integrations occur over web services seeing as how most vendors emphasize the "power and flexibility" of their web services.  While web services are usually the pasUNITY preferred method of interfacing various systems it is not the always the method we use.  Strange as it may seem the nearly half century old FTP is still a frequently employed means of moving data today
 
To pull back the curtain on this often-misunderstood protocol keep reading…

What is FTP?

FTP is short for File Transfer Protocol.  This standard was introduced to the world as Request for Comments #114 (RFC114) in 1971.  Since then it has it has evolved several times to keep up with innovations in technology (such as X.509 certificates and Secure Shell) and to introduce new features.  This has resulted in several technologies all calling themselves FTP (or some derivation thereof) some which are true evolutions of the original standard and others being entirely new: each with different strengths and weaknesses. 

Why FTP?

So how exactly is FTP still in such wide use today? Sometimes it is chosen out of ignorance and sometimes it just happens to be the best tool for the job.
 
Despite the "power and flexibility" of modern web services, FTP is often chosen instead due to the simplicity and reliability of this decades old technology to avoid several of the challenges still encountered today.  Many products on the market today have excellent web service implementations: secure, flexible, and easy to interface.  Unfortunately, not all web services are created equally and there is a plethora of vendors out there with room for improvement: vast room for improvement.  Vendor web services are often poorly documented and support is costly.  Many vendors bolt on web services as an afterthought to check off a marketing box somewhere and as a result while they may *have* web services they are not very useable or require additional exorbitant fees to enable or develop for.  Vendor web services are often reviewed and found to be lacking in security and not trusted for exposure to external vendors or toolsets.  Sometimes, vendor web services may not exist at all.  Even today, this is still common in many accounting-centric applications.
 
For any of the reasons above FTP may provide a cost-effective and secure alternative option for moving data.
 
Unfortunately, for as many reasons as there are to select FTP, sometime it is chosen when a customer is unaware that web services are an option.  Worse is when FTP is chosen when a customer did not understand web services. 
 
There are plenty of other reason for choosing web services over FTP.  Web services are often capable of validating data as they receive it and when business rules are violated they can often throw exceptions back to the caller with nice, clean, easy to understand error messages that help end-users correct mistakes in data and resubmit on their own.  This is less valuable on source systems but of major importance when writing to a destination system.

What are the various FTP implementations?

FTP comes in many flavors.  The three primary variants of FTP are FTP, FTPS, and SFTP. 

• FTP (File Transfer Protocol) is the original standard.
• FTPS (File Transfer Protocol Secure or File Transfer Protocol over SSL) adds encryption to the transport layer of the connection using X.509 certificates (commonly called SSL certificates) and is common in Windows environments.
• SFTP (Secure File Transfer Protocol or SSH File Transfer Protocol) is an implementation of the FTP protocol over highly trusted Secure Shell technology predominantly found in Linux and UNIX environments.

The similarities are evident immediately.  They each allow for the enumeration and transfer of files to and from a server by a client device using a common set of fully documented and standardized commands over the TCP/IP protocol. 
 
The differences are more subtle.  Under the hood, each of these protocols handles security a bit differently.  The authors of the original FTP standard could not have envisioned a word of ever-evolving security vulnerabilities and criminal intent on intercepting network traffic with the intent of stealing your important credentials and data.  As such, the original FTP standard had only rudimentary support for security.  As the years went by FTPS grew out of a need to bolt on a form of security that could be enabled after the connection was established.  SFTP grew out of a desire to encrypt all communication.
 
If selected an X509 certificate must exist, be valid, not be expired, and match the host name used to connect.

What are FTP Addresses?

The FTP address is what's used in the 'Server' or 'Host" field in an FTP setup. An FTP address looks a lot like an HTTP or website adress except it uses the prefix 'ftp', such as ftp.testclient.com. You also have the option to use an IP address which is a unique string of numbers separated by periods that identify a computer. You would just need to supply the IP address for the FTP server you are working with.

Firewall

Due to the way FTP works, there can be many challenges using the FTP service on a server behind a firewall. A command channel connection is opened when a standard mode FTP client initiates a session to a server. A file transfer is requested by the client by sending a port command to the server in which then the server attempts to initiate a “data channel” connection back to the client on TCP port number 20. A typical firewall running on the client sees this data channel connection request from the server as unsolicited and drops the packets, causing the file transfer to fail.
Windows Firewall with Advanced Security in Windows Vista and Windows Server 2008 support stateful FTP, which allows it to match inbound connection requests on port 20 with previous outbound port commands from the client. However, the inbound connection requests can get blocked if you use FTP over SSL to encrypt and secure the FTP traffic which causes the firewall to no longer be able to inspect the requests. To avoid this issue, FTP supports a “passive” mode (which is further explained in the next section) in which the client initiates the data channel connection. Instead of using the port command, the client sends a PASV command on the command channel. The server responds with the TCP port number to which the client should connect to establish the data channel.
By default, the server uses an available port within the range of 1025 through 5000. You can do things such as restrict the port range used by the FTP service and then creating a firewall rule that allows FTP traffic on only those allowed port numbers in order to better secure the server.

What is Active and Passive FTP?

FTP utilizes two ports, a 'data' port and a 'command' port (also known as the control port). Traditionally these are port 21 for the command port and port 20 for the data port. However, this may vary depending on which mode you are using. For SFTP the default port number is 22 and for FTPS the default port number is 990.
FTP and FTPS allow for two types of file transfer modes, active and passive. 
In active transfer, the client chooses a port and tells the server to send data to it.  The main problem with active mode FTP actually falls on the client side. The FTP client does not make the actual connection to the data port of the server--it simply tells the server what port it is listening on and the server connects back to the specified port on the client. From the client side firewall, this appears to be an outside system initiating a connection to an internal client, something that is usually blocked.
In passive mode, the client uses the control connection to send a “PASV” command to the server. This command requests the server to "listen" on a data port and to wait for a connection. The server will then open a random port and replies back to the client with the server IP address and server port number. This solves the problem of firewalls filtering the incoming data port connection to the client from the server

SFTP and Certificates

In our pasUnity software, you are given the option to use a Certificate as a mode of authentication when using the SFTP protocol. You can create your own certificate by using a tool called PuttyGen. You would then need to convert the certificate using the same tool to a usable format. When you create the certificate using the tool you will need to provide a passphrase (keep note of this as it will be needed when setting up the SFTP step when using the Certificate authentication mode). Now you have 2 files, one public and one private certificate. You would then be able to provide the public cert to whoever needs it, for example the bank for a BAI integration.
Now when you setup the SFTP step in pasUnity you can provide the path to the private certificate and its passphrase. The people on the other end of the integration (such as the Bank) will encrypt the data using their public key that you provided and then your private key plus the passphrase will decode it. This way the other side knows for certain that you are who you say you are.

Things to consider when selecting your file transfer protocol with pasUNITY:

Since we offer all three types of file transfer protocol at pasUnity, we can help you decide which one best fits your needs.  
 
If you are concerned about security and privacy of your information, then it is not recommended that you use FTP and that you select one of the other two methods.

Conclusion

In conclusion, this was just a brief overview of the different protocols we offer here at pasUNITY along with some benefits for using each one. We hope these options can be of use to you while working with your pasUnity jobs.