Use case of a multinational retail company
A multinational retail company operates in multiple countries and sells products both online and in brick-and-mortar stores. To better understand their customers' buying patterns, they need to collect and analyze data from various sources such as their point-of-sale (POS) systems, online transactions, customer feedback surveys, and social media mentions.
The challenge: Making better use of the efficiency benefits of digital transformation
To do this, the company employs an ETL-Agent, a managed file transfer tool that extracts files from various sources and transfer it into a data warehouse (A central location). The ETL-Agent is responsible for ensuring that the files are correctly transfered, and that it adheres to the company's data governance policies.
One of the company's main challenges is dealing with the sheer volume of files they collect. They need to be able to handle several gigabytes of file each day and transfer them quickly to get insights in near real-time. The ETL-Agent is designed to handle this scale and can scale up or down depending on the data load.
Another challenge is dealing with data from different sources. The ETL-Agent is responsible for transferring those files to a destination and it can be easily analyzed. For example, the ETL-Agent can pickup the files from SFTP, Google Storage, shared drive, local machine and transfer it to any of those.
The ETL-Agent is also responsible for ensuring data security. The ETL-Agent ensures that sensitive information that will be use for transferring like SFTP cridentials, SQL connection string etc. can be encoded.
Implementation
Point of Sales to SAP
The POS (Point of sale) software will export data and upload it to SFTP server. The data is usually encoded with md5 encryption.
The ETL agent gets the data from the SFTP to SAP on the OS level (Oracle Unix) and pushes the data via RFC in SAP.