I have 3 different formatted files that need to be imported and converted into two MySQL tables. The thought is to import into a table identical to the file format. Then export to the primary table and update, followed by a CSV dump to an FTP server. Then another server will poll the FTP and download file and update its database which is in a different format.
Challenges - One file has redundant data with some information that had been previously input updated. Some columns of the CSV have number limit that if exceeded the data would need to be substituted and a flag written to another column. The value of from one of the initial files column needs to multiplied by 1000 and converted, then the output from that table needs to a different column multiplied by 1000 for upload to the FTP server. Also there are some results in the file that are estimated so they need a flag written to another column.
The import files will be either retrieved from a FTP server or uploaded to the main database server.
Then a CSV file needs to be generated to export the revisions made for columns that exceed a certain threshold, based upon the initial file uploaded.
Another CSV needs to be generated for export truncating data in certain files and to prepare for import into a completely separate database.
One of similar projects i done was, A client have had 1.2GB Dump file. And we successfully imported that file, containing 2,73,451 Articles. Thats why I think I am more suitable then others !