Big data text file to database

I need to parse all lines of a very large (~650MB) text file into table rows of a SQLite3 database. Basically, the file contents are lines of text where individual pieces of data are separated by single spaces on each line. The data format looks like this: 4287053 06218896 N 19801222 19810901 19881222 M171 \r\n As seen above, each line is terminated by carriage return and line feed. Each row should be a maximum of 53 characters in length, although I know that some rows are missing the last field (e.g., M171 ), and it's possible (although I haven't seen it after parsing 500,000+ lines) that there may be other fields missing from some rows. Similarly, I know there are some rows that appear to be repeats. The file I need to parse is located at: [login to view URL], and documentation about the zipped file contents is located at [login to view URL] The [login to view URL] file is updated every week, and the changes are cumulative. The first "field" in the text line is a number representing a US patent. Rows are group by patent number in increasing order, except that the last entries in the file will start with "RE", although they will still be 7 characters in length. Thus, the entire file will look like this: 4287053 06218896 N 19801222 19810901 19881222 M171 \r\n ... 6497914 08193028 N 19940203 20021224 20030613 ASPN \r\n ... RE44712 13288586 N 20111103 20140121 20131219 ASPN \r\n The problem: The file contains millions of lines of text. The process of parsing this file to remove duplicate lines and create/update the database seems to be taking days on the hardware I have available. I'm looking for an optimized python script to do the following as quickly as possible according to hardware constraints: - perform an initial download and processing of the file to create a database of these records. Preferably, I'd like to eliminate duplicate rows by not importing them into the SQLite database. - download the weekly file update and parse it to update the database. Updated data is not appended to the existing file. Rather, the file is completely replaced with a new file that includes any updates to any of the previously listed patents, as well as the addition of any new patents that may have issued since the previous update. Therefore, a new record could appear almost anywhere in the file. This should also be optimized to run as quickly as possible according to hardware constraints. Specs: SQLite database table looks like this: id patent_num app_num small_entity app_filing_date patent_issue_date event_date event_code 1 4287053 06218896 N 19801222 19810901 19881222 M171 id = primary key pat_num - is text app_num is text small_entity is boolean app_filing_date is a date/datetime patent_issue_date is a date/datetime event_date is a date/datetime event_code is text