Hi all,

I have a script that reads the contents of a remote CSV file, iterates over the lines, and adds the data items to a database. This file has on average about 3000 lines, and therefore 3000 products.

To make a few things clear:

• I DO NOT have control over the data in the CSV file beforehand
• I DO NOT have access to / control over the manner in which this CSV file is cretaed
• The CSV file is dynamically generated once a day, from data in a MySQL database

The problem:

My script only iterates over about 1300 lines then stops, no errors, nothing. All text is enclosed in double quotes, and generally the CSV file seems correctly formatted. The weird thing is this: If I download the CSV file, open it in Notepad++ and change the encoding to UTF-8 WITHOUT BOM, upload that to a test server and run my script on THAT file, I get the FULL 3000 items and all is fine.

So, I am assuming that the people generating this file need to insert the data as UTF-8? Because I cannot control that process, I would like to know if there is a fairly simple manner in which I can apply the UTF-8 WITHOUT BOM encoding to that file, or at least read the file contents into a variable and re-encode that?

Many thanks

Originally asked by: webfac on Stack Overflow