Shell script remove uft8 file bom

Posted on | In
Solve special character when merge files by shell script

Solve special character when merge files by shell script

Problem encountered

When using jenkins merge the SQL scripts in Ubuntu server, the result of the sql file contains special character like below

With this special character, the sqlplus not able run the script normally which report error

1
SP2-0734: unknown command beginning "delete fr..." - rest of line ignored.

Solution

The porblme caused by utf8 file contains the bom \xEF,\xBB,\xBF, it is cause the result contains special character after sed or cat files.

We can use sed to remove replace the bom to empty for the file

1
sed -i 's/^\xEF\xBB\xBF//g' test.txt

Or use grep to find all files (include sub folders) with utf8 bom and replace the bom to empty

1
grep -r -i -l $'^\xEF\xBB\xBF' . | xargs sed -i 's/^\xEF\xBB\xBF//g'