Repository navigation
Filetype detection differences on Windows #12
Description
Some of the tests have different results on Windows:
FAILED tests/typecode/test_contenttype.py::TestContentTypeComplex::test_size
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_archive_e_tar_gz_4
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_archive_file_4_26_1_diff_gz_5
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_code_c_netdb_h_44
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_code_java_appender_java_53
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_code_java_commonviewersitefactory_jad_55
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_code_java_logger_java_56
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_code_java_contenttype_java_57
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_code_python___init___py_59
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_config_defconfig_ar531x_jffs2_71
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_doc_office_glitch_erd_vsd_84
FAILED tests/typecode/test_types.py::TestFileTypesDataDriven::test_filetest_doc_office_word_doc_91
In the case of test_filetest_code_java_logger_56, different filetype, mimetypes, and file size were detected:
Expected result for test_filetest_code_java_logger_56:
filetype_file: Java source, ASCII text
mimetype_file: text/x-java
mimetype_python: text/x-java-source
filetype_pygment: Java
programming_language: Java
is_file: yes
is_regular: yes
size: 6800
is_text: yes
contains_text: yes
is_source: yes
Result for test_filetest_code_java_logger_56:
filetype_file: ASCII text, with CRLF line terminators
mimetype_file: text/plain
mimetype_python: text/x-java-source
filetype_pygment: Java
programming_language: Java
is_file: yes
is_regular: yes
size: !!int '7013'
is_text: yes
contains_text: yes
is_java_source: yes
is_source: yes
The detected types and size should be the same. The other failing tests have similar issues.
Metadata
Metadata
Assignees
Labels
No labels
Activity
JonoYang commentedon Oct 27, 2020
We use
os.path.getsizeto get the size of a file. I ranos.path.getsize('configure.bat')on Windows and Ubuntu to see if we get a difference in size. On Windows,configure.batis 4185 bytes and on Ubuntu, it is 4064 bytes. The size disrepency is due to the line ending differences between Windows and Linux. Text files on Windows end in/r/n, rather than just/nas in Linux. So for every line in a text file on windows, there is an extra byte.On a freshly cloned repo,
typecode/tests/typecode/data/contenttype/size/dir/a.txtis 2 bytes in Ubuntu and 3 bytes in Windows.pombredanne commentedon Oct 28, 2020
@JonoYang this difference would be quite a surprise!
pombredanne commentedon Oct 28, 2020
I did a quick check and the size is the same for me, given the exact same file (sha1-wise) as an input.
IMHO you must have by tripped by using different checkouts or branches on each OSes :)
JonoYang commentedon Oct 28, 2020
@pombredanne so
configure.batis 4064 bytes on windows and ubuntu for you?On a freshly cloned typecode repo:
I think github is cloning the repo and adding the crlf endings automatically to text files on Windows. The same thing is happening on the azure ci wrt TestContentTypeComplex.test_size
JonoYang commentedon Oct 29, 2020
@pombredanne Got around the size detection difference by modifying the windows azure pipeline job template to run
git config --global core.autocrlf falsebefore checking out the repo so it doesn't replace the line endings when checking out files. I also had to add a .gitattributes file that setsconfigure.bat eol=crlf. I found out that the script did not run properly when windows runs a batch script that has LF line endings rather than CRLF.There are still filetype/mimetype detection differences remaining.