Redmine本体のテストをruby/rubyのmasterブランチ(2.7)で動かすと落ちていたのでcsv gemにPull Requestを送った
言いたいこと
来月ruby 2.7リリースなのでそれまでにお手元のRailsアプリでruby 2.7 preview3を試していい感じに動くか確認しましょう
やったこと
Encoding.default_internal が設定された状態で CSV.generate に第1引数が渡された場合、encodingオプションが効いてなさそうだったのを直した。
調べたこと
stringio、この辺から Encoding.default_internal を設定している場合に StringIO#external_encoding がそれになってしまい結果かえってくる文字列のエンコーディングが変わってしまうようになった
% ruby -EUTF-8:UTF-8 -rstringio -ve 'p StringIO.new(String.new(encoding: Encoding::Big5)).external_encoding' ruby 2.7.0dev (2019-11-23T07:06:30Z master b563439274) [x86_64-linux] #<Encoding:UTF-8> % ruby -EUTF-8:UTF-8 -rstringio -ve 'p StringIO.new(String.new(encoding: Encoding::Big5)).external_encoding' ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux] #<Encoding:Big5>
この動作の影響で内部でStringIO使っているcsvが影響うけていてissueが立っており
ここで CSV#initialize の中で StringIO.new する場合はなおったんだけど
CSV.generate の第1引数に文字列渡すと、その中でも予め StringIO.new を読んでおり、そこではエンコーディングの指定してない
def generate(str=nil, **options) # add a default empty String, if none was given if str str = StringIO.new(str) str.seek(0, IO::SEEK_END) else encoding = options[:encoding] str = +"" str.force_encoding(encoding) if encoding end csv = new(str, **options) # wrap yield csv # yield for appending csv.string # return final String end
https://github.com/ruby/csv/blob/8a1f1719deb76fbced8a5cfd9f52cb86af3cc2ed/lib/csv.rb#L533-L546
CSV.generate の第1引数に文字列渡すとすでにStringIOとして渡ってくるので set_encoding が呼ばれない感じ
def initialize(data, col_sep: ",", row_sep: :auto, quote_char: '"', field_size_limit: nil, converters: nil, unconverted_fields: nil, headers: false, return_headers: false, write_headers: nil, header_converters: nil, skip_blanks: false, force_quotes: false, skip_lines: nil, liberal_parsing: false, internal_encoding: nil, external_encoding: nil, encoding: nil, nil_value: nil, empty_value: "", quote_empty: true, write_converters: nil, write_nil_value: nil, write_empty_value: "", strip: false) raise ArgumentError.new("Cannot parse nil as CSV") if data.nil? if data.is_a?(String) @io = StringIO.new(data) @io.set_encoding(encoding || data.encoding) else @io = data end @encoding = determine_encoding(encoding, internal_encoding)
https://github.com/ruby/csv/blob/8a1f1719deb76fbced8a5cfd9f52cb86af3cc2ed/lib/csv.rb#L921-L954
直し方
とりあえず encoding オプション受け取っている場合は StringIO#set_encoding を呼ぶようにしておいた
def generate(str=nil, **options) encoding = options[:encoding] # add a default empty String, if none was given if str str = StringIO.new(str) str.seek(0, IO::SEEK_END) str.set_encoding(encoding) if encoding else str = +"" str.force_encoding(encoding) if encoding end csv = new(str, **options) # wrap yield csv # yield for appending csv.string # return final String end
落ちていたログ
なんかCSVと関係なさそうなのも落ちてたので後で見直して直す
Bazaar test repository NOT FOUND. Skipping functional tests !!!
CVS test repository NOT FOUND. Skipping functional tests !!!
Filesystem test repository NOT FOUND. Skipping functional tests !!!
Git test repository NOT FOUND. Skipping functional tests !!!
Mercurial test repository NOT FOUND. Skipping functional tests !!!
Subversion test repository NOT FOUND. Skipping functional tests !!!
Git test repository NOT FOUND. Skipping integration tests !!!
(Test LDAP server not configured)
Bazaar test repository NOT FOUND. Skipping unit tests !!!
Cvs test repository NOT FOUND. Skipping unit tests !!!
Filesystem test repository NOT FOUND. Skipping unit tests !!! See doc/RUNNING_TESTS.
Git test repository NOT FOUND. Skipping unit tests !!!
Mercurial test repository NOT FOUND. Skipping unit tests !!!
Subversion test repository NOT FOUND. Skipping unit tests !!!
Bazaar test repository NOT FOUND. Skipping unit tests !!!
CVS test repository NOT FOUND. Skipping unit tests !!!
Filesystem test repository NOT FOUND. Skipping unit tests !!! See doc/RUNNING_TESTS.
Git test repository NOT FOUND. Skipping unit tests !!!
Mercurial test repository NOT FOUND. Skipping unit tests !!!
Subversion test repository NOT FOUND. Skipping unit tests !!!
Skipping LDAP tests.
Run options: --seed 61600
# Running:
..................................................................................................................................................................................................................................................................................................................SS......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S............................................................................................................................................................................................................................................................................................................................................S...............................................................................................................................................................................................................................................F
Failure:
AttachmentsControllerTest#test_thumbnail_for_pdf_should_be_png [/home/sei/src/github.com/redmine/redmine/test/functional/attachments_controller_test.rb:480]:
Expected response to be a <2XX: success>, but was a <404: Not Found>
Response body:
bin/rails test test/functional/attachments_controller_test.rb:472
.......................................................................E
Error:
IssuesControllerTest#test_index_csv_cannot_convert_should_be_replaced_big_5:
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and Big5
test/functional/issues_controller_test.rb:836:in `include?'
test/functional/issues_controller_test.rb:836:in `block in test_index_csv_cannot_convert_should_be_replaced_big_5'
test/test_helper.rb:100:in `with_settings'
test/functional/issues_controller_test.rb:820:in `test_index_csv_cannot_convert_should_be_replaced_big_5'
bin/rails test test/functional/issues_controller_test.rb:819
......................................................................................................................E
Error:
IssuesControllerTest#test_index_csv_big_5:
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and Big5
test/test_helper.rb:217:in `include?'
test/test_helper.rb:217:in `assert_include'
test/functional/issues_controller_test.rb:813:in `block in test_index_csv_big_5'
test/test_helper.rb:100:in `with_settings'
test/functional/issues_controller_test.rb:799:in `test_index_csv_big_5'
bin/rails test test/functional/issues_controller_test.rb:798
..................................................................................................S................................................................................................S..............................S.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F
Failure:
TimelogReportTest#test_csv_big_5 [/home/sei/src/github.com/redmine/redmine/test/functional/timelog_report_test.rb:297]:
--- expected
+++ actual
@@ -1,3 +1 @@
-# encoding: Big5
-# valid: true
-"\x{A5CE}\x{A4E1},2011-11-11,\x{A475}\x{AEC9}\x{C160}\x{AD70}"
+"用戶,2011-11-11,工時總計"
bin/rails test test/functional/timelog_report_test.rb:262
.F
Failure:
TimelogReportTest#test_csv_cannot_convert_should_be_replaced_big_5 [/home/sei/src/github.com/redmine/redmine/test/functional/timelog_report_test.rb:341]:
--- expected
+++ actual
@@ -1,3 +1 @@
-# encoding: Big5
-# valid: true
-"\x{A5CE}\x{A4E1},2011-11-11,\x{A475}\x{AEC9}\x{C160}\x{AD70}"
+"用戶,2011-11-11,工時總計"
bin/rails test test/functional/timelog_report_test.rb:308
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E
Error:
AttachmentTest#test_thumbnail_should_generate_the_thumbnail:
TypeError: no implicit conversion of nil into String
test/unit/attachment_test.rb:451:in `basename'
test/unit/attachment_test.rb:451:in `block (2 levels) in test_thumbnail_should_generate_the_thumbnail'
test/unit/attachment_test.rb:448:in `each'
test/unit/attachment_test.rb:448:in `block in test_thumbnail_should_generate_the_thumbnail'
test/unit/attachment_test.rb:447:in `test_thumbnail_should_generate_the_thumbnail'
bin/rails test test/unit/attachment_test.rb:434
..................................................................................................................................................................................................................................................................................................................................................................................................................................
Finished in 293.530429s, 15.6781 runs/s, 66.5417 assertions/s.
4602 runs, 19532 assertions, 3 failures, 3 errors, 7 skips
You have skipped tests. Run with --verbose for details.
