Spark DataFrame 在列数超过200的时候使用 DataFrame.cache 存在bug

问题

事情的起因是在新增几百维特征之后,与 label 数据 join 之后存为 Parquet 格式的数据, 再读进来的时候, 发现数据全为空了。但是如果只 select 其中一部分特征来 join 就没问题。
然后尝试了 改变特征数据的列数,最后发现在小于等于200的时候就是完全没问题的,在超过200列的时候数据会全部变成空。然后开始各种尝试找到问题,调整executor的数量,partition的数量啊都没办法解决。尝试了很多修改之后在使用spark-shell 的时候发现将数据读进来之后 如果使用了 dataframe.cache 数据就编变成了空,初步定为跟cache有关。去掉代码中的 cache, 发现竟然正常了。得到结论是:在 Spark DataFrame 列数超过 200 列的时候, 使用 cache 会导致数据清空。

复现

启动 spark-shell

创建一个201列的 DataFrame

1
2
3
4
5
6
7
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
val values = List.tabulate(201)(x => x)
val schema = StructType(values.map(x => StructField(s"column$x", IntegerType, true)))
val row = Row.fromSeq(values)
val rdd = sc.parallelize(List(row))
val df = sqlContext.createDataFrame(rdd, schema)

看一下 cache 之前的结果

1
df.show
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|column0|column1|column2|column3|column4|column5|column6|column7|column8|column9|column10|column11|column12|column13|column14|column15|column16|column17|column18|column19|column20|column21|column22|column23|column24|column25|column26|column27|column28|column29|column30|column31|column32|column33|column34|column35|column36|column37|column38|column39|column40|column41|column42|column43|column44|column45|column46|column47|column48|column49|column50|column51|column52|column53|column54|column55|column56|column57|column58|column59|column60|column61|column62|column63|column64|column65|column66|column67|column68|column69|column70|column71|column72|column73|column74|column75|column76|column77|column78|column79|column80|column81|column82|column83|column84|column85|column86|column87|column88|column89|column90|column91|column92|column93|column94|column95|column96|column97|column98|column99|column100|column101|column102|column103|column104|column105|column106|column107|column108|column109|column110|column111|column112|column113|column114|column115|column116|column117|column118|column119|column120|column121|column122|column123|column124|column125|column126|column127|column128|column129|column130|column131|column132|column133|column134|column135|column136|column137|column138|column139|column140|column141|column142|column143|column144|column145|column146|column147|column148|column149|column150|column151|column152|column153|column154|column155|column156|column157|column158|column159|column160|column161|column162|column163|column164|column165|column166|column167|column168|column169|column170|column171|column172|column173|column174|column175|column176|column177|column178|column179|column180|column181|column182|column183|column184|column185|column186|column187|column188|column189|column190|column191|column192|column193|column194|column195|column196|column197|column198|column199|column200|
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|      0|      1|      2|      3|      4|      5|      6|      7|      8|      9|      10|      11|      12|      13|      14|      15|      16|      17|      18|      19|      20|      21|      22|      23|      24|      25|      26|      27|      28|      29|      30|      31|      32|      33|      34|      35|      36|      37|      38|      39|      40|      41|      42|      43|      44|      45|      46|      47|      48|      49|      50|      51|      52|      53|      54|      55|      56|      57|      58|      59|      60|      61|      62|      63|      64|      65|      66|      67|      68|      69|      70|      71|      72|      73|      74|      75|      76|      77|      78|      79|      80|      81|      82|      83|      84|      85|      86|      87|      88|      89|      90|      91|      92|      93|      94|      95|      96|      97|      98|      99|      100|      101|      102|      103|      104|      105|      106|      107|      108|      109|      110|      111|      112|      113|      114|      115|      116|      117|      118|      119|      120|      121|      122|      123|      124|      125|      126|      127|      128|      129|      130|      131|      132|      133|      134|      135|      136|      137|      138|      139|      140|      141|      142|      143|      144|      145|      146|      147|      148|      149|      150|      151|      152|      153|      154|      155|      156|      157|      158|      159|      160|      161|      162|      163|      164|      165|      166|      167|      168|      169|      170|      171|      172|      173|      174|      175|      176|      177|      178|      179|      180|      181|      182|      183|      184|      185|      186|      187|      188|      189|      190|      191|      192|      193|      194|      195|      196|      197|      198|      199|      200|
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+

cache 之后

1
2
df.cache
df.show
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|column0|column1|column2|column3|column4|column5|column6|column7|column8|column9|column10|column11|column12|column13|column14|column15|column16|column17|column18|column19|column20|column21|column22|column23|column24|column25|column26|column27|column28|column29|column30|column31|column32|column33|column34|column35|column36|column37|column38|column39|column40|column41|column42|column43|column44|column45|column46|column47|column48|column49|column50|column51|column52|column53|column54|column55|column56|column57|column58|column59|column60|column61|column62|column63|column64|column65|column66|column67|column68|column69|column70|column71|column72|column73|column74|column75|column76|column77|column78|column79|column80|column81|column82|column83|column84|column85|column86|column87|column88|column89|column90|column91|column92|column93|column94|column95|column96|column97|column98|column99|column100|column101|column102|column103|column104|column105|column106|column107|column108|column109|column110|column111|column112|column113|column114|column115|column116|column117|column118|column119|column120|column121|column122|column123|column124|column125|column126|column127|column128|column129|column130|column131|column132|column133|column134|column135|column136|column137|column138|column139|column140|column141|column142|column143|column144|column145|column146|column147|column148|column149|column150|column151|column152|column153|column154|column155|column156|column157|column158|column159|column160|column161|column162|column163|column164|column165|column166|column167|column168|column169|column170|column171|column172|column173|column174|column175|column176|column177|column178|column179|column180|column181|column182|column183|column184|column185|column186|column187|column188|column189|column190|column191|column192|column193|column194|column195|column196|column197|column198|column199|column200|
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+
|      0|      0|      0|      0|      0|      0|      0|      0|      0|      0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|       0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|        0|
+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+---------+

数据完全变成0了, 如果数据本身的类型是其他类型,也会对应其他的类型的 空值 null.asInstanceOf[T]

还有一个现象就是虽然 .cache 之后使用 .show 或者 .write.parquet(path) 数据会变空,但是如果进行 filter, select 或者其他操作却还是能得到正确的结果的, 因为是惰性执行的, 也就是说是在用了cache之后,在最后执行 action 的时候,如果列数还是大于 200 的,那么内容就会变成空,否则的话是能获取到正确结果的。

1
2
df.filter($"column1" === 1).count
df.select($"column1").show
res5: Long = 1

+-------+
|column1|
+-------+
|      1|
+-------+

后续

想着这肯定是 Spark 的bug啊, 怎么可能会没人提呢,就再去搜索了一下,搜到了这个
Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

之前刚发现问的时候也去网上搜了一下,由于当时连着公司的 vpn, 然后我自己的 shadowsocks 代理用不了,就使用 bing 搜索了一下,没有找到有用信息,然后今天断掉vpn之后用昨天差不多的关键字去google搜竟然能搜到 上面的那个页面了!! 看来还是得用 google 搜索啊