Substrait Modifications in Gluten

Substrait is a project aiming to create a well-defined, cross-language specification for data compute operations. Since it is still under active development, there are some lacking representations for Gluten needed computing operations. At the same time, some existing representations need to be modified a bit to satisfy the needs of computing.

In Gluten, the base version of Substrait is v0.23.0. This page records all the Gluten changes to Substrait proto files for reference. It is preferred to upstream these changes to Substrait, but for those cannot be upstreamed, alternatives like AdvancedExtension could be considered.

Modifications to algebra.proto

  • Added JsonReadOptions and TextReadOptions in FileOrFiles(#1584).
  • Changed join type JOIN_TYPE_SEMI to JOIN_TYPE_LEFT_SEMI and JOIN_TYPE_RIGHT_SEMI(#408).
  • Added WindowRel, added column_name and window_type in WindowFunction, changed Unbounded in WindowFunction into Unbounded_Preceding and Unbounded_Following, and added WindowType(#485).
  • Added output_schema in RelRoot(#1901).
  • Added ExpandRel(#1361).
  • Added GenerateRel(#574).
  • Added PartitionColumn in LocalFiles(#2405).
  • Added WriteRel (#3690).

Modifications to type.proto

  • Added Nothing in Type(#791).
  • Added names in Struct(#1878).
  • Added PartitionColumns in NamedStruct(#320).
  • Remove PartitionColumns and add column_types in NamedStruct(#2405).

Back to top

Copyright © 2024 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. Apache Gluten, Gluten, Apache, the Apache feather logo, and the Apache Gluten project logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. All other marks mentioned may be trademarks or registered trademarks of their respective owners.

Apache Gluten is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

Privacy Policy