Nemo Release 0.1

Mar 2, 2019 • Won Wook SONG

Nemo 0.1 is an initial release that brings several features and performance enhancements. The most visible features are the frontend support for Beam and Spark; IR DAG with execution properties, optimization passes and policies; support for loops; dynamic optimization; multiple-job submission; and various features of the execution runtime.

You can view the JIRA-generated release note here: link.

New features / Improvements

Frontend

Beam
Official Support for Apache Beam (details available on this link)
Spark

Core

Intermediate representation (IR)
Execution properties
Passes
Policies
Loop support
Dynamic optimization
Multi-job submission

Execution Runtime

Data stores
Metadata management
Inter-executor data transfer (memory/disk/gluserFS for batch, pipe for streaming)
Data encoding & decoding
Partitioners
Block manager
Data location based scheduling
Scheduler (based on different execution properties including locality, site, anti affinity, …)
RDD caching

Other Components

Web UI

Known Issues

A number of bugs & unresolved JIRA issues

Miscellaneous Fixes

Website
CI integration
Documentations

Maven Artifacts

Nemo is available in Maven Central, making it easier to link into your programs without having to build as a JAR. Use the following Maven identifiers to add it to a project:

<!-- https://mvnrepository.com/artifact/org.apache.nemo/nemo-project -->
<dependency>
   <groupId>org.apache.nemo</groupId>
   <artifactId>nemo-project</artifactId>
   <version>0.1</version>
   <type>pom</type>
</dependency>

List of Contributors

Nemo 0.1 was the work of many contributors:

Arun Lakshman R
Byung-Gon Chun
Davor Bonaci
Eunji Jeong
Geon Woo Kim
Gyewon Lee
Jae Hyeon Park
JangHo Seo
Jangho Seo
Jean-Baptiste Onofré
Jeongyoon Eo
John Yang
Joo Yeon Kim
Kenn Knowles
Markus Weimer
Minhyeok Kweun
Sanha Lee
Seonghyun Park
Soojeong Kim
Taegeon Um
Won Wook SONG
Wooyeon Lee
Yunseong Lee

RECENT POSTS