Spark UDAF could be an option!

Another real-life use case and methods to consider when optimizing a Spark job, this time using User Defined Aggregation Functions (UDAF).