Skip to main content

Apache Spark Is a Superior In-Memory Analytics Engine

Apache Spark is a superior in-memory analytics engine that has gained momentum in the big data and analytics industry.

"Ask the Expert" in white letters against a teal background.

Q: What's the best in-memory analytics engine? Why?

Apache Spark is one of the best in-memory analytics engines out there. It’s gained a lot of momentum in the big data and analytics industry, and I expect that momentum to continue to accelerate for several reasons.

For one thing, Apache Spark is a fast in-memory analytics engine with a complete, functional runtime library. Its performance is scalable in terms of doing in-memory analytics. It provides business value by helping organizations process data more efficiently, and it helps keep pace with the influx of data. In addition, Apache Spark reduces the time it takes to convert raw data to business insights, offering a strong competitive advantage to enterprises that use it.

The fact that Apache Spark supports the four popular languages of big data—Java*, Scala, Python and R—makes it very attractive for developers. It increases developer productivity because it provides very consumable APIs.

With Apache Spark, you can use the same APIs to process streaming data as well as historical batch data, so it provides a unified development environment for data scientists, statisticians and application developers who want to analyze data.

Analytics applications, as we know them, are very memory intensive, but Apache Spark can provide considerable performance and scalability benefits to a greater extent than competing in-memory analytics engines.

The last point I’d add is that IBM included Apache Spark as a main component of its IBM Open Data Analytics for z/OS* offering for good reason. With Apache Spark, along with its other two primary components—Anaconda and Python—IBM Open Data Analytics for z/OS can analyze data in place, gain additional business insight and preserve the security of where data originates.



Stay on top of all things tech!
View upcoming & on-demand webinars →