[Beijing website production] Clojure language challenge Scala language, we are not afraid of memory overflow

I'm here to share with you some experience gained from the World Singles system

As early as November 2009, we began to use the Scala language. We have a long running operation that takes a large amount of data change information from the membership information database and sends it to the custom search engine in the form of XML packaging files. Mapping a large number of data tables from a database to an XML rule file is a rather complex operation. In the past, the company tried various methods, some of which were easy to use and some of which were not. I recommend Scala language to the company because of its efficiency, parallelism, type safety, convenience and applicability (especially XML has a corresponding basic data type in Scala language).

We have used Scala programming as the background release program of our products for almost two years. Most of the time, it runs very well, but when the pressure is high, it will overflow memory. After a lot of observation and testing, we slowly confirmed that (at least part of the reason) the Scala language itself is the cause of this problem. Scala is preparing to incorporate the Akka library. We have been considering migrating the program to Akka

Because we started using Clojure this year (probably last May, we began to try this language), we think we should focus on developing a Clojure version of Scala program to see how it works.

The development of the background release program of Clojure version took us about 15 hours and passed all tests. Today, we conducted a "fatigue test" to process the information of nearly 300000 members at a time. If it is a Scala program, when the amount of data processed at a time reaches 50000 copies (sometimes less), it will report a memory overflow error. The Clojure program, on the other hand, went smoothly -- so we decided to replace the Scala program with it in the next product release.

Another interesting aspect is that the Scala version has a total length of about 1000 lines (about 31000 characters). The Clojure version has only 260 lines (about 11500 characters). Even the annotations are simple (* ahem * - this is not what I want to show off, just to say that this comparison is not biased). Our comparison has not included the unit test code, because the Clojure version was written in a hurry. The code form of Clojure is very similar to Scala code, and many functions are the same -- Clojure is a more powerful Scala -- it reconstructs some auxiliary functions to make them more modular and maintainable. ( Beijing website production )

The result is that obviously, I will use the release program of Clojure version in the product and completely abandon Scala.

This is the credit of Rich Hickey (the inventor of Clojure language) and the Clojure/core development team, who created such a wonderful and applicable programming language to help people solve big problems - thank you!

label: Beijing website production High end website construction