Hi everyone!
Here is another report about our progress for all potential clients and testers.
We have done a lot within the previous month. We have almost finished reworking the structure of our database, wrote the procedure for reports building and click saving. We completed most of the tasks we planned and now we need to unite all the elements in one script, solving some issues and eliminating bugs along the way.
We made an operating test yesterday which rendered us amazing results. I am going to tell about it a bit more.
We used our good old server for testing with the following characteristics:
Intel® Xeon™ E3-1245 Quad-Core incl. – 32 GB DDR3 RAM ECC, 2 x 3 TB 6 Gb/sec 7200 RPM HDD SATA3
ОС: Ubuntu 12.04 Precise LTS 64 bit
Apache server with ISPManager has not been optimized for our database yet.
The operation was made by a multithreaded emulator which I wrote in python. It emulates IP, ISP, Device, tokens, in general, all the parameters of clicks.
Tracker worked with a click, recorded it in the database, handled redirects (test was made with rules) and it showed the time of performing different stages of work on the lander.
The average result was the following:
Click #4160
detect tokens: 0.00010395050048828
detect camp: 0.0002138614654541
detect device: 0.026785850524902
detect operator: 0.010999202728271
save detect: 0.00013208389282227
detect load: 0.038197994232178
save click: 0.00028395652770996
function redirect: 0.00063610076904297
all time: 0.041412115097046
And that was with 100 flows, approximately 120 clicks per second! (432k per hour, 10 million per day). All the results are in seconds, thus total time = 41мs
This testing does not take the speed of the lander download into account. That means we only measured the time between click on track-URL and lander opening. Roughly speaking, this is the time difference as if you were to streamed the lander directly without any tracker. Our tracker adds 41 milliseconds which is quite long for it as the tests showed.
Then we put only “all time” and tested on different streams of traffic. Informing you in advance – we didn’t manage to “kill” our server or the tracker (my Mac’s processor failed and we needed 100 MB more for the Internet-channel).
The testing looks like this:
We measured the average response and maximum time. I will be straightforward saying that the maximum came to about 100000 clicks quite often during each of the tests, the average number shows it.
Testing results
We checked the server’s capabilities, report generation on all stages of the test (opened websites, landers, etc.). The database kept on being accessible for reading, owing to the transitional system of clicks and driver perseverance. Consequently, the reports were generated without delays and didn’t influence clicks loss (how it is in imobitrack).
The most important is that clicks loss was just 0%. Each test consisted of 100000 clicks, and all of them were delivered to the database even when we cracked down on my poor Mac violently and sent the packs as hard as we could.
At the same time, running memory was used moderately. The processor wasn’t overloading. We will provide more accurate results after the final test of release on our partners VPS to recommend our clients different tariffs according to their volume of traffic.
Besides, we are going to optimize clicks’ sheet next week (will replace user-agent with id). It will speed our redirects even further.
What’s more? Except for “ultra-fast-redirects” (how they call them), our tracker will have an ultra-fast report making system. The database had been reworked to speed up the tracker workflow. Advanced algorithms of recording are responsible for redirect’s speed, not the database structure. Apart from database optimization, we use so-called intermediate tables (google it).
The point is, there is minimal database load while recording to it. We can operate our tracker, make it generate the main reports on campaigns, offers, and landers (keep in mind, that the statistics in the tracker are not updated immediately, but once in 30-60 seconds because of clicks recording algorithms). Therefore, as a user, reports will be generated instantly
For example, you are examining the statistics of a particular campaign and return to the general statistics of the campaign (the most frequent action). You don’t have to wait. Campaign statistics are immediately generated. You can choose another campaign and keep working fluidly. iMobitrack uses a similar model, but they don’t use any intermediate tables of course. They remember the previous statistics and show it from the cache (to see the up-to-date result you need to press “refresh”).
With this optimization, we will try to touch all frequent elements of the interaction with the tracker. Minimize waiting time and simplify your work on the interface level, and others. We do not have to save up our resources as we are not Saas.
What means the most to us, is your time. We are committed on saving your time, this is our job.
Good luck to us in this complicated task, I do hope we will be able to offer the first versions of our tracker testing soon!