Recently we've experienced a large surge in the size of our data and it's called into question some of our querying approaches and node configuration.
We had been using
2i for most of our querying and a combination of "
Data Point Objects" and MapReduce for our more analytical needs. However when our MapReduce started bombing we questioned/reviewed our querying approach.
(for the record, it was bombing due to pref_list exhausted, currently resolved. more on this in a subsequent post)
It didn't take long to find posts like this on Google:
Be aware that key filters are just a thin layer on top of full-bucket key listings. You'd be better off storing the field you want to filter in a secondary index, which more efficiently supports range queries (note that only the LevelDB storage engine currently supports secondary indexes). Barring that, you could use the special "$key" index for a range query on the key. - Sean Cribbs (http://riak-users.197444.n3.nabble.com/preflist-exhausted-error-td3935922.html)
Key Filters are the backbone of our querying approach. We were/are under the impression this is the best way to get a subset of the data for MapReduce. Since most of our queries are over date ranges all of our keys are prefixed with
yyyymmddhhmmss. This allows us to use date based
Key Filters in our Map Reduce. According to the post however, we should be using 2i and the special
$key field for performance reasons.
First of all, what is
$key??? Here is what the
Riak Handbook has to say about the
$key field:
There are special field names at your disposal too, namely the field $key, which automatically indexes the key of the Riak object. Saves you the trouble of specifying it twice. Riak automatically indexes the key as a binary field for your convenience ...
So I put it to the test. For the record we are using Ruby and Rails and the
Ripple(ODM) with
Ripplr (RiakSearch). I opened a rails console and ran my Key Filter based MapReduce. It returned in 31,830 rows in a little over a minute on average. I then queried the same bucket using 2i and $key with the same parameters with no Reduce phase. It ran for
over 3 minutes before I canceled it! For a sanity check I shrunk the range to two months that I knew had much less data ... it returned results over a minute later!
The code for our Key Filter:
The code for the comparable 2i range query:
So my question to all of you is why is 2i/$key being recommended over Key Filters? What am I missing here?
As a side note ... Riak handled an extraordinary spike in load like a champ writing and reading data. Our concerns strictly revolve around how best to query that data now that we have it.
To demonstrate, Nyan created 4 teams and had each team play "The Dot Game". The game has the team divided up into 8 roles and the team measures how fast they can assemble the "product". The demonstration showed that adding more work in progress only resulted in less being delivered. Nyan then changed the rules of the game such that there was less work in progress by requiring each role to only work on one product at a time and repeated the exercise. Each of the 4 teams saw an average of 8x improvement in Cycle Time, a huge improvement in quality, and increase in the amount of product produced.
Several stations were assembled, each with their own mouth watering trays of bacon and selection of toppings. 350lbs of bacon were consumed in a very short amount of time and no heart attacks were reported. Thanks to Josh Walsh and Designing Interactive for coming up with this great idea and sponsoring the activity!!
I was however surprised that Duct Tape beat Bacon 34-29 in the first round of Manifest's MashMadness. Duct Tape even went on to beat Gandolf the Grey in the championship round. Gonzaga??
Riak recently introduced Secondary Indexes (a.k.a. 2i). 2i allows objects in Riak to be stored with additional queryable values. At Validas we have leveraged 2i for validating uniqueness instead of relying on MapReduce or RiakSearch to perform the check, among other things. This worked great for us in our local dev environment.