leveled_bookie:status/1

[hmmr]

This implements martinsumner#478.

When called here, it returns a map containing reportable items described in the issue above.

(dev2@127.0.0.1)12> riak_kv_vnode:vnode_status(riak_core_vnode_manager:all_index_pid(riak_kv_vnode)).
[{1347321821914426127719021955160323408745312813056,
  [{backend_status,riak_kv_leveled_backend,
                   #{ledger_cache_size => 4,
                     journal_last_compaction_result => {0,0.0},
                     get_sample_count => 0,get_body_time => 0,
                     head_sample_count => 0,head_rsp_time => 0,
                     put_sample_count => 0,put_prep_time => 0,put_ink_time => 0,
                     put_mem_time => 0}},
   {vnodeid,<<"}ÍyرYdå">>},
   {counter,140003},
   {counter_lease,150000},
   {counter_lease_size,10000},
   {counter_leasing,false}]},
...

Not all items can be available at the time of calling, in which case the backend_status will not include the corresponding keys.

In riak_control, the results will show as follows:

[hmmr]

Marking as WIP pending work on the last item ("Mean level for recent fetches").

[martinsumner]

Can we add a test to the ct tests, or perhaps just extend one of the tests in basic_SUITE.

It would be good to verify that the results returned are as expected. Some may just need to be range checked, but you can get the PIDs of the Inker/Penciller via leveled_bookie:book_returnactors/1 to absolutely verify things like the penciller cache size.

Obviously fails the coverage check at the moment as book_status/1 is never called:

|------------------------|------------|
  |                module  |  coverage  |
  |------------------------|------------|
  |        leveled_bookie  |       99%  |
  |           leveled_cdb  |      100%  |
  |         leveled_codec  |      100%  |
  |        leveled_ebloom  |      100%  |
  |          leveled_eval  |      100%  |
  |        leveled_filter  |      100%  |
  |          leveled_head  |      100%  |
  |        leveled_iclerk  |      100%  |
  |     leveled_imanifest  |      100%  |
  |         leveled_inker  |      100%  |
  |           leveled_log  |      100%  |
  |       leveled_monitor  |       97%  |
  |        leveled_pclerk  |      100%  |
  |     leveled_penciller  |      100%  |
  |     leveled_pmanifest  |      100%  |
  |          leveled_pmem  |      100%  |
  |        leveled_runner  |      100%  |
  |         leveled_setop  |      100%  |
  |           leveled_sst  |      100%  |
  |      leveled_sstblock  |      100%  |
  |        leveled_tictac  |      100%  |
  |          leveled_tree  |      100%  |
  |          leveled_util  |      100%  |
  |------------------------|------------|
  |                 total  |       99%  |
  |------------------------|------------|

There are tests in basic_SUITE that prompt journal compaction, so you cna follow the pattern to check results are expected after a compaction.

[martinsumner]

The PUT will call leveled_monitor:maybe_time/2 - so sometimes it will be timed, and sometimes not. So I'm not sure that the test will pass reliably.

I do think it should try a number of operations, so that we can check all the values change to the right order of magnitude. There are a lot of load functions, or you can add a fetch and validation of the book status into one of the other tests.

Also, what happened to reporting the journal compaction scores and results?

[martinsumner]

One other thing to note, which came up in a thread about stats - is the riak admin vnode-status command.

I think it would be worth looking at the formatting of the output of this CLI command as part of this. There's an open issue from the basho days: basho/riak_kv#343

As we have the OTP json module available, perhaps we could fix this by just pretty-printing json, so that we don't end up coupling some presentation logic in the CLI to the format of the status output.

[martinsumner]

I think you still need to increment on open_reader. I expected you needed an additional increment on cdb_roll, not to remove the open_reader counts.

Otherwise, when a store is restarted, it will simply not count all the old cdb files it starts as read-only. Also, it will not count new files created by compaction.

[martinsumner]

Some comments on some of the stats:

• recent_putgethead_counts; doesn't seem to be connected (i.e. can't be updated).
• avg_compaction_score; doesn't seem to be connected (i.e. can't be updated).
  • Would be preferable I think to avg_compaction_score_sample, as avg_compaction_score_sample is not a sample in that it is not a random slice of the journal scores (the scores are in-order so it will always show one end of the Journal only).
  • Would be better to have avg_compaction_score and max_compaction_score, drop the sample scores.
• penciller_work_backlog_status - I think this probably needs to be updated in all three case statement in leveled_penciller not just the final one. The first case would be an update of {0, false, false}, the second {WC, false, true} and the third case is OK as-is.
• journal_last_compaction_time/penciller_last_merge_time - would it be better to present this in a human readable format? Especially if we're to sue this more generally other than for riak_control.

[hmmr]

I have

• removed recent_putgethead_counts, which was a resurfaced leftover from early work (those counts exist as individual items).
• did the suggested to penciller_work_backlog_status, which will provide it with a more meaningful value than undefined.
• on compaction score, I am now reporting min_ and max_compaction_score, calculated on every avg_compaction_score_update; the sample is no longer included in the report.
• I would still keep the various times in the report as unixtime milliseconds, and only convert at the sites where they are to be presented, i.e., in Elm code in riak_control, and in the json printed by riak admin vnode-status (in the works).

[martinsumner]

When a CDB file is rolled - it should also +1 to the n_active_journal_files_update