OLTP Optimization Cheat Sheets for dbdevs and DBAs

[[This is Chapter 27(c) from “beta” Volume VII of the upcoming book "Development&Deployment of Multiplayer Online Games", which is currently being beta-tested. Beta-testing is intended to improve the quality of the book, and provides free e-copy of the "release" book to those who help with improving; for further details see "Book Beta Testing". All the content published during Beta Testing, is subject to change before the book is published.

To navigate through the "1st beta" of the book, you may want to use Development&Deployment of MOG: Table of Contents.]]

After we discussed some trivial and not-so-trivial aspects of DBMS execution plans in two previous posts (1,2) – we can summarize some of generic (and actually very basic) things to be kept in mind when optimizing your DB. NB: DON’T expect anything novel here, it is just a list of things which every DBA should know by heart; still – it is easy to forget about one of these things very easily. In addition, it is VERY desirable for database developers to remember at least about half of these things – but from what I’ve seen, even in 2017 such dbdevs-who-know-this-stuff, are extremely rare <very-sad-face />.

Another NB: we’re still speaking about OLTP; that is – a database with LOTS of updates (a typical OLTP DB can be anywhere between 20-80 and 80-20 split between reads and writes, while analytical ones can easily have 100x more reads than writes).

General DB Cheat Sheet for dbdevs and DBAs

Before even starting to discuss optimizations, we MUST ensure that a few very basic things are complied with:

• Every table has a primary key (PK); this is a prerequisite to having your table to comply with First Normal Form (1NF). Seriously, if you have trouble agreeing with this concept – you’ll have LOTS of trouble with relational databases in general.
• You DO write your SQL by hand (not allowing a “magical” 3^rd-party tool such as ORM to generate it for you¹).
• You DO have your SQL transactions aligned with business transactions (and consistency requirements). For example – for a bank we MUST NOT have separate transactions “take money from user account” and “add money to user account”; instead – we should have transactions such as “transfer money from user A to user B” (or “transfer money from user A to our own account” <wink />).
• You DO use prepared statements (it is necessary both to avoid injections, and to improve performance).

OLTP Optimization Cheat Sheet for dbdevs and DBAs

As it was already noted above – in spite of practice-which-is-common-for-certain-companies where database developers just write SQL, and then it comes to DBAs to make it work fast – I insist that

Database developers and DBAs have joint responsibility for the database performance.

Otherwise, it leads to developers-who-don’t-care-about-performance writing outright atrocious SQL statements – which cannot possibly be optimized. Or, looking at the same thing from a bit different angle – we can think about the “joint responsibility” statement above as of incarnation of DevOps (~=”collaboration between development and operations”).

With this in mind, here goes a short list of the points which IMNSHO both database developers and DBAs should keep in mind:

1. DO think in terms of execution plans
2. DO make 100% sure that you DO understand execution plans
3. Have I already mentioned that you DO need to understand execution plans?
   • More seriously – for any query is executed frequently enough, we have to be sure that an “Ideal Execution Plan” for such a query doesn’t have any O(N) with N being in millions of rows; in fact, as a rule of thumb for OLTP database O(N) with N>=1000 spells as “trouble” (though if it happens rarely, it may fly), and O(N) for N>=100’000 qualifies as a “disaster”. For more discussion on “Ideal Execution Plans” and big-O asymptotic – see [[TODO]] section above.
   • If we can optimize big-O asymptotic – as a Very Big Fat Rule of Thumb™, such an optimization will trump any other optimization, so optimizing big-O behaviors is the very first thing we should try
4. Indexes are the most important thing to enable “Ideal Execution Plans”. However:
   • DO understand which indexes are necessary to execute your queries, yourself
     • Sure, it is not possible to be right 100% of the time – but with practice, you can reach 80-90% accuracy, and it already tends to help a Damn Lot™
     • DO keep in mind that two indexes by (A) and by (B) are not the same as index by (A,B) (it might be crazy, but I’ve heard a claim that “we have (A) and (B) so we don’t need (A,B)” from a real-world dbdev). OTOH, see also below about b-tree index (A,B) acting as an index on (A)
   • DON’T rely on advice from magical “index advisors” (“wizards”, etc.). While they often list that-index-you-need among those recommended, I have yet to see a single “index wizard” which wouldn’t advise to create a dozen of indexes (most of them with non-key columns) where one will suffice.² With regards to “why one index is better” – see the next item.
   • DO keep in mind that maintaining indexes has its own cost (both in terms of additional load, and in terms of size – the latter including very valuable cache size(!)).
     • As a rule of thumb – that one additional index which you MAY really need to optimize big-O asymptotic of your query, is not too bad. There are dozens of not-exactly-necessary indexes – in particular, those one falling under “the law of diminishing returns” which needs to be avoided.
     • DO keep in mind that in most cases, b-tree index by (A,B) can be used pretty much as the index by (A). As a result:
       • If you have both index by (A) and another index by (A,B) – consider dropping the one by (A)
       • if you already have an index by (A), converting it to another index by (A,B)³ is often a very viable option.
     • At this level (while we’re thinking in terms of big-O optimizations) – DON’T think about stuff such as clustered indexes or non-key columns in indexes (leave this stuff to later stages and to DBAs).

5. DO avoid functions-which-can-be-replaced-with-range-restrictions, in your queries. For example – DON’T use something like WHERE YEAR(some_field) == 2017 (unless SQL compiler is unusually smart, it is very likely to cause full index scan instead of range index scan – and the difference can be like 1000x easily); use something like WHERE some_field >= ‘2017-01-01 00:00:00’ AND some_field < ‘2018-01-01 00:00:00’⁴
   • If you still happen to need a function in your index (not in an example such as above, where it can be avoided) – DO consider index on function. Keep in mind though, that even if supported by your RDBMS, such indexes usually cannot be reused for other queries, so use them even more sparingly than usual ones.
6. DO avoid indexes on columns which have different-selectivity-for-different-values-in-the-index. While quite a few modern RDBMS try to handle it (by using value-level stats for such columns) – it is cumbersome and tends to be rather fragile. Symptoms of such indexes are having some kind of ‘type’ column, with number of rows being vastly different for different ‘types’. Overall, very often such tables and indexes exhibit database design problems.⁵
7. DO avoid foreign keys. Yes, I know it is a kind of fallacy in theoretical world – but still all the seriously-loaded-DBs-I’ve-seen – avoided them for performance reasons.
   • It means that enforcing constraints (these and many others) belong to app-level. TBH, I do NOT see it as a serious practical problem (in particular, because enforcing-FK-like-constraints tends to represent only a tiny portion of the consistency-rules-to-be-enforced-by-app-level-anyway).

OLTP Optimization Cheat Sheet (DBAs only)

While I am arguing for dbdevs to avoid outright-inefficient SQL statements – I am sure that cluttering their mind with relatively-minor details would push it too far (in particular – beyong magic 7+-2 cognitive limit). As a result, I am all for further optimizations to be performed by DBAs; in other words – I see that dbdevs should enable optimizations (by providing reasonable statements to start with) and DBAs should perform those optimizations.

For DBAs, at least the following list should be kept in mind:

1. DO run RUNSTATS/ANALYZE/UPDATE STATISTICS on regular basis (normally – daily)
2. If there is a situation when that “Ideal Execution Plan” discussed above is not picked up by your SQL compiler – DO use hints to force it (this also includes de-facto hints such as adding “OR 1=0” to WHERE clause).
   • If nothing else helps – DO consider denormalization to improve performance. For a discussion of examples when denormalization can help – see, for example, [http://www.vertabelo.com/blog/technical-articles/denormalization-when-why-and-how] (unfortunately, denormalization examples are never simple, so discussing them won’t really fit into this book).
     • Keep in mind that with each denormalization, app-level gets yet another responsibility for maintaining DB consistency.
   • In extreme cases, you may be tempted to manipulate stats to achieve desired result. I’ve seen it working, but it is sooooo fragile, that it should be used as a Really Last Resort™.
3. DO consider “clustered” indexes
   • Note that “clustered” indexes can also be considered under our “C++/Java/Knuth approximations” (in case of a clustered index, we can say that our table is no longer a list, but std::map<>/TreeMap/tree, with all the other indexes referring to the table by index, that’s it).
   • Beware: costs of updating a clustered index might be significantly higher than that of the non-clustered one
   • On a plus side – pretty often, clustered index is a Good Thing™ for those tables with PK being something like time, incremental ID, etc. (i.e. for a pretty much any historical table) – and having very few indexes (this may vary). In such cases, costs of updating a clustered index may be even lower than for the non-clustered index, plus locality during range index scans will be usually significantly better.
4. DO consider non-key fields in indexes
   • The idea here is to avoid the need for reading table pages while performing range index scans; the same thing can also be observed in our “C++/Java/Knuth approximations” (and while big-O asymptotic won’t change – we will be able to avoid referring from index to table data).
   • They can help quite a bit (though not as much as improving your big-O asymptotic) – but beware of including too-long non-key fields (especially VARCHARS); this may cause index becoming too large and using too much of valuable cache space.
5. DO consider hash indexes.
   • Keep in mind that hash indexes cannot be used for Range Index Scans.
   • NB: departure from our “approximations”. Note that while in theory, complexity difference between hash index and b-tree index is a difference between O(1) and O(log(N)) – in practice, b-tree index won’t really have more than 6 levels of pages even for a billion-row table. This, in turn, means that we can say that b-tree index is merely 6x slower than hash index – so it is actually also O(1). On the other hand:
     • 6x is also considerable.
     • for a completely cached DB role of hash indexes tends to increase further (as we’re no longer completely dominated by page read costs)
   • Overall – if you won’t need to range scan (never ever) – then hash indexes DO have an advantage.
6. DO consider converting some of your statements into stored procedures
   • Personally, I don’t really like storage procedures because of them causing very severe Vendor-Lock-In (and from my experience, careful use of prepared statements allows to achieve comparable performance while being MUCH more DB-independent). Still, I do recognize stored procedures as a legitimate optimization technique (especially when it can be demonstrated that they do help performance-wise); also – with heterogeneous app-level replication (as will be discussed in [[TODO]] section below), Vendor-Lock-In won’t prevent you from using a different DB for your reporting/analytical replicas, so it won’t be that
   • Keep in mind that even for stored procedures, we DO need to use prepared statements to call them.
7. DO optimize your DB physical layout (and underlying hardware). In particular:
   • DO dedicate one completely-isolated RAID array (ideally – even the whole RAID card) to your DB logs
   • DO use BBWC RAID (or NVMe) at least for that-RAID-which-handles-DB-logs
     • No, simple SSD-sitting-on-SATA won’t provide the-best-possible-performance (though BBWC RAID over such SSDs will).
   • DO keep an eye on your DB caches.
     • DO ensure that you’re NOT double-caching (once at DB level and another time at OS level). In general – whenever possible, DO prefer DB-level caching.
     • DO try to cache your whole OLTP DB (we’ll discuss how to limit its size in [[TODO]] section below)
     • If caching your whole DB isn’t feasible – DO consider separating your caches for different tables (and then give more caching for tables such as USERS, and less caching for historical tables where it won’t do much good). In addition to improved overall caching of the all-important USERS table, this will also allow to prevent “cache poisoning” from scanning a historical table from affecting your USERS cache.
   • Wherever possible – DO use DB-level containers (even if they’re lying on top of OS files); this tends to reduce OS-level fragmentation greatly.
     • If DB-level containers are not supported by your RDBMS – DO pay special attention to the underlying file system. DO use that-filesystem-which-is-recommended-by-your-RDBMS-vendor.
   • DO consider reorganizing your tables and/or indexes (unfortunately, they do degrade with time <sad-face />). Note that the need and frequency of required reorgs heavily depends on specifics of your RDBMS.
     • While doing it – DO consider parameters such as PCTFREE or FILL FACTOR; while they’re unlikely the need to eliminate reorgs completely, they may help to make reorgs less frequent. Be ready to experiment (this is one thing which is very difficult to get right at the first attempt; on the other hand – penalties for doing it not-so-perfectly are rather mild too <phew />).
   • When populating data into a non-operational table⁶ – DO consider dropping index before populating, and creating it later (believe it or not, but it will be significantly faster⁷).

TL;DR for OLTP DB Optimization 101

This concludes our discussion on “OLTP DB Optimization 101”. In the next section, we’ll proceed to “OLTP DB Optimization 102” – including things such as app-level caches and app-level heterogeneous replication (which in turn allows to truncate our OLTP DB so it fits into 100G or so – and 100G can be fully-cached these days rather easily).

[[To Be Continued…

This concludes beta Chapter 27(c) from the upcoming book “Development and Deployment of Multiplayer Online Games (from social games to MMOFPS, with social games in between)”.

Stay tuned for beta Chapter 27(d), where we’ll continue our discussion into less-known ways for OLTP optimization, including app-level caches and heterogeneous asynchronous replicas]]

Acknowledgement

Cartoons by Sergey Gordeev from Gordeev Animation Graphics, Prague.

P.S.

Don't like this post? Criticize↯

P.P.S.

We've tried to optimize our feed for viewing in your RSS viewer. However, our pages are quite complicated, so if you see any glitches when viewing this page in your RSS viewer, please refer to our original page.