How to change category h3 font size?

Your changes are not applied because the selector in the categories-list.scss file is more specific. If you want your styles to apply, you need to change your selector so that it either has the same specificity or higher.

Try this instead:

.category-list tbody .category h3 {
  font-size: $font-up-3;
  line-height: $line-height-medium;
  font-family: Calibre-Regular, Helvetica, Arial, sans-serif;

Also as a side note, if you want to use the font-size variables, you don’t actually need to add all of these to your theme:

$font-up-6: 2.296em;
$font-up-5: 2em;
$font-up-4: 1.7511em;
$font-up-3: 1.5157em;
$font-up-2: 1.3195em;
$font-up-1: 1.1487em; // 2^(1/5)
$font-0: 1em;
$font-down-1: 0.8706em; // 2^(-1/5)
$font-down-2: 0.7579em; // Smallest size we use based on the 1em base
$font-down-3: 0.6599em;
$font-down-4: 0.5745em;
$font-down-5: 0.5em;
$font-down-6: 0.4355em;

// Common line-heights
$line-height-small: 1;
$line-height-medium: 1.2; // Headings or large text
$line-height-large: 1.4; // Normal or small text

You only need to add this:

@import "common/foundation/variables";

and all the variables (including font-size) should be available for you to use.