When should we upgrade to Rails 4?

(Jeff Atwood) #52

Figuring out why the front page is 20% slower under Rails 4 than it currently is with Rails 3.

(Sam Saffron) #53

This may also interest @steveklabnik @tenderlove @novemberkilo and @rafaelfranca

I set up a reasonably simple process to get a “profile” instance of Discourse up.

http://meta.discourse.org/t/benchmarking-discourse-locally/9070

Using those steps, when I run in Rails 3 mode I get:

sam@ubuntu:~/Source/discourse$ ab -n 100 http://localhost:3000/  
This is ApacheBench, Version 2.3 <$Revision: 655654 $>  
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/  
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        thin  
Server Hostname:        localhost  
Server Port:            3000

Document Path:          /  
Document Length:        46714 bytes

Concurrency Level:      1  
Time taken for tests:   3.337 seconds  
Complete requests:      100  
Failed requests:        0  
Write errors:           0  
Total transferred:      4701700 bytes  
HTML transferred:       4671400 bytes  
Requests per second:    29.97 [#/sec] (mean)  
Time per request:       33.366 [ms] (mean)  
Time per request:       33.366 [ms] (mean, across all concurrent requests)  
Transfer rate:          1376.09 [Kbytes/sec] received

Connection Times (ms)  
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0  
Processing:    27   33  10.6     30      82  
Waiting:       27   33  10.6     30      82  
Total:         27   33  10.6     31      82

Percentage of the requests served within a certain time (ms)  
  50%     31
  66%     31
  75%     32
  80%     33
  90%     35
  95%     76
  98%     78
  99%     82
 100%     82 (longest request)

When I run in Rails 4 mode I get:

sam@ubuntu:~/Source/discourse$ ab -n 100 http://localhost:3000/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        thin
Server Hostname:        localhost
Server Port:            3000

Document Path:          /
Document Length:        46963 bytes

Concurrency Level:      1
Time taken for tests:   3.831 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      4739400 bytes
HTML transferred:       4696300 bytes
Requests per second:    26.10 [#/sec] (mean)
Time per request:       38.307 [ms] (mean)
Time per request:       38.307 [ms] (mean, across all concurrent requests)
Transfer rate:          1208.21 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    34   38   9.1     36      85
Waiting:       34   38   9.1     36      85
Total:         34   38   9.1     36      85

Percentage of the requests served within a certain time (ms)
  50%     36
  66%     37
  75%     37
  80%     37
  90%     38
  95%     42
  98%     83
  99%     85
 100%     85 (longest request)

This means the median request time for the front page in Rails 4 is raised from 31ms to 36ms. This is pretty significant and I would not like to move to Rails 4 until we figure out why this is happening (could be our fault) and get it fixed.

Yesterday I spent an hour profiling, I looked at flame graphs and query counts, the sad thing is that nothing really struck me as problematic. Only major oddity I found was that Rails 4 had a significantly higher in memory object count.

2 Likes
(Régis Hanol) #54

I’m seeing the same thing on the median request time for the front page:

  • 98ms for Rails 3
  • 120ms for Rails 4
(steveklabnik) #55

Bummer. :frowning: I haven’t seen a whole lot of benchmarks, but they’ve mostly been faster, not slower.

(Sam Saffron) #56

This could be a benchmarking or an upgrade issue

(Navin) #57

So I think I’ve fixed all remaining deprecation warnings.

But there is a spec failure on Rails 4 now that seems non-trivial. PostAnalyzer#cooked_document looks different under Rails3 and Rails4 - will investigate further.

2 Likes
(Navin) #58

@sam the issue appears to be with string encoding. In Rails3, Rails.cache.read returns strings encoded as UTF-8 but in Rails4, as ASCII-8BIT - so PostAnalyzer#cook is giving different results from a post with raw content containing a URL for a gif (for example). To reproduce the failure do:

 RAILS4=true bundle exec rspec spec/components/post_revisor_spec.rb:209

From several hours spent spelunking around today, this gist contains the salient bits of pry sessions (all around Oneboxer#render_from_cache) in Rails4 and Rails3 modes.

Run out of time today and figuring you might know a thing or two about the setup for Rails.cache and redis-store :wink:

Got any thoughts?


Update (28 Aug): Sorry to be the bearer of weird news but this appears to be a Heisenbug :unamused: @sam confirmed seeing this failure as well but all specs pass in Rails4 mode today. In the interim I’ve restarted my machine and done a RAILS4=true bundle install on current master and it’s gone.

3 Likes
(Sam Saffron) #59

Found another 2% of our 20% perf regression. We are slowly getting there.

1 Like
(Sam Saffron) #60

Another couple of fixes:

And

###Rails 3

---
home_page:
  50: 28
  75: 29
  90: 31
  99: 78
topic_page:
  50: 49
  75: 51
  90: 54
  99: 100
timings:
  load_rails: 2277
ruby-version: 2.0.0-p247
rails4?: false
architecture: amd64
operatingsystem: Ubuntu
kernelversion: 3.8.0
memorysize: 5.89 GB
physicalprocessorcount: 1
processor0: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
virtual: vmware

###Rails 4

---
---
home_page:
  50: 33
  75: 33
  90: 35
  99: 79
topic_page:
  50: 47
  75: 49
  90: 50
  99: 99
timings:
  load_rails: 2389
ruby-version: 2.0.0-p247
rails4?: true
architecture: amd64
operatingsystem: Ubuntu
kernelversion: 3.8.0
memorysize: 5.89 GB
physicalprocessorcount: 1
processor0: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
virtual: vmware

Turns out that one of my fixes to Discourse sped up Rails 3 quite a lot as well.

3 Likes
(Sam Saffron) #61

Update, I have been busy cleaning up all sorts of hotspots see:

I also improved the benching script to perform tests as admin as well.

Current results are:

##Rails 4

---
home_page:
  50: 27
  75: 28
  90: 29
  99: 77
topic_page:
  50: 45
  75: 47
  90: 51
  99: 94
home_page_admin:
  50: 36
  75: 37
  90: 38
  99: 85
topic_page_admin:
  50: 61
  75: 64
  90: 68
  99: 180
timings:
  load_rails: 2632
ruby-version: 2.0.0-p247
rails4?: true
architecture: amd64
operatingsystem: Ubuntu
kernelversion: 3.8.0
memorysize: 5.89 GB
physicalprocessorcount: 1
processor0: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
virtual: vmware

Rails 3

---
home_page:
  50: 25
  75: 27
  90: 29
  99: 77
topic_page:
  50: 43
  75: 44
  90: 47
  99: 94
home_page_admin:
  50: 33
  75: 34
  90: 35
  99: 86
topic_page_admin:
  50: 64
  75: 68
  90: 75
  99: 166
timings:
  load_rails: 2471
ruby-version: 2.0.0-p247
rails4?: false
architecture: amd64
operatingsystem: Ubuntu
kernelversion: 3.8.0
memorysize: 5.89 GB
physicalprocessorcount: 1
processor0: Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
virtual: vmware

Many of my optimisations were applicable to Rails 3 as well, so our overall perf is better. Most importantly though the distance between Rails 3 and 4 is closing down.

3 Likes
(Sam Saffron) #62

I think we are ready to upgrade to Rails 4.

This change was very significant.

Here is a sample bench I got just now:

###Rails 4

---
home_page:
  50: 26
  75: 27
  90: 27
  99: 76
topic_page:
  50: 41
  75: 42
  90: 44
  99: 90
home_page_admin:
  50: 36
  75: 39
  90: 44
  99: 90
topic_page_admin:
  50: 62
  75: 64
  90: 69
  99: 181
timings:
  load_rails: 2478

###Rails 3

---
home_page:
  50: 25
  75: 26
  90: 27
  99: 77
topic_page:
  50: 52
  75: 58
  90: 70
  99: 115
home_page_admin:
  50: 35
  75: 37
  90: 44
  99: 87
topic_page_admin:
  50: 63
  75: 66
  90: 69
  99: 168
timings:
  load_rails: 2531
ruby-version: 2.0.0-p247
rails4?: false

So, our home page is a smidgen slower (between 1% and say 4%) however our topic page is consistently faster depending on the test run it can be 14% faster. Our topic page is our money page, its where most the traffic ends up.

Further more, I did not spend a lot of time optimising the our actual pages, I have been mostly focused on the framework. I am pretty confident we can halve load times across the board and probably even look at far more intelligent caching mechanisms.

I am happy to give the next stable release of Rails 4 the Discourse stamp of approval.

Our next goal will be to get Discourse running on Rails master, so we can keep checking for performance regression, as opposed to attacking them when a stable version is released.

I would like to aim to move to Rails for as soon as the team release the next point release.

cc @tenderlove @rafaelfranca @steveklabnik

9 Likes
(Dave McClure) #63

Interesting.

For the saike of clarity, Is your goal be to have discourse run on the head of rails master for all instances for the undetermined future? Or would that be limited to some development environment or deployment?

(rafaelfranca) #64

Cool! I’m happy to hear that. I’ll prepare a new point release this week.

1 Like
(Sam Saffron) #65

I would like a constantly running benchmark, somewhat like http://speed.pypy.org/ that tests a bunch of stuff. Mainly the Discourse bench on a bunch of environments. Rails head, Ruby head, latest versions of all the gems we depend on. Who knows, maybe one day Rubinius and JRuby. It would have to run on constant hardware, so no VMs.

1 Like
(Sam Saffron) #66

We are now shipping with Rails 4 as the default:

We will drop rails 3 support in the next few weeks, if you want to run under Rails 3 type:

RAILS3=1 bundle exec rails s

6 Likes
(Neng Xu) #67

Great! Would like to try discourse now, so obviously a newbie. How about Ruby version? Default to Ruby 2?

I would like to run discourse under Rubinius 2.1. Anyone tried that?

Could you explain this constraint? Thanks.

(Sam Saffron) #68

Benchmarks need to be reliably reproduced, VMs have inconsistent perf. I have a machine provisioned by ninefold, will set it up over the next few weeks.

(Dan Porter) #69

Are there any stipulations or requirements for upgrading to Rails 4? I’m a few commits behind the OTB default change, will following the ‘update to latest commit and type these commands’ guide for ubuntu be sufficient for updating to rails 4, or is there some manual steps that need done?

Also, has anyone tested upgrading the rails framework on a live system, and were there any issues?

(Sam Saffron) #70

We went through the upgrade here, only thing I would possibly recommend is dumping redis cache redis-cli flushall cause the wire protocol changed in Rails 4, its not critical as the cache will expire anyway and flushing redis has other side effects.

There are no new steps needed, the standard git pull && bundle install --deployment && bundle exec rake assets:precompile should do the trick

1 Like
(Sam Saffron) #71

yes default to ruby 2, that is the best version to use. We do not work under rubinius, however I would gladly accept patches that add support for rubinius / jruby.