Website is unreachable because of Status Code:429 Too Many Requests. What should I do?


#1

The site is http://iosre.com. The site remains unreachable after setting up fail2ban. What now? Thanks!


(Rafael dos Santos Silva) #2

The error is Status Code:429 Too Many Requests.

Try tailing the nginx logs.

cd /var/discourse
./launcher logs app

#3
Failed password for root from 123.249.0.130 port 4467 ssh2
Failed password for root from 123.249.0.130 port 4467 ssh2
Failed password for root from 123.249.0.130 port 4467 ssh2
Failed password for root from 123.249.0.130 port 4467 ssh2
Failed password for root from 123.249.0.130 port 4467 ssh2
Connection reset by 123.249.0.130 port 4467 [preauth]
Failed password for root from 123.249.0.130 port 2442 ssh2
Failed password for root from 123.249.0.130 port 2442 ssh2
Failed password for root from 123.249.0.130 port 2442 ssh2
Failed password for root from 123.249.0.130 port 2442 ssh2
Failed password for root from 123.249.0.130 port 2442 ssh2
Connection reset by 123.249.0.130 port 2442 [preauth]
Failed password for root from 123.249.0.130 port 3546 ssh2
Failed password for root from 123.249.0.130 port 3546 ssh2
Failed password for root from 123.249.0.130 port 3546 ssh2
Failed password for root from 123.249.0.130 port 3546 ssh2
Failed password for root from 123.249.0.130 port 3546 ssh2
Connection reset by 123.249.0.130 port 3546 [preauth]
Failed password for root from 123.249.0.130 port 4056 ssh2
Failed password for root from 123.249.0.130 port 4056 ssh2
Failed password for root from 123.249.0.130 port 4056 ssh2
Failed password for root from 123.249.0.130 port 4056 ssh2
Failed password for root from 123.249.0.130 port 4056 ssh2
2016-08-25 15:27:59 UTC [460-2] discourse@discourse LOG:  duration: 175.706 ms  execute <unnamed>: SELECT  "incoming_referers".* FROM "incoming_referers" WHERE "incoming_referers"."path" = '/' AND "incoming_referers"."incoming_domain_id" = 197 LIMIT 1
Connection reset by 123.249.0.130 port 4056 [preauth]
2016-08-25 15:27:59 UTC [460-3] discourse@discourse LOG:  duration: 161.061 ms  execute <unnamed>: INSERT INTO "incoming_links" ("incoming_referer_id", "post_id", "ip_address", "created_at") VALUES (653, 12924, '123.240.0.97/32', '2016-08-25 15:27:59.736731') RETURNING "id"
Failed password for root from 123.249.0.130 port 2168 ssh2
Failed password for root from 123.249.0.130 port 2168 ssh2
Failed password for root from 123.249.0.130 port 2168 ssh2
Connection reset by 123.249.0.130 port 2168 [preauth]

Does it mean someone is trying to attack my server? @Falco


#4

Installed fail2ban according to this post, and there was no failed ssh in the log anymore. But the problem remains; here’s the new log:

2016-08-25 15:28:49 UTC [511-2] discourse@discourse LOG:  duration: 278.178 ms  execute <unnamed>: SELECT  "incoming_referers".* FROM "incoming_referers" WHERE "incoming_referers"."path" = '/c' AND "incoming_referers"."incoming_domain_id" = 132 LIMIT 1
2016-08-25 15:28:49 UTC [460-4] discourse@discourse LOG:  duration: 279.357 ms  statement: INSERT INTO topic_views (topic_id, ip_address, viewed_at, user_id)
                 SELECT 4087, '111.82.111.82', '2016-08-25', NULL
                 WHERE NOT EXISTS (
                   SELECT 1 FROM topic_views
                   WHERE (ip_address = '111.82.111.82' AND topic_id = 4087 AND user_id IS NULL)
                 )
2016-08-25 15:28:53 UTC [3186-1] discourse@discourse LOG:  duration: 171.986 ms  execute <unnamed>: SELECT COUNT(*) AS count_all, "user_visits"."visited_at" AS user_visits_visited_at FROM "user_visits" WHERE (visited_at >= '2016-07-25' and visited_at <= '2016-08-25') GROUP BY "user_visits"."visited_at"  ORDER BY "user_visits"."visited_at" ASC
2016-08-25 15:28:54 UTC [3186-2] discourse@discourse LOG:  duration: 103.462 ms  execute <unnamed>: SELECT COUNT(*) AS count_all, date(viewed_at) AS date_viewed_at FROM "user_profile_views" WHERE (viewed_at >= '2016-07-25' AND viewed_at < '2016-08-26') GROUP BY date(viewed_at)  ORDER BY date(viewed_at)
2016-08-25 15:28:54 UTC [3186-3] discourse@discourse LOG:  duration: 138.670 ms  bind <unnamed>: SELECT COUNT(*) AS count_all, DATE(created_at) AS date_created_at FROM "email_logs" WHERE "email_logs"."skipped" = 'f' AND (created_at BETWEEN '2016-07-25 00:00:00.000000' AND '2016-08-25 23:59:59.999999') GROUP BY DATE(created_at)  ORDER BY DATE(created_at)
2016-08-25 15:28:55 UTC [3186-4] discourse@discourse LOG:  duration: 830.905 ms  execute <unnamed>: SELECT COUNT(*) AS count_all, DATE(created_at) AS date_created_at FROM "email_logs" WHERE "email_logs"."skipped" = 'f' AND (created_at BETWEEN '2016-07-25 00:00:00.000000' AND '2016-08-25 23:59:59.999999') GROUP BY DATE(created_at)  ORDER BY DATE(created_at)
2016-08-25 15:28:55 UTC [3186-5] discourse@discourse LOG:  duration: 112.757 ms  execute <unnamed>: SELECT COUNT(*) FROM "email_logs"
2016-08-25 15:28:56 UTC [3186-6] discourse@discourse LOG:  duration: 283.972 ms  execute <unnamed>: SELECT COUNT(*) AS count_all, users.username AS users_username FROM "incoming_links" INNER JOIN "users" ON "users"."id" = "incoming_links"."user_id" WHERE (incoming_links.created_at > '2016-07-26 15:28:55.878454' AND incoming_links.user_id IS NOT NULL) GROUP BY users.username
2016-08-25 15:28:56 UTC [3186-7] discourse@discourse LOG:  duration: 434.612 ms  execute <unnamed>: SELECT  COUNT(*) AS count_all, incoming_domains.name AS incoming_domains_name FROM "incoming_links" INNER JOIN "incoming_referers" ON "incoming_referers"."id" = "incoming_links"."incoming_referer_id" INNER JOIN "incoming_domains" ON "incoming_domains"."id" = "incoming_referers"."incoming_domain_id" WHERE (incoming_links.created_at > '2016-07-26 15:28:56.250453') GROUP BY incoming_domains.name  ORDER BY count_all DESC LIMIT 10
2016-08-25 15:28:56 UTC [3186-8] discourse@discourse LOG:  duration: 164.986 ms  execute <unnamed>: SELECT COUNT(DISTINCT posts.topic_id) AS count_distinct_posts_topic_id, incoming_domains.name AS incoming_domains_name FROM "incoming_links" INNER JOIN "incoming_referers" ON "incoming_referers"."id" = "incoming_links"."incoming_referer_id" INNER JOIN "incoming_domains" ON "incoming_domains"."id" = "incoming_referers"."incoming_domain_id" INNER JOIN "posts" ON "posts"."id" = "incoming_links"."post_id" AND ("posts"."deleted_at" IS NULL) WHERE (incoming_links.created_at > '2016-07-26 15:28:56.733367' AND incoming_domains.name IN ('www.google.com.tw','www.baidu.com','www.google.com.hk','www.google.com','www.oschina.net','m.douban.com','www.sogou.com','weibo.com','www.v2ex.com','www.google.co.jp')) GROUP BY incoming_domains.name
43:M 25 Aug 15:30:43.079 * 10 changes in 300 seconds. Saving...
43:M 25 Aug 15:30:43.080 * Background saving started by pid 3792
3792:C 25 Aug 15:30:43.505 * DB saved on disk
3792:C 25 Aug 15:30:43.506 * RDB: 30 MB of memory used by copy-on-write
43:M 25 Aug 15:30:43.581 * Background saving terminated with success
43:M 25 Aug 15:35:44.083 * 10 changes in 300 seconds. Saving...
43:M 25 Aug 15:35:44.084 * Background saving started by pid 5479
5479:C 25 Aug 15:35:44.705 * DB saved on disk
5479:C 25 Aug 15:35:44.705 * RDB: 32 MB of memory used by copy-on-write
43:M 25 Aug 15:35:44.787 * Background saving terminated with success
2016-08-25 15:37:04 UTC [5920-1] discourse@discourse LOG:  duration: 262.623 ms  statement: DELETE FROM directory_items
                  USING directory_items di
                  LEFT JOIN users u ON u.id = user_id
                  WHERE di.id = directory_items.id AND
                        u.id IS NULL AND
                        di.period_type = 5
2016-08-25 15:37:06 UTC [5920-2] discourse@discourse LOG:  duration: 1894.536 ms  statement: WITH x AS (SELECT
                      u.id user_id,
                      SUM(CASE WHEN ua.action_type = 2 THEN 1 ELSE 0 END) likes_received,
                      SUM(CASE WHEN ua.action_type = 1 THEN 1 ELSE 0 END) likes_given,
                      COALESCE((SELECT COUNT(topic_id) FROM topic_views AS v WHERE v.user_id = u.id AND v.viewed_at >= '2016-08-24 15:37:04.271443'), 0) topics_entered,
                      COALESCE((SELECT COUNT(id) FROM user_visits AS uv WHERE uv.user_id = u.id AND uv.visited_at >= '2016-08-24 15:37:04.271443'), 0) days_visited,
                      COALESCE((SELECT SUM(posts_read) FROM user_visits AS uv2 WHERE uv2.user_id = u.id AND uv2.visited_at >= '2016-08-24 15:37:04.271443'), 0) posts_read,
                      SUM(CASE WHEN ua.action_type = 4 THEN 1 ELSE 0 END) topic_count,
                      SUM(CASE WHEN ua.action_type = 5 THEN 1 ELSE 0 END) post_count
                    FROM users AS u
                    LEFT OUTER JOIN user_actions AS ua ON ua.user_id = u.id
                    LEFT OUTER JOIN topics AS t ON ua.target_topic_id = t.id AND t.archetype = 'regular'
                    LEFT OUTER JOIN posts AS p ON ua.target_post_id = p.id
                    LEFT OUTER JOIN categories AS c ON t.category_id = c.id
                    WHERE u.active
                      AND NOT u.blocked
                      AND COALESCE(ua.created_at, '2016-08-24 15:37:04.271443') >= '2016-08-24 15:37:04.271443'
                      AND t.deleted_at IS NULL
                      AND COALESCE(t.visible, true)
                      AND p.deleted_at IS NULL
                      AND (NOT (COALESCE(p.hidden, false)))
                      AND COALESCE(p.post_type, 1) = 1
                      AND u.id > 0
                    GROUP BY u.id)
        UPDATE directory_items di SET
                 likes_received = x.likes_received,
                 likes_given = x.likes_given,
                 topics_entered = x.topics_entered,
                 days_visited = x.days_visited,
                 posts_read = x.posts_read,
                 topic_count = x.topic_count,
                 post_count = x.post_count
        FROM x
        WHERE
          x.user_id = di.user_id AND
          di.period_type = 5 AND (
          di.likes_received <> x.likes_received OR
          di.likes_given <> x.likes_given OR
          di.topics_entered <> x.topics_entered OR
          di.days_visited <> x.days_visited OR
          di.posts_read <> x.posts_read OR
          di.topic_count <> x.topic_count OR
          di.post_count <> x.post_count )
  
        
2016-08-25 15:37:42 UTC [5920-3] discourse@discourse LOG:  duration: 120.478 ms  execute <unnamed>: SELECT "topic_links".* FROM "topic_links" WHERE "topic_links"."user_id" = 3980
2016-08-25 15:37:43 UTC [5920-4] discourse@discourse LOG:  duration: 100.058 ms  execute <unnamed>: INSERT INTO "user_histories" ("acting_user_id", "context", "action", "email", "ip_address", "details", "created_at", "updated_at") VALUES (-1, 'Automatically deleted as abandoned, deactivated account', 1, '2461191065@q.com', '115.192.216.139', 'id: 3980
  username: 2461191065
  name: 
  created_at: 2016-08-11 12:17:21 UTC
  trust_level: 0
  last_seen_at: 
  last_emailed_at: 2016-08-11 12:22:19 UTC', '2016-08-25 15:37:43.068720', '2016-08-25 15:37:43.068720') RETURNING "id"
43:M 25 Aug 15:40:45.070 * 10 changes in 300 seconds. Saving...
43:M 25 Aug 15:40:45.071 * Background saving started by pid 7141
7141:C 25 Aug 15:40:45.525 * DB saved on disk
7141:C 25 Aug 15:40:45.526 * RDB: 32 MB of memory used by copy-on-write
43:M 25 Aug 15:40:45.571 * Background saving terminated with success
Shutting Down
run-parts: executing /etc/runit/3.d/01-nginx
ok: down: nginx: 1s, normally up
run-parts: executing /etc/runit/3.d/02-unicorn
exiting
ok: down: unicorn: 0s, normally up
run-parts: executing /etc/runit/3.d/10-redis
43:signal-handler (1472139851) Received SIGTERM scheduling shutdown...
43:M 25 Aug 15:44:11.619 # User requested shutdown...
43:M 25 Aug 15:44:11.619 * Saving the final RDB snapshot before exiting.
43:M 25 Aug 15:44:12.050 * DB saved on disk
43:M 25 Aug 15:44:12.050 # Redis is now ready to exit, bye bye...
ok: down: redis: 0s, normally up
run-parts: executing /etc/runit/3.d/99-postgres
2016-08-25 15:44:12 UTC [39-2] LOG:  received smart shutdown request
2016-08-25 15:44:12 UTC [57-2] LOG:  autovacuum launcher shutting down
2016-08-25 15:44:13 UTC [54-1] LOG:  shutting down
2016-08-25 15:44:13 UTC [54-2] LOG:  database system is shut down
ok: down: postgres: 0s, normally up
run-parts: executing /etc/runit/3.d/99-ssh
Received signal 15; terminating.
ok: down: sshd: 1s, normally up
When using programs that use GNU Parallel to process data for publication please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.
Or you can get GNU Parallel without this requirement by paying 10000 EUR.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

ok: down: nginx: 5s, normally up
ok: down: postgres: 2s, normally up
ok: down: redis: 3s, normally up
ok: down: cron: 0s, normally up
ok: down: rsyslog: 0s, normally up
ok: down: sshd: 2s, normally up
ok: down: unicorn: 4s, normally up
run-parts: executing /etc/runit/1.d/00-ensure-links
run-parts: executing /etc/runit/1.d/00-fix-log-permissions
run-parts: executing /etc/runit/1.d/anacron
run-parts: executing /etc/runit/1.d/cleanup-pids
Cleaning stale PID files
run-parts: executing /etc/runit/1.d/copy-env
run-parts: executing /etc/runit/1.d/enable-brotli
Started runsvdir, PID is 28
sh: echo: I/O error
rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 37
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
37:M 25 Aug 15:44:17.017 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
37:M 25 Aug 15:44:17.021 # Server started, Redis version 3.0.6
37:M 25 Aug 15:44:17.021 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
37:M 25 Aug 15:44:17.219 * DB loaded from disk: 0.197 seconds
37:M 25 Aug 15:44:17.219 * The server is now ready to accept connections on port 6379
ok: run: redis: (pid 37) 1s
ok: run: postgres: (pid 39) 1s
2016-08-25 15:44:17 UTC [53-1] LOG:  database system was shut down at 2016-08-25 15:44:13 UTC
2016-08-25 15:44:17 UTC [53-2] LOG:  MultiXact member wraparound protections are now enabled
2016-08-25 15:44:17 UTC [39-1] LOG:  database system is ready to accept connections
2016-08-25 15:44:17 UTC [57-1] LOG:  autovacuum launcher started
supervisor pid: 42 unicorn pid: 59
37:M 25 Aug 15:49:17.013 * 10 changes in 300 seconds. Saving...
37:M 25 Aug 15:49:17.014 * Background saving started by pid 2034
2034:C 25 Aug 15:49:17.472 * DB saved on disk
2034:C 25 Aug 15:49:17.473 * RDB: 30 MB of memory used by copy-on-write
37:M 25 Aug 15:49:17.515 * Background saving terminated with success
2016-08-25 15:49:51 UTC [1726-1] discourse@discourse LOG:  duration: 106.094 ms  statement:       UPDATE post_revisions
           SET number = pr.rank
          FROM (SELECT id, 1 + ROW_NUMBER() OVER (PARTITION BY post_id ORDER BY number, created_at, updated_at) AS rank FROM post_revisions) AS pr
         WHERE post_revisions.id = pr.id
           AND post_revisions.number <> pr.rank
  
2016-08-25 15:49:51 UTC [1726-2] discourse@discourse LOG:  duration: 122.600 ms  statement:       UPDATE posts
           SET version = 1 + (SELECT COUNT(*) FROM post_revisions WHERE post_id = posts.id),
               public_version = 1 + (SELECT COUNT(*) FROM post_revisions pr WHERE post_id = posts.id AND pr.hidden = 'f')
         WHERE version <> 1 + (SELECT COUNT(*) FROM post_revisions WHERE post_id = posts.id)
            OR public_version <> 1 + (SELECT COUNT(*) FROM post_revisions pr WHERE post_id = posts.id AND pr.hidden = 'f')
  
2016-08-25 15:50:17 UTC [1726-3] discourse@discourse LOG:  duration: 25300.461 ms  statement: UPDATE user_stats SET topics_entered = X.c
               FROM
              (SELECT v.user_id, COUNT(topic_id) AS c
               FROM topic_views AS v
               WHERE v.user_id IN (
                  SELECT u1.id FROM users u1 where u1.last_seen_at > '2016-08-25 02:49:51.864811'
               )
               GROUP BY v.user_id) AS X
              WHERE
                      X.user_id = user_stats.user_id AND
                      X.c <> topics_entered
      
2016-08-25 15:50:17 UTC [1726-4] discourse@discourse LOG:  duration: 624.268 ms  statement: UPDATE user_stats SET posts_read_count = X.c
                FROM
                (SELECT pt.user_id,
                        COUNT(*) AS c
                 FROM users AS u
                 JOIN post_timings AS pt ON pt.user_id = u.id
                 JOIN topics t ON t.id = pt.topic_id
                 WHERE u.last_seen_at > '2016-08-25 02:49:51.864811' AND
                       t.archetype = 'regular' AND
                       t.deleted_at IS NULL
                 GROUP BY pt.user_id) AS X
                 WHERE X.user_id = user_stats.user_id AND
                       X.c <> posts_read_count
      
37:M 25 Aug 15:54:18.022 * 10 changes in 300 seconds. Saving...
37:M 25 Aug 15:54:18.023 * Background saving started by pid 3694
3694:C 25 Aug 15:54:18.454 * DB saved on disk
3694:C 25 Aug 15:54:18.455 * RDB: 30 MB of memory used by copy-on-write
37:M 25 Aug 15:54:18.526 * Background saving terminated with success
37:M 25 Aug 15:59:19.034 * 10 changes in 300 seconds. Saving...
37:M 25 Aug 15:59:19.035 * Background saving started by pid 5432
5432:C 25 Aug 15:59:19.500 * DB saved on disk
5432:C 25 Aug 15:59:19.501 * RDB: 30 MB of memory used by copy-on-write
37:M 25 Aug 15:59:19.538 * Background saving terminated with success
Shutting Down
run-parts: executing /etc/runit/3.d/01-nginx
ok: down: nginx: 0s, normally up
run-parts: executing /etc/runit/3.d/02-unicorn
exiting
ok: down: unicorn: 0s, normally up
run-parts: executing /etc/runit/3.d/10-redis
37:signal-handler (1472140769) Received SIGTERM scheduling shutdown...
37:M 25 Aug 15:59:29.448 # User requested shutdown...
37:M 25 Aug 15:59:29.448 * Saving the final RDB snapshot before exiting.
37:M 25 Aug 15:59:30.083 * DB saved on disk
37:M 25 Aug 15:59:30.083 # Redis is now ready to exit, bye bye...
ok: down: redis: 0s, normally up
run-parts: executing /etc/runit/3.d/99-postgres
2016-08-25 15:59:30 UTC [39-2] LOG:  received smart shutdown request
2016-08-25 15:59:30 UTC [57-2] LOG:  autovacuum launcher shutting down
2016-08-25 15:59:31 UTC [54-1] LOG:  shutting down
2016-08-25 15:59:31 UTC [54-2] LOG:  database system is shut down
ok: down: postgres: 0s, normally up
run-parts: executing /etc/runit/3.d/99-ssh
Received signal 15; terminating.
ok: down: sshd: 1s, normally up
When using programs that use GNU Parallel to process data for publication please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.
Or you can get GNU Parallel without this requirement by paying 10000 EUR.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

ok: down: nginx: 4s, normally up
ok: down: postgres: 1s, normally up
ok: down: redis: 2s, normally up
parallel: SIGTERM received. No new jobs will be started.
parallel: Waiting for these 2 jobs to finish. Send SIGTERM again to stop now.
parallel: sv force-stop cron
parallel: sv force-stop rsyslog
Shutting Down
run-parts: executing /etc/runit/3.d/01-nginx
fail: nginx: runsv not running
run-parts: /etc/runit/3.d/01-nginx exited with return code 1
run-parts: executing /etc/runit/3.d/02-unicorn
fail: unicorn: runsv not running
run-parts: /etc/runit/3.d/02-unicorn exited with return code 1
run-parts: executing /etc/runit/3.d/10-redis
fail: redis: runsv not running
run-parts: /etc/runit/3.d/10-redis exited with return code 1
run-parts: executing /etc/runit/3.d/99-postgres
fail: postgres: runsv not running
run-parts: /etc/runit/3.d/99-postgres exited with return code 1
run-parts: executing /etc/runit/3.d/99-ssh
fail: sshd: runsv not running
run-parts: /etc/runit/3.d/99-ssh exited with return code 1
When using programs that use GNU Parallel to process data for publication please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; and it won't cost you a cent.
Or you can get GNU Parallel without this requirement by paying 10000 EUR.

To silence this citation notice run 'parallel --bibtex' once or use '--no-notice'.

fail: cron: runsv not running
fail: nginx: runsv not running
fail: postgres: runsv not running
fail: redis: runsv not running
fail: rsyslog: runsv not running
fail: sshd: runsv not running
fail: unicorn: runsv not running
/sbin/boot: line 9: kill: (28) - No such process
run-parts: executing /etc/runit/1.d/00-ensure-links
run-parts: executing /etc/runit/1.d/00-fix-log-permissions
run-parts: executing /etc/runit/1.d/anacron
run-parts: executing /etc/runit/1.d/cleanup-pids
Cleaning stale PID files
run-parts: executing /etc/runit/1.d/copy-env
run-parts: executing /etc/runit/1.d/enable-brotli
Started runsvdir, PID is 28
fail: redis: runsv not running
sh: echo: I/O error
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 3.0.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 44
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

44:M 25 Aug 16:00:22.865 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
44:M 25 Aug 16:00:22.865 # Server started, Redis version 3.0.6
44:M 25 Aug 16:00:22.865 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
rsyslogd: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScript command (v6+ config)? [v8.16.0 try http://www.rsyslog.com/e/2222 ]
rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
rsyslogd: activation of module imklog failed [v8.16.0 try http://www.rsyslog.com/e/2145 ]
rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com/e/2039 ]
Server listening on 0.0.0.0 port 22.
Server listening on :: port 22.
44:M 25 Aug 16:00:23.361 * DB loaded from disk: 0.496 seconds
44:M 25 Aug 16:00:23.361 * The server is now ready to accept connections on port 6379
ok: run: redis: (pid 44) 1s
ok: run: postgres: (pid 38) 1s
2016-08-25 16:00:23 UTC [55-1] LOG:  database system was shut down at 2016-08-25 15:59:31 UTC
2016-08-25 16:00:24 UTC [55-2] LOG:  MultiXact member wraparound protections are now enabled
2016-08-25 16:00:24 UTC [38-1] LOG:  database system is ready to accept connections
2016-08-25 16:00:24 UTC [63-1] LOG:  autovacuum launcher started
supervisor pid: 50 unicorn pid: 65
2016-08-25 16:00:41 UTC [94-1] discourse@discourse LOG:  duration: 148.114 ms  parse <unnamed>:               SELECT t.oid, t.typname, t.typelem, t.typdelim, t.typinput, r.rngsubtype, t.typtype, t.typbasetype
                FROM pg_type as t
                LEFT JOIN pg_range as r ON oid = rngtypid
                WHERE
                  t.typname IN ('int2', 'int4', 'int8', 'oid', 'float4', 'float8', 'text', 'varchar', 'char', 'name', 'bpchar', 'bool', 'bit', 'varbit', 'timestamptz', 'date', 'time', 'money', 'bytea', 'point', 'hstore', 'json', 'jsonb', 'cidr', 'inet', 'uuid', 'xml', 'tsvector', 'macaddr', 'citext', 'ltree', 'interval', 'path', 'line', 'polygon', 'circle', 'lseg', 'box', 'timestamp', 'numeric')
                  OR t.typtype IN ('r', 'e', 'd')
                  OR t.typinput = 'array_in(cstring,oid,integer)'::regprocedure
                  OR t.typelem != 0
  
2016-08-25 16:00:42 UTC [94-2] discourse@discourse LOG:  duration: 170.123 ms  statement: SELECT name, data_type, value FROM site_settings
2016-08-25 16:00:43 UTC [94-3] discourse@discourse LOG:  duration: 204.982 ms  bind <unnamed>: SELECT COUNT(count_column) FROM (SELECT  1 AS count_column FROM "users" LIMIT 20) subquery_for_count
2016-08-25 16:00:49 UTC [94-4] discourse@discourse LOG:  duration: 166.841 ms  bind <unnamed>: SELECT  "posts".* FROM "posts" WHERE ("posts"."deleted_at" IS NULL)  ORDER BY "posts"."id" ASC LIMIT 1
2016-08-25 16:00:50 UTC [94-5] discourse@discourse LOG:  duration: 115.054 ms  bind <unnamed>: SELECT  "top_topics".* FROM "top_topics"  ORDER BY "top_topics"."id" ASC LIMIT 1
2016-08-25 16:00:55 UTC [161-1] discourse@discourse LOG:  duration: 2007.011 ms  execute <unnamed>: SELECT  "topics"."id" FROM "topics" LEFT OUTER JOIN "categories" ON "categories"."id" = "topics"."category_id" LEFT OUTER JOIN topic_users AS tu ON (topics.id = tu.topic_id AND tu.user_id = -1) WHERE (categories.id = 1 OR (categories.parent_category_id = 1 AND categories.topic_id <> topics.id)) AND (topics.archetype <> 'private_message') AND (COALESCE(categories.topic_id, 0) <> topics.id) AND "topics"."visible" = 't' AND ("topics"."id" IS NOT NULL) AND (topics.deleted_at IS NULL) AND (pinned_at IS NULL OR category_id <> 1)  ORDER BY topics.bumped_at DESC LIMIT 3
2016-08-25 16:00:56 UTC [161-2] discourse@discourse LOG:  duration: 170.230 ms  statement:        UPDATE posts p
          SET score = x.score
         FROM (
          SELECT posts.id, COALESCE(posts.reply_count, 0) * 5 + COALESCE(posts.like_score, 0) * 15 + COALESCE(posts.incoming_link_count, 0) * 5 + COALESCE(posts.bookmark_count, 0) * 2 + COALESCE(posts.avg_time, 0) * 0.05 + COALESCE(posts.reads, 0) * 0.2 as score FROM posts
          join topics on posts.topic_id = topics.id
          WHERE (posts.score IS NULL OR posts.score <> COALESCE(posts.reply_count, 0) * 5 + COALESCE(posts.like_score, 0) * 15 + COALESCE(posts.incoming_link_count, 0) * 5 + COALESCE(posts.bookmark_count, 0) * 2 + COALESCE(posts.avg_time, 0) * 0.05 + COALESCE(posts.reads, 0) * 0.2) AND (topics.bumped_at > '2016-08-24 16:00:55.854364' )
          limit 20000
         ) AS x
         WHERE x.id = p.id
  
2016-08-25 16:00:56 UTC [161-3] discourse@discourse LOG:  duration: 399.797 ms  statement: UPDATE posts
  SET percent_rank = X.percent_rank
  FROM (
    SELECT posts.id, Y.percent_rank
    FROM posts
    JOIN (
      SELECT id, percent_rank()
                   OVER (PARTITION BY topic_id ORDER BY SCORE DESC) as percent_rank
      FROM posts
     ) Y ON Y.id = posts.id
     JOIN topics ON posts.topic_id = topics.id
    WHERE (posts.percent_rank IS NULL OR Y.percent_rank <> posts.percent_rank) AND (topics.bumped_at > '2016-08-24 16:00:55.854364' )
    LIMIT 20000
  ) AS X
  WHERE posts.id = X.id
  
2016-08-25 16:01:02 UTC [230-1] discourse@discourse LOG:  duration: 176.259 ms  execute <unnamed>: SELECT  "topics".* FROM "topics" WHERE ("topics"."deleted_at" IS NULL) AND "topics"."archetype" = 'banner'  ORDER BY "topics"."id" ASC LIMIT 1
2016-08-25 16:01:02 UTC [230-2] discourse@discourse LOG:  duration: 109.091 ms  execute <unnamed>: SELECT "users".* FROM "users" WHERE "users"."id" IN (2412, 2057)
2016-08-25 16:01:02 UTC [229-1] discourse@discourse LOG:  duration: 211.190 ms  execute <unnamed>: SELECT "posts".* FROM "posts" WHERE ("posts"."deleted_at" IS NULL) AND "posts"."topic_id" = 4087 AND "posts"."id" IN (12924, 12963, 12996, 12998, 13000, 13001, 13021, 13022, 13024, 13025, 13027, 13055, 13063, 13083, 13180, 14701) AND "posts"."post_type" IN (1, 2, 3)  ORDER BY sort_order
2016-08-25 16:01:02 UTC [229-2] discourse@discourse LOG:  duration: 152.650 ms  statement: SELECT
                        l.post_id,
                        l.url,
                        l.clicks,
                        COALESCE(t.title, l.title) AS title,
                        l.internal,
                        l.reflection,
                        l.domain
                FROM topic_links l
                LEFT JOIN topics t ON t.id = l.link_topic_id
                LEFT JOIN categories AS c ON c.id = t.category_id
                WHERE (t.deleted_at IS NULL) AND (COALESCE(t.archetype, 'regular') <> 'private_message') AND (l.post_id IN (12924,12963,12996,12998,13000,13001,13021,13022,13024,13025,13027,13055,13063,13083,13180,14701)) AND (NOT COALESCE(c.read_restricted, false))
                ORDER BY reflection ASC, clicks DESC
2016-08-25 16:01:03 UTC [230-3] discourse@discourse LOG:  duration: 316.099 ms  statement: INSERT INTO topic_views (topic_id, ip_address, viewed_at, user_id)
                 SELECT 2353, '123.125.71.94', '2016-08-25', NULL
                 WHERE NOT EXISTS (
                   SELECT 1 FROM topic_views
                   WHERE (ip_address = '123.125.71.94' AND topic_id = 2353 AND user_id IS NULL)
                 )
2016-08-25 16:01:03 UTC [231-1] discourse@discourse LOG:  duration: 472.055 ms  execute <unnamed>: SELECT COUNT(*) FROM "notifications" WHERE "notifications"."user_id" = 1 AND (read = false)
2016-08-25 16:01:05 UTC [231-2] discourse@discourse LOG:  duration: 1776.251 ms  statement:     SELECT 
             u.id AS user_id,
             topics.id AS topic_id,
             topics.created_at,
             highest_post_number,
             last_read_post_number,
             c.id AS category_id,
             tu.notification_level
      FROM topics
      JOIN users u on u.id = 1
      JOIN user_stats AS us ON us.user_id = u.id
      JOIN user_options AS uo ON uo.user_id = u.id
      JOIN categories c ON c.id = topics.category_id
      LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
      WHERE u.id = 1 AND
            topics.archetype <> 'private_message' AND
            ((1=0) OR ("topics"."deleted_at" IS NULL AND topics.created_at >= GREATEST(CASE
                    WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
                    WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(u.previous_visit_at,u.created_at)
                    ELSE ('2016-08-25 16:01:03.538407'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
                 END, us.new_since, '2016-07-25 01:56:32.000000') AND tu.last_read_post_number IS NULL AND COALESCE(tu.notification_level, 2) >= 2)) AND
            (topics.visible OR u.admin OR u.moderator) AND
            topics.deleted_at IS NULL AND
            ( NOT c.read_restricted OR u.admin OR category_id IN (
                SELECT c2.id FROM categories c2
                JOIN category_groups cg ON cg.category_id = c2.id
                JOIN group_users gu ON gu.user_id = 1 AND cg.group_id = gu.group_id
                WHERE c2.read_restricted )
            )
            AND NOT EXISTS( SELECT 1 FROM category_users cu
                            WHERE last_read_post_number IS NULL AND
                                 cu.user_id = 1 AND
                                 cu.category_id = topics.category_id AND
                                 cu.notification_level = 0)
  
  
  UNION ALL
  
      SELECT 
             u.id AS user_id,
             topics.id AS topic_id,
             topics.created_at,
             highest_post_number,
             last_read_post_number,
             c.id AS category_id,
             tu.notification_level
      FROM topics
      JOIN users u on u.id = 1
      JOIN user_stats AS us ON us.user_id = u.id
      JOIN user_options AS uo ON uo.user_id = u.id
      JOIN categories c ON c.id = topics.category_id
      LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id
      WHERE u.id = 1 AND
            topics.archetype <> 'private_message' AND
            (("topics"."deleted_at" IS NULL AND tu.last_read_post_number < topics.highest_post_number AND COALESCE(tu.notification_level, 1) >= 2) OR (1=0)) AND
            (topics.visible OR u.admin OR u.moderator) AND
            topics.deleted_at IS NULL AND
            ( NOT c.read_restricted OR u.admin OR category_id IN (
                SELECT c2.id FROM categories c2
                JOIN category_groups cg ON cg.category_id = c2.id
                JOIN group_users gu ON gu.user_id = 1 AND cg.group_id = gu.group_id
                WHERE c2.read_restricted )
            )
            AND NOT EXISTS( SELECT 1 FROM category_users cu
                            WHERE last_read_post_number IS NULL AND
                                 cu.user_id = 1 AND
                                 cu.category_id = topics.category_id AND
                                 cu.notification_level = 0)
  
  
2016-08-25 16:02:18 UTC [230-4] discourse@discourse LOG:  duration: 131.154 ms  execute <unnamed>: SELECT  "topics".* FROM "topics" WHERE "topics"."id" = 187  ORDER BY "topics"."id" ASC LIMIT 1
2016-08-25 16:02:37 UTC [161-4] discourse@discourse LOG:  duration: 123.073 ms  statement: WITH category_definition_topic_ids AS (
                    SELECT COALESCE(topic_id, 0) AS id FROM categories
                  ), invisible_topic_ids AS (
                    SELECT id
                    FROM topics
                    WHERE deleted_at IS NOT NULL
                       OR NOT visible
                       OR archetype = 'private_message'
                       OR archived
                       OR id IN (SELECT id FROM category_definition_topic_ids)
                  )
                  DELETE FROM top_topics
                  WHERE topic_id IN (SELECT id FROM invisible_topic_ids)
2016-08-25 16:02:39 UTC [161-5] discourse@discourse LOG:  duration: 1674.056 ms  statement: UPDATE top_topics
                  SET daily_views_count = c.count
                  FROM top_topics tt
                  INNER JOIN (SELECT topic_id, COUNT(*) AS count
               FROM topic_views
               WHERE viewed_at >= '2016-08-24 16:02:37.929224'
               GROUP BY topic_id) c ON tt.topic_id = c.topic_id
                  WHERE tt.topic_id = top_topics.topic_id
                    AND tt.daily_views_count <> c.count
2016-08-25 16:03:27 UTC [231-3] discourse@discourse LOG:  duration: 133.151 ms  execute <unnamed>: SELECT  "notifications".* FROM "notifications" LEFT JOIN topics ON notifications.topic_id = topics.id WHERE "notifications"."user_id" = 1 AND (topics.id IS NULL OR topics.deleted_at IS NULL)  ORDER BY notifications.created_at desc LIMIT 10
44:M 25 Aug 16:05:23.084 * 10 changes in 300 seconds. Saving...
44:M 25 Aug 16:05:23.085 * Background saving started by pid 2071
2071:C 25 Aug 16:05:23.719 * DB saved on disk
2071:C 25 Aug 16:05:23.720 * RDB: 32 MB of memory used by copy-on-write
44:M 25 Aug 16:05:23.786 * Background saving terminated with success
2016-08-25 16:06:05 UTC [230-5] discourse@discourse LOG:  duration: 168.899 ms  statement: INSERT INTO topic_views (topic_id, ip_address, viewed_at, user_id)
                 SELECT 4585, '207.46.13.127', '2016-08-25', NULL
                 WHERE NOT EXISTS (
                   SELECT 1 FROM topic_views
                   WHERE (ip_address = '207.46.13.127' AND topic_id = 4585 AND user_id IS NULL)
                 )
2016-08-25 16:09:11 UTC [231-4] discourse@discourse LOG:  duration: 131.902 ms  execute <unnamed>: INSERT INTO "incoming_links" ("incoming_referer_id", "post_id", "ip_address", "created_at") VALUES (102, 12924, '111.82.39.156/32', '2016-08-25 16:09:10.993862') RETURNING "id"
44:M 25 Aug 16:10:24.057 * 10 changes in 300 seconds. Saving...
44:M 25 Aug 16:10:24.057 * Background saving started by pid 3699
3699:C 25 Aug 16:10:24.537 * DB saved on disk
3699:C 25 Aug 16:10:24.538 * RDB: 30 MB of memory used by copy-on-write
44:M 25 Aug 16:10:24.559 * Background saving terminated with success

Any ideas? Thanks!


(Matt Palmer) #5

I can’t see any indication of a 429 response in that log; try looking in /var/docker/shared/app/log/rails/production.log. However, as far as I can tell from a connection to your site, you’re not actually hitting Discourse. The nginx server that is replying is version 1.4.6, which is waaaaaay older than what we run in the Discourse container. Take a look at whatever’s proxying requests into Discourse first. Track the problem down methodically, rather than jumping to conclusions.


#6

I don’t have this file:

root@iosre:~# ls /var/docker/shared/app/log/rails/production.log
ls: cannot access /var/docker/shared/app/log/rails/production.log: No such file or directory

But I do have:

root@iosre:~# find / -name production.log
/var/lib/docker/aufs/diff/3a210005fb876a229df30d25fc2fa60dd919ba65657f82bd0ebda4e02676ffd9/var/www/discourse/log/production.log
/var/lib/docker/aufs/mnt/c63da2a37a8233329021f97f6f0adc16d2884a20926e8e3fe21a1eaed9eb3e30/var/www/discourse/log/production.log
/var/discourse/shared/standalone/log/rails/production.log

And I found something suspicious in the last production log:


Started GET “/notifications?recent=true&limit=13&=1471500405578" for 110.83.62.1 at 2016-08-26 00:18:39 +0000
Processing by NotificationsController#index as JSON
Parameters: {“recent”=>“true”, “limit”=>“13”, "
”=>“1471500405578”}
Completed 403 Forbidden in 1ms (Views: 0.1ms | ActiveRecord: 0.0ms)
Started GET “/notifications?recent=true&limit=13&=1471500405581" for 110.83.62.1 at 2016-08-26 00:18:39 +0000
Processing by NotificationsController#index as JSON
Parameters: {“recent”=>“true”, “limit”=>“13”, "
”=>“1471500405581”}
Completed 403 Forbidden in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms)
Started GET “/notifications?recent=true&limit=13&=1471500405582" for 110.83.62.1 at 2016-08-26 00:18:39 +0000
Processing by NotificationsController#index as JSON
Parameters: {“recent”=>“true”, “limit”=>“13”, "
”=>“1471500405582”}
Completed 403 Forbidden in 1ms (Views: 0.1ms | ActiveRecord: 0.0ms)
Started GET “/notifications?recent=true&limit=13&=1471500405583" for 110.83.62.1 at 2016-08-26 00:18:40 +0000
Processing by NotificationsController#index as JSON
Parameters: {“recent”=>“true”, “limit”=>“13”, "
”=>“1471500405583”}
Completed 403 Forbidden in 1ms (Views: 0.1ms | ActiveRecord: 0.0ms)
(End of file)

The GET notifications requests from 110.83.62.1 started at yesterday afternoon and got called several times per second:

Started GET “/notifications?recent=true&limit=13&_=1471495755711” for 110.83.62.1 at 2016-08-25 12:50:08 +0000

Is this the problem?

And should I update nginx now, before tracking down the problem?

Thanks!


(Matt Palmer) #7

Well, a 403 isn’t a 429, so I’d say it’s a relatively low probability that’s the problem.

The fact that the nginx that is responding to the HTTP request with a 429 is older than we use isn’t necessarily a problem, it’s simply an indication that whatever is responding isn’t Discourse, but is rather something in front of Discourse. So, this doesn’t look like a Discourse problem, and we really can’t help you debug whatever is causing the problem if it isn’t Discourse.


#8

BTW, my observation is that when I was running some time-consuming commands (say /var/discourse/shared/standalone/log/rails/production.log, launcher log apps, or scp root@iosre.com:/var/discourse/shared/standalone/backups/default/xxx.tar.gz, it was more likely to visit the site.

Does that indicate anything? Thanks!


#9

I’ve updated nginx to 1.10.1 but problem persists. If that’s not a discourse problem, can you gimme some suggestions where to ask for help? Thanks!


(Matt Palmer) #10

Perhaps superuser.com? Nginx mailing list / forum? Honestly not sure where people ask random nginx-related questions, because I tend to fix my problems myself.


#11

If I stop discourse:

root@iosre:/var/discourse# ./launcher stop app
+ /usr/bin/docker stop -t 10 app
app

And visit the website, it shows:

502 Bad Gateway
nginx/1.10.1

i.e. it’s 502 bad gateway instead of error 429. Does that mean it’s a discourse problem rather than an nginx problem?


(Matt Palmer) #12

I’d say it makes it more likely, but who knows what crazy stuff something upstream might be doing? Another place you could check for log messages, by the way, is /logs on your instance (when it’s running, of course…)


#13
root@iosre:/var/discourse# find / -name logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/app/assets/javascripts/admin/templates/logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-2.3.7/apis/logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-2.3.22/apis/logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/.git/logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/public/assets/admin/templates/logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/plugins/docker_manager/.git/logs
/var/lib/docker/aufs/diff/d335459aeaba6ab4232dafbc3cbb5585bedbae3f83ddff0026780a4035065d7a/var/www/discourse/plugins/discourse-solved/.git/logs
/var/lib/docker/aufs/diff/203aa2a3d8cbd578067cf56200ec86661631e0259042bb39865d4d6747be66aa/var/www/discourse/app/assets/javascripts/admin/templates/logs
/var/lib/docker/aufs/diff/203aa2a3d8cbd578067cf56200ec86661631e0259042bb39865d4d6747be66aa/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-2.3.7/apis/logs
/var/lib/docker/aufs/diff/203aa2a3d8cbd578067cf56200ec86661631e0259042bb39865d4d6747be66aa/var/www/discourse/.git/logs
/var/lib/docker/aufs/diff/55d9a9ec7c7bfa9243d728ee84f8a1a3b37b16e26cf1c7ccd5f0931d726d6a73/pups/.git/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/app/assets/javascripts/admin/templates/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-2.3.7/apis/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-2.3.22/apis/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/.git/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/public/assets/admin/templates/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/plugins/docker_manager/.git/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/var/www/discourse/plugins/discourse-solved/.git/logs
/var/lib/docker/aufs/mnt/31c0b138ff96ef17e3c9252238246b65484311bb0f919061f102a19ee208e952/pups/.git/logs
/var/discourse/.git/logs

Which one are you referring to?


#14

Also I’ve tried to rebuild discourse but no luck fixing the problem.


(Matt Palmer) #15

Visit http://iosre.com/logs in your web browser of choice.


#16

Got error 429 again :cry:


(Matt Palmer) #17

I’m pretty sure that means the rate limiting is happening upstream of Discourse. The /logs endpoint isn’t protected by the Discourse built-in rate limiting, as far as I can see.


#18

Any suggestions what should I do next to track the problem?


(Matt Palmer) #19
  1. Formulate a falsifiable hypothesis as to what the problem may be;
  2. Devise an experiment that will test the hypothesis;
  3. Run the experiment;
  4. Evaluate the data.

Repeat as required.


#20

I’m not a server expert, just a random discourse user who happen to know some basic computer operations.

Can you gimme a more concrete suggestion? Like first run some commands and paste the output here, then … etc etc.

Thanks!