Bootstrap fails on CentOS 7

Installation
1

Hi!

I am currently trying to setup a discourse instance on CentOS 7 w/ OverlayFS… I seem to be getting this error when running discourse-setup, or manually trying to bootstrap with launcher.

I, [2016-08-20T20:52:04.113288 #15]  INFO -- : > cd /var/www/discourse && chown -R discourse /var/www/discourse
chown: cannot access '/var/www/discourse/app/assets/images/favicons/google_branding/logo_docs_128px.png': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/images/favicons/google_branding/logo_drive_128px.png': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/images/favicons/google_branding/logo_forms_128px.png': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/images/favicons/google_branding/logo_sheets_128px.png': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/images/favicons/google_branding/logo_slides_128px.png': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/images/grippie.png': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/defer/html-sanitizer-bundle.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/components/poster-name.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/autolink_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/bbcode_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/bold_italics_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/category_hashtag_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/censored_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/code_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/html_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/mention_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/nested_link_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/newline_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/onebox_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/quote_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/dialects/table_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/censored-words.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/ember_compat_handlebars.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/emoji/emoji-groups.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/emoji/emoji-toolbar.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/emoji/emoji.js.erb': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/markdown.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/onebox.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/lib/utilities.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/mixins/ajax.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/models/input_validation.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/models/login_method.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/models/static_page.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/templates/user/user.hbs': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse/views/application.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/discourse.js': No such file or directory
chown: cannot access '/var/www/discourse/app/assets/javascripts/preload_store.js': No such file or directory
chown: cannot access '/var/www/discourse/app/helpers/deferred_scripts_helper.rb': No such file or directory
chown: cannot access '/var/www/discourse/app/jobs/scheduled/clean_up_digest_keys.rb': No such file or directory
chown: cannot access '/var/www/discourse/app/jobs/scheduled/daily_performance_report.rb': No such file or directory
chown: cannot access '/var/www/discourse/app/models/digest_unsubscribe_key.rb': No such file or directory
chown: cannot access '/var/www/discourse/lib/freedom_patches/ember_compat_handlebars.rb': No such file or directory
chown: cannot access '/var/www/discourse/plugins/discourse-details/assets/javascripts/details_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/plugins/poll/assets/javascripts/poll_dialect.js': No such file or directory
chown: cannot access '/var/www/discourse/script/nginx_analyze.rb': No such file or directory
chown: cannot access '/var/www/discourse/spec/fixtures/emails/bounced_email_2.eml': No such file or directory
chown: cannot access '/var/www/discourse/spec/models/digest_unsubscribe_key_spec.rb': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/adapters/topic-list-test.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/admin/models/api-key-test.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/admin/models/flagged-post-test.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/helpers/current_user.js': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/helpers/site.js': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/lib/bbcode-test.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/lib/markdown-test.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/lib/onebox-test.js.es6': No such file or directory
chown: cannot access '/var/www/discourse/test/javascripts/mdtest/mdtest.js.erb': No such file or directory
chown: cannot access '/var/www/discourse/vendor/assets/javascripts/md5.js': No such file or directory

I, [2016-08-20T20:52:15.476668 #15]  INFO -- : 
I, [2016-08-20T20:52:15.477149 #15]  INFO -- : Terminating async processes
I, [2016-08-20T20:52:15.477205 #15]  INFO -- : Sending INT to HOME=/var/lib/postgresql USER=postgres exec chpst -u postgres:postgres:ssl-cert -U postgres:postgres:ssl-cert /usr/lib/postgresql/9.5/bin/postmaster -D /etc/postgresql/9.5/main pid: 44
I, [2016-08-20T20:52:15.477300 #15]  INFO -- : Sending TERM to exec chpst -u redis -U redis /usr/bin/redis-server /etc/redis/redis.conf pid: 157
2016-08-20 20:52:15 UTC [44-2] LOG:  received fast shutdown request
2016-08-20 20:52:15 UTC [44-3] LOG:  aborting any active transactions
157:signal-handler (1471726335) Received SIGTERM scheduling shutdown...
2016-08-20 20:52:15 UTC [51-2] LOG:  autovacuum launcher shutting down
2016-08-20 20:52:15 UTC [48-1] LOG:  shutting down
2016-08-20 20:52:15 UTC [48-2] LOG:  database system is shut down
157:M 20 Aug 20:52:15.545 # User requested shutdown...
157:M 20 Aug 20:52:15.545 * Saving the final RDB snapshot before exiting.
157:M 20 Aug 20:52:15.546 * DB saved on disk
157:M 20 Aug 20:52:15.546 # Redis is now ready to exit, bye bye...


FAILED
--------------------
Pups::ExecError: cd /var/www/discourse && chown -R discourse /var/www/discourse failed with return #<Process::Status: pid 279 exit 1>
Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'
exec failed with the params {"cd"=>"$home", "hook"=>"web", "cmd"=>["gem update bundler", "chown -R discourse $home"]}
221c77edf0b2ca79fd7d0714be191161744b53ad8d92d04355b0002697b4bc38
** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

What is this related to?

Also, here is my docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.703 GiB
Name: discourse
ID: E6F3:PEGQ:6W6H:UQXO:MPVE:Q32C:763B:Q4K2:ZFGA:TD24:J32J:A7HS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Thanks.

2

Bumping this, trying to get this resolved ASAP.

3

It is possible you need a newer kernel. Even on Ubuntu 16, Docker still uses aufs filesystem in a clean install.

4

Could this be discourse specific? I’ve never had a problem using Overlay with anything else.

I refuse to use device mapper, for obvious reasons. :heart_eyes:

I suppose I can roll with Fedora, if need be.

5

We have found that Docker is only fully stable with very recent Kernel versions. Not sure beyond that.

In theory overlayfs should work. In practice clean installs of Docker today still use aufs. Read into that what you will.

6

That is bizarre. I don’t know why chown wouldn’t work. I do know that it has nothing to do with Discourse.

7

I’d like to add, this is being executed in the docker container… not on the actual host.

This is what leads me to think it’s something to do with the storage driver.

8

That sounds right. And broken. We gave up on another hosting service because of some other strange problem. (I think it had to do with networking.)

You might search for those terms with your hosting provider or docker and centos.

9

Not relevant in this case, self virtualized under ESXi. It’s a straight out of ISO install, with the exception of EPEL.

10

Trying with mainline kernel, will report back.

11

Same thing is occurring on mainline kernel, reverting to device mapper to see if outcome is successful… Definitely won’t be using that in production, however.

Update: Works with devicemapper, must be a problem between overlay and discourse.

I have tried with SELinux disabled, to no avail… as expected.

12

Devicemapper will corrupt your data over time in our experience, every time.

13

I have run overlay just fine in dev but not backed by xfs, only backed by ext4fs. It was certainly more quirky than running aufs, you got to be careful with inode counts.

14

I’ve heard that before (in regards to the inode counts). Device mapper is not something I will use in production.

I’m not sure exactly how to track this down any further. I will attempt to use Fedora 24 and Overlay tomorrow, although I highly doubt it will help, seeing as I was using a newer kernel on CentOS anyways.

15

Hi everyone,

I’m having the same problem here, trying to install discourse following the beginner install guide on an OpenStack CentOS 7 VM, getting exactly the same output & errors (http://pastebin.com/PJBcSJcH).

This is my docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.5
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.609 GiB
Name: discoursetest
ID: Y367:5ONL:JYSR:BACX:2ZD2:J3GF:VLS3:EFKA:YBYZ:NLVF:74AB:75SA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

I tried updating the kernel and still getting the same errors as @Luke. Docker info following:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.5
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.9.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.765 GiB
Name: discoursetest.novalocal
ID: Y367:5ONL:JYSR:BACX:2ZD2:J3GF:VLS3:EFKA:YBYZ:NLVF:74AB:75SA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Any new insights on this issue? The VM is sill deployed if extra info or testing is needed.

16

Any help or useful information about this problem would be greatly appreciated :slight_smile:

17

Docker file systems are, let’s say, complicated.

We choose AUFS and we guarantee you can host your data there and everything will work out.

We can’t afford to go after every distro+os fs+docker fs, so I can’t really help you on that.

1 Like
18

Useful information greatly appreciated!

Would be nice to mention that in the hardware requirements of the install guide to avoid time loss (and maybe maintain a compatibility table to track problematic & successful configurations tested by the community).

19

It is a limitation of your distro – you should be asking them why Docker is not stable on their platform, not us.

1 Like
20

Most probably it’s because the backing xfs filesystem was not created with ftype=1 option. This is very common if you let the OS installer format the / fs and you just have /var/lib/docker on /.

On newer docker verision (1.13?) with docker info you can see support d_type = false and that indicates your xfs was created with ftype=0 instead of the required ftype=1. This will cause weird issues if you use overlayfs on top of it.

See below issues for detail:
https://github.com/docker/docker/issues/27358
https://github.com/docker/docker/issues/9572

The solution:

  1. The easy way - just create another fs for /var/lib/docker with proper option if you have free space on disk. There are two options:
    a. Create a ext4 fs with default option and it will be fine.
    b. If you prefer xfs, create it like this: mkfs.xfs -n ftype=1 /path/to/your/device

  2. Or reinstall OS and during installation, make sure you manually create root fs with proper option. You cannot specify that option in the GUI/TUI installer so have to run the command by hand in the terminal provided by the installer. The command is mkfs -t xfs -n ftype=1 /PATH/TO/DEVICE per RH document.

  3. Or change your current root fs option from the default ftype=0 to ftype=1. Unfortunately there is no online conversion way exists so you can only do a xfsdump/xfsrestore. Basically you need to prepare a new replacement root partition or disk, manually create xfs on it with proper option, then dump original root fs to the new one, and boot from the new root fs finally.

With any of above solutions applied, in docker info output you will see support d_type = true. Then you won’t run into the issue.

Edit in 2017/02/24: the symptom you may see with overlayfs on a d_type=false fs may be similar - strange errors during some very basic file operations, such as chown, chmod, rm etc… If you happens to run into such problems, and you find d_type=false with docker info, then above solution is for you.

9 Likes