Data lost with standalone docker install

Hi,

I manage a Discourse instance for a collaborative project, and this morning, when I’ve logged me into the forum, all recent discussion where gone (since the end of 2018).
It’s like all the forum have gone back in a previous state.

There is two backup archives, one from 2019-04-11 and the other from 2019-04-18, but when I extract them and read the dump.sql file, there is no trace of all the subjects that have gone…

I’ve had recently a RAID problem with one of my disc, but the RAID array haven’t been comprise and the other services are running without problem since this incident. The only thing is that I was forced to reboot the server with the kernel sysrq keys.

UPDATE :

  • Now I’m sure that no discourse auto-backup contains my missing data
  • docker ps --all show me no other discourse container
  • So, the problem seems to be that my shared directory have been override by something, but WHAT ? If I well understand the discourse workflow, the shared directory is not managed by the discourse container, so it seems that it’s not directly a container issue.
  • What happen when discourse start (my systemd service run : /var/discourse/launcher start app) ? Is there anything that can throw my data away ?

Here is some information :

# docker ps

CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                                                                                NAMES
1504dd58ccaf        local_discourse/app   "/sbin/boot"             5 months ago        Up 3 days                                                                                                                app
# cat /var/discourse/containers/app.yml

## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
  - "templates/web.socketed.template.yml" 
## Uncomment these two lines if you wish to add Lets Encrypt (https)
#  #- "templates/web.ssl.template.yml"
#  #- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
#expose:
#  - "80:80"   # http
#  - "443:443" # https

params:
  db_default_text_search_config: "pg_catalog.french"

  ## Set db_shared_buffers to a max of 25% of the total memory.
  ## will be set automatically by bootstrap based on detected RAM, or you can override
  db_shared_buffers: "3072MB"

  ## can improve sorting performance, but adds memory usage per-connection
  #db_work_mem: "40MB"

  ## Which Git revision should this container use? (default: tests-passed)
  #version: tests-passed

env:
  LANG: fr_FR.UTF-8
  # DISCOURSE_DEFAULT_LOCALE: en

  ## How many concurrent web requests are supported? Depends on memory and CPU cores.
  ## will be set automatically by bootstrap based on detected CPUs, or you can override
  UNICORN_WORKERS: 4

  ## TODO: The domain name this Discourse instance will respond to
  ## Required. Discourse will not work with a bare IP number.
  DISCOURSE_HOSTNAME: ********
  DISCOURSE_RELATIVE_URL_ROOT: /forum

  ## Uncomment if you want the container to be started with the same
  ## hostname (-h option) as specified above (default "$hostname-$config")
  #DOCKER_USE_HOSTNAME: true

  ## TODO: List of comma delimited emails that will be made admin and developer
  ## on initial signup example 'user1@example.com,user2@example.com'
  DISCOURSE_DEVELOPER_EMAILS: '****'

  ## TODO: The SMTP mail server used to validate new accounts and send notifications
  # SMTP ADDRESS, username, and password are required
  # WARNING the char '#' in SMTP password can cause problems!
  DISCOURSE_SMTP_ADDRESS: ******
  DISCOURSE_SMTP_PORT: 587
  DISCOURSE_SMTP_USER_NAME: *******
  DISCOURSE_SMTP_PASSWORD: "******"
  #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

  ## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
  #LETSENCRYPT_ACCOUNT_EMAIL: me@example.com

  ## The CDN address for this Discourse instance (configured to pull)
  ## see https://meta.discourse.org/t/14857 for details
  #DISCOURSE_CDN_URL: //discourse-cdn.example.com

## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git

## Any custom commands to run after building
run:
    - exec: echo "Beginning of custom commands"
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  #- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
    - exec:
        cd: $home
        cmd:
          - mkdir -p public/forum
          - cd public/forum && ln -s ../uploads && ln -s ../backups
    - replace:
       global: true
       filename: /etc/nginx/conf.d/discourse.conf
       from: proxy_pass http://discourse;
       to: |
          rewrite ^/(.*)$ /forum/$1 break;
          proxy_pass http://discourse;
    - replace:
       filename: /etc/nginx/conf.d/discourse.conf
       from: etag off;
       to: |
          etag off;
          location /forum {
             rewrite ^/forum/?(.*)$ /$1;
          }
    - replace:
         filename: /etc/nginx/conf.d/discourse.conf
         from: $proxy_add_x_forwarded_for
         to: $http_your_original_ip_header
         global: true
    - exec: echo "End of custom commands"

Is there any chance to recover the forum’s data ? And in your opinion, what cloud have happen ?

Thank you for your help.

There was another topic this past weekend that had a similar complaint of Discourse reverting to an old version.

I can’t imagine how it could happen and don’t know if these events are related. I can’t find it right now.

Hi,

I’ve search the post you’ve mentioned but was unable to find it, can you link it here ?

No. I can’t find it either. It seems to have been deleted. I don’t see it in what I believe is the user’s profile page.

I emailed the person whose site it was and asked if they had resolved it.

It seems highly improbable that your backups would not have the missing posts in them if they were indeed on the site since then.

If you have a budget and would like for me to take a look, my contact info is in my profile. Since I don’t know that I can solve your problem I will have to charge you for my time and not a solution to your problem.

2 Likes

Thank you for contacting him.

I’ve post some updates, and I’m now trying to find out if there is any files that have been lost by my filesystem (ext4) during the RAID outrage.

Unfortunately we cannot afford this kind of help.

1 Like