# Legal Tools Plugin
**URL:** https://meta.discourse.org/t/legal-tools-plugin/87966
**Category:** Plugin
**Tags:** pavilion, broken
**Created:** [May 21, 2018, 12:30am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966 "2018-05-21T00:30:57Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [May 21, 2018, 12:30am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/1 "2018-05-21T00:30:57Z")
Repository: [GitHub - paviliondev/discourse-legal-tools: Tools to help with legal compliance when using Discourse · GitHub](https://github.com/paviliondev/discourse-legal-tools)
This plugin provides tools to assist with legal compliance when running a Discourse forum. Tools will be added on an ongoing basis.
Please note the disclaimer below. This plugin provides no guarantee of legal compliance.
**Extended User Download**
The extended user download is a single CSV with the following entries, each separated by two blank lines:
- A header (can be edited: Customize \> Text Content \> “csv\_export.extended.title”):
- A note below the header (can be edited: Customize \> Text Content \> “csv\_export.extended.note”):
- Posts. The default information included in the user download.
- Account. Account and Profile information.
- External Accounts. Information from external accounts (if it exists).
- Statistics. Statistical information stored concerning the user.
- Login and Login History. Information about user logins.
- Searches. All ip-logged records of searches performed by the user.
- Topic Views. All ip-logged topic views of the user.
- Topic Link Clicks. All ip-logged topic links clicked by the user.
- Profile Views. All ip-logged profile views of the user.
- Actions. All actions taken by the user.
- History. All ip-logged records of loggable actions involving the user.
There are two site settings that enable the extended download:
- `legal extended user download`: When enabled, the “Download All” feature in the User Activity page becomes an extended download.
- `legal extended user download admin`. When enabled, permitted staff can download all the information of any user of the site. They will see an new “Download All” button at the top of the the admin user information of each user.
The two settings are severable, i.e. you can enable `legal extended user download` without giving admins or staff the ability to download all the information of every user, or you can enable `legal extended user download admin` without giving users the ability to download all of their information.
The background for this feature is the EU’s GDPR. See in particular:
- [GDPR countdown and compliance](https://meta.discourse.org/t/gdpr-countdown-and-compliance/87190)
- [Providing data for GDPR](https://meta.discourse.org/t/providing-data-for-gdpr/83595)
- [Problematic IP address fields](https://meta.discourse.org/t/problematic-ip-address-fields/83785)
- [https://meta.discourse.org/t/gdpr-compliance-solution-pls-critique/87908](https://meta.discourse.org/t/gdpr-compliance-solution-pls-critique/87908)
Please note, that you should consider the security implications of allowing users, staff and / or admins the ability to download all of the listed information. This feature may not be suitable in all circumstances. Further information on this issue can be found in the topics above.
An alternative to directly downloading user information is for a relevant staff member (a “Data Protection Officer”) to compile the information via database queries.
**Disclaimer**
The Legal Tools plugin (the “Plugin”), and its author Angus McLeod (the “Author”), is not a lawyer or a substitute for a lawyer or legal advice. Communications between you and the Author are not protected by lawyer-client privilege or as work product. The Plugin, and its Author, cannot provide any kind of advice, explanation, opinion, recommendation or surety about possible legal rights, remedies, defences, options, selection of forms, or strategies.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [May 21, 2018, 12:35am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/2 "2018-05-21T00:35:43Z")
@codinghorror how much of this do you think belongs in core? Should we just amend our default “download all my data” to include all this stuff?
---
### Author:  [@codinghorror](https://meta.discourse.org/u/codinghorror)
#### Post date: [May 21, 2018, 1:10am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/4 "2018-05-21T01:10:29Z")
No, we need @riking to come through with the removal of IP where it’s unnecessarily recorded and should not be first. If that can’t be achieved in a reasonable timeframe we need to get someone else to do it.
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [May 22, 2018, 9:52pm UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/5 "2018-05-22T21:52:38Z")
Update here.
**Scope of the download**
All user-related records in `user activity` (record of likes, bookmarks, topics, replies etc) have been added to the download ([commit](https://github.com/angusmcleod/discourse-legal-tools/commit/7f9718a71e3a0159c66b5fe3a78f194bf72026e4)).
I initially didn’t add this as it seemed like overkill. However @RGJ raised it with me and we had a productive exchange on the question.
Essentially, we decided that the best approach for the purposes of this feature was to include all records of activity tied to the user’s identifier that don’t entail countervailing concerns about the privacy of others or similar rights.
I would emphasise “for the purposes of this feature”, as the purpose of this feature is to take a ‘maximalist’ approach to possible interpretations of the GDPR. It does not attempt to parse ‘likely’ approaches. I’ve laid out some of my own views on the ‘likely’ approaches in [this topic](https://meta.discourse.org/t/providing-data-for-gdpr/83595) (which remain unchanged).
The specifics of the reasoning behind this ‘maximalist’ approach are:
- The broadest interpretation of A.4.1 (the definition of ‘Personal Data’ in the GDPR) as it applies to Discourse is _any_ record in the db that contains the user’s `user_id`, i.e.:
- Read literally, this definition doesn’t care about how the data is produced (e.g. whether the user is acting or not). It merely requires the data to be related to the user’s identifier in some way.
- However, applying that literally would produce a fair amount of duplication (e.g. the records in the `directory_items` table are duplicative of various other entries).
- The point of the extended download is to guard against even the small risk that Article 4.1 could receive a very broad interpretation by _some_ authority or court in Europe.
- The factors against including it - size of download, potentially security (?) - do not outweigh the possible benefit of including it.
We also considered whether to include ‘administrative’ records with the user’s `user_id` such as flags, complaints and staff whispers. We decided against this, reasoning as follows:
- They’re already in the territory of information associated with the user purely by their identifier. They are not information about the user per se (i.e. name, email, age, location etc). This is already assuming a wide interpretation of A.4.1.
- Whether _administrative_ records intrude on the privacy of other parties, or other relevant concerns (i.e. R. 63.5 & A.15.4) must be determined on a case-by-case basis.
- Other parties, such as Facebook, do no include such data in their user download functionality.
---
### Author:  [@bartv](https://meta.discourse.org/u/bartv)
#### Post date: [May 24, 2018, 4:39pm UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/6 "2018-05-24T16:39:30Z")
Hey Angus,
I’m having some trouble getting this to work. I’ve enabled the ‘legal extended user download’ setting, refreshed the page and clicked the ‘Download all’ button on the activity page. This results in an archive that contains one CSV file with topics in it. I checked both as an admin and as a regular user. What is the expected output - multiple CSV files, each for a different table?
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [May 24, 2018, 11:16pm UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/7 "2018-05-24T23:16:14Z")
The expected output is a single csv with headers for each item mentioned in the first post in this topic. If you try it out on my sandbox, this is what you’ll get.
Do you see any errors at `/sidekiq`?
@RGJ have you had this issue?
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [May 25, 2018, 5:43am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/8 "2018-05-25T05:43:24Z")
> [@angus](#):
>
> @RGJ have you had this issue?
Nope. We had a few users with the same kind of complaint though but it turned out to be a false alarm.
I guess they never scrolled down past the posts. Maybe it’s an idea to move the posts section (i.e. the most unstructured / multiline content) to be the last section.
---
### Author:  [@bartv](https://meta.discourse.org/u/bartv)
#### Post date: [May 25, 2018, 6:24am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/9 "2018-05-25T06:24:21Z")
Could you provide a few of the header rows as an example so I can do a search for them?
---
### Author:  [@RGJ](https://meta.discourse.org/u/RGJ)
#### Post date: [May 25, 2018, 7:08am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/10 "2018-05-25T07:08:44Z")
See the ‘separator’ lines [here](https://github.com/angusmcleod/discourse-legal-tools/blob/master/lib/export_csv_file_extension.rb#L192-L234).
---
### Author:  [@bartv](https://meta.discourse.org/u/bartv)
#### Post date: [May 25, 2018, 7:12am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/11 "2018-05-25T07:12:05Z")
Thanks Richard. Definitely not seeing those. So just to be sure I’m doing the right thing: this plugin is supposed to replace the ‘Download’ button in the Profile \> Activity sidebar, right?
---
### Author:  [@tm2017](https://meta.discourse.org/u/tm2017)
#### Post date: [May 25, 2018, 10:00am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/12 "2018-05-25T10:00:29Z")
First words of admiration go to @angus👏 ❤ 👍 for doing such a great job helping everyone here to get those highly useful tools (not only this plugin but other plugins too).
I’ve got one question though: wouldn’t it be better to have this export available to admins only (at least as option for those concerned)?
Isn’t it potentially risky in case when given account password is compromised and then ‘all activity’ is easily downloaded by unauthorized person? (Sorry that is two questions 🙂 )
---
### Author:  [@bartv](https://meta.discourse.org/u/bartv)
#### Post date: [May 25, 2018, 11:59am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/13 "2018-05-25T11:59:43Z")
I just did a few more tests and tried it on your sandbox too. My results are just the default download - they don’t match your new format. I don’t _think_ I see any errors in sidekiq (at least not in ‘Failed’ - ‘Errors’ is not clickable). Any suggestions how I can best find more information here?
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [May 25, 2018, 11:28pm UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/14 "2018-05-25T23:28:21Z")
> [@bartv](#):
>
> I just did a few more tests and tried it on your sandbox too. My results are just the default download - they don’t match your new format
Please post (or PM) a screenshot of the csv you got from my sandbox with the ip addresses blacked out (if any).
> [@tm2017](#):
>
> I’ve got one question though: wouldn’t it be better to have this export available to admins only (at least as option for those concerned)?
I think I’ll make this a setting.
> [@tm2017](#):
>
> Isn’t it potentially risky in case when given account password is compromised and then ‘all activity’ is easily downloaded by unauthorized person?
Potentially. This is partly what I meant by
> [@angus](#):
>
> Please note, that you should consider the security implications of allowing users to directly download all the listed information.
Nevertheless, some people will need / want the functionality of allowing users to directly download their info. So I think a setting is the move here.
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [May 26, 2018, 5:36am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/15 "2018-05-26T05:36:51Z")
There are now two site settings that enable the extended download:
- `legal extended user download`: When enabled, the “Download All” feature in the User Activity page becomes an extended download.
- `legal extended user download admin`. When enabled, permitted staff can download all the information of any user of the site. They will see an new “Download All” button at the top of the the admin user information of each user.
The two settings are severable, i.e. you can enable `legal extended user download` without giving admins or staff the ability to download all the information of every user, or you can enable `legal extended user download admin` without giving users the ability to download all of their information.
Due to the security implications of allowing users (even if they’re admins) to download all the information of other users, I’ve been particularly careful with protecting the admin extended user download server methods, however I would appreciate a second set of eyes on that aspect of the changes, particularly considering this feature seems to be quite popular. Perhaps you could take a look @riking if you have the time? (i.e. particularly the [changes to the Guardian](https://github.com/angusmcleod/discourse-legal-tools/blob/master/lib/export_csv_file_extension.rb#L21)).
I’ve also added text to the top of the extended csv:
- A header (can be edited: Customize \> Text Content \> “csv\_export.extended.title”):
- A note below the header (can be edited: Customize \> Text Content \> “csv\_export.extended.note”):
This has all been tested on my [sandbox](https://discourse.angusmcleod.com.au/) where it is currently live.
@bartv Let me know if these updates fix your issue.
---
### Author:  [@bartv](https://meta.discourse.org/u/bartv)
#### Post date: [May 26, 2018, 7:06am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/16 "2018-05-26T07:06:32Z")
This new version solves the issue I had before. Thanks for all your hard work on this, Angus!
The extended note is interesting; could you shed some light on which information has been excluded and for which reasons?
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [May 26, 2018, 7:20am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/17 "2018-05-26T07:20:53Z")
That is primarily referring to the exclusion mentioned below, and also helps if we’ve not included something some authority happens to consider relevant.
> [@angus](#):
>
> We also considered whether to include ‘administrative’ records with the user’s `user_id` such as flags, complaints and staff whispers. We decided against this, reasoning as follows:
>
> - They’re already in the territory of information associated with the user purely by their identifier. They are not information about the user per se (i.e. name, email, age, location etc). This is already assuming a wide interpretation of A.4.1.
>
> - Whether _administrative_ records intrude on the privacy of other parties, or other relevant concerns (i.e. R. 63.5 & A.15.4) must be determined on a case-by-case basis.
>
> - Other parties, such as Facebook, do no include such data in their user download functionality.
---
### Author:  [@riking](https://meta.discourse.org/u/riking)
#### Post date: [September 30, 2020, 3:15pm UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/18 "2020-09-30T15:15:25Z")
6 posts were split to a new topic: [User data export failing (transaction aborted)](https://meta.discourse.org/t/user-data-export-failing-transaction-aborted/165838)
---
### Author:  [@GuidoD](https://meta.discourse.org/u/GuidoD)
#### Post date: [January 8, 2021, 2:48pm UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/22 "2021-01-08T14:48:37Z")
As admin I just tried to export data from a user (a GDPR request) by clicking on the right button (download all) but I got a message from the system that the data export did fail and that I should control the logs.
I cannot figure out if this is now a problem inside the plugin or a problem of Discourse itself.
Update January 11 17.40 UTC: The discourse team just said that looking at the backtrace it is a problem in a third party plugin. @angus can you please have a look at this, thanks.
The Discourse instance is running on 2.7.0.beta1 ( [4f72830eb0](https://github.com/discourse/discourse/commits/4f72830eb02ec063866a2cd882b5c2dd0ede5800) )
Update January 11th 16:15 UTC: now on version 2.7.0.beta1( [422f395042](https://github.com/discourse/discourse/commits/422f395042ef7f32d528a2c11e8da619fdbd5641) ) but the error remains the same.
This is what is configured inside the plugin:
In the logs I do see the following error:
Job exception: undefined method `collect’ for nil:NilClass
```
/usr/local/lib/ruby/2.7.0/csv/writer.rb:46:in `<<'
/usr/local/lib/ruby/2.7.0/csv.rb:1230:in `<<'
/var/www/discourse/app/jobs/regular/export_csv_file.rb:66:in `block (3 levels) in execute'
/var/www/discourse/plugins/discourse-legal-tools/lib/export_csv_file_extension.rb:267:in `user_archive_export_extended'
/var/www/discourse/plugins/discourse-legal-tools/lib/export_csv_file_extension.rb:226:in `admin_user_archive_export'
/var/www/discourse/app/jobs/regular/export_csv_file.rb:66:in `each'
/var/www/discourse/app/jobs/regular/export_csv_file.rb:66:in `block (2 levels) in execute'
/usr/local/lib/ruby/2.7.0/csv.rb:658:in `open'
/var/www/discourse/app/jobs/regular/export_csv_file.rb:64:in `block in execute'
/var/www/discourse/app/jobs/regular/export_csv_file.rb:63:in `each'
/var/www/discourse/app/jobs/regular/export_csv_file.rb:63:in `execute'
/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'
rails_multisite-2.5.0/lib/rails_multisite/connection_management.rb:76:in `with_connection'
/var/www/discourse/app/jobs/base.rb:221:in `block in perform'
/var/www/discourse/app/jobs/base.rb:217:in `each'
/var/www/discourse/app/jobs/base.rb:217:in `perform'
sidekiq-6.1.2/lib/sidekiq/processor.rb:196:in `execute_job'
sidekiq-6.1.2/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
sidekiq-6.1.2/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'
/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'
sidekiq-6.1.2/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'
sidekiq-6.1.2/lib/sidekiq/middleware/chain.rb:143:in `invoke'
sidekiq-6.1.2/lib/sidekiq/processor.rb:163:in `block in process'
sidekiq-6.1.2/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/job_retry.rb:111:in `local'
sidekiq-6.1.2/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq.rb:38:in `block in '
sidekiq-6.1.2/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/processor.rb:257:in `stats'
sidekiq-6.1.2/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/job_logger.rb:13:in `call'
sidekiq-6.1.2/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
sidekiq-6.1.2/lib/sidekiq/job_retry.rb:78:in `global'
sidekiq-6.1.2/lib/sidekiq/processor.rb:124:in `block in dispatch'
sidekiq-6.1.2/lib/sidekiq/logger.rb:10:in `with'
sidekiq-6.1.2/lib/sidekiq/job_logger.rb:33:in `prepare'
sidekiq-6.1.2/lib/sidekiq/processor.rb:123:in `dispatch'
sidekiq-6.1.2/lib/sidekiq/processor.rb:162:in `process'
sidekiq-6.1.2/lib/sidekiq/processor.rb:78:in `process_one'
sidekiq-6.1.2/lib/sidekiq/processor.rb:68:in `run'
sidekiq-6.1.2/lib/sidekiq/util.rb:15:in `watchdog'
sidekiq-6.1.2/lib/sidekiq/util.rb:24:in `block in safe_thread'
```
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [January 12, 2021, 5:55am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/23 "2021-01-12T05:55:39Z")
Apologies for the slow reply here, I’ve been away 🏖.
I repro’d this and fixed it. Please update and try the download again. If possible, let me know how the request goes.
[https://github.com/paviliondev/discourse-legal-tools/commit/a15193e9267ee5aee2d7eb8e5dc6c3b0d18d3b23](https://github.com/paviliondev/discourse-legal-tools/commit/a15193e9267ee5aee2d7eb8e5dc6c3b0d18d3b23)
---
### Author:  [@GuidoD](https://meta.discourse.org/u/GuidoD)
#### Post date: [January 12, 2021, 8:47am UTC](https://meta.discourse.org/t/legal-tools-plugin/87966/24 "2021-01-12T08:47:13Z")
> [@angus](#):
>
> I’ve been away
no problem, it was a weekend and nobody has to react right away.
I am thankful for this plugin.
> [@angus](#):
>
> Please update and try the download again. If possible, let me know how the request goes.
I did update and tried it again.
It worked fine, thanks a lot for the fix, really appreciated your good work.
If I would like to also include ‘administrative’ records with the user’s `user_id` such as flags, complaints and staff whispers, what would be the best SQL statements to get the data with the [data explorer](https://meta.discourse.org/t/32566?silent=true)?
If needed I could then provide this data separately to the user who requested the data.
[Next page](https://meta.discourse.org/t/legal-tools-plugin/87966.md?page=2)