Ah, fantastic! Let me start to look over the development docs and I’ll see what I can do. The example usage is spot on, even better than the old school “log in and generate” that I had in mind!
Do you have any tips/ examples/ templates that could help me? I have no idea what I’m doing, the most I can tell is that I might add
@headless = params[:headless] but then I don’t actually know what that’s doing… I think I would want to:
- add the parameter headless, with option to be empty (default) if not provided by the user (not headless mode)
- based on the API key, use the API secret to encrypt it? (Where is that?)
- if headless is true, print to the screen
heyo! My awesome collaborator is helping me out, so hopefully we can put our heads together and have a PR, maybe sometime over the weekend? I’ll ping you here if we run into trouble! His main language is Ruby, and he’s awesome, so I think for now we’re good! (and having fun too!)
hey @sam if we have a custom discourse repo, what is the quickest way to get a development server running? I’m looking at the docker-compose (and associated Docker images) from the main discourse image, and it’s quite a hairball. I don’t see any clear or easy way to combine the discourse base, the minidb, and minidb extras, and all the other dependencies that are stacked into there. What would you suggest?
Easiest way to get started is
hey @sam! This is fine if I just want to run the released discourse, but if I want to run a (development) version, from my Github repository, well I’ve been parsing through the dockerfiles this morning and it’s a nightmare - everything is a custom tool or command that has so many layers to reverse engineer I’m pulling my hair out. Ideally, we need an image (Dockerfile) that takes a discourse repository as an argument, and then generates the equivalent discourse:latest but with the custom respository. I can start with this docker-compose.yml
version: '2' services: postgresql: image: 'bitnami/postgresql:9.6' volumes: - 'postgresql_data:/bitnami' redis: image: 'bitnami/redis:latest' environment: - ALLOW_EMPTY_PASSWORD=yes volumes: - 'redis_data:/bitnami' discourse: image: 'bitnami/discourse:latest' labels: kompose.service.type: nodeport ports: - '80:3000' depends_on: - postgresql - redis volumes: - 'discourse_data:/bitnami' environment: - POSTGRESQL_HOST=postgresql - POSTGRESQL_ROOT_USER=postgres - POSTGRESQL_CLIENT_CREATE_DATABASE_NAME=bitnami_application - POSTGRESQL_CLIENT_CREATE_DATABASE_USERNAME=bn_discourse - POSTGRESQL_CLIENT_CREATE_DATABASE_PASSWORD=bitnami1 - DISCOURSE_POSTGRESQL_NAME=bitnami_application - DISCOURSE_POSTGRESQL_USERNAME=bn_discourse - DISCOURSE_POSTGRESQL_PASSWORD=bitnami1 sidekiq: image: 'bitnami/discourse:latest' depends_on: - discourse volumes: - 'sidekiq_data:/bitnami' command: 'nami start --foreground discourse-sidekiq' environment: - DISCOURSE_POSTGRESQL_NAME=bitnami_application - DISCOURSE_POSTGRESQL_USERNAME=bn_discourse - DISCOURSE_POSTGRESQL_PASSWORD=bitnami1 - DISCOURSE_HOST=discourse - DISCOURSE_PORT=3000 volumes: postgresql_data: driver: local redis_data: driver: local discourse_data: driver: local sidekiq_data: driver: local
And I’d ideally just want to replace the name of the image with one that I’ve built. That’s all I’ve been done (for 4 hours now this morning) and I’m not making any progress because it’s so complicated. Is there no simple Dockerfile to do this? It might seem easy to you because you wrote all this / have developed it too for years, so the most I can do is tell you how challenging this is for (someone with pretty good container and development experience) to unravel. I’m very frustrated that this is so complicated. How can we do better?
As an example, when I develop some Django application, and let’s say the repository is downloaded to /code, I can just map a volume at
$PWD:/code and then a restart to the container (via docker-compose) will update the app with my changes. Here in order to get to the source code addition I have to go all the way back to the base container:
And download the entire set of files that go into there, but then the container that uses it (discourse_dev) needs a whole set up of yaml configurations (also from the repository) h
which seem to be set up by various ruby scripts and there is no clear list of instructions for “how to deploy this with your own discourse.” If I try a different strategy to reproduce the discourse:latest (that has two base images of minideb and minideb-extras I again am hit with a custom install script from a packaged version, downloaded from a server that isn’t Github
# Install required system packages and dependencies RUN install_packages advancecomp ghostscript gifsicle hostname imagemagick jhead jpegoptim libbsd0 libc6 libcomerr2 libcurl3 libedit2 libffi6 libgcc1 libgcrypt20 libgmp-dev libgmp10 libgnutls30 libgpg-error0 libgssapi-krb5-2 libhogweed4 libicu57 libidn11 libidn2-0 libjpeg-progs libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libldap-2.4-2 liblzma5 libncurses5 libnettle6 libnghttp2-14 libp11-kit0 libpq5 libpsl5 libreadline7 librtmp1 libsasl2-2 libssh2-1 libssl1.0.2 libssl1.1 libstdc++6 libtasn1-6 libtinfo5 libunistring0 libxml2 libxml2-dev libxslt1-dev libxslt1.1 optipng pngcrush pngquant zlib1g zlib1g-dev RUN bitnami-pkg install ruby-2.4.5-0 --checksum 6b8fe1a5db54cc642125e96caf239329d27b2871cd0830a4158c312668a2afc7 RUN bitnami-pkg unpack postgresql-client-9.6.11-0 --checksum f1825273d8f7a0c88fecf6e3f91e9bc940ce1f8b69add7dfa14aca51d15209aa RUN bitnami-pkg install git-2.19.2-0 --checksum 24817a90223cfc91ce38cdc459dc6647dcf36754bf0300433ddb00ec276757ff RUN bitnami-pkg unpack discourse-sidekiq-2.1.4-0 --checksum 52ca8aabdf383a6b3958f7e85015e3c438bf6a5b431c510ded9642ac3532247a RUN bitnami-pkg unpack discourse-2.1.3-0 --checksum 714bb0e0a0f47a01fb5a61c1cdf71af5c6779abbfd5045f3d603e4a4c7539f9f
You do not want to run your own version of Discourse from your own github repo. I promise. For many of the reasons that you have pointed out. You want to develop a plugin. Maybe you should see Beginners Guide to Install Discourse for Development using Docker and Beginner's Guide to Creating Discourse Plugins - Part 1
See also Can Discourse ship frequent Docker images that do not need to be bootstrapped?. Bitnami is out of date and not supported here.
haha yes, that describes all my frustrations quite well.
So - an update to the controller script that @sam pointed out here:
- discourse/user_api_keys_controller.rb at 06d1b19ca2d8994d444ebfa3bb17634a4ae5ea7b · discourse/discourse · GitHub
You are saying this should be done by way of a plugin? And I can’t update the code there, and then test it locally? is there no logical way? As a developer, and given the discourse/discourse is the core codebase, I would suggest that there needs to be an avenue to do this.
And given yes to the above - how can the software stack be reproducible?
The only case that it makes sense to change the codebase rather than make changes in a plugin is if your change will be accepted in core. Otherwise, you’ll have to merge Discourse changes into your own, everyone who has done so was very sorry and spent lots of time and money undoing those changes.
You want to make changes in a plugin.
Correct - that is my use case. I am testing code that will be a PR to core, and I’d like to have a development server to test before doing a PR. I could PR blindly but that is not something I have ever done, it seems like bad practice.
Then follow any of the development #howto documents.
Please read the topic closely, that system is out there for hacking on you locally checked out source tree, strongly recommend not doing this docker compose bitnami hack
okay, I didn’t realize the bitnami code was a hack!
@sam I have a few more questions on the API call. How does the client_key fit in to this? Do you mean that helpme is registered as an application somewhere, and then given a key? And how is this distributed to the user?
I’m trying to map this to my understanding of OAuth2. With OAuth2 the provider would register the application (with a client id and secret) and the user doesn’t need to create a key. For example, to get a Github OAuth2 for Singularity Hub I:
- register the application in Github
- am given a client secret and key
- add the secret and key to Singularity Hub
- Singularity hub uses the key/secret to create tokens (token and refresh) on the user’s behalf
In our case, we are only dealing with one provider - discourse - and so there is no “Github” to register the application to. But maybe something like this?
- register the helpme client on a discourse board (this is like the board giving approval for helpme to be used)
- the discourse board turns on the endpoint to accept generation of tokens (and generated a client_key? where does it go / how is it used?)
- the user goes through the authentication flow (accepting in browser) to get a token
- the token is valid for some scope to post questions, etc.
I’m not sure where a public and private key fit in here, because it’s not typical to ask every user to do that, and actually would probably be too much and deter people from using it. The first validation is by way of the user accepting in the web interface, and the following come from the API token used after that. Could we talk about the generation and then use of the various keys/ secrets, and how you see this working?
I’m working on this a bit as well.
What’s I’ve got so far is:
- Discourse admin configures “Allowed user api auth redirects” to their own host ( i.e. https://your-discourse.org/user-api-key ).
- An additional show action in the user_api_key controller shows the payload ( which the user will use to copy-paste into an environment variable)
Sounds like there’s a complication with the requirement for an RSA key pair. Since this isn’t a bundled mobile app, users would have to generate their own keys to get the token. Would it be possible to make this key pair optional if we are just redirecting back to the discourse app instead of a 3rd party? Likelihood of a man in the middle seems low, but I’m definitely unfamiliar with the requirements of this.
Keep in mind nothing is bundled, you hand the server your public key, it then encrypts your token using it. I know it may feel a bit overkill but I do not want to amend the protocol here. So when you “show” the api key to the user actually show the encrypted api key. Your CLI can then decrypt it cause it holds the private key.
hey everyone! A quick question. We have a development discourse set up, and I’m testing a call to request a (terminal given) token. So my question - what is a nonce? It looks like a unique id of some kind, but I’m wondering how / when / where it’s generated (so I can do it on behalf of my user, etc.) Thanks!
A nonce can be anything, a random string. You use is to protect against replay attack.
When we give you your encrypted payload you check the nonce matches the one you expect, if it does not well, someone is replaying an old encrypted token. Clearly for your use this level of security is … a bit … overkill…
Cool thanks! I’ll generate a uuid.uuid4() string. I think @fitz and I are probably close - he’s in a different time zone and we will touch bases soon, but hopefully we will have something working and for review! And if we have more questions, thank you again in advance for being so quick to help.
Ok, looping back to this…
I’ve put something together here : Make auth_redirect param options on user_api_keys · cfitz/discourse@f5c631d · GitHub
Makes the auth_redirect param optional, so that the payload will just be returned after the UserApiKey is created.
Does this look alright?
Should I add something to allow admins to disable this feature? Should I add some additional text to be displayed in the new.html.erb?
Thanks @fitz! I can confirm that I can get a token with the helpme client I’m developing - should it be a string on multiple lines that we join by newline or just empty space (to generate one long string)?
Currently, I’m sending the api_username and api_token as part of the body (in data) and the requests that I made previously (using my Dockerized discourse with an admin key) are returning 403, Forbidden. Is your instance expecting a different post type (perhaps in headers or something different?) Or am I not parsing the key from the web interface correctly?
I’m overall wondering how the public key helps in this interaction. I’m guessing that since I’m logged into the discourse instance in my browser, an association is made between my login and the token that is generated? Didn’t we want to do some kind of token generation and/or using a key to sign, encrypt, and send, and then the server decodes?
We put together a PR here
Hope this helps.