Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add built-in image proxy & misc stuff #91

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add built-in image proxy & misc stuff #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/assets/js/missingroutes.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ document
const json = await res.json();
const missingRoutes = json.routes;

document.getElementById("counter").innerHTML =
document.getElementById("counter").textContent =
`We implement ${json.discord - json.missing}/${
json.discord
} endpoints from Discord.com ` +
Expand Down
8 changes: 4 additions & 4 deletions docs/contributing/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

## Structure

{{ project.name }} is written in Typescript and is comprised of 4 main parts:
{{ project.name }} is written in TypeScript and is comprised of 4 main parts:

- REST HTTP API server
- Websocket Gateway server for realtime communication with clients
Expand All @@ -29,7 +29,7 @@ Generally, the approach is to just see what the Discord.com client sends and rec
and guessing about any functionality server-side, if it's undocumented.

For a lot of things it's pretty simple to guess, `GET /api/users/@me` returns private details about your user for example.
This route is also detailed in [Discords own documentation](https://discord.com/developers/), [here specifically](https://discord.com/developers/docs/resources/user#get-current-user).
This route is also detailed in [Discords own documentation](https://discord.com/developers/docs/intro), [here specifically](https://discord.com/developers/docs/resources/user#get-current-user).

Discord generally does not document anything that is not related to application/bot development, though.
As an example, `GET /api/updates?platform={}` which returns the `url`, `pub_date`, `name` and any `notes` about the latest client release for a platform.
Expand All @@ -40,5 +40,5 @@ Easy fix though, just edit the `DeveloperOptionsStore` localStorage key so that

!!! warning

Make sure you rerun `npm run build` every time you edit source code. Additionally, make sure you run `npm run generate:schema` whenever you change a
schema. If you want to do both, there's a shortcut: `npm run setup`.
Make sure you rerun `npm run build` every time you edit source code, or just use `npm run watch` to make TypeScript automatically recompile on code changes.
Wenn making changes to schemas or HTTP routes, run `npm run generate:schemas` and `npm run generate:openapi` to update the schemas used for validating incoming requests and generating the API documentation.
10 changes: 5 additions & 5 deletions docs/contributing/instances.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ Your instance:
5. Must have at least regular uptime, meaning it is available at a consistent time of day.
6. Must have a valid and monitored [`general_correspondenceEmail` config](/setup/server/configuration) set.
7. Must not have default [rights](/setup/server/security/rights) that include operator or other administrative rights.
8. Enable [Imagor](/setup/server/configuration/imagor), as no image proxy allows attackers to learn user IP addresses.
8. Use an [image proxy](/setup/server/configuration/imageProxy), e.g. Imagor, as no image proxy allows attackers to learn user IP addresses.
9. Have a valid SSL/TLS certificate for all endpoints.

We recommend (not required) that you:

- Enable [Email verification](/setup/server/email), for anti-spam purposes
- Enable [Captcha](/setup/server/security/captcha), for anti-spam purposes
- Run your instance under [SystemD](/setup/server/systemd) or a similar system in your distro, for automatic restarting
- Provide some mechanism for users to report content. This may be as simple as more openly advertising your correspondence email (i.e. outside `GET /api/policies/instance` or `/api/ping`)
- Enable [Email verification](/setup/server/email), for anti-spam purposes.
- Enable [Captcha](/setup/server/security/captcha), for anti-spam purposes.
- Run your instance under [SystemD](/setup/server/systemd) or a similar system in your distro, for automatic restarting.
- Provide some mechanism for users to report content. This may be as simple as more openly advertising your correspondence email (i.e. outside `GET /api/policies/instance` or `/api/ping`).
- Provide some mechanism for instance status, such as [Grafana](https://grafana.com/).
- Host a [`/.well-known/spacebar`](/setup/server/wellknown) file on the domain you wish users associate with your instance, e.g. `spacebar.chat`.
If doing so, use this domain as the `url` field in your community instances PR.
6 changes: 3 additions & 3 deletions docs/contributing/server/missingroute.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Missing Routes

Below is a list of routes available on Discord.com {{ project.name }}-server does not currently implement.
Below is a list of routes available on Discord.com that the {{ project.name }} server does not currently implement.

It does not account for different HTTP methods (GET, POST, etc). A single method implemented by {{ project.name }} will remove it from this list,
It does not account for different HTTP methods (GET, POST, etc). A single method implemented will remove it from this list,
so be sure to double check in the {{ project.name }} [source code]({{ repositories.base_url }}/{{ repositories.server }}/tree/master/src/api/routes).

It is generated daily by [{{ repositories.missing_routes }}]({{ repositories.base_url }}/{{ repositories.missing_routes }}/),
It is generated automatically by [{{ repositories.missing_routes }}]({{ repositories.base_url }}/{{ repositories.missing_routes }}/),
by scraping the latest Discord.com client.

<div>
Expand Down
16 changes: 8 additions & 8 deletions docs/setup/bots/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ with the appropriate URLs of the instance you want to connect to.

You can get them from a client or from the [well-known](server/wellknown) instance endpoint.

### Discord.js
### discord.js

The `Client` class constructor accepts a `http` object, which you can use to change
the endpoints used.
The `Client` class constructor accepts configuration options that can be used to change
the endpoints.

```js
const { Client } = require("discord.js");
Expand All @@ -36,7 +36,7 @@ const client = new Client({
client.login("your token here");
```

### Discord.py
### discord.py

```py
import discord
Expand All @@ -57,13 +57,13 @@ client.run("your token here")
@NotNull
@Override
public String getGateway() {
return "wss://{REPLACE HERE WITH YOUR GATEWAY SERVER URL}/?encoding=json&v=9&compress=zlib-stream";
return "wss://gateway.{{ project.domain }}/?encoding=json&v=9&compress=zlib-stream";
}
```
5. Finally, configure JDA to use your RestConfig & SpacebarSessionController, like this:
```java
JDA jda = JDABuilder.createDefault("your token here")
.setRestConfig(restConfig)
.setSessionController(new SpacebarSessionController())
.build();
.setRestConfig(restConfig)
.setSessionController(new SpacebarSessionController())
.build();
```
5 changes: 3 additions & 2 deletions docs/setup/server/configuration/embeds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Embeds in {{ project.name }} are external content that is displayed within your
{{ project.name }} tries its best to fetch content from these external sites,
but sometimes we require an API key or other authentication with the site to display their content (or nicer looking embeds).

For external images, it's best to set up [Imagor](imagor.md) for image resizing.
For external images, it's best to set up [Imagor or another image proxy](imageProxy.md) for image resizing.
The client may or may not fetch images directly from their source if this is not set up,
and as such some users may not see all images.
In the case where a client does fetch an image from it's source, without Imagor an attacker
Expand All @@ -15,5 +15,6 @@ may be able to learn the IP addresses of users.
Go to the [Twitter developer portal](https://developer.twitter.com/) and sign up for developer access.
You must have a phone number attached to your account to sign up.
Create an application for use with the Twitter API (We don't need to post messages to Twitter itself).
Make sure to create a bearer token for authentication.
Make sure to create a Bearer token for authentication.

Once you have your API key, set the `external_twitter` [config](index.md) value to your API key, wrapped in quotes.
36 changes: 18 additions & 18 deletions docs/setup/server/configuration/env.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,21 @@ Below is a list of environment variables used by {{ project.name }}.
You can set environment variables easily by creating an `.env` file
in the `{{ project.name.lower() }}-server` folder, with the format `NAME=VALUE` with each on new lines.

| Name | Value | Description |
| ---------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------- |
| THREADS | number | Number of threads to run {{ project.name }} on when using bundle. Make sure you've enabled RabbitMQ if using more than one |
| PORT | number | Port to listen on. Used by all components, including bundle. If using bundle, all components run under the same port |
| DATABASE | string | Database connection string. Defaults to SQlite3 at project root |
| DB_SYNC | bool | Unsafely synchronise database model on startup. ***Not recommended for use in production environments!*** |
| CONFIG_PATH | string | File path for JSON config, if not using `config` db table |
| WS_LOGEVENTS | boolean | If set, log websocket events except messages from gateway |
| WS_VERBOSE | boolean | If set, log websocket messages sent/received by gateway |
| WS_DUMP | boolean | If set, dump websocket messages sent/received to disk |
| CDN | string | Lowest priority value for public CDN annoucements |
| GATEWAY | string | Lowest priority value for public gateway annoucements |
| STORAGE_LOCATION | string | CDN storage location. File path or S3 bucktet |
| STORAGE_PROVIDER | "s3" or "file" | CDN storage provider |
| STORAGE_BUCKET | string | S3 bucket name |
| STORAGE_REGION | string | S3 storage region |
| DB_LOGGING | boolean | if "true" logs all SQL queries to the terminal |
| LOG_REQUESTS | filter | What requests to log, per response code (eg. `-200` to log every non-200 response code, or `404` to log requests with a not found status code) |
| Name | Value | Description |
| ---------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| THREADS | number | Number of threads to run {{ project.name }} on when using bundle. Make sure you've enabled RabbitMQ if using more than one |
| PORT | number | Port to listen on. Used by all components, including bundle. If using bundle, all components run under the same port |
| DATABASE | string | Database connection string. Defaults to SQLite3 at project root |
| DB_SYNC | bool | Unsafely synchronise database model on startup. **Not recommended for use in production environments!** |
| CONFIG_PATH | string | File path for JSON config, if not using `config` db table |
| WS_LOGEVENTS | boolean | If set, log websocket events except messages from gateway |
| WS_VERBOSE | boolean | If set, log websocket messages sent/received by gateway |
| WS_DUMP | boolean | If set, dump websocket messages sent/received to disk |
| CDN | string | Lowest priority value for public CDN announcements |
| GATEWAY | string | Lowest priority value for public gateway announcements |
| STORAGE_LOCATION | string | CDN storage location. File path or S3 bucket |
| STORAGE_PROVIDER | "s3" or "file" | CDN storage provider |
| STORAGE_BUCKET | string | S3 bucket name |
| STORAGE_REGION | string | S3 storage region |
| DB_LOGGING | boolean | If "true", logs all SQL queries to the terminal |
| LOG_REQUESTS | filter | What requests to log, per response code (eg. `-200` to log every non-200 response code, or `404` to log requests with a Not Found status code) |
94 changes: 94 additions & 0 deletions docs/setup/server/configuration/imageProxy.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Image proxy

If the configuration `cdn_imagorServerUrl` is empty (which is the default), `proxy_url` is set to the same value as the original `url`, causing clients to download images directly from the image host.
This may lead to privacy concerns, as an attacker is able to learn users IP addresses.

To combat this, {{ project.name }} supports using an image proxy to serve and resize images.

## Imagor

[Imagor](https://github.com/cshum/imagor) is a "fast, secure image processing server"
used by {{ project.name }} for external image resizing, primarily by embeds from other websites when linked in a message.

### Dependencies

- [Docker](https://www.docker.com/)

### Setup

To setup Imagor for {{ project.name }}, first grab the `security_requestSignature` config value from {{ project.name }}'s database,
and create a `imagor.env` file somewhere safe, with the following content.
**Make sure to edit the file with the correct information**. Your requestSignture should _not_ start or end with `"`.

```
IMAGOR_SECRET=security_requestSignature value from your {{ project.name }} config
PORT=8000
```

You can now start Imagor with

```bash
docker run --env-file ./imagor.env -p 8000:8000 shumc/imagor
```

`8000` here is our port. Make sure that it'd available to people outside your network.
If you would like to change the port Imagor listens on, be sure to change both the PORT value in `imagor.env`,
and the `-p` value used in docker.

If you're using a [reverse proxy](../reverseProxy.md) such as Nginx for {{ project.name }} already, you could add this to your config's `server` block

```nginx
location /media/ {
# If you changed the port, be sure to change it here too
proxy_pass http://127.0.0.1:8000/;
}
```

Along with any additional config you already have, of course.
Alternative (and perhaps the better choice) would be to create a new domain, say `media.example.com` specifically for Imagor.

```nginx
server {
# Change the server_name to reflect your true domain
server_name media.example.com;

add_header Last-Modified $date_gmt;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_set_header X-Forwarded-For $remote_addr;

location / {
# If you had changed the port, change it here as well
proxy_pass http://127.0.0.1:8000;
}
}
```

Our last step is to simply tell {{ project.name }} about Imagor. Just set the `cdn_imagorServerUrl` config value to your public endpoint for Imagor, wrapped in quotes.

For example, if you used the `/media` location in your existing nginx config, it will look something like `"https://{{ project.domain }}/media"`.
If you used a subdomain, it will look like `"https://media.{{ project.domain }}"`.
Don't include a trailing backslash.

Congrats! After a restart, you've now got Imagor resizing your images!

## Built-in image proxy

While we recommend using Imagor, {{ project.name }} has a built-in image proxy that can be used instead of Imagor if you just want to test the server or are unable to use Docker.

To use it, simply set the `cdn_imagorServerUrl` to your API URL, but use `/imageproxy` instead of `/api`.
For example, if the API is running at `https://api.{{ project.domain }}/api`, you would set `cdn_imagorServerUrl` to `"https://api.{{ project.domain }}/imageproxy"`.

Note that to support resizing the image to the given format, it's required to install either [sharp](https://www.npmjs.com/package/sharp) (preferred) or [jimp](https://www.npmjs.com/package/jimp):

```bash
npm install sharp
```

## Your own image proxy

You can also point the `cdn_imagorServerUrl` [config](index.md) to your own image proxy, if you have one that supports the Imagor URL format:

1. Construct URL path: `<Image width>x<Height>/<Unencoded URL without https://>`
2. Create a SHA-1 HMAC of the URL path using `security_requestSignature`
3. Construct final `proxy_url`: `<cdn_imagorServerUrl configuration>/<SHA-1 hash>/<Path from above>`
Loading