Gran cantidad de errores en /user-badges.json

Soporte Instalación
1

Aquí estoy de nuevo molestandoles a ustedes.

Nuestros usuarios están formados por una buena cantidad de personas con conocimientos técnicos y han planteado este problema que yo había notado hace algún tiempo, pero decidí posponerlo ya que también estoy ocupado con asuntos que no son de administración del foro :roll_eyes:

De todos modos, hoy es un día en el que tengo gente empacando nuestra casa antes de una mudanza, así que de todas formas no puedo trabajar.

Hace bastante tiempo que noté esto, pero no logro entender por qué ocurre solo para el endpoint específico que devuelve las insignias de los usuarios.

Tengo un registro de 224k filas que puedo pasar al soporte si es necesario, pero un extracto que lo resume se puede publicar aquí:

2023/07/23 00:31:58 [error] 69#69: *12563 limiting requests, excess: 12.164 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Quiroga.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:58 [error] 67#67: *7122 limiting requests, excess: 12.140 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/kAr.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *11749 limiting requests, excess: 12.844 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/LorenzoLamas.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13835 limiting requests, excess: 12.844 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Predy.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *12563 limiting requests, excess: 12.808 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/licher.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 67#67: *7122 limiting requests, excess: 12.772 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/JaKo.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *8391 limiting requests, excess: 12.592 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/Quiroga.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 68#68: *13829 limiting requests, excess: 12.664 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/OrangE.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13842 limiting requests, excess: 12.412 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/CrazyWildhog.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *11749 limiting requests, excess: 12.424 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Clive.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13835 limiting requests, excess: 12.400 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/MARGIO.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 67#67: *7122 limiting requests, excess: 12.176 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/licher.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 68#68: *13829 limiting requests, excess: 12.164 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Cardoza.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *11749 limiting requests, excess: 12.164 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Lord_Phobos.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13835 limiting requests, excess: 12.140 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Nomeacaso.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *12559 limiting requests, excess: 12.936 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/MARGIO.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13836 limiting requests, excess: 12.924 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/Nemo.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 68#68: *13830 limiting requests, excess: 12.020 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/LorenzoLamas.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *11749 limiting requests, excess: 12.544 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Nightmare.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13835 limiting requests, excess: 12.484 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/MARGIO.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13836 limiting requests, excess: 12.232 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/Lord_Phobos.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *12563 limiting requests, excess: 12.316 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/inglo.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *8391 limiting requests, excess: 12.148 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/PogueMahone.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 68#68: *13830 limiting requests, excess: 12.316 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/OrangE.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 67#67: *7122 limiting requests, excess: 12.100 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/JaKo.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *11749 limiting requests, excess: 12.088 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/licher.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13835 limiting requests, excess: 12.016 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Sheika.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *13836 limiting requests, excess: 12.680 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/CrazyWildhog.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 69#69: *8391 limiting requests, excess: 12.296 by zone "flood", client: 162.158.129.16, server: _, request: "GET /user-badges/Lord_Phobos.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 68#68: *13830 limiting requests, excess: 12.380 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/OrangE.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 67#67: *7122 limiting requests, excess: 12.128 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/licher.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 66#66: *11749 limiting requests, excess: 12.056 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/RisVIII.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"
2023/07/23 00:31:59 [error] 68#68: *13829 limiting requests, excess: 12.420 by zone "flood", client: 162.158.129.17, server: _, request: "GET /user-badges/Chuck.json HTTP/2.0", host: "OMITTED.COM", referrer: "https://OMITTED.COM/t/la-volta-in-cui-ah-non-avevo-capito-un-cazz/838164/78"

OMITTED.COM es, por supuesto, el nombre de host real :stuck_out_tongue:

Esto es lo que las métricas de Prometheus también me muestran, que, en mi opinión, es un número de solicitudes de error definitivamente demasiado alto.

image
image960×301 53.6 KB

image
image952×297 65.9 KB

Pensé en los límites de tasa, pero esto solo está ocurriendo para ese endpoint específico (quiero decir, significativamente más al menos) y, si fuera un problema de límite de tasa, ¿no debería ocurrir también en otras GET(s)?

Estoy tratando de entender qué podría estar causando esto, ante todo. Se agradece cualquier pista o solicitud de información adicional.

Solo ten en cuenta que es posible que no sea tan receptivo esta semana, ya que literalmente me estoy mudando de casa en los próximos 3-4 días :weary:

¡Saludos!

2

Por si sirve de ayuda, y esperando que no sea inoportuno, aquí tenéis una instantánea de ese hilo, que tiene 185 publicaciones:

image
image591×166 46.8 KB

3

Gracias @Ed_S, siempre que el nombre de host no esté en texto plano, todo está bien. Simplemente no quiero que los bots/aspirantes a hackers se lo pongan demasiado fácil y se diviertan un poco con un pequeño foro comunitario.

Ya estoy dedicando demasiado tiempo en comparación con lo que esperaba, para ello :laughing:

4

Parece que está usando la función de proxy inverso HTTP de Cloudflare, pero no tiene nginx configurado para usar las IP reales que los servidores de Cloudflare envían para las solicitudes, lo que podría causar problemas (si cada solicitud parece provenir de la misma fuente, todos sus usuarios se verán afectados por los límites de velocidad, incluso si solo uno está enviando solicitudes excesivas).

5

Gracias por la pista, ¿tienes algún tema/guía que pueda leer sobre cómo solucionar esto?

El proxy inverso ha sido una solicitud de quien realmente posee el VPS para no exponer la IP real.

6

¿Añadir esa plantilla hace que funcione correctamente?

7

Reconstruyendo ahora. Lo dejaré funcionando durante la noche y volveré a revisar mañana/lunes (todavía estoy en medio de la mudanza y desempaquetando :weary:)\n\nEditar: Una rápida comprobación inmediatamente después de la reconstrucción parece haber resuelto el problema :+1: \n\n¡Actualizaré en un par de días de todos modos!

8

«Un par de días después»: sí, habilitar la plantilla adicional para Cloudflare resolvió el problema de que esas solicitudes de insignias fallaran.

También se resolvieron otros problemas, como el filtro de IP para administradores/moderadores que mostraba a todos viniendo de la misma IP.

Sin embargo, el monitoreo que obtengo de los scrapes de Prometheus muestra muchos errores de todos modos.
Tendré que encontrar algo de tiempo para revisar los registros de nginx, supongo.

image
image963×305 45.4 KB

image
image943×295 73.9 KB