Plugin to scrape news from other sites

[f1r4s (Eddie)]

Hello,

Is there any plugin to scrab news from other sites ?

[merefield (Robert)]

There’s: News Plugin 📰

Which uses RSS feeds

[Falco]

The Configure the Discourse RSS Polling Plugin can scrape many sites, like blogs, youtube channels and playlists, and automatically create new topics just a few minutes after content is posted.

[f1r4s (Eddie)]

I installed the plugin; and follow the guide as below;

i put 3 sections to test working rss from bleeping security site;

1st category rss source
https://www.bleepingcomputer.com/news/security
category filter: none.
username selected; and category to post rss selected.

second category rss source
BleepingComputer
category filter: security
username selected; and category to post rss selected.

also i use another site rss

darkreading
category filter: vulnerabilities-threats
username selected; and category to post rss selected.

but nothing work with me ??

[tobiaseigen]

Did you go to sidekiq to run the process? If not then it will take a while, depending on your admin settings. Also look in the logs for any errors.

You also have to be sure to use an rss url. If you open thise urls they don’t seem to be rss. There is also a setting that makes imported topics unlisted by default.

[f1r4s (Eddie)]

sorry, i miss that part; the logs show me this

[f1r4s (Eddie)]

I see now only the third rss (darkreading.com) is working to start posting; and it’s post pretty good more than 100+ post, but all the posts looks like this

value:

https://www.darkreading.com/rss.xml
category filter: vulnerabilities-threats
username selected; and category to post rss selected.

how can i include full post with pictures ?

[justin (Justin DiRose)]

@f1r4s and @Jagster, keep it civil here.

While there are legitimate uses for wanting to pull in content from other sites, such as for an internal community where you want to monitor important security news, we do not condone copyright infringement.

Discourse community owners are responsible to run their site in accordance with all governing laws and host terms of service, just like with any other site on the web.

[Falco]

@f1r4s

how can i include full post with pictures ?

Try toggling the embed truncate site setting.

@f1r4s

also i use another site rss

…

but nothing work with me ??

Some sites have a weird RSS markup, so you will have to manually debug broken ones.

[f1r4s (Eddie)]

I recommended using second RSS but the images are broken in scrapping…