Hector
(Héctor Fernández)
January 10, 2020, 1:22pm
6
It seems that some Instagram pages have a canonical link with a different url that requires login. The code in the onebox library prefers the canonical url.
end
def self.fetch_html_doc(url, headers = nil, body_cacher = nil)
response = (fetch_response(url, headers: headers, body_cacher: body_cacher) rescue nil)
Nokogiri::HTML(response)
end
def self.fetch_response(location, redirect_limit: 5, domain: nil, headers: nil, body_cacher: nil)
redirect_limit = Onebox.options.redirect_limit if redirect_limit > Onebox.options.redirect_limit
raise Net::HTTPError.new('HTTP redirect too deep', location) if redirect_limit == 0
uri = Addressable::URI.parse(location)
uri = Addressable::URI.join(domain, uri) if !uri.host
use_body_cacher = body_cacher && body_cacher.respond_to?('fetch_cached_response_body')
For example, these are the canonical urls of the instagrams in this topic:
https://www.instagram.com/shaparakgroup.ir/p/B3jG277hve9/caption/backhand_index_pointing_right-chp-shprkh-ok_hand-black_small_square-khrt-wyzyt-hy-brsh-khs
https://www.instagram.com/dogsofinstagram/p/B6gUJwDl-lQ/caption/happy-holidays-writes-konaandcamper-misslolathegoldendoodle-camera_with_flash-by-sh
6 Likes