I am looking to semi-automate the process of turning a nominations thread into a table of nominations. I am using python (specifically the beautiful soup library) to parse the html. The nominations thread consists of a bunch of posts by users in which they link to a thread or topic that they like. I have successfully written the code to scrape the nominations thread to find the username of the nominator, the link, and the picture(s) of the project. My routine can even handle posts with more than one link.
The roadblock I have reached is that if I follow the links, the resultant page will have a number of posts from before the post being linked. I assume this is to have enough previous info so that the user can scroll up after following the link. I can’t figure out how to spot the linked post or alternately modify the link so that it shows only the post that was linked. Anyone got any suggestions?
P.S. it’s a little rough right now, but I’ll be happy to share my code when I get it working.