scraping the site for fun

Loneliness, Depression & Relationship Forum

Help Support Loneliness, Depression & Relationship Forum:

This site may earn a commission from merchant affiliate links, including eBay, Amazon, and others.

mickey

Well-known member
Joined
Jan 4, 2015
Messages
432
Reaction score
2
I have developed an interest in applied statistics, especially data science (if you accept that the concept of data science is valid, which not everyone does). Right now I'm informally studying math, including statistics, as well as programming and general data science techniques. It will probably take me about two more years before I have the skills needed to do actual research, and even then I'll be doing it solo for fun, but in the meantime I'd like to practise some of the skills involved.

One practice involved in some kinds of data science is called "scraping," which is when you download webpages into html files on your hard drive. It's usually automated with a computer program or ready-made software, which is fairly easy to find. It's done all the time but, except when used by for-profit businesses to do research on competitors, it's considered ethical to ask permission of a website's owner before scraping the website. What I'd like to do is ask permission of everyone involved in A Lonely Life Forums, including all users who choose to respond, to scrape the public, non-identifying, non-sensitive and non-confidential parts of the forum just to practise data gathering, and also data cleaning (which is assembling the data into tables). If anyone whatsoever is not comfortable with this, and/or if the website's owner says no, I shan't do it. Let me know.
 
Well it's a negative from myself on anyone "scraping" so called "data" on anything i have posted on here,99%of which would be considered crap anyway, but,personaly,i see it as "Big Brother" tactic's. Any and all postings or threads put on here,are for people to find help,advice,support,and friendship,anything that has been posted,should not be "harvested" for "fun" or any other reason.My own personal feeling is that it is a intrusion.So please leave my crap where it is!

Have a nice day. :)
 
I don't care. It's nice of you to ask permission though. Scraping can be fun. I made a scraper for a tech recruiting company once. It would go on linkedin and find all of the qualified people for whatever they were searching for. Then it would make an easy to read CSV file with all of the candidate's contact information and a few other descriptions from their profile. The best part was exploiting the "people you might know" feature of the site to get access to all of the profiles, even if you weren't linked to them in some way.

Have fun. The data is public so you can so whatever you want with it. The most useful thing to learn to help with cleaning up data is how to work with the DOM and how to use Xpath to get to specific information. Using Xpath can be a little tricky at first but you almost need to know how to use it to do most things.
 
You do not have my permission to harvest any of my data from ALL. Some of the stories I've shared about life got published in the past, so they are my copyright.

As if the Google spider isn't pesky enough. :(
 
Since people have said no I'm not going to do it. But you shouldn't be under the delusion that no one else is doing it without bothering to inform you of what they're doing. And they don't care about "spying on you" personally. They're gathering generic data in which you're just a statistic and considered individually uninteresting. Data science is not espionage but statistics. Just correcting a misimpression people seem to have. Still, as I said, since you said no I'm not going to do it, even if it's almost a certainty other people are doing it. Have a nice day.
 
I don't doubt people have already copied my data, but if they publish on the internet without my express permission, then they shall bend over and receive my size 7's where the sun don't shine.

All I did was post an opinion for gawd's sake. Time to edit my Ignore List.
 
"...public, non-identifying, non-sensitive and non-confidential parts of the forum "

But you'd be the one deciding what part of the website fit this criteria, where other people might not agree.
 
h3donist said:
"...public, non-identifying, non-sensitive and non-confidential parts of the forum "

But you'd be the one deciding what part of the website fit this criteria, where other people might not agree.

Yeah I agree with this. I wouldn't be comfortable with that at all.
 
It's data is out there to anyone that visits the site. There isn't really anything to stop someone from doing it.
 

Latest posts

Back
Top