View Code? Open in Web Editor
NEW
A web-scraping-based python package that enables you to scrape data from various platforms like GitHub, Twitter, Instagram, or any useful website.
Home Page: https://pypi.org/project/scrape-up/
License: MIT License
scrape-up's Issues
Proposed Method:
.opened_by()
Description
Get the name of the user who opened the particular issue.
To what class it belongs?:
Issue
Make changes at
src/scrape_up/github/issue.py
Adding a feature, so that the program also fetches user bio. I want to work on this issue under JWOC
Proposed Method: .get_contributors()
Description
.get_contributors()
will scrape out the contributors of a repository.
To what class it belongs?: Repository
Example Usage
from scrape_up import github
repo = github .Repository (username = "Clueless-Community" , repository = "scrape-up" )
print (repo .get_contributors ())
# Output - Starts listing all contributors......
I want to work on this issue under JWOC.
Write a method to fetch the numbers of contributions done by a user in the last year (a 365 days time frame)
Example
Proposed Method:
.commits()
Description
Get the
To what class it belongs?:
PullRequest()
Note
Add the method in github/pull_request.py
Is your feature request related to a problem? Please describe.
I can see that the website related to this repo is not deployed anywhere.
Describe the solution you'd like
Deploy the website on cloud using Render since Heroku is now paid.
Assign this issue to me under JWOC
Proposed Method:
.reviewers()
Description
Get the list of reviewers assigned in a PR, if any.
To what class it belongs?:
PullRequest()
Note
Add the method in github/pull_request.py
Proposed Method: .
fork_count()
Description
Create a method that returns the fork count of a repository.
To what class it belongs?:
Repository
Proposed Method:
.tags()
Description
Create a method to get the list of tags used in a repository.
To what class it belongs?:
Repository
Proposed Method:
.labels()
Description
Returns a list of labels given to an issue.
To what class it belongs?:
Issue
Make changes at
src/scrape_up/github/issue.py
Proposed Method:
.commits()
Description
Create a method to get the number of commits made in a repository.
To what class it belongs?:
Repository
Proposed Method:
.files_changed()
Description
Get the number of files changed in a PR made.
To what class it belongs?:
PullRequest()
Note
Add the method in github/pull_request.py
Proposed Method:
.issues_count()
Description
Create a method to get the number of issues opened in a repository.
To what class it belongs?:
Repository
Adding Instagram with follower post
Proposed Method:
.about()
Description
Create a method to get the details of a repository, that is the about the section.
To what class it belongs?:
Repository
Proposed Method:
.labels()
Description
Returns the list of labels given in a PR, return an empty list of no labels are given.
To what class it belongs?:
PullRequest()
Note
Add the method in github/pull_request.py
Proposed Method:
.opened_at()
Description
Returns a string containing the time range when the issue was opened, like "yesterday", "last week", etc.
You can find this at
To what class it belongs?:
Issue
Make changes at
src/scrape_up/github/issue.py
Proposed Method:
.languages()
Description
Create a method to get the list of all the languages used in a repository.
To what class it belongs?:
Repository
Adding a feature to fetch all repositories along with used tech stack for it. I want to work on this issue under JWOC
Proposed Method: get_starred_repos()
Description
With the help of get_starred_repos(), one can easily scrape out the list of starred repos of any user.
To what class it belongs?: Users
Example Usage
from scrape_up import github
user = github .Users (username = "nikhil25803" )
print (user .get_starred_repos ())
# Output - Starts listing all starred repos.......
I want to work on this issue under JWOC.
We can add a method to retrieve the list of pinned repositories for a given user.
Proposed Method:
.stars_count()
Description
Create a method to get the number of stars, a repository has.
To what class it belongs?:
Repository
Proposed Method: get_techstack_repo()
Description
with the help of get_techstack_repo(), one can easily know main tech stack used for each repository of given user, in which it helps to pick repositories to contribute based on Tech Stack we are interested in.
To what class it belongs?: Users
Example Usage
from scrape_up import github
user = github .Users (username = "PalaVenkiReddy" )
# Call the techstack function
print (user .get_techstack_repo ())
# Output - {repo_name : tech_stack_used , . . . . . . . }
Achievements are a major part of a user's profile.
So, this too can be web-scrapped using python...
Proposed Method:
.tags()
Description
Create a method to get the tags of a repository, that is the title tags.
To what class it belongs?:
Repository
Proposed Method: .get_pull_requests()
Description
scrape all the pull requests of given repository
To what class it belongs?: Repository
Example Usage
# Import the required module
from scrape_up import github
# Instantiate an object with username provided.
user = github .Repository (username = "Clueless-Community" , repository = "scrape-up" )
# Cal the followers function
print (user .get_pull_requests ())
Output :
Returns all the pull requests of given repository
Proposed Method
.peoples()
Description
Create a method to scrape the number of peoples who are the part of the orgnaization.
To what class it belongs?
Organization
Note
Make all the changes in the github/organization.py
file.
Use a private method __scrape_page
to scrape the page.
Include the method in the documentation.md
file.
Upload a screenshot with the PR, demonstrating the method.
Proposed Method: .get_issues()
Description
Scrape the list of issues open in a repository.
To what class it belongs?: Repository
Example Usage
from scrape_up import github
repo = github .Repository (username = "Clueless-Community" , repository = "scrape-up" )
print (repo .get_issues ())
Output :
Returns a list of all open issues in the repository.
Hey @nikhil25803 I would like to work in this issue under DWOC '23 :) ๐
We need to make an issue and pull a request template for this project. Here are a few things which we need to include
Issue template
What method and you want to include?
To what class it belongs?
PR Templates
Some checkpoints
Have you followed the code convention?
Have you included the method/function in the documentation.md
file?
This is just a brief, you can include relevant points on your own. But please do not dump any random template. Keep it short and precise.
Proposed Method: get_followers_list()
Description
returns list of followers for given github username
To what class it belongs?: Users
Example Usage
# Import the required module
from scrape_up import github
# Instantiate an object with username provided.
user = github .Users (username = "nikhil25803" )
# Cal the followers function
print (user .get_followers_list ())
# Output: Returns List of Followers
Please assign this issue to me under JWOC
Create a method in the Users
class to get the following count of a user.
Download all repo of the user
Hello, i am a python developer and i like to work on automation and scripting. I want to contribute to this project.
Get the names of the organization, a user is a part of displayed on the profile page of a user.
Return a list with the names of the organizations, else return an empty list.
As the methods of the Users
class is done, I need someone to make a test file, which can test if all the methods as working correctly or not.
Create a test
folder in /src
Within that directory, create a github_test.py
.
Include all your script there, testing the methods that have been created till now.
Proposed Method:
.title()
Description
Get the title of a pull request.
To what class it belongs?:
PullRequest()
Note
Add the method in github/pull_request.py
Proposed Method:
.avatar()
Description
Create a method to scrape the URL of a GitHub organization
To what class it belongs?:
Organization
Note
Make all the changes in the github/organization.py
file.
Use a private method __scrape_page
to scrape the page.
Include the method in the documentation.md
file.
Upload a screenshot with the PR, demonstrating the method.
Proposed Method:
.releases()
Description
Create a method to get Releases of a repository.
To what class it belongs?:
Repository
As the folder structure of the project is now changed, you have to make minor changes in the contribution. md
file.
Proposed Method: .is_milestone()
Description
Returns the milestone, if the issue is part of one or 'No milestone', if it's not.
To what class it belongs?: Issue
Proposed Method:
.title()
Description
Get the title of an issue.
To what class it belongs?:
Issue
Make changes at
src/scrape_up/github/issue.py
Proposed Method:
.assignees()
Description
Returns a list of users, who are assigned to an issue. Else returns an empty list.
To what class it belongs?:
Issue
Make changes at
src/scrape_up/github/issue.py
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Proposed Method
.followers()
Description
Create a method to scrape the number of followers an organization has.
To what class it belongs?
Organization
Note
Make all the changes in the github/organization.py
file.
Use a private method __scrape_page
to scrape the page.
Include the method in the documentation.md
file.
Upload a screenshot with the PR, demonstrating the method.
I have found a typo error while running the commands stated for demo. actually class name is different from written one, so the command is not working properly. pls assign me this issue
Is your feature request related to a problem? Please describe.
We may need to have readme file for a user
Describe the solution you'd like
Scrape the users profile and get its readme and save it to directory of the user.
Describe alternatives you've considered
No alternatives
Additional context
Create a method with the Users
class to fetch the number of repositories and star count of a user.
For example, 45 repo count and 68 stars count as shown below for the user nikhil25803
README.md needs fixing in the "How to use?" section. The link to the provided methods is broken and doesn't open "documentation.md".
Proposed Method:
.pull_requests()
Description
Create a method to get the number of pull requests opened in a repository.
To what class it belongs?:
Repository