<Doc> Tags are not removed. about wikiextractor HOT 4 CLOSED

You can remove the tags with a simple sed script, eg.

sed  '/^<[^>]*>$/d’

—

On 23 mag 2016, at 10:22, kylemadigan [email protected] wrote:

Hi ,

I have tried cleaning my document using this: python WikiExtractor.py testfile.xml -o extractedcommand. And I got the following result:

Category:Scientific method

A scientific method is a sequence or collection of processes that are considered characteristic of scientific investigation and the acquisition of new scientific knowledge based upon physical evidence.

Category:Scientific terminology Category:Science and engineering awards

This category includes all prizes and awards relating to science, engineering, technology, and mathematics.

Scientific criticisms are a body of analysis of scientific methodologies, philosophies, and possible negative roles media and politics play in scientific research. Criticism of science is distinct from the academic positions of antiscience or anti-intellectualism which seek to reject entirely the scientific method. Rather, criticism is made to address and refine problems within the sciences in order to improve science as a whole, and its role in society.

That is with the tags still visible, how do i remove all xml tags using wikiextractor?

I have tried :

python WikiExtractor.py testfile.xml -o --no-templates --escapedoc

but unfortunately it does not work either. Please help me.

Thank you

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

from wikiextractor.

Hi I am not very sure what you mean? Can you be more specific please. Thank you.

from wikiextractor.

Hi @kylemadigan!

You can use the feature I added in the #59 pull request.

Let me know if that work for you.

from wikiextractor.

"""
import matplotlib.pyplot as plt
from bs4 import BeautifulSoup
import nltk
from nltk import word_tokenize
from nltk.corpus import stopwords
import re
from nltk.util import ngramstext
from collections import Counter
from nltk.stem import PorterStemmer
from nltk import ngrams
from nltk.probability import FreqDist
soup = BeautifulSoup(open("C:\Users\x\Downloads\wiki_00", encoding="utf8"), "html.parser")
text = soup.get_text()

from wikiextractor.