"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Example Task: count number of references on a page in Wikipedia\n",
"\n",
"* Use the wikidumps, as this information is not otherwise available\n",
"* We count using a regular expression.\n",
"* The tricky thing is that the references are in the running text at the point where the footnote appears.\n",
" * They are not \"where you see them on the page at the end\"\n",
"* **Technique:** We stream in the bz2 file and read it page by page using lxml iterparse. \n",
" \n",
" \n",
" "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2018-01-24 08:36:31-- https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles1.xml-p10p30302.bz2\n",
"Resolving dumps.wikimedia.org... 208.80.154.11, 2620::861:1:208:80:154:11\n",
"Connecting to dumps.wikimedia.org|208.80.154.11|:443... connected.\n",
"WARNING: cannot verify dumps.wikimedia.org's certificate, issued by `/C=US/O=Let\\'s Encrypt/CN=Let\\'s Encrypt Authority X3':\n",
" Unable to locally verify the issuer's authority.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 168503832 (161M) [application/octet-stream]\n",
"Saving to: `enwiki-latest-pages-articles1.xml-p10p30302.bz2'\n",
"\n",
"100%[======================================>] 168,503,832 262K/s in 5m 47s \n",
"\n",
"2018-01-24 08:42:19 (474 KB/s) - `enwiki-latest-pages-articles1.xml-p10p30302.bz2' saved [168503832/168503832]\n",
"\n"
]
}
],
"source": [
"# inspect the en wikipedia dump\n",
"large_wiki= 'https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2'\n",
"small_wiki= 'https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles1.xml-p10p30302.bz2'\n",
"#!wget --no-check-certificate $small_wiki"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\r\n",
" \r\n",
" Wikipedia\r\n",
" enwiki\r\n",
" https://en.wikipedia.org/wiki/Main_Page\r\n",
" MediaWiki 1.31.0-wmf.17\r\n",
" first-letter\r\n",
" \r\n",
" Media\r\n",
" Special\r\n",
" \r\n",
" Talk\r\n",
" User\r\n",
" User talk\r\n",
" Wikipedia\r\n",
" Wikipedia talk\r\n",
" File\r\n",
" File talk\r\n",
" MediaWiki\r\n",
" MediaWiki talk\r\n",
" Template\r\n",
" Template talk\r\n",
" Help\r\n",
" Help talk\r\n",
" Category\r\n",
" Category talk\r\n",
" Portal\r\n",
" Portal talk\r\n",
" Book\r\n",
" Book talk\r\n",
" Draft\r\n",
" Draft talk\r\n",
" Education Program\r\n",
" Education Program talk\r\n",
" TimedText\r\n",
" TimedText talk\r\n",
" Module\r\n",
" Module talk\r\n",
" Gadget\r\n",
" Gadget talk\r\n",
" Gadget definition\r\n",
" Gadget definition talk\r\n",
" \r\n",
" \r\n",
" \r\n",
" AccessibleComputing\r\n",
" 0\r\n",
" 10\r\n",
" \r\n",
" \r\n",
" 767284433\r\n",
" 631144794\r\n",
" 2017-02-25T00:30:28Z\r\n",
" \r\n",
" Godsy\r\n",
" 23257138\r\n",
" \r\n",
" [[Template:This is a redirect]] has been deprecated, change to [[Template:Redirect category shell]].\r\n",
" wikitext\r\n",
" text/x-wiki\r\n",
" #REDIRECT [[Computer accessibility]]\r\n",
"\r\n",
"{{Redirect category shell|\r\n",
"{{R from move}}\r\n",
"{{R from CamelCase}}\r\n",
"{{R unprintworthy}}\r\n",
"}}\r\n",
" ds1crfrjsn7xv73djcs4e4aq9niwanx\r\n",
" \r\n",
" \r\n",
" \r\n",
" Anarchism\r\n",
" 0\r\n",
" 12\r\n",
" \r\n",
" 820703495\r\n",
" 820025687\r\n",
" 2018-01-16T03:44:44Z\r\n",
" \r\n",
" Ira Leviton\r\n",
" 25046916\r\n",
" \r\n",
" \r\n",
" Deleted 'of course'. Please see [[Wikipedia:Manual_of_Style#Instructional_and_presumptuous_language]], which cautions against use of this term.\r\n",
" wikitext\r\n",
" text/x-wiki\r\n",
" {{Redirect2|Anarchist|Anarchists|the fictional character|Anarchist (comics)|other uses|Anarchists (disambiguation)}}\r\n",
"{{pp-move-indef}}\r\n",
"{{Use British English|date=January 2014}}\r\n",
"{{Anarchism sidebar}}\r\n",
"{{Basic forms of government}}\r\n",
"\r\n",
"'''Anarchism''' is a [[political philosophy]] that advocates [[self-governance|self-governed]] societies based on voluntary institutions. These are often described as [[stateless society|stateless societies]],<ref>"ANARCHISM, a social philosophy that rejects authoritarian government and maintains that voluntary institutions are best suited to express man's natural social tendencies." George Woodcock. "Anarchism" at The Encyclopedia of Philosophy</ref><ref>"In a society developed on these lines, the voluntary associations which already now begin to cover all the fields of human activity would take a still greater extension so as to substitute themselves for the state in all its functions." [http://www.theanarchistlibrary.org/HTML/Petr_Kropotkin___Anarchism__from_the_Encyclopaedia_Britannica.html Peter Kropotkin. "Anarchism" from the Encyclopædia Britannica]</ref><ref>"Anarchism." The Shorter Routledge Encyclopedia of Philosophy. 2005. p. 14 "Anarchism is the view that a society without the state, or government, is both possible and desirable."</ref><ref>Sheehan, Sean. Anarchism, London: Reaktion Books Ltd., 2004. p. 85</ref> although several authors have defined them more specifically as institutions based on non-[[Hierarchy|hierarchical]] or [[Free association (communism and anarchism)|free associations]].<ref>"as many anarchists have stressed, it is not government as such that they find objectionable, but the hierarchical forms of government associated with the nation state." Judith Suissa. ''Anarchism and Education: a Philosophical Perspective''. Routledge. New York. 2006. p. 7</ref><ref name="iaf-ifa.org">{{cite web |url=http://www.iaf-ifa.org/principles/english.html |title=IAF principles |publisher=[[International of Anarchist Federations]] |archiveurl=https://web.archive.org/web/20120105095946/http://www.iaf-ifa.org/principles/english.html |archivedate=5 January 2012 |deadurl=yes |quote=The IAF – IFA fights for : the abolition of all forms of authority whether economical, political, social, religious, cultural or sexual.}}</ref><ref>"That is why Anarchy, when it works to destroy authority in all its aspects, when it demands the abrogation of laws and the abolition of the mechanism that serves to impose them, when it refuses all hierarchical organization and preaches free agreement — at the same time strives to maintain and enlarge the precious kernel of social customs without which no human or animal society can exist." [[Peter Kropotkin]]. [http://www.theanarchistlibrary.org/HTML/Petr_Kropotkin__Anarchism__its_philosophy_and_ideal.html Anarchism: its philosophy and ideal]</ref><ref>"anarchists are opposed to irrational (e.g., illegitimate) authority, in other words, hierarchy — hierarchy being the institutionalization of authority within a society." [http://www.theanarchistlibrary.org/HTML/The_Anarchist_FAQ_Editorial_Collective__An_Anarchist_FAQ__03_17_.html#toc2 "B.1 Why are anarchists against authority and hierarchy?"] in [[An Anarchist FAQ]]</ref> Anarchism holds the [[state (polity)|state]] to be undesirable, unnecessary, and harmful.<ref name="definition">\r\n",
"{{cite journal |last=Malatesta|first=Errico|title=Towards Anarchism|journal=MAN!|publisher=International Group of San Francisco|location=Los Angeles|oclc=3930443|url=http://www.marxists.org/archive/malatesta/1930s/xx/toanarchy.htm|archiveurl=https://web.archive.org/web/20121107221404/http://marxists.org/archive/malatesta/1930s/xx/toanarchy.htm|archivedate=7 November 2012 |deadurl=no|authorlink=Errico Malatesta |ref=harv}}\r\n",
"{{cite journal |url=https://www.theglobeandmail.com/servlet/story/RTGAM.20070514.wxlanarchist14/BNStory/lifeWork/home/\r\n",
"|archiveurl=https://web.archive.org/web/20070516094548/http://www.theglobeandmail.com/servlet/story/RTGAM.20070514.wxlanarchist14/BNStory/lifeWork/home |archivedate=16 May 2007 |deadurl=yes |title=Working for The Man |journal=[[The Globe and Mail]] |access-date=14 April 2008 |last=Agrell |first=Siri |date=14 May 2007 |ref=harv }}\r\n",
"{{cite web |url=http://www.britannica.com/eb/article-9117285|title=Anarchism|year=2006|work=Encyclopædia Britannica|publisher=Encyclopædia Britannica Premium Service|access-date=29 August 2006| archiveurl= https://web.archive.org/web/20061214085638/http://www.britannica.com/eb/article-9117285| archivedate= 14 December 2006<!--Added by DASHBot-->}}\r\n",
"{{cite journal |year=2005|title=Anarchism|journal=The Shorter [[Routledge Encyclopedia of Philosophy]]|page=14|quote=Anarchism is the view that a society without the state, or government, is both possible and desirable. |ref=harv}}\r\n",
"The following sources cite anarchism as a political philosophy:\r\n",
"{{cite book | last = Mclaughlin | first = Paul | title = Anarchism and Authority | publisher = Ashgate | location = Aldershot | year = 2007 | isbn = 978-0754661962 |page=59}}\r\n",
"{{cite book | last = Johnston | first = R. | title = The Dictionary of Human Geography | publisher = Blackwell Publishers | location = Cambridge | year = 2000 | isbn = 0-631-20561-6 |page=24}}</ref><ref name=slevin>Slevin, Carl. "Anarchism." ''The Concise Oxford Dictionary of Politics''. Ed. Iain McLean and Alistair McMillan. Oxford University Press, 2003.</ref>\r\n",
"\r\n",
"While opposition to the [[State (polity)|state]] is central,<ref>"Anarchists do reject the state, as we will see. But to claim that this central aspect of anarchism is definitive is to sell anarchism short."[https://books.google.com/books?id=kkj5i3CeGbQC&printsec=frontcover#v=onepage&q&f=false ''Anarchism and Authority: A Philosophical Introduction to Classical Anarchism'' by Paul McLaughlin. AshGate. 2007. p. 28]</ref> anarchism specifically entails opposing authority or hierarchical organisation in the conduct of all human relations.<ref>"Anarchy is the condition of existence of adult society, as hierarchy is the condition of primitive society. There is a continual progress in human society from hierarchy to anarchy."[https://www.panarchy.org/proudhon/state.html ''The State: Its Nature, Object, and Destiny'' by Pierre-Joseph Proudhon.]</ref><ref>"In a word, we reject all legislation, all authority, and all privileged, licensed, official, and legal influence, even though arising from universal suffrage, convinced that it can turn only to the advantage of a dominant minority of exploiters against the interests of the immense majority in subjection to them. This is the sense in which we are really Anarchists."[https://theanarchistlibrary.org/library/michail-bakunin-god-and-the-state ''God and the State'' by Mikhail Bakunin.]</ref><ref>"In practice, any individual who, because of his or her temperament or because of conscious and serious reflection, repudiates all external authority or coercion, whether of a governmental, ethical, intellectual, or economic order, can be considered an anarchist. Everyone who consciously rejects the domination of people by other people or by the social ambiance, and its economic corollaries, can be said to be an anarchist as well."[http://theanarchistlibrary.org/library/emile-armand-anarchist-individualism-and-amorous-comradeship#toc17 ''Anarchist Individualism and Amorous Comradeship'' by Emile Armand]</ref> Anarchism is usually considered a [[Far-left politics|far-left]] ideology<ref name=brooks>{{cite book |quote= Usually considered to be an extreme left-wing ideology, anarchism has always included a significant strain of radical individualism, from the hyperrationalism of Godwin, to the egoism of Stirner, to the libertarians and anarcho-capitalists of today |last= Brooks |first= Frank H. |year= 1994 |title= The Individualist Anarchists: An Anthology of Liberty (1881–1908) |publisher= Transaction Publishers |page= xi|isbn= 1-56000-132-1}}</ref><ref>{{cite journal |author= Joseph Kahn|title= Anarchism, the Creed That Won't Stay Dead; The Spread of World Capitalism Resurrects a Long-Dormant Movement |year= 2000|journal= [[The New York Times]]|issue= 5 August |ref= harv}}{{cite journal |author= Colin Moynihan |title= Book Fair Unites Anarchists. In Spirit, Anyway|year= 2007|journal= New York Times|issue= 16 April |ref= harv}}</ref> and much of [[anarchist economics]] and [[anarchist law|anarchist legal philosophy]] reflects [[Libertarian socialism|anti-authoritarian interpretations]] of [[anarcho-communism|communism]], [[collectivist anarchism|collectivism]], [[anarcho-syndicalism|syndicalism]], [[Mutualism (economic theory)|mutualism]] or [[participatory economics]].<ref>\r\n",
""The anarchists were unanimous in subjecting authoritarian socialism to a barrage of severe criticism. At the time when they made violent and satirical attacks these were not entirely well founded, for those to whom they were addressed were either primitive or "vulgar" communists, whose thought had not yet been fertilized by Marxist humanism, or else, in the case of Marx and Engels themselves, were not as set on authority and state control as the anarchists made out." Daniel Guerin, ''[http://theanarchistlibrary.org/library/daniel-guerin-anarchism-from-theory-to-practice#toc2 Anarchism: From Theory to Practice]'' (New York: Monthly Review Press, 1970)</ref>\r\n",
"\r\n",
"Anarchism does not offer a fixed body of doctrine from a single particular world view, instead fluxing and flowing as a philosophy.<ref>{{cite book |last=Marshall|first=Peter|title=[[Demanding the Impossible: A History of Anarchism]]|year=2010|publisher=PM Press|location=Oakland, CA|isbn=978-1-60486-064-1|page=16}}</ref> Many types and traditions of anarchism exist, not all of which are mutually exclusive.<ref>{{cite book |last= Sylvan |first= Richard |chapter= Anarchism |title= A Companion to Contemporary Political Philosophy |editors= Goodwin, Robert E. and Pettit |publisher= Philip. Blackwell Publishing |year= 1995 |page= 231}}</ref> [[Anarchist schools of thought]] can differ fundamentally, supporting anything from extreme [[individualism]] to complete collectivism.<ref name=slevin /> Strains of anarchism have often been divided into the categories of [[social anarchism|social]] and [[individualist anarchism]] or similar dual classifications.<ref name="black dict">[[Geoffrey Ostergaard|Ostergaard, Geoffrey]]. "Anarchism". ''The Blackwell Dictionary of Modern Social Thought''. Blackwell Publishing. p. 14.</ref><ref name=socind>{{cite book |authorlink= Peter Kropotkin |last= Kropotkin |first= Peter |title= Anarchism: A Collection of Revolutionary Writings |publisher= Courier Dover Publications |year= 2002 |page= 5|isbn= 0-486-41955-X}}{{cite journal |author= R.B. Fowler|title= The Anarchist Tradition of Political Thought|year= 1972|journal= Western Political Quarterly|volume= 25|issue= 4|pages= 738–752|doi= 10.2307/446800|publisher= University of Utah|jstor= 446800 |ref= harv}}</ref>\r\n",
"\r\n",
"==Etymology and terminology==\r\n",
"{{Related articles|Anarchist terminology}}\r\n",
"\r\n",
"The word ''[[wikt:anarchism|anarchism]]'' is composed from the word ''[[anarchy]]'' and the suffix ''[[-ism]]'',<ref>[http://www.etymonline.com/index.php?term=anarchism&allowed_in_frame=0 Anarchism], [[Online etymology dictionary]].</ref> themselves derived respectively from the Greek {{lang|grc|ἀναρχία}},<ref>{{cite EB1911|wstitle=Anarchism|volume=1|page=914}}</ref> i.e. ''anarchy''<ref>{{LSJ|a)narxi/a|ἀναρχία|ref}}</ref><ref>[http://www.merriam-webster.com/dictionary/anarchy Anarchy], [[Merriam-Webster]] online.</ref><ref>[http://www.etymonline.com/index.php?term=anarchy&allowed_in_frame=0 Anarchy], [[Online etymology dictionary]].</ref> (from {{lang|grc|ἄναρχος}}, ''anarchos'', meaning "one without rulers";<ref>{{LSJ|a)/narxos|ἄναρχος|ref}}</ref> from the [[privative]] prefix [[privative alpha|ἀν]]- (''an-'', i.e. "without") and {{lang|grc|ἀρχός}}, ''archos'', i.e. "leader", "ruler";<ref>{{LSJ|a)rxo/s|ἀρχός|ref}}</ref> (cf. ''[[archon]]'' or {{lang|grc|ἀρχή}}, ''arkhē'', i.e. "authority", "sovereignty", "realm", "magistracy")<ref>{{LSJ|a)rxh/|ἀρχή|ref}}</ref>) and the suffix {{lang|grc|-ισμός}} or {{lang|grc|-ισμα}} (''-ismos'', ''-isma'', from the verbal [[infinitive]] suffix -ίζειν, ''-izein'').<ref>[http://www.etymonline.com/index.php?term=-ism&allowed_in_frame=0 -ism], [[Online etymology dictionary]].</ref> The first known use of this word was in 1539.<ref>"Origin of ANARCHY\r\n",
"Medieval Latin anarchia, from Greek, from anarchos having no ruler, from an- + archos ruler — more at arch-\r\n",
"First Known Use: 1539" [http://www.merriam-webster.com/dictionary/anarchy "Anarchy" at Merriam Webster dictionary online]</ref> Various factions within the [[French Revolution]] labelled opponents as anarchists (as [[Maximilien de Robespierre|Robespierre]] did the [[Hébertists]])<ref>{{cite book|last=Deleplace|first=Marc|editor=Annie Geffroy|title=Dictionnaire des usages socio-politiques (1770–1815)|chapterurl=https://books.google.com/books?id=GOkSKE57mdYC&pg=PA9|year=1990|publisher=ENS Editions|language=French|isbn=9782252026946|pages=9–34|chapter=Anarchie–Anarchiste; Germinal–Fructidor An III (21 mars – 16 septembre 1795)}}</ref> although few shared many views of later anarchists. There would be many revolutionaries of the early nineteenth century who contributed to the anarchist doctrines of the next generation, such as [[William Godwin]] and [[Wilhelm Weitling]], but they did not use the word ''anarchist'' or ''anarchism'' in describing themselves or their beliefs.<ref>{{cite book |title=The Anarchists |last=Joll |first=James |year=1964 |publisher=Harvard University Press |isbn=0-674-03642-5 |pages=27–37}}</ref>\r\n",
"\r\n",
"The first political philosopher to call himself an anarchist was [[Pierre-Joseph Proudhon]], marking the formal birth of anarchism in the mid-nineteenth century. Since the 1890s, and beginning in France,<ref>{{cite book |title=A Short History of Anarchism |last=Nettlau |first=Max|authorlink=Max Nettlau |year=1996 |publisher=Freedom Press |isbn=0-900384-89-1 |page=162}}</ref> the term "[[libertarianism]]" has often been used as a synonym for anarchism<ref>"At the end of the century in France, Sebastien Faure took up a word used in 1858 by one Joseph Dejacque to make it the title of a journal, Le Libertaire. Today the terms "anarchist" and "libertarian" have become interchangeable." [http://www.theanarchistlibrary.org/HTML/Daniel_Guerin__Anarchism__From_Theory_to_Practice.html#toc2 ''Anarchism: From Theory to Practice''] [[Daniel Guérin]]</ref> and was used almost exclusively in this sense until the 1950s in the United States;<ref>Russell, Dean. [http://www.boogieonline.com/revolution/politics/name.html ''Who is a Libertarian?''], [[Foundation for Economic Education]], "Ideas on Liberty," May 1955.</ref> its use as a synonym is still common outside the United States.<ref>\r\n",
"* Ward, Colin. Anarchism: A Very Short Introduction. Oxford University Press 2004 p. 62\r\n",
"* Goodway, David. [[Anarchist Seeds Beneath the Snow]]. Liverpool Press. 2006, p. 4\r\n",
"* MacDonald, Dwight & Wreszin, Michael. Interviews with [[Dwight Macdonald]]. University Press of Mississippi, 2003. p. 82\r\n",
"* Bufe, Charles. The Heretic's Handbook of Quotations. See Sharp Press, 1992. p. iv\r\n",
"* Gay, Kathlyn. Encyclopedia of Political Anarchy. ABC-CLIO / University of Michigan, 2006, p. 126\r\n",
"* Woodcock, George. Anarchism: A History of Libertarian Ideas and Movements. Broadview Press, 2004. (Uses the terms interchangeably, such as on page 10)\r\n",
"* [[Alexandre Skirda|Skirda, Alexandre]]. Facing the Enemy: A History of Anarchist Organization from Proudhon to May 1968. AK Press 2002. p. 183.\r\n",
"* Fernandez, Frank. Cuban Anarchism. The History of a Movement. See Sharp Press, 2001, page 9.</ref> On the other hand, some use libertarianism to refer to individualistic free-market philosophy only, referring to free-market anarchism as [[libertarian anarchism]].<ref>Morris, Christopher. 1992. ''An Essay on the Modern State''. Cambridge University Press. p. 61. (Using "libertarian anarchism" synonymously with "individualist anarchism" when referring to individualist anarchism that supports a [[market economy|market society]]).</ref><ref>{{cite book |last=Burton |first=Daniel C. |title= Libertarian anarchism |url= http://www.libertarian.co.uk/lapubs/polin/polin168.pdf |publisher=[[Libertarian Alliance]] |year= |page=|access-date=2009-03-04}}</ref>\r\n",
"\r\n",
"==History==\r",
"\r\n",
"{{Main|History of anarchism}}\r\n",
"\r\n",
"===Origins===\r\n",
"[[File:Levellers declaration and standard.gif|thumb|upright|Woodcut from a [[Diggers]] document by [[William Everard (Digger)|William Everard]]]]\r\n",
"<!--Anarcho-communist Joseph Déjacque, the first person to use the term "libertarian" in a political sense and self-proclaimed advocate of libertarianism, needs to be added here. His work and stances on anarchism are very relevant to this particular section of the article. Additionally, his criticisms of Pierre-Joseph Proudhon's mutualism are very relevant here.-->\r\n",
"The earliest<ref name="LVM">{{cite web|url=https://mises.org/daily/2054#1.10|title=Mises Daily|work=Mises Institute|access-date=2014-09-13|archive-url=https://web.archive.org/web/20140914050357/https://www.mises.org/daily/2054|archive-date=2014-09-14}}</ref> anarchist themes can be found in the 6th century BC among the works of [[Taoism|Taoist]] philosopher [[Laozi]]<ref name="EB1910">Peter Kropotkin, [http://dwardmac.pitzer.edu/Anarchist_Archives/kropotkin/britanniaanarchy.html "Anarchism"], ''[[Encyclopædia Britannica Eleventh Edition|Encyclopædia Britannica]]'' 1910.</ref> and in later centuries by [[Zhuang Zhou|Zhuangzi]] and Bao Jingyan.<ref name="wordpress">{{cite web|url=http://robertgraham.wordpress.com/anarchism-a-documentary-history-of-libertarian-ideas-volume-one-from-anarchy-to-anarchism-300ce-1939/|title=Anarchism: A Documentary History of Libertarian Ideas, Volume One: From Anarchy to Anarchism (300CE–1939)|work=Robert Graham's Anarchism Weblog|access-date=2011-03-05|archive-url=https://web.archive.org/web/20101130131904/http://robertgraham.wordpress.com/anarchism-a-documentary-history-of-libertarian-ideas-volume-one-from-anarchy-to-anarchism-300ce-1939/|archive-date=2010-11-30}}</ref> Zhuangzi's philosophy has been described by various sources as anarchist.<ref>"The priority of dao over tiannature:sky underwrites the themes of dependency and relativism that pervade the Zhuangzi and ultimately the skepticism, the open-minded toleration and the political anarchism (or disinterest in political activity or involvement)." [http://plato.stanford.edu/entries/taoism/ "Taoism" at the Stanford Encyclopedia of Philosophy]</ref><ref>"Doing nothing [wu wei] is the famous Daoist concept for natural action, action in accord with Dao, action in which we freely follow our own way and allow other beings to do likewise. Zhuangzi, the great anarchic Daoist sage, compared it to "riding on the wind." [http://www.theanarchistlibrary.org/HTML/Max_Cafard__Zen_Anarchy.html Max Cafard. "Zen Anarchy"]</ref><ref>"Zhuangzi helps us discover an anarchistic epistemology and sensibility. He describes a state in which "you are open to everything you see and hear, and allow this to act through you."[45] Part of wuwei, doing without doing, is "knowing without knowing," knowing as being open to the things known, rather than conquering and possessing the objects of knowledge. This means not imposing our prejudices (whether our own personal ones, our culture's, or those built into the human mind) on the Ten Thousand Things." [http://www.theanarchistlibrary.org/HTML/Max_Cafard__The_Surre_gion_alist_Manifesto_and_Other_Writings.html#toc24 Max Cafard. ''The Surre(gion)alist Manifesto and Other Writings'']</ref><ref>"The next group of interpreters have also become incorporated into the extant version of the text. They are the school of anarchistically inclined philosophers, that Graham identifies as a "Primitivist" and a school of "Yangists," chapters 8 to 11, and 28 to 31. These thinkers appear to have been profoundly influenced by the Laozi, and also by the thought of the first and last of the Inner Chapters: "Wandering Beyond," and "Responding to Emperors and Kings." There are also possible signs of influence from Yang Zhu, whose concern was to protect and cultivate one's inner life-source. These chapters combine the anarchistic ideals of a simple life close to nature that can be found in the Laozi with the practices that lead to the cultivation and nurturing of life. " [http://www.iep.utm.edu/zhuangzi/ "Zhuangzi (Chuang-Tzu, 369–298 BCE)" at the Internet Encyclopedia of Philosophy]</ref> Zhuangzi wrote: "A petty thief is put in jail. A great [[wikt:brigand|brigand]] becomes a ruler of a Nation".<ref>{{cite web |url=https://www.mises.org/journals/jls/9_2/9_2_3.pdf|format=PDF|title=Concepts of the role of intellectuals in social change toward laissez faire|author=[[Murray Rothbard]]|access-date=28 December 2008| archiveurl= https://web.archive.org/web/20081216214953/https://www.mises.org/journals/jls/9_2/9_2_3.pdf| archivedate= 16 December 2008<!--Added by DASHBot-->}}</ref> [[Diogenes of Sinope]] and the [[Cynicism (philosophy)|Cynics]], as well as their contemporary [[Zeno of Citium]], the founder of [[Stoicism]], also introduced similar topics.<ref name="EB1910" /><ref>{{IEP|c/cynics.htm|Cynics|Julie Piering}}</ref> [[Jesus]] is sometimes considered the first anarchist in the [[Christian anarchism|Christian anarchist]] tradition. Georges Lechartier wrote: "The true founder of anarchy was [[Jesus]] [[Christ (title)|Christ]] and ... the first anarchist society was that of the [[Apostle (Christian)|apostles]]".<ref>Cited in George Woodcock, ''Anarchism: A History of Libertarian Ideas and Movements'' (Cleveland: Meridian Books, 1962), p. 38.</ref> In early [[History of Islam|Islamic history]], some manifestations of anarchic thought are found during the [[Second Fitna|Islamic civil war]] over the [[Caliphate]], where the [[Kharijites]] insisted that the [[imamate]] is a right for each individual within the Islamic society.<ref>{{cite web|url=http://theanarchistlibrary.org/library/mohamed-jean-veneuse-anarca-islam#fn_back31|title=Anarca-Islam|work=theanarchistlibrary.org|access-date=2014-02-19|archive-url=https://web.archive.org/web/20140225023248/http://theanarchistlibrary.org/library/mohamed-jean-veneuse-anarca-islam|archive-date=2014-02-25}}</ref>\r\n",
"\r\n",
"The French [[Renaissance humanism|renaissance]] political philosopher [[Étienne de La Boétie]] wrote in his most famous work the ''Discourse on Voluntary Servitude'' what some historians consider an important anarchist precedent.<ref>Several historians of anarchism have gone so far as to classify La Botie's treatise itself as anarchist, which is incorrect since La Botie never extended his analysis from tyrannical government to government per se. But while La Botie cannot be considered an anarchist, his sweeping strictures on tyranny and the universality of his political philosophy lend themselves easily to such an expansion.[https://books.google.com/books?id=6o-8P3iqf7IC&pg=PA18&lpg=PA18&dq=anarchism+la+boetie&source=bl&ots=z79GU1rW1t&sig=4ini7oZUie2U8-P0BpMLogXYWPs&hl=en&sa=X&ei=eAOFUJCYJ4Le9AT_iYG4DA&ved=0CD0Q6AEwBQ#v=onepage&q=anarchism%20la%20boetie&f=false ''Introduction to The Politics of Obedience: The Discourse of Voluntary Servitude'' by Murray Rothbard. Ludwig Von Mises Institute. p. 18]</ref><ref>"Quite rightly, La Boëtie recognizes the potential for domination in any democracy: the democratic leader, elected by the people, becomes intoxicated with his own power and teeters increasingly towards tyranny. Indeed, we can see modern democracy itself as an instance of voluntary servitude on a mass scale. It is not so much that we participate in an illusion whereby we are deceived by elites into thinking we have a genuine say in decision-making. It is rather that democracy itself has encouraged a mass contentment with powerlessness and a general love of submission."[http://theanarchistlibrary.org/library/saul-newman-voluntary-servitude-reconsidered-radical-politics-and-the-problem-of-self-dominatio "Voluntary Servitude Reconsidered: Radical Politics and the Problem of Self-Domination"] [[Saul Newman]]</ref> The radical [[Protestant Christianity|Protestant Christian]] [[Gerrard Winstanley]] and his group the Diggers are cited by various authors as proposing anarchist social measures in the 17th century in England.<ref>"Anarchists have regarded the secular revolt of the [[Diggers]], or True Levellers, in seventeenth-century England led by Gerrard Winstanley as a source of pride. Winstanley, deeming that property is corrupting, opposed clericalism, political power and privilege. It is economic inequality, he believed, that produces crime and misery. He championed a primitive communalism based on the pure teachings of God as comprehended through reason." [http://www.theanarchistlibrary.org/HTML/Kenneth_C._Wenzer__Godwin_s_Place_in_the_Anarchist_Tradition___a_Bicentennial_Tribute.html Kenneth C. Wenzer. "Godwin's Place in the Anarchist Tradition — a Bicentennial Tribute"]</ref><ref>"It was in these conditions of class struggle that, among a whole cluster of radical groups such as the Fifth Monarchy Men, the [[Levellers]] and the Ranters, there emerged perhaps the first real proto-anarchists, the Diggers, who like the classical 19th-century anarchists identified political and economic power and who believed that a social, rather than political revolution was necessary for the establishment of justice. Gerrard Winstanley, the Diggers' leader, made an identification with the word of God and the principle of reason, an equivalent philosophy to that found in [[Tolstoy]]'s ''[[The Kingdom of God is Within You]]''." Marlow. "Anarchism and Christianity"</ref><ref>"Although Proudhon was the first writer to call himself an anarchist, at least two predecessors outlined systems that contain all the basic elements of anarchism. The first was Gerrard Winstanley (1609 – c. 1660), a linen draper who led the small movement of the Diggers during the Commonwealth. Winstanley and his followers protested in the name of a radical Christianity against the economic distress that followed the Civil War and against the inequality that the grandees of the New Model Army seemed intent on preserving. In 1649–1650 the Diggers squatted on stretches of common land in southern England and attempted to set up communities based on work on the land and the sharing of goods." George Woodcock Anarchism The Encyclopedia of Philosophy</ref> The term "anarchist" first entered the English language in 1642, during the [[English Civil War]], as a [[Pejorative|term of abuse]], used by [[Cavalier|Royalists]] against their [[Roundhead]] opponents.<ref name=bbc>[http://www.bbc.co.uk/radio4/history/inourtime/inourtime_20061207.shtml "Anarchism"], [[BBC Radio 4]] program, [[In Our Time (BBC Radio 4)|In Our Time]], Thursday 7 December 2006. Hosted by [[Melvyn Bragg]] of the BBC, with John Keane, Professor of Politics at [[University of Westminster]], [[Ruth Kinna]], Senior Lecturer in Politics at [[Loughborough University]], and [[Peter Marshall (author)|Peter Marshall]], philosopher and historian.</ref> By the time of the French Revolution some, such as the ''[[Enragés]]'', began to use the term positively,<ref>Sheehan, Sean. ''Anarchism'', London: Reaktion Books Ltd., 2004. p. 85.</ref> in opposition to [[Jacobin (politics)|Jacobin]] centralisation of power, seeing "revolutionary government" as [[oxymoron]]ic.<ref name=bbc /> By the turn of the 19th century, the English word "anarchism" had lost its initial negative connotation.<ref name=bbc />\r\n",
"\r\n",
"Modern anarchism emerged from the secular or religious thought of the [[Age of Enlightenment|Enlightenment]], particularly [[Jean-Jacques Rousseau]]'s arguments for the moral centrality of freedom.<ref name=Encarta>"Anarchism", ''[[Encarta]] Online Encyclopedia'' 2006 (UK version).</ref>\r\n",
"\r\n",
"As part of the political turmoil of the 1790s in the wake of the French Revolution, William Godwin developed the first expression of modern anarchist thought.<ref name="Everhart, Robert B 1982. p. 115">Everhart, Robert B. The Public School Monopoly: A Critical Analysis of Education and the State in American Society. Pacific Institute for Public Policy Research, 1982. p. 115.</ref><ref name="godwinsep" /> Godwin was, according to [[Peter Kropotkin]], "the first to formulate the political and economical conceptions of anarchism, even though he did not give that name to the ideas developed in his work",<ref name="EB1910" /> while Godwin attached his anarchist ideas to an early [[Edmund Burke]].<ref>Godwin himself attributed the first anarchist writing to Edmund Burke's '' [[A Vindication of Natural Society]]''. "Most of the above arguments may be found much more at large in Burke's ''Vindication of Natural Society''; a treatise in which the evils of the existing political institutions are displayed with incomparable force of reasoning and lustre of eloquence ..." – footnote, Ch. 2 ''[[Political Justice]]'' by William Godwin.</ref>\r\n",
"\r\n",
"[[File:WilliamGodwin.jpg|left|thumb|upright|William Godwin, "the first to formulate the political and economical conceptions of anarchism, even though he did not give that name to the ideas developed in his work".<ref name="EB1910" />]]\r\n",
"Godwin is generally regarded as the founder of the school of thought known as 'philosophical anarchism'. He argued in ''[[Political Justice]]'' (1793)<ref name="godwinsep" /><ref name="Adams, Ian 2001. p. 116">Adams, Ian. Political Ideology Today. Manchester University Press, 2001. p. 116.</ref> that government has an inherently malevolent influence on society, and that it perpetuates dependency and ignorance. He thought that the spread of the use of reason to the masses would eventually cause government to wither away as an unnecessary force. Although he did not accord the state with moral legitimacy, he was against the use of revolutionary tactics for removing the government from power. Rather, he advocated for its replacement through a process of peaceful evolution.<ref name="godwinsep">{{cite SEP |url-id=godwin |title=William Godwin |last=Philip |first=Mark|date=2006-05-20}}</ref><ref>{{Cite book|title=[[Political Justice|Enquiry Concerning Political Justice and its Influence on Modern Morals and Manners]] |last=Godwin |first=William |authorlink=William Godwin |year=1796 |publisher=G.G. and J. Robinson |oclc=2340417 |origyear=1793}}</ref>\r\n",
"\r\n",
"His aversion to the imposition of a rules-based society led him to denounce, as a manifestation of the people's 'mental enslavement', the foundations of law, [[property rights]] and even the institution of marriage. He considered the basic foundations of society as constraining the natural development of individuals to use their powers of reasoning to arrive at a mutually beneficial method of social organisation. In each case, government and its institutions are shown to constrain the development of our capacity to live wholly in accordance with the full and free exercise of private judgement.\r\n",
"\r\n",
"The French Pierre-Joseph Proudhon is regarded as the first ''self-proclaimed'' anarchist, a label he adopted in his groundbreaking work, ''[[What is Property?]]'', published in 1840. It is for this reason that some claim Proudhon as the founder of modern anarchist theory.<ref>Daniel Guerin, ''Anarchism: From Theory to Practice'' (New York: Monthly Review Press, 1970).</ref> He developed the theory of [[spontaneous order]] in society, where organisation emerges without a central coordinator imposing its own idea of order against the wills of individuals acting in their own interests. His famous quote on the matter is "Liberty is the mother, not the daughter, of order". In ''What is Property?'' Proudhon answers with the famous accusation "[[Property is theft]]." In this work, he opposed the institution of decreed "property" (''propriété''), where owners have complete rights to "use and abuse" their property as they wish.<ref name="proudhon-prop">Proudhon, Pierre-Joseph. ''"[http://www.marxists.org/reference/subject/economics/proudhon/property/ch03.htm Chapter 3. Labour as the efficient cause of the domain of property]"'' from ''"[[What is Property?]]"'', 1840</ref> He contrasted this with what he called "possession," or limited ownership of resources and goods only while in more or less continuous use. Later, however, Proudhon added that "Property is Liberty" and argued that it was a bulwark against state power.<ref>Edwards, Stewart. Introduction to ''Selected Writings of Pierre-Joseph Proudhon'', Anchor Books, Doubleday & Company, Inc. 1969, p. 33</ref> His opposition to the state, organised religion, and certain capitalist practices inspired subsequent anarchists, and made him one of the leading social thinkers of his time.\r\n",
"\r\n",
"The anarcho-communist [[Joseph Déjacque]] was the first person to describe himself as "libertarian".<ref name="Dejacque">Joseph Déjacque, [http://joseph.dejacque.free.fr/ecrits/lettreapjp.htm De l'être-humain mâle et femelle - Lettre à P.J. Proudhon par Joseph Déjacque] (in French)</ref> Unlike Pierre-Joseph Proudhon, he argued that, "it is not the product of his or her labour that the worker has a right to, but to the satisfaction of his or her needs, whatever may be their nature."<ref>"l'Echange", article in ''Le Libertaire'' no 6, 21 September 1858, New York. [http://joseph.dejacque.free.fr/libertaire/n06/lib01.htm]</ref> In 1844 in Germany the post-hegelian philosopher [[Max Stirner]] published the book, ''[[The Ego and Its Own]]'', which would later be considered an influential early text of individualist anarchism.<ref name="SEP-Stirner">{{cite SEP|url-id=max-stirner|title=Max Stirner|last=Leopold|first=David|date=2006-08-04}}</ref> French anarchists active in the [[1848 Revolution]] included [[Anselme Bellegarrigue]], Ernest Coeurderoy, Joseph Déjacque<ref name="Dejacque">Joseph Déjacque, [http://joseph.dejacque.free.fr/ecrits/lettreapjp.htm De l'être-humain mâle et femelle - Lettre à P.J. Proudhon par Joseph Déjacque] (in French)</ref> and [[Pierre Joseph Proudhon]].<ref>{{cite web|url=http://theanarchistlibrary.org/library/pierre-joseph-proudhon-toast-to-the-revolution|title=Toast to the Revolution|work=theanarchistlibrary.org|access-date=2012-11-13|archive-url=https://web.archive.org/web/20121003195513/http://theanarchistlibrary.org/library/pierre-joseph-proudhon-toast-to-the-revolution|archive-date=2012-10-03}}</ref><ref>{{cite web|url=https://books.google.com/books/about/L_acitivit%C3%A9_d_un_socialiste_de_1848.html?id=wbrfSAAACAAJ&redir_esc=y|title=L'acitivité d'un socialiste de 1848|work=google.com.ec|access-date=2016-06-05}}</ref>\r\n",
"{{Clear}}\r\n",
"\r\n",
"===First International and the Paris Commune===\r\n",
"{{Main|International Workingmen's Association|Paris Commune}}\r\n",
"[[File:Bakunin.png|thumb|upright|Anarchist [[Mikhail Bakunin]] opposed the [[Marxist]] aim of [[dictatorship of the proletariat]] in favour of universal rebellion, and allied himself with the federalists in the First International before his expulsion by the Marxists.<ref name=bbc />]]\r\n",
"\r\n",
"In Europe, harsh reaction followed the [[revolutions of 1848]], during which ten countries had experienced brief or long-term social upheaval as groups carried out nationalist uprisings. After most of these attempts at systematic change ended in failure, conservative elements took advantage of the divided groups of socialists, anarchists, liberals, and nationalists, to prevent further revolt.<ref>{{cite book |last=Breunig |first=Charles |title=The Age of Revolution and Reaction, 1789–1850 |year=1977 |publisher=[[W. W. Norton & Company]] |location=New York, N.Y |isbn=0-393-09143-0 }}</ref> In Spain Ramón de la Sagra established the anarchist journal ''El Porvenir'' in La Coruña in 1845 which was inspired by Proudhon´s ideas.<ref name="britannica.com">{{cite web|url=http://www.britannica.com/EBchecked/topic/22753/anarchism/66525/Anarchism-in-Spain#ref539322|title=anarchism :: Anarchism in Spain|work=Encyclopædia Britannica|access-date=2014-04-24}}</ref> The [[Catalan people|Catalan]] politician [[Francesc Pi i Margall]] became the principal translator of Proudhon's works into Spanish<ref>[[George Woodcock]]. ''Anarchism: a history of libertarian movements''. Pg. 357</ref> and later briefly became president of Spain in 1873 while being the leader of the Democratic Republican Federal Party. According to George Woodcock "These translations were to have a profound and lasting effect on the development of Spanish anarchism after 1870, but before that time Proudhonian ideas, as interpreted by Pi, already provided much of the inspiration for the federalist movement which sprang up in the early 1860's."<ref>George Woodcock. ''Anarchism: a history of libertarian movements''. Pg. 357</ref> According to the ''[[Encyclopædia Britannica]]'' "During the Spanish revolution of 1873, Pi y Margall [[Cantonal Revolution|attempted to establish a decentralised, or "cantonalist," political system]] on Proudhonian lines."<ref name="britannica.com" />\r\n",
"\r\n",
"In 1864 the [[International Workingmen's Association]] (sometimes called the "First International") united diverse revolutionary currents including French followers of Proudhon,<ref>{{cite book | last = Blin | first = Arnaud | title = The History of Terrorism | publisher = University of California Press | location = Berkeley | year = 2007 | isbn = 0-520-24709-4 |page=116}}</ref> [[Blanquism|Blanquists]], [[Philadelphes]], English trade unionists, socialists and [[social democrats]]. Due to its links to active workers' movements, the International became a significant organisation. [[Karl Marx]] became a leading figure in the International and a member of its General Council. Proudhon's followers, the mutualists, opposed Marx's [[state socialism]], advocating political [[abstentionism]] and small property holdings.<ref>{{cite book | last = Dodson | first = Edward | title = The Discovery of First Principles: Volume 2 | publisher = Authorhouse | page=312 | year = 2002 | isbn = 0-595-24912-4 }}</ref><ref>{{cite book | last = Thomas | first = Paul | title = Karl Marx and the Anarchists | publisher = Routledge & Kegan Paul |ref=harv | location = London | year = 1985 | isbn = 0-7102-0685-2 |page=187}}</ref> Woodcock also reports that the American individualist anarchists [[Lysander Spooner]] and [[William B. Greene]] had been members of the [[First International]].<ref name="Woodcock">{{cite book|last=Woodcock|first=G.|authorlink=George Woodcock|title=Anarchism: A History of Libertarian Ideas and Movements|year=1962|publisher=Penguin|location=Melbourne|page=460}}</ref> In 1868, following their unsuccessful participation in the [[League of Peace and Freedom]] (LPF), Russian revolutionary Mikhail Bakunin and his collectivist anarchist associates joined the First International (which had decided not to get involved with the LPF).<ref>{{cite book | last = Thomas | first = Paul | title = Karl Marx and the Anarchists | publisher = Routledge and Kegan Paul | location = London | year = 1980 | isbn = 0-7102-0685-2 |page=304}}</ref> They allied themselves with the [[federalist]] socialist sections of the International,<ref>{{cite book | last = Bak | first = Jǹos | title = Liberty and Socialism | publisher = Rowman & Littlefield Publishers | location = Lanham | year = 1991 | isbn = 0-8476-7680-3 |page=236}}</ref> who advocated the revolutionary overthrow of the state and the collectivisation of property.\r\n",
"\r\n",
"At first, the collectivists worked with the Marxists to push the First International in a more revolutionary socialist direction. Subsequently, the International became polarised into two camps, with Marx and Bakunin as their respective figureheads.<ref>{{cite book | last = Engel | first = Barbara | title = Mothers and Daughters | publisher = Northwestern University Press | location = Evanston | year = 2000 | isbn = 0-8101-1740-1 |page=140}}</ref> [[Mikhail Bakunin]] characterised Marx's ideas as [[centralism|centralist]] and predicted that, if a Marxist party came to power, its leaders would simply take the place of the [[ruling class]] they had fought against.<ref name="bakuninmarx" /><ref>{{cite book |last=Bakunin|first=Mikhail|authorlink=Mikhail Bakunin|origyear=1873|year=1991|title=Statism and Anarchy |publisher=Cambridge University Press|isbn=0-521-36973-8}}</ref> Anarchist historian George Woodcock reports that "The annual Congress of the International had not taken place in 1870 owing to the outbreak of the Paris Commune, and in 1871 the General Council called only a special conference in London. One delegate was able to attend from Spain and none from Italy, while a technical excuse – that they had split away from the Fédération Romande – was used to avoid inviting Bakunin's Swiss supporters. Thus only a tiny minority of anarchists was present, and the General Council's resolutions passed almost unanimously. Most of them were clearly directed against Bakunin and his followers."<ref name="Anarchism 1962">[[George Woodcock]]. ''Anarchism: A History of Libertarian Ideas and Movements'' (1962)</ref> In 1872, the conflict climaxed with a final split between the two groups at the [[Hague Congress (1872)|Hague Congress]], where Bakunin and [[James Guillaume]] were expelled from the International and its headquarters were transferred to New York. In response, the federalist sections formed their own International at the [[Anarchist St. Imier International|St. Imier Congress]], adopting a revolutionary anarchist programme.<ref name=Graham-05>Graham, Robert '[http://www.blackrosebooks.net/anarism1.htm ''Anarchism''] (Montreal: Black Rose Books 2005) {{ISBN|1-55164-251-4}}.</ref>\r\n",
"\r\n",
"The [[Paris Commune]] was a government that briefly ruled Paris from 18 March (more formally, from 28 March) to 28 May 1871. The Commune was the result of an uprising in Paris after France was defeated in the Franco-Prussian War. Anarchists participated actively in the establishment of the Paris Commune. They included [[Louise Michel]], the Reclus brothers, and [[Eugene Varlin]] (the latter murdered in the repression afterwards). As for the reforms initiated by the Commune, such as the re-opening of workplaces as co-operatives, anarchists can see their ideas of associated labour beginning to be realised ... Moreover, the Commune's ideas on federation obviously reflected the influence of Proudhon on French radical ideas. Indeed, the Commune's vision of a communal France based on a federation of delegates bound by imperative mandates issued by their electors and subject to recall at any moment echoes Bakunin's and Proudhon's ideas (Proudhon, like Bakunin, had argued in favour of the "implementation of the binding mandate" in 1848 ... and for federation of communes). Thus both economically and politically the Paris Commune was heavily influenced by anarchist ideas.<ref>{{cite web|url=http://flag.blackened.net/revolt/anarchism/writers/anarcho/commune.html|title=The Paris Commune|work=blackened.net|access-date=2012-04-30|archive-url=https://web.archive.org/web/20120625232340/http://flag.blackened.net/revolt/anarchism/writers/anarcho/commune.html|archive-date=2012-06-25}}</ref> George Woodcock states:\r\n",
"{{quote|a notable contribution to the activities of the Commune and particularly to the organization of public services was made by members of various anarchist factions, including the mutualists Courbet, Longuet, and Vermorel, the libertarian collectivists Varlin, Malon, and Lefrangais, and the bakuninists Elie and [[Elisée Reclus]] and Louise Michel.<ref name="Anarchism 1962" />}}\r\n",
"\r\n",
"===Organised labour===\r\n",
"{{Main|Anarcho-syndicalism|International Workers' Association|Anarchism in Spain|Spanish Revolution}}\r\n",
"The anti-authoritarian sections of the First International were the precursors of the anarcho-syndicalists, seeking to "replace the privilege and authority of the State" with the "free and spontaneous organization of labour."<ref>Resolutions from the St. Imier Congress, in ''Anarchism: A Documentary History of Libertarian Ideas'', Vol. 1, p. 100 [http://www.blackrosebooks.net/anarism1.htm]</ref> In 1886, the [[Federation of Organized Trades and Labor Unions]] (FOTLU) of the United States and Canada unanimously set 1 May 1886, as the date by which the [[Eight-hour day|eight-hour work day]] would become standard.<ref name=foner />\r\n",
"[[File:ChicagoAnarchists.jpg|left|thumb|upright|A sympathetic engraving by [[Walter Crane]] of the executed "Anarchists of Chicago" after the [[Haymarket affair]]. The Haymarket affair is generally considered the most significant event for the origin of international May Day observances.]]\r\n",
"In response, unions across the United States prepared a [[general strike]] in support of the event.<ref name=foner /> On 3 May, in Chicago, a fight broke out when [[strikebreaker]]s attempted to cross the picket line, and two workers died when police opened fire upon the crowd.<ref>{{cite book |last=Avrich |first=Paul |title=[[The Haymarket Tragedy]] |year=1984 |publisher=Princeton University Press |location=Princeton |isbn=0-691-00600-8 |page=190}}</ref> The next day, 4 May, anarchists staged a rally at Chicago's Haymarket Square.<ref>{{cite book |last=Avrich |title=[[The Haymarket Tragedy]] |page=193 |isbn=0-691-04711-1 }}</ref> A bomb was thrown by an unknown party near the conclusion of the rally, killing an officer.<ref>{{cite web |url=http://www.odmp.org/officer/3972-patrolman-mathias-j.-degan |title=Patrolman Mathias J. Degan |access-date=19 January 2008 |publisher=The Officer Down Memorial Page, Inc | archiveurl= https://web.archive.org/web/20080118084649/http://www.odmp.org/officer/3972-patrolman-mathias-j.-degan| archivedate= 18 January 2008<!--Added by DASHBot-->}}</ref> In the ensuing panic, police opened fire on the crowd and each other.<ref>''[[Chicago Tribune]]'', 27 June 1886, quoted in {{cite book |last=Avrich |title=[[The Haymarket Tragedy]] |page=209 |isbn=0-691-04711-1 }}</ref> Seven police officers and at least four workers were killed.<ref name="the bomb">{{cite web |url=http://www.chicagohistory.org/dramas/act2/act2.htm |title=Act II: Let Your Tragedy Be Enacted Here |access-date=19 January 2008 |year=2000 |work=The Dramas of Haymarket |publisher=Chicago Historical Society | archiveurl= https://web.archive.org/web/20080115030929/http://www.chicagohistory.org/dramas/act2/act2.htm| archivedate= 15 January 2008<!--Added by DASHBot-->}}</ref> Eight anarchists directly and indirectly related to the organisers of the rally were arrested and charged with the murder of the deceased officer. The men became international political celebrities among the labour movement. Four of the men were executed and a fifth committed suicide prior to his own execution. The incident became known as the Haymarket affair, and was a setback for the labour movement and the struggle for the eight-hour day. In 1890 a second attempt, this time international in scope, to organise for the eight-hour day was made. The event also had the secondary purpose of memorialising workers killed as a result of the Haymarket affair.<ref>{{cite book |last=Foner |title=May Day |page=42 |isbn=0-7178-0624-3 }}</ref> Although it had initially been conceived as a once-off event, by the following year the celebration of [[International Workers' Day]] on May Day had become firmly established as an international worker's holiday.<ref name=foner>{{cite book | last = Foner | first = Philip Sheldon | title = May day: a short history of the international workers' holiday, 1886–1986 | publisher = International Publishers | location = New York | year = 1986 | isbn = 0-7178-0624-3 |page=56}}</ref>\r\n",
"\r\n",
"In 1907, the [[International Anarchist Congress of Amsterdam]] gathered delegates from 14 different countries, among which were important figures of the anarchist movement, including [[Errico Malatesta]], [[Pierre Monatte]], [[Luigi Fabbri]], [[Benoît Broutchoux]], [[Emma Goldman]], [[Rudolf Rocker]], and [[Christiaan Cornelissen]]. Various themes were treated during the Congress, in particular concerning the organisation of the anarchist movement, [[popular education]] issues, the general strike or [[antimilitarism]]. A central debate concerned the relation between anarchism and [[syndicalism]] (or [[trade union]]ism). Malatesta and Monatte were in particular disagreement themselves on this issue, as the latter thought that syndicalism was revolutionary and would create the conditions of a [[social revolution]], while Malatesta did not consider syndicalism by itself sufficient.<ref>[http://www.fondation-besnard.org/article.php3?id_article=225 Extract of Malatesta's declaration] {{webarchive |url=https://web.archive.org/web/20070928002329/http://www.fondation-besnard.org/article.php3?id_article=225 |date=September 28, 2007 }} {{fr icon}}</ref> He thought that the trade-union movement was [[reformism|reformist]] and even [[Conservatism|conservative]], citing as essentially bourgeois and anti-worker the phenomenon of professional union officials. Malatesta warned that the syndicalists aims were in perpetuating syndicalism itself, whereas anarchists must always have anarchy as their end and consequently refrain from committing to any particular method of achieving it.<ref>{{cite book | last = Skirda | first = Alexandre | title = [[Facing the Enemy|Facing the enemy: a history of anarchist organization from Proudhon to May 1968]] | publisher = A. K. Press| year = 2002 | isbn = 1-902593-19-7 |page=89 }}</ref>\r\n",
"\r\n",
"The [[Federación Anarquista Ibérica|Spanish Workers Federation]] in 1881 was the first major anarcho-syndicalist movement; anarchist trade union federations were of special importance in Spain. The most successful was the [[Confederación Nacional del Trabajo]] (National Confederation of Labour: CNT), founded in 1910. Before the 1940s, the CNT was the major force in Spanish working class politics, attracting 1.58 million members at one point and playing a major role in the [[Spanish Civil War]].<ref>{{Cite book\r\n",
" |last=Beevor\r\n",
" |first=Antony\r\n",
" |authorlink=Antony Beevor\r\n",
" |year=2006\r\n",
" |title=The Battle for Spain: The Spanish Civil War 1936–1939\r\n",
" |publisher=Weidenfeld & Nicolson\r\n",
" |location=London\r\n",
" |isbn=978-0-297-84832-5\r\n",
" |page=24\r\n",
" }}</ref> The CNT was affiliated with the [[International Workers Association]], a federation of anarcho-syndicalist trade unions founded in 1922, with delegates representing two million workers from 15 countries in Europe and Latin America. In Latin America in particular "The anarchists quickly became active in organising craft and industrial workers throughout South and Central America, and until the early 1920s most of the trade unions in [[Anarchism in Mexico|Mexico]], [[Anarchism in Brazil|Brazil]], Peru, Chile, and Argentina were anarcho-syndicalist in general outlook; the prestige of the Spanish C.N.T. as a revolutionary organisation was undoubtedly to a great extent responsible for this situation. The largest and most militant of these organisations was the [[Federación Obrera Regional Argentina]] ... it grew quickly to a membership of nearly a quarter of a million, which dwarfed the rival socialdemocratic unions."<ref name="Anarchism 1962" />\r\n",
"{{Clear}}\r\n",
"\r\n",
"===Propaganda of the deed and illegalism===\r\n",
"{{Main|Propaganda of the deed|Illegalism|Expropriative anarchism}}\r\n",
"[[File:Lugi Gallean2.jpg|right|thumb|upright|[[Italian-American]] anarchist [[Luigi Galleani]]. His followers, known as Galleanists, carried out a series of bombings and assassination attempts from 1914 to 1932 in what they saw as attacks on 'tyrants' and 'enemies of the people']]\r\n",
"Some anarchists, such as [[Johann Most]], advocated publicising violent acts of retaliation against counter-revolutionaries because "we preach not only action in and for itself, but also action as propaganda."<ref>{{cite web |url=http://dwardmac.pitzer.edu/Anarchist_Archives/bright/most/actionprop.html |title="Action as Propaganda" by Johann Most, 25 July 1885 |publisher=Dwardmac.pitzer.edu |date=21 April 2003 |access-date=20 September 2010|archive-url=https://web.archive.org/web/20110521211430/http://dwardmac.pitzer.edu/Anarchist_Archives/bright/most/actionprop.html |archive-date=2011-05-21 }}</ref> Scholars such as Beverly Gage contend that this was not advocacy of mass murder, but [[targeted killings]] of members of the [[ruling class]] at times when such actions might garner sympathy from the population, such as during periods of heightened government repression or labor conflicts where workers were killed.<ref>{{cite book|last=Gage|first=Beverly|title=The Day Wall Street Exploded: A Story of America in its First Age of Terror|location=New York|publisher=[[Oxford University Press]]|year=2009|ISBN=978-0199759286|page=48}}</ref> However, Most himself once boasted that "the existing system will be quickest and most radically overthrown by the annihilation of its exponents. Therefore, massacres of the enemies of the people must be set in motion."<ref>{{cite web |url=https://www.vice.com/en_us/article/kill-the-banker-0000525-v21n12|title=When Revolution Came to America|last=Ketcham |first=Christopher |date= December 16, 2014|website=[[Vice (magazine)|Vice]] |publisher= |access-date=April 8, 2017}}</ref> Most is best known for a pamphlet published in 1885: ''The Science of Revolutionary Warfare'', a how-to manual on the subject of making explosives, based on knowledge he acquired while working at an explosives plant in New Jersey.<ref>{{cite book |last=Most|first=Johann|date=1978 |title=Science of Revolutionary Warfare|url= |location= |publisher=Desert Publications|page=v|isbn=0879472111|author-link=Johann Most}}</ref>\r\n",
"\r\n",
"By the 1880s, people inside and outside the anarchist movement began to use the slogan, "propaganda of the deed" to refer to individual bombings, [[regicide]]s, and [[tyrannicide]]s. From 1905 onwards, the Russian counterparts of these anti-syndicalist anarchist-communists become partisans of economic terrorism and illegal '[[Confiscation|expropriations]]'."<ref>{{cite web|url=http://www.zabalaza.net/theory/txt_anok_comm_ap.htm |title="Anarchist-Communism" by Alain Pengam |access-date=2009-03-12 |deadurl=yes |archiveurl=https://web.archive.org/web/20090312022528/http://www.zabalaza.net/theory/txt_anok_comm_ap.htm |archivedate=2009-03-12 |df= }}</ref> [[Illegalism]] as a practice emerged and within it "The acts of the anarchist bombers and assassins ("[[propaganda by the deed]]") and the anarchist burglars ("[[individual reappropriation]]") expressed their desperation and their personal, violent rejection of an intolerable society. Moreover, they were clearly meant to be ''exemplary'' invitations to revolt.".<ref name="Imrie">{{cite web|url=http://recollectionbooks.com/siml/library/illegalistsDougImrie.htm |title=The Illegalists |first=Doug|last= Imrie |work=recollectionbooks.com |deadurl=yes |archiveurl=https://web.archive.org/web/20150908072801/http://recollectionbooks.com/siml/library/illegalistsDougImrie.htm |archivedate=2015-09-08 |access-date=2010-12-09 }}</ref> France's [[Bonnot Gang]] was the most famous group to embrace illegalism.\r\n",
"\r\n",
"However, as soon as 1887, important figures in the anarchist movement distanced themselves from such individual acts. Peter Kropotkin thus wrote that year in ''Le Révolté'' that "a structure based on centuries of history cannot be destroyed with a few kilos of dynamite".<ref>quoted in Billington, James H. 1998. ''Fire in the minds of men: origins of the revolutionary faith'' New Jersey: Transaction Books, p 417.</ref> A variety of anarchists advocated the abandonment of these sorts of tactics in favour of collective revolutionary action, for example through the trade union movement. The anarcho-syndicalist, [[Fernand Pelloutier]], argued in 1895 for renewed anarchist involvement in the labour movement on the basis that anarchism could do very well without "the individual dynamiter."<ref>{{cite web |url=http://blackrosebooks.net/anarism1.htm |title=Table Of Contents |publisher=Blackrosebooks.net |date= |access-date=20 September 2010}}</ref>\r\n",
"\r\n",
"[[Political repression|State repression]] (including the infamous 1894 French ''[[lois scélérates]]'') of the anarchist and [[labour movement]]s following the few successful bombings and assassinations may have contributed to the abandonment of these kinds of tactics, although reciprocally state repression, in the first place, may have played a role in these isolated acts. The dismemberment of the French [[socialist movement]], into many groups and, following the suppression of the 1871 Paris Commune, the execution and exile of many ''[[communards]]'' to [[penal colonies]], favoured individualist political expression and acts.<ref>Historian [[Benedict Anderson]] thus writes: <blockquote> "In March 1871 the Commune took power in the abandoned city and held it for two months. Then [[Adolphe Thiers|Versailles]] seized the moment to attack and, in one horrifying week, executed roughly 20,000 Communards or suspected sympathizers, a number higher than those killed in the recent war or during [[Robespierre]]'s '[[Reign of Terror|Terror]]' of 1793–1794. More than 7,500 were jailed or deported to places like New Caledonia. Thousands of others fled to Belgium, England, Italy, Spain and the United States. In 1872, stringent laws were passed that ruled out all possibilities of organising on the left. Not till 1880 was there a general amnesty for exiled and imprisoned Communards. Meanwhile, the Third Republic found itself strong enough to renew and reinforce [[Napoleon III of France|Louis Napoleon]]'s imperialist expansion – in Indochina, Africa, and Oceania. Many of France's leading intellectuals and artists had participated in the Commune ([[Gustave Courbet|Courbet]] was its quasi-minister of culture, [[Arthur Rimbaud|Rimbaud]] and [[Pissarro]] were active propagandists) or were sympathetic to it. The ferocious repression of 1871 and thereafter, was probably the key factor in alienating these milieux from the Third Republic and stirring their sympathy for its victims at home and abroad." {{Cite journal | last = Anderson | first = Benedict | author-link = Benedict Anderson | title = In the World-Shadow of Bismarck and Nobel | journal = [[New Left Review]] | volume = II | issue = 28 | pages = 85–129 | publisher = New Left Review | date = July–August 2004 | url = http://newleftreview.org/II/28/benedict-anderson-in-the-world-shadow-of-bismarck-and-nobel | ref = harv | postscript = .| access-date = 2016-01-07 | archive-url = https://web.archive.org/web/20151219130121/http://newleftreview.org/II/28/benedict-anderson-in-the-world-shadow-of-bismarck-and-nobel | archive-date = 2015-12-19 }} </blockquote> According to some analysts, in [[History of Germany since 1945|post-war Germany]], the prohibition of the [[Communist Party of Germany|Communist Party]] (KDP) and thus of institutional far-left political organization may also, in the same manner, have played a role in the creation of the [[Red Army Faction]].</ref>\r\n",
"\r\n",
"Numerous heads of state were assassinated between 1881 and 1914 by members of the anarchist movement, including Tsar [[Alexander II of Russia]], President [[Marie François Sadi Carnot|Sadi Carnot]] of France, [[Empress Elisabeth of Austria]], King [[Umberto I of Italy]], President [[William McKinley]] of the United States, King [[Carlos I of Portugal]] and King [[George I of Greece]]. McKinley's assassin [[Leon Czolgosz]] claimed to have been influenced by anarchist and [[feminist]] Emma Goldman.<ref>{{cite web|title = American Experience {{!}} Emma Goldman {{!}} Transcript {{!}} PBS|url = https://www.pbs.org/wgbh/amex/goldman/filmmore/pt.html|website = www.pbs.org|access-date = 2016-01-12}}</ref>\r\n",
"\r\n",
"===Russian Revolution and other uprisings of the 1910s===\r\n",
"{{Main|Anarchism in Russia|Russian Revolution (1917)||Revolutionary Insurrectionary Army of Ukraine|Revolutions of 1917–23}}\r\n",
"[[File:Makhno group.jpg|thumb|left|[[Nestor Makhno]] with members of the anarchist [[Revolutionary Insurrectionary Army of Ukraine]]]]\r\n",
"Anarchists participated alongside the [[Bolshevik]]s in both [[February Revolution|February]] and [[October Revolution|October revolutions]], and were initially enthusiastic about the Bolshevik revolution.<ref>{{cite book | last = Dirlik | first = Arif | title = Anarchism in the Chinese Revolution | publisher = University of California Press | location = Berkeley | year = 1991 | isbn = 0-520-07297-9 }}</ref> However, following a political falling out with the Bolsheviks by the anarchists and other left-wing opposition, the conflict culminated in the 1921 [[Kronstadt rebellion]], which the new government repressed. Anarchists in central Russia were either imprisoned, driven underground or joined the victorious Bolsheviks; the anarchists from Petrograd and Moscow fled to [[Ukraine]].<ref>{{cite book | last = Avrich | first = Paul | title = [[The Russian Anarchists]] | publisher = AK Press | location = Stirling | year = 2006 | isbn = 1-904859-48-8 |page=204}}</ref> There, in the [[Free Territory (Ukraine)|Free Territory]], they fought in the [[Russian Civil War|civil war]] against the [[White movement|Whites]] (a grouping of monarchists and other opponents of the October Revolution) and then the Bolsheviks as part of the Revolutionary Insurrectionary Army of Ukraine led by Nestor Makhno, who established an anarchist society in the region for a number of months.\r\n",
"\r\n",
"\r\n",
"bzip2: I/O or other error, bailing out. Possible reason follows.\r\n",
"bzip2: Broken pipe\r\n",
"\tInput file = enwiki-latest-pages-articles1.xml-p10p30302.bz2, output file = (stdout)\r\n"
]
}
],
"source": [
"# inspecting the file on the command line (only works on Mac and Linux)\n",
"\n",
"local_wiki='enwiki-latest-pages-articles1.xml-p10p30302.bz2'\n",
"! bzip2 -d --stdout $local_wiki |head -200"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import pandas as pd\n",
"import numpy as np\n",
"import seaborn as sns\n",
"\n",
"import re\n",
"from lxml import etree \n",
"from bz2file import BZ2File\n",
"import codecs\n",
"from tqdm import tqdm_notebook "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Install a conda package in the current Jupyter kernel\n",
"import sys\n",
"#!conda install --yes --prefix {sys.prefix} bz2file"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(7,\n",
" [\""ANARCHISM, a social philosophy that rejects authoritarian government and maintains that voluntary institutions are best suited to express man's natural social tendencies." George Woodcock. "Anarchism" at The Encyclopedia of Philosophy\",\n",
" '"In a society developed on these lines, the voluntary associations which already now begin to cover all the fields of human activity would take a still greater extension so as to substitute themselves for the state in all its functions." [http://www.theanarchistlibrary.org/HTML/Petr_Kropotkin___Anarchism__from_the_Encyclopaedia_Britannica.html Peter Kropotkin. "Anarchism" from the Encyclopædia Britannica]',\n",
" '"Anarchism." The Shorter Routledge Encyclopedia of Philosophy. 2005. p. 14 "Anarchism is the view that a society without the state, or government, is both possible and desirable."',\n",
" 'Sheehan, Sean. Anarchism, London: Reaktion Books Ltd., 2004. p. 85',\n",
" \""as many anarchists have stressed, it is not government as such that they find objectionable, but the hierarchical forms of government associated with the nation state." Judith Suissa. ''Anarchism and Education: a Philosophical Perspective''. Routledge. New York. 2006. p. 7\",\n",
" '"That is why Anarchy, when it works to destroy authority in all its aspects, when it demands the abrogation of laws and the abolition of the mechanism that serves to impose them, when it refuses all hierarchical organisation and preaches free agreement — at the same time strives to maintain and enlarge the precious kernel of social customs without which no human or animal society can exist." [[Peter Kropotkin]]. [http://www.theanarchistlibrary.org/HTML/Petr_Kropotkin__Anarchism__its_philosophy_and_ideal.html Anarchism: its philosophy and ideal]',\n",
" '"anarchists are opposed to irrational (e.g., illegitimate) authority, in other words, hierarchy — hierarchy being the institutionalisation of authority within a society." [http://www.theanarchistlibrary.org/HTML/The_Anarchist_FAQ_Editorial_Collective__An_Anarchist_FAQ__03_17_.html#toc2 "B.1 Why are anarchists against authority and hierarchy?"] in [[An Anarchist FAQ]]'])"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"# The regex which extracts the references (depending on the string encoding)\n",
"reference_extractor=r'(.*?)' # r'<ref>(.*?)</ref>'\n",
"\n",
"\n",
"sample='''\n",
"Anarchism is a [[political philosophy]] that advocates [[self-governance|self-governed]] societies with voluntary institutions. These are often described as [[stateless society|stateless societies]],<ref>"ANARCHISM, a social philosophy that rejects authoritarian government and maintains that voluntary institutions are best suited to express man's natural social tendencies." George Woodcock. "Anarchism" at The Encyclopedia of Philosophy</ref><ref>"In a society developed on these lines, the voluntary associations which already now begin to cover all the fields of human activity would take a still greater extension so as to substitute themselves for the state in all its functions." [http://www.theanarchistlibrary.org/HTML/Petr_Kropotkin___Anarchism__from_the_Encyclopaedia_Britannica.html Peter Kropotkin. "Anarchism" from the Encyclopædia Britannica]</ref><ref>"Anarchism." The Shorter Routledge Encyclopedia of Philosophy. 2005. p. 14 "Anarchism is the view that a society without the state, or government, is both possible and desirable."</ref><ref>Sheehan, Sean. Anarchism, London: Reaktion Books Ltd., 2004. p. 85</ref> but several authors have defined them more specifically as institutions based on non-[[Hierarchy|hierarchical]] [[Free association (communism and anarchism)|free associations]].<ref>"as many anarchists have stressed, it is not government as such that they find objectionable, but the hierarchical forms of government associated with the nation state." Judith Suissa. ''Anarchism and Education: a Philosophical Perspective''. Routledge. New York. 2006. p. 7</ref><ref name="iaf-ifa.org"/><ref>"That is why Anarchy, when it works to destroy authority in all its aspects, when it demands the abrogation of laws and the abolition of the mechanism that serves to impose them, when it refuses all hierarchical organisation and preaches free agreement — at the same time strives to maintain and enlarge the precious kernel of social customs without which no human or animal society can exist." [[Peter Kropotkin]]. [http://www.theanarchistlibrary.org/HTML/Petr_Kropotkin__Anarchism__its_philosophy_and_ideal.html Anarchism: its philosophy and ideal]</ref><ref>"anarchists are opposed to irrational (e.g., illegitimate) authority, in other words, hierarchy — hierarchy being the institutionalisation of authority within a society." [http://www.theanarchistlibrary.org/HTML/The_Anarchist_FAQ_Editorial_Collective__An_Anarchist_FAQ__03_17_.html#toc2 "B.1 Why are anarchists against authority and hierarchy?"] in [[An Anarchist FAQ]]</ref> Anarchism holds the [[state (polity)|state]] to be undesirable, unnecessary, or harmful.<ref name="definition">\n",
"{{cite journal |last=Malatesta|first=Errico|title=Towards Anarchism|journal=MAN!|publisher=International Group of San Francisco|location=Los Angeles|oclc=3930443|url=http://www.marxists.org/archive/malatesta/1930s/xx/toanarchy.htm|archiveurl=https://web.archive.org/web/20121107221404/http://marxists.org/archive/malatesta/1930s/xx/toanarchy.htm|archivedate=7 November 2012 |deadurl=no|authorlink=Errico Malatesta |ref=harv}}\n",
"{{cite journal |url=http://www.theglobeandmail.com/servlet/story/RTGAM.20070514.wxlanarchist14/BNStory/lifeWork/home/\n",
"|archiveurl=https://web.archive.org/web/20070516094548/http://www.theglobeandmail.com/servlet/story/RTGAM.20070514.wxlanarchist14/BNStory/lifeWork/home |archivedate=16 May 2007 |deadurl=yes |title=Working for The Man |journal=[[The Globe and Mail]] |accessdate=14 April 2008 |last=Agrell |first=Siri |date=14 May 2007 |ref=harv }}\n",
"'''\n",
"\n",
"\n",
"sum ( 1 for _ in re.finditer(r'<ref>(.*?)</ref>', sample)), re.findall(r'<ref>(.*?)</ref>',sample)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# from http://stackoverflow.com/questions/12160418/why-is-lxml-etree-iterparse-eating-up-all-my-memory\n",
"# We use the ideas in this function in our own iter function:\n",
" # clear the context element\n",
" # remove all now-empty references from the root node to the contxt element\n",
"\n",
"def fast_iter(context, func, *args, **kwargs):\n",
" \"\"\"\n",
" http://lxml.de/parsing.html#modifying-the-tree\n",
" Based on Liza Daly's fast_iter\n",
" http://www.ibm.com/developerworks/xml/library/x-hiperfparse/\n",
" See also http://effbot.org/zone/element-iterparse.htm\n",
" \"\"\"\n",
" for event, elem in context:\n",
" func(elem, *args, **kwargs)\n",
" # It's safe to call clear() here because no descendants will be\n",
" # accessed\n",
" elem.clear()\n",
" # Also eliminate now-empty references from the root node to elem\n",
" for ancestor in elem.xpath('ancestor-or-self::*'):\n",
" while ancestor.getprevious() is not None:\n",
" del ancestor.getparent()[0]\n",
" del context"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# We use the pageid http://dbpedia.org/ontology/wikiPageID as key value for the page\n",
"# Note the use of the (default) namespace all the time.\n",
"\n",
"def count_references(f,inset): # ,outfile):\n",
" '''With f a wikipedia dump and inset a list of pageids, count for each page in the dump \\cap inset the number of references\n",
" on the page and return as a dict with pageid:reference_count key-value pairs.'''\n",
" with BZ2File(f) as xml_file:\n",
" context = etree.iterparse(xml_file, tag= '{http://www.mediawiki.org/xml/export-0.10/}page')\n",
" pages_references_dict={} \n",
" # use this when you want to write away the results to file\n",
" #with codecs.open(outfile,'w', encoding='utf-8') as f: \n",
" #f.write('WikiPageID;Number_of_references\\n')\n",
" \n",
" c=0\n",
" for _, elem in tqdm_notebook(context):\n",
" #title= elem.findtext('{http://www.mediawiki.org/xml/export-0.10/}title') \n",
" page_id = elem.findtext('{http://www.mediawiki.org/xml/export-0.10/}id')\n",
" try: \n",
" page_id= int(page_id)\n",
" if page_id in inset: # we only do this for pages in inset\n",
" pagetext=elem.findtext('{http://www.mediawiki.org/xml/export-0.10/}revision/{http://www.mediawiki.org/xml/export-0.10/}text')\n",
" ref_count= sum ( 1 for _ in re.finditer(reference_extractor, pagetext))\n",
" #if page_id and ref_count: # just store those with a count > 0\n",
" pages_references_dict[page_id]= ref_count \n",
" # use this when you want to write away the results to file\n",
" #f.write(str(page_id)+';'+str(ref_count)+'\\n')\n",
" except:\n",
" True\n",
" \n",
" # now get rid of the element and also delete all its ancestors \n",
" # from http://stackoverflow.com/questions/12160418/why-is-lxml-etree-iterparse-eating-up-all-my-memory\n",
" elem.clear()\n",
" # Also eliminate now-empty references from the root node to elem\n",
" for ancestor in elem.xpath('ancestor-or-self::*'):\n",
" while ancestor.getprevious() is not None:\n",
" del ancestor.getparent()[0] \n",
" # for debugging and seeing how far the code is already\n",
" c+=1\n",
" if c% 10**3==0:\n",
" print(c) # \n",
" #break #\n",
" del context \n",
" return pages_references_dict\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"# do the counting and save dataframe as pickle\n",
"\n",
"def Count_References():\n",
" # wikidump\n",
" inputfile= local_wiki\n",
" #output='en_ref_count.csv'\n",
" # our list of wikipageid of our persons\n",
" #wikiPageID= pd.read_pickle('../wikiPageID.pkl')\n",
" wikiPageID_set= set(range(10**8)) # set(wikiPageID.values) # we just take all\n",
" # do the counting\n",
" en_ref_count= count_references(inputfile,wikiPageID_set)# ,output)\n",
" # turn into a dataframe and pickle\n",
" ef= pd.DataFrame.from_dict(en_ref_count, orient='index')\n",
" #ef=pd.Series(en_ref_count) # nicer\n",
" ef.columns=['Number_of_references']\n",
" ef.to_pickle('en_ref_count.pkl')\n",
" return True\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/admin/anaconda/envs/python3.6/lib/python3.6/site-packages/ipykernel_launcher.py:15: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0\n",
"Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`\n",
" from ipykernel import kernelapp as app\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2c70abb965544e828da996f48eddba67",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(IntProgress(value=1, bar_style='info', max=1), HTML(value='')))"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"1000\n",
"2000\n",
"3000\n",
"4000\n",
"5000\n",
"6000\n",
"7000\n",
"8000\n",
"9000\n",
"10000\n",
"11000\n",
"12000\n",
"13000\n",
"14000\n",
"15000\n",
"16000\n",
"17000\n",
"18000\n",
"19000\n",
"\n",
"CPU times: user 48.8 s, sys: 14.1 s, total: 1min 2s\n",
"Wall time: 1min 5s\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%time Count_References()"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"10000\n",
"peak memory: 8153.01 MiB, increment: 836.18 MiB\n"
]
}
],
"source": [
"# memory usage See Chapter 1 of the handbook\n",
"\n",
"%load_ext memory_profiler\n",
"%memit Count_References()"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"10000\n",
" "
]
}
],
"source": [
"%prun Count_References()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"