opencog / miner Goto Github PK

8.0 8.0 14.0 77 MB

Frequent and surprising subhypergraph pattern miner for the AtomSpace.

License: Other

CMake 2.80% C++ 45.36% Scheme 51.61% GDB 0.03% Shell 0.21%

miner's Introduction

OpenCog

This repo is no longer maintained! Please use the following, instead:

Obsolete! As of 2021, the most interesting and actively maintained parts of this git repo have been split off into their own distinct git repos. What is left here is a mish-mash of unmatained stuff that is in the process of bit-rotting. Some unit tests fail. Some unit tests won't run. Some code won't compile. Perhaps there's some good stuff in here. Perhaps it can be brought back to life and used for something or other. However... for the most part, it is obsolete.

This git repository contains the "OpenCog Framework", which has served as a (scientific, technical) laboratory for researching, exploring and learning how to integrate AI algorithms and systems into humanoid robotic systems. Most of the activity within this particular repo has focused on integrating natural language chat, common-sense reasoning, assorted learning algorithms, and motor control of humanoid robots.

A stated goal of the OpenCog project is to develop artificial general intelligence (AGI) systems. This is all and well; however, what can be found here, in this particular repo, is very far from that. The code here really is ... a laboratory for integrating various types of AI systems. As such, it is a compilation of several decades of work by a large and varying collection of students, researchers, professors and software engineers. As a laboratory, it is filled with all sorts of devices in varying states of working order, from well-polished to mostly-broken.

Overview

Most of the basic components used in OpenCog are distributed across various git repos, (mostly) grouped under https://github.com/opencog

This git repository contains a crude natural language processing pipeline, several embodied chatbots, and some control/action-selection mechanisms. These include:

Ghost, a Chatscript-compatible chatbot with additional capabilities for accepting visual sensory input, and for controlling robot movements.
OpenPsi, a model of psychological states. Its currently a mashup of two unrelated ideas: a generic rule-class action-selection and planning system, and a model of human psychological states. An open to-do item is to untangle these two.
An assortment of natural language processing subsystems, including:
- Natural language generation (for expressing thoughts as sentences).
- Natural language input (for reading and hearing).
- Relex2logic, converting natural language to logic expressions.
- Assorted chatbots, some of which are embodied.
- A Lojban tool.

Prerequisites

To build and run the system here, the packages listed below are required. Users of Ubuntu may use the dependency installer from the /opencog/octool repository. Docker containers with OpenCog preconfigured can be found in the opencog/docker repo.

cogutil

Common OpenCog C++ utilities. https://github.com/opencog/cogutil It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

atomspace

OpenCog Atomspace, a sophisticated (hyper-)graph database. https://github.com/opencog/atomspace It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

cogserver

OpenCog CogServer Network Server. https://github.com/opencog/cogserver It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

attention

OpenCog Attention Allocation subsystem. https://github.com/opencog/attention It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

URE

OpenCog Unified Rule Engine. https://github.com/opencog/ure Required for PLN It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

pln

OpenCog Probabilistic Logic Networks reasoning system. https://github.com/opencog/pln It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

spacetime

OpenCog Spacetime Server - locations of objects in space and time. https://github.com/opencog/spacetime It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

ros-behavior-scripting

Visual and auditory senses, robot motor control. https://github.com/opencog/ros-behavior-scripting It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

lg-atomese

Natural Language Parser for English, Russian, other languages. Required for natural language generation, and the chatbot. https://github.com/opencog/lg-atomese It uses exactly the same build procedure as this package. Be sure to sudo make install at the end.

Building OpenCog

Perform the following steps at the shell prompt:

    cd to project root dir
    mkdir build
    cd build
    cmake ..
    make

Libraries will be built into subdirectories within build, mirroring the structure of the source directory root.

Unit tests

To build and run the unit tests, from the ./build directory enter (after building opencog as above):

    make test

miner's People

Contributors

Stargazers

Watchers

Forkers

ngeiswei mhatta singnet stjordanis leungmanhin linas ntoxeg kasimebrahim baerxxl rustyshackleforth superfluouss

miner's Issues

MinerUTest fails 3 of 51 tests

gcc --version: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

This is the output of cmake ..

-- Build type: Release
-- CogUtil version 2.0.3 found.
-- AtomSpace found.
-- URE found.
-- Boost version 106900 found.
-- Guile (2.2.3 >= 2.2.2) was found.
-- Valgrind Prefix: 
-- VALGRIND was found.
-- VALGRIND devel headers found.

Building for Ubuntu 18.04.5 LTS

The following components will be built:
-----------------------------------------------
   Miner  - URE based hypergraph pattern miner.

-- Configuring done
-- Generating done

build works 100%. Atomspace, opencog, cogutil, ure, other repo.. works

Output of miner make test:

[ 38%] Built target miner
[ 50%] Built target guile-miner
[ 61%] Built target minerutestutils
[ 77%] Built target SurprisingnessUTest
[ 94%] Built target MinerUTest
[ 94%] Built target tests
[100%] Running tests...
Test project /home/michele/OpenCog_ALL/miner/build/tests
    Start 1: MinerUTest
1/2 Test #1: MinerUTest .......................***Failed    4.97 sec
Running cxxtest tests (51 tests)[INFO] BEGIN TEST: test_partitions
.[INFO] BEGIN TEST: test_is_blk_syntax_more_abstract_1
.[INFO] BEGIN TEST: test_is_blk_syntax_more_abstract_2
.[INFO] BEGIN TEST: test_is_blk_syntax_more_abstract_3
.[INFO] BEGIN TEST: test_is_pat_syntax_more_abstract
.[INFO] BEGIN TEST: test_is_pat_more_abstract_1
.[INFO] BEGIN TEST: test_is_pat_more_abstract_2
.[INFO] BEGIN TEST: test_is_pat_more_abstract_3
.[INFO] BEGIN TEST: test_is_more_abstract_foreach_var
.[INFO] BEGIN TEST: test_remove_useless_clauses_1
.[INFO] BEGIN TEST: test_remove_useless_clauses_2
.[INFO] BEGIN TEST: test_remove_useless_clauses_3
.[INFO] BEGIN TEST: test_compose_1
.[INFO] BEGIN TEST: test_compose_2
.[INFO] BEGIN TEST: test_compose_3
.[INFO] BEGIN TEST: test_compose_4
.[INFO] BEGIN TEST: test_expand_conjunction_disconnect
.[INFO] BEGIN TEST: test_expand_conjunction_1
.[INFO] BEGIN TEST: test_expand_conjunction_2
.[INFO] BEGIN TEST: test_expand_conjunction_3
.[INFO] BEGIN TEST: test_expand_conjunction_4
.[INFO] BEGIN TEST: test_shallow_abstract
.[INFO] BEGIN TEST: test_empty
.[INFO] BEGIN TEST: test_A
.[INFO] BEGIN TEST: test_AB
.[INFO] BEGIN TEST: test_AB_redundant_cnj
.[INFO] BEGIN TEST: test_AB_AC
.[INFO] BEGIN TEST: test_AB_AC_BC
.[INFO] BEGIN TEST: test_AB_ABC
.[INFO] BEGIN TEST: test_ABCD
.[INFO] BEGIN TEST: test_ABAB
.[INFO] BEGIN TEST: test_AAAA
.[INFO] BEGIN TEST: test_glob

In MinerUTest::test_glob:
/home/michele/OpenCog_ALL/miner/build/tests/miner/../../../tests/miner/MinerUTest.cxxtest:1345: Error: Assertion failed: content_eq(ure_results, ure_expected)
[INFO] BEGIN TEST: test_transitivity
.[INFO] BEGIN TEST: test_long_transitivity
.[INFO] BEGIN TEST: test_no_transitivity
.[INFO] BEGIN TEST: test_evaluation
.[INFO] BEGIN TEST: test_variable_factorization
.[INFO] BEGIN TEST: test_type_support_1

In MinerUTest::test_type_support_1:
/home/michele/OpenCog_ALL/miner/build/tests/miner/../../../tests/miner/MinerUTest.cxxtest:1544: Error: Assertion failed: content_eq(ure_results, ure_expected)
[INFO] BEGIN TEST: test_type_support_2
In MinerUTest::test_type_support_2:
/home/michele/OpenCog_ALL/miner/build/tests/miner/../../../tests/miner/MinerUTest.cxxtest:1581: Error: Assertion failed: content_eq(ure_results, ure_expected)
[INFO] BEGIN TEST: test_typed_glob
.[INFO] BEGIN TEST: test_2conjuncts_1
.[INFO] BEGIN TEST: test_2conjuncts_2
.[INFO] BEGIN TEST: test_2conjuncts_3
.[INFO] BEGIN TEST: test_2conjuncts_4
.[INFO] BEGIN TEST: test_2conjuncts_5
.[INFO] BEGIN TEST: test_2conjuncts_6
.[INFO] BEGIN TEST: test_InferenceControl
.[INFO] BEGIN TEST: test_SodaDrinker
.[INFO] BEGIN TEST: test_SodaDrinker_incremental
.[INFO] BEGIN TEST: test_vqa
.
Failed 3 and Skipped 0 of 51 tests
Success rate: 94%

Errors in file MinerUTest.cxxtest at lines 1345-1544-1581.
The method content_eq(ure_results, ure_expected) in file HandleTree.cc seems not to work in those 3 tests but I tested the output both by hand and using the method itself (I add it into the MinerUTest file and test using it) and so they work.

is there any error in my opencog component installations?

Store frequency + other attributes in patterns of the pattern miner

Currently the new pattern miner does not keep any intermediary information leading to its discovery such as its frequency, eventually its surprisingness. This should be kept track of, either by attaching Values to the atom corresponding to the pattern, or explicitly via EvaluationLink in the atomspace.

Signal6 issue with pattern miner

Today i've tried to launch firstly simple example from the site (https://wiki.opencog.org/w/Pattern_miner) and on the line "opencog> loadmodule opencog/learning/PatternMiner/libPatternMinerAgent.so" i've received following output in server's terminal window:

"Listening on port 17001
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;; or pass the --no-auto-compile argument to disable.
;;; compiling /home/cics/OpenCog/opencog/build/../opencog/learning/PatternMiner/ugly_male_soda-drinker_corpus.scm
;;; compiled /home/cics/.cache/guile/ccache/2.0-LE-8-2.0/home/cics/OpenCog/opencog/opencog/learning/PatternMiner/ugly_male_soda-drinker_corpus.scm.go
terminate called after throwing an instance of 'opencog::AssertionException'
what(): (/home/cics/OpenCog/opencog/opencog/learning/PatternMiner/Parameters.cc:47)
[2018-01-18 10:37:35:266] [ERROR] Caught signal 6 (Aborted) on thread 139683083409472
Stack Trace:
2: basic_string.h:539 ~basic_string()
3: CogServerMain.cc:81 _Z7sighandi()
4: [0x36cb0] ??() ??:0
5: raise.c:56 __GI_raise()
6: abort.c:91 __GI_abort()
7: [0x60535] ??() ??:0
8: [0x5e6d6] ??() ??:0
9: [0x5e703] ??() ??:0
10: [0x5e922] ??() ??:0
11: oc_assert.cc:59 opencog::cassert(char const*, bool)
12: Parameters.cc:47 opencog::PatternMining::Parameters::reSetAllSettingsFromConfig()
13: PatternMiner.cc:2682 opencog::PatternMining::PatternMiner::PatternMiner(opencog::AtomSpace&)
14: PatternMinerAgent.cc:66 opencog::PatternMinerAgent::PatternMinerAgent(opencog::CogServer&)
15: Factory.h:71 opencog::Factory<opencog::PatternMinerAgent, opencog::Agent>::create(opencog::CogServer&) const
16: Registry.h:104 opencog::Registryopencog::Agent::create(opencog::CogServer&, std::string const&)
17: shared_ptr_base.h:448 __shared_countopencog::Agent*()
18: shared_ptr_base.h:545 ~__shared_count()
19: CogServer.cc:481 opencog::CogServer::loadModule(std::string const&)
20: LoadModuleRequest.cc:55 opencog::LoadModuleRequest::execute()
21: CogServer.cc:261 opencog::CogServer::processRequests()
22: CogServer.cc:224 opencog::CogServer::runLoopStep()
23: CogServer.cc:199 opencog::CogServer::serverLoop()
24: CogServerMain.cc:139 main()
25: libc-start.c:321 __libc_start_main()
26: [0x4070fc] ??() ??:0"

Also tried to do similar thing using python's scheme_eval. Like this:

"from opencog.atomspace import AtomSpace, TruthValue
from opencog.atomspace import types
from opencog.type_constructors import *
from opencog.bindlink import satisfying_set, bindlink
from opencog.utilities import initialize_opencog
from opencog.scheme_wrapper import load_scm, scheme_eval, scheme_eval_h

a = AtomSpace()
initialize_opencog(a)
set_type_ctor_atomspace(a)
TV = TruthValue(1, 1)
data = ["opencog/atomspace/core_types.scm",
"opencog/scm/utilities.scm"]
for item in data:
load_scm(a, item)
scheme_eval(a, "(add-to-load-path "/usr/local/share/opencog/scm")")
scheme_eval(a,"(use-modules (opencog) (opencog nlp) (opencog nlp chatbot) (opencog nlp relex2logic) (opencog atom-types))")
scheme_eval(a,"(use-modules (opencog patternminer))")
scheme_eval(a, "(clear)")
scheme_eval (a,'(load "../opencog/learning/PatternMiner/testdatasmall.scm")')"

and got same signal 6 problem on "scheme_eval(a,"(use-modules (opencog patternminer))")" line.
"Process finished with exit code 134 (interrupted by signal 6: SIGABRT)"

If anyone could help with that i would really appreciate that.
Have a nice day!

Optimize pattern frequency calculation

Problem

Currently, pattern frequency (required for calculating the empirical probability during surprisingness evaluation) is calculated by enumerating all its matches and dividing by the universe count. Such enumeration is costly, especially in RAM. On a real world dataset, such as used in

https://github.com/opencog/miner/tree/master/examples/miner/mozi-ai

https://github.com/ngeiswei/reasoning-bio-as-xp

it easily maxes out 32GB of RAM. This has been improved by subsampling/bootstrapping the dataset based on an estimate of the empirical probability. Such estimate can be very wrong though, leading to under or over subsampling, thus innacurracies or RAM explosions.

Solutions

Improve the subsampling/bootstrapping mechanism, maybe auto-tuned via binary search, etc.
Introduce a dedicated pattern matcher callback that takes less memory, maybe only saving the atom hashes rather than the atoms themselves, or maybe saving nothing at all but still somehow guarantying not to recount matches.

Pattern Miner considering too many groundings

MinerUtils::restricted_satisfying_set should only check whether the given text matches or not, rather than considering subhypergraphs, as well as permutations due to unordered links, which is contrary to how empirical probability is calculated.