purpose -

to filter the input csv data as per the provided arguments

filter types

• invalid data
• all words
• specific topic
• specific alphabet (which will be initial of the english word)
• TBD (more can be added)

about invalid data filter

the invalid data filter is to be run always before running any other filter, as it eliminates those data elements which have insufficient or invalid data

• insufficient data - any of the english or marathi word is missing

• invalid data - if english word contains non english characters (this can be thought of later, and is low priority right now), same for marathi word.

IMP

• make separate functions for each filter type
• as per the passed argument of the "filter type" call the relavant function

steps

1. take passed csv data and filter type as argument
2. generate a truncated csv data structure as per the target filter

• call the specific filter function here

3. return this truncated data to the calling function

from marathi-shabd.

Hey so when you say "insufficient data" do you mean missing English or Marathi words exclusively, or does it also include missing examples and tags?

from marathi-shabd.

only the main 2 words. 1 en and 1 mr.

from marathi-shabd.

Thanks. Also could you explain what the "all words" filter is supposed to do?

from marathi-shabd.

Thanks. Also could you explain what the "all words" filter is supposed to do?

"All words" basically means no filtering (other than the invalid/insufficient data, of course).

from marathi-shabd.

So I would just call the invalid/insufficient data scripts when the filter type is "All words"?

from marathi-shabd.

Yes. Pretty much.

from marathi-shabd.

Also, the filter by topic function will require the topic as an argument. Do you want me to add an optional topic argument to the main filter function?

from marathi-shabd.

Yes you can do it in whichever way that makes the functions easy to use and also reusable.

What I've written in the gen-out.py file is just a basic example.
You can fill in the details and missing gaps.

from marathi-shabd.

pending issues from PR #32

among this priority ones are the # 1 and # 3

from marathi-shabd.