codigofuerte / geonames-mysql-dataimport Goto Github PK
View Code? Open in Web Editor NEWShell Script to download and import geonames.org dumps into a mysql database.
Home Page: http://codigofuerte.github.io/GeoNames-MySQL-DataImport/
Shell Script to download and import geonames.org dumps into a mysql database.
Home Page: http://codigofuerte.github.io/GeoNames-MySQL-DataImport/
The cursor blinks after the "Loading allCountries.txt" is shown twice, and that's it. Nothing else appears to be happening.
What is the expected output I should see after ./geonames_importer.sh -a import-dumps is run?
Should the process take minutes, or hours?
Hi again,
It would be great if we can specify a 'data-dir' (or somenthing like that) to keep downloaded files away from the files provieded in -tgz file.
Thanks in advance.
The footer on the project site (says: "Hosted on GitHub Pages using the Dinky theme") is overlaying the bottom two buttons, so the links for TAR and GitHub aren't clickable.
Look at this:
http://download.geonames.org/export/dump/admin1CodesASCII.txt
code length for GR.% for example, is major than 6
in my db i set char(15)
Thank you for your HUGE work!
when it try to download allCountries.zip script failed with error:
./geonames_importer.sh: line 52: (-4): substring expression < 0
wget version:
GNU Wget 1.17 built on darwin15.3.0.
GeoNames also provides a dump of postal codes here http://download.geonames.org/export/zip/ would would be nice to include in this.
When run under the latest version of MySQL (8.0.12) (and perhaps others) import-dumps
fails.
However, adding the following to the top of geonames_import_data.sql
seems to fix it,
though I do not think this solution is good practice (with respect to security).
SET GLOBAL local_infile = 1;
...
/Applications/MAMP/Library/bin/ is needed in front of the mysql command in geonames_importer.sh or there should be instructions to make sure there is an alias to mysql that is set properly.
I'm not one to bitch about be free code on github, but why they hell do you have indexes on every frkn column?
All the data downloaded in the script is in utf8mb4 and the creation of the tables and columns must be in this codification . Replace all uftf8 with utf8mb4 and it will work.
you might consider to adding the following here https://github.com/codigofuerte/GeoNames-MySQL-DataImport/blob/master/geonames_importer.sh#L189
*)
RED='\033[0;31m'
NC='\033[0m' # No Color
printf "${RED}Invalid ACTION! \n Please choose one of \n${NC}"
usage
exit 127
this will help n00b users like me understand why the program says everything okay but not having end results
It's present in the data downloaded from geonames, importer doesn't import it.
Hi guys. I'm trying to adapt this to update existing data.
Since the process is time expensive (downloading / importing), truncating and reimporting would result in DB downtime.
Geonames exposes their changes throught a modifications.txt file:
modifications-.txt : all records modified on the previous day, the date is in yyyy-MM-dd format. You can use this file to daily synchronize your own geonames database.
I guess we could run a a cronjob that fetches that file and runs the modifications.
Any idea how to tackle this?
The schema creation scripts correctly create the tables using the utf8 character set. However there is no specific collation order set, which means that collation would default to utf8_general_ci. However the general_ci incorrectly orders many non-English character sets - see https://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci for a discussion.
The solution is to declare a collation in the table DDL: COLLATE utf8_unicode_ci
. With this collation, the ordering is correct.
Hi,
I't would be nice if you add '-c' option to wget command to ensure we only download the portion we don't already have in our local filesystem.
From 'man wget':
-c
--continue
Continue getting a partially-downloaded file. This is useful when you want to finish up a download started by a previous instance of Wget, or by another program.
Thanks in advance.
Keep the good job on this software.
Thanks for this project! Would you kindly add a license to your code? As Geonames is licensed under the CC4.0 Attribution License I suggest this as an adequate license.
Hi!
I am proposing this instead of making a PR because I am not so good with shell programming to make the change.
But I'd be sweet to be able to add prefixes to tables. Just for grouping and keeping tables organized in a big legacy database.
I just downloaded ES.txt and try to import it in a geoname table. There was many :"Incorrect integer value: '' for column 'elevation' at row 1 "
It seems at first glance that Either the database is corrupted, or the sql schema should accept a string. Or we should change the import to finetune that field and accept '' leads to NULL
./geonames_importer.sh: line 125: /geonames_db_struct.sql: No such file or directory [FAILED]
I get this error when running ./geonames_importer.sh -a create-db. To fix it, I just edit the geonames_importer.sh file and add a new line below the commented out dir name.
dir=pwd
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.