From 3e57a1902f7bf6884662fb2aca403e13787c2d26 Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Tue, 7 Dec 2021 15:30:18 -0800 Subject: Update README with logging info --- README.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 575bb05..53061b6 100644 --- a/README.md +++ b/README.md @@ -33,8 +33,11 @@ access) are also supported with the use of the `--username` and `--password` flags. More detailed information, such as the site used and full URL of the file, can -be displayed with `-v` or `--verbose`. Use `-vv` to display even more detail. -`-q` can be used to silence warnings. +be displayed with `-v` or `--verbose`. Use `-vv` to display even more detail, +mainly debugging information or API messages. `-q` can be used to silence warnings. +A logfile can be specified with `-l` or `--logfile`. If this option is present, the +logfile will contain the same information as `-v` along with timestamps. New log +entries will be appended to an existing logfile. By default, the program won't overwrite existing files with the same name as the target, but this can be forced with `-f` or `--force`. Additionally, the file @@ -55,6 +58,7 @@ wikiget https://en.wikipedia.org/wiki/File:Example.jpg -o test.jpg ## Future plans +- download multiple files at once in batch mode - continue batch download even if input is malformed or file doesn't exist (possibly by raising exceptions in `download()`) - batch download by (Commons) category or user uploads @@ -62,7 +66,7 @@ wikiget https://en.wikipedia.org/wiki/File:Example.jpg -o test.jpg ## Contributing -Pull requests or bug reports are more than welcome. +Pull requests, bug reports, or feature requests are more than welcome. It's recommended that you use a [virtual environment manager](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/) -- cgit v1.2.3 From 80633b5e864cf5705a4636dd05601531a5c33a45 Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Tue, 26 Sep 2023 13:31:48 -0700 Subject: Update link in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'README.md') diff --git a/README.md b/README.md index a0ccf8d..029aaf7 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ source venv/bin/activate ``` Then run `pip install -e .` to invoke an -["editable" install](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs), +["editable" install](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs) meaning any changes made to the source will be reflected immediately in the executable script. Unit tests can be run with `pytest` (make sure to run `pip install pytest` in the virtual environment first.) -- cgit v1.2.3 From b6fac1b7c0962e48a8f708efc9f535bb8552c9c6 Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Tue, 3 Oct 2023 09:11:10 -0700 Subject: Update README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'README.md') diff --git a/README.md b/README.md index fd4d464..67fe0a2 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ Requires Python 3.7+. Get it with `pip install --user wikiget` or `pipx install ## Usage `wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-p PATH] [--username USERNAME] -[--password PASSWORD] [-o OUTPUT | -a] FILE` +[--password PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] FILE` If `FILE` is in the form `File:Example.jpg` or `Image:Example.jpg`, it will be fetched from the default site, which is "commons.wikimedia.org". If it's the -- cgit v1.2.3 From 485df31f095a9b629a1dcc04af13956325856d8c Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Tue, 3 Oct 2023 09:51:58 -0700 Subject: Update README and do some code cleanup --- README.md | 102 ++++++++++++++++++++++++++------------------------------------ 1 file changed, 42 insertions(+), 60 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 67fe0a2..876c319 100644 --- a/README.md +++ b/README.md @@ -3,49 +3,39 @@ [![Python package](https://github.com/clpo13/wikiget/actions/workflows/python.yml/badge.svg?branch=master)](https://github.com/clpo13/wikiget/actions/workflows/python.yml) [![PyPI version](https://badge.fury.io/py/wikiget.svg)](https://badge.fury.io/py/wikiget) -Something like wget for downloading a file from MediaWiki sites (like Wikipedia -or Wikimedia Commons) using only the file name or the URL of its description -page. +Something like wget for downloading a file from MediaWiki sites (like Wikipedia or Wikimedia Commons) using only the +file name or the URL of its description page. Requires Python 3.7+. Get it with `pip install --user wikiget` or `pipx install wikiget`. ## Usage -`wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-p PATH] [--username USERNAME] -[--password PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] FILE` - -If `FILE` is in the form `File:Example.jpg` or `Image:Example.jpg`, it will be -fetched from the default site, which is "commons.wikimedia.org". If it's the -fully-qualified URL of a file description page, like -`https://en.wikipedia.org/wiki/File:Example.jpg`, the file is fetched from the -specified site, in this case "en.wikipedia.org". Full URLs may contain -characters your shell interprets differently, so you can either escape those -characters with a backslash `\` or surround the entire URL with single `'` or -double `"` quotes. Use of a fully-qualified URL like this may require setting -the `--path` flag (see next paragraph). - -The site can also be specified with the `--site` flag, though this will not have -any effect if the full URL is given. Non-Wikimedia sites should work, but you -may need to specify the wiki's script path with `--path` (where `index.php` and -`api.php` live; on Wikimedia sites it's `/w/`, but other sites may use `/` or -something else entirely). Private wikis (those requiring login even for read -access) are also supported with the use of the `--username` and `--password` -flags. - -More detailed information, such as the site used and full URL of the file, can -be displayed with `-v` or `--verbose`. Use `-vv` to display even more detail, -mainly debugging information or API messages. `-q` can be used to silence warnings. -A logfile can be specified with `-l` or `--logfile`. If this option is present, the -logfile will contain the same information as `-v` along with timestamps. New log -entries will be appended to an existing logfile. - -By default, the program won't overwrite existing files with the same name as the -target, but this can be forced with `-f` or `--force`. Additionally, the file -can be downloaded to a different name with `-o`. - -Files can be batch downloaded with the `-a` or `--batch` flag. In this mode, -`FILE` will be treated as an input file containing multiple files to download, -one filename or URL per line. If an error is encountered, execution stops +`wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-p PATH] [--username USERNAME] [--password PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] FILE` + +The only required parameter is `FILE`, which is the file you want to download. It can either be the name of the file on +the wiki, including the namespace prefix, or a link to the file description page. If `FILE` is in the form +`File:Example.jpg` or `Image:Example.jpg`, it will be fetched from the default site, which is "commons.wikimedia.org". +If it's the fully-qualified URL of a file description page, like `https://en.wikipedia.org/wiki/File:Example.jpg`, the +file is fetched from the site in the URL, in this case "en.wikipedia.org". Note: full URLs may contain characters your +shell interprets differently, so you can either escape those characters with a backslash `\` or surround the entire URL +with single `'` or double `"` quotes. Use of a fully-qualified URL like this may require setting the `--path` flag (see +next paragraph). + +The site can also be specified with the `--site` flag, though this will not have any effect if the full URL is given. +Non-Wikimedia sites should work, but you may need to specify the wiki's script path with `--path` (where `index.php` and +`api.php` live; on Wikimedia sites it's `/w/`, but other sites may use `/` or something else entirely). Private wikis +(those requiring login even for read access) are also supported with the use of the `--username` and `--password` flags. + +More detailed information, such as the site used and full URL of the file, can be displayed with `-v` or `--verbose`. +Use `-vv` to display even more detail, mainly debugging information or API messages. `-q` can be used to silence +warnings. A logfile can be specified with `-l` or `--logfile`. If this option is present, the logfile will contain the +same information as `-v` along with timestamps. New log entries will be appended to an existing logfile. + +By default, the program won't overwrite existing files with the same name as the target, but this can be forced with +`-f` or `--force`. Additionally, the file can be downloaded to a different name with `-o`. + +Files can be batch downloaded with the `-a` or `--batch` flag. In this mode, `FILE` will be treated as an input file +containing multiple files to download, one filename or URL per line. If an error is encountered, execution stops immediately and the offending filename is printed. ### Example usage @@ -70,13 +60,11 @@ Pull requests, bug reports, or feature requests are more than welcome. It's recommended that you use a [virtual environment manager](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/) -like venv or [virtualenv](https://virtualenv.pypa.io/en/latest/) to create an -isolated environment in which to install this package's dependencies so as not -to clutter your system Python environment: +like venv or [virtualenv](https://virtualenv.pypa.io/en/latest/) to create an isolated environment in which to install +this package's dependencies so as not to clutter your system Python environment: ```bash -# if you plan on submitting pull requests, fork the repo on GitHub -# and clone that instead +# if you plan on submitting pull requests, fork the repo on GitHub and clone that instead git clone https://github.com/clpo13/wikiget cd wikiget @@ -97,28 +85,22 @@ source venv/bin/activate Then run `pip install -e .` to invoke an ["editable" install](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs) -meaning any changes made to the source will be reflected immediately in the executable -script. Unit tests can be run with `pytest` (make sure to run `pip install pytest` -in the virtual environment first.) +meaning any changes made to the source will be reflected immediately in the executable script. Unit tests can be run +with `pytest` (make sure to run `pip install pytest` in the virtual environment first.) -Alternatively, using [Hatch](https://hatch.pypa.io/latest/), simply clone the repository -and run `hatch run test` to create the environment and run pytest. Also try `hatch shell` -or `hatch run wikiget --help`. +Alternatively, using [Hatch](https://hatch.pypa.io/latest/), simply clone the repository and run `hatch run test` to +create the environment and run pytest. Also try `hatch shell` or `hatch run wikiget --help`. ## License Copyright (C) 2018-2023 Cody Logan and contributors -This program is free software: you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. +This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public +License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later +version. -This program is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. +This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied +warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. -You should have received a copy of the GNU General Public License -along with this program (see [LICENSE](LICENSE)). If not, see -. +You should have received a copy of the GNU General Public License along with this program (see [LICENSE](LICENSE)). +If not, see . -- cgit v1.2.3 From e18222daecca1656390652cbd1c7f6985080241a Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Tue, 3 Oct 2023 09:58:37 -0700 Subject: Add short user and pass options Swapped path short option from -p to -P and added -u for username and -p for password --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'README.md') diff --git a/README.md b/README.md index 876c319..9bf9250 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Requires Python 3.7+. Get it with `pip install --user wikiget` or `pipx install ## Usage -`wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-p PATH] [--username USERNAME] [--password PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] FILE` +`wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-P PATH] [-u USERNAME] [-p PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] FILE` The only required parameter is `FILE`, which is the file you want to download. It can either be the name of the file on the wiki, including the namespace prefix, or a link to the file description page. If `FILE` is in the form -- cgit v1.2.3 From 0f45fe2526d7be48d77ef60b7505a196d533b4f4 Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Thu, 5 Oct 2023 15:27:46 -0700 Subject: Add new options to readme and man page --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 9bf9250..4faadf4 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Requires Python 3.7+. Get it with `pip install --user wikiget` or `pipx install ## Usage -`wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-P PATH] [-u USERNAME] [-p PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] FILE` +`wikiget [-h] [-V] [-q | -v] [-f] [-s SITE] [-P PATH] [-u USERNAME] [-p PASSWORD] [-o OUTPUT | -a] [-l LOGFILE] [-j THREADS] FILE` The only required parameter is `FILE`, which is the file you want to download. It can either be the name of the file on the wiki, including the namespace prefix, or a link to the file description page. If `FILE` is in the form @@ -36,7 +36,10 @@ By default, the program won't overwrite existing files with the same name as the Files can be batch downloaded with the `-a` or `--batch` flag. In this mode, `FILE` will be treated as an input file containing multiple files to download, one filename or URL per line. If an error is encountered, execution stops -immediately and the offending filename is printed. +immediately and the offending filename is printed. For large batches, the process can be sped up by downloading files +in parallel. The number of parallel downloads can be set with `-j`. For instance, with `-a -j4`, wikiget will download +four files at once. Without `-j` or with `-j` by itself without a number, wikiget will download the files one at a +time. ### Example usage @@ -48,7 +51,6 @@ wikiget https://en.wikipedia.org/wiki/File:Example.jpg -o test.jpg ## Future plans -- download multiple files at once in batch mode - continue batch download even if input is malformed or file doesn't exist (possibly by raising exceptions in `download()`) - batch download by (Commons) category or user uploads -- cgit v1.2.3 From 87052196874cc1bf82f70a6f5aa8e6df59bc1537 Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Fri, 13 Oct 2023 11:13:31 -0700 Subject: Revise batch file parsing to ignore blank and commented lines Previously, blank lines would cause an error and lines prepended with "#" would be downloaded like any other, assuming they were valid. Now, "#" can be used to mark ignored files or comments. --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) (limited to 'README.md') diff --git a/README.md b/README.md index 4faadf4..eecea90 100644 --- a/README.md +++ b/README.md @@ -35,11 +35,11 @@ By default, the program won't overwrite existing files with the same name as the `-f` or `--force`. Additionally, the file can be downloaded to a different name with `-o`. Files can be batch downloaded with the `-a` or `--batch` flag. In this mode, `FILE` will be treated as an input file -containing multiple files to download, one filename or URL per line. If an error is encountered, execution stops -immediately and the offending filename is printed. For large batches, the process can be sped up by downloading files -in parallel. The number of parallel downloads can be set with `-j`. For instance, with `-a -j4`, wikiget will download -four files at once. Without `-j` or with `-j` by itself without a number, wikiget will download the files one at a -time. +containing multiple files to download, one filename or URL per line. Blank lines and lines starting with "#" are +ignored. If an error is encountered, execution stops immediately and the offending filename is printed. For large +batches, the process can be sped up by downloading files in parallel. The number of parallel downloads can be set with +`-j`. For instance, with `-a -j4`, wikiget will download four files at once. Without `-j` or with `-j` by itself without +a number, wikiget will download the files one at a time. ### Example usage -- cgit v1.2.3 From 45a550899e0adf6958764d8a5133da4e21aa7fea Mon Sep 17 00:00:00 2001 From: Cody Logan Date: Fri, 13 Oct 2023 14:14:12 -0700 Subject: Add note to README about dev branch --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'README.md') diff --git a/README.md b/README.md index eecea90..bd8ece2 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,9 @@ Something like wget for downloading a file from MediaWiki sites (like Wikipedia or Wikimedia Commons) using only the file name or the URL of its description page. -Requires Python 3.7+. Get it with `pip install --user wikiget` or `pipx install wikiget`. +Requires Python 3.7+ and pip. Get it with `pip install wikiget` or `pipx install wikiget`. For the latest features, at +the risk of bugs and undocumented behavior, you can try the dev branch: +`pip install https://github.com/clpo13/wikiget/archive/refs/heads/dev.zip` ## Usage -- cgit v1.2.3